An Augmented Knitting Machine for Operational Assistance and Guided Improvisation

Within Interactive Fabrication, methods for locating a user interface directly on the fabrication hardware, such as via projections or Augmented Reality headsets, have been proposed as a way of helping users understand a fabrication machine’s output in real-world dimensions [25, 48] and situate irregular materials directly into the context of fabrication [7, 38], and as a way of exploring context-specific CAD paradigms that do not translate as well to a screen [50]. Research in this area has particularly target unique fabrication techniques like formless heat-molding [44] for which a traditional CAD pipeline would be inadequate, as well as techniques with deep histories of expert use, such as woodcarving [67] and machining with a lathe [60].

We are inspired by the range of this work as well as its recognition of the strengths of hands-on, experiential fabrication. We similarly focus on a somewhat unusual domain and hope to highlight the possibilities for other fabrication paradigms that may be overlooked in HCI. Our work specifically proposes drawing on an existing, mature fabrication technology, with implications for the vast number of other such technologies currently in use.

In this work, we are exploring adding computational capabilities to a not-otherwise-computerized machine. This allows us to draw on a wealth of existing fabrication technologies and communities of practice, but integrating with existing mechanisms – especially complex ones, like knitting machines – can require strategic attention. Augmented reality research has long shown how helpful information can be overlaid onto physical surfaces [64], boosting users’ ability to navigate complex, domain-specific, and critical tasks [54]. Like our work, the “Drill Sergeant” [52] and “Adroid” [59] systems augment a manual fabrication task with real-time guidance. Our system broadens the focus from helping a user complete a specific task to encouraging the user’s understanding and confident agency with the underlying machine. As such, we foreground interpretation of the recent past and current state of the machine alongside suggestions of potential near-futures as flexible modules for a variety of modes of use.

Machine knitting has received attention recently in HCI, with research predominantly focused on fully automatic computational knitting, which has been used as a tool to generate databases of knit material properties [26], and to create complex technical materials for wearable [34, 40] and/or robotic contexts [3, 4],[35]. Design tools targeting fully automatic computational knitting have been studied in HCI as well as within the computer graphics research community, and approaches span from low-level compilation [42] and generating patterns from 3D models [47],[29, 45] to knit design interfaces [31] drawing on sewing pattern notations to simplify user interaction.

While our system does support outputting a record of the user’s knitting via the Knitout file format that was initially developed as a target for general-purpose knitting compilation [41], the needs of our system diverge greatly from these tools in several major aspects: 1) we needed to encapsulate and present to the user a very different set of knittability constraints than fully automatic machines support; 2) the output of our system is primarily not the Knitout record, but rather guidance to the human user; 3) our system is intended to be used interactively as knitting progresses. A more closely related work is the eLoominate system, which is a purpose-build peg knitting loom – a type of jig often used for teaching hand-knitting – with LED indicators to guide users through simple two-color patterns designed in advance on an accompanying application [22].

Less-automatic machines have primarily been explored on the hardware side. All Yarns Are Beautiful (AYAB) [10] is an open-source project which documents an Arduino-based replacement controller for the 1980’s Brother ElectroKnit series of computerized home knitting machines – these machines are manually operated (non-motorized), but the computer controls specific patterning (typically used for two-color “pixel art”-style patterning). Another open-source project, OpenKnit [20], seeks to make it possible for a hobbyist to build a knitting machine from off-the-shelf and 3D printed parts. Depending on the build, an OpenKnit machine can be fully manual or mostly automatic. (OpenKnit is now a hardware startup, Kniterate [21], which makes fully-automatic machines.) AYAB and OpenKnit are both long-running projects with active communities (as of this writing, the OpenKnit Instructable has over 111,000 views [19], and the AYAB discussion group on Ravelry has 362 members [17]); other, smaller projects include small-run specialty tools for automating color changes [56] and repeating patterns across the width of a knit [36]. These hobbyist-led innovations have supported interactive art [51] as well as experimental architecture research [11]. Together, these projects highlight both a community interest in manual machine knitting as well as opportunities for creative practice.

We chose lightweight methods to capture the machine’s settings at a given time. In designing the sensing method, we prioritized flexible approaches which would be straightforward to deploy and could be modified to suit other similar machines. Additionally, because we are augmenting an existing vintage machine, which is itself a lovely and complete artifact, we made only reversible changes to the machine.

To capture the racking position, we mounted a simple 3-axis accelerometer (GY-61 ADXL335) to the racking lever at the side of the machine (Figure 6, left). To sense carriage position, we mounted Hall effect sensors at the left and right sides of the machine to be triggered by magnets attached to the carriage. These sensors are mounted on rails with binder clips and are positioned to be just outside the knitting area for a given task (e.g. for a narrower fabric, they can be brought closer to the center of the machine). We sense the left and right positions separately to support “leaving one position but not yet arriving at the other” as an input gesture. We use an Arduino to debounce these hardware sensor inputs and send change event notifications over USB serial.

Because the cam switches are mechanically complex and somewhat numerous, we decided against hardware sensing for their positions. Instead, we used computer vision: we mounted two webcams to the bow of the carriage (Figure 6, right), with one each pointed to the front and back carriages (Figure 5). During system use, a Processing sketch captures data within calibrated crop areas of the webcams.

For image classification, we used TensorFlow.js [1] with a separate model for each of the four switch sets (front leftward and rightward, and back leftward and rightward). Each model has ten classes: one for each of nine setting combinations (as listed in Table 1), plus one for “hands visible in the image,” to minimize updating the switch position display while the knitter is in the middle of adjusting a switch. We captured approximately 320 images of each switch set position group (e.g. “tuck on all needles, for back bed rightward rows”) using a second Processing sketch to manage the webcams and organize the data for each class. During image capture, we stored webcam input slightly outside the calibrated crop areas so that we could later augment our image data with randomly-chosen sub-crops at the final image size. This process took approximately a half hour. We manually sorted out images with hands visible into a separate “hand” class for each switch set, then augmented the approximately 250 images remaining in each other class with randomized crops, blurring, and image contrast to a total of approximately 1600 images per knob set. Using a basic Keras model on a personal computer with an RTX 3070 GPU, we trained a three layer convolutional neural network with 1.6 million parameters. Training took thirty seconds per model, and reported 99.32% accuracy when reserving 20% of input images as validation data. We did not fine-tune our approach beyond what was suggested in an online tutorial [58], suggesting that comparable results do not require particular machine learning expertise.

We coordinate these sensors with a server written in Node which accepts the carriage and rack change events from the Arduino as well as image data from the Processing webcam sketch, uses Tensorflow.js to classify the image data, and passes machine state events to the frontend user interface over a websocket.

We use a computational model of knitting machine state to track knitting progress. This model includes carriage position, yarn carriers, bed rack position, and a graph representation of the knit fabric being formed; together, these factors capture the current and past state of knitting, and can be used to project possible futures. These are particularly useful in visualizing the recent past (that is, the knitting which is currently hidden between the needle beds) and generating warnings about undesired next moves.

In addition to tracking direct operator actions, our underlying machine model retains a low-level history in the Knitout knitting machine operation language spec [41], allowing knit structures to be “replayed” on any Knitout-compatible computer-controlled knitting machine (Figure 7). This could allow a knitter to design interactively, then use an automated knitting machine to create multiple duplicates, or to knit at a different stitch size. On top of the basic needle-by-needle abstraction of Knitout, we model the carriage cam settings and needle types (“high” or “low”). Lastly, we maintain both 1) committed machine states, representing operations the knitter has already taken, and 2) potential machine states, representing possible futures given changes in the machine settings.

To communicate machine state to the knitter, our front end system comprises several visualization modules which are written as interoperable JavaScript classes. (These can also can be used as input devices themselves – while not part of the main on-machine interface scope of this work, this capability does allow a user to practice knitting virtually.)

We render carriage, rack, and yarn settings diagrammatically, with textual labels for the switch settings, Figure 8. When the user changes a cam or rack setting on the physical machine, this view is automatically updated to indicated the current settings. We render the machine as a simplified needle bed, with the needles aligned according to the current rack position and a symbol on each needle showing which operation would be applied at that needle if a row were made with the current settings.

The in-progress knitting is visualized using a mass-spring simulation, with the back bed yarn connections shaded slightly darker than the front. We abstract the stitch connections in the fabric into simple nodes and edges, instead of showing a literal yarn path, for readability. (We chose the mass-spring simulation for its particular suitability in showing how columns of stitches deflect in the “racked” patterning we highlight in section 6.3.) In this view, the rows that have already been knit are displayed in a yarn color, and future row predictions/suggestions are tinted yellow.

We also created a sequential panel representation of our “pattern rows” notation. Each panel shows the cam and rack settings, carriage direction, and yarn carrier needed to reproduce a particular row. When displayed as part of live instruction set, each panel highlights the changes the knitter would need to make to follow that instruction.

These machine state and instruction panel views form the visual basis of the patterning interface modules we describe in Section 6.

Because we model the fabric being formed and its relation to the machine, we can add error checking for problematic operations. For example, in Figure 8, the interface shows an error that “Needles on back bed would have 3 loops!” Needles can only hold so many loops, so when additional loops are added by successive tucking operations without intervening knit operations, the knitter runs a risk of overloading the needle, leading to dropped or torn loops. Because we model each needle with a reference to the loops in the

The underlying model can provide warning for certain conditions in either the committed or projected machine states. For example, the needle bed model can detect when many consecutive “tuck” operations are performed at the same needle – needles can only hold so many loops, and this can interfere with proper fabric takedown.

First, we created a view that provides an interpretation for the knitter of the interconnected machine settings and their effects on the next rows to be knit. In this view, the diagram of the cam and rack settings is shown live alongside a simulation of the existing fabric and a preview of what the next two rows of knitting would look like with the current settings. The cam settings are labeled with the name of their position (“all”/“some”/“none” and “knit”/“kt”/“tuck”), and the diagrammatic view of the needles displays the operations as the would occur in the next pass at the current settings. When the knitter changes a cam or rack setting, these views update accordingly. The fabric display shows the recent rows that are still hidden from physical view behind the machine beds. Lastly, this module displays error checking messages to warn the knitter about potentially risky operations they have performed or would perform. This module therefore collects and displays information about the recent past, present, and potential near future states of the machine and fabric, giving the knitter information but not imposing any particular guidance.

Our second module is intended to help a knitter produce a specific outcome. We focused on producing fabrics with two-layer “tubular knit” areas, which could be used as open pockets or as closed regions to contain other materials. This knitting style requires the user to plan the locations of High and Low needles, and to change cam settings at the beginning and end of the pocket section. If the user wants to knit a pocket which is open on one side of the knit, they will additionally need to switch cam settings every other row, even within the pocket section.

Our “Pockets” module provides a simple sketch-like interface to plan pocket locations, Figure 10. During knitting, it shows the knitter’s progress through the plan and provides row-by-row cam setting guidance.

In Figure 11, we showcase an advantage of manual knitting machines over industrial knitting: a greater range of possibilities for integrating additional materials into the knit. (This would be dangerous and difficult with a high-gauge, fast-moving, delicate industrial machine.) In particular, items slimmer than the gap between the beds (6mm at knit time, which can be temporarily increased to 12mm while knitting is paused) can be embedded in the fabric by designing a closed pocket and inserting the object just before the end of pocket knitting.

Our third module targets open-ended exploration with a greater depth of complexity than the first module. Because of the necessary presence of the knitter, manually-operated machine knitting presents a great opportunity for real-time creativity. However, the effects of particular cam setting choices can take a few rows to become clear, and a beginner may not have much basis for understanding their range of options. With the additional complication that recently-knit rows aren’t even visible to the knitter yet, the knitter might not have enough information to make improvisational choices.

To show how the knitter’s understanding of complex possible outcomes could be supported, we produced an interface module which generates and simulates a set of “path options” for the knitter to consider pursuing. Each path generates its instructions using its own sequence generation algorithm, and it is displayed as a sliding sequence of instruction panels alongside a fabric simulation with the hypothetical stitches that would be generated by that path highlighted in yellow. As with the Pockets interface, instruction panels show a live view of which settings the knitter needs to change to pursue that instruction. As knitting progresses, the set of path options is updated accordingly. Paths whose “next step” corresponded to the action just taken by the knitter are advanced to show the following step; paths which did not include that action are recomputed starting from the new step.

The path options module is written to be flexible and extensible with respect to which generative algorithms are used. We implemented three:

We aimed to study 1) basic usability: whether participants could understand the annotations and use them to reason about machine operation; 2) improvisational usability: whether the system sufficiently scaffolded real-time decision-making; and 3) overall participant attitudes toward hand fabrication, computational mediation, and improvisational practice, both in their own work and as they experienced these aspects of our system. The first two questions are assessments of our specific technical system, while the last question relates to the broader possibilities for augmented manual machines and exploratory use of interactive fabrication.

To avoid biasing the results on basic usability, we recruited participants with no machine knitting background, and no or minimal hand-knitting experience. For safety reasons, and to mitigate novelty effects from interacting with computational creation overall, we required experience in other computational production systems: six had 3D printed and/or laser-cut, and the remaining one has used computational systems for creative image generation. In order to meet these qualifications, and in accordance with covid-related limitations on visitors, we recruited participants within our department, or family members of department members, who were not textiles researchers. Our participants ranged in age from 20 to approximately 40.

For each session: after asking the user to practice moving the carriage, we introduced the basic “interpretation” view (Figure 8) and gave a verbal explanation of the carriage settings. The user was encouraged to interact with the settings and knit as many new rows as they liked until they were “ready to learn another capability,” at which point we introduced the racking lever. The user was given the option to view a swatch of several “named” patterns (rib, tube, cardigan, half-milano, and a mock interlock structure) along with a paper printout of instructions for how to knit each. Finally, we introduced the “suggested paths” view (Figure 12) and again encouraged each user to interact for as long as they liked with the system. In all, users spent approximately an hour each interacting with the system. After this, we conducted a semi-structured interview with each user, focusing on their experience of the system, how it compared to past fabrication experiences (both computational and manual), and their creative decision-making throughout their process. While our interview was semi-structured, we asked each participant at least the following questions:

We recorded the audio from our interviews, photographs of knit artifacts, and time-stamped system logs. The system log data includes all user actions perceived by the system, such as changing a cam or rack setting and moving the carriage. (Note that this data is messy, because it is not debounced e.g. to remove moments when the classification system mis-categorized a cam setting – because categorization is done many times per second and is generally accurate, these only appear as brief flickers to a user, but would be recorded as “changes” in the system.)

To assess basic usability, we viewed the artifacts and system logs. In the artifacts, we starting by looking for egregious knitting errors of the type which our error-checking (Section 5.4) was designed to help avoid. We found that all participants did successfully avoid these errors. While we did not deliberately include a comparison case with error-checking turned off, several of our participants discovered another, comparably predictable common error which we had not built checks for. These participants each encountered the same problem multiple times, suggesting that it was a difficult problem to avoid without tool assistance.

We found that users made many more one-off setting changes within the first part of their knitting, including much more changing of cam/rack settings without moving the carriage, to see the effect of these without committing it to the knit. This implies a process of initially gaining literacy with the system.

To assess improvisational usability and participants attitudes about computationally-mediated hand fabrication, we analyzed the interviews. One author performed a reflexive thematic analysis [12] by segmenting the interview transcripts and producing first highlights, then initial codes in a spreadsheet, then performing an iterative bottom-up coding. Because our questions largely centered on the experience of fabrication, our analysis is constructionist. We organize our observations of participant experiences and attitudes into themes in the following subsections.

Our participants were novices to both knitting in general and machine knitting in particular. Most participants described their learning as initially undirected, and they expressed that the system made it possible to manipulate the machine without needing to first form a complete understanding of its operation. Indeed, participants described being able to operate the machine before understanding much at all: “This isn’t something I’d typically do and it’s nice to have something like this where I can just kind of jump in and I am very confused about a lot of things but eventually I will pick it up. With the help of the computer [...] I get a more intuitive sense as to what is happening under the surface as opposed to needing to be explained every little part of what’s happening.” (P6)

Similarly, P3 mentioned an initial period of knitting to get accustomed to the machine, before branching out: “It took me a little bit to get comfortable just going back and forth, but once I started being able to see what was happening, it was like, ’Oh, I can change stuff up.”’

Depending on their goals, a knitter might find these modules to have too low of a learning ceiling. P2, who was mostly interested in gaining and refining a mental model of how the machine worked, expressed concern that they might not truly be learning and summarized their interaction with the Paths module as “Well, I’m kind of just following instructions.” (We discuss this possible negative outcome in section 8.3.)

However, other participants balanced their priorities between gaining a deeper understanding and generating an interesting artifact (in P1’s words: “I don’t really like to feel like I’m making garbage”). P6 enjoyed the system because “it was nice to see that I could put something together relatively easily and have some sort of guidance [...] and actually make something that looks like it was designed with purpose and intention,” implicitly regardless of whether it was their own purpose and intention.

Participants touched on feelings of “stress” (P1), “confidence” (P1, P7), “trust” (P1, P2, P6), and “self-reliance” (P6) to describe how they viewed their relationship to the system over the course of their session. P1 described the interpretation assistance as a kind of “re-assurance” and an “encouragement.”

In relation to how they thought of fully-automatic systems, they remarked on the relative power and also responsibility of hybrid interactive systems. Despite the usual premise that fully-automated systems aim to be reliable and predictable, every participant with computational fabrication experience mentioned that, when a problem does occasionally arise, the user typically doesn’t know until after it occurs. P2 compared using a fully automatic machine to “the handoff that happens [when you] give a plan or geometry to a secondary fabricator and trust that happens.” P6 gave a longer explanation that was also suggested by P1, P2, P3, and P7: "Since I’m physically at the system the whole time, working with it in this hybrid approach, it’s much easier to avoid any issues that might come up. With a 3d printer, with a lot of automated fabrication, there’s a kind of expectation that, well it’s automated for a reason; I don’t need to necessarily watch too much, within reason. [...]That is not always the case. Even printers that are industry standard sometimes can just have wild things happen to them. Things can go wrong and that is definitely something that is not likely going to happen with this hybrid approach. One, because it’s telling me where things might go wrong, and two because I am constantly there at the machine [...] For example if I’m pulling the machine across and I feel all the resistance building up that’s a pretty good indication that something is going wrong and I should be careful."

The benefits and drawbacks of interactivity were summarized by P4: “If I just give something to a printer, the output is predictable all the time. But the thing is, if I play with something like this, I have the control. [...] So I have the rights to make a mistake as well [...] If I play something with my hands, putting more effort on it, I feel like I did something really by myself.”

In addition to the complex interpretive expertise of understanding the machine’s settings and operations, a manual machine knitter must learn the haptic and auditory cues of successful operation. Every participant remarked on gaining this knowledge over the course of the session. For example, from P6: “Knowing how hard to push – I would say it definitely faded back into my subconscious by the end.” And from P2: “even if you’re following [the guided improvisation module], at the start there is a lot of experiencing the difficulty in the the haptics and understanding what feels right, and not, and the sort of rhythm you get into with switching the gear. Even if you’re not thinking about all those switches, you’re building that physical memory of the interaction with the machine how everything should feel and sound.”

This embodied experience of using the manual machine was generally seen as a positive. In comparison to the fully automatic process of using a 3D printer, P6 said “Assuming in a perfect world that your [3D printer] is going to work well, you can just walk away from it and come back later once it’s done. But [having to physically operate the machine] isn’t always necessarily a bad thing in my mind. [...] I think there is an aspect to it, sometimes you just really want to zone in on one thing and make sure you’re doing that one thing really well.”

All participants at some point in their conversation made a full-body “moving the carriage with both arms” motion, and P5 did so with an onomatopoetic “shunk” sound as well. P3 made the gesture while saying “I was having fun with the process once I got more of a handle on it,” and later summarized the experience with “there’s a lot of satisfaction to it.”

The hands-on aspect of the process also prompted feelings of pride, or ownership. P6 was very enthusiastic about the aspect of handcraft in our system: “I think that there is something really really special about being able to make something... I say ‘by hand’ – I’m putting some giant air quotes around that because it’s using the machine – but, you know, something that you crafted yourself.”

Drawing on related work in Interactive Fabrication, we proposed that on-machine interaction is especially suitable for contexts in which a hands-on experience is desirable, such as for personal or context-specific fabrication. With a manual knitting machine, the hands-on labor is not optional; however, this does not necessarily make it less desirable. Instead of being an unfortunate drawback of a less-expensive system, the need to physically operate the machine can be a creative opportunity [5]. Participants connected hands-on production to ideas of labor as a locus of value, for example suggesting that they might use the system to make nice gifts for loved ones which would be more valuable that something store-bought or fabricated automatically (P2, P6). In addition to the perceived social value of the artifact itself, hands-on systems can mediate valuable experiences – for example, an experience of learning [60] or of agentive collaboration with a machine [9].

In building our system, we made several design decisions to emphasize the experience of knitting. We positioned the machine itself as the only input – technically, the front-end interfaces could also be used with mouse clicks, for debugging or for explaining machine operation to someone without their own machine, but we typically deployed the system without either keyboard or mouse visible. We arranged the computer screen physically very close to the bed, to allow quick glances between the two and to reinforce the interaction metaphor of the machine itself as interface. (Future iterations of this work could conceivably close the distances even further with either projected imagery or with an Augmented Reality headset.) This closeness underpins a sense of immediacy, a private collaboration, between user and fabrication machine.

Additionally, participants, as well as the authors of this paper and anecdotally numerous lab visitors, have found the physical sensation of manual machine-knitting delightful. The auditory and haptic cues, along with the smoothly repetitive motion and feelings of control over a complex mechanism, add up to a uniquely satisfying experience.

We also proposed that augmenting manual machines could increase the availability of machine processes to creators beyond those with the ability to purchase and maintain expensive automated machines. Researchers can lower financial barriers to computational fabrication through engineering new, lower-cost mechanical systems [7] or by using software approaches to squeeze latent functionality out of existing low-cost systems [23, 37]. Our participants mentioned that the system allowed them to find value in a machine that they may not have otherwise interacted with, either because it was intimidating or because, as practitioners of computational fabrication, they found the idea of purely mechanical machines boring. While we do not share this latter opinion and are not of the belief that our system inherently “elevates” the knitting machine, we do see this as evidence that the system widens access, bringing new attention to a mature and fascinating fabrication machine.

Augmentation does not need to destroy or subjugate the underlying machine. For both pragmatic and conceptual reasons, we found it important to choose entirely reversible hardware interventions, and we designed our modular software systems to offer flexible amounts of support.

A drawback of computational tools is that they can “water down” or de-skill production processes: if a user is simply enacting system instructions, they lose creative agency. This concern has become particularly topical as increasing use of machine learning techniques in creativity support has spurred a new wave of discourse on the relative roles of creators and computational systems.

Because we view machine augmentation as a possible way to scaffold learning, the idea that a creator could over-rely on a computational system to the detriment of developing their own intuition is concerning. Indeed, one participant mentioned exactly this concern. (See section 7.1.) While each participant’s engagement with our system was too brief to produce deep expertise, we did observe that participants did not rely uniformly on each computational aspect of the system. They did lean heavily on basic usability assistance like error checking, which was explained by their fear of breaking the system (P2, P4, P6, P7). However, they followed higher-level suggestions (in the “Paths” interface) much less strictly. This implies that they were able to view these appropriately as suggestions, which they had more agency to reject.