Abstract
1. Introduction
Stuffed toys are often a child’s first friend and play an important role in a child’s cognitive, physical, and emotional development. They are also essential for building social skills through pretend play and role-playing. For example, when children groom or feed a stuffed toy, they mimic everyday interactions which then transition into the social world. During the process of caring for a stuffed toy, they also build empathy and kindness. Such interactions also play an important role in language skills, since children act out stories and scenarios with their toys.
While highly instrumented stuffed toys can be useful for parents and experts to observe and understand how children are developing in their natural environment, a challenge is how to incorporate sensing elements in smart toys without changing their feel and texture. Despite significant development in high-tech toys that incorporate a variety of sensors and actuators, they lose the soft feel and touch due to the need for rigid sensor elements to be placed near the surface of the toy. As a result, they are not as attractive to children who are drawn to softer and more squishy plush toys.5 As a compromise, smart-toy manufacturers usually only place a small number of sensors (typically one or two) on the toy’s surface. This diminishes the ability to measure fine-grained interaction with different regions of the toy.
Recent works on fabric-based sensors8,11,20,21 present a promising alternative to overcome traditional limitations in designing instrumented soft toys. These sensing techniques use familiar garments made of cotton and silk thread, and imperceptibly adapt them to enable sensing of pressure and touch signals to yield natural fabric-based sensors. In addition, fabric-based sensors have improved in their sensitivity, which now makes it possible to instrument soft toys with such sensors beneath the surface of the felt on the toy, thereby making them even less obtrusive.
These advances present a new possibility to create highly instrumented toys that integrate a large number of sensors to measure fine-grained and spatial-temporally complex interactions. In contrast to interaction with most rigid toys, interaction with soft toys is much more complex, often involving both hands and body contact. These interactions can range from holding, patting, or tickling the toy while simultaneously squeezing or holding it. The high-dimensional nature of the interaction possibilities and methods, dynamic pressure intensity based on the type of interaction and the location it is applied to, and varying sensing-region surface area require an equivalently high degree of instrumentation in the toy to accurately determine the type of interaction and makes single-threshold based pressure sensing methods obsolete. As a result, conventional binary buttons fail to track such complex interactions, while rigid pressure sensors change the feel of the toy and can interrupt natural interaction with the toy.
In this paper, we describe an end-to-end hardware and software design of such a highly instrumented soft toy, FabToy (Fabric-enhanced Toy, see Figure 1), and how we tackle the myriad challenges at the different system-design layers. In designing FabToy, we tackle engineering and design challenges across the stack, from form-factor optimization to sensor data acquisition, low-power analytics, and low-power communication.
From a sensor-design perspective, the array of 24 sensors is embedded under the felt surface of the toy to capture a range of complex interactions. From a hardware design perspective, we design ultra-low power and small-factor hardware that is placed deep within the toy to acquire signals, and amplify, filter, and reject powerline noise.
In terms of data analytics, we consider both local and remote processing—local processing would be suitable to execute autonomously and remote processing allows it to stream data to a broader range of interactive storytelling applications. To optimize local processing, we use machine learning (ML) to fuse the sensor data to classify simple and spatial-temporally complex interactions, as well as localize these interactions. We optimize this model to deal with several issues, such as crosstalk between sensor elements, while also ensuring it is lightweight enough to fit within the resource constraints of a low-power micro-controller. To optimize the remote processing pipeline, we compress the multi-channel data to transmit this over a Bluetooth low energy (BLE) radio via an auto-encoder to reduce transmission overhead.
Our end-to-end implementation and evaluation shows that:
FabToy can classify single interactions with an accuracy of 86% and complex interactions with an accuracy of 83%. This accuracy can further increase to 92%–94% for medium- or coarser-grained classification.
In the case of local processing, the use of early exit reduces processing power consumption by 45% while sacrificing only 4% accuracy.
In the case of remote processing, we show that dynamic channel reduction reduces transmission power consumption over a BLE radio by 43%, while sacrificing only 2% accuracy.
We implement the full FabToy system from sensor to processor to radio. Our hardware power and latency benchmarks on the nRF52840 low-power microcontroller with BLE radio show that our implementation is lightweight and can be executed with low delay and low power consumption and is practical for real-world deployment.
2. FabToy Hardware Design
The hardware architecture of FabToy is designed to achieve three main goals. First, the final prototype must look identical to a typical plush toy. Second, we wish to achieve complete coverage across the toy to capture a wealth of complex interactions across different locations of the toy. Third, we wish to ensure that we obtain high signal quality across various pressures applied during the interaction, from very gentle pressure during tickling, moderate pressure during rubbing, and orders of magnitude higher pressure while squeezing the toy.
Fabric-based sensors on the FabToy. We designed FabToy by leveraging recent developments in textile-based sensors. The fabric pressure sensors are implemented using a design presented in Homayounfar et al.9 and Kiaghadi et al.11 The sensors’ structure, electrical equivalent, and measurement method is shown in Figure 2. The sensors detect pressure applied on two electrodes by changing the resistance of the layer within.
Since typical fabrics present very high resistance, we lowered the resistance of the middle layer by coating the fabric with an ion-conductive polymer that provides ion conductivity to its fibers. As a result, the fabric’s resistance can be lowered by multiple orders of magnitude, and the sensitivity can be increased by introducing more conductive paths between electrodes. This is important since the textile sensors are beneath the felt surface and need to be extra-sensitive to capture a range of interactions from gentle tickling to squeezing.
Using fabric as the sensing substrate enables the sensors to conform to the toy’s shape to maximize sensing efficiency in the region of interest, which provides the flexibility to fit into odd 3D shapes, such as noses and ears.
Array of fabric-based sensors. To achieve fine-grained sensing of interaction, the toy needs to be covered with a large number of sensors to detect which part is being interacted with and the specific type of interaction. High spatial fidelity is vital given that interactions with soft toys tend to involve both hands and sometimes also the torso. A single large sensor that covers a large surface of the toy would only capture the overall pressure on the toy, so we would lose both information about the location of interaction and more nuanced information about interactions such as tickling a toy with one hand while holding the toy with the other.
Figure 3 shows a high-level view of the multi-sensor array: 24 sensors placed at strategic locations such as hands, arms, feet, ears, stomach, nose, and so on.
Amplified and unamplified sensor streams. While designing FabToy, we considered how to obtain a quality signal from the wide variety of pressure across the toy. Soft toys can be handled in a variety of ways, from very gentle pressure to a tight squeeze; this results in a few orders of magnitude changes in signal strength.
While increasing the sensitivity of the fabric-based sensor partially achieves this goal, it is not sufficient. Therefore, we also split each textile sensor into two streams: an un-amplified stream for medium to tight handling and an amplified stream for very gentle handling. The amplified data stream uses a band-pass filter to increase the signal-to-noise ratio (SNR) and helps us acquire very weak signals, such as during tickling.
Analog multiplexing the sensor channels. The 48 channels introduce several downstream challenges in sampling and processing the signal. At the hardware level, we need to deal with the fact that typical micro-controllers used on low-power devices have only a few analog-to-digital converters (ADCs). However, we can take advantage of the fact that we do not need very high sampling rates for the individual sensors.
The data acquisition pipeline is shown in Figure 3. We start with a voltage divider that senses the resistance of the pressure sensors. Output voltage represents the pressure applied on the fabric surface. Then, we use analog multiplexers in our design to uniformly sample all the channels using control signals issued by the micro-controller.
While low sampling rates can be adequate for capturing the interactions of interest, filtering out powerline noise poses a limitation. Powerline noise is generally minimal for rigid electronic force sensors with small surface areas. However, the larger surface area and relatively high impedance of textile sensors in FabToy make them particularly susceptible to electromagnetic interference. To mitigate this, we use a sufficiently high sampling rate of 160Hz per channel, which helps filter out powerline noise. Additionally, a simple moving average filter with a cutoff frequency of approximately 12Hz is applied within the micro-controller.
Hardware power consumption. To keep power consumption as low as possible, we carefully chose ultra-low power op-amps, regulators, and analog multiplexers that, in combination, consume two orders of magnitude lower power than a low-power microcontroller.
Computational challenges. The choices made in our FabToy hardware impact the design of the downstream modules in several ways:
Large number of channels. The overarching problem is that the large number of sensor channels increases computation and communication overhead. Even though each channel is sampled at a low rate, the fact that we have 48 channels makes the cumulative sampling rate quite high, which increases overhead downstream for both analytics on the low-power microcontroller and communication via a low-power radio.
Cross-talk between sensors. In contrast to electronic rigid force sensors, the fabric pressure sensors we use are double-sided, that is, they sense pressure applied from both sides of the sensor. As a result, they are also affected by internal movements of the toy stuffings, which might be caused by interactions with other sensors. Crosstalk also occurs when interaction with one sensor leads to pressure being applied on other sensors. For instance, the close proximity of nose and mouth sensors will lead to interactions with the toy’s nose to cause changes in the mouth sensor as well.
Cross-talk also occurs during complex interactions that involve holding the toy with one hand while interacting with the other hand. As a result, while the child is interacting, a reverse force is also applied to the anchor sensor.
An example of cross-talk is presented in Figure 4. We see that the chest and stomach sensors clearly pick up their corresponding interaction signals. However, the stomach sensor is also affected by swiping the chest.
Effect of humidity. A second challenge is that fabric-based pressure sensors are affected by humidity as well as pressure, that is, these sensors will have reduced resistance in a humid environment, similar to being under pressure. As a result, the output baseline will depend on base pressure as well as humidity.
3. Data Analytic Pipeline
The aim of the data analytic pipeline is to overcome the data-volume and cross-talk challenges to achieve high accuracy while optimizing power consumption.
Signal processing pipeline overview. The overall computational pipeline with the local and remote processing branches is shown in Figure 5. The initial stage is a triggering stage wherein sampled data from 48 analog channels is fed into a trigger block to ignore idle states when no interaction is happening with the toy.
Once the interaction is detected, we have two possible downstream pipelines depending on whether the data is locally or remotely processed. The first scenario is locally processing on a low-power microcontroller without an external device, while the second is remote processing, where raw data is transmitted to a smartphone-class device to offload computation and serve as part of a larger storytelling or educational platform.
Wake-up trigger. Since interactions tend to be in burst, the first stage of our signal-processing pipeline is a wake-up trigger to detect when the FabToy is in an idle state, where there is no interaction, versus active state, when there is interaction. The triggering module is based on the intuition that any activities happening on the toy will cause signal distortions, especially on the amplified signals. As a result, the standard deviation of the amplified channels is a simple indicator of activity.
To adapt this block to dynamic changes, such as changes in ambient powerline noise, and the fabric resistance due to temperature and humidity, we use a dynamic threshold based on the summation of the standard deviations of all amplified channels.
Local processing. The local processing model in FabToy consists of five layers of neural networks with batch norms and ReLu layers. We introduce early exits between neural network layers to optimize computation power consumption.
Convolutional layers. In our design, we apply and stack multiple convolutional layers to help learn the crosstalk between sensors because a single convolutional kernel can only capture the patterns in its Receptive Field and fail to capture correlations from crosstalk sensors that are far away. The early layers with smaller receptive fields can capture near-field patterns of the sensors; the latter layers could perceive the potential crosstalk relationship between the sensors that are far away in the data plane.
Batch norm. To overcome humidity-related artifacts, we embed normalization layers to standardize the internal features after each convolutional layer. The normalization layer helps rectify the biased input due to the humidity during model inference time.
Local early exit. To reduce the computation time and energy cost, our computational pipeline endows the network with early exits,12 thereby allowing us to obtain predictions at intermediate stages in the pipeline. The intuition is that the majority of data cases can be classified with only a few layers, and only a few more complex data cases require the entire deep-learning pipeline.
We make the prediction based on 48× raw sensor data at the granularity of a second. The classification result will only be generated once within the given time window, which indicates the categories of the interaction and position for the given time period. In our experiments, we use a time window of 3s with 2s of overlap between windows.
Remote classification model. The remote model is similar to the local model except for two differences. First, we introduce a data-reduction module to minimize communication overhead. Second, we remove the early exit modules since we do not have as stringent resource limitations at the remote device.
Dynamic channel aggregation. To reduce communication cost, we reduce the input dimension by down-sampling time series data for all channels. We aggregate channels by using an auto-encoder that aggregates the streams to reduce the number of data streams to the desired value. The auto-encoder structure consists of an encoder/decoder pair, which contains several convolutional layers. In our setting, the encoder will be on the IoT devices to efficiently aggregate and encode the original streams into smaller size.
The number of output streams directly affects the size of required data to be transmitted through the radio, and as a result, directly reduces communication power consumption. The trade-off between reducing data streams and the drop in system’s accuracy is studied in Section 5.3.
Remote computational model. The remote classification model that we use is similar to the local one, however, it does not have the early exit modules since compute resources are not as limited in the remote device. We note that one of the major advantages of remote processing is the ability to leverage more complex models.
4. Implementation
In this section, we describe the implementation of our FabToy prototype. We used a plush teddy bear to implement FabToy, shown in Figure 5. The figure also highlights the placement of 24 fabric sensors on the toy. We note that the fabric sensors are placed underneath the felt; therefore, there is no change to the feel and texture of the exterior of the toy.
Fabric sensor design and placement. The textile pressure patches are made from three layers. The two outer layers are silverized nylon fabric acting as electrodes that cover the middle layer, which is the functionalized cotton gauze. The size of these patches varies from 2×2cm2 to 3×3cm2 depending on their placement. The cotton gauze is sonicated in deionized water for 15 minutes, rinsed with isopropanol, then heated at 100∘C for two hours. The treated cotton gauze patches are rinsed once more in isopropanol and dried overnight. Finally, this layer is coated with perfluorosilane using vapor deposition to add wash-stability to the sensor.
To measure the resistance of the sensors, they are connected to a voltage divider circuit where one of the electrodes is grounded (as shown in Figure 2). For shielding purposes, we place the grounded electrode outward and closer to human skin during interaction. This is due to the fact that human body carries electrical charge, which can be coupled into the sensors and confound the interaction signal. By grounding the outer plane, we make sure this extra charge is routed to ground and that it will not show up in the output signal.
Electronics board. The sensors are internally routed to a PCB. It receives signals from 24 sensors, filters them, and then creates two sub-channels: one amplified and one unamplified. Then the resulting 48 channels are multiplexed into eight ADC channels of the MCU using four analog multiplexer ICs. Each multiplexer outputs two out of its 12 inputs according to the address bits provided by the microcontroller. Finally, the microcontroller digitizes and transmits the data to a laptop using BLE (Figure 3).
The microcontroller runs an application that provides address control signals for the multiplexers, reads the analog channels from 8 analog input pins, creates packets from the samples and an index for packet loss detection. A moving average filter over seven samples and duty cycling is used to reduce power consumption.
Implementation of classification model. For the classification task, we have both single interactions (interaction at one location) and complex interactions (two concurrent interactions). For the given time window, we use the tuple (interaction@position) that is most frequently shown as the ground truth for the single interaction; and similarly for complex interactions, we use the most frequently seen two single interactions as the ground truth.
We use leave one out as cross validation for the model training. Due to the fact that many of the ground truth labels are “no interaction” in the collected data and the labels are unbalanced, we use a data sampling technique to balance the labels in the dataset for the model training.
We implemented the local and remote execution pipelines of FabToy on the nRF52840 platform.16 nRF52840 is a low-power ARM-based embedded device with BLE protocol support, which allows us to implement both the local and remote processing pipelines. The local model requires 18.5KB of RAM and 153.7KB of flash memory, corresponding to 7% and 15.3% of available space in the MCU, respectively.
5. Evaluation
To validate the performance of FabToy, we performed a series of experiments that highlight the benefits of our hardware platform and machine-learning pipeline and provide a breakdown of the contribution of various building blocks in FabToy.
User study. We asked 18 participants to perform several interactions with a toy as part of an IRB-approved study. The interactions include holding, patting, tickling, and swiping the toy. We chose this combination as an example of meaningful interactions that can be performed with a toy (for example, holding a hand while patting the head). This separation of single and complex interactions allows us to monitor the impact of complex interactions and compare the performance of FabToy with other models. We chose 37 single interactions from the pool of possible interaction/location pairs to prove the possibility of detecting the interaction and its location. In addition, we chose 65 complex interactions that include holding one of the toy’s hands and performing the other interactions with the user’s free hand. Including the idle state, we have 103 different combinations of single and complex interaction/location as labels.
User study. The users varied in age (25–35) and gender (six females). They were free to choose their own method of performing each specific interaction, including the action’s speed and intensity.
The feel and look of the toy convey no clue regarding the whereabouts of sensors underneath the felt, and users were not informed about it either so as not to alter users behaviors. We placed a video camera zoomed in on the toy to be used as ground truth for labeling the data. Each user went through slightly more than 16 minutes of study where they were asked to perform a series of interactions covering all the 103 labels, each during a ten-second window with 3–4s rest, counted toward idle case. Overall, we gathered around five hours of data from our participants while they were interacting with FabToy.
Overall classification performance. We start by looking at the overall performance of FabToy. Since crosstalk is present between adjacent sensors, we look at the classification performance for three spatial granularities:
Fine-grained, where we want to precisely determine which of the 24 sensor locations is being interacted with.
Medium-grained, where we merged some adjacent sensors into one location, for example, node and mouse, resulting in 24 sensor locations being merged into eight regions of interaction.
Coarse-grained, where we merged the following sensors: both arms (hand and arm on each side), both feet (foot and thigh on each side), head (including nose, mouth, cheeks, ears, and forehead and top of the head), body (including chest, stomach, waist, and back) and counted them as one to transform 24 sensor locations into four coarser regions.
Figure 6a shows the results. We see that for precise fine-grained classification, we achieve 86% accuracy for single interactions and 83% for complex interactions.
As we progressively coarsen the prediction granularity, classification performance increases from 86% to 94% for single interactions and from 83% to 93% for complex interactions. Thus, we see that FabToy can be very effective at both fine-grained and coarse-grained classification with increasing performance as the spatial fidelity reduces.
Comparison against alternate models. We now demonstrate the superiority of the FabToy model to some other well-established ML models. We compare against relatively lightweight models that can be executed on a microcontroller platform. The Multi-Layer Perception (MLP) model14 uses features extracted from lightweight convolutional layers, which are fed into multiple fully connected layers in the MLP model to compute the predictions. For the other models (Random Forest, xgBoost, and Nearest Neighbors), we pre-process the raw sensor data to help them better capture the time-series contextual information. We then use histogram density features to map the time-series data from each sensor into a 10-bin histogram. The histogram features (10×48=480) are fed into the three ML pipelines to train the models. The Random Forest model13 constructs a collection of decision trees and performs an ensemble classification; the xg-Boost model4 boosts decision trees via applying gradients to correct the previous mistakes and minimize the losses; the k-Nearest Neighbors model3 deterministically finds k instances in the dataset which are most close to the input data and use the most commonly seen label of the k instance as the label of the input.
Figure 6b compares FabToy versus other models for fine-grained classification. As can be seen, the FabToy model outperforms all others both for single and complex interactions. The FabToy model can achieve more than 5% higher accuracy than the MLP model and more than 10% higher accuracy than the other three ML methods.
Importance of amplified and non-amplified channels. Second, we look at the advantage of using both amplified and unamplified streams in terms of overall performance. To evaluate, we separately choose amplified and non-amplified signals to train our model and check the performance against the combined version that uses both data streams. For the amplified-only and non-amplified only cases, we modified the FabToy model so that it could accept input signals that are half the size.
The result is shown in Figure 6c. We see that the combination of amplified and non-amplified signals provides more information for the ML model compared with either of them separately. For single interactions, using both streams gives us 3%–4% improvement in accuracy over using one of the streams. For complex interactions, we get 1%–5% improvement over just using one of the streams. This plot shows the importance of generating amplified versions of the 24 base channels.
Benefit of early exit. The effect of early exit is studied on model accuracy and latency on the nRF52840 (Figure 7a).
We see that the model exiting early reduces accuracy by about 4% but has around 8× better latency compared to executing the full model. Early exits at intermediate layers between these ends provide progressively better accuracy but offer less gains in performance.
Benefit of adaptive aggregation. Figure 7b illustrates the trade-off between the number of streams transmitted and the model’s accuracy. Accuracy rapidly increases until about 10 streams and then plateaus. This allows us to reduce transmission power consumption by about 2× compared to a system that transmits all channels without compression.
Specific to the remote model, we employ data dimensionality reduction by adaptive aggregation. We now look at the performance of this module. We produced this benchmark using Nordic Online Power Profiler for BLE.a These numbers also agree with the empirical power breakdown figure, shown in Figure 8.
Comparison of local and remote processing. Having analyzed the local and remote versions of FabToy, we can compare the performance of these methods. Figure 7c shows the results. We see that the local model is generally more efficient than the remote model. This is unsurprising since the radio consumes more power than the MCU. However, the gap narrows in the regime where higher accuracies are desirable since the remote version is able to take advantage of more complex models being used on the remote device. We note that the choice between local and remote processing may be to enable specific applications rather than power consumption. From this perspective, the main takeaway is that both methods are viable at low power.
Power benchmarks. Figure 8 shows the processing power consumption breakdown for FabToy across the different blocks. Since power consumed for the remote model depends on the number of channels transmitted and power for the local model depends on the early exit point, we provide numbers for three different channel aggregation values and three different early exit points.
Overall, FabToy consumes from 2.9mW to 4mW depending on the number of channels transmitted or the early exit point. This amount of power consumption corresponds to more than a month of operation on a small 950mAh rechargeable battery(3cm×5cm) before it needs recharging.
We see that a significant fraction of the power is consumed by the sampling block. This is the case since we are sampling each channel at 160Hz (to remove powerline noise), hence the system is sampling at 7.68kHz. We have not focused on optimizing this subsystem in this paper but expect that this can be optimized with better hardware design. The triggering block can also improve efficiency if done in analog before the MCU is turned on.
Finally, we note that since the board and battery can be placed inside the toy, they are physically isolated using waterproof packaging and as a result, battery capacity can be increased to enable months of operation per full charge.
6. Related Work
In this section, we describe relevant state of the art in smart toys and the usage of fabric-based sensors in measuring human health and activity.
Augmenting toys with sensing capabilities. There has not been much work on exploring the use of arrays of sensors in detecting user interactions with toys. Vega-Barbas et al. implemented a prototype toy that can measure grip and stretch in a toy using fabric sensors.17 Yonezawa et al.22 incorporate textile pressure sensors to detect the intensity and duration of pressure on toy’s surface and localize it as head, body, or back. Using the information they gather from the pressure sensors, they estimate a user’s emotional state. There has also been some work to enable toys to gather biomedical information on the toddler playing with the toy.18
Using an array of fabric sensors to detect physical and physiological signals. Recent advances in smart fabrics have led to sensing, actuation, and energy-harvesting fabric elements. Kiaghadi et al. designed and implemented loose-fitting sleepwear that can track sleeping posture, heart rate, and respiration rate.11 Fabric-based sensors have also been used to detect various human motions,19 perspiration,10 respiration rate,2 or inputs.1
However, none of these efforts enables what we have proposed: fine-grained interaction detection and localization on plush toys. Our application and the techniques we use to deal with large-scale sensor arrays are not addressed in prior work.
Early exit and autoencoder. The ML components we use, early exit15 and autoencoder, are well-known techniques in edge AI. Reducing data dimension using autoencoders has also been studied in literature.7 Our contribution is that we leverage these techniques in the context of a unique smart-toy system rather than the ML components by themselves.
7. Discussion
In our design of the FabToy, we have encountered several roadblocks and opportunities for future work. We briefly discuss these issues.
Application studies involving toddlers and children. Our dataset is collected from adults to enable repeatable and dense data collection since we have a large number of {interaction, location} pairs that we need to distinguish with our model. The study was started during the COVID pandemic, and due to safety considerations we did not conduct a user study with small children who would normally play with plush toys. Our work paves the way for future work that studies how small children interact with FabToys in naturalistic conditions.
Multi-toy interaction. One of the exciting new opportunities presented by FabToy is detecting a much richer set of interactions with soft toys. So far, we have focused on a single FabToy but our work can potentially be extended to interactions with more than one toy. This can allow us to explore more complex storytelling applications that involve interaction with different toys that play different roles in the story.6 While such multi-toy interaction with smart toys can be enabled by today’s technology, the key advantage in our approach is the naturalistic feel for the smart toy making it more engaging for children.
Platform for introducing computational thinking. Programmable robots are extensively used for introducing computational thinking in early childhood education, where children learn about computing via physical interactions with robots (for example, Bee-Botb). Unlike rigid robotic toys, FabToy is a plush toy that can engage with children more naturally, thereby providing an alternate platform to engage children physically while teaching computational techniques. For example, children can map voice responses from the plush toy to interactions performed in the physical world, such as patting or tickling.
Sensing and actuation in FabToy. While FabToy is focused on dense sampling of a soft toy, there are many new opportunities if we can pair it with other modalities, such as audio, to expand the vocabulary of interaction. These have been significant advances in natural language understanding and dialog, and we can potentially pair richer tactile sensing of interactions with the FabToy sensors with more sophisticated audio-based dialog methods to enhance how children interact with smart toys.
Dynamics in natural environments. We have designed, implemented, and analyzed an interaction-aware toy and validated the performance in semi-stationary scenarios. However, there may be a broader range of interactions in the natural environment. For example, a child may interact with a toy during walking. This can create new signal dynamics, as walking causes vibrations all over the toy, which will be seen by all the channels. To deal with this issue, we will need to add active motion-artifact-canceling methods, where an algorithm detects and recreates the motion signal and subtracts it from the original.
8. Conclusion
In this paper, we presented FabToy, an end-to-end platform for detecting and localizing users’ interactions with soft toys in a fine-grained real-time manner. Our design addresses several challenges, including ensuring unobtrusiveness and a natural look and feel while achieving high signal quality, high spatial-temporal fidelity, and low power operation. To enable this, we have optimizations across the hardware-software stack, including a highly optimized array of fabric sensors, low-power signal conditioning, acquisition, and low-power embedded ML and data compression. Our evaluation shows that the device can enable accurate detection across a range of simple and complex interactions across the entire surface of the toy. Overall, FabToy offers a promising path forward for intelligent plush toys and has significant potential to enable a new class of interactive toys for kids.
Acknowledgements
This work was funded by the National Science Foundation under agreement CSR Medium 1763524. We also acknowledge support from the National Science Foundation under agreements 1815347 and 1839999.
Join the Discussion (0)
Become a Member or Sign In to Post a Comment