WO2019216941A1 - Quality inference from living digital twins in iot-enabled manufacturing systems - Google Patents

Abstract

A method for quality inference in an Internet of things (IoT)-enabled manufacturing system includes generating a fingerprint library. Each fingerprint corresponds to a digital twin of a product and comprises (i) stored feature clusters; and (ii) a stored silhouette coefficient for each feature cluster. Sensor values are collected from IoT sensors monitoring a physical twin of the manufacturing system. Feature values are extracted from the sensor values. Parameters defining the digital twin are identified and used to segment the feature values into feature subgroups. Next, feature clusters are estimated for the subgroups using the fingerprint library. A current silhouette coefficient is determined for each feature cluster. If the current silhouette coefficient deviates from a corresponding stored silhouette coefficient, the parameters of the digital twin corresponding to the feature cluster are identified and designated as being potentially anomalous. An alert may then be provided to users identifying the potentially anomalous parameters.

Description

QUALITY INFERENCE FROM LIVING DIGITAL TWINS IN IOT- ENABLED MANUFACTURING SYSTEMS

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] TECHNOLOGY field This application claims the benefit of U.S. Provisional Application Serial No. 62/668,327, filed May 8, 2018, which is incorporated herein by reference in its entirety.

GOVERNMENT RIGHTS STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

[0002] This invention was made with government support under NSF CPS CNS-l 546993 awarded by National Science Foundation. The government has certain rights in the invention.

[0003] The present invention relates generally to methods, systems, and apparatuses related to a methodology for building Digital Twin of manufacturing systems from the Internet of Things (IoT) enabled sensors. The methodologies described herein may be applied, for example, to perform quality inference and fault localization in various manufacturing environments.

BACKGROUND

[0004] A Digital Twin is the digital representation of a physical system. A typical Digital Twin comprises data and models generated throughout the lifecycle of a system. Building a Digital Twin begins long before the physical system even exists in the real world and ends long after the system ceases to exist. During design, the Digital Twin provides an as- designed blueprint of the system consisting of requirements, conceptual designs, simulations, and system models. During manufacturing, the Digital Twin provides an as-produced view of the individual instances of the system consisting of their unique properties such as manufacturing data. Finally, during operation, the Digital Twin provides an as-operated view of the individual instances of the system from insights generated from sensor data, service records, logs, etc. Although the narrative of the Digital Twin is usually told chronologically from the beginning to the end, the most powerful aspect in the Digital Twin concept are the feedback loops that arise when the different views are compared. For example, the system operation can be optimized by analyzing how the system is currently operating (as-operated) against how it should be operating (as- designed). Similarly, the time-to-market of subsequent system instances can be accelerated by analyzing how previous instances were manufactured (as-manufactured) and deviate from the designed (as-designed). Hence, Digital Twin is expected to play an important role in the next industrial revolution (Industry 4.0).

[0005] Digitalization is transforming the manufacturing industry. Government organizations (e.g., NASA, the Air Force, etc.) and businesses like Siemens™ are currently building Digital Twins of gas turbines, wind turbines, engines, and airplanes that allow them to manage the assets, optimize the system and fleets, and to monitor the system health and provide prognostics. The concept of the Digital Twin was first used by NASA to describe a digital replica of physical systems in space maintained for diagnosis and prognosis. A key enabler for creating a Digital Twin is the availability of a large number of built-in sensors and their historical data. Previously, due to the lack of cheap sensors and high-speed and reliable network, acquiring the data for the Digital Twin was costly. Today, thanks to the availability and affordability of Internet-of-Things (IoT) sensors, it is becoming easier and cheaper for system operators to acquire large amounts of data on-the-go from their physical systems. However, for systems that do not have built-in sensors and lack historical data, building the Digital Twin with IoT sensors is still a challenge.

SUMMARY

[0006] The present invention, as described in various embodiments herein, describes a systems, methods, and apparatuses that perform quality inference from living Digital Twins in IoT-enabled manufacturing systems. Briefly, methodology described herein constructs a living Digital Twin of the manufacturing system by utilizing various IoT sensors (e.g., acoustic, vibration, magnetic, and/or power sensors). Based on these signals, a clustering algorithm is used to generate a fingerprint library that effectively represents the physical status or the Physical Twin of the system. The Digital Twin is used for localizing the anomalous physical emissions that have the potential of resulting in key performance indicator (KPI) variation. Moreover, some embodiments of the present invention employ an algorithm for updating the Digital Twin, and inferring the KPI deviation. Compared to the state-of-the-art methods (which do not consider model aliveness), the methodology described herein is able to update itself, infer KPI deviation and localize anomalous faults in the manufacturing system.

[0007] According to some embodiments, a method for quality inference in an Internet of things (IoT)-enabled manufacturing system includes generating a fingerprint library comprising a plurality of fingerprints. Each fingerprint corresponds to a digital twin of a product and comprises (i) a plurality of stored feature clusters; and (ii) a stored silhouette coefficient for each feature cluster. Sensor values are collected from a plurality of IoT sensors monitoring a physical twin of the manufacturing system. New feature values are extracted from the sensor values. Parameters defining the digital twin of the product are identified, and the new feature values are segmented into new feature subgroups based on the parameters. Next, an anomaly detection process is performed that includes estimating new feature clusters for the new feature subgroups using the fingerprint library. A current silhouette coefficient is determined for each new feature cluster. If the current silhouette coefficient for a new feature cluster deviates from a corresponding stored silhouette coefficient, the parameters of the digital twin of the product corresponding to the new feature cluster are identified. Additionally, the parameters are designated as being potentially anomalous. An alert may then be provided to one or more users identifying the parameters of the digital twin designated as being potentially anomalous. For example, in one embodiment, the alert is presented on a human machine interface (HMI) or other display. Alternatively, rather than providing an alert, a record may be created (e.g., in a log file or a database) indicating the potentially anomalous parameters.

[0008] According to other embodiments, an article of manufacture for quality inference in an IoT-enabled manufacturing system comprises a non-transitory, tangible computer- readable medium holding computer-executable instructions for performing the method discussed above.

[0009] According to other embodiments, a system for quality inference in an IoT-enabled manufacturing system includes a database, IoT sensors, and a computing system comprising one or more processors. The database stores a fingerprint library. The IoT sensors generate sensor values corresponding to a physical twin of the manufacturing system. The processors of the computing system extract new feature values from sensor values, identify parameters defining the digital twin of the product, and segment the new feature values are segmented into new feature subgroups based on the parameters. The processors then perform anomaly detection process to determine whether the parameters of the digital twin are potentially anomalous. If the parameters are potentially anomalous, a record is created or an alert is provided to a user.

[0010] Additional features and advantages of the invention will be made apparent from the following detailed description of illustrative embodiments that proceeds with reference to the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

[0001] The foregoing and other aspects of the present invention are best understood from the following detailed description when read in connection with the accompanying drawings. For the purpose of illustrating the invention, there is shown in the drawings embodiments that are presently preferred, it being understood, however, that the invention is not limited to the specific instrumentalities disclosed. Included in the drawings are the following Figures:

[0002] FIG. 1 shows an example Digital Twin for manufacturing, according to some embodiments;

[0003] FIG. 2 shows a Digital Twin modeling methodology that may be used in some embodiments;

[0004] FIG. 3 presents an algorithm for fingerprint library generation for a Digital Twin that may be used in some embodiments;

[0005] FIG. 4 presents an algorithm for localizing deviation and checking for Digital Twin updating that may be used in some embodiments;

[0006] FIG. 5 shows an experimental setup for modeling a Digital Twin;

[0007] FIG. 6 presents an experimental setup for sensor position exploration;

[0008] FIG. 7 shows classification accuracy score for various sensor positions;

[0009] FIG. 8 shows scatter plots of clusters plotted with the first two principal components of the features for five clusters; [0010] FIG. 9 presents the silhouette coefficient of various clustering algorithms that may be used in different embodiments of the present invention;

[0011] FIG. 10 shows a test DTobject created using a CAD tool for checking anomaly localization capability of a Digital Twin;

[0012] FIG. 11 shows example average receiver operating characteristic (ROC) curves for anomaly localization through four sensor data;

[0013] FIG. 12 illustrates the accuracy of the Digital Twin's anomaly localization for each channel;

[0014] FIG. 13 presents degradation test result for the Digital Twin;

[0015] FIG. 14 shows the accuracy of the Digital Twin's KPI inference model;

[0016] FIG. 15 presents a qualitative comparative study of various methods compared to the methodology employed by various embodiments of the present invention; and

[0017] FIG. 16 illustrates an exemplary computing environment which may execute an application for performing quality inference from living digital twins in IoT-enabled manufacturing systems, according to some embodiments.

DETAILED DESCRIPTION OF ILLUSTRATIVE EMBODIMENTS

[0018] The following disclosure describes the present invention according to several embodiments directed at methods, systems, and apparatuses related to modeling and maintaining a living Digital Twin of a manufacturing system using off-the-shelf multi-sensors in Internet-of- Things (IoT) devices. In contrast to conventional techniques, the methodology described herein is able to update itself and improve its fault localization and inferring dimensional tolerances.

[0019] FIG. 1 illustrates a framework 100 for applying digital twins to manufacturing applications, according to some embodiments. As briefly explained earlier, in a manufacturing environment, we have Digital and Physical Twin of product and system DTobject, DTsystem, PTobject, and PTsystem, respectively (as labeled in FIG. 1). The Digital Twin of the product DTobject starts its lifecycle in the design phase, where computer aided design and computer aided manufacturing tools are used to represent the product in the cyber-domain. These product Digital Twins from design phase (Xi) then go through the production, where the Physical Twin of the system PTsystem takes raw materials, energy, and the DTobject to create its corresponding Physical Twins (Yi). The Physical Twin of the system PTsystem maybe represented as a Cyber- Physical System (CPS), which comprises cyber-domain computing processes and physical domain components that communicate through a network. We represent the whole CPS system as a Physical Twin of the manufacturing system. We may consider dividing the manufacturing system into corresponding cyber and physical domains, and further consider only the physical domain components as the actual Physical Twin of the system. However, since the cyber domain of the manufacturing system reacts in a deterministic manner corresponding to the product Digital Twin, for simplicity we consider combination of cyber and physical domain as the PTsystem. The PTsystem is influenced by the manufacturing environment in a stochastic manner.

[0020] Let ^object ^~

represent the parameters that define the Digital Twin of the product (e.g., dimension, surface roughness, mechanical strength etc.), RΊ system—

- , b^h } n€ L>o,/> ^€ S. represent the parameters of the

Physical Twin of the manufacturing system, and let ^s stem =

P^€ ^>0. Y ^e & represent the environmental factors affecting the manufacturing system (e.g., temperature, humidity, pressure, etc.). The framework 100 shown in FIG. 1 captures the interaction between these parameters using one or more IoT sensors. Using the data collected from multiple modalities (e.g., acoustic, vibration, magnetic, power, etc.), a stochastic function /(. ) is modeled that performs three tasks. First, the deviation in the DTobject parameter is localized from its Physical Twin PTobject. Second, the DTsystem is confirmed to be up-to-date (i.e., alive). Third, the Key Performance Indicator (KPI) deviation is inferred for the DTobject before creating the PTobject. Moreover, the DTobject may interrogate the DTsystem to infer about tolerance deviation due to the current status of the PTsystem.

[0021] The manufacturing system comprises cyber and the physical domain. The computing components in the cyber-domain have processes that communicate with the physical domain. A cross-domain signal that is passed from the cyber-domain to the physical-domain has the possibility of impacting the physical domain characteristics. This phenomenon is more prominent in manufacturing system where the Digital Twin of the product causes the Physical Twin of the system to behave in a certain deterministic manner. However, due to these characteristics, there exist physical emissions (e.g., acoustic, vibration, magnetic, etc.) which also leak information about the Digital Twin of the product. We denote these emissions as side- channels, as they indirectly reveal the information about the cyber-domain interactions due to the particular physical implementation of the system. For building the Digital Twin of the manufacturing system that captures the interaction between the product Physical Twin, the environment, and the system’s Physical Twin, these side-channels play a crucial role in providing the necessary information. In this work, we analyze four such analog emissions which potentially behave as side-channels.

[0022] Let Sa(t), Sv(t), Sp(t), and sm(t) represent acoustic, vibration, power and magnetic emissions from the manufacturing system. Then we define each of these signals as:

[0023] Equations 1 -4 represent the analog emissions as a result of deterministic function d(.) which is influenced by the Digital Twin parameters of the product DTobject and the Physical Twin parameters of the System PTsystem, and the non-deterministic environmental parameters E system. Moreover, for each of the analog emissions, the total number of parameters (a,b,g) may not be same. Traditionally, non-trivial simulation based approach such as finite element analysis is utilized to model the deterministic part and explore relation between the DTobject, PTobject, and PTsystem. However, the PTsystem parameters vary over time, and Esystem parameters affect the PTobject in a stochastic manner. Hence, we explore the possibility of using IoT sensors to model and maintain a live DTsystem for product quality control.

[0024] The Digital Twin can be used for various purposes. As one example, the paragraphs that follow demonstrate that by maintaining a living Digital Twin once can infer about the possible deviation in quality of the product. One of the Key Performance Indicator (KPI) that is utilized is the dimension (Kd) of the product.

[0025] FIG. 2 shows a methodology 200 for building the digital twin, according to some embodiments. As mentioned earlier, we build the Digital Twin from the IoT sensor data to perform three tasks: run-time localization of faults, regular update of the system Digital Twin, and KPI inference for product’s Digital Twin. For run-time localization, in this example, we create and maintain an active fingerprint library of the individual IoT sensor data corresponding to the DTobject parameters. This fingerprint also captures the PTsystem and Eastern parameters during run-time. Then for localizing the faults, the deviation of the run-time IoT sensor data is compared with the fingerprint. For updating the Digital Twin, a voting scheme is used to check if majority of the fingerprints are deviating corresponding to few fingerprints. To infer about the deviation in KPI, we estimate a function ^

where the KPI is a function of DTobject, PTsystem and E system parameters, and the IoT sensor data.

[0026] For generating the fingerprint of the DTobject from the IoT sensor data, first of all it is parsed to its corresponding parameters

· · ^Sm The parsed values will depend on the type of manufacturing system. In the experimental section provided below, we will present the parsing for additive manufacturing system that utilizes G/M-codes. These codes are the instructions that carry the process information(e.g., machine specific parameters, such as temperature, acceleration values for motors, etc.) and product information (for example the geometry description). The parsing will break down the individual parameters from the product Digital Twin.

[0027] For generating the fingerprint from the analog emissions, various time domain features may be analyzed from IoT sensor data including, without limitation, energy; energy entropy; mean amplitude; maximum amplitude; minimum amplitude; median amplitude; mode of amplitude; peak to peak features (i.e., highest peaks, peak widths, peak prominence, etc.); root mean square values; skewness; standard deviation; zero crossing rate; kurtosis and frequency domain features such as mean frequency; median frequency; signal-to-noise ratio; power bandwidth; spectral centroid; spectral entropy; spectral flux; spectral roll off from short term 50 millisecond time domain windows (also known as short term Fourier transform) and continuous wavelet transform (CWT) (140 in total); 20 mel-frequency cepstral coefficients (MFCCs); etc. From these features, a technique such as Principal Component Analysis (PCA) may be performed to reduce the dimension of the features.

[0028] Before clustering is performed, the features are synchronized and segmented into subgroups based on the parsed DTobject parameters

For instance, the features are segmented based on conditions such as presence or absence of particular component’s movement (e.g., motors responsible for moving the 3D printer nozzle in X, Y, Z-Axes). By segmenting based on the parsed parameters, the features may be reduced into smaller groups. This allows for further reducing the complexity in acquiring the fingerprints.

[0029] For generating the fingerprint of the parsed parameters of DTobject, a clustering algorithm is used to generate clusters that group the similar features into a single cluster. For analyzing the clustering algorithm and the corresponding fitness of cluster number, silhouette coefficient is calculated for each sample. It measures the similarity of the feature to its assigned cluster compared to other clusters, with high value representing its close match to the assigned cluster. It may be calculated as follows:

where a(i) is the average intra-cluster distance, and b(i) is the mean of the nearest cluster distance (lowest average distance of i with all other points in other cluster where i is not a member). The clustering is carried out for each group of the features for all the analog emissions. The cluster centers, cluster number, and the corresponding average silhouette coefficient of all the analog emissions are stored in a library, effectively representing the fingerprint for the given parsed DTobject parameter.

[0030] FIG. 3 illustrates an algorithm 300 for generating the cluster fingerprint, according to some embodiments. As shown in this algorithm 300, Line 3 estimates k clusters for each of the group in all channels and stores it in a library along with the corresponding average silhouette score. This clusters and the corresponding silhouette scores are used as the fingerprint to determine if the future analog emissions match the given product Digital Twin parameters. [0031] The fingerprint library calculated using the algorithm 300 shown in FIG. 3 is used for detecting and localizing the anomalous physical signals corresponding to the DTobject while printing. The algorithm 400 for detecting and localizing the deviation from the stored fingerprint is shown in FIG. 4. Algorithm 400 parses the features of the DTobject either run time or after the product’s Physical Twin has been created. Then, using the fingerprint library it estimates the new cluster labels for the parsed features in line 5. Using these labels and the features, the new silhouette coefficient for the parsed features are calculated in line 6 using Equation 5. If the calculated silhouette coefficient is less than the stored silhouette coefficient ± threshold SCihreshoid then then the DTobject segment corresponding to the feature is marked as deviating from the previous fingerprint, and returned as containing possible anomaly.

[0032] For updating the Digital Twin model, the library of fingerprint for the DTobject has to be updated. However, before updating the library, it should be checked if the anomaly in the fingerprint is temporary or it is due to the degradation of the machine over time. In order to update the Digital Twin by updating the library, lines 10 in algorithm 400 keep tracks of all the DTobject variables that deviated. Then line 12 checks if more than featureThreshoid of the DTobject parameters deviated from the previous fingerprint. Then line 15 checks if more than groupThreshoid of the groups deviated from the previous fingerprint. Finally, line 18 checks if more than channelThreshoid of all the channels deviated. If these conditions are met then in line 19 the library for the Digital Twin is updated. These thresholds for checking the deviation from the fingerprint can be varied for different channels and groups based on the amount of information leaked by each of the side-channels.

[0033] As briefly mentioned earlier the KPI inference is done using an estimated function

¾ - / (<*> P- U ¾( > ½{^/), ^$m(f)k ¾»{t)) However, instead of the signals, the features extracted for corresponding DTobject and PTsystem parameters are used. This function may be solved using various techniques known in the art. For example, in some embodiments, gradient boosting based regressor is used to estimate the function /(. ). It uses ensemble of decision trees based on regression models. This ensemble generates a new tree against the negative gradient of the loss function and combine weak learner to control over-fitting. Hence, they are robust to outliers. This estimation function is also updated when the Digital Twin update algorithm reaches a consensus that all the fingerprints are outdated. [0034] To demonstrate the applicability of the methodology described herein for building and maintaining a Digital Twin, FIG. 5 shows an experimental setup for modeling the Digital Twin in this system, according to some embodiments. In this example, an additive manufacturing system is utilized. A fused deposition modeling based Cartesian additive manufacturing system is selected due to its widespread use in rapid prototyping 3D objects layer by layer. The Digital Twin of the product (i.e., the 3D object being created) is initially described using a Computer Aided Design (CAD) tools. The CAD tool then produces StereoLithography (STL) files which comprises geometry description of the object in coordinate space. Then, a Computer Aided Manufacturing (CAM) tool takes the STL file and slices it into multiple layers and finds a trace to be followed to print the object in each layer. The output of the CAM layer is the G/M-code. In the experiment shown in FIG. 5, we consider that the DTobject is described using the G/M-code.

[0035] For analyzing the analog emissions from the side-channels, four acoustic (AT2021 cardioid condenser and a contact microphone), one vibration (Adafruit triple-axis accelerometer), one magnetic (Honeywell’s magnetometer HMC5883L) and current (Pico current clamp) sensors are placed non-intrusively without hampering the normal operation of the system. These types of sensors are available in IoT devices. The placement of these sensors is performed by position exploration in Cartesian coordinate. The vibration and magnetic sensors measure signals in X, Y, and Z axis. Hence, in total there are four acoustic, three vibration, three magnetic, and a current sensors. We consider them as 11 separate channels. Analog emissions from the additive manufacturing system (or a 3D printer) are automatically collected using National Instruments Data Acquisition (NI DAQ) system whenever a print command is given to it.

[0036] The sample G/M-code (DTobject) comprises maximum of six parameters, G/M code specifying whether it is machine instruction or coordinate geometry information, travel feed-rate F of the nozzle head, the coordinates in XYZ-Axes each and amount of extrusion E. The parsing algorithm in this case separates the DTobject based on presence or absence of 5 of these parameters, G/M, X, Y, Z, and E. Hence,

^. _<r_S2) We consider that the PT_system and E _system implicitly affect the signals ^sa{^f), s_v(t), s_m{t), and sp(t^}_ Hence, an exhaustive list of PT system and E system need not be explored for the modeling. From each of these signals time and frequency domain features are extracted. Moreover, the ^^T biect= ,·,^. ·^,««} comprises a timestamp to segment and synchronize the features. For initial training phase, various G/M-code of the 3D-objects (e.g., cube, pyramid, cylinder, etc.) are given to 3D printer and their corresponding analog emissions are collected. From them, we proceed to generate the fingerprint library for maintaining the Digital Twin of the system.

[0037] FIG. 6 shows the experimental setup for sensor position exploration. One of the challenges in IoT sensor based information extraction is figuring out non-intrusive position of the sensors. This task may also be machine specific. In our experiment, the 3D printers’ external surface is considered for non-intrusively placing the sensors. Total of 28 uniform positions are selected.

[0038] For each of the positions, vibration, acoustic, and magnetic sensors are placed and data is collected for various DTobject parameters. Then a gradient boosted random forest is used to create a simple classifier to estimate the accuracy of the model based on various sensor location data. The accuracy of the classifier (Accuracy = - TruePosⁱtwes - ^, j_s taken

TruePositives+TrueNegative as a metric for determining the placement of the sensors around the 3D printer. Accuracy score of IoT sensor data is shown in FIG. 7. It may be noticed that for different positions the classification accuracy is different. Based on these values single position for each of the sensors are selected. However, since four acoustic sensors are used, positions with top four classification accuracy are selected for the sensor placement.

[0039] Various algorithms may be used for creating clusters to generate the fingerprint including, without limitation, Mini batch K-means, Spectral Clustering, Ward, Agglomerative Clustering, Birch, and the Gaussian Mixture method. For each of the clustering algorithm, varying numbers of clusters are initialized, and the corresponding silhouette coefficient is calculated for measuring the fitness of the features into these clusters.

[0040] The average silhouette coefficients of the clustering algorithms for all groups and channels are shown in FIG. 9, and the corresponding scatter plots for cluster number five is shown in FIG. 8. Note that, although the Agglomerative Clustering has a higher silhouette coefficient value from the scatter plot, the clusters are not well distributed in the scatter plot. However, the Birch clustering algorithm has relatively higher silhouette coefficient with better spread of the cluster centers. Hence, Birch algorithm is selected in this example for generating the clusters for fingerprinting the DTobject.

[0041] For testing the accuracy of the Digital Twin and for detecting the anomalous signals that can possibly cause deviation in the quality of the product, a specialized test 3D object is designed. FIG. 10 shows the test DTobject created using a CAD tool for checking anomaly localization capability of the Digital Twin.

[0042] Flowrate should be maintained for uniform deposition of the filament while printing in fused deposition modeling based 3D printers. However, due to sudden slippage, faulty filament, etc., the flow of the filament may deviate from its nominal value. The flowrate is a process specific parameter which is calculated as follows:

where W is the width and H is the height of the line-segment being printed on the XY-plane, Q is the constant volumetric flow rate of the material. Q is estimated based on die swelling ration, pressure drop value and buckling pressure of the filament vfeed is the feed velocity of the filament and is calculated as:

Vfeed ~ 0>_T * R_f (7) where cor is the angular velocity of the pinch rollers, and Rr is the radius of the pinch rollers. Based on these values, the pressure drop is calculated as follows:

where Pmotor is the pressure applied by the stepper motors, DR is the pressure drop. Hence, pressure applied by the motor needs to be maintained for the constant volumetric flow rate. However, this pressure needs to be less than buckling pressure calculated as follows:

where E is the elastic modulus of the filament, df is the diameter of the filament, and Lf is the length of the filament from the roller to the entrance of the liquefier present in the nozzle. Sudden change in the pressure can cause the uniform flow of the filament to be disrupted. For validating the application of the Digital Twin in anomaly localization, the flow rate is varied outside the optimal range (80% to 100%).

[0043] The Digital Twin comprises fingerprint for the optimal flowrate in its library, and the corresponding clusters of the individual channels. When the object is printed, the corresponding features are passed to the Digital Twin, and the silhouetted coefficients corresponding to the DTobject are calculated. Based on algorithm 400 (shown in FIG. 4), the analog emissions in each channel is labeled as either being within the deviation limit or exceeding the deviation limit of the silhouette coefficient. For selecting the optimal threshold for making this decision, initially the threshold is varied and the corresponding accuracy of the detection mechanism is measured. Corresponding to the varying threshold of the Receiver Operating Characteristic (ROC) curve for some of the channels is presented in FIG. 11.

[0044] Based on the highest accuracy acquired for each of the channels, the threshold is set and the corresponding classification accuracy for the segments that has been degraded is calculated. The accuracy of each channel in detecting the anomalous flowrate is shown in FIG. 12. Since the features are time stamped, the corresponding section of the DTobject may be calculated after the Digital Twin has marked the features to be anomalous.

[0045] From FIG. 12, it can be seen that analog emissions from microphone number four are more accurate in detecting the degradation of the flow rate. This is due to the fact that this emission is collected by the contact microphone attached near the extruder’s stepper motor. Moreover, the average accuracy across all the channels in detecting the anomalous flowrate is 83.093.

[0046] For detecting the degradation of the system, and hence the need for updating the Digital Twin, the flowrate for the entire DTobject is varied beyond the optimal range. From equations 6 - 9, it is evident that various mechanical degradation (e.g., worn out rollers), stepper motor degradation over time, etc., may cause the flowrate to be reduced over time. To check if the Digital Twin model gets updated to reflect the current status of the system, we perform two experiments. In the first experiment, the current Digital Twin with its fingerprint library is used to predict the class labels (0 for update and 1 for do not update) for the degraded flowrate (60%). Then based on the result of algorithm 2, the updated (or the old) Digital Twin is used to predict the class labels again for the same degraded flowrate (60%) to see if the Digital Twin gets updated again.

[0047] FIG. 13 presents a table with the results for degradation analysis. The table shows true negative rate, false positive rate and update decision taken for each channel for the old cluster. When the system degrades, we expect the Digital Twin to find higher negative labels being generated as the silhouette score will be lower than the average silhouette score stored for all the channels and groups. It can be seen that out of eleven channels four of them had the decision of not updating the cluster, and seven of them opting for updating the clusters. Hence, the clusters are updated by algorithm 400 (see FIG. 4). On the other hand, once the cluster have been updated, the analog emissions are labeled as true, hence we expect to see a higher true positive rate and a lower false negative rate. In the table shown in FIG. 13, it can be seen that only three of the channels gave the decision for updating the clusters again, however eight of them opted for not updating the cluster. This shows that the Digital Twin is able to update itself during degradation that causes emissions in multiple side-channels to vary.

[0048] For checking the accuracy of the Digital Twin in inferring the deviation of KPI(Kd), first of all gradient boosting based ensemble of regressors (random forest) is used to estimate the function k_d = /(. ) for the optimal flowrate range (80% to 120%). Then it is used to infer the thickness of the 3D object (shown in FIG. 10) with varying flowrates. The accuracy of the DTsystemsKPI inference model is measured using mean absolute error value.

[0049] The result of the KPI inference is shown in FIG. 14. At first, the mean absolute error value of the inference model trained with optimal flowrate range is measured. It can be seen in the figure that for optimal flowrate ranges the mean absolute error value is around 0.5 mm. Then, at each consecutive step the flowrate of the PTsystem is varied with step size of +10% in positive direction (> 120%) and at the same time +10% in negative direction (< 80%). It may be seen that in both directions when the system ages (degrades with increase or decrease in the flowrate), the DTsystem first has increase mean absolute error first. This is intuitive as the DTsystem has not been updated to the new fingerprint. However, once it has been updated the mean absolute error is lower. It may also be noticed that when the system degraded with flowrates at 160% and 50%, the wrong decision was taken by the algorithm 400 (see FIG. 4) in not updating the KPI inference model. Due to this, a large increase in mean absolute error was observed for the KPI for other DTobject KPI prediction. However, this faulty decision was recovered in the consecutive stages. Moreover, the average mean absolute error in predicting the KPI was 0.59 mm (calculated by averaging the mean absolute errors of the inference model after update decision).

[0050] Although the present invention focuses on building a living Digital Twin using IoT based sensors, there has been considerable amount of work in quality prediction in additive manufacturing or manufacturing systems in general. In this section, we provide a qualitative comparative study of various methods compared to the methodology described herein. The result of comparison is shown in the table presented in FIG. 15. It may be observed that there are three general categories of research effort in maintaining the quality. The first is the first principle based approach (simulation), where the model of the KPI based on the process and design parameters are performed, and the KPI deviation is predicted. These models although are accurate, they do not account for system degradation over time, and requires non-trivial formulation of physics based equations. The second category involves in-situ process monitoring methodologies. These methods monitor the process variation using high-end acoustic and piezoelectric sensors. Compared to these high-end sensor based methods, our method is able to keep the model updated using low-end sensor data for fault localization and KPI inference. The third category involves process monitoring using low-end sensor placement. They focus either on specific anomaly detection or KPI variation detection. However, these methods do not consider checking the aliveness (up-to-date model) of the model and are mostly limited to anomaly detection. Each of these techniques have their own merit, hence, the methodology described herein is not intended to function independently but in conjunction with various approaches to fully realize the concept of Digital Twin. [0051] To validate the methodology described herein to build the Digital Twin of the manufacturing system, flowrate was used to detect anomalous system behavior and overall system degradation behavior. However, in real life, there can be multiple PTsystem parameters that might affect the KPI. However, our methodology can be adjusted overtime to consider variation in other PTsystem parameters over time. Moreover, we have considered only dimension (thickness of a simple 30 object) as the key performance indicator; however, for building the full scale DTsystem, multiple KPIs may be used.

[0052] In the example experiment described above, a low sampling rate and resolution sensors for magnetic, vibration and power sensing were utilized. Additionally, in the experiment described above the number of sensors were limited as well. It should be understood that the number of sensors of resolution may be increased, as desired, to increase the accuracy of the Digital Twin. Moreover, in some embodiments, the methodology described herein may utilize IoT sensor arrays.

[0053] FIG. 16 illustrates an exemplary computing environment 1600 which may execute an application for performing quality inference from living digital twins in IoT-enabled manufacturing systems, according to some embodiments. The computing environment 1600 includes computer system 1610, which is one example of a computing system upon which embodiments of the invention may be implemented. Computers and computing environments, such as computer system 1610 and computing environment 1600, are known to those of skill in the art and thus are described briefly herein.

[0054] As shown in FIG. 16, the computer system 1610 may include a communication mechanism such as a system bus 1621 or other communication mechanism for communicating information within the computer system 1610. The computer system 1610 further includes one or more processors 1620 coupled with the system bus 1621 for processing the information. The processors 1620 may include one or more central processing units (CPUs), graphical processing units (GPUs), or any other processor known in the art.

[0055] The computer system 1610 also includes a system memory 1630 coupled to the bus 1621 for storing information and instructions to be executed by processors 1620. The system memory 1630 may include computer readable storage media in the form of volatile and/or nonvolatile memory, such as read only memory (ROM) 1631 and/or random access memory (RAM) 1632. The system memory RAM 1632 may include other dynamic storage device(s) (e.g., dynamic RAM, static RAM, and synchronous DRAM). The system memory ROM 1631 may include other static storage device(s) (e.g., programmable ROM, erasable PROM, and electrically erasable PROM). In addition, the system memory 1630 may be used for storing temporary variables or other intermediate information during the execution of instructions by the processors 1620. A basic input/output system (BIOS) 1633 contains the basic routines that help to transfer information between elements within computer system 1610, such as during start-up, may be stored in ROM 1631. RAM 1632 may contain data and/or program modules that are immediately accessible to and/or presently being operated on by the processors 1620. System memory 1630 may additionally include, for example, operating system 1634, application programs 1635, other program modules 1636 and program data 1637. The application programs 1635 may include, for example, the one or more executable applications for performing quality inference.

[0056] The computer system 1610 also includes a disk controller 1640 coupled to the system bus 1621 to control one or more storage devices for storing information and instructions, such as a hard disk 1641 and a removable media drive 1642 (e.g., compact disc drive, solid state drive, etc.). The storage devices may be added to the computer system 1610 using an appropriate device interface (e.g., a small computer system interface (SCSI), integrated device electronics, Universal Serial Bus (USB), or FireWire).

[0057] The computer system 1610 may also include a display controller 1665 coupled to the bus 1621 to control a display 1666, such as a liquid crystal display (LCD), for displaying information to a computer user such as anomaly detection alerts or other descriptive information related to production (KPIs). The computer system includes an input interface 1660 and one or more input devices, such as a keyboard 1662 and a pointing device 1661, for interacting with a computer user and providing information to the processors 1620. The pointing device 1661 may be, for example, a mouse or a pointing stick for communicating direction information and command selections to the processors 1620 and for controlling cursor movement on the display 1666. The display 1666 may provide a touch screen interface which allows input to supplement or replace the communication of direction information and command selections by the pointing device 1661.

[0058] The computer system 1610 may perform a portion or all of the processing steps of embodiments of the invention in response to the processors 1620 executing one or more sequences of one or more instructions contained in a memory, such as the system memory 1630. Such instructions may be read into the system memory 1630 from another computer readable medium, such as a hard disk 1641 or a removable media drive 1642. The hard disk 1641 may contain one or more datastores and data files used by embodiments of the present invention. Datastore contents and data files may be encrypted to improve security. The processors 1620 may also be employed in a multi-processing arrangement to execute the one or more sequences of instructions contained in system memory 1630. In alternative embodiments, hard-wired circuitry may be used in place of or in combination with software instructions. Thus, embodiments are not limited to any specific combination of hardware circuitry and software.

[0059] As stated above, the computer system 1610 may include at least one computer readable medium or memory for holding instructions programmed according to embodiments of the invention and for containing data structures, tables, records, or other data described herein. The term“computer readable medium” as used herein refers to any medium that participates in providing instructions to the processor 1620 for execution. A computer readable medium may take many forms including, but not limited to, non-volatile media, volatile media, and transmission media. Non-limiting examples of non-volatile media include optical disks, solid state drives, magnetic disks, and magneto-optical disks, such as hard disk 1641 or removable media drive 1642. Non-limiting examples of volatile media include dynamic memory, such as system memory 1630. Non-limiting examples of transmission media include coaxial cables, copper wire, and fiber optics, including the wires that make up the bus 1621. Transmission media may also take the form of acoustic or light waves, such as those generated during radio wave and infrared data communications.

[0060] When used in a networking environment, computer system 1610 may include modem 1672 for establishing communications over a network 1671 with components of the manufacturing system. In the example of FIG. 16, a 3D Printer 1680 is shown as one component that may be connected to the computer system 1610. Modem 1672 may be connected to bus 1621 via user network interface 1670, or via another appropriate mechanism. It should be noted that, although the 3D Printer 1680 is illustrated as being connected to the computer system 1610 over the network 1671 in the example presented in FIG. 16, in other embodiments of the present invention, the computer system 1610 may be directly connected to the 3D Printer 1680. For example, in one embodiment the computer system 1610 and the 3D Printer 1680 are co-located in the same room or in adjacent rooms, and the devices are connected using any transmission media generally known in the art.

[0061] Network 1671 may be any network or system generally known in the art, including the Internet, an intranet, a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a direct connection or series of connections, a cellular telephone network, or any other network or medium capable of facilitating communication between computer system 1610 and other computers (e.g., robot controller 1680). The network 1671 may be wired, wireless or a combination thereof. Wired connections may be implemented using Ethernet, Universal Serial Bus (USB), RJ-l l or any other wired connection generally known in the art. Wireless connections may be implemented using Wi-Fi, WiMAX, and Bluetooth, infrared, cellular networks, satellite or any other wireless connection methodology generally known in the art. Additionally, several networks may work alone or in communication with each other to facilitate communication in the network 1671.

[0062] The embodiments of the present disclosure may be implemented with any combination of hardware and software. In addition, the embodiments of the present disclosure may be included in an article of manufacture (e.g., one or more computer program products) having, for example, computer-readable, non-transitory media. The media has embodied therein, for instance, computer readable program code for providing and facilitating the mechanisms of the embodiments of the present disclosure. The article of manufacture can be included as part of a computer system or sold separately.

[0063] While various aspects and embodiments have been disclosed herein, other aspects and embodiments will be apparent to those skilled in the art. The various aspects and embodiments disclosed herein are for purposes of illustration and are not intended to be limiting, with the true scope and spirit being indicated by the following claims.

[0064] Unless stated otherwise as apparent from the following discussion, it will be appreciated that terms such as “applying,” “generating,” “identifying,” “determining,” “processing,”“computing,”“selecting,” or the like may refer to the actions and processes of a computer system, or similar electronic computing device, that manipulates and transforms data represented as physical (e.g., electronic) quantities within the computer system's registers and memories into other data similarly represented as physical quantities within the computer system memories or registers or other such information storage, transmission or display devices. Embodiments of the methods described herein may be implemented using computer software. If written in a programming language conforming to a recognized standard, sequences of instructions designed to implement the methods can be compiled for execution on a variety of hardware platforms and for interface to a variety of operating systems. In addition, embodiments of the present invention are not described with reference to any particular programming language. It will be appreciated that a variety of programming languages may be used to implement embodiments of the present invention.

[0065] An executable application, as used herein, comprises code or machine readable instructions for conditioning the processor to implement predetermined functions, such as those of an operating system, a context data acquisition system or other information processing system, for example, in response to user command or input. An executable procedure is a segment of code or machine readable instruction, sub-routine, or other distinct section of code or portion of an executable application for performing one or more particular processes. These processes may include receiving input data and/or parameters, performing operations on received input data and/or performing functions in response to received input parameters, and providing resulting output data and/or parameters.

[0066] A graphical user interface (GUI), as used herein, comprises one or more display images, generated by a display processor and enabling user interaction with a processor or other device and associated data acquisition and processing functions. The GUI also includes an executable procedure or executable application. The executable procedure or executable application conditions the display processor to generate signals representing the GUI display images. These signals are supplied to a display device which displays the image for viewing by the user. The processor, under control of an executable procedure or executable application, manipulates the GUI display images in response to signals received from the input devices. In this way, the user may interact with the display image using the input devices, enabling user interaction with the processor or other device.

[0067] As used herein, the term“component” can refer to either or both of: (i) a software component that causes an electronic device to accept various inputs and generate certain outputs; or (ii) an electronic input/output interface, such as a panel, frame, textbox, window or other portion of a GUI.

[0068] The functions and process steps herein may be performed automatically or wholly or partially in response to user command. An activity (including a step) performed automatically is performed in response to one or more executable instructions or device operation without user direct initiation of the activity.

The system and processes of the figures are not exclusive. Other systems, processes and menus may be derived in accordance with the principles of the invention to accomplish the same objectives. Although this invention has been described with reference to particular embodiments, it is to be understood that the embodiments and variations shown and described herein are for illustration purposes only. Modifications to the current design may be implemented by those skilled in the art, without departing from the scope of the invention. As described herein, the various systems, subsystems, agents, managers and processes can be implemented using hardware components, software components, and/or combinations thereof. No claim element herein is to be construed under the provisions of 35 U.S.C. 112(f) unless the element is expressly recited using the phrase “means for.”

Claims

CLAIMS We claim:

1. A method for quality inference in an Internet of things (IoT)-enabled manufacturing system, the method comprising:

generating a fingerprint library comprising a plurality of fingerprints, wherein each fingerprint corresponds to a digital twin of a product and comprises (i) a plurality of stored feature clusters; and (ii) a stored silhouette coefficient for each feature cluster;

collecting a plurality of sensor values from a plurality of IoT sensors monitoring a physical twin of the manufacturing system;

extracting a plurality of new feature values from the plurality of sensor values;

identifying a plurality of parameters defining the digital twin of the product;

segmenting the new feature values into a plurality of new feature subgroups based on the parameters of the digital twin of the object;

performing an anomaly detection process comprising:

estimating new feature clusters for the new feature subgroups using the fingerprint library;

determining a current silhouette coefficient for each new feature cluster;

if the current silhouette coefficient for a new feature cluster deviates from a corresponding stored silhouette coefficient, (a) identifying the parameters of the digital twin of the product corresponding to the new feature cluster and (b) designating the parameters as being potentially anomalous; and

providing an alert to one or more users identifying the parameters of the digital twin designated as being potentially anomalous.

2. The method of claim 1, further comprising:

during the anomaly detection process, determining a count of parameters of the digital twin designated as being potentially anomalous;

if the count of parameters exceeds a predetermined parameter threshold, updating the fingerprint library with the new feature clusters and the current silhouette coefficient.

3. The method of claim 2, wherein (a) the plurality of parameters defining the digital twin are segmented into a plurality of groups; (b) the count of parameters of the digital twin designated as being potentially anomalous is determined on a group-by-group basis; and (c) the fingerprint library is only updated if the total number of groups with the count of parameters exceeding the predetermined parameter threshold exceeds a predetermined group threshold.

4. The method of claim 3, wherein (a) the plurality of parameters defining the digital twin is received across a plurality of channels; (b) the count of parameters of the digital twin designated as being potentially anomalous is further determined on a channel-by-channel basis; and (c) the fingerprint library is only updated if:

the total number of groups with the count of parameters exceeding the predetermined parameter threshold exceeds the predetermined group threshold and

the total number of channels with the count of parameters exceeding the predetermined parameter threshold exceeds a predetermined channels threshold.

5. The method of claim 1, further comprising:

estimating a key performance indicator (KPI) inference function based on (a) the plurality of parameters defining the digital twin of the product, (b) a plurality of parameters representing parameters of the physical twin of the manufacturing system, and (c) a plurality of parameters representing environmental factors affecting the manufacturing system;

determining a deviation of one or more KPIs using the KPI inference function; and providing a notification to a user of the deviation of one or more KPIs.

6. The method of claim 5, wherein the KPI inference function is estimated using a gradient boosting-based regressor.

7. The method of claim 1 , wherein the parameters defining the digital twin of the product are represented in G-code or M-code initialized from a Computer Aided Manufacturing (CAM) tool.

8 The method of claim 1, further comprising; prior to segmenting the new feature values, performing Principal Component analysis on the new feature values to reduce the dimension of the new feature values.

9. The method of claim 1, wherein the manufacturing system is an additive manufacturing system comprising a 3D printer and the product is an object printed by the 3D printer.

10. An article of manufacture for quality inference in an IoT-enabled manufacturing system, the article of manufacture comprising a non-transitory, tangible computer-readable medium holding computer-executable instructions for performing a method comprising:

retrieving a fingerprint library comprising a plurality of fingerprints, wherein each fingerprint corresponds to a digital twin of a product and comprises (i) a plurality of stored feature clusters; and (ii) a stored silhouette coefficient for each feature cluster;

collecting a plurality of sensor values from a plurality of IoT sensors monitoring a physical twin of the manufacturing system;

extracting a plurality of new features values from the plurality of sensor values;

identifying a plurality of parameters defining the digital twin of the product;

segmenting the new feature values into a plurality of new feature subgroups based on the parameters of the digital twin of the object;

performing an anomaly detection process comprising:

estimating new feature clusters for the new feature subgroups using the fingerprint library;

determining a current silhouette coefficient for each new feature cluster;

if the current silhouette coefficient for a new feature cluster deviates from a corresponding stored silhouette coefficient, (a) identifying the parameters of the digital twin of the product corresponding to the new feature cluster and (b) designating the parameters as being potentially anomalous; and

creating a record identifying the parameters of the digital twin designated as being potentially anomalous.

11. The article of manufacture of claim 10, wherein the method further comprises: during the anomaly detection process, determining a count of parameters of the digital twin designated as being potentially anomalous;

if the count of parameters exceeds a predetermined parameter threshold, updating the fingerprint library with the new feature clusters and the current silhouette coefficient.

12. The article of manufacture of claim 11, wherein (a) the plurality of parameters defining the digital twin are segmented into a plurality of groups; (b) the count of parameters of the digital twin designated as being potentially anomalous is determined on a group-by-group basis; and (c) the fingerprint library is only updated if the total number of groups with the count of parameters exceeding the predetermined parameter threshold exceeds a predetermined group threshold.

13. The article of manufacture of claim 12, wherein (a) the plurality of parameters defining the digital twin is received across a plurality of channels; (b) the count of parameters of the digital twin designated as being potentially anomalous is further determined on a channel-by- channel basis; and (c) the fingerprint library is only updated if:

the total number of groups with the count of parameters exceeding the predetermined parameter threshold exceeds the predetermined groups threshold and

the total number of channels with the count of parameters exceeding the predetermined parameter threshold exceeds a predetermined channels threshold.

14. The article of manufacture of claim 10, wherein the method further comprises:

estimating a KPI inference function based on (a) the plurality of parameters defining the digital twin of the product, (b) a plurality of parameters representing parameters of the physical twin of the manufacturing system, and (c) a plurality of parameters representing environmental factors affecting the manufacturing system;

determining a deviation of one or more KPIs using the KPI inference function; and providing a notification to a user of the deviation of one or more KPIs.

15. The article of manufacture of claim 14, wherein the KPI inference function is estimated using a gradient boosting-based regressor.

16. The article of manufacture of claim 10, wherein the parameters defining the digital twin of the product are represented in G-code or M-code initialized from a Computer Aided

Manufacturing (CAM) tool.

17. The article of manufacture of claim 10, wherein the method further comprises:

prior to segmenting the new feature values, performing Principal Component analysis on the new feature values to reduce the dimension of the new feature values.

18. The article of manufacture of claim 10, wherein the manufacturing system is an additive manufacturing system comprising a 3D printer and the product is an object printed by the 3D printer.

19. A system for quality inference in an IoT-enabled manufacturing system, the system comprising:

a database storing a fingerprint library comprising a plurality of fingerprints, wherein each fingerprint corresponds to a digital twin of a product and comprises (i) a plurality of stored feature clusters; and (ii) a stored silhouette coefficient for each feature cluster;

a plurality of IoT sensors generating a plurality of sensor values corresponding to a physical twin of the manufacturing system;

a computing system comprising one or more processors configured to:

extract a plurality of new feature values from the plurality of sensor values; identify a plurality of parameters defining the digital twin of the product;

segment the new feature values into a plurality of new feature subgroups based on the parameters of the digital twin of the object;

perform an anomaly detection process comprising:

estimating new feature clusters for the new feature subgroups using the fingerprint library;

determining a current silhouette coefficient for each new feature cluster; if the current silhouette coefficient for a new feature cluster deviates from a corresponding stored silhouette coefficient, (a) identifying the parameters of the digital twin of the product corresponding to the new feature cluster and (b) designating the parameters as being potentially anomalous; and

create a record identifying the parameters of the digital twin designated as being potentially anomalous.