Real-Time Quality Control in Thin Glass Forming Using Infrared Thermography and Deep Learning

Towards the growing trends in lightweight, flexible, and optical advantages, thin glasses become key components in numerous applications such as consumer electronics like foldable smartphones, or automotive interiors. Nonisothermal glass molding promises a viable technology for the cost-efficient production of precision glass components. In the existing production, the quality of the glass products can only be accessed at the end of the hot forming process. Due to high rates of product failures often appeared in the precision glass molding processes, the current quality control of the produced optical products suffers low process efficiency. This work introduces an enabling approach for monitoring the product quality in real-time using thermography and machine learning. Specifically, the acquisition of the temperature fields of the glass components during the hot forming stage enabled by an infrared thermographic camera allows machine learning to predict the final shape of the molded components at the end of the forming process. Several transfer learning models have been investigated to demonstrate the proposed method. To further enhance the prediction performance, self-built convolutional neural network models were developed using different types of image data. By incorporating the time-series image data as an input to the learning models, the prediction performance was achieved. The model built in the present work demonstrates an excellent prediction accuracy where the difference between the measured and predicted shapes of the glass products can be kept at low double-digit micrometers. Such accuracy achieved by our self-developed machine learning model promisingly satisfy the quality control in serial productions of numerous precision optical glass components in automotive and consumer electronics sectors.


Introduction
The automotive industry is undergoing a rapid transformation. Technological advancements for autonomous driving, efficient energy consumption, as well as increasingly safe and affordable driving experiences are shaping the future of automobiles. Today, many concept cars strive for making the future automobiles lighter, safer, and smarter. This fact pushes automakers to innovate lighter, tougher, and more optically and functionally advanced automotive interiors and exteriors [1]. Due to its mechanical and optical advantages, thin glass holds a key material for such inevitable innovation [2].
Glass components for the automotive future are setting ever-increasing demands for glass manufacturers towards highly precision and geometrically complex but low-cost products. Fabrications of brittle glass components with high precision at large volumes essentially require advancements in glass processing technology. In fact, the most common method via grinding and polishing is not applicable for thin glass due to the unavoidable deformation or cracks caused by thermo-mechanical stress on the glass surface and between the glass and clamping parts [3]. Instead, a replicative manufacturing, namely Nonisothermal Glass Molding (NGM), has been developed and becomes a viable technology for the cost-efficient production of glass optics [4,5]. In more recent years, the nonisothermal molding principle has been extended for thin glass forming by applying suitable thermal-mechanical loading means such as thermal slumping, vacuum pressure, press bending, and deep drawing [6]. The implementation of various loading enablers allows this newly developed glass forming process for the production of complex, precision, [7][8][9] and microstructureintegrated optical components [10][11][12][13].  1 introduces the process chain of thin glass forming using vacuum assistance. The process starts with a heating step of the glass and mold parts. As soon as the glass temperature is sufficient for forming, vacuum pressure is applied, and the glass part is deformed into the mold cavity to a desired shape in a couple of seconds. Afterwards, it is demolded and then undergoes an annealing step where its temperature is slowly decreased to room temperature in order to release the internal stress. After annealing, the molded component is ready-to-use and requires no further post-processing steps such as grinding or polishing. The entire molding process of a glass unit takes place in a few minutes allowing a cost-efficient solution for serial production in optical glass industry.
In the existing production, accuracy and possible glass defects can only be realized at the end of the forming process by relevant off-line measurements when the molded components are entirely cooled down at room temperature [14][15][16]. It is, however, emphasized that imperfections of the molded components such as form deviation, chill ripples, or glass cracks are primarily driven by the molding step [17][18][19][20]. Those are the common defects observed in the nonisothermal molding process, mainly resulted from the high heat exchanges at the glass-mold interface [21,22] as well as the complex thermo-viscoelastic material behaviors of glass at elevated molding temperatures [23][24][25]. In serial production, if the defects are not detected during the molding step, it commonly results in a large number of glass rejects and high energy cost vain to anneal the glass failures [26]. Accordingly, the glass optic manufacturing industry is steadily raising demands on real-time quality control in the hot forming of thin glasses.
This work presents an enabling approach for the real-time monitoring and quality control of the glass components during the molding process. The essence of this research is based on the hypothesis stating that there exists an underlying relationship between the form deviation and temperature fields of the glass components. To demonstrate, an infrared (IR) thermographic camera was employed to record the temperature fields of the glass components right after the molding step. In the following, image processing techniques were carried out to extract the temperature data and essential features that eventually enable the discovery of the relationship between the temperature field and the final shape of the molded glass. To this end, we implemented machine learning (ML) and established Convolution Neural Networks (CNN) to accomplish the relationship. By using the time-series temperature images after the molding step, the CNN model developed by this study reveals an excellent prediction accuracy of the final glass shapes after annealing. The findings make it possible for the real-time quality control in glass molding process by exploiting the IR-thermographic and machine learning techniques.

Methodology
Experimental procedure and data acquisition. To demonstrate our solution approach for the enabling real-time quality control in thin glass forming, a display mirror was chosen as an optics demonstrator (Fig. 2a). For this demonstrator, vacuum pressure was applied to deform glass plates. Having introduced in Fig. 1, the glass plate and mold system were heated up, and then vacuum was applied when glass reached a temperature sufficient for molding. After molding, the deformed glass was quickly driven out from the heating furnace where its temperature was recorded for 10 seconds by the IR-camera (Fig. 2b) before it is transferred to the annealing oven. The VarioCAM ® HD Head produced by InfraTec was used to measure the glass temperature. The camera lens firstly collects the IR radiation emitted from the measuring objects at the same time and then reproduces the thermal radiation on the detector elements. The detector absorbs this radiation in the spectral range from 7.5 to 14 μm. The temperature change gathered by the detector, thereof, results in a signal that can be analyzed electronically. This camera enables the measurement accuracy of ±1 °C. A full-frame rate of 30 Hz with high resolution of 1.024 × 768 pixels was used for the temperature measurement [27]. A total of 121 molding experiments was conducted to produce the data for this study. The experiments consisted of different sets of the molding parameters including glass temperature, mold temperature, vacuum pressure, and holding time while applying vacuum. For each experiment, with the sampling rate 30 Hz and the recording time of 10 seconds, a total of 300 images were collected, accordingly. Fig. 3(a) introduces the time-series images showing the temperature fields of the molded glass components. Goal of this research is to utilize the time-series images collected by the IR-camera during the recording period for predicting the final shape of the molded glass mirrors using machine learning. In other words, by relying on the IR-images collected directly after the molding step makes, it is possible to control the product quality in real-time.

Achievements and Trends in Material Forming
Data pre-processing. The images captured by the IR-camera were given as inputs to the machine learning models. However, a pre-processing of those images (raw data) is necessary before they can be used sufficiently in the learning models. As observed in Fig. 3(a), the raw IR-images indicate the presence of surrounding noise in the vicinity of the glass specimen, which needs to remove so that only the temperature data of glass were taken in the inputs. To this end, Canny Edge Detection algorithm was employed to detect the glass boundary. A mask was generated for each image that contains information about the position of the glass specimen in the image. Subsequently, masking was done on the original image which was then followed by cropping. At the end of pre-processing step, the image was transformed to the final resolution of 500 × 500 pixels from the initial resolution of 768 × 1024 pixels. The image was resized as per the requirement of the CNN model. This procedure can be visualized in Fig. 3(b), and the input data for the learning models are shown in Fig. 3(c).
Furthermore, the final shapes of the molded glass optics were measured after the annealing step by employing a tactile measuring device. The resulting shape measurement contains 69,000 data points which was reduced to 100 data points by means of a standardization procedure. The standardization algorithm included end trimming, rotation, offset elimination and finally size reduction through cubic spline interpolation. The measured shape with reduced data points used as output vectors is shown in Fig. 4. Both the input images and the output vectors were normalized to values between 0 and 1 to obtain optimum performance from the CNN models. Then, the dataset was split into a training set and test set, each of which has 80% and 20% of the total data, respectively. In addition, a validation set containing 20% of the training dataset was given to control the overfitting nature of the model.

Selection of learning models.
Machine learning algorithms for any domain are typically selected by considering the type of input and output data, and the size of the dataset. A convolutional neural network model is a commonly used algorithm when the input data is an image due to its ability to learn the spatial relationship between different data points. The performance of the CNN model is mainly dependent on the network architecture and the dataset. In this work, three approaches were considered to build the predictive models. The first approach was the use of transfer learning where the feature representation from a CNN model trained on a larger dataset in another domain can be reused for our domain dataset [28,29]. The rationale of this approach is that the lower layers of the CNN model trained on a very large dataset mostly detect generic features [30,31], which can be used for other domains as well. Secondly, a self-built CNN model architecture was targeted. The topology of the model was chosen with the help of a hyperparameter tuning algorithm such that the validation error of the model with optimum hyperparameters is minimum. It is noted that all aforementioned models took only one single frame as the input, meaning that the time-series data were not considered in developing those models. In this study, the first frame was chosen. Lastly, we developed a 3D CNN model where all images as time-series data collected over the entire data recording period were taken in the input.
Transfer learning models. Five CNN models that are pre-trained on the ImageNet dataset [32] were chosen in this work with taking their accuracy on the ImageNet dataset and their model complexity into account. All the models are available in the Keras API of TensorFlow. The fully connected layers of the pre-trained models were not included; instead, randomly initialized dense layers were added to the model. The weights of the pre-trained layers were frozen and only the weights in the additional dense layers were trained during the training process. The hyperparameters of the model such as the number of fully connected layers, the number of neurons, and the activation function were decided by searching the entire hyperparameter space. The parameters of each model are given in Table 1.  (CNN-2D), consists of repeating convolution modules followed by fully connected layers (Fig. 4). Each convolution module consists of two 2D convolutional layers and a 2D max-pooling layer. The input to this model was a two-dimensional tensor containing the grayscale image of the temperature data. The model hyperparameters such as filter size or pool size were chosen through the randomized search algorithm. Each model was trained for 100 epochs and the model with the best 5-fold cross-validation error was selected. The hyperparameter grid as well as the optimum parameters are given in Table 2.  3D CNN model. During the data recording period, the IR-camera generated 300 frames of thermal images per experiment. The 3D CNN model (CNN-3D) was built by taking the entire serial frames over the recording time as the output. Due to the heat exchanges to environment during the recording time, each frame, corresponding to a time interval, has a different temperature field. The evolution of temperature distribution for each molded glass mirror was also different due to the individual set of parameters chosen for each experiment. Based on the assumption that the evolution of temperature distribution influences on the final shape of the glass products, we built a three-dimensional input Achievements and Trends in Material Forming tensor for each experiment where the third dimension refers to the time or, in other words, the frame number (Fig. 3c). A similar architecture to CNN-2D was built, except 3D convolutional layers and 3D max-pooling layers were employed here instead. Since it is time-intensive for training CNN-3D, the use of a random search algorithm for hyperparameter tuning is not feasible. For this reason, the hyperparameters were determined through the hyperband algorithm from the KerasTuner library.
Hyperband is the algorithm that adopts random search with adaptive resource allocation techniques such that the only promising models are trained for a higher number of epochs [33]. The resulting optimum hyperparameters are indicated in Table 2.
All the models are trained with the Adam optimization method [34] with a learning rate of 0.001 and using mean squared error as the loss function [35,36]. Overfitting of the models was prevented by the early stopping method [37].

Results and Discussions
All models were implemented in Python and TensorFlow. The training and evaluation of each model were repeated by 20 times to account for their variability. Performance of training and testing set of those models evaluated by root mean squared error (RMSE) is presented in Fig. 5(a-b). Of those models, DenseNet-169 revealed the best performance because of its least mean and variance of errors. By computing the difference between the prediction results of the DenseNet-169 and the experimentally measured shapes for the test set, an average deviation of 58.45 micron was achieved, which is acceptable for the form deviation required for many of the optical display components produced by the glass molding processes. The shape prediction enabled by the DenseNet-169 and the measured shape for one molded glass mirror are demonstrated in Fig. 5(c). Datapoint index [-] (c)

Measured shape
Predicted shape RMSE = 0.0048 RMSE [1] It was observed that MobileNet produced the second best generalization error on the test set, followed by ResNet50. It is emphasized that despite being trained only on a small dataset, CNN-2D is able to outperform other pre-trained models, demonstrating the power of the self-built CNN model developed in this study. Furthermore, besides its sufficiently accurate prediction, the CNN-2D has significantly low model complexity and consequently training time, making it more practical for realtime applications over either DenseNet or MobileNet.
As CNN-2D was able to provide well prediction accuracy with the first frame of the thermal image data, it raises a question whether the choice of other frames has any effect on the model performance.
To answer it, CNN-2D was trained individually using the 1 st , 150 th and 300 th frame. The result of the model evaluation for different frames is exhibited in Fig. 6. We observed that the model has comparable means and standard deviations of error regardless of the chosen frame number. Hence, it can be concluded that the choice of the frame is not critical. Finally, this study aims to discern whether the incoporation of the time-series image data is beneficial for further enhancement of the shape prediction. For this purpose, all 300 frames collected by the IR-camera over the entire recording period were taken as inputs to CNN-3D. Performance of this model, in comparison with to CNN-2D, is presented in Table 3. Overall, CNN-3D enables lower mean and minimum errors achieved by both training and test sets, implying that it has better generalization ability than CNN-2D. This finding clearly points out that the time-series image data contain more informative correlations between the temperature and the final product shape than a single frame image. During the recording period, the molded glass components undergo continuous thermal exchanges with surrounding environment, and the process of temperature drops due to the heat loss to the environment decisively influences on the final shape. The availability of the data during this cooling period prompts the model to better learn the influences of the environment on the final shape.

Achievements and Trends in Material Forming
However, it is noteworthy pointing out that the use of time-series data significantly increases the model complexity and consequently long training process. In addition, we observed high variance of the CNN-3D model, indicating that its performance is highly sensitive to its initial weights and stochastic nature of the training [38]. Therefore, depending on the targeted accuracy of the final products, relevant learning models should be decided by compromising the prediction accuracy of the model and time required for the training process.

Summary
This work demonstrates an enabling approach for real-time quality control in the production of precision optical glass components using the infrared thermographic technique and machine learning. The acquisition of temperature fields during the molding stage enabled by the IR-camera allows us to immediately realize the final shape of the molded glass components at the end of the hot forming process. The prediction of the final shape is permitted in real-time by the implementation of machine learning with relevant convolution neural network architectures. Based on the self-built learning models, we demonstrated that the difference of low two micro digits between the predicted and experimentally measured shapes is promised. The prediction accuracy achieved by developed machine learning model is typically sufficient for quality control of numerous precision components in the optical glass markets today. Therefore, the proposed method is highly promising for industrial applications to reduce the efforts for measuring the glass products after hot forming process, to increase the process efficiency, and to enable the process automation.