Performance analysis of machine learning for fault diagnosis of ballast water treatment system

2.1 Fault diagnosis

Industrial facilities are connected to building systems, with some facilities having complex arrangements of various complex devices. If a device malfunctions, the performance of the entire system may be affected, and the system may undergo damage. Monitoring techniques for devices and facilities are continuously being developed to prevent such incidents. Such techniques utilize various types of sensors to diagnose faults and predict device lifespans. Such techniques are collectively referred to as prognostic and health management (PHM) [10].

The PHM technique consists of four components. The components are as follows: (1) data sensing, in which the characteristics of facilities are analyzed, and sensor systems are used to detect abnormal conditions; (2) preprocessing and feature generation, in which statistical and physical characteristics related to faults are generated from the data collected during data sensing; (3) diagnosis, in which normal and fault conditions are determined based on the features generated during preprocessing and feature generation; (4) prognosis, which involves predicting the lifespan of the system or equipment in terms of the time until the next fault occurs.

The data used in this study were collected from five types of sensors and measured in intervals of 1 s at the BWTS “GloEn-Patrol” facility manufactured by Panasia Co., Ltd using the UV projection method. Therefore, the data generation step of the PHM process was not required. Feature generation through window-size-driven transformation and diagnosis was performed using six machine learning models. Table 1 lists the five types of sensors used for data acquisition.

Table 1:

Types of sensors used for data sensing

2.2 SVM

SVM is an algorithm that performs binary classification by determining the optimal linear decision boundary [8]. A decision boundary is a hyperplane with the maximum margin between the data labels. The maximum margin is the maximum Euclidean distance between the nearest data and the hyperplane discovered by the SVM. The hyperplane is expressed by Equation (1), where x is the x-axis value on the hyperplane, w^T is the gradient of the decision boundary, and b is the maximum margin.

Binary classification is conducted to identify the maximum margin, and learning is performed to achieve the correct classification of new data (u₁) based on the value of Equation (2), which multiplies the classification target of Equation (1).

Multiclassification is performed by one-versus-rest (OvR) for a decision boundary for each label type or by one-versus-one (OvO) for the binary classification decision boundary in all cases [11]. The most significant difference between OvR and OvO is the number of decision boundaries. Because the OvR method has only one decision boundary, one vector is classified into only one class in the vector space. The OvO method constructs decision boundaries corresponding to each pair of existing classes. Therefore, a vector can be divided into two or more classes according to the decision boundary. The final classification result of the OvO method is determined by the class most classified by all decision boundaries.

2.3 Logistic Regression

Logistic regression is a binary classification algorithm that outputs a probability between zero and one using a logistic equation and a latent variable [12]. Logistic regression differs from SVM primarily in that logistic regression explores decision boundaries using logistic equations. The hyperplane is determined using Equation (3), where $$ \overrightarrow{x}$$ is the input vector, Y denotes the classified class of $$ \overrightarrow{x}$$, and $$ \overrightarrow{{\beta }^T}$$ is the gradient vector corresponding to the values inside $$ \overrightarrow{x}$$.

Binary classification is performed based on if the right side of Equation (3) is less or greater than zero.

Multiclassification is performed by subtracting the probability that the data correspond to a specific label from 1 using the OvR method of SVM.

2.4 k-Nearest Neighbors (k-NN)

A k-NN algorithm is a machine learning algorithm that performs classification based on the Euclidean distance between each data point using a user-defined constant k [13]. Each labeled multidimensional characteristic space vector is used as the learning data element. The Euclidean distance for each data element is a k label based on the user-defined constant k.

2.5 Multilayer Perceptron

A perceptron is a basic neural network model whose workings are based on neuronal activities. It is an algorithm that produces an output from multiple inputs [14]. Each perceptron outputs a value of one if the weighted sum of the inputs exceeds a given threshold; otherwise, the output value is zero.

A multilayer perceptron is a structure with several layers stacked to ensure that the output of a perceptron is used as an input into another perceptron. When a multilayer perceptron is used in a multiclassification task, as many perceptrons as the number of labels in the last layer (zero and one are output according to each label) are arranged.

2.6 Random Forest

A random forest is an ensemble machine learning model based on a decision tree algorithm [15] that prevents the overfitting of decision tree tasks. It completes the process of bootstrap aggregation, in which the number of learning data elements for generating each decision tree and its features are selected as a subset of the total number.

decision tree algorithm proceeds with learning in the direction in which the information gain is maximized for each branch node. The information gain is determined using Equation (4), where S is the data set arriving at a node, S^f is the data set entering the left or right child node of the S node, and H(S) is the Shannon entropy of the S data set.

This value is obtained by subtracting the entropy value of the child nodes from the entropy value of the parent node. Maximizing information gain refers to learning in a direction that reduces the complexity of the classification result as the number of tree branches increases.

2.7 eXtream Gradient Boosting (XGBoost)

XGBoost is an ensemble machine learning model based on a boosting method, unlike the random forest algorithm, which is based on the bootstrap aggregation method. In contrast to bootstrap aggregation, boosting uses a decision tree as a weak classifier and reteaches the following decision tree by assigning weights to data elements that have failed to be classified precisely [16]. The objective function is expressed by Equation (5), and minimal learning should proceed.

In Equation (5), Ω is the weight function used for regularization to prevent overfitting, u_t is the correct answer class of the input data, $$ u_t^{\mathrm{\text{'}}}$$ is the class predicted by the model, and the l function is the loss function for the difference between u_t and $$ u_t^{\mathrm{\text{'}}}$$.

3.1 Data preprocessing

The machine learning models used in this study were classified according to the algorithm formula of each model. However, if significant differences exist between the input feature values, classification performance may deteriorate, or overfitting may occur. For the sensor data, the range of values calculated for the sensors may vary significantly, depending on the fields or units measured using the sensors. Table 2 list an excerpt from typical sensor data.

Table 2:

Parts of acquired normal data

The #S F_DP sensor had a value ranging between 0.12 and 0.26, but the #S FLOW sensor had a value ranging between 478.0 and 513.0 (Table 2). If the difference is used as learning data for the machine learning model without scaling, performance degradation or overfitting may occur.

Methods for adjusting the numerical ranges include standard scaling and minmax scaling [4]. In this study, the sensor data were scaled to floating-point numbers between zero and one using minmax scaling, as expressed by Equation (6).

In Equation (6), x is the raw data value measured by a sensor at a specific time, and x_max and x_min are the maximum and minimum values, respectively, measured by the sensor throughout the entire period.

3.2 Feature generation

Time-series data were collected repeatedly for a specific process over a particular period. A typical deep learning model that uses time-series data is the RNN, which uses all data input before each learning step. A feature is generated to predict the current situation using a number n of previous observations, considering the specificity of the industrial sites. This approach is in contrast to neural network-driven models that require extensive learning resources.

Suppose a row of sensor data is x. All the acquired time-series sensor data can be represented by (x₁, x₂, …, x_n). Because of the characteristics of time-series data, using only one row x for learning may adversely affect the performance of machine learning models. Therefore, in this study, the data for the stage belonging to a window were generated as new features while moving the window, whose size was fixed at values ranging from 1 to 10. Window refers to the continuous number of x to be used as the input data. For example, if the window size was set to 2, features were generated in the form of [(x₁, x₂), (x₂, x₃), …, (x_n-2, x_n-1), (x_n-1, x_n)], and if the window size was expanded to 3, the features were generated in the form of [(x₁, x₂, x₃), (x₂, x₃, x₄), …, (x_n-2, x_n-1, x_n)].

In this feature-generation method, an effective window size analysis is necessary for each machine learning model. As the window size increases, the time-series data characteristics improve, and the data size used by the machine learning models also increases. Therefore, the classification accuracy is evaluated according to the window size.

3.3 Model training

Unlike in previous studies in which fault diagnosis was performed using SVM only, five additional machine learning models were used in this study. Table 3 lists the machine learning models used for fault diagnosis.

Table 3:

Machine learning models applied to fault diagnosis system

The BWTS fault diagnosis task is converted into a multi-classification process that produces 11 labels as outputs, that is, the normal state and 10 fault states for five types of sensor data from the machine learning models. Scikit-learn [17], a module of the Python programming language, was used to implement, learn, and evaluate each model, and default values were borrowed for all model parameters.

SVM and LR models use the OvO (R) method to perform multiclassification. OvO (R) is a method that splits multiple classifications into all available binary classifications. A k-NN model borrows labels for the k-nearest data by measuring the Euclidean distance between each learning data element. Classification is preceded by designating the largest number of labels as the labels for the step data. Perceptron produces binary outputs, such as SVM and LR models. Therefore, MLP sets the number of perceptrons in the last layer equal to the number of labels to perform multiclassification tasks. Each perceptron is allocated to a label and learns as many binary classifications as the number of labels. The RF model performs ensemble learning using the bootstrap aggregation method on multiple decision trees. Each decision tree is trained by dividing the learning data and features into multiple subsets. The XGBoost model is an ensemble model that uses a boosting method on multiple decision trees. Unlike bootstrap aggregation, which teaches multiple decision trees at once, the boosting method trains only one decision tree at a time, weighing nodes of the tree that fails classification and then trains the next decision tree. Therefore, a single strong classifier is generated by the multiple decision trees of an ensemble, which performs the role of weak classifiers.

3.4 Fault diagnosis

The trained machine learning models use evaluation data to diagnose the normal state and 10 fault states based on each data state. Table 4 presents the state of the BWTS based on the labels and descriptions.

Table 4:

Status and description of diagnosed BWTS by label

4.1 Experimental setup

The BWTS sensor data are crucial for learning and evaluating machine learning models. Figure 2 shows the UV unit of the “GloEn-Patrol” BWTS facility of Panasia Co., Ltd., which provided the sensor data used in this study.

Figure 2:

UV unit of “GloEn-Patrol” BWTS of Panasia Co., Ltd

Out of the 47,430 sensor data elements acquired, 45,911 were normal data, and 1,519 were fault data. These data reflect an imbalance, that is, a very low proportion of fault data to normal data. Thus, the proportions of the normal and fault data in the learning and evaluation data sets were set differently for each label to reduce the imbalance and improve the suitability of the evaluation results for the machine learning models. In the training step, the number of normal data was increased to be as similar as possible to the actual installation site of the BWTS. In addition, the ratio of the fault data was higher than that of the normal data in the evaluation step to determine whether the trained model performed fault diagnosis correctly. The statistics of the data division for each label are listed in Table 5.

Table 5:

Statistics of data by label

We used the Python programming language for feature generation and machine-learning model learning. For feature generation for window sizes of 1–10, Numpy [18] and Pandas [19]modules were used. The Scikit-learn module was used to implement and test each machine learning model. All model parameters borrowed default values, except for “OvO” in SVM and LR.

4.2 Feature generation performance by window size

The sensor data used for learning and evaluation were time-series data observations obtained in intervals of 1 s. To evaluate the suitability based on the window size during feature generation, we measured the accuracy of the machine learning models with window sizes ranging from 1 to 10. Table 6 presents the classification accuracy results based on the window size for the six models.

Table 6:

Classification accuracy of six models based on window size

The accuracy of the SVM model tended to decrease with increasing window size. This is the OvO method, which performs binary classification between label pairs. It appeared that an increase in the number of dimensions of support vectors generated a hyperplane that increased the classification complexity. In contrast, the accuracy of the LR model tended to with an increase in the window size. This trend occurred owing to the OvR method used by the LR model in performing multiple classifications. Unlike the OvO method, which carries out multiple classifications by performing binary classification between label pairs for all cases, the OvR method performs multiple classifications by removing the probability of the remaining labels from the total probability. Therefore, it appeared that the transfer of more features improved the accuracy of the probability calculations for the hyperplane determination of the LR model.

The k-NN, MLP, RF, and XGBoost models showed improved accuracy up to specific window size, after which the accuracy remained similar to that for a window size of 1. Although the effects were weaker than those of the SVM and LR models, the classification accuracy tended to increase as the window size was increased to reflect the time-series data. Furthermore, the RF model demonstrated the highest accuracy. This result could be attributed to unbalanced learning progression of the bootstrap aggregation ensemble method, despite the minimal amount of fault data.

The conclusive differences compared to the results of previous studies are encouraging. Unlike previous studies that demonstrated fault diagnosis using only the SVM model, the results of this study show that feature generation according to window size is valid for other machine learning models. By comparing window sizes ranging from 1 to 5, we confirm that a window size of 2 is suitable for the SVM model. However, by comparing window sizes ranging from 1 to 10, we demonstrate that the feature generation method is unsuitable for the SVM model.

4.3 Performance analysis of RF model

We evaluated the performance of the RF model using a window size of six, which yielded the highest accuracy. The evaluation showed improved accuracy under all conditions compared to those of previous studies. Table 7 lists the confusion matrix for the experimental results of the RF model. Table 8 presents the six evaluation measures [20] obtained using the values listed in Table 7.

Table 7:

Confusion matrix for fault diagnosis of RF model

Table 8:

Evaluation metrics based on Table 7

A true positive is a frequency at which a steady state is predicted accurately, whereas a false positive is a frequency at which a fault is predicted as normal. A false negative is the frequency of a fault being predicted, even in a normal state, whereas a true negative is the frequency of a fault being predicted accurately. The positive predictive value is called precision in the same term, and it refers to the ratio of correct predictions among the model predicted to the normal state. The true positive rate is called recall in the same term and refers to the ratio of correct predictions among actual normal state data.

Precision is a score indicating the prediction accuracy of the state of a BWTS. The RF model exhibited the best performance, as indicated by a precision value of 100%. The recall rate, which is a measure of the accuracy of the predicted fault state, was 99.46%. This value indicated that approximately 0.5% of the faults were undetected. The F1 score of 99.73% was the harmonic mean of precision and recall. The false alarm rate is defined as the rate at which the normal state is incorrectly predicted as a fault. The false alarm rate of the RF model was 0%, indicating the case in which the normal state was classified as a fault. Specificity, the rate at which the fault state is predicted to have a fault, was determined by subtracting the false alarm rate from 1, recorded as 100%. In summary, the false alarm rate and specificity, including precision and recall, reflected excellent performance. We believe that this model is appropriate for application in the industrial sector. However, further confirmation and monitoring by experts are essential, as some areas need to be further evaluated to implement complete unmanned systems.

Author Contributions

Conceptualization, H. M. Park, S. G. Cheon, J. H. Kim, and S. D. Lee; Methodology, H. M. Park, J. H. Kim, and S. D. Lee; Hardware and Data Generation; S. G. Cheon, and S. D. Lee; Software, H. M. Park, J. H. Kim; Validation, H. M. Park, J. H. Kim, and S. D. Lee; Data Management, H. M. Park, and S. G. Cheon; Writing-Original Draft Preparation, H. M. Park;Writing-Review & Editing, H. M. Park, S. G. Cheon, J. H. Kim, and S. D. Lee.