Decision support for preventing safety violations

Aim. The aim of the paper is to examine the experience of reducing the effect of the human factor on business processes, to develop the structure and software of the decisionsupport system for preventing safety violations by train drivers using machine learning and to analyse the findings. Methods. The study presented in the paper uses machine learning, statistical analysis and expert analysis. In terms of machine learning, the following methods were used: logistical regression, random forests, gradient boosting over decision trees with frequency-domain representation of categorical features, neural networks. Results. A set of indicators characterizing a train driver’s operation were identified and are to be used as part of the system under development. The term “train driver’s reliability” was defined as the ability not to violate train traffic safety over a certain number of trips. Algorithms were designed and examined for predicting violations in a train driver’s operation that are used in defining reliability groups and lists of preventive measures recommended for the reduction of the number of safety violations in a train driver’s operation. Major violations with proven guilt of the driver that may be committed within the following 3, 7, 10, 20, 30, 60 days were chosen as attributes for the purpose of safety violation prediction. Analysis of the results on the test sample revealed that the model based on gradient boosting over decision trees with frequency-domain representation of categorical features shows the best results for binary classification on the prediction horizon of 30 and 60 days. The developed algorithm made a correct prediction in 76% of cases with the threshold value of 0.7 and horizon of 30 days and in 82% of cases with the threshold value of 0.9 and horizon of 60 days. The solution of the problem can be found in the integration of different approaches to predicting safety violations in a train driver’s operation. Additionally, 10 of the most significant indicators of a train driver’s operation were identified with the best of the considered models, i.e., gradient boosting over decision trees with frequency-domain representation of categorical features. Conclusion. The paper presents an overview of methods and systems of assessing human reliability and the effect of the human factor on the safety of transportation systems. It allowed choosing the most promising directions and methods of predictive analysis of a train driver’s operation, including methods of machine learning. The resulting set of indicators of a train driver’s operation that take into consideration the changes in the quality of such operation allowed obtaining initial data for training the models implemented as part of the system under development. The implemented models enabled the aggregation of information on train drivers and adoption of targeted and temporary preventive measures recommended for improving driver reliability. The resulting approach to the definition of preventive measures has been implemented in three depots of JSC RZD in trial operation mode.


Introduction
One of the subjects matters of the research of the human factor as part of man-machine systems is the problem of human errors in the process of operations and their predictability. Today, train drivers' operation has no objective assessment system. Recording and defining the significance of the indicators that characterize the quality of a train driver's operation primarily depend on his/her direct superior, i.e., the presence of the human factor can be observed. Therefore, a decision support system (DSS) should be developed for preventing safety violations in the driver's operations that would enable their objective assessment by predicting possible violations and defining preventive measures recommended for improving the driver's reliability.
Let us introduce the concept of "train driver's reliability" that will be defined as the ability not to violate train traffic safety over R trips, R≥1. A traffic safety violation will be understood as any of a set of incidents causing the violation of a provision of the current classifier of violations and emergencies of JSC RZD. The list of incidents is made by decoding speed tapes, whose information is recorded in the network-level information system for registration, analysis and investigation of traffic safety violations [1]. The probability of no traffic safety violations is calculated through the probability of the opposite event: where P N (R) is the probability of traffic safety violation within R trips. Identifying this value requires knowing the correlation between the driver's performance and the committed violations.
The concepts of structural and functional dependability are examined in detail in [2,3]. For the purpose of this study, it would be logical to rely on the concept of functional dependability that is defined as the driver's readiness to perform the predefined tasks within R trips. The tasks can be defined as follows: -ensuring the performance of the traffic schedule; -performing the established procedures of train driving and shunting operations; -when driving passenger trains, ensuring quality service of passengers, preventing violations of smooth train running, ensuring electric heating/ventilation of cars, boarding and alighting of passengers.

Source overview
Despite the advances in automated control systems (ACS), it is still impossible to completely eliminate the human involvement in business processes. Human reliability analysis (HRA) is a relatively new discipline. HRA methods are applied in many industries. These methods aim to assess human reliability and the human factor affecting ACS. A number of HRA methodologies have been developed by the scientific community over the last few years [4,5,6]. The developed methodologies can be divided into two macrocategories, i.e., the first and second-generation methods.
The first generation includes 35 to 40 methods of ensuring human reliability. Many of them are modifications of one method. The common theoretical foundation of most first-generation methods is: the method of error classification according to the concept of "inaction"; definition of the "performance influencing factors"; cognitive model (based on skills, rules and knowledge). The most popular first-generation theory for identifying and classifying incorrect actions is the error classification method according to the concept examined in [7] that is based on the "action -inaction" principle. In accordance with those principles, "inaction" defines an action that has not been performed or was performed late. An "action" is an action performed by a person that is not required for the process. Based on that principle, the first-generation prediction models were developed. THERP became the most popular out of them [8]. That is a method for generating predictions of error by a person based on the frequency of past errors of the same person. The method was developed for a nuclear power plant for the purpose of probabilistic risk assessment. Using this technique, the authors quantified the probability of human error.
The second-generation methods (a term first coined in [8,9,10]) were developed for the purpose of overcoming the limitations of the first-generation methods. These methods are based on mental process models developed in the cognitive psychology. They extended the ways an error can be described beyond a simple binary classification. The paper considers the dynamic aspects of human -machine interaction and their application as the foundation for the development of operator simulators.
The examined subject matter is associated with railway traffic safety that is one of the key concerns of JSC RZD. There is a great number of works dealing with various aspects of safety. Let us look into some of them [11,12,13,14]. In railway transportation, safety in terms of control and driver's behaviour supervision is ensured through a number of means. The comprehensive on-board safety and protection capabilities include the following components: -All-Purpose Automatic Train Operation System ( USAVP) [15]; -Automatic Brake Control System (SAUT-TsM) [16]; -Integrated On-Board Train Protection System (KLUB-U) [17]; -Remote Driver Vigilance Supervision System ( TSKBM) [18]; -Driver Vigilance Handle (RBM) [19]. In addition to the safety systems that use indirect control and restriction of driver behaviour, there are methods for assessing the personal characteristics of drivers, as well as methods designed for psychological support of drivers' professional activities. Using the above methods, a detailed description was prepared of the motivation and personal qualities that characterize the sample of drivers, the correlation was analysed between the selected characteristics and the rate of accidents in the drivers' operations [20].
Among the works dedicated to the evaluation of human activity in railway transportation, [21] deserves a special attention. Its authors rely on the methods of expert assessment used for determining the significance coefficients of indicators, evaluates the risks of potential -caused by the human factor -disruptions in the business processes of railway stations. The work revealed that the primary cause (50-75% of the total number of causes) of incidents in railway transportation is the technical staff errors.
JSC RZD has already rated its train drivers [22]. The algorithm was based on collecting and analysing experts' opinions. The experts selected the features and rated their significance. A classical linear combination of a feature vector and weight vector was used that was normalized by the number of trips made by a driver during a month. The method's significant limitation consists in the subjectivity of experts' opinions that may cause a bias or a strong spread of the estimates of the quality of the driver's work.
This paper's findings can be integrated into the intelligent system for centralized traffic management of rapid transit under heavy traffic [23].

The methods
The DSS for preventing safety violations by train drivers includes the following units ( Fig. 1): 1. The object of control, a driver or group of drivers that share the same depot or railway line. The unit receives inputs in the form of control actions [E 1 , E 2 , …, E n ] and outputs a set of reactions by the object of control, 2. The information collection module that records (measures) information on the drivers and saves it to the database. Importantly, the information is recorded in various ACSs of JSC RZD. The module outputs set D.
3. The database, a single physical storage of big data collected from various ASCs of JSC RZD using the Automated System Trusted Environment of the Locomotive Service [24]. This storage contains raw information on the drivers and the calculation data on their performance. The unit outputs the vector of indicators [F 1 , F 2 , …, F n ].
4. The violation prediction module calculates the probability of violations by drivers based on their performance indicators. The unit outputs probability vector [P 1 , P 2 , …, P n ] that stores information on the probability of a major violation and probabilities of specific violations.
5. The driver operations analysis module aggregates driver ratings [25], number and types of committed violations, driver risk groups, driver's medical indicators in past trips. In addition to the above, the following information is supplied to the module's input: -C v , the classifier of recorded violations and emergencies identified as the result of decoding of speed tapes and other media; -C s , the safety requirements approved by JSC RZD; , the vector of driver's features and characteristics.
The module outputs values [R 1 , R 2 , …, R n ] of a driver's reliability group membership and the criteria of the list of preventive measures recommended for improving driver reliability. ComD, comparing device; ConD, control device; ED, executive device 6. The measure planning module defines the list of preventive measures recommended for improving the reliability of a particular driver, depot, railway. In addition to the above information, the module receives the list of preventive measures C a that can be recommended for improving a driver's reliability.
The module outputs a list of measures and actions [A 1 , A 2 , …, A n ] aimed at improving traffic safety.
7. The measure implementation module creates a set [E 1 , E 2 , …, E n ] of control actions that affect the object of control.
Based on the generated structure diagram of the DSS for preventing safety violations by train drivers (see Fig. 1), let us define the problems whose solution is examined in this paper: a) identification of the set of indicators of a driver's operation used in the DSS; b) development of the algorithm for predicting violations in the drivers' operations for the purpose of defining reliability groups within the DSS; c) development of the algorithm for defining the list of preventive measures recommended for improving driver reliability based on the analysis of the outputs of the algorithms for predicting violations in the driver's operations; d) implementation of DSS for preventing safety violations in drivers' operations as part of JSC RZD's automated information management system.

Identified set of driver performance indicators used in the DSS for preventing safety violations in drivers' operations
The study analysed 90 indicators that characterize a driver's operations obtained from seven ACSs of JSC RZD. All indicators can be classified into the following groups: fuel and energy consumption; disciplinary (associated with past safety violations); medical; operational discipline; interaction with the assistant; level of knowledge; interaction with train driving instructor; basic information (e.g., service record, class, etc.). In total, data for over 4.2 million trips between 01.01.2020 and 01.08.2020 were analysed.

Development of the algorithm for predicting violations in the drivers' operations for the purpose of defining reliability groups within the DSS for preventing safety violations in drivers' operations
The algorithm for predicting violations in the drivers' operations is intended for binary classification as part of prediction of imminent violations by a driver. The test sample consisted of about 850 ths driver trips.
Major violations with proven guilt of the driver that may be committed within the following 3, 7, 10, 20, 30, 60 days were chosen as attributes for the purpose of safety violation prediction. The driver sample is unbalanced. This problem and its possible solutions are covered in [26].
Solving the problem of binary classification involves using only such method of assessing the model performance that reflects the objective reality. In the context of the problem at hand, accuracy, harmonic mean (F-measure) and area under the receiver operating characteristics curve (AUCROC) should be used. Table 1 shows the prediction outputs using various machine learning algorithms and representation of categorical attributes [27]. Fig. 2 shows the outputs of gradient boosting over decision trees with frequency-domain representation of categorical features under various prediction horizons.
The results of binary classifier training show (Fig. 3) that identifying the probability of a violation within the following few days is not a trivial and easy task. The results are given for two prediction horizons, i.e., 30 and 60 days. It turned out that the algorithm made a correct prediction: in 76% of cases with the threshold value of 0.7 and horizon of 30 days; in 82% of cases with the threshold value of 0.9 and horizon of 60 days. The solution of the problem can be found in the integration of different approaches to predicting safety violations in a train driver's operation. Gradient boosting over decision trees with frequency-domain representation of categorical features showed the best data processing results.
Additionally, 10 of the most significant indicators of a train driver's operation were identified with the best of the considered models, i.e., gradient boosting over decision trees with frequency-domain representation of categorical features (Fig. 4).

Development of the algorithm for defining the list of preventive measures recommended for improving driver reliability based on the analysis of the outputs of the algorithms for predicting violations in the driver's operations
A method is proposed of defining driver reliability groups based on quantiles of the distribution of estimates of the likelihood of violations and identification of imminent reliability. The following quantiles were taken as delimiters of reliability groups defined based on the probability values of the absence of traffic safety violations by the driver: 0.  Besides the algorithm for predicting violations themselves, an algorithm was developed that allows predicting the type of the safety violation. This algorithm is covered in sufficient detail in [28]. The approaches used for predicting the type of violation belong to the domain of advisory systems and are built on neural networks.

Implementation of DSS for preventing safety violations in drivers' operations as part of JSC RZD's automated information management system
Let us examine the approach to defining the list of recommended preventive measures that is based on the analysis of a driver's membership in a reliability group, probability of a major violation, past and predicted violations.
The operation of the algorithm (Fig. 5) is initiated by a user of the information system who requests the assignment of a driver to a trip. The depth of past violations check (number of days) N is a parameter specified by the management of the relevant Central Directorate of JSC RZD. The recommended default setting in the system is N=20, which is the average monthly number of a driver's trips. If no violations were identified in N days, the probability of imminent violation is calculated (Block 7). The vector of the driver's features and characteristics is used as the input information for this block F. Its output is the calculation of the probability of an imminent major violation P. Then, after calculating P, it is compared with the permissible threshold r that is to be specified by the management of the relevant Central Directorate of JSC RZD. If P did not exceed threshold r, the driver is allowed to drive.
If violations have been identified within N days, the advisory subsystem or model (Block 8) is initiated and makes a list of violations that the same driver is likely to commit in the future. The input information for this block is vector F and the prediction horizon m (configurable parameter). The output is an m-long list of predicted violations V sorted by significance. The list of violations V is then used as the input for Block 9 where the number Q of major violations or violations whose weight is above the specified threshold coefficient w is calculated. If Q>n, the imminent violation probability algorithm is initiated. If Q<n, the driver is allowed to drive after an interview with the manager. The input for Block 12 is the list of violations V, probability P and the driver's current rating R. Based on values V and R, the set of recommended measures is defined. P can be interpreted as a value that completes the driver's level of reliability with respect to one.
The recommendations and actions for the driver are based on the information on the predicted and past violations. Each violation is characterized by two groups of factors, i.e., the general characteristic (major or minor violation, with breach of regulations, with possible violation of safety, with violation of safety) and the human factor (insufficient knowledge, lack of experience, carelessness, distraction, haste, negligence).
The measures pertaining to the driver are divided into two classes: "short-term", i.e., before the trip; "long-term", i.e., after the trip.
The algorithm for defining the measures consists of the following steps: 1. For all the violations committed by driver K in the last N days, the accumulated significance levels of the above factors multiplied by the weights (2) are summarized: , (2) where w i is the weight of the i-th violation, f i is the vector of the factors of the i-th violation, F v is the sum vector of the levels of effect (vector of ranks). 2. The cooccurrence matrix of factors f and events E is multiplied with the vector of the levels of effect (vector of ranks) F v .
3. All measures are sorted in nondecreasing order of importance. Each measure is ranked.
The importance of a measure is determined by its rank. The higher the rank, the more important is the measure for the driver.

Conclusion
The paper presents an overview of methods and systems of assessing human dependability and the effect of the human factor on the safety of transportation systems. It allowed choosing the most promising directions and methods of predictive analysis of a train driver's operation, including methods of machine learning.
As part of the structure of the DSS for preventing safety violations by train drivers, machine learning models were constructed and implemented that allow identifying a driver's level of reliability, probability of future violations, as well as defining preventive measures recommended for improving a driver's operational reliability.
A set of indicators defining a train driver's operations that are used for determining the effect of the driver's reliability on the traffic safety were identified by creating and applying a method for estimating drivers' performance that takes into account the past dynamics of a driver's quality of operation, which allowed collecting input data for the construction of mathematical models for the DSS for preventing safety violations by train drivers.
An analysis was carried out of the outputs of machine learning algorithms on a test sample that revealed that a model based on gradient boosting over decision trees with frequency-domain representation of categorical features shows the best results for binary classification on the prediction horizon of 30 and 60 days. The models enabled the aggregation of information on train drivers and adoption of targeted and temporary preventive measures recommended for improving driver dependability.
The resulting approach to the definition of preventive measures has been implemented in three depots of JSC RZD in trial operation mode.