Methodology for detecting and removing outliers in statistical studies

The paper presents a calculation method for detecting and eliminating outlying values. It is shown that its effectiveness depends on the amount of a priori information on the examined process. The proposed method is used for cases whereas the process is stationary and has a Gaussian probability density law. When analysing non-stationary random processes, the existing methods and algorithms rely on the fact that the outlying component is additive and the characteristics of the outlying values are known a priori. The work used the statistical decisions theory that allows formalising the verification algorithms and selecting a criterion for detecting outlying values. Both parametric and non-parametric methods were proposed. In the first case, it is required to have a priori information both on the function of the useful component and on the distribution law of the outlying component of the process, as well as its parameters. It is postulated that the use of non-parametric processing methods requires significantly less a priori information, but their effectiveness is defined by the processing parameters that, in turn, depend on the function of the useful and the distribution law of the outlying components of the process. It is noted that an outlier may prove to be one of the extreme values of the probability distribution of a random variable. The authors outline the problems of ambiguity of input data in case of classical computing. The paper examines the way the external factors affect the dependability and the degree to which such factors are taken into consideration in the existing methods. Methods for assessing the life of the examined items are presented, among which control chart-based methods hold a prominent place. It is shown that the range proves to be a more convenient measure for data dispersion calculation than the standard deviation. Plotting the range of sample on a control chart along with the expectation makes it easier to notice an anomaly. The range is a rough measure of the rate of change of the monitored variable and its value may exceed the control limits on the range chart and inform of an anomaly much earlier than the change in the mean that may still be within the specified control limits.

1. Gnedenko B.V., Beliaev Yu.K., Soloviev A.D. [Mathematical methods in the dependability theory]. Moscow: Nauka; 1965. (in Russ.)

2. Sidnyaev N.I. [Experimental design theory and statistical data analysis: a study guide]. Moscow: ID Yurayt; 2011. (in Russ.)

3. Morozov D.V., Chermoshentsev S.F. Method of improving the functional dependability of the control systems of an unmanned aerial vehicle in flight in case of failure in the onboard test instrumentation. Dependability 2019;19(1):30-35.

4. Sidnyaev N.I., Sadykhov G.S., Savchenko V.P. [Models and methods of estimation of the residual operating life of electronics]. Moscow: Bauman MSTU Publishing; 2015. (in Russ.)

5. Mоrris S.F. Use and application of MIL-HDBK-217. Solid Slate Technology 1990;33(6):65-69.

6. Sidnyaev N.I. [Mathematic simulation of dependability estimation of complex technical systems]. Problemy mashinostroyeniya i nadiozhnosti mashin 2003;4:24-31. (in Russ.)

7. Вrennоm T.R. Should US MIL-HDBK-217 be 8888. IEEE Trans. Reliab. 1988;37(5):474-475.

8. Sidnyaev N.I. [Overview and research of physics of failure for the estimation of the dependability indicators of today’s radar electronics]. Physical Bases of Instrumentation 2017;2(23):4-52. (in Russ.)

9. Barlow R., Proschan F. Mathematical theory of reliability. Moscow: Sovetskoye radio; 1969.

10. RD 50-690-89. [Guidelines. Dependability of technology. Methods of estimation of dependability indicators based on experimental data]. Moscow: State committee of the USSR for products quality management and standards; 1990. (in Russ.)

11. Sidnyaev N.I., Makridenko L.A., Gecha V.Ya., Onufriev V.N. [Factors of space weather affecting the airborne devices of low-orbiting spacecraft]. In: Electromechanical matters. VNIIEM studies. Proceedings of the Fourth International Science and Technology Conference Topical Issues of the Design of Space-Based Earth Remote Sensing Systems. Moscow: VNIIEM Corporation; 2016. Pp. 90-100. (in Russ.)

12. Antonov S.G., Klimov S.M. Method for risk evaluation of functional instability of hardware and software systems under external information technology interference. Dependability 2017;17(1):32-39.