A system framework for non-intrusive monitoring of HMI states for detecting human-in-the-loop error precursors

Abstract

Past industrial accidents suggest that increased automation has had an adverse effect on operator situational awareness (SA) and Human-in-the-Loop (HITL) errors. More so, any current automation is not yet capable of providing the level of situational awareness that a human operator can provide and close the ethical responsibility-gap. The objective of this research is to design a system for detecting HITL error precursors to ascertain operator SA (as a function of operator activity index) in real-time via non-intrusive monitoring. The proposed system frameworks are ViDAQ and EYE-on-HMI. ViDAQ is a visual data acquisition system that uses computer vision techniques to capture dynamic visual feedback from the industrial human-machine interface (HMI) (e.g., control panels) states. For example, ViDAQ results demonstrate approximately 90% accuracy at 1 meter acquisition distance when reading a multi-dial rotary-style meter. Positive results, coupled with real-world control room settings that offer constant lighting and a vibration-free environment, support the future efficacy of ViDAQ. The EYE-on-HMI is a novel expert supervisory system that models HMI state patterns (as captured from ViDAQ) under normal and abnormal conditions, to detect HITL error precursors in real-time. It is developed using a variety of modeling techniques: linear regression (ARIMA), recurrent, and convolutional neural network (RNN and CNN) for HMI time-series modeling; and Seq2Seq deep learning natural language processing (NLP) models for HMI discrete event system model. As an example, relative root-mean-square-error in forecast accuracy is RMSE ~ [30%; 80%; 100%] for N - ahead > 10 time-step forecast window. This suggests regression-based models are closely followed by RNN, CNN, and NLP models in order of forecast accuracy achieved using synthetic HMI state datasets. Nevertheless, RNN and CNN are more versatile and scalable than the regression models during the training and evaluation phases, which is also anticipated for large scale multi-variate industrial HMI datasets. Moreover, the NLP based models embed contextual dependencies as semantic relations between HMI states for complex event patterns to improve HITL error precursor detection accuracy. Lastly, the proof-of-concept HITL error precursor detection using CANDUTM Nuclear Control Room Operator training simulator is demonstrated.