Industrial Augmented Reality

- Innovative operator assistance in collaboration with Augmented Reality -

Yosuke Ishii¹ Kenji Ooishi¹ Yasuki Sakurai¹

Plant operators are being forced to handle more advanced and complicated systems with fewer people because of the aging workforce problem. This problem has become a worldwide epidemic and has been inducing critical accidents caused by human errors. Yokogawa has been evaluating a series of Proof of Concepts (PoC) for new operator assistance systems. In particular, the concepts applying Augmented Reality (AR) technology have been highly anticipated by the customers, as AR enhances operator's visual perception, which is the most relied upon of the five human senses. This paper introduces Yokogawa's new operator assistance concepts, based on human-centered design and AR, as well as introducing activities for launching the products, and solutions for achieving above stated goals.

1. Research & Development Division, Innovation Headquarters

INTRODUCTION

In industrialized countries, social trends such as the aging of workers and the decrease in working population are becoming obvious. Substantial problems such as a lack of skilled engineers and increases in the labor cost of knowledge workers are significantly impacting industrial automation users. These problems cause greater economic loss to the users. In particular, economic loss due to human error including operator faults or oversight is said to be one of the biggest issues for plant operations. In fact, several surveys have reported statistical data showing that more than 70% of current industrial accidents are caused by human errors⁽¹⁾.

In order to overcome this situation, users are intending to secure operators and improve the quality of operators by reducing labor cost through employing human resources from developing countries and enhancing education and training for young plant operators. However, as labor costs are already rising and training to become a skilled engineer takes as long as five to ten years, any satisfactory solutions are currently found neither in the short term nor in the long term.

Yokogawa believes that simply securing and training human resources is not enough for resolving this issue. Recently, reflecting this opinion, bodies for activities relating to human machine interfaces (HMI) taking human factors into consideration have been established with the participation of both users and vendors^(2),(3), and various investigation and research activities have been carried out for improving the performance of plant operators, as well as health, safety and environment (HSE) in plants.

Under such circumstances, Yokogawa is reconsidering the human factors in plants, and conducting studies for resolving the problem of human errors in which users are most interested. In this paper, verification activities for a new operator assistance concept using augmented reality (AR) and their results are reported as an example of those studies. AR is a technology which greatly enhances operators' visual perception, which is the most relied upon of the five human senses, and provides intuitive user interfaces not available until now. This paper describes analysis and hypotheses aimed at resolving the human error problem using AR in the first section. The following sections describe examples of a proof of concept (PoC) conducted to verify the hypotheses; technical issues in the AR and activities for resolving them for commercialization; and the current status of the prototype and its field test, as well as future development, respectively in this order.

ANALYSIS OF THE HUMAN ERROR PROBLEM, AND APPROACHES FOR RESOLVING IT

Defining the problem: human errors due to carelessness

Figure 1 The number of human errors in each category (Data source: statistics of accidents and cases in fiscal year 2007 by Ministry of Health, Labor and Welfare, Japan)

Human errors caused by individuals are classified into the following six categories from the perspective of the cause of an incident⁽⁴⁾.

1. Unavoidable: errors beyond human capability
2. Slip: mistakes in judgment such as misidentifying or misunderstanding
3. Forgetting: forgetting to do something, lapse of memory
4. Lack in capability: lack in capability or skills for performing a task
5. Insufficient knowledge: no knowledge nor understanding of a task to be carried out
6. Violation: omission or negligence

Because human errors in category 1) and 6) mainly depend on management of concerned organizations and human morals, appropriate measures should be taken as activities in those organizations as needed. The intrinsic measures against human errors in category 4) and 5) are education and training using manuals or a training simulator, or transfer of knowledge and technical skills through OJT.

In contrast, errors referred to as careless mistakes such as slips and forgetting, categorized into 2) and 3), cannot be prevented by traditional training or discipline for thorough conformance to manuals. Committing these errors does not depend on the years of experience of the operators, but depends on their physical and psychological conditions while working, and so prior education or training cannot prevent them. In fact, the analysis results of human errors indicate that careless mistakes account for the largest ratio in all the human error as shown in Figure 1. Thus, we have concluded that there are few effective solutions for reducing human errors at present.

Improving reliability of operators by controlling human errors

Figure 2 Error Troika by Professor R. Helmreich of the University of Texas

Although this study aims to prevent human errors, it is essentially impossible to completely eliminate human errors themselves. To be accurate, the objective of this study is executing 'error management', one of the principles in research on human factors. That is, the objective focuses not on completely eliminating human errors but on minimizing damage caused by them by controlling them. As measures for error management, the following three-stage measures to control human errors are proposed by Professor R. Helmreich of the University of Texas and others as shown in Figure 2^(5),(6).

1. Avoid Error: Reducing frequency of occurrence of human errors by making the work simpler. Investigation and consideration of human recognition capability, operators' five senses and sharing recognition between members involved are required.
2. Trap Error: Letting operators be aware of anomalies to avoid accidents before suffering damage. Mutual suggestions among members involved and warning through machinery or tools are required.
3. Mitigate Consequences of Error: Mitigating damage and preventing a trivial accident from becoming a serious one. Systematic measures are required for preparing for rare emergency accidents by arranging preliminary analysis and procedures for them.

In this study, under the hypothesis that visual enhancement by AR greatly contribute to measures 1) and 2) above, we have created concepts supporting operators taking advantage of AR, such as its potential for intuitive operational feeling and flexible expressive power of information, and then we have evaluated their user value.

PoC: CoNCEPTS USING AR TECHNOLOGY FOR EVALUATING USER VALUE

Figure 3 BuddySystem concept

AR is applicable to various areas, and its definition and interpretation are somewhat different depending on its application area and usage. However, we define AR as a technology for superimposing digital data collected by computers onto the real-world images captured by cameras. Nowadays, typical devices which can be equipped with AR applications are what are called smart devices such as tablets, smartphones and head mounted displays (HMD). Although the concepts and technology of AR have been known for a while now, the spread of high-performance mobile devices and high-speed wireless data communication has rapidly expanded AR's applicability to various areas, including industrial use⁽⁷⁾.

We have developed multiple prototypes for concept demonstration to provide images of plant operations to which AR has been applied, and have evaluated its value via surveys from users. Four examples among these are introduced below. The names of these concepts were coined by Yokogawa.

1. BuddySystem: Double-check assistance with a computer
   The BuddySystem as shown in Figure 3 aims to achieve real-time collaboration, sharing images and voice provided by displays for AR purpose between multiple operators at a distance from each other. Visual effects owing to AR make it more effective for sharing situations at sites between operators and sending instructions to each other, which has been difficult with only conversation by voice and still images. Human errors at a plant site often occur during patrol or maintenance work by a field operator working alone. In order to prevent this, such a system that can secure two or more operators to perform double-checking for an important task is required, but it is not realistic to apply such a system to every task from the view point of cost. In addition, when multiple operators work together as with valve operation during flow path switching, communication between physically isolated operators often fails, leading to mistakes. This concept supports collaboration between operators in such situations, by fully leveraging the visual effects owing to the AR. Skilled workers can visually support newcomers' work from a remote location. Furthermore, instructions using handwritten drawings as needed improve efficiency and safety of work at a site assigned to one worker.

2. Figure 4 ARMan concept (Top: electronic manuals; Bottom: icon for diagnosis)

   ARMan: "Just-in-Time" & "Just-in-Place" instruction
   In the ARMan, information of plant equipment, process diagnostics and operation procedures is displayed on demand for every workplace and work item as shown Figure 4. This enables operators to immediately and intuitively obtain information needed for their work, and greatly improves the efficiency of patrols and maintenance work. It is a heavy burden for operators to carry many weighty and thick manuals and find out the necessary information from them during their work. ARMan provides not just electronic manuals but also necessary information in an easily understood format according to a workplace and work item, significantly reducing the work load of operators at sites.
3. RadioLens: Visualizing wireless communication network status
   The RadioLens graphically visualizes information relating to the state of wireless communication between wireless field devices such as radio field intensity as shown in Figure 5. This can support proper installation, operation and maintenance of wireless networks, which are invisible to human beings and intrinsically unstable. Wireless field instruments such as those provided by Yokogawa are limited in screen display capability due to such reasons as installation environment and power consumption. Therefore, it is very difficult to evaluate communication quality at sites compared with usual mobile devices, and knowledge related to wireless is required to intuitively understand its quality. However, the RadioLens enables accurate evaluation of transmission quality at sites by providing the communication status of wireless devices superimposed over devices in the real world.
4. TetheredDisplay: More information at remote locations
   The TetheredDisplay displays the content displayed in a graphical user interface (GUI) of field instruments on a smart device as shown in Figure 6. This can display information on extremely expressive smart devices instead of field instruments with limited displaying capabilities. As described in the RadioLens case, the display of instruments provided by Yokogawa is limited in display performance due to constraints at sites or limitation of power consumption. Thus, the available information is sometimes limited, and this affects the efficiency of the work of operators at sites. The TetheredDisplay recognizes and identifies instruments at sites by using AR first, collecting information from the instruments via a wireless network or short-distance communications, performing calculations on them, and then displays them on the GUI of the smart devices, enabling extensive information presentation and its flexible customization that traditional field instruments cannot achieve.

TECHNICAL ISSUES AND ACTIVITIES FOR RESOLVING THEM

Figure 5 RadioLens concept

When applying AR technology, related objects are recognized in an image on a mobile terminal, and then their contents are overlaid onto them by using authoring techniques. Although the overlaying process can mostly depend on tools on the market, we consider that the recognition process includes unresolved problems in fundamental technologies at present. Objective recognition methods are classified into two main approaches: image recognition and location identification. The most popular means for them are attaching markers such as QR code and positioning by GPS, respectively. As every means has advantages and disadvantages, these means need to be used in combination according to each situation. However, in the plant environment which is the target area of Yokogawa as described in this paper, technologies available today include many problems. For example, image recognition is sometimes difficult due to many shady or dark areas in a plant. In addition, there exist blind spots for GPS in a plant, and vertical positioning accuracy by GPS is not sufficient. For these issues, Yokogawa needs to find ways to resolve them through our own efforts or outsourcing of technologies^(8),(9). Yokogawa is aiming to establish technologies for them through field tests, etc.

ACTIVITIES FOR PROTOTYPE DEVELOPMENT AND ITS FIELD TEST

Figure 6 TetheredDisplay concept

Aiming at practical use of AR in a plant, we are now developing a prototype system interconnected with various products including Yokogawa's major product CENTUM VP, and we are planning its field test. Figure 7 shows CENTUM VP and the prototype system interconnected. Although applying AR to plant sites still includes some technical problems as described in the previous section, a part of the prototype system is ready for practical use. Yokogawa will continue to accelerate research and development for reducing human errors, which is the theme of this study, and which results in improvement in operation efficiency and safety, i.e. the user value.

CONCLUSION

The use of high-performance mobile devices represented by smartphones and tablets is rapidly spreading. Their shipments are already exceeding those of PCs and it is expected that half of the companies in the world will adopt Bring Your Own Device (BYOD) by 2017⁽¹⁰⁾. Although use of mobile terminals in the industrial automation area includes a variety of restrictions such as being explosion proof, their diffusion in this area will progress in the future, and the day will undoubtedly come when plant operators routinely make effective use of them. As a system vendor, Yokogawa is responsible for supporting this diffusion and providing for best usage. The PoC activities introduced in this paper have proven that AR can be effectively applied for mobile terminals to be used at sites. Yokogawa will continue research and development focusing on collaboration with mobile devices, and making efforts to increase user value.

Figure 7 CENTUM VP and prototype system interconnect

REFERENCE

1. Interim Report on Industrial Accident Survey, Ministry of Health, Labor and Welfare, Japan 2003 in Japanese
2. Abnormal Situation Management (ASM) Consortium, http://www.asmconsortium.net/Pages/default.aspx
3. Center for Operator Performances (COP), https://operatorperformance.org/
4. A Komatsubara, Human Error (second edition), Maruzen Publishing, 2008 in Japanese
5. Robert L. Helmreich, et al., "The Evolution of Crew Resource Management Training in Commercial Aviation," The International Journal of Aviation Psychology, Vol. 9, No. 1, 1999, pp. 19-32
6. T Nakata, Wisdom for Preventing Human Errors: Can all mistakes be eliminated ?, Kagaku-Dojin Publishing, 2007 in Japanese
7. Nikkei Communications editorial office, All About AR: Technology to innovate mobile phones and the Internet, Nikkei Business Publishing, 2009 in Japanese
8. Kenji Oafish, Yaqui Sakurai, Yosuke Ishii, "An Adaptive AR Photographing Parameter Tuning for Industry Workspaces," SICE Annual Conference 2013, September 2013.
9. In-Location Alliance
10. Fabio Trevino, "The HMI of the future will look very familiar," Control Engineering, Vol. 60, No. 7, 2013, pp. 26-29

* CENTUM and CENTUM VP are the registered trademark of Yokogawa Electric Corporation.
* QR code is the registered trademark of DENSO WAVE INCORPORATED.