Yokogawa's Modular Procedural Automation is a Knowledge Preservation Tool for the Process Industries

 

Average Dollar Loss Per Major Incident by Cause
Millions of Dollars
Source: J & H Marsh & McLennan, Inc.

Operator Error Accounts for the Highest Dollar Losses Per Incident in the Process Industries

 

Procedure Automation is one of the Primary Elements of ARC's Model for Human Reliability

Overview of Operational Procedures

The world of process automation is governed by procedures. While we like to refer to the process industries as being largely "continuous", this could not be further from the truth. Process manufacturing is constantly in flux. Whether you are doing a startup, shutdown, grade change, or are in the middle of a maintenance turnaround, your plant is governed by procedures and transitional states that can either run smoothly and provide you with superior plant performance, a safe and orderly start-up/shutdown or they can cost you in terms of unplanned shutdowns, incidents, lost product, and lost opportunities.

But what is a procedure exactly? Procedures are nothing more than a predetermined set of tasks that must be completed in a set order and in a consistent manner every time to achieved a desired goal or end result. Startup, shutdown, and grade change are probably the most common. Of course, procedures vary in terms of complexity and duration, and requirements for plant procedures vary for each industry, but the common fact remains that plants cannot operate without them.

Bulk Chemicals	Polymers	Refining	Power
Startup	Startup	Startup	Startup
Shutdown	Shutdown	Shutdown	Shutdown
Grade Changes	Grade Changes	Crude Switchover	Generation
Line Switchover	Switchover	Regeneration	Industrial Boiler Control
Continuous Reactions	Transition Management	Decoking	Load Management
Cleaning		Transition Management	Grade Changes
Decoking			Cleaning
Transition Management			Descaling

Manual, Prompted, or Automated?

Today, operational procedures can be lumped into three primary categories – manual, prompted, and automated. In manual procedures, the operator performs the necessary actions required either through their own years of experience or through following Standard Operating Procedure (SOP) manuals. As one might expect, the consistency with which manual procedures are performed can vary greatly depending upon the level of experience of those carrying out the procedures. Manual procedures also call for manual records to be kept, which can equally vary in their effectiveness and quality. Electronic records are preferable, but their quality can also vary depending upon how well they were entered into the system, and only reflect the procedures that were purportedly carried out. There is no way to verify that th e manual procedures followed were in fact consistent with printed SOPs.

There is no equivalent to the ISA-88 standard in continuous process manufacturing, and operations such as starting up and shutting down a refinery operation, for instance, are normally considered by many end users to be a craft or an art form that relies heavily on experience and knowledge of the particular plant and its quirks

Prompted operational procedures go one step further, where the procedures are implemented in the process automation system and the operator is prompted to acknowledge that each step has been successfully completed in order to continue. Prompted procedures make it easier to keep electronic records and verify that procedures were followed correctly and can even decrease transition times and decrease variability.

Like prompted operational procedures, automated procedures are implemented in the process automation system. The difference is that automated procedures will go through the entire operational sequence before stopping, unless there is intervention from the operator or the system on an exception basis. Automated procedures can provide even more reductions in variability and transition time.

Many industries have been using prompted or automated procedures for some time. The batch industries such as life sciences and food and beverage have been using the ISA-88 standard for years, which defines a modular approach to batch automation and batch procedures. In the continuous process industries, however, prompted and automated procedures are not the established way of doing business. There is no equivalent to the ISA-88 standard in continuous process manufacturing, and operations such as starting up and shutting down a refinery are normally considered by many end users to be a craft or an art form that relies heavily on experience and knowledge of the particular plant and its quirks.

This is not to say that automated procedures are unknown in the continuous process industries. Many companies have implemented sequence logic that allows procedures to be automated. However, these have been done largely in an ad-hoc framework using custom programming methodologies that can become cumbersome when it comes time to upgrade the automation infrastructure. This ad-hoc approach also carries a high cost of ownership, since they have to be maintained by the end user. Changes made to the code over time can create a tangled mass of "spaghetti code" that can be impossible to translate.

Many end user companies in the process industries today are also the result of mergers and acquisitions. Along with that come the many system platforms and unstructured code implementations that have accumulated over the years. Clearly, this is not a sustainable way to do business. As a result, more and more end users are standardizing on their approaches.

Process Industry Issues Call for Modular Procedural Automation

An open and modular approach to automated procedures is required to overcome the issues associated with proprietary approaches based on proprietary code. Modular applications enable functions to be more manageable and standardized across plants, sites, and the entire business enterprise. A standardized approach means that engineering costs, labor costs, and total cost of ownership are reduced.

Modular procedures go beyond the software and systems that these procedures are implemented upon, however. It is unreasonable to think that all procedures can be automated. The right balance must be struck between manual, prompted, and automated procedures. The approach to procedure management should be a consultative methodology with an eye toward improving operations and reliability in th e plant. Specifically, the benefits include increased safety, energy savings, increased throughput, improved quality, faster transitions, and the ability to capture knowledge from a rapidly aging workforce. Modular procedural automation will be the way of the next generation of plant operations.

The following issues are facing process industries – keeping safety health and environment (SHE) at the forefront, aging workforce, improving availability, quality, increased operator workload. Distracted operators make mistakes. Operational error is the single biggest reason for unscheduled shutdowns.

Procedure Automation Enables Human Reliability

In demand limited industries like refining, the overarching objective is to improve utilization, which cannot be achieved without reducing unplanned downtime. Research has shown that the largest reason for unscheduled downtime is operational or human error, which accounts for approximately 42 percent of the unscheduled shutdowns in the process industries. Of that 42 percent, 16 percent is directly related to procedural error. When researching the role of operators in the Refinery of the Future, several major operating companies concluded that this can be addressed first by empowering a higher level of perspective, one which enables flawless intervention by exception and relieves operators of manual tasks, freeing time for more value added activities.

Distribution of Unscheduled Shutdowns and Slowdowns

Procedure automation was one of the key process automation system functions identified to provide the environment of flawless intervention, along with alarm management and the operational perspective. The know-how and operating level of experienced operators can be incorporated into automatic sequences and used to standardize operating methods and improve the efficiency of all operators.

ARC has a vision for the Collaborative Process Automation System (CPAS) of the 21 st century. An important part of this vision is that in developing an overall automation strategy, humans should be allowed to do what they do best and automation should be allowed to do what it does best. Humans are good at ad-hoc intervention and non-linear reasoning. They do best when empowered with an overall production cycle perspective. Machines and automation are good at repetitive functions, steady-state and transition management. Automation provides an environment for unbroken, precise execution, linear reasoning, and can consistently implement best practices through automated procedures.

Increased Plant Safety Modular Procedural

Automation also provides enhanced process safety. There is increased emphasis on health, safety, and environment in today's process plants. Accidents continue to occur in many manufacturing facilities even after the installation of safety systems and other safety protections that were initially considered adequate. Many operators, when faced with an abnormal situation, are faced with a landslide of information, and possibly a flood of alarms. It can be difficult to sort through all the noise to get to the right data and make a good decision in a crisis. Procedure based recovery from abnormal situations is faster and more reliable than recovery based on random operator knowledge – even from expert operators.

The Safety Lifecycle Model can benefit from Modular Procedural Automation

Modular Procedural Automation could possibly also stop abnormal events from happening in the first place. Several major incidents in the past few years were caused at least in part by plant personnel not following proper operating procedures, while under pressure. In addition, as we all know, operators and maintenance personnel do not always follow standard operating procedures when they are required to execute them manually. For every one of these incidents that occurs due to poor procedural operations, there are probably ten near misses.

Modular Procedural Automation can also help in the development of the safety lifecycle. A study conducted by the Health and Safety Executive in the UK states that many failures are due largely to inadequate safety management. To ensure safety, current safety standards specify safety lifecycle activities that need to be followed over the entire life of a production system. This is known as safety lifecycle management. It is a method or procedure that provides the means to specify, design, implement, and maintain safety systems in order to achieve overall safety in a documented and verified manner. Modular Procedural Automation can help implement the procedures required to developed safety lifecycle management in the same way it can help execute transition changes and other functions in the plant.

Constrained Worker Resources

Finding experienced personnel is a challenge for the process industries. The retiring wave of "baby boomers" in North America and the shortfall of qualified engineers in Asia and other parts of the world are creating a skilled labor shortage which will drive growth in demand for services and more sophisticated forms of automation.

Corporate Knowledge Sources

However, approximately 80 - 85% of corporate information is in an unstructured form. Source: Stratify, 80-20, Knowledge Management Research Center

In an interview, an executive at a major refining company stated they had lost 2,500 years of experience recently when 100 operators retired at one site, each with an average of 25 years of experience. As further evidence, a team at a major chemical company analyzed their plant demographics and found one of their larges t plants would lose 75 percent of its operating staff to retirement by the en d of 2009. Another major refiner has to fly its experienced peple to various sites all over the USA to augment the existing workforce during start-up s and another has to rehire retired workers to do unit startups because they do not have enough people left on site that know how to do these procedures.

Driving a Modular Procedural Automation strategy in your company can also help facilitate the process of capturing the knowledge of skilled employees before they retire or are laid off. Recent research shows that up to 42 percent of all corporate knowledge is actually stored in employee's brains.

Many firms continue to operate with lean technical staffs, yet the level of technological complexity is increasing. The timely flow of information, data, and knowledge is more important than ever in the process industries. This downward trend in staffing an d the increased demand for accurate, real time information will translate into additional systems and higher, more sophisticated levels of automation being applied, including Modular Procedural Automation. The smaller, less experienced workforce that will exist in future must be empowered with new technologies and workflows that can transfer knowledge on demand.

Workers Under Stress

Operators today can be under huge amounts of stress that will have an impact on their responses to abnormal situations. Under normal operating conditions, the operator can apply their training and basic knowledge successfully. Even at the first sign of a fault, the operator is capable of acknowledging the event and responding accordingly. As the situation begins to deteriorate, however, the ability of the operator to respond effectively rapidly deteriorates.

End Users See the Value of Sequence Based Operator Procedures

Transition Management Objectives for Operator Assistance

A 2008 survey by ARC indicated that continuous manufacturers are now seeing effective and repeatable transition management along with the use of sequence based operator procedures as a competitive advantage but in the continuous process industries, there is no prescriptive standard they can use to base their procedures on. Most plants do not always operate at steady-state and do not always rely upon applications such as advanced process control (APC) to manage transitions. Instead, as we mentioned earlier, they use varying degrees of manual and automated procedures to manage complex procedures during shutdown, startup, grade changes, and other planned and nonplanned unsteady-state events.

ARC's Views on State Logic Control

State transitions in the plant are a major subset of Modular Procedural Automation. Since state transitions are the most hazardous parts of operations, and the primary time where off-specification product and its resulting yield loss are generated, ARC's Collaborative Process Automation System (CPAS) model requires that there be no difference in the way continuous and batch control systems are defined. Economics should then determine if the state transition procedures are automated, prompted, or are made advisory in a manual state transition procedural operation. In all cases, continuous processes must be treated as though they were batch processes and the process logic must be defined for the state transitions of all processes.

One of the most common continuous controlled processes is the electric power steam boiler. The recovery boiler used in pulp mills is a particular form of steam boiler that has a record of being a hazardous process since there have been many recorded recovery boiler explosions in history. Recovery boilers are not hazardous during full load operation. All recorded explosions have occurred during startup or shutdown of the recovery boiler while operations were under manual control.

Even when the operation of the process under state change conditions is too complex for analysis based on first principles modeling, it is always possible to codify operations to automate the best operator actions

Distillation is the most common unit operation in the continuous controlled equipment category. Steady state distillation control has been the subject of complex Multivariable Process Control methods for process improvement and has been very successful in reducing operational variances. Rarely are there any automatic controls available that will effectively increase or decrease the throughput rate of a distillation column while holding constant product quality – this is done manually by the process operator. We know that some operators are far more skilled at "lining out" a column after a production rate change than others. We sometimes measure operator effectiveness for these manual operations in hours required. Yet, chemical engineers know a great deal of the physical chemistry involved during a production rate change and could easily build th e sequential control systems to make such changes automatically.

All control systems should provide the basic languages for controlling sequential and concurrent operations, and in ARC's view that language is a sequential function chart (SFC). Even when the operation of the process under state change conditions is too complex for analysis based on first principles modeling, it is always possible to codify operations to automate the best operator actions.

Procedural Automation Standard for Continuous Processes Submitted to ISA

Given the lack of a standard for Modular Procedure Automation, it seems only logical to propose one. As of November 2009, a new ISA standard has been proposed to provide industry with benchmarking data and design considerations for procedural automation in continuous processes. The proposed standard may address topics ranging from models and terminology to modularization of procedural steps with an eye toward reusability and lower cost of ownership. Other potential areas to be addressed include:

• Exception handling for handling abnormal situations
• State models for procedural logic
• Process unit orientation with operational perspective
• Compliance requirements
• Recommended best practices
• Implementation of start up, shutdown, abnormal situations, hold states, and transition logic
• Recommended target platform (i.e. control system vs. safety system) for different types of procedures
• Lifecycle management best practices
• Training and certification best practices
• Recommended display hierarchies

The committee has a lot of source material to draw from in creating this standard, including existing ISA standards such as the ISA-88 batch control standard, ISA-84 safety standard, ISA-18 alarm management standard ,ISA101 standard for human machine interfaces (HMI), and the ISA-95 manufacturing operations management standard. Other groups that could provide potential source material include EEMUA, NAMUR, and the Abnormal Situation Management Consortium (ASM). In theory, the final standard will incorporate elements of training, a certification process, best practices, as well as specific HMI strictures.

Yokogawa Toolsets Support MPA

As a major supporter of Modular Procedural Automation and the procedural automation standard, Yokogawa already has a toolset that can support Modular Procedural Automation (MPA), from applications as small as scheduling sequences consisting of many sequential function charts (SFC) steps to more complex an d integrated multiproduct solutions incorporating the Centum VP process automation system. SFC is the desirable means of defining the state relationships, concurrency, and transition conditions – otherwise known as the state model of the process – in the control system. With the application of additional procedural programming, the state model itself becomes the root of the control system.

The primary tool that Yokogawa has to implement MPA in the continuous process industries, however, is Exapilot– an automated procedural control solution now installed in over 1,000 plants around the world. Using Exapilot to describe steady state and non-steady-state operations by experienced operators as a graphical flow diagram of the sequence of actions, the operations can be fully or semi-automated. In addition to Exapilot, the Exaplog event analysis package helps to refine the controllability of a plant and identify parts of the plant operation that could be improved, by analyzing alarms and operator actions side by side. Combining Yokogawa's consulting skills with products such as Exapilot allows users to integrate operator actions and the control system into a single unified environment for best practice plant operation. Some operators have a lot of knowledge, but are not familiar with computers or Exapilot, so someone has to put this knowledge into Exapilot. Yokogawa knowledge engineers specialize in capturing this knowledge from experienced workers and driving it into the system.

Major Japanese Chemical Manufacturer Benefits from MPA

Yokogawa assisted a major Japanese chemical manufacturer in the implementation of MPA, which resulted in considerable benefits. The company implemented Yokogawa's Exapilot application as an operational support application used to incorporate operator expertise to automate operating sequences including reactor process startup, shutdown, and cleaning. MPA was implemented in conjunction with an overall reinstrumentation program.

As is the case with many companies today, this resin manufacturing plant had to change its production strategy from a single stream of product to a small batch multiproduct operation. The key difference is that this plant had to do it over ten years ago. As we mentioned earlier, Japanese companies had to go through the shift in workforce with mass retirements, companies cutting back because of the recession, and influx of younger more inexperienced workers back in the nineties, so they serve as a good example for what many companies are experiencing today on a global basis.

The shift to more flexible production, combined with grade changes in feedstock, increased the complexity of operations by a factor of around 1.5, with the overall DCS operations increasing to about 50 per day. The number of "non regular" operations increased to nearly 200 per day. According to the company, "if part of the plant is stopped or started, the process is switched, or cleaning performed, the variation in terms of number of operations can be as high as 3:1." Longer intervals between maintenance outages means less opportunity to perform startup/shutdown sequences. This means that less experienced operators would have fewer opportunities to do a startup or shutdown procedure. With the reduction of operator staff being a primary strategic objective for the company, such an increase in overall level and complexity of operations leaves little alternative other than to implement procedural automation.

So, in 1996, the company started automating operational sequences of the DCS and related applications. The Exapilot application was used to implement operational sequences such as pump start/stop, MV/SV ramping, control valve opening setting, and field operations. Phases of operations are represented as icons that the user can string together to create entire automated sequences. Emergency operational sequences were also part of the implementation, including Pause, Restart, Skip, and Abort/Skip.

As we said earlier, MPA is not just ab out automating all operations. It is a combination of manual, prompted, and automated procedures. The company also incorporated operator guidance for prompted tasks, field device maintenance related operations, alarm confirmation, and other prompted and manual operations.

The plant itself consists of several reactors. Impurities build up on the reactors and they must be cleaned every three to four days. The means stopping the reactor, cleaning it, and restarting each reactor on a weekly basis. Related operational sequences include line switching, adjustment of process variable fluctuations before and after switching, setting of production volume, and load adjustment. These operations alone represented 530 person hours of DCS operations and monitoring work per year. Cleaning also requires a lot of coordination between the instrument room and the field. One of the primary objectives of the project was to reduce overall DCS operations and monitoring time to one quarter of what it was previously. Phase time was also to be reduced by three hours, which would have a corresponding production increase of 200 tons per year.

Since the implementation of MPA through Exapilot, the number of persondays of engineering work has been reduced between 10 and 20 percent, and the number of days required from planning to implementation has been reduced to 1/5 of what it was previously. The functional icons are also a reusable software resource, which reduces the cost of future implementations.

Strategies and Challenges Moving Forward

Developing a successful standard for ISA, IEC, or any standards body is a challenging task. Fortunately, there is much source material that the future procedural automation standards committee could draw from, particularly ISA-88 and its application to continuous processes. ARC believes there is value in creating a procedural standard for end users, and that MPA has the ability to address many of the challenges that the process business will face and is facing today, from the shortage of qualified and experienced personnel to the increased focus on health, safety, and environment. Getting the standard completed and having the future standards committee bring it forward in a constructive way remains the primary challenge both for Yokogawa and the procedural automation standard. ARC believes that the future standard must be defined in a way that makes it easy for end users to deploy, relies on open technology to implement, and conforms to other existing standards.

Refining – Reduced crude switchover time by 69% – Faster Transition – 25% faster – Smoother Transition – 10% fewer problems Petrochemicals – Increased olefins reactor runtime by 150 hrs/yr – Improved quality variability by 27% – Reduced operator manual manipulations by 90% Polymer Process – Enables consistent product changeover time, previously varied by 50% – Safer start ups and shut downs – Reduced introduction time for proposed new product ASM Consortium – Unexpected events cost 3-8% of annual production capacity – Approx. $10 billion per year

Reported End User Benefits of Implementing Modular Procedure Automation

But what is the real value of creating a standard for procedural automation? Many suppliers have tools that can provide a path to procedural automation. The benefits of an MPA standard would be similar to those provided by the ISA 88 standard for batch automation. Design time and time to startup are both greatly reduce d. ISA-88 could be called the "Universal Batch Translator". Every supplier large and small currently offers an ISA-88 "aware" batch control system and everyone in the batch industry knows what control modules and equipment modules are. They know what the system commands are and what impact they should have when to go into hold or abort, when to go back to run and how and so on.

In business terms, this has saved many companies a lot of money – in terms of system development, changes to system design, engineering and the flexibility to respond to changing market conditions without having to reprogram the control system completely. ISA-88 is the most successful set of standards ever produced by ISA. A standard for MPA in the process industries would have similar benefits and would experience a similar level of success.

Analyst: Larry O'Brien
Editor: Dick Hill

Acronym Reference:
For a complete list of industry acronyms, refer to our web page at www.arcweb.com/Research/IndustryTerms/

API Application Program Interface
B2B Business-to-Business
BPM Business Process Management
CAGR Compound Annual Growth Rate
CAS Collaborative Automation System
CMM Collaborative Management Model
CPG Consumer Packaged Goods
CPM Collaborative Production Management
CRM Customer Relationship Management
DCS Distributed Control System
EAM Enterprise Asset Management
ERP Enterprise Resource Planning
HMI Human Machine Interface
IOp Interoperability
IT Information Technology
MIS Management Information System
OpX Operational Excellence
PAS Process Automation System
PLC Programmable Logic Controller
PLM Product Lifecycle Management
RFID Radio Frequency Identification
ROA Return on Assets
RPM Real-time Performance Management
SCM Supply Chain Management
WMS Warehouse Management System

CENTUM, ProSafe, STARDOM, VigilantPlant Services, and Vnet/IP are either trademarks or registered trademarks of Yokogawa Electric Corporation. All other company brand or product names in this report are trademarks or registered trademarks of their respective holders.

Founded in 1986, ARC Advisory Group is the leading research and advisory firm for industry. Our coverage of technology from business systems to product and asset lifecycle management, supply chain management, operations management, and automation systems makes us the go-to firm for business and IT executives around the world. For the complex business issues facing organizations today, our analysts have the industry knowledge and first-hand experience to help our clients find the best answers.

All information in this report is proprietary to and copyrighted by ARC. No part of it may be reproduced without prior permission from ARC. This research has been sponsored in part by Yokogawa. However, the opinions expressed by ARC in this paper are based on ARC's independent analysis.

You can take advantage of ARC's extensive ongoing research plus experience of our staff members through our Advisory Services. ARC's Advisory Services are specifically designed for executives responsible for developing strategies and directions for their organizations. For membership information, please call, fax, or write to:

ARC Advisory Group, Three Allied Drive, Dedham, MA 02026 USA
Tel: 781-471-1000, Fax: 781-471-1100
Visit our web pages at www.arcweb.com