Production Performance Ratings

Abstract

Key Performance Indicators (KPIs) are used in batch processing industries as measurements of production performance. Their use is one element in the current trend of real-time performance management. A single KPI used as the primary measurement of production can cause other dimensions of production performance lose importance. When multiple KPIs are used to measure a batch's production performance it can be difficult to reconcile differences between them for individual batches or for groups of batches. KPIs based upon meeting a target, or specification, measure absolute performance yet do not provide relative information regarding how a batch performed against its peers. The peer comparisons are important for monitoring variability of production performance, which is a critical factor in documenting ROI.

One method to compare batches with several KPIs is to calculate a composite KPI providing a single top level metric for easier identification of problem and exemplar batches. This concept can be extended to include KPIs that measure variability between peer batches and to normalize the value from 0 to 100% to provide a consistent, easily used value. This KPI is called the Production Performance Rating that reflects the production performance for a batch based upon its peers.

Introduction

Many different metrics, or Key Performance Indicators (KPIs) can be used to measure the effectiveness of batch production. Previous WBF papers by Luedecke, Sanders & Baker and by Jensen list several, including:

• Batch cycle times
• Actual results vs. planned/recipe expectations
• Capacity utilization
• Batch exceptions
• Attainment to schedule
• Attainment to standard

Some of these items are very general categories, so note that an item like "Attainment to standard" can include typical golden batch criteria such as critical process measurement trends meeting a "gold" standard", laboratory results meeting targets for a recipe or product, material additions meeting recipe targets, and labor costs for a batch meeting expectations.

Of course the ability to repeatedly produce on-spec and saleable product is absolutely critical, however once this is accomplished there remain many opportunities to improve production. As the aforementioned authors noted key benefits of focusing on KPIs beyond this basic level are lowering costs and increasing capacity without requiring large capital expenditures.

As real-time performance management receives increasing attention we see batch production KPIs exposed to higher levels of management in real-time on dashboards, web pages and in e-mails. One difficult aspect of using multiple KPIs is that each one usually does not have an equal importance with regards to overall production efficiency and costs. Compounding this, individual KPIs can give conflicting indications of performance and can make it difficult to judge overall performance.

One method to make handling multiple KPIs per batch easier is to calculate a composite KPI that provides a single top level metric for rating batches. The composite KPI can be refined to weight each input KPI, measure variability of input KPIs between batches and to normalize the KPI value from 0 to 100%. When this is done the composite KPI is called the Production Performance Rating. The addition of batch variability provides insight as to how batches compare against their peers.

Production Performance Ratings for batches can be rolled up to master recipe versions, master recipes, and products to enable comparisons regarding their production performance. While one normalized value cannot express all the subtleties of a batch's production it can act as a quick measurement of production performance and serve as a filter for finding top and bottom performing batches and products.

What is a Production Performance Rating?

A Production Performance Rating is a technique used to rate batch production performance on a 0-100% scale. The rating's inputs are individual KPIs based upon meeting targets and specifications as well as KPIs that measure production performance against a batch's peers, the other batches based upon the same master recipe version. This combination is weighted to meet the needs of each application and yields an overall percentage that can be used for quick identification of poor performing and top performing batches.

The criteria and calculation for the rating will vary per application, but using the ISA-88 standard a set of application independent KPIs can be developed that will provide a good starting point for most batch processing facilities. This set of KPIs can be refined with application specific criteria and customized calculations can be developed to provide more meaningful results. Some of the standard ISA-88 derived KPIs are:

• Cycle Time
• Number of times a batch was held
• Percent of time a batch was in hold
• Number of events associated with a batch

By themselves these measurements have no context so they must be compared against a target or against their peers.

In some cases comparison to a target makes sense, for example lab results may be required to meet a specific level or process measurements may need to meet specific criteria before a product can be released. These KPIs provide a base level for defining an acceptable batch of product. However they do not provide adequate insight into the batch's production performance.

For example a set of batches may meet the lab results and process measurement requirements to be released, but some of the batches may have been produced with much shorter cycle times and less operator involvement than other batches. Those batches produced with shorter cycle times and less operator involvement should be considered higher performing batches since they met the release requirements at a lower cost (less asset utilization and less labor cost).

Pure target based KPIs would not differentiate between the higher and lower cost batches. Peer based KPIs can be used to differentiate between these batches.

Used together target and peer based KPIs can be used as inputs to the Production Performance Rating calculation, a example of input KPIs is shown in Figure 1. The inputs should be weighted to reflect each application's needs. For example when meeting release targets is critical target based KPIs can be weighted to force a very low rating, even forced to 0, when any release criteria is not met. When the release criteria are met the peer based KPIs can be used to rate the batch's production performance against other batches of the same master recipe, or even master recipe version. This provides finer resolution that will enable identification of good and poor production characteristics.

Production Performance Ratings are normalized values from 0 to 100% in order to provide an easy to use scale. This normalization can require tuning the weighting factors and calculation in order to achieve meaningful results. Once tuned the normalized scale provides for quick identification of top performing batches, whether those above 95%, 90% or 80%, as appropriate for the application.

Figure 1 - Target and Peer Based KPI Inputs

The normalized rating value can lead to easier identification of golden, or top performing, batches. Traditionally many batch operating companies have used an individual batch as the "gold" standard. Some have identified one golden batch for each unit recipe and built up a composite golden batch by assembling unit recipes from different batches. Production Performance Ratings enable the rapid identification of the top performing batches and, if carried down to the unit recipe level, the top performing unit recipes. This permits process and automation engineers to move beyond the single golden batch concept to look at what production characteristics the top performing batches have in common – was it the equipment used, time of day, personnel, or ingredients?

Perhaps more important than examining the golden, or top performing, batches is the need to examine the bottom performing batches, or the "brown" batches. Not all of these brown batches necessarily made un-releasable or off-spec product, they may of made acceptable quality product, but perhaps they required more re-work resulting in longer cycle times and more operator involvement. Understanding the characteristics that lead to brown batches may enable corrective actions that will yield less production variability and lower costs faster than chasing the golden batches.

To facilitate identifying golden and brown batches the normalized rating value can be used as a filter criterion in a batch historian when selecting batches for analysis and reports.

Rating Calculation

Calculations used to determine a Production Performance Rating can vary in complexity. A straight forward calculation, as shown in Figure 2, may calculate a value for each input KPI based upon a comparison to a target value or the mean and standard deviation of its peers. This value is then weighted to give individual KPI inputs different influences in the final rating, the calculation is performed using a default percentage as a starting point and the result is normalized to 0-100%. This method can easily be expanded to include new KPIs.

Figure 2 – KPI Aggregation Formula

More complex formulas may be used but the key principles to use are:

1. Compare input KPIs to a target or its peers
2. Weight each input according to its importance to production performance
3. Normalize the result, clamping it to 0-100%

When a KPI is compared to those of its peers a simple rule of thumb can be used as a starting point. This rule, shown in Figure 3, draws upon concepts from six sigma in that low variability is desirable. When a batch's KPI, for example its cycle time, is within the upper and lower standard deviation of all its peer batches then this is good and should increase the rating. When the cycle time is outside the standard deviation limits this is poor behavior and should decrease the rating. In order to reward those batches with improvements those with cycle times below the mean and above the lower standard deviation limit should have a slightly higher rating.

This rule of thumb is intended to reward batches that provide consistent cycle times, with a slightly higher reward for those trending below the mean. When a cycle time is below the lower standard deviation limit this should be considered "too good to be true", perhaps it is due to a breakthrough in production performance, but this decision should be held until it is repeated enough for the mean and standard deviation limits to change sufficiently for these batches to be in the green zone.

Figure 3 - Peer Comparisons Using Standard Deviation

The number of standard deviations to use should be adjusted for based upon the industry, company and process. In most cases three standard deviations is a good starting point, but in processes with low variability this may not provide sufficient differentiation so a lower number may be desirable.

Peer batches need to be defined for each application. In the strictest sense all batches based upon the same master recipe version are peer batches. If the master recipe is revised the batches based upon the new version should be grouped separately from previous batches. In other applications batches from multiple master recipes may be considered peers if the recipes are similar enough.

As more peer batches are produced the mean and standard deviation values will drift as new data points appear. At some point the Production Performance Rating for all peer batches will need to be recalculated in order to provide a level comparison. Whether this should be done each time a batch completes, periodically or upon demand is an application specific decision. Depending upon the process and products involved it may be necessary to create different Production Performance Rating formulas for each master recipe or group of master recipes in order to customize them for differences in the processes or products.

Batch vs. Unit Recipe Ratings

Discussion so far has referred to batch level Production Performance Ratings. The batch level rating is the most visible but is often the unit recipe level can provide a more accurate reflection of key performance areas.

One weakness with batch metrics such as cycle time is that one key unit recipe, say the reaction unit recipe, can represent the true bottleneck or high cost/risk portion of a batch. Other unit recipes such as those for preparation, mixing, post-reaction processing, and drying can provide variability that is actually caused by another batch's reaction, or other key, unit recipe. When this occurs unit recipe Production Performance Ratings should be calculated and used as the primary method to compare batches. Alternatively instead of using the batch cycle time as a KPI input to the batch Production Performance Rating a key unit recipe's cycle time could be used.

While this could be carried down to the operation and phase level there may be diminishing returns in these cases. If they are calculated then comparisons could be made for very specific periods of a batch's execution.

Ratings Roll-Up

A powerful use of Production Performance Ratings is the ability to roll-up results. Roll-ups can be performed on many levels, for example:

Roll-up	Benefit
Batches based upon the same master recipe version	Compare if master recipe modifications are increasing or decreasing the Production Performance Rating, Compare Production Performance Ratings for different master recipes & products
Batches based upon the same master recipe, regardless of version	Compare the latest master recipe version's Production Performance Rating against the average of all versions for that master recipe
Batches based upon a group of master recipes	Compare ratings for products that have more than one master recipe or for families of products.

Rolling up ratings can provide a quick indication of trends. For example Figure 4 shows the Production Performance Rating for batches rolled up by master recipe version and master recipe.

Figure 4 - Production Performance Ratings Rolled Up By Master Recipe Version

This roll-up enables the quick comparison of the rating for each version of Recipe A or B thereby helping indicate a trend. When rolled up to the recipe level comparisons between Recipe's A and B can be made, assuming the same formula was used for both recipes.

Applications

Production Performance Ratings can be used as metrics for dashboards but can also provide a powerful indexing tool for production analysis. Using a batch and unit recipe's ratings as a filter it is easy to find top and bottom performing batches for a number of criteria such as production lines or units, time of day, products and material lots. Once used to create sets of batches the batches can be analyzed using the KPI inputs to gain a better understanding of what causes production problems and higher costs.

When used with a batch historian historical batch data can be analyzed for trends over time, not just on a recipe or product basis but also to detect other correlations such as if certain operations, phase classes, or units are commonly associated with high or low ratings.

For example a process engineer may find all batches of a product in the last quarter that had performance ratings below 50%, then drill down to find the root causes of the lower performance ratings. Or they may find the averages and std dev for performance ratings for all batches of a product, then compare different products to see which have the greatest variability. In this case peer comparisons can be made using batch's Production Performance Ratings as shown in Figure 5.

Figure 5 - Peer Comparisons of Batch's Production Performance Ratings

Conclusion

Production Performance Ratings provide a composite KPI that can be used to identify top and bottom performing batches. Ratings should not be used to reflect product quality levels, instead focusing on production performance of on-spec batches can lead to the root causes of production problems and detection of characteristics the lead to top performing batches.

Ratings can also be used to expand the concept of golden batches from one exemplary batch or unit recipe to a set of excellent, top-performing batches from which common traits and characteristics can be gleamed. The golden batch concept can also be expanded to include "brown" batches, the low performing batches which can be analyzed to identify corrective action to prevent future production problems.

Disclosure: Portions of this paper are based upon Japanese Patent # 02N0113, equivalent U.S. patent applied for.

References

Japanese Patent Number 02N0113

"Six Sigma and S88 Unite for Batch Automation Productivity Improvement" by Bruce Jensen, WBF North American Conference 2001

"Throughput Improvement at Henkel Surface Technologies" by Rick Luedecke, Doug Sanders, Dean D. Baker, WBF North American Conference 2002