Spend a Little to Reduce the Project Risk

Your team has established and communicated a financial justification for your site’s control system replacement project. Now all eyes are on you to ensure that the project is executed in a timely manner and within a certain budget. As Benjamin Franklin said, if you fail to plan, then you plan to fail. This is the second installment in a series on industrial control system migrations: Front End Loading.

 

Why Does Anyone do Front End Loading (FEL)?

FEL is all about minimizing the risk: The cost risk, the schedule risk, and the scope risk for any capital undertaking.  Early in a project, the cost impact of changes is low, but it increases, often drastically, in later stages. Implementing a stage-gated process gives an owner/operator the opportunity to make strategic choices instead of falling victim to a haphazard outcome.

FEL can be multiple stages: FEL 1, FEL 2, and FEL 3. Some owners opt to only do a single FEL. However, the key is to consider all factors and engage in a level of FEL that is reasonable. Regardless of the number of FEL phases performed, all business needs should be evaluated, and a complete business case should be identified to minimize cost/schedule/budget (scope risk).

Wise owners recognize the value of FEL and are more willing to invest strategically up front. The highest return on value at the lowest amount of risk can be easily attained through proper preparation and planning.

1. Execution Strategy and Plan

2. Resource Plan

3. Risk Management Plan

4. Change Management Plan

 

Rip and Replace vs Phased Approach?

Two strategies for a control system migration would typically be to either “rip and replace”- that is replace the entire system all at once, or to carry out a phased replacement. Removing the current system entirely and replacing it in one fell swoop during a major turnaround is an option that can be considered. However, often, the business is unable to incur a lengthy outage or is unwilling to accept the risk of a delayed restart if things don’t work out as planned. In these cases, a phased approach can be carried out that uses any combination of the following strategies:

1. Perform a piecemeal upgrade of the system by area. For example, a site has multiple controllers in the plant that are geographically dispersed across different plant areas. Perhaps one plant area can withstand a complete outage better than another area or perhaps one area of the plant is suffering more reliability issues due to the old control system. This would be a piecemeal approach or a “mini-rip and replace” over several phases.

2. Replace the “top layer”, the servers and operator stations for the entire system. Consideration must be given to the compatibility of new computer hardware and operating systems to the controllers and other interfaces.  Virtualization of the new top layer should be given strong consideration, as this topology is certainly the trend not just in industrial control but throughout the IT industry.  Then, in a subsequent phase, the controllers and I/O can be replaced by plant area. Note that sometimes old controllers are be replaced but the old I/O is kept in place. The reasons for this may be to reduce the field construction labor and the risk of cutting over the field device wiring during the turnaround. This approach should be very carefully considered as the plant could be left with long-term reliability issues associated with the old I/O. 

 

Another approach to keeping some of the legacy I/O sub-system something I refer to as “The alien approach.” Many may recall a scene from the movie Alien where the creature has its tentacles around a character's face. Similarly, in the alien approach, an adapter and harness are installed on top of old infrastructure, typically at the I/O rack, with the intent to significantly reduce the cutover time of an upgrade. Often such strategies retain a “Marshaling Terminal Assembly” or “Field Terminal Assembly”, which themselves have electronic components that can fail.  Thus, the trade-off is living with a portion of the old hardware and an old interface, and the adapter/harness strategy can also limit access to important troubleshooting points.

 

A Living Functional Specification is a Rare Beast

How well a system is documented over its lifetime is a major factor for any migration. Some owners do an outstanding job of keeping up not only their hardware documentation and wiring diagrams, but also the documentation for how the process is controlled.

Unfortunately, a living functional specification is actually a rare beast. In any case, functional specification has to be thought through during FEL. Is it sufficient to guide the migration of the existing controller logic to the new controller? Will the project be upgrading the control platform with a system of the same OEM, and does the OEM provide migration tools? Regardless, the automation engineers will need adequate documentation to guide the migration. It is imperative to discuss the owner’s expectations and the programmer’s capabilities to reach an agreement during FEL that determines responsibility for developing the functional specification. Identifying accountability during the FEL phase, prior to the detailed execution phase will mitigate stressors when time is short and pressure is up.

Ideally, the functional specification would either be developed, or at least well-outlined, in FEL. At a minimum, the functional specification should be developed as a very early-stage activity during the execution phase.

 

Managing Expectations

When we say migration, exactly what do we mean? Where will it start? How far will it go? Will we reuse marshaling terminals? What is our cutover plan?  Determining owner/operator expectations is imperative for success in any project but it is especially so for a controls migration project. The owner’s objectives guide the end-goal and the path taken, ultimately playing into the level of effort, configuration, testing plan, commissioning plan, commissioning duration, etc. Unfortunately, too often clients have not thought through their objectives for a migration project.

As one simple example, an owner may say that they would like to roll out the new software but retain the look and feel of the old graphics. Accomplishing this can require a considerable amount of effort and cost, if it’s even possible at all. In addition, this strategy can miss a great opportunity to improve the operator interface experience, which can yield benefits in productivity, safety, and even operator stress levels.

Another example involves the control logic itself. Does the owner want to replicate the existing logic in the new control system, or are they open to enhancements that optimize control or provide more operational flexibility? New and powerful control capabilities and features are available in today’s modern control systems. It may be short-sighted to implement a brand new, powerful control system while leaving the basic, 35-year-old control in place because that option appears to be easiest and cheapest. There is extraordinary benefit in exploring the capabilities of modern control systems, but this too should be done in an FEL phase.

Some sites have been tempted to move forward with their migration and keep their same control logic and graphics, with the intent of making improvements after the new system is up and running. Perhaps that will be the outcome, but it is unlikely that there will be a compelling event in the future that will trigger additional spending to replace adequately working logic with something new. If the window is missed to optimize the controls during the upgrade, it will be difficult to justify doing so later.

 

Conclusion

Achieving the desired objectives in a control system migration requires a holistic, principled approach that combines thorough planning, effective communication, and proactive problem-solving.  Strategic decision-making and meticulous planning are key to any control system migration, upgrade, or replacement.  The Front-End Loading (FEL) phase is a pivotal element in minimizing risks associated with cost, schedule, and scope. Additionally, this process involves managing expectations and clearly communicating on objectives and priorities.