Emergency Callout Service (ECS) at Christmas Time Quickly Resolves Problems in Plant’s Critical Operations
One early-afternoon during Christmas Season, a client contacted Malisko Engineering to have a qualified technical resource on a plane the very next morning to assist with getting a critical system backup and running; a system that Malisko had nothing to do with, at a plant that Malisko had not been to. One of Malisko’s dedicated and extremely capable senior engineers offered to accept the challenge.
Fix the problems on the control system and get plant operations back up and running quickly.
- Extremely aggressive schedule – “get it up and running NOW”.
- Little to no documentation on existing system.
- Had to heavily rely on the resident expert consultant to gain the tribal knowledge of desired system functionality.
- Connect EWS to FactoryTalk Direct and Primary HMI Server.
- Replicated Primary HMI Server to Secondary HMI Server and tested redundancy.
- Create a repository on EWS for all current controller programs, HMI Servers and PanelView runtime files.
- Acquire new server infrastructure via Malisko to support development of a PlantPAx server system solution that can provide high-availability (HA) or fault-tolerant (FT) redundancy and scalability.
- Provide alarming for the acid system and resolve existing L32E controller issues by replacing L32E with the recommended CompactLogix L35E controller.
- Update devices during the upgrade so produce-consume communication throughout the automation system recognizes the new controller and balance of plant systems remain stable.
- Improve system reliability in the interim by rebuilding a critical Industrial Control System (ICS) component in the Engineering Workstation and Primary HMI server.
- Identify and mitigate ICS system risk for Alarms and Events specific to the system.
- Restore device alarm functionality
- Verify remote technical support
- Successful EWS rebuild and configuration in a short timeframe thus getting plant operations back up and running.
- Client was extremely appreciative and thankful to Malisko’s Senior Engineer for making personal sacrifices to change personal plans and travel on such extreme short notice.
- Because of the dedication and commitment of Malisko’s Senior Engineer’s ECS support and results from providing a high level of technical support, expertise and efficiency while onsite during critical plant production downtime the Client is in the process of:
- Having Malisko Engineering upgrade the entire IACS and OT Network
- Discussions with Malisko underway to replace existing HMI servers (Primary/Secondary) with a fault-tolerant server capable of lasting 20-30 years
- Having Malisko perform HMI & PLC system re-control with Rockwell’s latest Library of Process Objects (LPO) solution utilizing most current FactoryTalk View SE version, and expanding HMI technology with ThinManager and Thin Client hardware
- Discussions underway with Malisko to determine path required to update existing Industrial Network to a Cisco Validated Design (CVD) referencing Converged Plantwide Ethernet (CPwE) methods and best practices in order to guarantee IACS and Industrial Network performance via improved communications latency benchmark.
Identifying the Root Cause of the Problem
- While restoring device alarm functionality on a CompactLogix L32E controller, Malisko’s Senior Engineer discovered an issue with “controller utilization”.
- Provided root cause to why L32E could not provide device alarms – max utilization caused runtime issues with plant equipment.
- Established and verified remote support by Malisko while onsite.
Malisko Engineering was given the opportunity to provide emergency callout service support to this new client based on the quality of work we provided in the past, and projects we successfully delivered to the client subsidiary company.
The client subsidiary company was instrumental in recommending Malisko during their time of difficulty. Malisko responded immediately and because of our quick action, have since built a strong impression with site management and its stakeholders about the quality of work we do for integration related applications.
Because of the quality and professional nature of our Senior Integrators’ effort in this client facing engagement, Malisko gained a new client who is highly reliant on our integration services.
500 N Broadway
St. Louis, MO 63102
Western Regional Office
4600 S Ulster Street
Denver, CO 80237
Northern Regional Office
3610 Oakwood Hills Parkway
Eau Claire, WI 54701