The Hidden Threat of Embedded Systems

by Kyle McCormick

In 1996, chip manufacturing companies shipped over 7 billion microprocessors and microcontrollers. If you count the ones shipped before 1996 the number is in the tens of billions which doesn’t even include the old computers like the PDP-11s, Siemans machines or machines from Texas Instruments. So why is this important? Because many of the chips and the older computers are not year 2000 compliant. This means that they will either fail or produce unpredictable results when the clock strikes 2000.

Microprocessors are embedded in machines that are doing their job every day, sometimes 24 hours a day. They are monitoring manufacturing operations, assembling something, controlling chemical additions, performing process control, checking schedules for maintenance, scheduling events, controlling heat and air conditioning, controlling the electric power plant, changing traffic lights, – well you get the-picture. These microprocessors have a heart beat that can’t skip a beat without impacting materials, process, operations or even human life in some instances.

Microprocessors are everywhere; inside microwaves, embedded in inter-continental ballistic missiles, in hospital equipment, automobiles, fire tracks and in airplanes too. How comfortable will you feel when the airplane your flying in at 35,000 feet has engine and navigation failure at the stroke of midnight on January 1,2000 because one of the hundreds of microprocessors aboard failed to rollover into year 2000 – and the pilot is helpless. Although this may sound alarmist, the problem is very real. KLM Airline ran a test recently and found over 120 onboard embedded chips that are not year 2000 compliant.

You will hear people mentioning a lot of different “microprocessor” type of terms. For our purposes, all of the terms are interchangeable with “Embedded Systems “. There are Microprocessors (computers, that plug into a circuit board), Micro-code (software that is burned in), Chips (standalone CPU or custom), DIPs (dual inline processors), E-Proms (Erasable programmable read only memory), Controllers, Logic Ladders, PLCs (programmable logic controls), DCSs (Distributed control systems) and Embedded (meaning hidden under the covers). With all of this high tech wizardry contained in a “black box” how can you tell if it is year 2000 compliant? Ask the company that built it – you reply?

Most systems and “black boxes” are constructed using many embedded systems from various manufactures, some of which are no longer in business. A very large chip manufacturer (Lucent), stated at a Year 2000 user group meeting that they could not tell if over 50% of the chips shipped from their facility were in fact year 2000 compliant. This is due to the “custom code” that is burned in for the customer ordering the chip (and supplied by the customer), which is out of the control of the original chip manufacturer. In many cases, only the original customer who ordered the chip will be able to tell if the specific version of the chip that you have in your hand is indeed year 2000 compliant.

So what do you do about this problem? The first step is to identify the areas that are suspect of containing embedded systems, anything with an electrical component should be suspect. Start with the highest level functions, such as “laboratory toxicology research”. Then inventory the equipment being used, noting the manufacturer, specific model, date of manufacture, function being performed and last inspection date (if appropriate).

Devices that are not sealed “black boxes” may require someone to get under the covers to inventory components, especially if they were custom assembled specifically for your organization. Avoid grouping devices together such as fax machines assuming that if one is compliant, they all will be. A fax machine has at least one embedded system to control date and time of in-bound messages and there are hundreds of brands, models and version levels. There have even been cases where two of the exact same machines, assembled on the same date, contained different chip levels and only one of them was year 2000 compliant.

Following the inventory, a plan should be developed that identifies the business process, the mission critical areas, the devices that support the business process and the time frames for evaluation – based on a business priority sequence. The safest bet is too assume that all components are guilty until proven innocent.

The plan and inventory are used to drive the analysis process. The first step is to contact the equipment manufactures to determine which devices are believed to be compliant, those suspect and those definitely not compliant. The process of determining year 2000 compliance may be very complex and may require a review of micro-code along with interactions with other “embedded systems” based on the use of the specific component. Once a manufacturer has determined the level of compliance, the plan should be updated with this information and strategies should be developed to test the “compliant” devices and deal with the non-compliant and suspect devices. Strategies to consider in this process include: updating the components with new versions from the vendor, replacing the entire-device with a compliant device, or modifying the “date” on a device that allows a user to change specific information. An example of this would be to use an encapsulation technique which changes the date by subtracting 28 from the year (this allows days of the week to be consistent as the calendar repeats every 28 years). This is a risky technique but may allow the device to function correctly assuming that it does not interface with any other device that also examines the date or time.

In many cases it may not be possible to obtain a vendor response as they are out of business or the product in question is no longer supported. In these cases an alternate manufacturer or a completely new automated process may be required. The plan should be updated to reflect the state of compliance of each component inventoried, action items identified and the projected time frames to achieve compliance. A contingency plan should be developed for each business function shown on the inventory, in the event the effort can not be completed prior to the century roll over.

In all cases a comprehensive test plan should be developed and executed to validate year 2000 compliance. The type of test and how the test is conducted will be specific to each component and should be developed by someone knowledgeable in the device. The testing process may involve taking the equipment “off-line” for a period of time in order to check the processing under a future date. In cases where you can not simulate a future date due to the nature of the component, contingency planning should be made for each non-testable device. This may include manual or replacement procedures and/or a complete process shutdown from December 31, 1999 through January 2, 2000 for devices that could cause business or physical harm if functioning incorrectly.

The final phase is deployment where the new systems are brought operational and training and support is provided for the new processes and procedures.

So what would be the consequences if one link in your company’s supply chain breaks due to Embedded Systems failure? Let’s assume that a mid-western local power utility’s grid control processor falls on January 1,2000. Since it is an “unknown” problem, diagnostic and repair time could be days, instead of hours. This could “trip” multiple grids in a domino effect that could cause an extensive power outage. Companies not on motor generating systems would be forced to close. Even those that remain open would be faced with heating requirements most likely over the capacity of the motor generators and telephone systems that may not operate. Since your company buys all of it’s copper wire for it’s electronic toy manufacturing operation from a single company located in this area, and your company uses a “just-in-time” inventory process, the electronic toy operation could be in jeopardy. Could your company survive this type of scenario? What are your contingency plans?

Kyle McCormick is a vice president with Computer Generated Solutions, Inc., a systems integration and consulting company. He can be reached at 213/625-2655.