Ryan Parkinson - Siemens
Jacob Cassinat - Siemens
Governing subsystem interactions is a fundamental challenge for system integrators. Despite defining exhaustive I/O limits, sometimes failures occur and it is not clear which subsystem interaction generated the destructive element. It is difficult to request subsystem vendor's resources to troubleshoot a problem that did not clearly originate from their equipment, and testing each system in isolation may not account for all interactions. In these cases, the system integrator may be best positioned to monitor the relevant parts of the overall system, isolate the problem source, and assign the appropriate resources to resolve the issue. The Siemens Rail Systems Division recently successfully performed this system integrator's task.
Over the past three years, one of our customers faced a recurring issue with our SD160 light-rail transit vehicles. Denver RTD, a bus and light-rail service operating in Denver, Colorado, has 170 Siemens vehicles in operation. These vehicles receive power from an overhead catenary system (OCS), which in turn receives power from RTD's power distribution network. The auxiliary power supplies (APS) on board each vehicle receive power from the OCS and condition it for use by most of the other onboard vehicle subsystems. This APS had a high failure rate, which caused a critical failure for the vehicle. The failure log reported a high-voltage transient on the power input to the APS, which led the vendor to believe that either Denver RTD or the onboard propulsion subsystem (which provides power to the APS during electro-dynamic braking) were providing power outside the acceptable transient limit. However, both RTD and the propulsion unit supplier confirmed that their systems should not generate such a transient. Each light-rail vehicle failure was extremely expensive and time-consuming for Siemens and our supplier, and the failures caused operating delays for our customer. We needed to monitor the situation, establish the root cause, and find a solution as quickly as possible.
Preliminary Diagnostic Efforts
Initially, engineers at RTD verified OCS voltage levels met specifications. Subsequently, engineers from the APS vendor confirmed voltage transients that could contribute to the equipment failures, although when inducing these transients through various test routines, the APS always performed as designed. This testing required removing the vehicle from passenger service so personnel could monitor portable scopes. This method was inconclusive because high-voltage transients don’t occur very frequently and it is unlikely that a rare, damaging transient would occur during a short test period. It became clear that more comprehensive testing on vehicles in transit was needed to accurately characterize actual operating conditions.
The APS vendor built its own remote data logger to permanently install on an SD160 vehicle. It could obtain snapshots of system-level voltage data, but the data was insufficient to understand the surrounding environment and what was causing the transients. These approaches helped us realize that we needed to see the complete picture to diagnose the issue. We decided to build on these earlier efforts and design a rugged, remote system to monitor the trains for long periods of time to find and correct the problem.
We needed a highly flexible, yet powerful monitoring system to accommodate the variety of sensors and communication protocols from the different subsystems. We defined the following requirements:
- Continuous multichannel voltage sampling at >10,000 Hz to monitor six inputs for high-voltage transients
- At least three different configurable sampling rates to optimize each signal class data rate and minimize storage requirements
- A serial input using standard protocols to interface with the GPS antenna and provide location information for events
- Real-time calculations to provide output responses to interact with the sensors
- Pre- and posttrigger (event) data recording without saving nontrigger data to optimize analysis and minimize storage needs
- Large storage capacity
- Video management
- Automatically synchronize all inputs regardless of data rate or format
- Automatic downloads for extended operation with minimal personnel interactions
- Vibration and temperature operating ranges acceptable for installation on a rail vehicle
- Small footprint for installation in an electrical compartment
We decided to instrument two light-rail vehicles with CompactRIO modules, which we were surprised to learn fulfilled all of our system requirements. We used two different vehicles to compare the collected data and monitor how the vehicles interacted. We installed high-voltage transducers and connected them to several train components, focusing on areas that before and after the power input filters. This helped us determine if the transients were being generated from RTD's power network (prefilter), or if a subsystem of the train (postfilter) was generating the transient.
Programming With LabVIEW
We programmed our system exclusively with NI LabVIEW system design software, using the LabVIEW Real-Time and LabVIEW FPGA modules. We programmed the FPGA to acquire high voltages, currents, and vehicle diagnostics. We programmed the processor to acquire GPS locations and vehicle speeds, to perform daily housekeeping, and to perform postprocessing which allowed us to erase nontrigger data and minimize storage requirements since we were recording about 1 GB of data every 30 minutes. With automated postprocessing, we stored only about 5 GB per day. NI has a great database of prewritten code. Plugging in GPS software modules and general templates for the FPGA and processor software layout saved us a significant amount of time. After attending LabVIEW Core 1 and Core 2 classes in San Diego, we progressed from first-time users to advanced programmers in only a few months. Due to the intuitive nature of LabVIEW and previous programming experience, we completed and tested the software in less than six months.
Benefits of CompactRIO
FPGA and Processor
Perhaps the greatest benefit of CompactRIO is the FPGA/processor combination. Because the FPGA is reconfigurable, the achievable data rates and sampling accuracy are comparable to most state-of-the-art scopes. We can perform real-time calculations and outputs with no processor delays. Once the data is timestamped and buffered, the processor advantages come into play. Software engineers can take advantage of the full breadth of the processor’s flexibility to achieve extended and remote FPGA operation and manage large data sets. The buffered data can be retrieved and written to a USB drive, making its storage capabilities comparable to a laptop. The GPS signal is monitored and recorded. Scripts are run to postprocess, erase nontrigger data, and prepare the data for analysis. Daily tasks are performed and automated FTP uploads to a server can be executed each evening.
Rugged and Reconfigurable
The CompactRIO exceeded all our environmental requirements. It handles a temperature range of -40 °C to 80 °C, so we mounted the unit externally in an electrical compartment. Its small footprint and excellent vibration/shock resistance allowed for easy, semipermanent installation.
CompactRIO is highly customizable. We knew we needed to conduct multiple phases of investigation, and the ability to reconfigure the system to hone in on potential problem areas was a significant benefit. After performing preliminary voltage analysis, we discovered that it would be beneficial to monitor two current signals. Adding these two signals was a very simple task. Using a CompactRIO with swappable input modules, we could monitor almost any conceivable input.
CompactRIO offers multiple data access options. We could use a router to access the data over a network via wireless downloads or manually access the data via a removable USB drive. To make project development manageable, we did not implement wireless downloads at first. We recorded data to the flash drive and personnel accessed it once a week by shutting off the program, removing the USB drive, and swapping it with a new one. We are currently working with RTD to implement daily, automated FTP transfers to a server, which will save time and make the data more quickly accessible.
To effectively measure and record the transients, we needed to implement a sample rate >10,000 Hz. Vehicle diagnostic data is only available every millisecond, so a 1,000 Hz rate is sufficient. A third sampling rate of 50 Hz can be added if temperature readings are necessary. The FPGA easily accommodates these different rates and writes the data to a buffer. A fourth rate of 1 Hz was needed for the GPS input, and was handled in the processor.
Another benefit of CompactRIO is the video processing. We originally planned to handle and record the video in the CompactRIO FPGA. However, as the project progressed, we realized the CompactRIO could not handle the buffering we required at the specified resolution. We bought a digital video recorder (DVR), but then ran into difficulty synchronizing the video in the DVR with the CompactRIO data. They had different internal clocks, and due to natural clock drift, we could not rely on them for our required synchronization level. The solution here was simple. We bought an NI 9401 bidirectional digital input module to plug into the CompactRIO chassis. As the system records the voltages, the FPGA runs a simple algorithm to determine if a voltage transient is occurring. If it finds one, it sends out a binary signal to the DVR via the NI 9401. This occurs in real time with no processor delays. The DVR records the binary signal as a trigger with the video file, so we only have to download the video files that show active triggers, which reduces video storage and download times. This also helps us perfectly synchronize the video data with the CompactRIO voltage and GPS data by aligning the triggers.
Analyzing Data With DIAdem
Recording data with various rates and formats is only half the challenge; making sense of the data and effectively analyzing it is the other half. NI DIAdem data analysis software provided the ideal platform for the analysis task.
DIAdem supports the Technical Data Management Streaming (TDMS) file structure, which provides binary storage advantages. DIAdem uses automatically stored metadata to open, navigate, zoom, and perform computations on extremely large files very quickly. Figure 3 shows high-voltage channels in the top chart (three transient events occurred); elevation in the middle chart; and GPS, vehicle speed, and diagnostic data in the bottom charts.
DIAdem also supports scripting. Because we ran our monitoring system for more than three months, we generated hundreds of gigabytes of data and it was not feasible to open each file and manually analyze it. After determining the critical data needed to establish the root cause of the transients, we wrote a script that opened each file, looked for critical events, and summarized the findings (returned maximum transient events, durations, APS responses, trended location data, and so on).
At the conclusion of our study, we determined that transients exist in our customer's power distribution network and identified their causes, but we established that these transients lie within the required limits for magnitude and duration. The damaging transients were found to be generated in the APS. With this information, our vendor immediately analyzed its system further, believes it has identified the root cause and is correcting the issue.
Currently, we are working on a new project in Portland, Oregon where we are monitoring brake disc temperatures and hydraulic fluid line pressures to study disc wear. We can use the same program layout we wrote for the RTD voltage transient project with only simple changes to the sample rate, the input cards, the input scaling, and the trigger to record data. In addition, we are implementing some lessons learned from our Denver testing. After our initial investment for the Denver project, our cost and timeline to reconfigure and redeploy in Portland are significantly lower.