How to Overcome Production Test Challenges for Complex DUTs
The products we buy and use every day are so much more complex than they were a few years ago. Trends in IoT, technology convergence, voice control and more are pushing device functionality higher and higher, and with that comes pressure on test teams. When was the last time you had a straightforward test coverage requirement that didn’t include new measurements and didn't change over time? How complex is your test-line setup, compared to just a few product-releases ago?
Five years ago, the room you are sitting in probably was lit by a simple incandescent bulb or a fluorescent tube. Testing a light bulb is easy—you power it up, measure the lumens, and you’re finished. Chances are, though, that now you're being lit by LEDs. (Look up at your ceiling and see!)
Figure 1: The complexity of even our light fixtures is increasing rapidly with wireless and audio functionality.
Testing an LED fixture is more complicated. You have a driver board, which needs ICT and functional test. And, of course, as soon as every fitting required a printed circuit board assembly, lighting manufacturers were quick to add “differentiating functionality,” because adding circuits to an existing green board is a good use of space and improves product margin. You can imagine the conversation in R&D—
"Why not add wireless access points? That seems useful.”
“Well, if we have wireless, let’s add Bluetooth, too!”
“And then, we can add a speaker, and....”
And the list goes on.
Long story short, the test engineering team that once had to power-up a glass tube is now faced with power electronics, wireless connectivity, acoustics, and more. I imagine this sounds familiar, as it’s a similar story told from consumer electronics to industrial machines and life science applications. This new DUT complexity creates two fundamental problems for test engineering teams:
- The breadth and depth of test coverage requirements goes up, often requiring new station architectures, instrumentation, and software. However, budget and time-to-market schedules seldom increase proportionally, forcing either change, conflict, or both.
- A wider domain proficiency is required to implement systems. Specialism doesn’t happen overnight, so a combination of personal development, support, and abstraction must be found in order to build stations effectively.
It falls on the leaders of test organization to have a plan to solve these problems, and the path forward is not always obvious. Let’s talk about a few strategies available, alongside best practice from leading test groups.
Test Coverage - A Story of COTS Hardware and Software
The challenge of additional complexity is not just technical—e.g., meeting ambitious specifications—it’s financial. Let’s look at an extreme example of this technical/budgetary balance. The Saab Gripen E supersonic jet was aggressively optimized for both performance and cost effectiveness. The result was the total project cost for this airplane was a fraction (>10% by some estimates) of some other similar projects. Saab was focused on breaking the direct relationship between increased functionality and increased cost of test – they called this the cost curve, and their methodology is one we can all learn from.
Figure 2: The Saab Gripen E Aircraft adopted an open COTS approach to test.
Having a COTS product means we can contain development and maintenance costs, promoting the Saab initiative to break the cost curve.
-Anders Tunströmer, SAAB Aeronautics
They found the solution by adopting not just a COTS platform, but an open COTS platform for both hardware and software. This allowed them to save development costs for the 90% use case and integrated a point solution for the 10% unique requirement. Saab saved weeks of documentation per test system and cut test costs by 30%. Read more about how SAAB’s approach to test here.
By adopting a standard that closely integrates instruments for most measurements needs while operating within an ecosystem that fulfills niche requirements, you can confidently accept any new test specification safe in the knowledge that it will have minimal impact on existing functions, footprint, and processes. This philosophy serves as the foundation for PXI, which is based on an open, modular architecture.
Figure 3: The PXI foundation maintains an open modular standard ensuring flexibility for users.
Don't fall for the common misconception that COTS benefits are limited to hardware. To avoid limitations from proprietary, vendor-defined systems, production engineers often turn to a customized solution. While this might translate to huge coverage, cycle-time, and cost advantages, it also means they must reinvent the wheel every time they code a measurement.
That wouldn't matter as much if these measurements were simple and quick to code, but they often involve complex algorithms and require deep signal and sensor understanding. This is where a modular COTS approach once again shines. In audio test alone, numerous specialists offer great open software products such as CATS (CIM.AS) or Audioexpert (MegaSig). The benefit is summed up will by the sentiment from the CIM team:
Our goal through our CATS software is to be the electro-acoustic testing expert on your team so you can concentrate in being the expert in your product.
-Dennis Morini. Business Manager, CIM.AS
One roadblock to COTS adoption is the shift of budget from operational expense to upfront capital. If you add the price of a new test station for every new measurement type, the steepening cost curve soon becomes unappetizing. Neil Evans, a longtime Philips Healthcare employee, explains it well, as he has witnessed the ultrasound products he works on exponentially rise in functionality. Every year, he’s challenged with increasing test capability without skyrocketing his test budget.
The ability to articulate the business value that a test organization could deliver was critical. In this case the exponential development and sustaining costs could be forecast in line with increased product complexity. A vision of breaking the relationship between product complexity and test system cost provided executive buy-in.
-Neil Evans, Senior Manager, Philips
Read how Philips Healthcare successfully adopted COTS within production test.
The cost of not changing often is more apparent once you consider operational expenses at a level spanning from organizational to sustaining decisions.
Failing to “break the cost curve” and hiding it behind a lot of small test station decisions does not make it go away. Best-practice suggests to elevate decisions to a wider strategy whenever possible; this can slow them due to increased stakeholders but has been shown to return long term benefit.
Figure 4: Failure to break the cost curve and allow the cost of test to raise in line with product complexity soon becomes uneconomical to the business.
Bottom line: If you’re facing an increasing mix of test coverage requirements, the established best practice is to adopt an open COTS platform wherever possible. Look for interoperability with a wide ecosystem from multiple vendors so your team can focus on testing the DUT, rather than fixing compatibility issues or debugging analysis algorithms. Make budgetary decisions based on a total cost of ownership to deliver maximum returns.
Proficiency Investment: Critical to Complex Test
Any test strategy technology is only as good as the team that implements it. As DUT functionality increases, so does the expectation that test engineering teams. When hiring additional resources isn’t an option, each team member must be prepared to do more. This is a well-documented product-design challenge.
People are trying to bring much more complex products to market with an insufficient talent pool. We cannot rely on people who only know a single field; we need engineers that are trained in system-thinking.
-Prof. Alberto Sangiovani-Vincentelli. Professor of Engineering UC Berkley & Co-Founder of Cadence & Synopsys
Professor Vincentelli, an expert on this subject, concluded the key to success is abstracting domain-specific details away from the engineer, so they can work holistically and contribute across the system. Translating this theory into production test, the takeaway is that engineers need the ability to add value across measurement domains, taking test ownership of the entire DUT to ensure errors don't occur in the gaps between siloed systems.
By prioritizing system proficiency above task proficiency, the effectiveness of each engineer is maximized. Best practice to facilitate this is to establish a single set of processes and tools that scale to meet product-line test coverage requirements. To efficiently drive process adoption and easily implement more complex test lines, stations, or architectures, you need buy-in from every team member. Some engineers will naturally see wider system proficiency as an opportunity for personal growth, while others will balk at short-term frustrations.
To succeed, everyone must understand how the change will be implemented and who will do it—they need to share your future vision and feel supported to make it happen. Strong technical leadership should identify and course-correct technical challenges on the ground, as well as actively mentor team members. This leadership can be hired into the team permanently or supplied part-time through consultants – but it is essential. Chris Cilino, veteran test software engineer, states:
I've worked with all kinds of companies, from semiconductor to consumer electronics. Most people get stage one correct: They standardize on a single tool, such as LabVIEW, and a single set of development guidelines. Where they fail is stage two: Implementing a process for every engineer to adhere to those guidelines and supporting initial success until it becomes a habit.
-Chris Cilino, Founder & Owner, PetranWay
There is never a good moment to take time out from projects to invest in team proficiency. Any manager will tell you the test engineers on their team are their most valuable assets. Great managers back this up by prioritizing their proficiency, outsourcing their “busy work,” and supporting them to become system- or even organizational-level thinkers.
To learn more about any of the technology, business strategies, and test solutions we’ve discussed, get in touch with us at NI today. Let’s dive into how increased functionality in your DUTS is causing test challenges, and how NI can help.
Next steps:
If you’re interested in creating a functional test solution, learn more about how NI can help.
Learn more about the Philips case study discussed in this article.
Find out more information on LabVIEW, TestStand, and other software tools specifically designed so that engineers can work effectively across complex multidisciplinary test challenges.