Broadening Embedded Software Test Coverage With Test Automation and Automated Test-Case Generation

Publish Date: Dec 21, 2015 | 0 Ratings | 0.00 out of 5 | Print

Overview

As embedded software adoption grows, the demands on engineers increase. Manual operation of test parameters limits the amount of testing you can accomplish and still achieve tight time-to-market requirements. You can choose from several possible strategies to improve test coverage and reduce overall test time. Test automation can help you reduce the time you spend manually adjusting test system parameters and greatly speed up regression testing because you can easily re-create tests. With automated test-case generation, you can optimize the test creation process by using high-level abstraction tools to intelligently create test scripts based on expected usage patterns and potential software vulnerabilities.

Table of Contents

  1. Test Automation
  2. Automated Test-Case Generation
  3. Summary
  4. Additional Resources

1. Test Automation

One step you can take to optimize your test system is to automate tests by scripting system parameter values according to time. With test scripting, technicians can run tests faster than manual testing and run signal-level tests unsupervised. You also can place these test scripts in libraries so that they can be reused in multiple test systems and for regression testing, so the return on the investment of creating a test script increases as it is reused.

You can choose from a variety of scripting tools, which can be used to modify real-time system parameters and run tests. NI VeriStand software includes a sophisticated tool for creating and running real-time test scripts. This tool, the Stimulus Profile Editor, can help you create scripts including looping, branching, and multitasking. You also can use it to perform data logging, pass/fail analysis, and report generation. To learn more about the Stimulus Profile Editor in NI VeriStand, view Creating Real-Time Stimulus Profiles in NI VeriStand.

The adoption of certification standards such as DO-178 and ISO 26262 has placed a new emphasis on requirements traceability, data analysis, and standardized report generation. Requirements documents must include specific entries for safety-critical component testing, and you must test and document the performance of electronic control modules (ECMs) to be placed in safety-critical systems. For these types of tests, a framework that links tests to requirements and automates test result analysis and customized report generation greatly improves the development process and makes safety-critical ECM validation much more efficient. NI TestStand software provides enterprise-level connectivity to a variety of software environments. You can use it to deploy NI VeriStand real-time stimulus profiles; link to requirements documents in IBM DOORS, MS Word, or MS Excel using NI Requirements Gateway; and perform data analysis automatically to generate custom reports with NI DIAdem software. To learn more about using NI VeriStand with NI TestStand, view Automating Real-Time Tests Using NI VeriStand and NI TestStand.

Figure 1. An NI TestStand sequence that deploys an NI VeriStand project and conducts a series of tests.

 

Real-time test automation improves testing efficiency and makes regression testing much easier. Software integration frameworks such as NI TestStand make it possible to create system architectures including requirements traceability, real-time stimulus deployment and data logging, data analysis, and customized report generation. However, while test automation makes running real-time tests easier, you face the up-front cost of test-case generation. You must create test scripts based on your understanding of the potential uses for the device under test. To improve test coverage and efficiency when generating and running tests, you can use an automated test generation tool.

Back to Top

2. Automated Test-Case Generation

For applications that require extensive test coverage, creating real-time test scripts can be a significant challenge. While test automation vastly improves test execution efficiency, you may face significant costs when developing a library of test scripts to adequately cover testing requirements. Automated test-case generation tools further improve efficiency and broaden test coverage by helping you intelligently script test scenarios based on operational requirements. You can choose from a variety of tools to automate real-time test generation.

MaTeLo

MaTeLo is a model-based testing solution that uses Markov chains to describe the test model of the system under test (SUT) implemented for embedded software validation.

With the MaTeLo Usage Model Editor, you can create test models that describe the SUT use cases, and the model designer can assign usage probabilities so that tests focus more on real-world use cases. The MaTeLo Testor scripts test cases based on the usage model, which ensures that test scripts adequately represent how the SUT is used when it is deployed. The MaTeLo Test Campaign Analysis feature provides information such as test coverage (requirements traceability, model) or SUT reliability.

Using MaTeLo, you can switch from test scripting to test engineering and significantly improve your test maturity level. You can build test models incrementally and use them from the beginning of the project to achieve all validation campaigns starting from the model up to the real system.

The MaTeLo systematic testing approach provides ordered and efficient test cases. The Test Campaign Analysis feature tracks and offers metrics on product quality, which helps during the release decision. MaTeLo significantly improves validation productivity and makes it possible to drastically reduce a product's recall rate.

Figure 2. A usage model in MaTeLo that is used to script a series of tests based on use cases, which ensures optimal test coverage.

 

MaTeLo integrates with NI TestStand. Learn more about MaTeLo.

TestWeaver

TestWeaver is a tool for automated validation of simulated systems. It generates, runs, and evaluates thousands of tests automatically to provide comprehensive test coverage without requiring extensive libraries of hand-coded test scripts. TestWeaver intelligently generates tests by “learning” about system behavior from the results of past simulations. The states reached during simulations are continuously classified according to coverage goals and test acceptance criteria. This knowledge is used to reactively guide the generation of future test scenarios with the goals to (a) increase the coverage of system states and (b) increase test density by focusing on suboptimal scenarios. To learn about how Hydrive Engineering used TestWeaver to improve their testing process read the case study: Hydrive Expands Hardware-in-the-Loop Test Coverage With TestWeaver and NI VeriStand.

When a system approaches a predefined limit, TestWeaver actively attempts to cause system instability. This method of test-case generation makes TestWeaver ideal for identifying hard-to-find bugs in embedded software, and it is often used after traditional test methods have failed to discover a software flaw, as reported, for instance, by Mercedes-Benz, ZF, and others (cf. ref). TestWeaver also supports classical test automation methods including macrobased recording and replay as well as user-defined test scripts on model-in-the-loop (MIL), software-in-the-loop (SIL), and hardware-in-the-loop (HIL) simulation platforms.

Figure 3. TestWeaver automatically generates test cases based on potential software weaknesses and adapts test cases based on test results.

 

TestWeaver integrates natively with NI VeriStand. Learn more about TestWeaver.

Back to Top

3. Summary

Test automation improves the test process by helping you create test scripts. You can use test scripts to automate system parameter changes, map tests to requirements documents, and create reports that include pass/fail analysis. These test scripts can then be reused to replicate a testing scenario or perform regression testing. Automated test-case generation tools provide additional functionality that generates test scripts based on usage models and potential flaws in embedded software. You can use them to more efficiently create tests and to ensure that each test covers the most important use cases for the device under test.

Back to Top

4. Additional Resources

Learn more about NI VeriStand.

Learn more about NI TestStand.

Automating real-time tests using NI VeriStand and NI TestStand.

Back to Top

Bookmark & Share


Ratings

Rate this document

Answered Your Question?
Yes No

Submit