Testing the Performance of Touch-Enabled Smartphone User Interfaces

Hans Kuosmanen, Optofidelity

"CompactRIO offers a real-time OS that we can program with LabVIEW, which is a big benefit to us. Also, we need the field-programmable gate array (FPGA) part of CompactRIO to include online filtering functions, the I2C/SPI communication protocol, and precise timing for latency measurements. "

- Hans Kuosmanen, Optofidelity

The Challenge:

Automatically measuring all the user interface (UI) actions for the touch screen enabled smartphones, which are constantly gaining more functions and features and causing more challenges for UI testing.

The Solution:

Using NI LabVIEW, the LabVIEW Real-Time Module, and CompactRIO platform to easily build an automated, non-intrusive UI test platform to use with many different applications and hardware.

Measuring the User Experience With Automated Systems

The smartphone industry is one of the hottest industries today. UI testing and measuring the user experience is a critical part of testing smartphones. At OptoFidelity, a Finland-based NI Partner, we specialize in optical measurement systems and have long experience in using LabVIEW software and NI hardware.


In this case, our challenge was to build a system that can operate touch screens the same way a user does. The system needed to provide high level of automation to make sure the actions and test methods were repeatable and reproducible. Throughout the R&D process, it is crucial to verify product functionality from the user perspective. Without an automated test system, mobile phone and touch screen engineers have to perform an excessive amount of extremely slow and inaccurate manual UI testing during the R&D cycle. Time to market is also a crucial factor for any company, especially those in the mobile phone industry, which makes test system automation the only choice.


At OptoFidelity, we are committed to creating automated test systems for measuring the user experience. We already have extensive experince with mobile phone UI testing and it is a natural step to include touch screen testing in our platforms.


R&D Process Acceleration With a Two-Finger Robot

Throughout the R&D process, we must verify product functionality from the user perspective. Our two-finger robot helps engineers dedicate more time to the actual engineering tasks than they would be able to if they also need to perform manual testing. By concentrating on providing visual and intuitive user interfaces to the test automation it is easy to increase testing volumes when needed, and R&D time can be spent more wisely on tasks such as analyzing results instead of configuring the test system.


The two-finger robot consists of a desktop three-axis robot with an artificial finger and a camera that provides a live video stream to a computer. On the controlling computer, we have a UI that we developed in LabVIEW. Camera provides the UI with a live image of the touch screen of the device-under-test. This live image combined with a touch-screen monitor on the computer allows the operator to build the test sequences very intuitively; similar to using the real product. Test sequences can then be repeated and test result reports can be created automatically. A CompactRIO system with two NI 9550 C Series modules controls the three-axis robot and the two-finger actuator. A NI 9234 dynamic signal acquisition (DSA) C Series module measures the accelerometer sensor for haptic feedback testing. Because of the modularity of the CompactRIO platform, adding e.g. current and audio measurements are very straight-forward via additional modules, if needed.


Quality of Experience

Quality of experience (QoE), also known as quality of user experience, is a subjective measure of a customer’s experiences with a vendor. User opinion on a product is mostly based on the QoE perceived while using the product. We confirmed a high QoE in the released product when UI delays and latency times were continuously analyzed and optimized with our test platform through the R&D phase. In addition to measuring UI response times, our platform also objectively quantifies user reaction time to information on the UI. With the LabVIEW programming environment and CompactRIO platform, we created an intuitive, flexible test system platform for assessing the QoE of a product. 


Smartphone UI Testing Is Getting More Complex

Mobile phones, especially smartphones, are getting more functions and features all the time, which is driving more challenges for UI testing. A phone is not just for making calls or sending text messages anymore. It’s now a computer-based multimedia device. With wireless networks, you can browse the Internet, watch video streams, take high-definition video or high-resolution pictures, play games like Angry Birds, have a video conference with your colleagues, or navigate with a GPS. Abundant options mean more testing.


When vendors are adding new features or updating the OS, they need to test those changes. With our test platform, it’s easy to track the effect of those changes. Vendors can accurately measure the execution speeds, haptic feedback vibration, sounds, and the UI behavior. Because the test steps are reproducible, a true comparison is possible.


We have focused on making our test methods non-intrusive, i.e. that they do not require any test or debug –specific features from the product. This means that our customers can use the same test methods for any version or maturity level of their product design. It also makes it possible to do benchmarking against other products in the market.


In between improving the time to market of our customers, we also designed a system, just for fun, that plays the popular smartphone game Angry Birds automatically. Because we have extensive experience in touch panel testing and performance testing for mobile devices using video and optical measuring systems, it was quite easy to implement an application for this particular need. The most difficult part of the project was to play through the levels of Angry Birds. Fortunately, plenty of volunteer engineering resources were readily available for this. Although we developed the system for fun, it shows the test platform’s capability. Playing Angry Birds 1,000 times and always getting the same results is not possible to do manually.


Developing a Flexible Test System With CompactRIO and LabVIEW

To build a successful test platform, we need to make it as modular as possible. With CompactRIO, we can easily add different measurement and control modules to our test system, which assures that we can fulfill our customer’s future measurement needs. For example, haptic feedback functionality and 3D screens will be more popular in the future, but we won’t need to rebuild our platform from the scratch. We can just add new test modules as needed. With LabVIEW, we can easily connect to these new sensors and modules because of the easy hardware integration offered by the software.


Because we need to have real-time measurements, it is necessary to use a real-time OS. CompactRIO offers a real-time OS that we can program with LabVIEW, which is a big benefit to us. Also, we need the field-programmable gate array (FPGA) part of CompactRIO to include online filtering functions, the I2C/SPI communication protocol, and precise timing for latency measurements. Again, the ability to use LabVIEW to program the FPGA saves a lot of our software resources and time, as we don’t have to spend time in learning new development tools. 


Author Information:

Hans Kuosmanen
Hermiankatu 8 D
Tampere 33720


An NI Partner is a business entity independent from NI and has no agency, partnership, or joint-venture relationship with NI.

Hans Kuosmanen from OptoFidelity introduces the UI tester concept.
The OptoFidelity two-finger robot plays the popular smartphone game Angry Birds. (link: http://www.optofidelity.com/videos)