Picture Quality Analysis: Real-Time Measurements for Objective Video Quality

The first step in understanding picture quality analysis is to look at the types of defects that occur in digital video systems. The most recognizable defects for a user and, thus, the most important to test for are defects like macroblocking, pixelation, blurring, stuck or dropped frames, audio clipping, and LipSync.

Macroblocking – Digital content is usually compressed in block sizes in multiples of 8x8 using codecs like MPEG 4 or H.264. If the decompression algorithm does not have a high correlation to the compression algorithm, then areas of defined edges can occur. This is also known as macroblocking.

.

Figure 1. Macroblocking causes defined edges between sets of decompressed pixels due to poor decompression and retrieval of the original content.

Pixelation – If data is lost during transmission, the decoder cannot properly read the block of data to retrieve the original pixel data. This causes distinguishable errors of off-color blocks. While these errors are much less common than macroblocking, they are still noticeable to users.

Figure 2. Distinguishable pixelation errors are caused by data loss and incorrect decoding.

Blurring – During decoding or scaling, high-frequency components can be lost, causing a loss in definition between edges, or blurring. This artifact causes the video signal to lose clarity or sharpness.

Figure 3. Blurring causes less distinguishable edges in a video signal. The image on the left has less definition between the black and white edges.

While macroblocking, pixilation, and blurring are represented in a single image as shown in figures 1 through 3, these defects often don’t appear in every frame and must be measured over time as the device is stressed. Other defects, such as stuck or dropped video frames, audio clipping, and LipSync (audio/video synchronization), also require multiple frames to analyze.

Picture quality analysis is the measure of overall quality of a video stream. It starts with having a set of trained technicians watch a video stream and subjectively score the content according to their experience to create Differential Mean Opinion Scores (DMOS). There are many challenges with this approach, such as the cost of training technicians and the recurring cost of operators watching a display. Another key challenge is the quality of the subjective test. In manufacturing, it is not practical to hire video quality experts to watch every device that is produced, so less-trained operators are chosen to watch and score the video. Conditions such as fatigue, lack of concentration, or eye irritation can easily cause false failures or poor products to be passed, forcing engineers to rethink the way they test for picture quality.

A better approach to picture quality analysis is to apply repeatable, objective measurements to measure audio and video quality. Many algorithms have varying degrees of correlation to the subjective scores of trained operators. Two common measurements are the peak signal-to-noise ratio (PSNR), which is based on the mean-squared error (MSE) and the Structural Similarity (SSIM) Index developed by Dr. Al Bovik and the team at the University of Texas Laboratory for Image and Video Engineering (LIVE). This index has proven to have one of the highest correlations to perceived video quality. By applying algorithms such as PSNR and SSIM, you can easily score video quality using automated, repeatable measurements.

Figure 4. This comparison of image fidelity measures for the “Einstein” image was altered with different types of distortions: a) reference image, b) mean contrast stretch, c) luminance shift, d) Gaussian noise contamination, e) impulsive noise contamination, f) JPEG compression, g) blurring, h) spatial scaling (zooming out), i) spatial shift (to the right), j) spatial shift (to the left), k) rotation (counter-clockwise), and l) rotation (clockwise).

Download “Image Quality Assessment: From Error Visibility to Structural Similarity” by Dr. Zhou Wang and Dr. Alan C. Bovik, which won the most prestigious paper award presented by the IEEE Signal Processing Society.

Another approach to testing video quality is to target specific artifacts in the audio and video content. You can also apply defect-specific algorithms that focus on one specific error within the video, such as blockiness or stuck/dropped frames. Many production applications do not demand high-performance tests such as PSNR or SSIM. Instead, they generally guarantee there is no macroblocking, audio clipping, or stuck video. Using defect-specific measurements, you can quickly qualify a device as passing or failing.

The NI Digital Video Analyzer with NI Picture Quality Analysis software provides defect-specific measurements and total picture quality measurements such as PSNR and SSIM. You can also apply your own custom algorithms with the user-defined measurement to create a completely customizable picture quality analysis application.

Figure 5. A 1080p60 video stream is acquired and analyzed for macroblocking levels in the horizontal and vertical space.

There are three ways to apply picture quality analysis measurements to an audio/video stream.

1. Nonreference test: The benefit to this approach is that you can apply the measurements to any stream, regardless of the content. The challenge with this approach is that the measurement results can vary greatly depending on the content being analyzed.

2. Reduced-reference test: Reduced-reference tests require a "golden," or ideal, device. Measurements are made on the audio/video quality from the golden device, and then all devices being tested are compared against those results to define a pass/fail for the system. 

3. Full-reference test: Full-reference tests also require a "golden" device. Instead of measuring the test stream and reference stream independently, as in a reduced-reference test, the two streams are analyzed together on a pixel-to-pixel comparison. Full-reference tests are the most powerful measurements that can be applied, but they add the challenge of complex processing and storage/retrieval of a reference stream for analysis. PSNR and SSIM measurements use the full-reference approach. 

Figure 6. You can perform picture quality analysis in three ways: 1) a nonreferenced test for which the video stream is unknown or unrepeatable (left), 2) a reduced-reference test during which measurements made on a reference stream are compared to measurements made on a test stream (middle), and 3) a pixel-to-pixel comparison between test and reference streams, known as a full-reference test (right).

Triggering

When using reduced-reference or full-reference analysis, you need to synchronize the test stream with the reference stream to get an accurate comparison. You can choose from several ways to do this, but there are a few approaches to synchronization that are most ideal:

Optical character recognition (OCR) can be applied if there is text that can be identified in a region of interest (ROI). The best example of using OCR is if there is a frame-counter on the video stream. It is easy to set a trigger on any specific frame with the OCR.

Intensity threshold is an ideal way to trigger when there is a sharp change in the brightness of a stream, such as a camera angle change or other luminance spike. The key to this approach is to identify an ROI for the trigger that will not trigger at any other point in the sequence.

Pattern match is another approach that is commonly considered for triggering acquisition between reference and test streams. The challenge with pattern matching is that often frame N and frame N+1 are so similar that a pattern match trigger triggers on incorrect frames. Often the best way to apply a pattern match trigger is to insert a known pattern into a video stream (such as a color bar) for a single frame. You can then use this frame as the pattern match.

Figure 7. The three most ideal approaches to synchronization are pattern matching, intensity threshold, and optical character recognition.

NI Picture Quality Analysis software has these triggering options and more to fit any application and any video stream you need to analyze. There is also a user-defined trigger that can help you write your own triggering algorithm for the analyzer.

The NI Digital Video Analyzer and NI Picture Quality Analysis software provide the hardware and software required for HDMI streaming audio/video test applications. Based on PXI Express technology, the hardware acquires 1080p60, HDCP-encrypted content and applies a library of proven measurements to objectively measure picture quality.

The NI Digital Video Analyzer with NI Picture Quality Analysis software offers powerful algorithms for referenced video testing during multimedia device design and validation. For higher throughput applications, NI Picture Quality Analysis software offers real-time high-performance measurements to remove operator error and subjective tests. You can use the configurable interface to choose referenced or nonreferenced testing, select prebuilt measurements, set limits, and automate applications.

White Paper: Test Engineer's Guide to HDMI 1.4