Home>Characterization Test Center>Test How-To

DTV needs automated testing

John Maguire- September 27, 2012

Film historian Paul Burns has documented the evolution of cinematography over 2500 years, from antiquity through to 1900. It makes fascinating reading to see how the world gradually moved from drawings through static image cameras and onto silent movies. Toward the end of that period, the world was already starting to envisage what might happen once pictures could be made to move and be combined with audio.

A famous cartoon in Punch magazine in 1879 (Figure 1) extrapolates from the advances demonstrated by Edison on telephones ran as follows:


Figure 1. An 1879 newspaper cartoon predicted what would happen once sound and video were readily available in every home.

With the accompanying text reading:

EDISON'S TELEPHONOSCOPE (TRANSMITS LIGHT AS WELL AS SOUND).

    (Every evening, before going to bed, Pater- and Materfamilias set up an electric camera-obscura over their bedroom mantel-piece, and gladden their eyes with the sight of their Children at the Antipodes, and converse gaily with them through the wire.)

    Paterfamilias (in Wilton Place). "Beatrice, come closer. I want to whisper."  

    Beatrice (from Ceylon). "Yes, Papa dear."

    Paterfamilias. "Who is that charming young Lady playing on Charlie's side?"

    Beatrice. "She's just come over from England, Papa. I'll introduce you to her as soon as the Game's over!"


Since Edison’s time, rapid innovation has led to the point of high-definition color DTV (digital television) in nearly every every house in the developed world. Technology progression shows no signs of slowing down, leading to the emergence over the last 10 years of rapidly increasing rates of multi-screen and second-screen viewing experiences.

For commercial broadcasters, this last step in the technological evolution of the DTV space represents a new direction from what has come before. This step presents significant challenges and opportunities regarding test strategies. For the first hundred years or so of commercial television delivery, the focus was on delivering the best quality content from the operator’s head-end infrastructure through generally a single type of network infrastructure, through to a largely homogenous mix of receiver devices (typically discrete Set Top Boxes connected to the primary TVs). Over the last few years, we have seen the start of a significant shift away from this platform topology. On the content side, we see customers demanding access to a richer variety of content both directly from the operator and “over-the-top” (if sourced from the Internet and accessed by the consumer independent of the operator) or “through-the-middle” (if sourced from the Internet but integrated as part of the operator offering).

At the transport network, we see people consuming content via operator-provided applications over a wider array of network types. No longer can broadcaster and operators rely on a single broadcast point-to-multipoint network over QAM (quadrature amplitude modulation), QPSK (quadrature phase-shift keying) or equivalent protocols. Content is now delivered over Wi-Fi and 3G networks outside the home in addition to wired and wireless IP inside the home. Furthermore, consumers want to view content across an ever-expanding diversity of devices that now include laptops and tablet PCs, connected TVs, gaming platforms, and mobile phones in addition to the traditional Set-top box.

All three of these changes result in much more complex content delivery platform topologies that need to be tested. We can make a rough estimation of how much this testing challenge may have increased in just the last decade for a sample cable operator who, in the year 2000, was offering VOD (voice-over data) in addition to linear TV but is now rolling out the full gamut of multi-screen experience (Table 1).


Table 1. Network video delivery platforms and tests, pre and post 2000.

This represents a 60x increase from 2 to 125. It can be argued that the Test Combinations count calculated in the above table underestimates the number of potential combinations because it doesn’t include the interdevice testing. Many operators are seeking to deploy receiver environments that provide shared access to content and shared control over the consumption of that content. One regularly quoted example is the use-case where a viewer starts to watch a movie in the living room on the primary TV and wants to pause it and resume viewing from the same point on a second TV in the bedroom. There are many more use cases where the potential arises for simultaneous or sequential operations on different media consumption devices to cause unexpected or erroneous operation. All of this serves to further increase the testing complexity of modern digital TV platforms.

Replicating all of the possible combinations of {content sources, transport networks, and consumption devices that will exist in a given population of customers is becoming more difficult. Furthermore, the vast majority of testing performed by operators around the world today is still carried out manually. Teams of testers manually run through typically a few thousand tests to achieve a reasonable coverage test set against a single release of a modern digital set-top box. This manual testing effort simply can’t be cost-effectively scaled to achieve the levels of desired test coverage across all of the possible test combinations in a modern multi-screen environment. This is why we are seeing an increased interest in test automation from operators and vendors to this space from around the world. Judicious use of test automation techniques can remove much of the requirement for repetitive and error-prone manual testing, while not completely rendering it obsolete by any means.

Another changing aspect of the requirements for testing digital TV services is around the definition of just what viewers mean when they say that they want high quality of service. The generation of people growing up now has always been exposed to a variety of content from a number of different sources each with its own unique value, cost and expected quality levels. A number of factors contribute to the perceived quality level including overall resolution and pixel quality of the video image, the number of channels in the audio tracks, the length of time if any spent waiting for content to buffer and so forth.

In return for the convenience of having a vast array of high-definition movies available on demand, many people are willing to wait for a sufficient amount of content to have been buffered up by their Apple TV or equivalent device before it starts to play in spite of the fact that they have paid for the movie. In the very same period on their couch, these same viewers may then switch to YouTube content for a while where they are happy to look at very low-resolution user generated content but will show a very low tolerance for any delay in the content starting to play. And finally they may switch to the linear TV broadcast to tune into some vital end-of-season football game where they demand the highest resolution video, in real time, with absolutely no buffering or service outages.

Each of these different use cases and service level quality expectations are valid and ultimately customer-driven business requirements. This means they have to be tested against the appropriate quality metrics for each different service type to ensure that the overall end-to-end content delivery platform is delivering to the customer’s expectations. As well as testing each individual part of the platform, a complete end-to-end system test strategy has to be put in place (Figure 2).


Figure 2. Networks deliver content from sources to consumer devices such as set-top boxes, connected TVs, gateways, and mobile devices. End-to-end testing helps assure acceptable high quality of service.

As outlined in many other articles on this site such as “Engineers know when to automate,” the question of whether “to automate or not” needs to be carefully considered for every individual test environment.  Because of the complexity as outlined above, DTV testing is one environment where you can justify test automation. In fact, automation may even be critical to cope with the increasing system complexity. With an average of over four hours of TV viewing daily in the US and beyond, there are significant commercial drivers to underpin the investment required to put such an automated test infrastructure in place.