Testing right from the start
Engineers at Force10 Networks develop test plans and automation scripts while products are still in development.
By Martin Rowe, Senior Technical Editor -- Test & Measurement World, 8/1/2009 2:00:00 AM
|
See other articles from our August 2009 issue. |
SAN JOSE, CA—Internet service providers, enterprises, data centers, and research labs process billions of bits of data per second. To make sure all those bits, bytes, and packets reach their destinations, these organizations rely on combination Ethernet switches and backbone routers from Force10 Networks.
 Figure 1. All switches and routers from Force10 Networks, regardless of configuration, share a common code base. |
A high-end switch/router from Force10 can process more than 32 billion packets per second through 1-Gbit Ethernet and 10-Gbit Ethernet ports that connect thousands of network devices to network backbones. The company’s product line consists of chassis with removable line cards as well as fixed, stackable units (Figure 1). A high-end line card can have up to 90 1-GigE ports that connect to servers, and a chassis can have up to 140 10-GigE ports for connecting to a network backbone. Each system has a processor board with three processors, and each line card has its own processor, all of which run FTOS (Force10 operating system). FTOS is based on the NetBSD open-source kernel.
Reliability is paramount in network equipment, which is why Force10’s test engineers get involved early in a product’s development—so early, in fact, that they write software product definitions, test plans, and automated test scripts. Tushar Patel, VP of engineering test and quality assurance, heads a team of engineers who perform software testing on processor cards, line cards, integrated systems, and complete networks. They validate both new-feature software releases and maintenance releases.
Network backgrounds
Testing at Force10 Networks differs from that at other network-equipment companies in that the test engineers have networking backgrounds, which enables them to get involved early in product definition and development. “Our test engineers not only know protocols, but they have done network design, installation, and operation,” said Patel.
 Tushar Patel manages a department of about 50 test engineers who evaluate new software builds. Photo by Gary Laufman/Getty Images. |
Patel joined Force10 soon after the company was formed in 1999, and all of the test managers who work for him have been with the company for eight or nine years. “I joined Force10 because the founders recognized early that testing is important,” said Patel. “They didn’t wait until they had a product to begin building a test organization. I was able to build a testing team earlier than I could at most companies. I think this helps us maintain higher-quality code as we add features and functionality to FTOS.”
Force10 software engineers produce two major releases per year, with maintenance releases as needed. All of the company’s switch/routers use the same software code base. That streamlines software development and ensures that a bug fix will cross product lines. Each system has a set of software drivers for communicating between FTOS and the hardware. The common code base results in greater stability, manageable code maintenance, and simpler product training.
Test engineers are responsible for evaluating new software features, and they perform regression testing to assure customers that new code won’t adversely affect existing installations. Table 1 highlights the test-engineering departments, their managers, and some of their functions.
Figure 2 shows the path of a software build through the test groups. The main branch represents the software code base that is used across all of the company’s platforms. A private development branch refers to unverified code prior to its merging with the main code branch, whether the new code provides bug fixes or adds new features.
 Figure 2. Test engineers evaluate new code in a private branch before merging it with existing code in the main branch and running system tests. |
A private branch starts with software developers who perform unit test and work closely with the engineers who perform platform tests. Engineers in the platform-test group evaluate how new code interacts with hardware, often working on individual processor cards or line cards. They merge code into the existing main branch for system test and application test.
Engineers in the system-test group run the new code on an integrated switch/router after the private branch is merged with the main branch. If they approve the new code, it will remain part of the main branch for new products. “While system test and development test teams run through a release check list,” said Patel, “the applications test group performs application testing.”
Test engineers in all groups use test beds that consist of Force10 products, competitors’ products, and network traffic generators from Ixia, Spirent Communications, and Agilent Technologies. They use proprietary software tools to write scripts without needing to know the scripting language’s syntax. Through the scripts that the proprietary tools generate, engineers can control systems under test and the traffic generators because the scripting tool includes application programming interfaces to traffic generators. Scripts verify the data that passes through the test bed. Engineers use a command-line interface to run the scripts on a Linux- or Unix-based computer. Because test engineers use Force10’s scripting tool to write high-level scripts, they can test an entire switch/router in a test bed with just a few commands.
Platform test
Software developers write code and perform initial functional tests. From there, the platform-test group, managed by Subarrao Karavadi, tests how new FTOS code handles protocols and how it controls hardware functions such as power and cooling. This group tests the software that controls cooling fans and how the hardware switches to backup power supplies if needed.
 Subarrao Karavadi’s group of test engineers evaluate the interactions between new software and hardware. Photo by Gary Laufman/Getty Images. |
Platform-test engineers also test hot swapping of all field-replaceable units, including line cards, control modules, switch fabric modules, power supplies, and fan trays. “Hot-swap testing of these components is very critical because customers must be able to add or replace them without affecting a switch’s operation,” said Karavadi. “FTOS needs to know when each component is installed or removed. Traffic should pass to a new line card whenever it’s inserted.” Platform-test engineers also test control modules to ensure that a backup control module takes over without any traffic loss should a primary control module fail.
While engineers in Karavadi’s group don’t test for initial board start-up, they do test to make sure that FTOS boots a board’s processors. An FPGA (field-programmable gate array) can load the FTOS image from a network through FTP (file transfer protocol), a local Compact Flash card, or a USB “thumb” drive. His group also checks operation of hardware such as flash memory, optics, and processor chip sets but does not test for memory leaks or initial booting. Hardware engineers led by Jim Miller handle that. (See “Under the software.”)
Engineers in the platform-test group also measure packet throughput, latency, and jitter. They use several network configurations such as full mesh, many to one, one to many, and many to many for these measurements. Such tests often require hundreds of traffic-generator ports, which can get expensive.
 Figure 3. One traffic generator can test layer 2 switching on all 48 LAN ports. |
“We accomplish our tests by using relatively few traffic-generator ports in combination with a proprietary algorithm, “said Karavadi. “Engineers program hardware in 'snake’ configurations where data from one port passes to an adjacent port in the line card.” Figure 3 shows a configuration for a 48-port line card. Engineers can expand the test to cover every port in a chassis.
Karavadi’s group also performs basic layer 2 and layer 3 protocol testing. Once they’re satisfied with how a new software build works with the hardware, they pass it on to the development-test engineers. Development-test engineers, managed by Narmada Chenna, run functional tests on new code features. They also run stability tests to ensure that the code won’t get into undesired conditions. Interoperability tests let them check whether Force10’s products will communicate to network equipment from other companies. Development-test engineers also run regression tests, which can uncover bugs when new features interact with existing code.
System test
Software test manager Balu Ramappa leads the system-test group, whose engineers use the test beds to simulate complex networks found at Internet service providers and data centers. They perform a system test for about six weeks.
 Balu Ramappa manages the system-test group, which tests software in complete switch/router systems. Photo by Gary Laufman/Getty Images. |
“Our test beds are a superset of customer network configurations,” said Ramappa. “A typical data center has 20,000 servers that communicate with 200 routers. We can stress test our equipment by simulating 40,000 to 50,000 servers and 400 to 600 routers.”
Force10’s larger customers require that the company test software on test beds that emulate their networks. That job falls to Hitha Shetty, manager of customer applications and response. Shetty’s group uses test beds to simulate network traffic. “We go through customer topologies, configurations, and issues,” said Shetty. “We provide feedback to software developers and test engineers.”
When customers have software problems that first-line product-support engineers can’t recreate in the customer-support lab, Shetty’s engineers try to replicate the problem in the customer-application lab. Once the problem is identified, fixed, and verified, engineers apply the fix to all the release branches. If a customer situation demands a new code revision, engineers will issue a spot-patch release.
The difficulty in replicating a condition and identifying a problem depends on the symptom. For example, if a feature doesn’t work for a customer, engineers can usually isolate the problem and either recommend workarounds or request a code revision.
Other problems are more difficult to find. The most difficult involve networks where customers use equipment from two or more manufacturers. “We try to develop software with a focus on standards and interoperability” said Shetty. “But implementation differences exist when equipment manufacturers interpret the standard differently or deliver two different versions of the same standard.” When that happens, Force10 customers must contact both companies.
“An update to a communications standard can be a source of interoperability problems,” noted Shetty. “Not every customer upgrades their equipment with every new software release.”
Newer code may have a desired behavior change or bug fix, but customers, especially large customers, don’t like frequent upgrades because each upgrade requires that they test it prior to deployment. “Customers want to use a software release for as long as two years,” added Patel. “Small customers are more likely to upgrade than large customers.”
When to release
Test engineers at Force10 have considerable influence over software definitions and releases. Patel and his managers make the final call on when to release. Instead of applying only a strict definition of severity to bugs that must be resolved before a release, they again use their networking backgrounds to make a more nuanced call on which bugs actually are likely to impact the customer.
Test engineers set bug priorities and can delay a release if they are not satisfied with new code. Once satisfied, the system-test group performs a 72-hr stress test in the lab before signing off on the code. This test raises confidence that the code will perform as expected in customer networks. “If we find any critical bug,” said Ramappa, “we can return the code to the software developers. Once the bug is fixed and verified, this test starts again.
Test engineers at Force10 Networks manage software builds and releases, which gives them considerable control over the software quality. With their networking backgrounds, they bring a customer’s perspective to test. The company’s system of evaluating software has many processes, any of which can catch and report bugs before reaching customers’ networks.
| Table 1. Force10 test departments and their functions. | ||
| Department | Manager | Functions (partial listing) |
| Unit test | Development group | Unit testing, Software quality tool development, Code review |
| Platform test | Subarrao Karavadi | Hardware bring up, Diagnostic testing, Environment testing, Performance testing, Redundancy and high-availability testing, Interoperability, Regression |
| Development test | Narmada Chenna | Integration and merge, Routing and switching protocol functional test, Stability test, Interoperability, Regression |
| System test | Balu Ramappa | Robustness, Resiliency, Interoperability, Large-scale topologies |
| Application test | Hitha Shetty | Customer-specific application test, Proof-of-concept test |
-
I agree, very good.
Joe Swanson - 2009-4-9 10:34:00 EDT -
Woww!! Congrats on an awesome article!
- Aniket.
Aniket - 2009-30-8 16:41:00 EDT
No related content found.
- 0 rated items found.
Datasheets.com Electronic Parts & Inventory Search
185 million searchable parts
- Part Number
- Description
- Inventory
- Products
- Manufacturers
View All Webcasts
