Test scenario creation

Writing tests to determine whether a piece of code behaves as expected is an important part of the software development process. With Switchyard, it is possible to create a set of tests that verify whether a program attempts to receive packets when it should and sends the right packet(s) out the right ports. This section describes how to construct such tests.

A test scenario is Switchyard’s term for a series of tests that verify a program’s behavior. A test scenario is simply a Python source code file that includes a particular variable name (symbol) called scenario, which must refer to an instance of the class TestScenario. A TestScenario object contains the basic configuration for an imaginary network device along with an ordered series of test expectations. These expectations may be one of three types:

  • that a particular packet should arrive on a particular interface/port,

  • that a particular packet should be emitted out one or more ports, and

  • that the user program should time out when calling recv_packet because no packets are available.

To start off, here is an example of an empty test scenario:

An empty test scenario.
from switchyard.lib.userlib import *

scenario = TestScenario("test example")

If we run swyard in test mode using this test description and any Switchyard program, here’s the output we should see:

Results for test scenario test example: 0 passed, 0 failed, 0 pending


All tests passed!

Notice that in the above example code, we assigned the instance of the TestScenario class to a variable named scenario. An assignment to this variable name is required. If it is not found, you’ll get an ImportError exception. Notice also that there’s one parameter to TestScenario: this value can be any meaningful description of the test scenario.

There are two methods on TestScenario that are used to configure the test environment:

  • add_interface(name, macaddr, *ipaddrs, **kwargs)

    This method adds an interface/port to an imaginary network device that is the subject of the test scenario. For example, if you are creating a test for an IP router and you want to verify that a packet received on one port is forwarded out another (different) port on the device, you will need to add at least two interfaces. Arguments to the add_interface method are used to specify the interface’s name (e.g., en0), its hardware Ethernet (MAC) address, and zero or more IP addresses, which must be in the form addr/prefixlen, e.g., fe80::13/64.

    Two optional keyword arguments can also be given: ifnum can be used to explicitly specify the number (integer) associated with this interface, and iftype can be used to explicitly indicate the type of the interface. A value from the enumeration InterfaceType must be used, e.g., Wired, Wireless, Loopback, or Unknown. The type of an interface defaults to InterfaceType.Unknown.

  • add_file(filename, text)

    It is sometimes necessary to make sure that certain text files are available during a test that a user program expects, e.g., a static forwarding table for an IP router. This method can be used to specify that a file with the name filename and with contents text should be written to the current directory when the test scenario is run.

There is one method that creates a new test expectation in the test scenario:

  • expect(expectation_object, description)

    This method adds a new expected event to the test scenario. The first parameter must be an object of type PacketInputEvent, PacketInputTimeoutEvent, or PacketOutputEvent (each described below). The order in which expectations are added to a test scenario is critical: be certain that they’re added in the right order for the test you want to accomplish!

    The description parameter is a short text description of what this test step is designed to accomplish. In swyard test output, this description is what is printed for each step in both the abbreviated and verbose output: make sure it is descriptive enough so that the purpose of the test can be easily understood. At the same time, try to keep the text short so that it isn’t overwhelming to a reader.

The three event classes set up the specific expectations for each test, as described next.

  • PacketInputEvent(portname, packet, display=None, copyfromlastout=None)

    Create an expectation that a particular packet will arrive and be received on a port named portname. The packet must be an instance of the Switchyard Packet class. The portname is just a string like eth0. This port/interface must have previously be configured in the test scenario using the method add_interface (see above).

    The display argument indicates whether a particular header in the packet should be emphasized on output when Switchyard shows test output to a user. By default, all headers are shown. If a test creator wants to ignore the Ethernet header but emphasize the IPv4 header, he/she could use the argument display=IPv4. That is, the argument is just the class name of the packet header to be emphasized.

    The copyfromlastout argument can be used to address the situation in which a test scenario author wants to construct an incoming packet (that will be received by recv_packet) which has the same values in some packet header fields as the most recent packet emitted. For example, when creating a protocol stack, an application (socket) program might emit a packet with a source port number assigned by the socket emulation module. The destination port number in an arriving packet needs to be the same as the packet that was previously emitted in order for it to be handed to the correct application program. Thus, the copyfromlastout can be used to copy one or more packet header attributes from the last emitted packet to header fields in an incoming packet.

    copyfromlastout can take a tuple of 5 elements: the interface/port name out which the packet was sent, a header class name and attribute to copy from, and a header class name and attribute to copy to. For example, if we wanted to copy the UDP source port value from the last packet emitted out port en1 to the UDP destination port of the packet to be received, we could use the following:

    PacketInputEvent('en1', pkt, copyfromlastout('en1', UDP, 'src', UDP, 'dst'))

    Note that we would need to have created a Packet object named pkt which included a UDP header for this example to work correctly.

  • PacketInputTimeoutEvent(timeout)

    Create an expectation that the Switchyard user program will call recv_packet but time out prior to receiving anything. The timeout value is the number of seconds to wait within the test framework before raising the NoPackets exception in the user code. In order for this test expectation to pass, the user code must correctly handle the exception and must not emit a packet.

    To force a NoPackets exception, the timeout value given to this event must be greater than the timeout value used in a call to recv_packet. Note also that the test framework will pause for the entire duration of the given timeout. If a user program calls net.recv_packet(timeout=1.0) but the timeout given for a PacketInputTimeoutEvent is 5 seconds, the call to recv_packet will appear to have blocked for 5 seconds, not 1.

  • PacketOutputEvent(*args, display=None, exact=True, predicates=[], wildcard=[])

    Create an expectation that the user program will emit packets out one or more ports/interfaces. The only required arguments are args, which must be an even number of arguments. For each pair of arguments given, the first is a port name (e.g., en0) and the second is a reference to a packet object. Normally, a test wishes to establish that the same packet has been emitted out multiple interfaces. To do that, you could simply write:

    p = Packet()
# fill in some packet headers ...
PacketOutputEvent('en0', pkt, 'en1', pkt, 'en2', pkt)

    The above code expects that the same packet (named pkt) will be emitted out three interfaces (en0, en1, and en2).

    By default, the PacketOutputEvent class looks for an exact match between the reference packet supplied to PacketOutputEvent and the packet that the user code actually emits. In some cases, this isn’t appropriate or even possible. For example, you may want to verify that packets are forwarded correctly using standard IP (longest prefix match) forwarding rules, but you may not know the payload contents of a packet because another test element may modify them. As another example, in IP forwarding you know that the TTL (time-to-live) should be decremented by one, but the specific value in an outgoing packet depends on the value on the incoming packet, which the test framework may not know in advance. To handle these situations, you can supply exact, wildcard(s), and/or predicate(s) keyword arguments, as detailed below.

    • Exact vs. subset matching: Setting exact to True or False determines whether all packet header attributes are compared (exact=True) or whether a limited subset are compared (exact=False).

      The set of header fields that are compared when exact=False is specified are: Ethernet source and destination addresses, Ethernet ethertype field, Vlan vlanid and ethertype field, ARP target and sender protocol and hardware addresses (four fields), IPv4/IPv6 source and destination addresses and protocol, and TCP/UDP src/dst port numbers (or ICMP/ICMPv6 icmptype/icmpcode fields). Note that in subset matching no packet payloads are compared.

    • Wildcard fields: In addition to specifying the exact keyword parameter, it is possible to specify that some additional header fields should be wildcarded. That is, the wildcarded header fields are allowed to contain any value. Wildcards are specified using a tuple of two elements: a header class name and a field name.

      A single wildcard can be supplied (i.e., one 2-tuple) with the wildcard keyword parameter, or a list of 2-tuples can be supplied with the wildcards keyword. For example, the following line of code uses subset matching (exact=False) and one wildcard. For this example, assume that the packet pkt contains Ethernet, IPv4, and UDP headers:

      PacketOutputEvent('en0', pkt, exact=False, wildcard=(IPv4, 'src'))

      Note that for the above example, the only fields compared in the IPv4 header would be the destination address and protocol field (since other fields are already ignored with exact=False).

      Here is another example that ignores source addresses in the Ethernet, IPv4 and UDP fields, leaving only two fields in the Ethernet header to be compared (dst and ethertype), two fields to be compared in the IPv4 header (dst and protocol) and one field in UDP (dst). Again, assume that the packet pkt contains Ethernet, IPv4, and UDP headers:

      PacketOutputEvent('en0', pkt, exact=False, wildcards=[(Ethernet, 'src'), (IPv4, 'src'), (UDP, 'src')])

    Note

    Switchyard previously allowed certain strings (modeled on the Openflow 1.0 specification) to be used to indicate wildcarded fields. These strings can no longer be used in the current version of Switchyard. To specify wildcarded fields, you must use the (hdrclass, attribute) syntax.

    • Predicate functions: Lastly, predicate functions can be supplied to make arbitrary tests against packets. The predicate keyword argument can take a single lambda function in the form of a string, and the predicates keyword argument can take a list of lambda functions, each as strings. Each lambda given must take a single argument (the packet object to be inspected) and must yield a boolean value. (Note that internally, each lambda definition is eval'ed by Switchyard.)

      Here is one example that checks whether the IPv4 ttl field is between 32 and 34, inclusive. Note that this line of code contains a single predicate function as a string:

      PacketOutputEvent('en1', pkt, exact=False, predicate='''lambda p: p.has_header(IPv4) and 32 <= p[IPv4].ttl <= 34''')

      To provide multiple predicates, just use the predicates (plural) keyword and provide a list of lambdas-as-strings.

Test scenario examples

First, here is an example of a test scenario in which a packet is constructed and is expected to be received on port eth1, then sent back out the same port, unmodified. Notice in the example that the name scenario is required.

A test scenario in which a packet is received then sent back out the same port.
from switchyard.lib.userlib import *

scenario = TestScenario("in/out test scenario example")

# only one interface on this imaginary device
scenario.add_interface('eth0', 'ab:cd:ef:ab:cd:ef', '1.2.3.4/16', 
    iftype=InterfaceType.Wired)

# construct a packet to be received
p = Ethernet(src="00:11:22:33:44:55", dst="66:55:44:33:22:11") + \
    IPv4(src="1.1.1.1", dst="2.2.2.2", protocol=IPProtocol.UDP) + \
    UDP(src=5555, dst=8888) + b'some payload'

# expect that the packet is received
scenario.expect(PacketInputEvent('eth0', p), 
    "A udp packet should arrive on eth0")

# and expect that the packet is sent right back out
scenario.expect(PacketOutputEvent('eth0', p, exact=True), 
    "The udp packet should be emitted back out eth0")

Here is an additional example with a bit more complexity. The context for this example might be that we are implementing an IPv4 router. First, notice that we include in the scenario a static forwarding table file (forwarding_table.txt) to be written out when the scenario is executed. We construct a packet destined to a particular IP address and create an expectation that it arrives on port eth0. We then construct an expectation that the packet should be forwarded out port eth2 (note that according to the forwarding table, any packets destined to 2.0.0.0/8 should be forwarded out that port). We also include a predicate function to test that the IPv4 ttl is decremented by 1. Note that if we did not include this predicate, any ttl value would be accepted since we have specified exact=False. Note also that if we had set exact=True we would almost certainly need to wildcard several fields, e.g., checksums in the IPv4 and UDP headers, and would still need to include a predicate to check that ttl has been properly decremented. Furthermore, if we were writing a test scenario for an IP router, we would also want to include expectations that the correct ARP messages were sent in order to obtain the hardware address corresponding to the next hop IP address.

A simplified IP forwarding test scenario.
from switchyard.lib.userlib import *

scenario = TestScenario("packet forwarding example")

# three interfaces on this device
scenario.add_interface('eth0', 'ab:cd:ef:ab:cd:ef', '1.2.3.4/16')
scenario.add_interface('eth1', '00:11:22:ab:cd:ef', '5.6.7.8/16')
scenario.add_interface('eth2', 'ab:cd:ef:00:11:22', '9.10.11.12/24')

# add a forwarding table file to be written out when the test 
# scenario is executed
scenario.add_file('forwarding_table.txt', '''
# network   subnet-mask   next-hop      port
2.0.0.0     255.0.0.0     9.10.11.13    eth2
3.0.0.0     255.255.0.0   5.6.100.200   eth1
''')


# construct a packet to be received
p = Ethernet(src="00:11:22:33:44:55", dst="66:55:44:33:22:11") + \
    IPv4(src="1.1.1.1", dst="2.2.2.2", protocol=IPProtocol.UDP, ttl=61) + \
    UDP(src=5555, dst=8888) + b'some payload'

# expect that the packet is received
scenario.expect(PacketInputEvent('eth0', p), 
    "A udp packet destined to 2.2.2.2 arrives on port eth0")

# and subsequently forwarded out the correct port; employ 
# subset (exact=False) matching, along with a check that the
# IPv4 TTL was decremented exactly by 1.
scenario.expect(PacketOutputEvent('eth2', p, exact=False, 
    predicate='''lambda pkt: pkt.has_header(IPv4) and pkt[IPv4].ttl == 60'''),
    "The udp packet destined to 2.2.2.2 should be forwarded out port eth2, with an appropriately decremented TTL.")

Compiling a test scenario

A test scenario can be run directly with swyard or it can be compiled into a form that can be distributed without giving away the code which was used to construct it. Compiled test scenario files are, by default, given a .srpy extension; uncompiled test scenarios should just be regular Python (.py) files.

To compile a test scenario, you can simply invoke swyard with the -c flag, as follows:

swyard -c code/testscenario2.py

The output from this command should be a new file named code/testscenario2.srpy containing the obfuscated test scenario. This file can be used as the argument to the -t option when later running a Switchyard program against those tests.

Note

Note that if a scenario is compiled using a different version of Python than the one used to run a test scenario (especially a different major version, e.g., 3.4 vs. 3.5), you may get some mysterious errors. The errors are due to the fact that serialized representations of Python objects may change from one version to the next; if there are any changes, then the version used to run the test cannot correctly deserialize the various objects stored in the test scenario.