Test scenario creation¶
Writing tests to determine whether a piece of code behaves as expected is an important part of the software development process. With Switchyard, it is possible to create a set of tests that verify whether a program attempts to receive packets when it should and sends the right packet(s) out the right ports. This section describes how to construct such tests.
A test scenario is Switchyard’s term for a series of tests that verify a program’s behavior. A test scenario is simply a Python source code file that includes a particular variable name (symbol) called
scenario, which must refer to an instance of the class
TestScenario object contains the basic configuration for an imaginary network device along with an ordered series of test expectations. These expectations may be one of three types:
that a particular packet should arrive on a particular interface/port,
that a particular packet should be emitted out one or more ports, and
that the user program should time out when calling
recv_packetbecause no packets are available.
To start off, here is an example of an empty test scenario:
from switchyard.lib.userlib import * scenario = TestScenario("test example")
If we run
swyard in test mode using this test description and any Switchyard program, here’s the output we should see:
Results for test scenario test example: 0 passed, 0 failed, 0 pending All tests passed!
Notice that in the above example code, we assigned the instance of the
TestScenario class to a variable named
scenario. An assignment to this variable name is required. If it is not found, you’ll get an
ImportError exception. Notice also that there’s one parameter to
TestScenario: this value can be any meaningful description of the test scenario.
There are two methods on
TestScenario that are used to configure the test environment:
add_interface(name, macaddr, *ipaddrs, **kwargs)
This method adds an interface/port to an imaginary network device that is the subject of the test scenario. For example, if you are creating a test for an IP router and you want to verify that a packet received on one port is forwarded out another (different) port on the device, you will need to add at least two interfaces. Arguments to the
add_interfacemethod are used to specify the interface’s name (e.g.,
en0), its hardware Ethernet (MAC) address, and zero or more IP addresses, which must be in the form
Two optional keyword arguments can also be given:
ifnumcan be used to explicitly specify the number (integer) associated with this interface, and
iftypecan be used to explicitly indicate the type of the interface. A value from the enumeration
InterfaceTypemust be used, e.g.,
Unknown. The type of an interface defaults to
It is sometimes necessary to make sure that certain text files are available during a test that a user program expects, e.g., a static forwarding table for an IP router. This method can be used to specify that a file with the name
filenameand with contents
textshould be written to the current directory when the test scenario is run.
There is one method that creates a new test expectation in the test scenario:
This method adds a new expected event to the test scenario. The first parameter must be an object of type
PacketOutputEvent(each described below). The order in which expectations are added to a test scenario is critical: be certain that they’re added in the right order for the test you want to accomplish!
The description parameter is a short text description of what this test step is designed to accomplish. In
swyardtest output, this description is what is printed for each step in both the abbreviated and verbose output: make sure it is descriptive enough so that the purpose of the test can be easily understood. At the same time, try to keep the text short so that it isn’t overwhelming to a reader.
The three event classes set up the specific expectations for each test, as described next.
PacketInputEvent(portname, packet, display=None, copyfromlastout=None)
Create an expectation that a particular packet will arrive and be received on a port named
portname. The packet must be an instance of the Switchyard
portnameis just a string like
eth0. This port/interface must have previously be configured in the test scenario using the method
displayargument indicates whether a particular header in the packet should be emphasized on output when Switchyard shows test output to a user. By default, all headers are shown. If a test creator wants to ignore the Ethernet header but emphasize the IPv4 header, he/she could use the argument
display=IPv4. That is, the argument is just the class name of the packet header to be emphasized.
copyfromlastoutargument can be used to address the situation in which a test scenario author wants to construct an incoming packet (that will be received by
recv_packet) which has the same values in some packet header fields as the most recent packet emitted. For example, when creating a protocol stack, an application (socket) program might emit a packet with a source port number assigned by the socket emulation module. The destination port number in an arriving packet needs to be the same as the packet that was previously emitted in order for it to be handed to the correct application program. Thus, the
copyfromlastoutcan be used to copy one or more packet header attributes from the last emitted packet to header fields in an incoming packet.
copyfromlastoutcan take a tuple of 5 elements: the interface/port name out which the packet was sent, a header class name and attribute to copy from, and a header class name and attribute to copy to. For example, if we wanted to copy the UDP source port value from the last packet emitted out port
en1to the UDP destination port of the packet to be received, we could use the following:PacketInputEvent('en1', pkt, copyfromlastout('en1', UDP, 'src', UDP, 'dst'))
Note that we would need to have created a
pktwhich included a UDP header for this example to work correctly.
Create an expectation that the Switchyard user program will call
recv_packetbut time out prior to receiving anything. The timeout value is the number of seconds to wait within the test framework before raising the
NoPacketsexception in the user code. In order for this test expectation to pass, the user code must correctly handle the exception and must not emit a packet.
To force a
NoPacketsexception, the timeout value given to this event must be greater than the timeout value used in a call to
recv_packet. Note also that the test framework will pause for the entire duration of the given timeout. If a user program calls
net.recv_packet(timeout=1.0)but the timeout given for a
PacketInputTimeoutEventis 5 seconds, the call to
recv_packetwill appear to have blocked for 5 seconds, not 1.
PacketOutputEvent(*args, display=None, exact=True, predicates=, wildcard=)
Create an expectation that the user program will emit packets out one or more ports/interfaces. The only required arguments are
args, which must be an even number of arguments. For each pair of arguments given, the first is a port name (e.g.,
en0) and the second is a reference to a packet object. Normally, a test wishes to establish that the same packet has been emitted out multiple interfaces. To do that, you could simply write:p = Packet() # fill in some packet headers ... PacketOutputEvent('en0', pkt, 'en1', pkt, 'en2', pkt)
The above code expects that the same packet (named
pkt) will be emitted out three interfaces (
By default, the PacketOutputEvent class looks for an exact match between the reference packet supplied to PacketOutputEvent and the packet that the user code actually emits. In some cases, this isn’t appropriate or even possible. For example, you may want to verify that packets are forwarded correctly using standard IP (longest prefix match) forwarding rules, but you may not know the payload contents of a packet because another test element may modify them. As another example, in IP forwarding you know that the TTL (time-to-live) should be decremented by one, but the specific value in an outgoing packet depends on the value on the incoming packet, which the test framework may not know in advance. To handle these situations, you can supply
predicate(s)keyword arguments, as detailed below.
Exact vs. subset matching: Setting
Falsedetermines whether all packet header attributes are compared (
exact=True) or whether a limited subset are compared (
The set of header fields that are compared when
exact=Falseis specified are: Ethernet source and destination addresses, Ethernet ethertype field, Vlan vlanid and ethertype field, ARP target and sender protocol and hardware addresses (four fields), IPv4/IPv6 source and destination addresses and protocol, and TCP/UDP src/dst port numbers (or ICMP/ICMPv6 icmptype/icmpcode fields). Note that in subset matching no packet payloads are compared.
Wildcard fields: In addition to specifying the
exactkeyword parameter, it is possible to specify that some additional header fields should be wildcarded. That is, the wildcarded header fields are allowed to contain any value. Wildcards are specified using a tuple of two elements: a header class name and a field name.
A single wildcard can be supplied (i.e., one 2-tuple) with the
wildcardkeyword parameter, or a list of 2-tuples can be supplied with the
wildcardskeyword. For example, the following line of code uses subset matching (
exact=False) and one wildcard. For this example, assume that the packet
UDPheaders:PacketOutputEvent('en0', pkt, exact=False, wildcard=(IPv4, 'src'))
Note that for the above example, the only fields compared in the IPv4 header would be the destination address and protocol field (since other fields are already ignored with
Here is another example that ignores source addresses in the Ethernet, IPv4 and UDP fields, leaving only two fields in the Ethernet header to be compared (dst and ethertype), two fields to be compared in the IPv4 header (dst and protocol) and one field in UDP (dst). Again, assume that the packet
UDPheaders:PacketOutputEvent('en0', pkt, exact=False, wildcards=[(Ethernet, 'src'), (IPv4, 'src'), (UDP, 'src')])
Switchyard previously allowed certain strings (modeled on the Openflow 1.0 specification) to be used to indicate wildcarded fields. These strings can no longer be used in the current version of Switchyard. To specify wildcarded fields, you must use the
Predicate functions: Lastly, predicate functions can be supplied to make arbitrary tests against packets. The
predicatekeyword argument can take a single
lambdafunction in the form of a string, and the
predicateskeyword argument can take a list of
lambdafunctions, each as strings. Each lambda given must take a single argument (the packet object to be inspected) and must yield a boolean value. (Note that internally, each lambda definition is
eval'ed by Switchyard.)
Here is one example that checks whether the IPv4 ttl field is between 32 and 34, inclusive. Note that this line of code contains a single predicate function as a string:PacketOutputEvent('en1', pkt, exact=False, predicate='''lambda p: p.has_header(IPv4) and 32 <= p[IPv4].ttl <= 34''')
To provide multiple predicates, just use the
predicates(plural) keyword and provide a list of lambdas-as-strings.
Test scenario examples¶
First, here is an example of a test scenario in which a packet is constructed and is expected to be received on port
eth1, then sent back out the same port, unmodified. Notice in the example that the name
scenario is required.
from switchyard.lib.userlib import * scenario = TestScenario("in/out test scenario example") # only one interface on this imaginary device scenario.add_interface('eth0', 'ab:cd:ef:ab:cd:ef', '18.104.22.168/16', iftype=InterfaceType.Wired) # construct a packet to be received p = Ethernet(src="00:11:22:33:44:55", dst="66:55:44:33:22:11") + \ IPv4(src="22.214.171.124", dst="126.96.36.199", protocol=IPProtocol.UDP) + \ UDP(src=5555, dst=8888) + b'some payload' # expect that the packet is received scenario.expect(PacketInputEvent('eth0', p), "A udp packet should arrive on eth0") # and expect that the packet is sent right back out scenario.expect(PacketOutputEvent('eth0', p, exact=True), "The udp packet should be emitted back out eth0")
Here is an additional example with a bit more complexity. The context for this example might be that we are implementing an IPv4 router. First, notice that we include in the scenario a static forwarding table file (
forwarding_table.txt) to be written out when the scenario is executed. We construct a packet destined to a particular IP address and create an expectation that it arrives on port
eth0. We then construct an expectation that the packet should be forwarded out port
eth2 (note that according to the forwarding table, any packets destined to 188.8.131.52/8 should be forwarded out that port). We also include a predicate function to test that the IPv4 ttl is decremented by 1. Note that if we did not include this predicate, any ttl value would be accepted since we have specified
exact=False. Note also that if we had set
exact=True we would almost certainly need to wildcard several fields, e.g., checksums in the IPv4 and UDP headers, and would still need to include a predicate to check that ttl has been properly decremented. Furthermore, if we were writing a test scenario for an IP router, we would also want to include expectations that the correct ARP messages were sent in order to obtain the hardware address corresponding to the next hop IP address.
from switchyard.lib.userlib import * scenario = TestScenario("packet forwarding example") # three interfaces on this device scenario.add_interface('eth0', 'ab:cd:ef:ab:cd:ef', '184.108.40.206/16') scenario.add_interface('eth1', '00:11:22:ab:cd:ef', '220.127.116.11/16') scenario.add_interface('eth2', 'ab:cd:ef:00:11:22', '18.104.22.168/24') # add a forwarding table file to be written out when the test # scenario is executed scenario.add_file('forwarding_table.txt', ''' # network subnet-mask next-hop port 22.214.171.124 255.0.0.0 126.96.36.199 eth2 188.8.131.52 255.255.0.0 184.108.40.206 eth1 ''') # construct a packet to be received p = Ethernet(src="00:11:22:33:44:55", dst="66:55:44:33:22:11") + \ IPv4(src="220.127.116.11", dst="18.104.22.168", protocol=IPProtocol.UDP, ttl=61) + \ UDP(src=5555, dst=8888) + b'some payload' # expect that the packet is received scenario.expect(PacketInputEvent('eth0', p), "A udp packet destined to 22.214.171.124 arrives on port eth0") # and subsequently forwarded out the correct port; employ # subset (exact=False) matching, along with a check that the # IPv4 TTL was decremented exactly by 1. scenario.expect(PacketOutputEvent('eth2', p, exact=False, predicate='''lambda pkt: pkt.has_header(IPv4) and pkt[IPv4].ttl == 60'''), "The udp packet destined to 126.96.36.199 should be forwarded out port eth2, with an appropriately decremented TTL.")
Compiling a test scenario¶
A test scenario can be run directly with
swyard or it can be compiled into a form that can be distributed without giving away the code which was used to construct it. Compiled test scenario files are, by default, given a
.srpy extension; uncompiled test scenarios should just be regular Python (
To compile a test scenario, you can simply invoke
swyard with the
-c flag, as follows:
swyard -c code/testscenario2.py
The output from this command should be a new file named
code/testscenario2.srpy containing the obfuscated test scenario. This file can be used as the argument to the
-t option when later running a Switchyard program against those tests.
Note that if a scenario is compiled using a different version of Python than the one used to run a test scenario (especially a different major version, e.g., 3.4 vs. 3.5), you may get some mysterious errors. The errors are due to the fact that serialized representations of Python objects may change from one version to the next; if there are any changes, then the version used to run the test cannot correctly deserialize the various objects stored in the test scenario.