Live tests

Live tests are used to validate configurations built by ARouteServer and to test compliance between expected and real results.

A mix of Python unittest and Docker (and KVM too for OpenBGPD tests) allows to create scenarios where some instances of BGP speakers (the clients) connect to a route server whose configuration has been generated using this tool.

Some built-in tests are included within the project and have been used during the development of the tool; new custom scenarios can be easily built by users and IXP managers to test their own policies.

Example: in a configuration where blackhole filtering is enabled, an instance of a route server client (AS1) is used to announce some tagged prefixes (203.0.113.1/32) and the instances representing other clients (AS2, AS3) are queried to ensure they receive those prefixes with the expected blackhole NEXT_HOP (192.0.2.66).

def test_071_blackholed_prefixes_as_seen_by_enabled_clients(self):
  for inst in (self.AS2, self.AS3):
    self.receive_route(inst, "203.0.113.1/32", self.rs,
                       next_hop="192.0.2.66",
                       std_comms=["65535:666"], lrg_comms=[])

GitHub Actions log file contains the latest built-in live tests results. Since (AFAIK) OpenBGPD can’t be run on GitHub Actions platform, the full live tests results, including those run on OpenBGPD, can be found on this file. Starting with version 6.5, the Portable edition of OpenBGPD has been used to run some tests on GitHub Actions too.

A summary of the integration testing results and the BGP speakers which are tested can be found on the Integration testing coverage section of this documentation.

Setting up the environment to run live tests

To run live tests, Docker must be present on the system. Some info about its installation can be found on the External programs installation section.
In order to have instances of the route server and its clients to connect each other, a common network must be used. Live tests are expected to be run on a Docker bridge network with name arouteserver and subnet 192.0.2.0/24/2001:db8:1:1::/64. The following command can be used to create this network:

docker network create --ipv6 --subnet=192.0.2.0/24 --subnet=2001:db8:1:1::/64 arouteserver

Route server client instances used in live tests are based on BIRD 1.6.8, as well as the BIRD-based version of the route server used in built-in live tests; the pierky/bird:1.6.8 image is expected to be found on the local Docker repository. Also, for OpenBGPD Portable edition tests, pierky/openbgpd:6.6p0 must be there. Build the Docker image (or pull it from Dockerhub):

# build the image using the Dockerfile
# from https://github.com/pierky/dockerfiles
mkdir ~/dockerfiles
cd ~/dockerfiles
curl -o Dockerfile.bird -L https://raw.githubusercontent.com/pierky/dockerfiles/master/bird/1.6.8/Dockerfile
docker build -t pierky/bird:1.6.8 -f Dockerfile.bird .
curl -o Dockerfile.openbgpd -L https://raw.githubusercontent.com/pierky/dockerfiles/master/openbgpd/6.6p0/Dockerfile
docker build -t pierky/openbgpd:6.6p0 -f Dockerfile.openbgpd .

# or pull it from Dockerhub
docker pull pierky/bird:1.6.8
docker pull pierky/openbgpd:6.6p0

If there is no plan to run tests on the OpenBGPD-based version of the route server, no further settings are needed. To run tests on the OpenBGPD-based version too, the following steps must be done as well.

OpenBGPD live-tests environment

To run an instance of OpenBGPD, KVM is needed. Some info about its installation can be found on the External programs installation section.
Setup and install a KVM virtual-machine running one of the supported versions of OpenBSD. This VM will be started and stopped many times during tests: don’t use a production VM.
- By default, the VM name must be arouteserver_openbgpd60 or arouteserver_openbgpd61 or arouteserver_openbgpd62; this can be changed by setting the VIRSH_DOMAINNAME environment variable before running the tests.
- The VM must be connected to the same Docker network created above: the commands ip link show and ifconfig can be used to determine the local network name needed when creating the VM:
```
$ ifconfig
br-2d2956ce4b64 Link encap:Ethernet  HWaddr 02:42:57:82:bc:91
  inet addr:192.0.2.1  Bcast:0.0.0.0  Mask:255.255.255.0
  inet6 addr: fe80::42:57ff:fe82:bc91/64 Scope:Link
  inet6 addr: 2001:db8:1:1::1/64 Scope:Global
  inet6 addr: fe80::1/64 Scope:Link
  UP BROADCAST MULTICAST  MTU:1500  Metric:1
  ...
```
- In order to run built-in live test scenarios, the VM must be reachable at 192.0.2.2/24 and 2001:db8:1:1::2/64.
On the following example, the virtual disk will be stored in ~/vms, the VM will be reachable by connecting to any IP address of the host via VNC, the installation disk image is expected to be found in the install60.iso file and the network name used is br-2d2956ce4b64:
```
sudo virsh pool-define-as --name vms_pool --type dir --target ~/vms
sudo virsh pool-start vms_pool
sudo virt-install \
  -n arouteserver_openbgpd66 \
  -r 512 \
  --vcpus=1 \
  --os-variant=openbsd4.2 \
  --accelerate \
  -v -c install66.iso \
  -w bridge:br-2d2956ce4b64 \
  --graphics vnc,listen=0.0.0.0 \
  --disk path=~/vms/arouteserver_openbgpd66.qcow2,size=5,format=qcow2
```
Finally, add the current user to the libvirtd group to allow management of the VM:
```
sudo adduser `id -un` libvirtd
```
To interact with this VM, the live tests framework will use SSH; by default, the connection will be established using the root username and the local key file ~/.ssh/arouteserver, so the VM must be configured to accept SSH connections using SSH keys:
```
mkdir /root/.ssh
cat << EOF > .ssh/authorized_keys
ssh-rsa [public_key_here] arouteserver
EOF
```
The StrictHostKeyChecking option is disabled via command line argument in order to allow to connect to multiple different VMs with the same IP address.

The SSH username and key file path can be changed by setting the SSH_USERNAME and SSH_KEY_PATH environment variables before running the tests.

Be sure that the bgpd daemon will startup automatically at boot and that the bgpctl tool can be executed correctly on the OpenBSD VM:
```
echo "bgpd_flags=" >> /etc/rc.conf.local
chmod 0555 /var/www/bin/bgpctl
```

How to run built-in live tests

To run built-in live tests, the full repository must be cloned locally and the environment must be configured as reported above.

To test both the BIRD- and OpenBGPD-based route servers, run the Python unittest using pytest:

# from within the repository's root
pytest -vs tests/live_tests/

How it works

Each directory in tests/live_tests/scenarios represents a scenario: the route server configuration is stored in the usual general.yml and clients.yml files, while other BGP speaker instances (route server clients and their peers) are configured through the ASxxx.j2 files. These files are Jinja2 templates and are expanded by the Python code at runtime. Containers’ configuration files are saved in the local var directory and are used to mount the BGP speaker configuration file (currenly, /etc/bird/bird.conf for BIRD and /etc/bgpd.conf for OpenBGPD). The unittest code sets up a Docker network (with name arouteserver) used to attach instances and finally brings instances up. Regular Python unittest tests are then performed and can be used to match expectations to real results.

Details about the code behind the live tests can be found in the Live tests code documentation section.

Built-in scenarios

Some notes about the built-in scenarios that are provided with the program follow.

How to build custom scenarios

A live test scenario skeleton is provided in the pierky/arouteserver/tests/live_tests/skeleton directory.

It seems to be a complex thing but actually most of the work is already done in the underlying Python classes and prepared in the skeleton.

To configure the route server and its clients, please consider that the Docker network used by the framework is on 192.0.2.0/24 and 2001:db8:1:1::/64 subnets.

Initialize the new scenario into a new directory:

using the init-scenario command:

arouteserver init-scenario ~/ars_scenarios/myscenario

manually, by cloning the provided skeleton directory:

mkdir -p ~/ars_scenarios/myscenario
cp pierky/arouteserver/tests/live_tests/skeleton/* ~/ars_scenarios/myscenario

Document the scenario, for example in the README.rst file: write down which BGP speakers are involved, how they are configured, which prefixes they announce and what the expected result should be with regards of the route server’s configuration and its policies.
Put the general.yml, clients.yml and bogons.yml configuration files you want to test in the new directory.
Configure your scenario and write your test functions in the base.py file.
- Declare the BGP speakers you want to use in the _setup_rs_instance() and _setup_instances() methods of the base class.
  classmethod SkeletonScenario._setup_instances()
  
  Declare the BGP speaker instances that are used in this scenario.
  
  The cls.INSTANCES attribute is a list of all the instances that are used in this scenario. It is used to render local Jinja2 templates and to transform them into real BGP speaker configuration files.
  
  The cls.RS_INSTANCE_CLASS and cls.CLIENT_INSTANCE_CLASS attributes are set by the derived classes (test_XXX.py) and represent the route server class and the other BGP speakers class respectively.
  
  The first argument is the instance name.
  
  The second argument is the IP address that is used to run the instance. Here, the cls.DATA dictionary is used to lookup the real IP address to use, which is configured in the derived classes (test_XXX.py).
  
  The third argument is a list of files that are mounted from the local host (where Docker is running) to the container (the BGP speaker). The list is made of pairs in the form (local_file, container_file). The cls.build_rs_cfg and cls.build_other_cfg helper functions allow to render Jinja2 templates and to obtain the path of the local output files.
  
  For the route server, the configuration is built using ARouteServer’s library on the basis of the options given in the YAML files.
  
  For the other BGP speakers, the configuration must be provided in the Jinja2 files within the scenario directory.
  Example:
  @classmethod def _setup_instances(cls): cls.INSTANCES = [ cls._setup_rs_instance(), cls.CLIENT_INSTANCE_CLASS( "AS1", cls.DATA["AS1_IPAddress"], [ ( cls.build_other_cfg("AS1.j2"), "/etc/bird/bird.conf" ) ] ), ... ]
- To ease writing the test functions, set instances names in the set_instance_variables() method.
  SkeletonScenario.set_instance_variables()
  
  Simply set local attributes for an easier usage later
  
  The argument of self._get_instance_by_name() must be one of the instance names used in _setup_instances().
  
  Example:
  def set_instance_variables(self): self.AS1 = self._get_instance_by_name("AS1") self.AS2 = self._get_instance_by_name("AS2") self.rs = self._get_instance_by_name("rs")
- Write test functions to verify that scenario’s expectations are met.
  
  Some helper functions can be used:
  - LiveScenario.session_is_up(inst_a, inst_b)
    Test if a BGP session between the two instances is up.
    
    If a BGP session between the two instances is not up, the TestCase.fail() method is called and the test fails.
    
    Parameters:
    
    inst_a – the BGPSpeakerInstance instance where the BGP session is looked for.
    
    inst_b – the BGPSpeakerInstance instance that inst_a is expected to peer with.
    Example:
    def test_020_sessions_up(self): """{}: sessions are up""" self.session_is_up(self.rs, self.AS1) self.session_is_up(self.rs, self.AS2)
  - LiveScenario.receive_route(inst, prefix, other_inst=None, as_path=None, next_hop=None, std_comms=None, lrg_comms=None, ext_comms=None, local_pref=None, as_set=None, otc=None, filtered=None, only_best=None, reject_reason=None)
    Test if the BGP speaker receives the expected route(s).
    
    If no routes matching the given criteria are found, the TestCase.fail() method is called and the test fails.
    
    Parameters:
    
    inst – the BGPSpeakerInstance instance where the routes are searched on.
    
    prefix (str) – the IPv4/IPv6 prefix of the routes to search for.
    
    other_inst – if given, only routes received from this BGPSpeakerInstance instance are considered.
    
    as_path (str) – if given, only routes with this AS_PATH are considered.
    
    next_hop – can be a string or a BGPSpeakerInstance instance; if given, only routes that have a NEXT_HOP address matching this one are considered.
    
    std_comms (list) – if given, only routes that carry these BGP communities are considered. Use an empty list ([]) to consider only routes with no BGP comms.
    
    lrg_comms (list) – if given, only routes that carry these BGP communities are considered. Use an empty list ([]) to consider only routes with no BGP comms.
    
    ext_comms (list) – if given, only routes that carry these BGP communities are considered. Use an empty list ([]) to consider only routes with no BGP comms.
    
    local_pref (int) – if given, only routes with local-pref equal to this value are considered.
    
    as_set (str) – if given, only routes with this AS_SET are considered.
    
    otc (int) – if provided, only routes with the OTC attribute set to this value are considered. Use ‘0’ to match only routes NOT having the OTC value set.
    
    filtered (bool) – if given, only routes that have been (not) filtered are considered.
    
    only_best (bool) – if given, only best routes are considered.
    
    reject_reason (int) –
    valid only if filtered is True: if given the route must be reject with this reason code. It can be also a set of codes: in this case, the route must be rejected with one of those codes.
    
    The list of valid codes is reported in docs/CONFIG.rst or at https://arouteserver.readthedocs.io/en/latest/CONFIG.html#reject-policy
    Example:
    def test_030_rs_receives_AS2_prefix(self): """{}: rs receives AS2 prefix""" self.receive_route(self.rs, self.DATA["AS2_prefix1"], other_inst=self.AS2, as_path="2")
  - LiveScenario.log_contains(inst, msg, instances={}, opposite=False)
    Test if the BGP speaker’s log contains the expected message.
    
    This only works for BGP speaker instances that support message logging: currently only BIRD.
    
    If no log entries are found, the TestCase.fail() method is called and the test fails. If opposite is True, the failure is reported if a log entry is found.
    
    Parameters:
    
    inst – the BGPSpeakerInstance instance where the expected message is searched on.
    
    msg (str) – the text that is expected to be found within BGP speaker’s log.
    
    instances (dict) – a dictionary of pairs “<macro>: <BGPSpeakerInstance>” used to expand macros on the msg argument. Macros are expanded using the BGP speaker’s specific client ID or protocol name.
    
    opposite (bool) – when set to True, the call fails if a match is found.
    
    Example
    
    Given self.rs the instance of the route server, and self.AS1 the instance of one of its clients, the following code expands the “{AS1}” macro using the BGP speaker specific name for the instance self.AS1 and then looks for it within the route server’s log:
    
    self.log_contains(self.rs, "{AS1} bad ASN", {"AS1": self.AS1})
    
    On BIRD, “{AS1}” will be expanded using the “protocol name” that BIRD uses to identify the BGP session with AS1.
    Example:
    def test_030_rs_rejects_bogon(self): """{}: rs rejects bogon prefix""" self.log_contains(self.rs, "prefix is bogon - REJECTING {}".format( self.DATA["AS2_bogon1"])) self.receive_route(self.rs, self.DATA["AS2_bogon1"], other_inst=self.AS2, as_path="2", filtered=True) # AS1 should not receive the bogon prefix from the route server with self.assertRaisesRegex(AssertionError, "Routes not found"): self.receive_route(self.AS1, self.DATA["AS2_bogon1"])
Edit IP version specific and BGP speaker specific classes within the test_XXX.py files and set the prefix ID / real IP addresses mapping schema.

class pierky.arouteserver.tests.live_tests.skeleton.test_bird4.SkeletonScenario_BIRDIPv4(methodName='runTest')

BGP speaker specific and IP version specific derived class.

This class inherits all the test functions from the base class. Here, only IP version specific attributes are set, such as the prefix IDs / real IP prefixes mapping schema.

The prefix IDs reported within the DATA dictionary must be used in the parent class’ test functions to reference the real IP addresses/prefixes used in the scenario. Also the other BGP speakers’ configuration templates must use these IDs. For an example plase see the “AS2.j2” file.

The SHORT_DESCR attribute can be set with a brief description of this scenario.

Example:
```
class SkeletonScenario_BIRDIPv4(SkeletonScenario):

    # Leave this to True in order to allow pytest to use this class
    # to run tests.
    __test__ = True

    SHORT_DESCR = "Live test, BIRD, skeleton, IPv4"
    CONFIG_BUILDER_CLASS = BIRDConfigBuilder
    RS_INSTANCE_CLASS = BIRDInstanceIPv4
    CLIENT_INSTANCE_CLASS = BIRDInstanceIPv4
    IP_VER = 4

    DATA = {
        "rs_IPAddress":             "99.0.2.2",
        "AS1_IPAddress":            "99.0.2.11",
        "AS2_IPAddress":            "99.0.2.22",

        "AS2_prefix1":              "2.0.1.0/24",
        "AS2_bogon1":               "192.168.2.0/24"
    }
```

Edit (or add) the template files that, once rendered, will produce the configuration files for the other BGP speakers (route server clients) that are involved in the scenario (the skeleton includes two template files, AS1.j2 and AS2.j2).

Example:

router id 192.0.2.22;

# This is the path where Python classes look for
# to search BIRD's log files.
log "/var/log/bird.log" all;
log syslog all;
debug protocols all;

protocol device { }

# Prefixes announced by this BGP speaker to the route server.
#
# The Jinja2 'data' variable refers to the class 'DATA' attribute.
#
# IP prefixes are not configured directly here, only a reference
# to their ID is given in order to maintain a single configuration
# file that can be used for both the IPv4 and the IPv6 versions
# of the scenario.
protocol static own_prefixes {
	route {{ data.AS2_prefix1 }} reject;
	route {{ data.AS2_bogon1 }} reject;
}

protocol bgp the_rs {
	local as 2;
	neighbor {{ data.rs_IPAddress }} as 999;
	import all;
	export all;
	connect delay time 1;
	connect retry time 1;
}

Run the tests using pytest:
```
pytest -vs ~/ars_scenarios/myscenario
```

Details about the code behind the live tests can be found in the Live tests code documentation section.

Debugging live tests scenarios

To debug custom scenarios some utilities are provided:

the REUSE_INSTANCES environment variable can be set when executing pytest to avoid Docker instances to be torn down at the end of a run. When this environment variable is set, BGP speaker instances are started only the first time tests are executed, then are left up and running to allow debugging. When tests are executed again, the BGP speakers’ configuration is rebuilt and reloaded. Be careful: this mode can be used only when running tests of the same scenario, otherwise Bad Things (tm) may happen.

Example:
```
REUSE_INSTANCES=1 pytest -vs tests/live_tests/scenarios/global/test_bird4.py
```

once the BGP speaker instances are up (using the REUSE_INSTANCES environment variable seen above), they can be queried using standard Docker commands:

$ # list all the running Docker instances
$ docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS               NAMES
142f88379428        pierky/bird:1.6.3   "bird -c /etc/bird..."   18 minutes ago      Up 18 minutes       179/tcp             ars_AS101
26a9ec58dcf1        pierky/bird:1.6.3   "bird -c /etc/bird..."   18 minutes ago      Up 18 minutes       179/tcp             ars_AS2

$ # run 'birdcl show route' on ars_AS101
$ docker exec -it 142f88379428 birdcl show route

Some utilities are provided whitin the /utils directory to ease these tasks:

# execute the 'show route' command on the route server BIRD Docker instance
./utils/birdcl rs show route

# print the log of the route server
./utils/run rs cat /var/log/bird.log

The first argument (“rs” in the examples above) is the name of the instance as set in the _setup_instances() method.

the BUILD_ONLY environment variable can be set to skip all the tests and only build the involved BGP speakers’ configurations. Docker instances are not started in this mode.

Example:
```
BUILD_ONLY=1 pytest -vs tests/live_tests/scenarios/global/test_bird4.py
```