Test Specification

This file lists all test cases’ specifications.

class TimeSync
Description
This test suite is related with time sync.
Platform

Azure, Ready

Area

time

Category

functional

timesync_check_unbind_clocksource()
Description
This test is to check -
1. Check clock source name is one of hyperv_clocksource_tsc_page,
lis_hv_clocksource_tsc_page, hyperv_clocksource, tsc,
arch_sys_counter(arm64).
(there’s a new feature in the AH2021 host that allows Linux guests so use
the plain “tsc” instead of the “hyperv_clocksource_tsc_page”,
which produces a modest performance benefit when reading the clock.)
2. Check CPU flag contains constant_tsc from /proc/cpuinfo.
3. Check clocksource name shown up in dmesg.
4. Unbind current clock source if there are 2+ clock sources, check current
clock source can be switched to a different one.
Priority

2

timesync_chrony()
Description
This test is to check chrony works properly.
1. Restart chrony service.
2. Check and set server setting in config file.
3. Restart chrony service to reload with new config.
4. Check chrony sources and sourcestats.
5. Check chrony tracking.
Priority

2

timesync_ntp()
Description
This test is to check, ntp works properly.
1. Stop systemd-timesyncd if this service exists.
2. Set rtc clock to system time.
3. Restart Ntp service.
4. Check and set server setting in config file.
5. Restart Ntp service to reload with new config.
6. Check leap code using ntpq -c rv.
7. Check local time is synchronised with time server using ntpstat.
Priority

2

timesync_validate_ptp()
Description
This test is to check -
1. PTP time source is available on Azure guests (newer versions of Linux).
2. PTP device name is hyperv.
3. When accelerated network is enabled, multiple PTP devices will
be available, the names of ptp are changeable, create the symlink
/dev/ptp_hyperv to whichever /dev/ptp entry corresponds to the Azure host.
4. Chrony should be configured to use the symlink /dev/ptp_hyperv
instead of /dev/ptp0 or /dev/ptp1.
Priority

2

timesync_check_unbind_clockevent()
Description
This test is to check -
1. Current clock event name is ‘Hyper-V clockevent’ for x86,
‘arch_sys_timer’ for arm64.
2. ‘Hyper-V clockevent’ or ‘arch_sys_timer’ and ‘hrtimer_interrupt’
show up times in /proc/timer_list should equal to cpu count.
3. when cpu count is 1 and cpu type is Intel type, unbind current time
clock event, check current time clock event switch to ‘lapic’.
Priority

2

class VmHotResize
Description
This test suite tests vm behavior upon resizing without shutting down
Platform

Azure, Ready

Area

vm_hot_resize

Category

functional

verify_vm_hot_resize()
Description
This test case resizes the node and checks if it has the expected capabilities
(memory size and core count) after the resize
Steps:
1. Resize vm
2. Check the node’s core count and memory size against their expected values
Priority

1

Requirement

supported_features[Resize]

class HvModule
Description
This test suite covers test cases previously handled by LISAv2:
LIS-MODULES-CHECK, VERIFY-LIS-MODULES-VERSION,
INITRD-MODULES-CHECK, RELOAD-MODULES-SMP
It is responsible for ensuring the Hyper V drivers are all present,
are included in initrd, and are all the same version.
Platform

Azure, Ready

Area

core

Category

functional

verify_hyperv_modules()
Description
This test case will
1. Verify the presence of all Hyper V drivers using lsmod
to look for the drivers not directly loaded into the kernel.
Priority

1

verify_initrd_modules()
Description
This test case will ensure all necessary hv_modules are present in
initrd. This is achieved by
1. Skipping any modules that are loaded directly in the kernel
2. Find the path of initrd file
3. Use lsinitrd tool to check whether a necessary module is missing
Priority

2

reload_hyperv_modules()
Description
This test case will reload hyper-v modules as a stress test.
Priority

1

verify_lis_modules_version()
Description
This test case will
1. Verify the list of given LIS kernel modules and verify if the version
matches with the Linux kernel release number. (Drivers loaded directly in
to the kernel are skipped)
Priority

1

class Dhcp
Description
This test suite covers DHCP functionalities.
Platform

Azure, Ready

Area

core

Category

functional

verify_dhcp_client_timeout()
Description
This test case check the timeout setting of DHCP on Azure equals or more
than 300 seconds.
Priority

1

Requirement

supported_platform_type[AZURE, READY]

class AzureImageStandard
Description
This test suite is used to check azure image configuration.
Platform

Azure, Ready

Area

azure_image_standard

Category

functional

verify_default_targetpw()
Description
This test will verify that Defaults targetpw is not enabled in the
/etc/sudoers file.
If targetpw is set, sudo will prompt for the
password of the user specified by the -u option (defaults to root)
instead of the password of the invoking user when running a command
or editing a file. More information can be found here :
Steps:
1. Get the content of /etc/sudoers file.
2. Verify that Defaults targetpw should be disabled, if present.
Priority

1

Requirement

supported_platform_type[AZURE, READY]

verify_yum_conf()
Description
This test will verify content of yum.conf file on Fedora based distros
for version < 6.6.0
Steps:
1. Read the yum.conf file and verify that “http_caching=packages” is
present in the file.
Priority

1

Requirement

supported_platform_type[AZURE, READY]

verify_no_pre_exist_users()
Description
This test will check no pre added users existing in vm.
Steps:
1. Exclude current user from all users’ list.
2. Fail the case if the password of any above user existing.
Priority

1

Requirement

supported_platform_type[AZURE, READY]

verify_client_active_interval()
Description
This test will check ClientAliveInterval value in sshd config.
Steps:
1. Find ClientAliveInterval from sshd config.
2. Pass with warning if not find it.
3. Pass with warning if the value is not between 0 and 180.
Priority

1

Requirement

supported_platform_type[AZURE, READY]

verify_network_file_configuration()
Description
This test will verify that network file exists in /etc/sysconfig and networking
is enabled on Fedora based distros.
Steps:
1. Verify that network file exists.
2. Verify that networking is enabled in the file.
Priority

1

Requirement

supported_platform_type[AZURE, READY]

verify_network_manager_not_installed()
Description
This test will verify that network manager doesn’t conflict with the
waagent on Fedora based distros.
Steps:
1. Get the output of command rpm -q NetworkManager and verify that
network manager is not installed.
Priority

1

Requirement

supported_platform_type[AZURE, READY]

verify_bash_history_is_empty()
Description
This test will check the /root/.bash_history not existing or is empty.
Steps:
1. Check .bash_history exist or not, if not, the image is prepared well.
2. If the .bash_history existed, check the content is empty or not, if not, the
image is not prepared well.
Priority

1

Requirement

supported_platform_type[AZURE, READY]

verify_grub()
Description
This test will check the configuration of the grub file and verify that numa
is disabled for Redhat distro version < 6.6.0
Steps:
1. Verify grub configuration depending on the distro type.
2. For Redhat based distros, verify that numa is disabled for versions < 6.6.0
Priority

1

Requirement

supported_platform_type[AZURE, READY]

verify_udev_rules_moved()
Description
This test will verify that udev rules have been moved out in CoreOS
and Fedora based distros
Steps:
1. Verify that 75-persistent-net-generator.rules and 70-persistent-net.rules
files are not present.
Priority

1

Requirement

supported_platform_type[AZURE, READY]

verify_dhcp_file_configuration()
Description
This test will verify that dhcp file exists at
/etc/sysconfig/network/dhcp and DHCLIENT_SET_HOSTNAME is set
to no.
Steps:
1. Verify that dhcp file exists.
2. Verify that DHCLIENT_SET_HOSTNAME=”no” is present in the file.
Priority

1

Requirement

supported_platform_type[AZURE, READY]

verify_repository_installed()
Description
This test will check that repositories are correctly installed.
Steps:
1. Verify the repository configuration depending on the distro type.
Priority

1

Requirement

supported_platform_type[AZURE, READY]

verify_boot_error_fail_warnings()
Description
This test will check error, failure, warning messages from demsg,
/var/log/syslog or /var/log/messages file.
Steps:
1. Get failure, error, warning messages from dmesg, /var/log/syslog or
/var/log/messages file.
2. If any unexpected failure, error, warning messages excluding ignorable ones
existing, fail the case.
Priority

1

Requirement

supported_platform_type[AZURE, READY]

verify_hv_kvp_daemon_installed()
Description
This test will check that hv-kvp-daemon-init is installed on Debian based
distros. This is an optional requirement.
Steps:
1. Verify that list of running process matching name hv_kvp_daemon
has length greater than zero.
Priority

1

Requirement

supported_platform_type[AZURE, READY]

verify_serial_console_is_enabled()
Description
This test will check the serial console is enabled from kernel command line
in dmesg.
Steps:
1. Get the kernel command line from dmesg output.
2. Check expected setting from kernel command line.
2.1. Expected to see ‘console=ttyAMA0’ for aarch64.
2.2. Expected to see ‘console=ttyS0’ for x86_64.
Priority

1

Requirement

supported_platform_type[AZURE, READY]

verify_ifcfg_eth0()
Description
This test will verify contents of ifcfg-eth0 file on Fedora based distros.
Steps:
1. Read the ifcfg-eth0 file and verify that “DEVICE=eth0”, “BOOTPROTO=dhcp” and
“ONBOOT=yes” is present in network file.
Priority

1

Requirement

supported_platform_type[AZURE, READY]

class CPU
Description
This test suite is used to run CPU related tests.
Platform

Azure, Ready

Area

core

Category

functional

cpu_count_check()
Description
This test will check that vCPU count correctness.
Steps :
1. Get vCPU count.
2. Calculate vCPU count by core_per_socket_count * socket_count *
thread_per_core_count.
3. Judge whether the actual vCPU count equals to expected value.
Priority

2

l3_cache_check()
Description
This test case will check that L3 cache is correctly mapped
to NUMA node.
Steps:
1. Check if NUMA is disabled in commandline. If disabled,
and kernel version is <= 2.6.37, test is skipped as hyper-v
has no support for NUMA : https://t.ly/x8k3
2. Get the mappings using command :
lscpu –extended=cpu,node,socket,cache
3. Each line in the mapping corresponds to one CPU core. The L3
cache of each core must be mapped to the NUMA node that core
belongs to instead of the core itself.
Example :
Correct mapping:
CPU NODE SOCKET L1d L1i L2 L3
8 0 0 8 8 8 0
9 1 1 9 9 9 1
Incorrect mapping:
CPU NODE SOCKET L1d L1i L2 L3
8 0 0 8 8 8 8
9 1 1 9 9 9 9
Priority

2

verify_vmbus_interrupts()
Description
This test will verify if the CPUs inside a Linux VM are processing VMBus
interrupts by checking the /proc/interrupts file.
There are 3 types of Hyper-v interrupts : Hypervisor callback
interrupts, Hyper-V reenlightenment interrupts, and Hyper-V stimer0
interrupts.
Hyper-V reenlightenment interrupts are 0 unless the VM is doing migration.
Hypervisor callback interrupts are vmbus events that are generated on all
the vmbus channels, which belong to different vmbus devices. A VM with upto
4 vcpu on Azure/Hyper-V should have a NetVSC NIC, which normally has 4 VMBus
channel and should be bound to all the vCPUs.
Hyper-V Synthetic timer interrupts should be received on each CPU if the VM
is run for a long time. We can simulate this process by running CPU
intensive workload on each vCPU.
Steps:
1. Look for the Hyper-v timer property of each vCPU under /proc/interrupts
2. For Hyper-V reenlightenment interrupt, verify that the interrupt count
for all vCPU are zero.
3. For Hypervisor callback interrupt, verify that atleast min(#vCPU, 4)
vCPU’s are processing interrupts.
4. For Hyper-V Synthetic timer, run a CPU intensive command on each vCPU and
verify that every vCPU is processing the interrupt.
Priority

2

class Boot
Description
This test suite is to test VM working well after updating on VM and rebooting.
Platform

Azure, Ready

Area

core

Category

functional

verify_boot_with_debug_kernel()
Description
This test case will
1. Skip testing if the distro is not redhat/centos type, RHEL added this test
case, since they encounter an issue which is seeing call trace when boot
with debug kernel.
2. Install kernel-debug package and set boot with this debug kernel.
3. Reboot VM, check kernel version is debug type.
Priority

3

Requirement

supported_features[SerialConsole]

class LsVmBus
Description
This test suite is used to check vmbus devices and their associated vmbus channels.
Platform

Azure, Ready

Area

core

Category

functional

lsvmbus_count_devices_channels()
Description
This test case will
1. Check expected vmbus device names presented in the lsvmbus output.
- Operating system shutdown
- Time Synchronization
- Heartbeat
- Data Exchange
- Synthetic mouse
- Synthetic keyboard
- Synthetic network adapter
- Synthetic SCSI Controller
- Synthetic IDE Controller (gen1 only)
2. Check that each netvsc and storvsc SCSI device have correct number of vmbus
channels created and associated.
2.1 Check expected channel count of each netvsc is min (num of vcpu, 8).
2.2 Check expected channel count of each storvsc SCSI device is min (num of
vcpu/4, 64).
2.2.1 Caculate channel count of each storvsc SCSI device.
2.2.2 Cap of channel count of each storvsc SCSI device,
it is decided by host storage VSP driver.
Priority

1

Requirement

supported_platform_type[AZURE]

verify_vmbus_heartbeat_properties()
Description
This test case will
1. Looks for the VMBus heartbeat device properties.
2. Checks the properties can be read and that the folder structure exists.
3. Checks that the in_* files are equal to the out_* files
when read together.
4. Checks the interrupts and events values are increasing during
reading them.
Priority

4

Requirement

supported_platform_type[AZURE]

class Provisioning
Description
This test suite uses to verify if an environment can be provisioned correct or not.
- The basic smoke test can run on all images to determinate if a image can boot and
reboot.
- Other provisioning tests verify if an environment can be provisioned with special
hardware configurations.
Platform

Azure, Ready

Area

provisioning

Category

functional

verify_deployment_provision_standard_ssd_disk()
Description
This case runs smoke test on a node provisioned with standard ssd disk.
The test steps are same as smoke_test.
Priority

1

Requirement

supported_features[SerialConsole]

verify_deployment_provision_ephemeral_managed_disk()
Description
This case runs smoke test on a node provisioned with ephemeral disk.
The test steps are same as smoke_test.
Priority

1

Requirement

supported_features[SerialConsole]

verify_deployment_provision_sriov()
Description
This case runs smoke test on a node provisioned with sriov.
The test steps are same as smoke_test.
Priority

1

Requirement

supported_features[SerialConsole]

verify_reboot_in_platform()
Description
This case runs smoke test on a node provisioned.
The test steps are almost the same as smoke_test except for
executing reboot from Azure SDK.
Priority

1

Requirement

supported_features[SerialConsole, StartStop]

smoke_test()
Description
This case verifies whether a node is operating normally.
Steps,
1. Connect to TCP port 22. If it’s not connectable, failed and check whether
there is kernel panic.
2. Connect to SSH port 22, and reboot the node. If there is an error and kernel
panic, fail the case. If it’s not connectable, also fail the case.
3. If there is another error, but not kernel panic or tcp connection, pass with
warning.
4. Otherwise, fully passed.
Priority

0

Requirement

supported_features[SerialConsole]

verify_stop_start_in_platform()
Description
This case runs smoke test on a node provisioned.
The test steps are almost the same as smoke_test except for
executing stop then start from Azure SDK.
Priority

2

Requirement

supported_features[SerialConsole, StartStop]

verify_deployment_provision_premium_disk()
Description
This case runs smoke test on a node provisioned with premium disk.
The test steps are same as smoke_test.
Priority

1

Requirement

supported_features[SerialConsole]

verify_deployment_provision_synthetic_nic()
Description
This case runs smoke test on a node provisioned with synthetic nic.
The test steps are same as smoke_test.
Priority

1

Requirement

supported_features[SerialConsole]

class Storage
Description
This test suite is used to run storage related tests.
Platform

Azure, Ready

Area

storage

Category

functional

hot_add_disk_serial()
Description
This test case will verify that the standard hdd data disks disks can
be added one after other (serially) while the vm is running.
Steps:
1. Get maximum number of data disk for the current vm_size.
2. Get the number of data disks already added to the vm.
3. Serially add and remove the data disks and verify that the added
disks are present in the vm.
Priority

2

Requirement

disk

hot_add_disk_serial_premium_ssd()
Description
This test case will verify that the premium ssd data disks disks can
be added serially while the vm is running. The test steps are same as
hot_add_disk_serial.
Priority

2

Requirement

disk

hot_add_disk_serial_standard_ssd()
Description
This test case will verify that the standard ssd data disks disks can
be added serially while the vm is running. The test steps are same as
hot_add_disk_serial.
Priority

2

Requirement

disk

verify_disks_device_timeout_setting()
Description
This test will check that VM disks are provisioned
with the correct timeout.
Steps:
1. Find the disks for the VM by listing /sys/block/sd*.
2. Verify the timeout value for disk in
/sys/block/<disk>/device/timeout file is set to 300.
Priority

1

Requirement

supported_platform_type[AZURE]

hot_add_disk_parallel_premium_ssd()
Description
This test case will verify that the premium ssd data disks disks can
be added serially while the vm is running. The test steps are same as
hot_add_disk_parallel.
Priority

2

Requirement

disk

hot_add_disk_parallel_standard_ssd()
Description
This test case will verify that the standard ssd data disks disks can
be added serially while the vm is running. The test steps are same as
hot_add_disk_parallel.
Priority

2

Requirement

disk

hot_add_disk_parallel()
Description
This test case will verify that the standard HDD data disks can
be added in one go (parallel) while the vm is running.
Steps:
1. Get maximum number of data disk for the current vm_size.
2. Get the number of data disks already added to the vm.
3. Add maximum number of data disks to the VM in parallel.
4. Verify that the disks are added are available in the OS.
5. Remove the disks from the vm in parallel.
6. Verify that the disks are removed from the OS.
Priority

2

Requirement

disk

verify_swap()
Description
This test will check that the swap is correctly configured on the VM.
Steps:
1. Check if swap file/partition is configured by checking the output of
swapon -s and lsblk.
2. Check swap status in waagent.conf.
3. Verify that truth value in step 1 and step 2 match.
Priority

1

Requirement

supported_platform_type[AZURE]

verify_resource_disk_io()
Description
This test will check that the file IO operations are working correctly
Steps:
1. Get the mount point for the resource disk. If /var/log/cloud-init.log
file is present, mount location is /mnt, otherwise it is obtained from
ResourceDisk.MountPoint entry in waagent.conf configuration file.
2. Verify that resource disk is mounted from the output of mount command.
3. Write a text file to the resource disk.
4. Read the text file and verify that content is same.
Priority

1

Requirement

supported_platform_type[AZURE]

verify_resource_disk_mtab_entry()
Description
This test will check that the resource disk is present in the list of mounted
devices. Most VMs contain a resource disk, which is not a managed disk and
provides short-term storage for applications and processes. It is intended to
only store data such as page or swap files.
Steps:
1. Get the mount point for the resource disk. If /var/log/cloud-init.log
file is present, mount location is /mnt, otherwise it is obtained from
ResourceDisk.MountPoint entry in waagent.conf configuration file.
2. Verify that “/dev/<disk> <mount_point>` entry is present in
/etc/mtab file and the disk should not be equal to os disk.
Priority

1

Requirement

supported_platform_type[AZURE]

verify_os_partition_identifier()
Description
This test will verify that identifier of root partition matches
from different sources.
Steps:
1. Get the partition identifier from blkid command.
2. Verify that the partition identifier from blkid is present in dmesg.
3. Verify that the partition identifier from blkid is present in fstab output.
Priority

1

Requirement

supported_platform_type[AZURE]

class Kvp
Description
This test suite verify the KVP service runs well on Azure and Hyper-V
platforms. The KVP is used to communicate between Windows host and guest VM.
Platform

Azure, Ready

Area

kvp

Category

functional

verify_kvp()
Description
Verify KVP daemon installed, running, permission correct.
1. verify that the KVP Daemon is running.
2. run the KVP client tool and verify that the data pools are created and
accessible.
3. check kvp_pool file permission is 644.
4. check kernel version supports hv_kvp.
5. Check if KVP pool 3 file has a size greater than zero.
6. At least 11 items are present in pool 3, and verify record count is
correct.
Priority

1

class Floppy
Description
This test suite ensures the floppy driver is disabled.
The floppy driver is not needed on Azure and
is known to cause problems in some scenarios.
Platform

Azure, Ready

Area

core

Category

functional

check_floppy_module()
Description
The goal of this test is to ensure the floppy module is not enabled
for images used on the Azure platform.
This test case will
1. Dry-run modprobe to see if floppy module can be loaded
2. If “insmod” would be executed then the module is not already loaded
3. If module cannot be found then it is not loaded
If the module is loaded, running modprobe will have no output
Priority

1

class InfinibandSuit
Description
Tests the functionality of infiniband.
Platform

Azure, Ready

Area

hpc

Category

functional

verify_ping_pong()
Description
This test case will
1. Identify the infiniband devices and their cooresponding network interfaces
2. Run several ping-pong tests to check RDMA / Infiniband functionality
Priority

1

Requirement

min_count=2

verify_mvapich_mpi()
Description
This test case will
1. Ensure RDMA is setup
2. Install MVAPICH MPI
3. Set up ssh keys of server/client connection
4. Run MPI pingpong tests
5. Run other MPI tests
Priority

4

Requirement

min_count=2

verify_ibm_mpi()
Description
This test case will
1. Ensure RDMA is setup
2. Install IBM MPI
3. Set up ssh keys of server/client connection
4. Run MPI pingpong tests
Priority

4

Requirement

min_count=2

verify_hpc_over_sriov()
Description
This test case will
1. Determine whether the VM has Infiniband over SR-IOV
2. Ensure waagent is configures with OS.EnableRDMA=y
3. Check that appropriate drivers are present
Priority

2

Requirement

supported_features[Infiniband]

verify_open_mpi()
Description
This test case will
1. Ensure RDMA is setup
2. Install Open MPI
3. Set up ssh keys of server/client connection
4. Run MPI pingpong tests
5. Run other MPI tests
Priority

4

Requirement

min_count=2

verify_intel_mpi()
Description
This test case will
1. Ensure RDMA is setup
2. Install Intel MPI
3. Set up ssh keys of server/client connection
4. Run MPI pingpong tests
5. Run other MPI tests
Priority

4

Requirement

min_count=2

verify_hpc_over_nd()
Description
This test case will
1. Determine whether the VM has Infiniband over Network Direct
2. Ensure waagent is configures with OS.EnableRDMA=y
3. Check that appropriate drivers are present
Priority

2

Requirement

supported_features[Infiniband]

class XdpPerformance
Description
This test suite is to validate XDP performance.
Platform

Azure, Ready

Area

xdp

Category

performance

perf_xdp_ntttcp_latency()
Description
This case compare and record latency impact of XDP component.
The test use ntttcp to send tcp packets. And then compare the latency
with/without XDP component. If the gap is more than 40%, the test case
fails.
Priority

3

Requirement

network_interface

perf_xdp_lagscope_latency()
Description
This case compare and record latency impact of XDP component.
The test use lagscope to send tcp packets. And then compare the latency
with/without XDP component. If the gap is more than 40%, the test case
fails.
Priority

3

Requirement

network_interface

perf_xdp_rx_drop_multithread_sriov()
Description
This case tests the XDP drop performance by measuring Packets Per Second
(PPS) and received rate with multiple send threads.
* If the received packets rate is lower than 90% the test case fails.
* If the PPS is lower than 1M, the test case fails.
Priority

3

Requirement

network_interface

perf_xdp_rx_drop_singlethread_sriov()
Description
This case tests the XDP drop performance by measuring Packets Per Second
(PPS) and received rate with single send thread.
see details from perf_xdp_rx_drop_multithread_sriov.
Priority

3

Requirement

network_interface

perf_xdp_tx_forward_multicore_synthetic()
Description
This case tests the packet forwarded rate of XDP TX forwarding on the
multi core Syntethic networking.
Refer to perf_xdp_tx_forward_singlecore_synthetic for more details.
Priority

3

Requirement

network_interface

perf_xdp_tx_forward_singlecore_synthetic()
Description
This case tests the packet forwarded rate of the XDP TX forwarding on
the single core Synthetic networking. The pktgen samples in Linux code
base is used to generate packets.
The minimum cpu count is 8, it makes sure the performance is won’t too
low.
Three roles in this test environment, 1) sender is to send packets, 2)
the forwarder is to forward packets to receiver, 3) and the receiver is
used to receive packets and drop.
Finally, it checks how many packets arrives to the forwarder or
receiver. If it’s lower than 90%, the test fails. Note, it counts the
rx_xdp_tx_xmit (mlx5), rx_xdp_tx (mlx4), or dropped count for synthetic
nic.
Priority

3

Requirement

network_interface

perf_xdp_tx_forward_multicore_sriov()
Description
This case tests the packet forwarded rate of XDP TX forwarding on the
multi core SRIOV networking.
Refer to perf_xdp_tx_forward_singlecore_synthetic for more details.
The threshold of this test is lower than standard, it’s 85%. Because the
UDP packets count is big in this test scenario, and easy to lost.
Priority

3

Requirement

network_interface

perf_xdp_tx_forward_singlecore_sriov()
Description
This case tests the packet forwarded rate of XDP TX forwarding on the
single core SRIOV networking.
Refer to perf_xdp_tx_forward_singlecore_synthetic for more details.
Priority

3

Requirement

network_interface

class XdpFunctional
Description
This test suite is to validate XDP functionality.
Platform

Azure, Ready

Area

xdp

Category

functional

verify_xdp_with_different_mtu()
Description
It validates XDP with different MTU
1. Check current image supports XDP or not.
2. change MTU to 1500, 2000, 3506 to test XDP.
Priority

3

Requirement

min_count=2

verify_xdp_multiple_nics()
Description
It validates XDP with multiple nics.
1. Check current image supports XDP or not.
2. Install and validate xdpdump.
Priority

3

Requirement

min_nic_count=3

verify_xdp_sriov_failsafe()
Description
It validates the XDP works with Synthetic network when SRIOV is
disabled.
1. Test in SRIOV mode.
2. Disable the SRIOV.
3. Test in Synthetic mode.
Priority

2

Requirement

network_interface

verify_xdp_action_drop()
Description
It validates the XDP with action DROP.
1. start tcpdump with icmp filter.
2. start xdpdump.
3. run ping 5 times.
4. check tcpdump with 5 packets.
Priority

2

Requirement

min_count=2

verify_xdp_action_tx()
Description
It validates the XDP with action TX.
1. start tcpdump with icmp filter.
2. start xdpdump.
3. run ping 5 times.
4. check tcpdump with 5 packets, because the icmp is replied in xdp
level.
Priority

2

Requirement

min_count=2

verify_xdp_action_aborted()
Description
It validates the XDP with action ABORT.
1. start tcpdump with icmp filter.
2. start xdpdump.
3. run ping 5 times.
4. check tcpdump with 5 packets.
Priority

3

Requirement

min_count=2

verify_xdp_community_test()
Description
It runs all tests of xdp-tools. Check the official site for more
details.
Priority

3

verify_xdp_basic()
Description
It validates the basic functionality of XDP. It runs multiple times to
test the load/unload. It includes below steps,
1. Check current image supports XDP or not.
2. Install and validate xdpdump.
Priority

1

verify_xdp_synthetic()
Description
It validates the XDP works with Synthetic network.
The test step is the same as verify_xdp_basic, but it run once only.
Priority

2

Requirement

network_interface

verify_xdp_remove_add_vf()
Description
It validates the XDP works with VF hot add/remove from API.
1. Run xdp dump to drop and count packets.
2. Remove VF from API.
3. Run xdp dump to drop and count packets.
5. Add VF back from API.
6. Run xdp dump to drop and count packets.
Priority

3

Requirement

network_interface

class Dpdk
Description
This test suite check DPDK functionality
Platform

Azure, Ready

Area

dpdk

Category

functional

verify_dpdk_send_receive_netvsc()
Description
Tests a basic sender/receiver setup for direct netvsc pmd setup.
Sender sends the packets, receiver receives them.
We check both to make sure the received traffic is within the expected
order-of-magnitude.
Priority

2

Requirement

min_count=2

verify_dpdk_sriov_rescind_failover_send_only()
Description
test sriov failsafe during vf revoke (send only version)
Priority

2

Requirement

network_interface

verify_dpdk_ovs()
Description
Install and run OVS+DPDK functional tests
Priority

3

Requirement

network_interface

verify_dpdk_ring_ping()
Description
This test runs the dpdk ring ping utility from:
to measure the maximum latency for 99.999 percent of packets during
the test run. The maximum should be under 200000 nanoseconds
(.2 milliseconds).
Not dependent on any specific PMD.
Priority

2

Requirement

network_interface

verify_dpdk_send_receive_multi_txrx_queue_netvsc()
Description
Tests a basic sender/receiver setup for default failsafe driver setup.
Sender sends the packets, receiver receives them.
We check both to make sure the received traffic is within the expected
order-of-magnitude.
Priority

2

Requirement

min_count=2

verify_dpdk_build_failsafe()
Description
failsafe (azure default, recommended) version.
This test case checks DPDK can be built and installed correctly.
Prerequisites, accelerated networking must be enabled.
The VM should have at least two network interfaces,
with one interface for management.
Priority

2

Requirement

network_interface

verify_dpdk_multiprocess()
Description
Build and run DPDK multiprocess client/server sample application.
Requires 3 nics since client/server needs two ports + 1 nic for LISA
Priority

3

Requirement

network_interface

verify_dpdk_build_netvsc()
Description
netvsc direct pmd version.
This test case checks DPDK can be built and installed correctly.
Prerequisites, accelerated networking must be enabled.
The VM should have at least two network interfaces,
with one interface for management.
Priority

2

Requirement

network_interface

verify_dpdk_vpp()
Description
verify vpp is able to detect azure network interfaces
1. run fd.io vpp install scripts
2. install vpp from their repositories
3. start vpp service
4. check that azure interfaces are detected by vpp
Priority

2

Requirement

network_interface

verify_dpdk_sriov_rescind_failover_receiver()
Description
test sriov failsafe during vf revoke (receive side)
Priority

2

Requirement

min_count=2

verify_dpdk_nff_go()
Description
Install and run ci test for NFF-Go on ubuntu
Priority

3

Requirement

network_interface

verify_dpdk_send_receive_multi_txrx_queue_failsafe()
Description
Tests a basic sender/receiver setup for default failsafe driver setup.
Sender sends the packets, receiver receives them.
We check both to make sure the received traffic is within the expected
order-of-magnitude.
Priority

2

Requirement

min_count=2

verify_uio_binding()
Description
UIO basic functionality test.
- Bind interface to uio_hv_generic
- check that sysfs entry is created
- unbind
- check that the driver is unloaded.
Priority

2

Requirement

network_interface

verify_dpdk_send_receive_failsafe()
Description
Tests a basic sender/receiver setup for default failsafe driver setup.
Sender sends the packets, receiver receives them.
We check both to make sure the received traffic is within the expected
order-of-magnitude.
Priority

2

Requirement

min_count=2

class DpdkPerformance
Description
This test suite is to validate DPDK performance
Platform

Azure, Ready

Area

dpdk

Category

performance

perf_dpdk_failsafe_pmd_multi_core_huge_vm()
Description
DPDK Performance: failsafe mode, maximal core count, default queue settings
Priority

3

Requirement

min_core_count=48

perf_dpdk_failsafe_pmd_multi_queue_huge_vm()
Description
DPDK Performance: failsafe mode, maximum core count, maximum tx/rx queues
Run on a huge machine
Priority

3

Requirement

min_core_count=48

perf_dpdk_netvsc_pmd_dual_core()
Description
DPDK Performance: direct use of VF, minimal core count, default queues
Priority

3

Requirement

min_core_count=4

perf_dpdk_failsafe_pmd_multi_core()
Description
DPDK Performance: failsafe mode, maximal core count, default queue settings
Priority

3

Requirement

min_core_count=8

perf_dpdk_netvsc_pmd_multi_core_huge_vm()
Description
DPDK Performance: direct use of VF, maximum core count, default queues
Run on a big VM
Priority

3

Requirement

min_core_count=48

perf_dpdk_failsafe_pmd_dual_core()
Description
DPDK Performance: failsafe mode, minimal core count
Priority

3

Requirement

min_core_count=4

perf_dpdk_netvsc_pmd_multi_queue()
Description
DPDK Performance: direct use of VF, maximum core count, maximum tx/rx queues
Priority

3

Requirement

min_core_count=8

perf_dpdk_netvsc_pmd_multi_queue_huge_vm()
Description
DPDK Performance: direct use of VF, maximum core count, maximum tx/rx queues,
Run on a huge machine
Priority

3

Requirement

min_core_count=48

perf_dpdk_failsafe_pmd_multi_queue()
Description
DPDK Performance: failsafe mode, maximum core count, maximum tx/rx queues
Priority

3

Requirement

min_core_count=8

perf_dpdk_netvsc_pmd_multi_core()
Description
DPDK Performance: direct use of VF, maximum core count, default queues
Priority

3

Requirement

min_core_count=8

class Gpu
Description
This test suite runs the gpu test cases.
Platform

Azure, Ready

Area

gpu

Category

functional

verify_gpu_adapter_count()
Description
This test case verifies the gpu adapter count.
Steps:
1. Assert that node supports GPU.
2. If GPU modules are not loaded, install and load the module first.
3. Find the expected gpu count for the node.
4. Validate expected and actual gpu count using lsvmbus output.
5. Validate expected and actual gpu count using lspci output.
6. Validate expected and actual gpu count using gpu vendor commands
example - nvidia-smi
Priority

2

Requirement

supported_features

verify_load_gpu_driver()
Description
This test case verifies if gpu drivers are loaded fine.
Steps:
1. Validate if the VM SKU is supported for GPU.
2. Install LIS drivers if not already installed for Fedora and its
derived distros. Reboot the node
3. Install required gpu drivers on the VM and reboot the node. Validate gpu
drivers can be loaded successfully.
Priority

1

Requirement

supported_features[SerialConsole]

verify_gpu_cuda_with_pytorch()
Description
This test case will run PyTorch to check CUDA driver installed correctly.
1. Install PyTorch.
2. Check GPU count by torch.cuda.device_count()
3. Compare with PCI result
Priority

3

Requirement

supported_features

verify_gpu_rescind_validation()
Description
This test case will
1. Validate disabling GPU devices.
2. Validate enable back the disabled GPU devices.
Priority

2

Requirement

supported_features

class NetworkPerformace
Description
This test suite is to validate linux network performance.
Platform

Azure, Ready

Area

network

Category

performance

perf_tcp_ntttcp_sriov()
Description
This test case uses ntttcp to test sriov tcp network throughput.
Priority

3

Requirement

network_interface

perf_udp_1k_ntttcp_synthetic()
Description
This test case uses ntttcp to test synthetic udp network throughput.
Priority

3

Requirement

network_interface

perf_tcp_latency_sriov()
Description
This test case uses lagscope to test sriov network latency.
Priority

2

Requirement

network_interface

perf_tcp_single_pps_synthetic()
Description
This test case uses sar to test synthetic network PPS (Packets Per Second)
when running netperf with single port.
Priority

3

Requirement

network_interface

perf_udp_iperf_sriov()
Description
This test case uses iperf to test sriov udp network throughput.
Priority

3

Requirement

network_interface

perf_tcp_ntttcp_128_connections_synthetic()
Description
This test case uses ntttcp to test synthetic tcp network throughput for
128 connections.
Priority

2

Requirement

network_interface

perf_udp_iperf_synthetic()
Description
This test case uses iperf to test synthetic udp network throughput.
Priority

3

Requirement

network_interface

perf_tcp_latency_synthetic()
Description
This test case uses lagscope to test synthetic network latency.
Priority

2

Requirement

network_interface

perf_tcp_max_pps_synthetic()
Description
This test case uses sar to test synthetic network PPS (Packets Per Second)
when running netperf with multiple ports.
Priority

3

Requirement

network_interface

perf_tcp_max_pps_sriov()
Description
This test case uses sar to test sriov network PPS (Packets Per Second)
when running netperf with multiple ports.
Priority

3

Requirement

network_interface

perf_tcp_single_pps_sriov()
Description
This test case uses sar to test sriov network PPS (Packets Per Second)
when running netperf with single port.
Priority

3

Requirement

network_interface

perf_tcp_iperf_synthetic()
Description
This test case uses iperf3 to test synthetic tcp network throughput.
Priority

3

Requirement

network_interface

perf_tcp_ntttcp_synthetic()
Description
This test case uses ntttcp to test synthetic tcp network throughput.
Priority

3

Requirement

network_interface

perf_tcp_iperf_sriov()
Description
This test case uses iperf3 to test sriov tcp network throughput.
Priority

3

Requirement

network_interface

perf_udp_1k_ntttcp_sriov()
Description
This test case uses ntttcp to test sriov udp network throughput.
Priority

3

Requirement

network_interface

class KVMPerformance
Description
This test suite is to validate performance of nested VM using FIO tool.
Platform

Azure, Ready

Area

storage

Category

performance

perf_nested_hyperv_ntttcp_different_l1_nat()
Description
This script runs ntttcp test on two nested VMs on different L1 guests
connected with NAT
Priority

3

Requirement

supported_platform_type[AZURE, READY]

perf_nested_kvm_netperf_pps_nat()
Description
This script runs netperf test on two nested VMs on different L1 guests
connected with NAT
Priority

3

Requirement

disk

perf_nested_hyperv_storage_singledisk()
Description
This test case is to validate performance of nested VM in hyper-v
using fio tool with single l1 data disk attached to the l2 VM.
Priority

3

Requirement

disk

perf_nested_hyperv_storage_multidisk()
Description
This test case is to validate performance of nested VM using fio tool with raid0
configuration of 6 l1 data disk attached to the l2 VM.
Priority

3

Requirement

disk

perf_nested_kvm_storage_multidisk()
Description
This test case is to validate performance of nested VM using fio tool with raid0
configuration of 6 l1 data disk attached to the l2 VM.
Priority

3

Requirement

disk

perf_nested_kvm_ntttcp_different_l1_nat()
Description
This script runs ntttcp test on two nested VMs on different L1 guests
connected with NAT
Priority

3

Requirement

disk

perf_nested_kvm_storage_singledisk()
Description
This test case is to validate performance of nested VM using fio tool
with single l1 data disk attached to the l2 VM.
Priority

3

Requirement

disk

perf_nested_kvm_ntttcp_private_bridge()
Description
This test case runs ntttcp test on two nested VMs on same L1 guest
connected with private bridge
Priority

3

Requirement

disk

class StoragePerformance
Description
This test suite is to validate premium SSD data disks performance of Linux VM using
fio tool.
Platform

Azure, Ready

Area

storage

Category

performance

perf_storage_over_nfs_synthetic_tcp_4k()
Description
This test case uses fio to test performance of nfs server over TCP with
VM’s initialized with synthetic network interface.
Priority

3

Requirement

network_interface

perf_premium_datadisks_io()
Description
This test case uses fio to test vm with 24 data disks.
Priority

3

Requirement

disk

perf_premium_datadisks_4k()
Description
This test case uses fio to test data disk performance with 4K block size.
Priority

3

Requirement

disk

perf_storage_over_nfs_sriov_udp_4k()
Description
This test case uses fio to test performance of nfs server over UDP with
VM’s initialized with SRIOV network interface.
Priority

3

Requirement

network_interface

perf_premium_datadisks_1024k()
Description
This test case uses fio to test data disk performance using 1024K block size.
Priority

3

Requirement

disk

perf_storage_over_nfs_sriov_tcp_4k()
Description
This test case uses fio to test performance of nfs server over TCP with
VM’s initialized with SRIOV network interface.
Priority

3

Requirement

network_interface

perf_storage_over_nfs_synthetic_udp_4k()
Description
This test case uses fio to test performance of nfs server over UDP with
VM’s initialized with synthetic network interface.
Priority

3

Requirement

network_interface

class NvmePerformace
Description
This test suite is to validate NVMe disk performance of Linux VM using fio tool.
Platform

Azure, Ready

Area

nvme

Category

performance

perf_nvme()
Description
This test case uses fio to test NVMe disk performance.
Priority

3

Requirement

supported_features

class Docker
Description
This test suite runs the docker test cases for java, python, dotnet 3.1
, dotnet5.0, and wordpress.
Platform

Azure, Ready

Area

docker

Category

functional

verify_docker_python_app()
Description
This test case creates and runs a python app using docker
Steps:
1. Install Docker
2. Copy python dockerfile and program file to node
3. Create docker image and run docker container
4. Check results of docker run against python string identifier
Priority

3

verify_docker_dotnet50_app()
Description
This test case creates and runs a dotnet app using docker
Steps:
1. Install Docker
2. Copy dotnet dockerfile into node
3. Create docker image and run docker container
4. Check results of docker run against dotnet string identifier
Priority

2

verify_docker_compose_wordpress_app()
Description
This test case uses docker-compose to create and run a wordpress mysql app
Steps:
1. Install Docker and Docker-Compose on node
2. Copy docker-compose.yml into node
3. Start docker container with docker-compose
4. Run ps in the docker container and capture output
5. Check that “apache2” can be found in captured output
Priority

3

verify_docker_dotnet31_app()
Description
This test case creates and runs a dotnet app using docker
Steps:
1. Install Docker
2. Copy dotnet dockerfile into node
3. Create docker image and run docker container
4. Check results of docker run against dotnet string identifier
Priority

1

verify_docker_java_app()
Description
This test case creates and runs a java app using docker
Steps:
1. Install Docker
2. Copy java dockerfile and program file to node
3. Create docker image and run docker container
4. Check results of docker run against java string identifier
Priority

2

class Power
Description
This test suite is to test hibernation in guest VM.
Platform

Azure, Ready

Area

power

Category

functional

verify_hibernation_time_sync()
Description
This case is to verify vm time sync working after hibernation.
Steps,
1. Reset time using hwclock as 1 year after current date.
2. Hibernate and resume vm.
3. Check vm time sync correctly.
Priority

3

Requirement

supported_features

verify_hibernation_with_network_workload()
Description
This case is to verify hibernation with network workload.
Steps,
1. Run iperf3 network benchmark, make sure no issues.
2. Hibernate and resume vm.
3. Run iperf3 network benchmark, make sure no issues.
Priority

3

Requirement

supported_features

verify_hibernation_synthetic_network()
Description
This case is to verify vm hibernation with synthetic network.
Steps,
1. Install HibernationSetup tool to prepare prerequisite for vm
hibernation.
2. Get nics info before hibernation.
3. Hibernate vm.
4. Check vm is inaccessible.
5. Resume vm by starting vm.
6. Check vm hibernation successfully by checking keywords in dmesg.
6. Get nics info after hibernation.
7. Fail the case if nics count and info changes after vm resume.
Priority

3

Requirement

supported_features

verify_hibernation_sriov_network()
Description
This case is to verify vm hibernation with sriov network.
It has the same steps with verify_hibernation_synthetic_network.
Priority

3

Requirement

supported_features

verify_hibernation_with_memory_workload()
Description
This case is to verify hibernation with memory workload.
Steps,
1. Run stress-ng benchmark, make sure no issues.
2. Hibernate and resume vm.
3. Run stress-ng benchmark, make sure no issues.
Priority

3

Requirement

supported_features

verify_hibernation_with_storage_workload()
Description
This case is to verify hibernation with storage workload.
Steps,
1. Run fio benchmark, make sure no issues.
2. Hibernate and resume vm.
3. Run fio benchmark, make sure no issues.
Priority

3

Requirement

supported_features

class PowerStress
Description
This test suite is to test hibernation in guest vm under stress.
Platform

Azure, Ready

Area

power

Category

stress

verify_stress_hibernation()
Description
This case is to verify vm hibernation in a loop.
Priority

3

Requirement

supported_features[Hibernation]

class Synthetic
Description
This test suite uses to verify synthetic network functionality.
Platform

Azure, Ready

Area

network

Category

functional

synthetic_provision_with_max_nics_validation()
Description
This case verify VM works well when provison with max (8) synthetic nics.
Steps,
1. Provision VM with max network interfaces with synthetic network.
2. Check each nic has an ip address.
Priority

2

Requirement

network_interface

synthetic_add_max_nics_one_by_one_after_provision_validation()
Description
This case verify VM works well after attaching 7 extra synthetic nics
one by one.
Steps,
1. Provision VM with 1 network interface with synthetic network.
2. Add 7 extra network interfaces one by one.
3. Check each nic has an ip address.
Priority

2

Requirement

network_interface

synthetic_add_max_nics_one_time_after_provision_validation()
Description
This case verify VM works well after attaching 7 extra synthetic nics
in one time.
Steps,
1. Provision VM with 1 network interface with synthetic network.
2. Add 7 extra network interfaces in one time.
3. Check each nic has an ip address.
Priority

2

Requirement

network_interface

class NetInterface
Description
This test suite validates basic functionalities of Network Interfaces.
Platform

Azure, Ready

Area

network

Category

functional

validate_netvsc_reload()
Description
This test case verifies if synthetic network module - netvsc
can be reloaded gracefully when done multiple times.
Steps:
1. Validate netvsc isn’t built-in already. If it is then skip the test.
2. Unload and load netvsc module multiple times in a loop.
Priority

1

Requirement

network_interface

validate_set_static_mac()
Description
This test case verifies if the second network interface can be brought up
after setting static MAC address.
Steps:
1. Validate the second nic has IP address.
2. Bring down the second nic.
3. Set a random MAC address to the second nic.
4. Bring up the second nic.
Priority

3

Requirement

network_interface

class Sriov
Description
This test suite uses to verify accelerated network functionality.
Platform

Azure, Ready

Area

sriov

Category

functional

verify_sriov_max_vf_connection_max_cpu()
Description
This case needs 2 nodes, 8 nics and 64 Vcpus. And it verifies module of sriov
network interface is loaded and each synthetic nic is paired with one VF, and
check rx statistics of source and tx statistics of dest increase after send 200
Mb file from source to dest.
Steps,
1. Check VF of synthetic nic is paired.
2. Check module of sriov network device is loaded.
3. Check VF counts listed from lspci is expected.
4. Setup SSH connection between source and dest with key authentication.
5. Ping the dest IP from the source machine to check connectivity.
6. Generate 200Mb file, copy from source to dest.
7. Check rx statistics of source VF and tx statistics of dest VF is increased.
Priority

2

Requirement

network_interface

verify_sriov_max_vf_connection()
Description
This case needs 2 nodes and 8 nics. And it verifies module of sriov network
interface is loaded and each synthetic nic is paired with one VF, and check
rx statistics of source and tx statistics of dest increase after send 200 Mb
file from source to dest.
Steps,
1. Check VF of synthetic nic is paired.
2. Check module of sriov network device is loaded.
3. Check VF counts listed from lspci is expected.
4. Setup SSH connection between source and dest with key authentication.
5. Ping the dest IP from the source machine to check connectivity.
6. Generate 200Mb file, copy from source to dest.
7. Check rx statistics of source VF and tx statistics of dest VF is increased.
Priority

2

Requirement

network_interface

sriov_basic_validation()
Description
This case verifies module of sriov network interface is loaded and each
synthetic nic is paired with one VF.
Steps,
1. Check VF of synthetic nic is paired.
2. Check module of sriov network device is loaded.
3. Check VF counts listed from lspci is expected.
Priority

1

Requirement

network_interface

verify_sriov_interrupts_change()
Description
This case is to verify interrupts count increased after network traffic
went through the VF, if CPU is less than 8, it can’t verify the interrupts
spread to CPU evenly, when CPU is more than 16, the traffic is too light to
make sure interrupts distribute to every CPU.
Steps,
1. Start iperf3 on server node.
2. Get initial interrupts sum per irq and cpu number on client node.
3. Start iperf3 for 30 seconds with 128 threads on client node.
4. Get final interrupts sum per irq number on client node.
5. Compare interrupts changes, expected to see interrupts increased.
6. Get final interrupts sum per cpu on client node.
7. Compare interrupts changes, expected to see interrupts increased.
Priority

2

Requirement

node

verify_sriov_add_max_nics()
Description
This case verify VM works well after attached the max sriov nics after
provision.
Steps,
1. Attach 7 extra sriov nic into the VM.
2. Do the basic sriov testing.
Priority

2

Requirement

network_interface

verify_sriov_disable_enable()
Description
This case verify VM works well after disable and enable accelerated network in
network interface through sdk.
Steps,
1. Do the basic sriov check.
2. Set enable_accelerated_networking as False to disable sriov.
3. Set enable_accelerated_networking as True to enable sriov.
4. Do the basic sriov check.
Priority

2

Requirement

supported_platform_type[AZURE, READY]

verify_sriov_ethtool_offload_setting()
Description
This case verify below two kernel patches.
1. hv_netvsc: Sync offloading features to VF NIC
2. hv_netvsc: Allow scatter-gather feature to be tunable
Steps,
1. Change scatter-gather feature on synthetic nic,
verify the the feature status sync to the VF dynamically.
2. Disable and enable sriov,
check the scatter-gather feature status keep consistent in VF.
Priority

2

Requirement

network_interface

verify_sriov_disable_enable_on_guest()
Description
This case verify VM works well after down the VF nic and up VF nic inside VM.
Steps,
1. Do the basic sriov check.
2. Do network connection test with bring down the VF nic.
3. After copy 200Mb file from source to desc.
4. Check rx statistics of source synthetic nic and tx statistics of dest
synthetic nic is increased.
5. Bring up VF nic.
Priority

2

Requirement

network_interface

verify_sriov_disable_enable_pci()
Description
This case verify VM works well after disable and enable PCI device inside VM.
Steps,
1. Disable sriov PCI device inside the VM.
2. Enable sriov PCI device inside the VM.
3. Do the basic sriov check.
4. Do VF connection test.
Priority

2

Requirement

network_interface

verify_sriov_single_vf_connection()
Description
This case verifies module of sriov network interface is loaded and
each synthetic nic is paired with one VF, and check rx statistics of source
and tx statistics of dest increase after send 200 Mb file from source to dest.
Steps,
1. Check VF of synthetic nic is paired.
2. Check module of sriov network device is loaded.
3. Check VF counts listed from lspci is expected.
4. Setup SSH connection between source and dest with key authentication.
5. Ping the dest IP from the source machine to check connectivity.
6. Generate 200Mb file, copy from source to dest.
7. Check rx statistics of source VF and tx statistics of dest VF is increased.
Priority

1

Requirement

network_interface

verify_sriov_provision_with_max_nics()
Description
This case verify VM works well when provisioning with max (8) sriov nics.
Steps,
1. Provision VM with max network interfaces with enabling accelerated network.
2. Do the basic sriov testing.
Priority

2

Requirement

network_interface

verify_sriov_single_vf_connection_max_cpu()
Description
This case needs 2 nodes and 64 Vcpus. And it verifies module of sriov network
interface is loaded and each synthetic nic is paired with one VF, and check
rx statistics of source and tx statistics of dest increase after send 200 Mb
file from source to dest.
Steps,
1. Check VF of synthetic nic is paired.
2. Check module of sriov network device is loaded.
3. Check VF counts listed from lspci is expected.
4. Setup SSH connection between source and dest with key authentication.
5. Ping the dest IP from the source machine to check connectivity.
6. Generate 200Mb file, copy from source to dest.
7. Check rx statistics of source VF and tx statistics of dest VF is increased.
Priority

2

Requirement

network_interface

verify_sriov_reload_modules()
Description
This case verify VM works well during remove and load sriov modules.
Steps,
1. Provision VM with max network interfaces with enabling accelerated network.
2. Do the basic sriov testing.
3. Remove sriov module, check network traffic through synthetic nic.
4. Load sriov module, check network traffic through VF.
Priority

2

Requirement

network_interface

class Stress
Description
This test suite uses to verify accelerated network functionality under stress.
Platform

Azure, Ready

Area

sriov

Category

stress

verify_stress_sriov_iperf()
Description
This case is to check whether the network connectivity is lost after running
iperf3 for 30 mins.
Steps,
1. Start iperf3 on server node.
2. Start iperf3 for 30 minutes on client node.
3. Do VF connection test.
Priority

4

Requirement

node

class NetworkSettings
Description
This test suite runs the ethtool related network test cases.
Platform

Azure, Ready

Area

network

Category

functional

validate_device_enabled_features()
Description
This test case verifies required device features are enabled.
Steps:
1. Get the device’s enabled features.
2. Validate below features are in the list of enabled features-
rx-checksumming
tx-checksumming
tcp-segmentation-offload
scatter-gather
Priority

1

validate_ringbuffer_settings_change()
Description
This test case verifies if ring buffer settings can be changed with ethtool.
Steps:
1. Get the current ring buffer settings.
2. Change the rx and tx value to new_values using ethtool.
3. Get the settings again and validate the current rx and tx
values are equal to the new_values assigned.
4. Revert back the rx and tx value to their original values.
Priority

1

validate_device_statistics()
Description
This test case verifies that device statistics can be captured.
Steps:
1. Get all the device’s statistics.
2. Validate device statistics lists per queue statistics as well.
3. Run traffic using iperf3 and validate traffic is evenly distributed
among different VMBUS channel queues.
Priority

2

Requirement

node

validate_device_msg_level_change()
Description
This test case verifies whether setting/unsetting device’s
message level flag takes into affect.
Steps:
1. Verify Get/Set message level supported on kernel version.
2. Get all the device’s message level number and name setting.
2. Depending on current setting, set/unset a message flag by number
and name.
3. Validate changing the message level flag setting.
4. Revert back the setting to original value.
Priority

2

validate_device_rx_hash_level_change()
Description
This test case verifies whether changing device’s RX hash level
for tcp and udp takes into affect.
Steps:
Note: Same steps are used for both TCP and UDP.
1. Get all the device’s RX hash level status.
2. Depending on current setting, change to enabled/disabled.
3. Validate changing the hash level setting.
4. Revert back the settings to original values.
Priority

2

validate_device_gro_lro_settings_change()
Description
This test case verifies changing device’s GRO and LRO setting takes
into affect.
Steps:
1. Get all the device’s generic-receive-offload and large-receive-offload
settings.
2. If both GRO and LRO settings are “[fixed]” then skip testing specific
device.
3. Try flipping the GRO and LRO settings and validate it takes affect.
4. Revert back the settings to original values.
Priority

1

validate_device_channels_change()
Description
This test case verifies changing device channels count with ethtool.
Steps:
1. Get the current device channels info.
2 a. Keep Changing the channel count from min to max value using ethtool.
b. Get the channel count info and validate the channel count
value is equal to the new value assigned.
3. Revert back the channel count to its original value.
Priority

1

validate_device_rss_hash_key_change()
Description
This test case verifies changing device’s RSS hash key takes
into affect.
Steps:
1. Skip the test if the kernel version is any less than LTS 5.
2. Get all the device’s RSS hash key values.
3. Swap the last 2 characters of original hash key to make a new hash key.
4. Validate changing the hash key setting using the new hash key.
5. Revert back the settings to original values.
Priority

2

class CPUSuite
Description
This test suite is used to run cpu related tests.
Platform

Azure, Ready

Area

cpu

Category

functional

verify_cpu_offline_network_workload()
Description
This test will check cpu hotplug with network workload.
The cpu hotplug steps are same as verify_cpu_hot_plug test case.
Priority

4

Requirement

min_count=2

verify_cpu_offlined_channel_add()
Description
This test will check that the added channels to synthetic network
adapter do not handle interrupts on offlined cpu.
Steps:
1. Get list of offlined CPUs.
2. Add channels to synthetic network adapter.
3. Verify that the channels were added to synthetic network adapter.
4. Verify that the added channels do not handle interrupts on offlined cpu.
Priority

4

verify_cpu_offline_storage_workload()
Description
This test will check cpu hotplug with storage workload.
The cpu hotplug steps are same as verify_cpu_hot_plug test case.
Priority

4

Requirement

disk

verify_cpu_hot_plug()
Description
This test will check cpu hotplug.
Steps :
1. skip test case when kernel doesn’t support cpu hotplug.
2. set all vmbus channels target to cpu 0.
when kernel version >= 5.8 and vmbus version >= 4.1, code supports changing
vmbus channels target cpu, by setting the cpu number to the file
/sys/bus/vmbus/devices/<device id>/channels/<channel id>/cpu.
then all cpus except for cpu 0 are in idle state.
2.1 save the raw cpu number of each channel for restoring after testing.
2.2 set all vmbus channel interrupts go into cpu 0.
3. collect idle cpu which can be used for hotplug.
if the kernel supports step 2, now in used cpu is 0.
exclude the in used cpu from all cpu list to get idle cpu set which can be
offline and online.
if the kernel doesn’t step 2,
the idle cpu is quite rely on the cpu usage at that time.
4. skip testing when there is no idle cpu can be set offline and online.
5. set idle cpu offline then back to online.
6. restore the cpu vmbus channel target cpu back to the original state.
Priority

3

Requirement

min_core_count=32

class CPUStressSuite
Description
This test suite is used to run cpu related tests under stress.
Platform

Azure, Ready

Area

cpu

Category

stress

verify_cpu_hot_plug_stress()
Description
This test will check cpu hotplug under stress.
Detailed steps please refer case verify_cpu_hot_plug.
Priority

3

Requirement

min_core_count=32

class StorageTest
Description
This test suite is to validate storage function in Linux VM.
Platform

Azure, Ready

Area

storage

Category

functional

verify_disk_with_nobarrier()
Description
This test case is to
1. Make raid0 based on StandardHDDLRS disks.
2. Mount raid0 with nobarrier options.
Priority

3

Requirement

disk

class Nested
Description
This test suite is used to run nested vm related tests.
Platform

Azure, Ready

Area

nested

Category

functional

verify_nested_kvm_basic()
Description
This test case will run basic tests on provisioned L2 vm.
Steps:
1. Create L2 VM with Qemu.
2. Verify that files can be copied from L1 VM to L2 VM.
3. Verify that files from internet can be downloaded to L2 VM.
Priority

1

Requirement

disk

class Fips
Description
Tests the functionality of FIPS enable
Platform

Azure, Ready

Area

security

Category

functional

verify_fips_enable()
Description
This test case will
1. Check whether FIPS can be enaled on the VM
2. Enable FIPS
3. Restart the VM for the changes to take effect
4. Verify that FIPS was enabled propperly
Priority

3

class Xfstesting
Description
This test suite is to validate different types of data disk on Linux VM
using xfstests.
Platform

Azure, Ready

Area

storage

Category

community

xfstesting_azure_file_share_validation()
Description
This test case will run cifs xfstests testing against
azure file share.
Priority

3

Requirement

supported_platform_type[AZURE]

xfstesting_generic_standard_datadisk_validation()
Description
This test case will run generic xfstests testing against
standard data disk with xfs type system.
Priority

3

Requirement

disk

xfstesting_generic_nvme_datadisk_validation()
Description
This test case will run generic xfstests testing against
nvme data disk with xfs type system.
Priority

3

Requirement

supported_features[Nvme]

xfstesting_xfs_nvme_datadisk_validation()
Description
This test case will run xfs xfstests testing against
nvme data disk with xfs type system.
Priority

3

Requirement

supported_features[Nvme]

xfstesting_btrfs_nvme_datadisk_validation()
Description
This test case will run btrfs xfstests testing against
nvme data disk with btrfs type system.
Priority

3

Requirement

supported_features[Nvme]

xfstesting_ext4_standard_datadisk_validation()
Description
This test case will run ext4 xfstests testing against
standard data disk with ext4 type system.
Priority

3

Requirement

disk

xfstesting_btrfs_standard_datadisk_validation()
Description
This test case will run btrfs xfstests testing against
standard data disk with btrfs type system.
Priority

3

Requirement

disk

xfstesting_xfs_standard_datadisk_validation()
Description
This test case will run xfs xfstests testing against
standard data disk with xfs type system.
Priority

3

Requirement

disk

xfstesting_ext4_nvme_datadisk_validation()
Description
This test case will run ext4 xfstests testing against
nvme data disk with ext4 type system.
Priority

3

Requirement

supported_features[Nvme]

class Drm
Description
This test suite uses to verify drm driver sanity.
Platform

Azure, Ready

Area

drm

Category

functional

verify_drm_driver()
Description
This case is to check whether the hyperv_drm driver registered successfully.
Once driver is registered successfully it should appear in lsmod output.
Steps,
1. lsmod
2. Check if hyperv_drm exist in the list.
Priority

2

verify_dri_node()
Description
This case is to check whether the dri node is populated correctly.
If hyperv_drm driver is bind correctly it should populate dri node.
This dri node can be find at following sysfs entry : /sys/kernel/debug/dri/0.
The dri node name (/sys/kernel/debug/dri/0/name) should contain hyperv_drm.
Step,
1. Cat /sys/kernel/debug/dri/0/name.
2. Verify it contains hyperv_drm string in it.
Priority

2

class TvmTest
Description
This test suite is to validate secureboot in Linux VM.
Platform

Azure, Ready

Area

tvm

Category

functional

verify_measuredboot_compatibility()
Description
This case tests the image is compatible with Measured Boot.
Steps:
1. Enable repository azurecore from https://packages.microsoft.com/repos.
2. Install package azure-compatscanner.
3. Check image Measured Boot compatibility from output of mbinfo.
Priority

2

Requirement

supported_features

verify_secureboot_compatibility()
Description
This case tests the image is compatible with Secure Boot.
Steps:
1. Enable repository azurecore from https://packages.microsoft.com/repos.
2. Install package azure-security.
3. Check image Secure Boot compatibility from output of sbinfo.
Priority

2

Requirement

supported_features

class Nvme
Description
This test suite is to validate NVMe disk on Linux VM.
Platform

Azure, Ready

Area

nvme

Category

functional

nvme_max_disk_validation()
Description
This case runs nvme_basic_validation test against 10 NVMe disks.
The test steps are same as nvme_basic_validation.
Priority

2

Requirement

supported_features

nvme_function_validation()
Description
This test case will do following things for each NVMe device.
1. Get the number of errors from nvme-cli before operations.
2. Create a partition, filesystem and mount it.
3. Create a txt file on the partition, content is ‘TestContent’.
4. Create a file ‘data’ on the partition, get the md5sum value.
5. Umount and remount the partition.
6. Get the txt file content, compare the value.
7. Compare the number of errors from nvme-cli after operations.
Priority

2

Requirement

supported_features[Nvme]

nvme_manage_ns_validation()
Description
This test case will run commands 2-5, the commands are expected fail or not
based on the capabilities of the device.
1. Use nvme id-ctrl device command list the capabilities of the device.
1.1 When ‘Format NVM Supported’ shown up in output of ‘nvme id-ctrl device’,
then nvme disk can be format, otherwise, it can’t be format.
1.2 When ‘NS Management and Attachment Supported’ shown up in output of
‘nvme id-ctrl device’, nvme namespace can be created, deleted and detached,
otherwise it can’t be managed.
2. nvme format namespace - format a namespace.
3. nvme create-ns namespace - create a namespace.
4. nvme delete-ns -n 1 namespace - delete a namespace.
5. nvme detach-ns -n 1 namespace - detach a namespace.
Priority

3

Requirement

supported_features[Nvme]

nvme_blkdiscard_validation()
Description
This test case will
1. Create a partition, xfs filesystem and mount it.
2. Umount the mountpoint.
3. Run blkdiscard command on the partition.
4. Remount command should fail after run blkdiscard command.
Priority

3

Requirement

supported_features[Nvme]

nvme_rescind_validation()
Description
This test case will
1. Disable NVME devices.
2. Enable NVME device.
Priority

2

Requirement

supported_features[Nvme]

nvme_fstrim_validation()
Description
This test case will
1. Create a partition, xfs filesystem and mount it.
2. Check how much the mountpoint is trimmed before operations.
3. Create a 300 gb file ‘data’ using dd command in the partition.
4. Check how much the mountpoint is trimmed after creating the file.
5. Delete the file ‘data’.
6. Check how much the mountpoint is trimmed after deleting the file,
and compare the final fstrim status with initial fstrim status.
Priority

3

Requirement

supported_features[Nvme]

nvme_sriov_rescind_validation()
Description
This test case does following steps to verify VM working normally during
disable and enable nvme and sriov devices.
1. Disable PCI devices.
2. Enable PCI devices.
3. Get PCI devices slots.
4. Check PCI devices are back after rescan.
Priority

2

Requirement

supported_features[Nvme]

nvme_basic_validation()
Description
This test case will
1. Get nvme devices and nvme namespaces from /dev/ folder,
compare the count of nvme namespaces and nvme devices.
2. Compare the count of nvme namespaces return from nvme list
and list nvme namespaces under /dev/.
3. Compare nvme devices count return from lspci
and list nvme devices under /dev/.
4. Azure platform only, nvme devices count should equal to
actual vCPU count / 8.
Priority

1

Requirement

supported_features[Nvme]

class Lis
Description
This test suite contains tests that are dependent on an LIS driver
Platform

Azure, Ready

Area

lis

Category

functional

verify_lis_driver_version()
Description
Downloads header files based on the LIS version and compares the installed
LIS version to the expected one which is found in the header files.
Steps:
1. Check for RPM
2. Capture installed LIS version on the node
3. For each rhel version (5,6,7), it downloads the header file and compares
the LIS version in the header file with the LIS version installed
Priority

1

verify_lis_preinstall_disk_size_positive()
Description
This test case is to verify LIS RPM installation script to check the disk
space before proceeding to LIS install.
This avoids half / corrupted installations.
Steps:
1. Test leaves “bare minimum” size avaialble for LIS install and checks if
LIS installation is successful.
Priority

2

verify_lis_preinstall_disk_size_negative()
Description
This test case is to verify LIS RPM installation script to check the disk
space before proceeding to LIS install.
This avoids half / corrupted installations.
Steps:
1. Test leaves “non installable” size on disk and checks if ./install.sh
script skips the installation of not.
Priority

2

class KdumpCrash
Description
This test suite is used to verify if kernel crash dmup is effect, which is judged
through vmcore file is generated after triggering kdump by sysrq.
It has 7 test cases. They verfy if kdump is effect when:
1. VM has 1 cpu
2. VM has 2-8 cpus and trigger kdump on cpu 1
3. VM has 33-192 cpus and trigger kdump on cpu 32
4. VM has 193-415 cpus and trigger kdump on cpu 192
5. VM has more than 415 cpus and trigger kdump on cpu 415
6. crashkernel is set “auto”
7. crashkernel is set “auto” and VM has more than 2T memory
Platform

Azure, Ready

Area

kdump

Category

functional

kdumpcrash_validate_on_cpu415()
Description
This test case verfies if the kdump is effect when VM has more than 415 cpus,
and trigger kdump on the 416th cpu(cpu415), which is designed by a known issue.
The test steps are same as kdumpcrash_validate_single_core.
Priority

2

Requirement

node

kdumpcrash_validate_large_memory_auto_size()
Description
This test case verfies if the kdump is effect when crashkernel is set auto and
the memory is more than 2T. With the crashkernel=auto parameter, system will
reserved a suitable size memory for crash kernel. We want to see if the
crashkernel=auto can also handle this scenario when the system memory is large.
The test steps are same as kdumpcrash_validate_single_core.
Priority

2

Requirement

node

kdumpcrash_validate_on_random_cpu()
Description
This test case verfies if the kdump is effect when VM has any cores, and
trigger kdump on the random cpu.
The test steps are same as kdumpcrash_validate_single_core.
Priority

2

kdumpcrash_validate_single_core()
Description
This test case verfies if kdump is effect when VM has 1 cpu.
VM need 2G memory at least to make sure it has enough memory to load crash
kernel.
Steps:
1. Check if vmbus version and kernel configurations support for crash dump.
2. Specify the memory reserved for crash kernel in kernel cmdline, setting the
“crashkernel” option to the required value.
a. Modify the grub config file to add crashkrenel option or change the
value to the required one. (For Redhat 8, no need to modify grub config
file. It can specify crashkernel by using grubby command directly)
b. Update grub config
4. If needed, config the dump path.
3. Reboot system to make kdump effect.
4. Check if the crash kernel is loaded.
a. Check if kernel cmdline has crashkernel option and the value is expected
b. Check if /sys/kernel/kexec_crash_loaded file exists and the value is ‘1’
c. Check if /proc/iomem is reserved memory for crash kernel
5. Trigger kdump through ‘echo c > /proc/sysrq-trigger’, or trigger on
specified CPU by using command “taskset -c”.
6. Check if vmcore is generated under the dump path we configure after system
boot up.
Priority

1

Requirement

node

kdumpcrash_validate_auto_size()
Description
This test case verfies if the kdump is effect when crashkernel is set auto.
The test steps are same as kdumpcrash_validate_single_core.
Priority

2

kdumpcrash_validate_on_cpu32()
Description
This test case verfies if the kdump is effect when VM has 33~192 cpus and
trigger kdump on the 33th cpu(cpu32), which is designed by a known issue.
The test steps are same as kdumpcrash_validate_single_core.
Priority

2

Requirement

node

kdumpcrash_validate_on_cpu192()
Description
This test case verfies if the kdump is effect when VM has 193~415 cpus, and
trigger kdump on the 193th cpu(cpu192), which is designed by a known issue.
The test steps are same as kdumpcrash_validate_single_core.
Priority

2

Requirement

node

kdumpcrash_validate_smp()
Description
This test case verfies if the kdump is effect when VM has 2~8 cpus, and
trigger kdump on the second cpu(cpu1), which is designed by a known issue.
The test steps are same as kdumpcrash_validate_single_core.
Priority

2

Requirement

node