Test Specification

This file lists all test cases’ specifications.

class Dpdk
Description
This test suite check DPDK functionality
Platform

Azure, Ready

Area

dpdk

Category

functional

verify_dpdk_send_receive()
Description
Tests a basic sender/receiver setup.
Sender sends the packets, receiver receives them.
We check both to make sure the received traffic is within the expected
order-of-magnitude.
Priority

2

Requirement

min_count=2

verify_dpdk_ring_ping()
Description
This test runs the dpdk ring ping utility from:
to measure the maximum latency for 99.999 percent of packets during
the test run. The maximum should be under 200000 nanoseconds
(.2 milliseconds).
Priority

2

Requirement

network_interface

verify_dpdk_build()
Description
This test case checks DPDK can be built and installed correctly.
Prerequisites, accelerated networking must be enabled.
The VM should have at least two network interfaces,
with one interface for management.
Priority

2

Requirement

network_interface

class KdumpCrash
Description
This test suite is used to verify if kernel crash dmup is effect, which is judged
through vmcore file is generated after triggering kdump by sysrq.
It has 7 test cases. They verfy if kdump is effect when:
1. VM has 1 cpu
2. VM has 2-8 cpus and trigger kdump on cpu 1
3. VM has 33-192 cpus and trigger kdump on cpu 32
4. VM has 193-415 cpus and trigger kdump on cpu 192
5. VM has more than 415 cpus and trigger kdump on cpu 415
6. crashkernel is set “auto”
7. crashkernel is set “auto” and VM has more than 2T memory
Platform

Azure, Ready

Area

kdump

Category

functional

kdumpcrash_validate_auto_size()
Description
This test case verfies if the kdump is effect when crashkernel is set auto.
The test steps are same as kdumpcrash_validate_single_core.
Priority

2

kdumpcrash_validate_on_cpu415()
Description
This test case verfies if the kdump is effect when VM has more than 415 cpus,
and trigger kdump on the 416th cpu(cpu415), which is designed by a known issue.
The test steps are same as kdumpcrash_validate_single_core.
Priority

2

Requirement

node

kdumpcrash_validate_on_cpu32()
Description
This test case verfies if the kdump is effect when VM has 33~192 cpus and
trigger kdump on the 33th cpu(cpu32), which is designed by a known issue.
The test steps are same as kdumpcrash_validate_single_core.
Priority

2

Requirement

node

kdumpcrash_validate_single_core()
Description
This test case verfies if kdump is effect when VM has 1 cpu.
VM need 2G memory at least to make sure it has enough memory to load crash
kernel.
Steps:
1. Check if vmbus version and kernel configurations support for crash dump.
2. Specify the memory reserved for crash kernel in kernel cmdline, setting the
“crashkernel” option to the required value.
a. Modify the grub config file to add crashkrenel option or change the
value to the required one. (For Redhat 8, no need to modify grub config
file. It can specify crashkernel by using grubby command directly)
b. Update grub config
4. If needed, config the dump path.
3. Reboot system to make kdump effect.
4. Check if the crash kernel is loaded.
a. Check if kernel cmdline has crashkernel option and the value is expected
b. Check if /sys/kernel/kexec_crash_loaded file exists and the value is ‘1’
c. Check if /proc/iomem is reserved memory for crash kernel
5. Trigger kdump through ‘echo c > /proc/sysrq-trigger’, or trigger on
specified CPU by using command “taskset -c”.
6. Check if vmcore is generated under the dump path we configure after system
boot up.
Priority

1

Requirement

node

kdumpcrash_validate_smp()
Description
This test case verfies if the kdump is effect when VM has 2~8 cpus, and
trigger kdump on the second cpu(cpu1), which is designed by a known issue.
The test steps are same as kdumpcrash_validate_single_core.
Priority

2

Requirement

node

kdumpcrash_validate_large_memory_auto_size()
Description
This test case verfies if the kdump is effect when crashkernel is set auto and
the memory is more than 2T. With the crashkernel=auto parameter, system will
reserved a suitable size memory for crash kernel. We want to see if the
crashkernel=auto can also handle this scenario when the system memory is large.
The test steps are same as kdumpcrash_validate_single_core.
Priority

2

Requirement

node

kdumpcrash_validate_on_cpu192()
Description
This test case verfies if the kdump is effect when VM has 193~415 cpus, and
trigger kdump on the 193th cpu(cpu192), which is designed by a known issue.
The test steps are same as kdumpcrash_validate_single_core.
Priority

2

Requirement

node

class LsVmBus
Description
This test suite is used to check vmbus devices and their associated vmbus channels.
Platform

Azure, Ready

Area

core

Category

functional

lsvmbus_count_devices_channels()
Description
This test case will
1. Check expected vmbus device names presented in the lsvmbus output.
- Operating system shutdown
- Time Synchronization
- Heartbeat
- Data Exchange
- Synthetic mouse
- Synthetic keyboard
- Synthetic network adapter
- Synthetic SCSI Controller
- Synthetic IDE Controller (gen1 only)
2. Check that each netvsc and storvsc SCSI device have correct number of vmbus
channels created and associated.
2.1 Check expected channel count of each netvsc is min (num of vcpu, 8).
2.2 Check expected channel count of each storvsc SCSI device is min (num of
vcpu/4, 64).
2.2.1 Caculate channel count of each storvsc SCSI device.
2.2.2 Cap of channel count of each storvsc SCSI device,
it is decided by host storage VSP driver.
Priority

1

class Provisioning
Description
This test suite uses to verify if an environment can be provisioned correct or not.
- The basic smoke test can run on all images to determinate if a image can boot and
reboot.
- Other provisioning tests verify if an environment can be provisioned with special
hardware configurations.
Platform

Azure, Ready

Area

provisioning

Category

functional

verify_deployment_provision_standard_ssd_disk()
Description
This case runs smoke test on a node provisioned with standard ssd disk.
The test steps are same as smoke_test.
Priority

1

Requirement

supported_features[SerialConsole]

verify_deployment_provision_sriov()
Description
This case runs smoke test on a node provisioned with sriov.
The test steps are same as smoke_test.
Priority

1

Requirement

supported_features[SerialConsole]

verify_deployment_provision_ephemeral_managed_disk()
Description
This case runs smoke test on a node provisioned with ephemeral disk.
The test steps are same as smoke_test.
Priority

1

Requirement

supported_features[SerialConsole]

verify_deployment_provision_premium_disk()
Description
This case runs smoke test on a node provisioned with premium disk.
The test steps are same as smoke_test.
Priority

1

Requirement

supported_features[SerialConsole]

verify_reboot_in_platform()
Description
This case runs smoke test on a node provisioned.
The test steps are almost the same as smoke_test except for
executing reboot from Azure SDK.
Priority

1

Requirement

supported_features[SerialConsole, StartStop]

verify_deployment_provision_synthetic_nic()
Description
This case runs smoke test on a node provisioned with synthetic nic.
The test steps are same as smoke_test.
Priority

1

Requirement

supported_features[SerialConsole]

verify_stop_start_in_platform()
Description
This case runs smoke test on a node provisioned.
The test steps are almost the same as smoke_test except for
executing stop then start from Azure SDK.
Priority

2

Requirement

supported_features[SerialConsole, StartStop]

smoke_test()
Description
This case verifies whether a node is operating normally.
Steps,
1. Connect to TCP port 22. If it’s not connectable, failed and check whether
there is kernel panic.
2. Connect to SSH port 22, and reboot the node. If there is an error and kernel
panic, fail the case. If it’s not connectable, also fail the case.
3. If there is another error, but not kernel panic or tcp connection, pass with
warning.
4. Otherwise, fully passed.
Priority

0

Requirement

supported_features[SerialConsole]

class TimeSync
Description
This test suite is related with time sync.
Platform

Azure, Ready

Area

time

Category

functional

timesync_check_unbind_clocksource()
Description
This test is to check -
1. Check clock source name is one of hyperv_clocksource_tsc_page,
lis_hv_clocksource_tsc_page, hyperv_clocksource.
2. Check CPU flag contains constant_tsc from /proc/cpuinfo.
3. Check clocksource name shown up in dmesg.
4. Unbind current clock source if there are 2+ clock sources, check current
clock source can be switched to a different one.
Priority

2

timesync_validate_ptp()
Description
This test is to check -
1. PTP time source is available on Azure guests (newer versions of Linux).
2. PTP device name is hyperv.
3. When accelerated network is enabled, multiple PTP devices will
be available, the names of ptp are changeable, create the symlink
/dev/ptp_hyperv to whichever /dev/ptp entry corresponds to the Azure host.
4. Chrony should be configured to use the symlink /dev/ptp_hyperv
instead of /dev/ptp0 or /dev/ptp1.
Priority

2

timesync_ntp()
Description
This test is to check, ntp works properly.
1. Stop systemd-timesyncd if this service exists.
2. Set rtc clock to system time.
3. Restart Ntp service.
4. Check and set server setting in config file.
5. Restart Ntp service to reload with new config.
6. Check leap code using ntpq -c rv.
7. Check local time is synchronised with time server using ntpstat.
Priority

2

timesync_check_unbind_clockevent()
Description
This test is to check -
1. Current clock event name is ‘Hyper-V clockevent’.
2. ‘Hyper-V clockevent’ and ‘hrtimer_interrupt’ show up times in
/proc/timer_list should equal to cpu count.
3. when cpu count is 1 and cpu type is Intel type, unbind current time
clock event, check current time clock event switch to ‘lapic’.
Priority

2

timesync_chrony()
Description
This test is to check chrony works properly.
1. Restart chrony service.
2. Check and set server setting in config file.
3. Restart chrony service to reload with new config.
4. Check chrony sources and sourcestats.
5. Check chrony tracking.
Priority

2

class Floppy
Description
This test suite ensures the floppy driver is disabled.
The floppy driver is not needed on Azure and
is known to cause problems in some scenarios.
Platform

Azure, Ready

Area

core

Category

functional

check_floppy_module()
Description
The goal of this test is to ensure the floppy module is not enabled
for images used on the Azure platform.
This test case will
1. Dry-run modprobe to see if floppy module can be loaded
2. If “insmod” would be executed then the module is not already loaded
3. If module cannot be found then it is not loaded
If the module is loaded, running modprobe will have no output
Priority

1

class HvModule
Description
This test suite covers test cases previously handled by LISAv2:
LIS-MODULES-CHECK, VERIFY-LIS-MODULES-VERSION,
INITRD-MODULES-CHECK, RELOAD-MODULES-SMP
It is responsible for ensuring the Hyper V drivers are all present,
are included in initrd, and are all the same version.
Platform

Azure, Ready

Area

core

Category

functional

verify_hyperv_modules()
Description
This test case will
1. Verify the presence of all Hyper V drivers using lsmod
to look for the drivers not directly loaded into the kernel.
Priority

1

verify_initrd_modules()
Description
This test case will ensure all necessary hv_modules are present in
initrd. This is achieved by
1. Skipping any modules that are loaded directly in the kernel
2. Find the path of initrd file
3. Use lsinitrd tool to check whether a necessary module is missing
Priority

2

verify_lis_modules_version()
Description
This test case will
1. Verify the list of given LIS kernel modules and verify if the version
matches with the Linux kernel release number. (Drivers loaded directly in
to the kernel are skipped)
Priority

1

class Storage
Description
This test suite is used to run storage related tests.
Platform

Azure, Ready

Area

storage

Category

functional

verify_resource_disk_mtab_entry()
Description
This test will check that the resource disk is present in the list of mounted
devices. Most VMs contain a resource disk, which is not a managed disk and
provides short-term storage for applications and processes. It is intended to
only store data such as page or swap files.
Steps:
1. Get the mount point for the resource disk. If /var/log/cloud-init.log
file is present, mount location is /mnt, otherwise it is obtained from
ResourceDisk.MountPoint entry in waagent.conf configuration file.
2. Verify that “/dev/<disk> <mount_point>` entry is present in
/etc/mtab file and the disk should not be equal to os disk.
Priority

1

Requirement

supported_platform_type[AZURE]

verify_default_targetpw()
Description
This test will verify that Defaults targetpw is not enabled in the
/etc/sudoers file.
If targetpw is set, sudo will prompt for the
password of the user specified by the -u option (defaults to root)
instead of the password of the invoking user when running a command
or editing a file. More information can be found here :
Steps:
1. Get the content of /etc/sudoers file.
2. Verify that Defaults targetpw should be disabled, if present.
Priority

1

verify_root_device_timeout_setting()
Description
This test will check that VM root disk(os disk) is provisioned
with the correct timeout.
Steps:
1. Find the root disk (os disk) for the VM. The root disk
is the entry with mount point /’ in the output of `mount command.
2. Verify the timeout value for root disk in
/sys/block/<partition>/device/timeout file is set to 300.
Priority

1

Requirement

supported_platform_type[AZURE]

verify_resource_disk_io()
Description
This test will check that the file IO operations are working correctly
Steps:
1. Get the mount point for the resource disk. If /var/log/cloud-init.log
file is present, mount location is /mnt, otherwise it is obtained from
ResourceDisk.MountPoint entry in waagent.conf configuration file.
2. Verify that resource disk is mounted from the output of mount command.
3. Write a text file to the resource disk.
4. Read the text file and verify that content is same.
Priority

1

Requirement

supported_platform_type[AZURE]

verify_swap()
Description
This test will check that the swap is correctly configured on the VM.
Steps:
1. Check if swap file/partition is configured by checking the output of
swapon -s and lsblk.
2. Check swap status in waagent.conf.
3. Verify that truth value in step 1 and step 2 match.
Priority

1

Requirement

supported_platform_type[AZURE]

class nvme
Description
This test suite is to validate NVMe disk on Linux VM.
Platform

Azure, Ready

Area

nvme

Category

functional

nvme_blkdiscard_validation()
Description
This test case will
1. Create a partition, xfs filesystem and mount it.
2. Umount the mountpoint.
3. Run blkdiscard command on the partition.
4. Remount command should fail after run blkdiscard command.
Priority

3

Requirement

supported_features[Nvme]

nvme_fstrim_validation()
Description
This test case will
1. Create a partition, xfs filesystem and mount it.
2. Check how much the mountpoint is trimmed before operations.
3. Create a 300 gb file ‘data’ using dd command in the partition.
4. Check how much the mountpoint is trimmed after creating the file.
5. Delete the file ‘data’.
6. Check how much the mountpoint is trimmed after deleting the file,
and compare the final fstrim status with initial fstrim status.
Priority

3

Requirement

supported_features[Nvme]

nvme_rescind_validation()
Description
This test case will
1. Disable NVME devices.
2. Enable NVME device.
Priority

2

Requirement

supported_features[Nvme]

nvme_function_validation()
Description
This test case will do following things for each NVMe device.
1. Get the number of errors from nvme-cli before operations.
2. Create a partition, filesystem and mount it.
3. Create a txt file on the partition, content is ‘TestContent’.
4. Create a file ‘data’ on the partition, get the md5sum value.
5. Umount and remount the partition.
6. Get the txt file content, compare the value.
7. Compare the number of errors from nvme-cli after operations.
Priority

2

Requirement

supported_features[Nvme]

nvme_manage_ns_validation()
Description
This test case will run commands 2-5, the commands are expected fail or not
based on the capabilities of the device.
1. Use nvme id-ctrl device command list the capabilities of the device.
1.1 When ‘Format NVM Supported’ shown up in output of ‘nvme id-ctrl device’,
then nvme disk can be format, otherwise, it can’t be format.
1.2 When ‘NS Management and Attachment Supported’ shown up in output of
‘nvme id-ctrl device’, nvme namespace can be created, deleted and detached,
otherwise it can’t be managed.
2. nvme format namespace - format a namespace.
3. nvme create-ns namespace - create a namespace.
4. nvme delete-ns -n 1 namespace - delete a namespace.
5. nvme detach-ns -n 1 namespace - detach a namespace.
Priority

3

Requirement

supported_features[Nvme]

nvme_sriov_rescind_validation()
Description
This test case does following steps to verify VM working normally during
disable and enable nvme and sriov devices.
1. Disable PCI devices.
2. Enable PCI devices.
3. Get PCI devices slots.
4. Check PCI devices are back after rescan.
Priority

2

Requirement

supported_features[Nvme]

nvme_max_disk_validation()
Description
This case runs nvme_basic_validation test against 10 NVMe disks.
The test steps are same as nvme_basic_validation.
Priority

2

Requirement

supported_features

nvme_basic_validation()
Description
This test case will
1. Get nvme devices and nvme namespaces from /dev/ folder,
compare the count of nvme namespaces and nvme devices.
2. Compare the count of nvme namespaces return from nvme list
and list nvme namespaces under /dev/.
3. Compare nvme devices count return from lspci
and list nvme devices under /dev/.
4. Azure platform only, nvme devices count should equal to
actual vCPU count / 8.
Priority

1

Requirement

supported_features[Nvme]

class NetworkSettings
Description
This test suite runs the ethtool related network test cases.
Platform

Azure, Ready

Area

network

Category

functional

validate_ringbuffer_settings_change()
Description
This test case verifies if ring buffer settings can be changed with ethtool.
Steps:
1. Get the current ring buffer settings.
2. Change the rx and tx value to new_values using ethtool.
3. Get the settings again and validate the current rx and tx
values are equal to the new_values assigned.
4. Revert back the rx and tx value to their original values.
Priority

1

validate_device_rss_hash_key_change()
Description
This test case verifies changing device’s RSS hash key takes
into affect.
Steps:
1. Skip the test if the kernel version is any less than LTS 5.
2. Get all the device’s RSS hash key values.
3. Swap the last 2 characters of original hash key to make a new hash key.
4. Validate changing the hash key setting using the new hash key.
5. Revert back the settings to original values.
Priority

2

validate_device_rx_hash_level_change()
Description
This test case verifies whether changing device’s RX hash level
for tcp and udp takes into affect.
Steps:
Note: Same steps are used for both TCP and UDP.
1. Get all the device’s RX hash level status.
2. Depending on current setting, change to enabled/disabled.
3. Validate changing the hash level setting.
4. Revert back the settings to original values.
Priority

2

validate_device_enabled_features()
Description
This test case verifies required device features are enabled.
Steps:
1. Get the device’s enabled features.
2. Validate below features are in the list of enabled features-
rx-checksumming
tx-checksumming
tcp-segmentation-offload
scatter-gather
Priority

1

validate_device_msg_level_change()
Description
This test case verifies whether setting/unsetting device’s
message level flag takes into affect.
Steps:
1. Verify Get/Set message level supported on kernel version.
2. Get all the device’s message level number and name setting.
2. Depending on current setting, set/unset a message flag by number
and name.
3. Validate changing the message level flag setting.
4. Revert back the setting to original value.
Priority

2

validate_device_channels_change()
Description
This test case verifies changing device channels count with ethtool.
Steps:
1. Get the current device channels info.
2 a. Keep Changing the channel count from min to max value using ethtool.
b. Get the channel count info and validate the channel count
value is equal to the new value assigned.
3. Revert back the channel count to its original value.
Priority

1

validate_device_gro_lro_settings_change()
Description
This test case verifies changing device’s GRO and LRO setting takes
into affect.
Steps:
1. Get all the device’s generic-receive-offload and large-receive-offload
settings.
2. If both GRO and LRO settings are “[fixed]” then skip testing specific
device.
3. Try flipping the GRO and LRO settings and validate it takes affect.
4. Revert back the settings to original values.
Priority

1

class docker
Description
This test suite runs the docker test cases for java, python, dotnet, and wordpress.
Platform

Azure, Ready

Area

docker

Category

functional

docker_java_app()
Description
This test case creates and runs a java app using docker
Steps:
1. Install Docker
2. Copy java dockerfile and program file to node
3. Create docker image and run docker container
4. Check results of docker run against java string identifier
Priority

1

docker_python_app()
Description
This test case creates and runs a python app using docker
Steps:
1. Install Docker
2. Copy python dockerfile and program file to node
3. Create docker image and run docker container
4. Check results of docker run against python string identifier
Priority

3

class gpu
Description
This test suite runs the gpu test cases.
Platform

Azure, Ready

Area

gpu

Category

functional

validate_load_driver()
Description
This test case verifies if gpu drivers are loaded fine.
Steps:
1. Validate if the VM SKU is supported for GPU.
2. Install LIS drivers if not already installed for Fedora and its
derived distros. Reboot the node
3. Install required gpu drivers on the VM and reboot the node. Validate gpu
drivers can be loaded successfully.
Priority

1

Requirement

supported_features[Gpu, SerialConsole]

validate_gpu_adapter_count()
Description
This test case verifies the gpu adapter count.
Steps:
1. Assert that node supports GPU.
2. If GPU modules are not loaded, install and load the module first.
3. Find the expected gpu count for the node.
4. Validate expected and actual gpu count using lsvmbus output.
5. Validate expected and actual gpu count using lspci output.
6. Validate expected and actual gpu count using gpu vendor commands
example - nvidia-smi
Priority

2

Requirement

supported_features[Gpu]