Development setup

This document describes the existing developer tooling we have in place (and what to expect of it).

Environment Setup

Follow the Installation steps to prepare the source code. Then follow the steps below to set up the corresponding development environment.

Creating a LISA development virtual environment

Nox is used to manage virtual environments for LISA. See Using Nox for more information.

Nox can be installed with pip.

pip3 install nox toml

The following creates a virtual environment in .venv with an editable install of LISA. An editable install allows code changes in the repo to immediately affect the virtual environment. This is useful for iterative development.

From the root of the LISA repo, run the following to create or update virtual environment.

nox -vs dev

By default, the extra dependencies for Azure will be installed. On Linux, the extra dependencies for libvirt will also be installed. This behavior can be overridden by passing dependency groups as additional arguments.

For example, the following will only install extra dependencies for Azure.

nox -vs dev -- azure

Information on what extra dependency groups are supported can be found in LISA’s extras.

VSCode should automatically detect and activate the virtual environment on startup and you can manually activate it in a shell with the following commands.

Activate virtual environment (Bash)

source .venv/bin/activate

Activate virtual environment (Powershell)

.venv\Scripts\activate.ps1

Activate virtual environment (cmd)

.venv\Scripts\activate.bat

When the virtual environment is active, your command prompt will be prefixed with (lisa).

If you wish to deactivate the virtual environment, use the deactivate command.

deactivate

Visual Studio Code

  1. Click on the Python version at the bottom left of the editor’s window and select the Python interpreter which Poetry just created. If you do not find it, check FAQ and troubleshooting for extra instructions. This step is important because it ensures that the current workspace uses the correct Poetry virtual environment which provides all dependencies required.

  2. You can copy the settings below into .vscode/settings.json.

    {
   "markdown.extension.toc.levels": "2..6",
   "python.analysis.typeCheckingMode": "strict",
   "python.linting.pylintEnabled": false,
   "python.analysis.useLibraryCodeForTypes": false,
   "python.analysis.autoImportCompletions": false,
   "files.eol": "\n",
   "terminal.integrated.env.windows": {
      "mypypath": "${workspaceFolder}\\typings"
   },
   "python.analysis.diagnosticSeverityOverrides": {
      "reportUntypedClassDecorator": "none",
      "reportUnknownMemberType": "none",
      "reportGeneralTypeIssues": "none",
      "reportUnknownVariableType": "none",
      "reportUnknownArgumentType": "none",
      "reportUnknownParameterType": "none",
      "reportUnboundVariable": "none",
      "reportPrivateUsage": "none",
      "reportImportCycles": "none",
      "reportUnnecessaryIsInstance": "none",
      "reportPrivateImportUsage": "none",
      "reportUnusedImport": "none",
      "reportUnusedFunction": "none",
      "reportOptionalMemberAccess": "none"
   },
   "python.analysis.stubPath": "./typings",
   "python.languageServer": "Pylance",
   "flake8.importStrategy": "fromEnvironment",
   "mypy-type-checker.importStrategy": "fromEnvironment",
   "mypy-type-checker.args": [
      "--config-file",
      "pyproject.toml"
   ],
   "black-formatter.importStrategy": "fromEnvironment",
   "[python]": {
      "editor.defaultFormatter": "ms-python.black-formatter",
      "editor.formatOnSave": true,
      "editor.codeActionsOnSave": {
            "source.organizeImports": true
      },
   },
   "isort.importStrategy": "fromEnvironment",
   "isort.args": [
      "--settings-path",
      "pyproject.toml"
   ]
}

  3. Install extensions.

    • Install Python, Pylance to get best code intelligence experience.

    • Install Python extensions to get consistent error as CI pipelines, flake8, mypy, black, isort.

    • Install Rewrap to automatically wrap.

    • If there is need to update the documentation, it is recommended to install Markdown All in One. It helps to maintain the table of contents in the documentation.

    • Install reStructuredText to get a syntax checker for reStructuredText. To preview the document, see Local Documentation.

Emacs

Use the pyvenv package:

(use-package pyvenv
  :ensure t
  :hook (python-mode . pyvenv-tracking-mode))

Then run M-x add-dir-local-variable RET python-mode RET pyvenv-activate RET <path/to/virtualenv> where the value is the path given by the command above. This will create a .dir-locals.el file as follows:

;;; Directory Local Variables
;;; For more information see (info "(emacs) Directory Variables")

((python-mode . ((pyvenv-activate . ".venv"))))

Other setups

  • Install and enable ShellCheck to find bash errors locally.

Code checks

If the development environment is set up correctly, the following tools will automatically check the code. If there is any problem with the development environment settings, please feel free to submit an issue to us or create a pull request for repair. You can also run the check manually.

  • Black, the opinionated code formatter resolves all disputes about how to format our Python files. This will become clearer after following PEP 8 (official Python style guide).

  • Flake8 (and integrations), the semantic analyzer, used to coordinate most other tools.

  • isort, the import sorter, it will automatically divide the import into the expected alphabetical order.

  • mypy, the static type checker, which allows us to find potential errors by annotating and checking types.

  • rope, provides completion and renaming support for pyls.

Using Nox

Nox is test automation utility that allows running tests and utilities in virtual environments. This allows isolation and consistency for these actions.

Sessions

Nox tasks are called sessions. A number of Nox sessions have been configured for LISA. They can be displayed by running nox --list.

$  nox --list
Nox configuration file
See https://nox.thea.codes/en/stable/config.html

Sessions defined in /srv/Development/lisa/noxfile.py:

* test -> Run tests
* example -> Run example
* coverage -> Check test coverage
* black -> Run black
* isort -> Run isort
* flake8 -> Run flake8
* mypy -> Run mypy
* docs -> Build docs
* dev -> Create virtual environment for development

sessions marked with * are selected, sessions marked with - are skipped.

An individual session can be run with nox -vs <session>.

$ nox -vs flake8
nox > Running session flake8
nox > Creating virtual environment (virtualenv) using python3 in .nox/flake8
...
nox > flake8
nox > Session flake8 was successful.

Tags

Another way to call Nox sessions is with tags. Tags can not currently be listed on the command line, but the following have been define:

all

Runs various checks and tests to do before pushing a commit

format

Run formatting tools such as isort and black

linting

Run linting tools such as flake8

test

Run unit tests and test scenarios

typing

Run typing tools such as mypy

To execute all sessions with a given tag, use the -t option.

$ nox -vt format
nox > Running session black
...
nox > Running session isort
...
nox > Ran multiple sessions:
nox > * black: success
nox > * isort: success

To determine which sessions will be called for a tag without running them, use the --list option.

$ nox -t format --list
Nox configuration file
See https://nox.thea.codes/en/stable/config.html

Sessions defined in /srv/Development/lisa/noxfile.py:

- test -> Run tests
- example -> Run example
- coverage -> Check test coverage
* black -> Run black
* isort -> Run isort
- flake8 -> Run flake8
- mypy -> Run mypy
- docs -> Build docs
- dev -> Create virtual environment for development

Running with a different Python interpreter

The Nox sessions for LISA are configured to run with the same Python interpreter used to run Nox. To use a different interpreter, use the –force-python option. You can either specify a Python version or the path to an executable.

$ nox -vs test --force-python 3.11

$ nox -vs test --force-python /usr/bin/python3.11

Speeding up Nox

By default, Nox will recreate a virtual environment every time it runs. This ensures there are no stale dependencies, but is not always necessary. To reuse a virtual environment, use the -r option. To reuse the virtual environment without reinstalling any dependencies, use the -R option. This will have a greater impact for sessions with a large number of dependencies.

$ time nox -vs flake8
...
real    0m9.827s

$ time nox -vrs flake8
...
real    0m6.433s

$ time nox -vRs flake8
...
real    0m5.638s

Additional information

More information on Nox can be found here.

Local Documentation

It’s recommended to build the documentation locally using Sphinx for preview.

To do so, run

nox -vs docs

You can find all generated documents in ./lisa/docs/_build/html folder. Open them with a browser to view.

Extended reading