Test Maturity Model

Background

LISA test suites are growing in size and complexity, with frequent introduction of new tests and evolving validation coverage. Today, all discovered tests are treated equally unless explicitly filtered via runbook criteria, which makes it difficult to safely introduce immature or experimental tests without impacting validation pipelines.

To address this, a test maturity model is proposed to govern both lifecycle state and default execution behavior of tests.

Problem

New or evolving tests require staged validation before being considered production ready. Running such tests by default can:

• Introduce instability into production validation pipelines.

• Increase noise in failure signals.

• Require ad-hoc runbook changes to manage execution scope.

At the same time, developers need a simple and consistent way to control when tests are included in default execution.

Goals

• Introduce a single, unified concept for test readiness.

• Ensure production-ready tests continue to run by default.

• Prevent immature or unreliable tests from executing by default.

• Preserve backward compatibility with existing LISA test suites.

• Reuse the existing runbook selection model.

Proposal

Introduce an optional maturity field in TestCaseMetadata (and optionally TestSuiteMetadata for a suite-wide default):

If omitted, maturity defaults to stable, requiring no changes to existing tests.

Why Not Tags?

Tags could theoretically encode maturity (e.g. tags: ["preview"]), but a dedicated field is preferred for several reasons:

• Semantic clarity — maturity is a lifecycle property, not a topical grouping. Mixing the two in tags makes filtering ambiguous.

• Validated values — the maturity field accepts only the four defined levels (experimental, preview, stable, deprecated). Tags are free-form strings with no validation.

• Implicit gate — the stable gate is applied automatically after all other selection rules. Implementing this with tags would require every runbook to explicitly exclude non-stable tags, which is error-prone and not backward compatible.

• Inheritance — a suite-level maturity default is inherited by all its test cases unless overridden. Tags do not support this inheritance model.

Suite vs Case Maturity Precedence

When both TestSuiteMetadata and TestCaseMetadata specify a maturity value, the case-level value wins. If the case does not set maturity, it inherits the suite’s value. If neither is set, the default is stable.

@TestSuiteMetadata(
    area="example",
    category="functional",
    description="Suite with preview maturity",
    maturity="preview",       # suite-wide default
)
class ExampleSuite(TestSuite):

    @TestCaseMetadata(
        description="Inherits suite maturity (preview)",
        priority=2,
    )
    def test_inherits_preview(self, ...) -> None: ...

    @TestCaseMetadata(
        description="Overrides to stable",
        priority=2,
        maturity="stable",    # case-level override
    )
    def test_overrides_to_stable(self, ...) -> None: ...

Test Maturity Levels

The maturity field defines both lifecycle stage and default execution behavior.

Maturity	Description	Runs by Default
Experimental	Early-stage or actively evolving test.	No
Preview	Functionally complete but still under validation.	No
Stable (default)	Production-ready.	Yes
Deprecated	No longer recommended.	No

Note

experimental, preview, and stable describe an entry lifecycle of increasing readiness. deprecated is a separate retirement state — a test may be fully mature yet deprecated. All non-stable levels share the same default-execution behavior (not run by default) but remain distinct lifecycle stages.

Default Behavior

• Tests without a maturity value are treated as stable.

• Only stable tests run by default.

• Other maturity levels are excluded unless explicitly included.

Important

This changes the framework’s default selection behavior. Today, when no criteria are supplied, LISA selects all discovered cases. Under this proposal an implicit “stable-only” gate is applied to the final selection. This is backward compatible for test authors (an unset maturity defaults to stable) and produces no errors for existing runbooks, but the effective set of selected cases will shrink as tests are marked non-stable. This is by design.

Enabling Non-Stable Tests

Non-stable tests are included by referencing maturity in runbook criteria. This requires a new maturity criterion (see Changes Required); it is not supported by the existing criteria keys.

Benefits

• Backward compatible for test authors (unset maturity ⇒ stable).

• Safe introduction of new tests.

• Clear and explicit test lifecycle.

• No breaking changes or errors for existing runbooks (effective selection may shrink by design).

• Encourages gradual promotion (experimental → preview → stable).

Default & Override Behavior

Principle

An implicit maturity == stable gate is applied to the final selection. It is augmented (not replaced) by any explicit maturity reference and always overridden by forceInclude.

Truth table — does a test of maturity M run?

Test maturity	No maturity in runbook (default)	criteria maturity: [preview], include	forceInclude of this test (no maturity key)	forceExclude of this test
stable	Runs	Runs (still default)	Runs	Excluded
preview	Skipped	Runs (explicitly included)	Runs (force overrides gate)	Excluded
experimental	Skipped	Skipped (not referenced)	Runs (force overrides gate)	Excluded
deprecated	Skipped (+ optional warning if run)	Skipped	Runs + deprecation warning	Excluded

Precedence (highest to lowest):

1. forceExclude — always wins; the test never runs.

2. forceInclude — runs the test regardless of maturity (escape hatch for experimental/deprecated).

3. Explicit maturity criteria — include/exclude rules that name a maturity scope selection to those levels.

4. Implicit stable gate — applied last; drops any remaining non-stable case not covered above.

Test Selection Flow

The following describes the step-by-step order of operations when LISA selects test cases with the maturity model enabled:

1. Discover all test cases — LISA loads all registered TestCaseMetadata from the codebase. Each case has a resolved maturity (case-level → suite-level → stable).

2. Apply target OS pre-filter — if a target_os is configured, cases incompatible with that OS are dropped.

3. Process runbook filters sequentially — each filter rule in the runbook’s testcase list is applied in order:

   1. Criteria (name, area, category, priority, tags, maturity) are matched against the case metadata.

   2. The select_action (include / exclude / forceInclude / forceExclude) determines whether matched cases are added to or removed from the selection.

   3. If the filter contains a maturity criterion and the action is include or forceInclude, all matched cases are recorded in the maturity-approved set.

4. Apply the implicit stable gate — for every case still in the selection:

   • If maturity == stable → keep (always passes the gate).

   • If the case is in the force-included set → keep (force overrides the gate).

   • If the case is in the maturity-approved set → keep (explicitly approved by a maturity criterion).

   • Otherwise → drop (non-stable case with no explicit approval).

5. Emit warnings — any deprecated case that survived the gate (via forceInclude or maturity approval) triggers a deprecation warning in the log.

Note

When no filters are specified in the runbook, LISA returns all discovered cases. The implicit stable gate still applies, so only stable cases are selected.

Changes Required

• Add a maturity field to TestCaseMetadata (optional, defaults to stable).

• Add an optional suite-level maturity default in TestSuiteMetadata.

• Add a maturity criterion to the runbook selection model.

• Apply the implicit stable gate in the test selector.

Summary

The test maturity model provides a simple and safe way to introduce and manage tests in LISA. It preserves existing author behavior while enabling controlled onboarding and lifecycle progression for new and evolving tests.