Metrics reference

This page explains every figure the Console and the Station show, with the same words as How is this computed? in the Console.

This page is generated from the product code; do not edit by hand.

Gate metrics

A gate rule or a Quality health check can use these figures.

Pass rate

Passed and flaky results divided by all results that passed, were flaky, or failed, timed out or interrupted, over the whole period (never an average of daily rates). A flaky result (failed, then passed on retry) counts as passed.

Period: last 7 days; a gate uses the last 7 project days; a cycle covers its own executions
What counts: Skipped results are not runs. Executions with local variable overrides are left out. Quarantined tests are left out of both counts and shown as “N quarantined”.
Threshold: Quality health: healthy from 95%, critical under 85%.
Unit: percentage. Id: pass-rate

P1 pass rate

The pass rate of the catalog tests tagged @priority:P1: their passed and flaky results divided by their decisive results.

Period: last 7 days
What counts: Skipped results are not runs. Executions with local variable overrides are left out. Quarantined tests are left out of both counts and shown as “N quarantined”.
Unit: percentage. Id: p1-pass-rate

Failing P1 tests

P1 catalog tests whose latest decisive result (passed, flaky, or failed, timed out or interrupted) in the last 30 days failed.

Period: latest result in the last 30 days
What counts: Quarantined tests are not counted. A gate shows no data when no test ran in the last 7 days.
Threshold: Quality health (P1 tests): warning at 1 failing, any P1 not run or any test failing 3+ runs in a row; critical at 3 failing or a P1 test failing 3+ runs in a row.
Unit: count. Id: failing-p1

Tests failing 3+ runs in a row

Catalog tests that ran in the period and whose latest 3 or more results in a row all failed, timed out or interrupted.

Period: results of the last 30 days; the test must have run in the last 7 days
What counts: Skipped results are ignored in the streak. Tests that left the catalog or did not run in the period are not counted. Quarantined tests are left out and shown as “N quarantined”. A gate shows no data when no test ran in the last 7 days.
Threshold: Quality health: warning at 1, critical at 5; a P1 test among them makes the P1 check critical.
Unit: count. Id: persistent-failures

New failures, 7 days (project days)

Tests that failed in the period and did not fail in the 30 days before it. The 30 days look at every environment of the project; the period follows the environment of the screen or gate (ADR-094).

Period: last 7 days (project days, today included), plus the 30 days before
What counts: A failure is a result that failed, timed out or interrupted; a flaky result is not a failure. Failures made while a test was quarantined count in the 30 days before; quarantined tests are left out. Executions with local variable overrides and invalidated results are left out. A test that is also a persistent failure is counted in both.
Threshold: Quality health: warning at 3, critical at 10.
Unit: count. Id: new-failures

Tests whose stability is flaky, divided by the tests that ran in the period. See the stability classes: a test failing 3+ runs in a row is a persistent failure, not flaky.

Period: last 7 days
What counts: Skipped results are not runs. Executions with local variable overrides are left out. Quarantined tests are left out of both counts.
Threshold: Quality health: warning from 3%, critical above 8%; a risk is proposed above 8%.
Unit: percentage. Id: flaky-ratio

Duration change

The median of the daily average test duration over the period, compared with the median over the last 30 days, as a change in percent.

Period: last 7 days against the last 30 days
What counts: Skipped results are left out. Days are in the project’s time zone. Not used for release gates.
Threshold: Quality health: warning above +15%, critical above +40%; a risk is proposed above +40%.
Unit: percentage. Id: duration-change

Catalog coverage

Catalog tests whose latest decisive result passed (passed or flaky), divided by all catalog tests.

Period: latest result in the last 30 days
What counts: Quarantined tests, tests whose latest decisive result failed, and tests with only skipped results or none count as not passing.
Unit: percentage. Id: catalog-coverage

Cycle completion (releases)

Catalog tests in the cycle’s scope that ran at least once, divided by all catalog tests in scope.

Period: the cycle; a release gate uses the latest started cycle
What counts: Skipped results do not count as a run. Applies to release gates only; project and environment gates skip it.
Unit: percentage. Id: regression-completion

Requirement coverage (releases)

Active requirements of the release scope whose linked tests’ latest decisive results in the release’s cycles on the environment all passed, divided by the active requirements of the release scope.

Period: the release’s cycles
What counts: A test links to a requirement with a @req:KEY tag. Quarantined tests never verify a requirement. Draft, retired and archived requirements are left out. No data when the release scope has no active requirement, or on an environment that is not a sign-off environment of the release (every release environment counts when none is marked). Applies to release gates only.
Unit: percentage. Id: requirement-coverage

Test stability

Each test is in one stability class. These counts are the tests in each class.

Stable

A test with at least 2 runs, all of them passed.

Period: last 7 days
What counts: Skipped results are not runs. A failure is a result that failed, timed out or interrupted.
Unit: count. Id: stability-stable

Failing

A test with at least 2 runs that failed and never passed, fewer than 3 times in a row.

Period: last 7 days
What counts: Skipped results are not runs. A failure is a result that failed, timed out or interrupted.
Unit: count. Id: stability-failing

Flaky

A test Playwright reported flaky at least once (failed, then passed on retry), or one with at least 2 runs that both passed and failed.

Period: last 7 days
What counts: Skipped results are not runs. A test failing 3+ runs in a row is a persistent failure instead.
Unit: count. Id: stability-flaky

Persistent failure

A test whose latest 3 or more results all failed. This class wins over every other.

Period: latest results, whatever their age
What counts: Skipped results are ignored in the streak. A failure is a result that failed, timed out or interrupted.
Unit: count. Id: stability-persistent-failure

Not enough data

A test with fewer than 2 runs, no flaky result and no failure streak: too little to judge.

Period: last 7 days
What counts: Skipped results are not runs.
Unit: count. Id: stability-insufficient-data

Test debt

The Debt tab of Quality counts tests in each debt category.

Test debt: quarantine overdue

Quarantined tests whose entry is older than 14 days, expires within 7 days, or skips the test.

Period: now; the days are project settings
What counts: Active quarantine entries of catalog tests.
Threshold: No threshold: the Debt tab shows the count, the P1 count and a 90-day trend.
Unit: count. Id: debt-quarantine-overdue

Test debt: flaky

Tests whose stability class is flaky: the Flaky tab population, so a test that left the catalog but ran in the window counts too.

Period: last 7 days
What counts: Quarantined tests are left out, as in the other flaky figures.
Threshold: No threshold: the Debt tab shows the count, the P1 count and a 90-day trend.
Unit: count. Id: debt-flaky

Test debt: persistent failure

Catalog tests whose latest 3 or more results all failed.

Period: latest results, whatever their age
What counts: Quarantined tests still count.
Threshold: No threshold: the Debt tab shows the count, the P1 count and a 90-day trend.
Unit: count. Id: debt-persistent-failure

Test debt: not run

Catalog tests without any result on any environment in the window.

Period: last 30 days by default (project setting)
What counts: A skipped result is a result. Executions with local variable overrides are left out.
Threshold: No threshold: the Debt tab shows the count, the P1 count and a 90-day trend.
Unit: count. Id: debt-not-run

Test debt: always skipped

Catalog tests whose results in the window were all skipped.

Period: last 30 days by default (project setting)
What counts: Executions with local variable overrides are left out.
Threshold: No threshold: the Debt tab shows the count, the P1 count and a 90-day trend.
Unit: count. Id: debt-always-skipped

Test debt: getting slower

Catalog tests whose average passed duration over the last 7 project days is 20% or more above the 7 days before, with at least 3 passed results on both sides.

Period: last 7 days, against the 7 days before
What counts: Passed results only. Executions with local variable overrides and archived workloads are left out.
Threshold: No threshold: the Debt tab shows the count, the P1 count and a 90-day trend.
Unit: count. Id: debt-getting-slower

Test debt: no @id

Catalog tests without an @id tag.

Period: the newest catalog
What counts: A malformed @id counts as none.
Threshold: No threshold: the Debt tab shows the count, the P1 count and a 90-day trend.
Unit: count. Id: debt-no-id

Test debt: no explicit owner

Catalog tests with no explicit owner: no orbit.owner annotation and no CODEOWNERS entry. The manifest team alone does not count.

Period: the newest catalog
What counts: Every catalog test of the live workloads.
Threshold: No threshold: the Debt tab shows the count, the P1 count and a 90-day trend.
Unit: count. Id: debt-no-owner

Test debt: no requirement

Catalog tests without a @req tag whose @feature is the feature of an active requirement.

Period: the newest catalog
What counts: Counted only when the project has requirements; tests without a @feature tag are left out.
Threshold: No threshold: the Debt tab shows the count, the P1 count and a 90-day trend.
Unit: count. Id: debt-no-requirement

Test debt: orphaned quarantine or triage

Quarantine entries and triage statuses whose test is no longer in the newest catalog.

Period: now
What counts: Active quarantine entries and current triage statuses.
Threshold: No threshold: the Debt tab shows the count, the P1 count and a 90-day trend.
Unit: count. Id: debt-orphaned-config

Test debt: old known issue

Failure groups marked Known issue for more than 30 days.

Period: now; the days are a project setting
What counts: Groups of archived workloads are left out.
Threshold: No threshold: the Debt tab shows the count, the P1 count and a 90-day trend.
Unit: count. Id: debt-known-issue-old

Test debt: requirement without a test

Active requirements that no catalog test links to with a @req tag.

Period: the newest catalog
What counts: Draft, retired and archived requirements are left out.
Threshold: No threshold: the Debt tab shows the count, the P1 count and a 90-day trend.
Unit: count. Id: debt-uncovered-requirement

Other figures

New in this cycle

Tests failing now in the cycle’s scope set that were not failing at the end of the previous cycle. The previous cycle is the latest earlier cycle of the same release and the same environment that has started and has at least one decisive result in its own scope set. Without one, every failing test counts and the figure says “no previous cycle”.

Period: the cycle, compared with the end of the previous cycle
What counts: Failing now is the latest decisive official result of the test in the cycle (failed, timed out or interrupted). Quarantined tests are left out. Executions with local variable overrides and invalidated results are left out. A closed cycle keeps the value stored when it closed.
Threshold: A release gate rule on this figure keeps its threshold.
Unit: count. Id: new-in-cycle

Requirement coverage

Active requirements whose linked tests’ latest decisive results all passed, divided by all active requirements, with the same figure for P1 requirements. A requirement is verified when that holds on some environment and no environment has a failing linked test.

Period: latest result in the last 30 days
What counts: A test links to a requirement with a @req:KEY tag in its code. Quarantined tests never verify a requirement. Draft, retired and archived requirements, archived workloads and executions with local variable overrides are left out.
Unit: percentage. Id: project-requirement-coverage

P1 tests not run

P1 catalog tests without any result in the last 30 days.

Period: last 30 days
What counts: A test with only skipped results has run. Quarantined tests are not counted.
Unit: count. Id: p1-not-run

Failing now

Tests whose latest result in the cycle failed, timed out or interrupted.

Period: this cycle
What counts: Only results in the cycle’s scope. Executions with local variable overrides are left out.
Unit: count. Id: failing-now

Flaky tests

Tests whose stability class is flaky in the period: Playwright reported them flaky, or they both passed and failed. The same count on every page.

Period: last 7 days, or the cycle
What counts: Skipped results are not runs. Executions with local variable overrides are left out. Quarantined tests are counted separately, never here.
Unit: count. Id: flaky-tests

Flaky results

Results Playwright reported flaky: the test failed, then passed on retry.

Period: last 7 days
What counts: Counts results, not tests. Quarantined tests and executions with local variable overrides are left out.
Unit: count. Id: flaky-reported

Failure signatures

Distinct error signatures (failures grouped by normalized message and top frame) seen on failed results in the period.

Period: last 7 days
What counts: Failed results only. Quarantined tests and executions with local variable overrides are left out.
Unit: count. Id: failure-signatures

Open failure groups

Failure groups (one per error signature) whose status is New or Triaged. A resolved group that fails again opens as New.

Period: now
What counts: Groups of archived workloads are left out.
Unit: count. Id: open-failure-groups

Failure groups past SLA

Open failure groups past their SLA: no owner within the owner days, or not resolved within the resolve days, counted in business days (Monday to Friday, project time zone) by the highest priority of the affected tests.

Period: now
What counts: Open groups only. No holiday calendar. The SLA is informational and never affects gates or readiness.
Unit: count. Id: sla-breached-groups

Recovered failure groups

Open failure groups where no test that failed with the error in the last 30 days still fails with it: the latest decisive result of each test and environment passed or failed differently.

Period: tests that failed with the error in the last 30 days
What counts: Executions with local variable overrides and archived workloads are left out. A group recovered for 14 days is resolved automatically.
Unit: count. Id: recovered-groups

Flaky queue

Tests whose stability is flaky or failing that are not quarantined and not marked Known flaky after their newest flaky run.

Period: last 7 days
What counts: Executions with local variable overrides and archived workloads are left out.
Unit: count. Id: flaky-queue

Average test duration

The total duration of the results of a day divided by their number.

Period: each day of the last 30 days, in the project’s time zone
What counts: Skipped results are left out. Executions with local variable overrides are left out.
Unit: duration. Id: avg-duration

Executions

Completed executions that started in the period. A cycle counts its member executions with a result in its scope.

Period: last 7 days, or the cycle
What counts: Executions with local variable overrides are left out.
Unit: count. Id: executions

Environments down

Environments whose current status is down: over the last 10 minutes, the latest sample of at least 2 sources (Stations or the Console probe) was unreachable, or the only source was unreachable twice.

Period: last 10 minutes
What counts: Samples are idle Station probes, run starts and the opt-in Console probe. An HTTP 5xx answer counts as reachable (degraded, not down). Without a sample in 30 minutes the status is unknown, never down; maintenance windows are left out.
Unit: count. Id: environments-down

Uptime

The share of the period an environment was not inside a down incident, from the daily health rollup.

Period: last 30 days on the environment page; last 7 days on Station Insights
What counts: Degraded, unknown and maintenance time count as up. Days before the first sample are left out.
Unit: percentage. Id: environment-uptime

Stations online

Stations that contacted the Console in the last 20 minutes and ran the project (or the customer’s projects) in the last 14 days.

Period: last 20 minutes; runs of the last 14 days
What counts: Removed Stations never count. Admin → Stations shows the organisation-wide count.
Unit: count. Id: stations-online

Runs today

Executions the Console recorded today in the environment (or in every environment of the project), from every Station and person, finished or not.

Period: today, in the project’s time zone; the change is against yesterday
What counts: Every execution counts, including runs with local variable overrides (custom options): they are real activity.
Unit: count. Id: runs-today

People today

People who started at least one execution today in the environment (or in every environment of the project).

Period: today, in the project’s time zone; the change is against yesterday
What counts: Every execution counts, including runs with local variable overrides.
Unit: count. Id: people-today

Stations online in the environment

Stations that ran in the chosen environment in the last 30 days and contacted the Console in the last 10 minutes, out of all Stations that ran there in those 30 days. When no environment is chosen, Stations that ran anywhere in the project.

Period: last 10 minutes; runs of the last 30 days
What counts: Removed and revoked Stations never count.
Unit: count. Id: environment-stations-online

Getting slower