Measure use-case coverage

Use case: the release question is never “is line coverage ≥ 80%?” — it’s “can a customer still check out?” You want a report that maps user-facing use cases to the tests that prove them, and flags the use cases nothing proves.

▶ Runnable sample: measure-use-case-coverage.spec.ts — how to run: examples/README

Why this is hard

Code coverage tools measure lines executed, which says nothing about scenarios proven: 90% line coverage can coexist with “password reset is untested”. Use cases live in specs and heads, not in any artifact a tool can count — until you make them one.

Recipe

1. Make use cases first-class: a registry

One file, owned like code, reviewed like code:

// usecases.ts — the product's testable surface
export const USECASES = {
  'checkout.guest': 'Guest completes a purchase',
  'checkout.member.discount': 'Member discount applies at checkout',
  'auth.password-reset': 'User resets a forgotten password',
  'auth.passkey.login': 'User signs in with a passkey',
  'billing.invoice.pdf': 'User downloads a correct invoice PDF',
} as const;
export type UseCaseId = keyof typeof USECASES;

2. Tag tests with the use cases they prove

Playwright’s tags carry the link (and --grep gives you scenario-scoped runs for free):

test('guest can buy a shirt', { tag: '@uc:checkout.guest' }, async ({ page }) => {
  // …
});

// Unit tests prove use cases too — a type-checked annotation:
describeUseCase('checkout.member.discount', () => { /* table-driven cases */ });

The typed ID matters: a renamed use case becomes a compile error, not a silently orphaned tag.

3. Aggregate with a custom reporter

import type { Reporter, TestCase, TestResult } from '@playwright/test/reporter';
import { USECASES } from './usecases';

export default class UseCaseReporter implements Reporter {
  private hits = new Map<string, { passed: number; failed: number }>();

  onTestEnd(test: TestCase, result: TestResult) {
    for (const tag of test.tags.filter((t) => t.startsWith('@uc:'))) {
      const id = tag.slice(4);
      const h = this.hits.get(id) ?? { passed: 0, failed: 0 };
      h[result.status === 'passed' ? 'passed' : 'failed']++;
      this.hits.set(id, h);
    }
  }

  onEnd() {
    const rows = Object.entries(USECASES).map(([id, title]) => {
      const h = this.hits.get(id);
      const state = !h ? '🔴 UNCOVERED' : h.failed ? `🟡 failing` : `🟢 ${h.passed} test(s)`;
      return `| ${id} | ${title} | ${state} |`;
    });
    require('node:fs').writeFileSync(
      'usecase-coverage.md',
      `| Use case | Description | Status |\n|---|---|---|\n${rows.join('\n')}`,
    );
  }
}

The output is the report stakeholders actually wanted from “coverage”:

Use case	Description	Status
checkout.guest	Guest completes a purchase	🟢 3 test(s)
auth.password-reset	User resets a forgotten password	🔴 UNCOVERED

4. Enforce the interesting direction

Failing CI on “uncovered use case” invites tag-spam. Enforce the honest subset instead: new entries in USECASES must gain a test within a sprint; deleting a test that is a use case’s only prover fails CI immediately. That’s the regression that matters.

Caveats

A tag is a claim. Periodically audit that the test would actually fail if the use case broke (mutation testing on the app, or targeted sabotage on a branch).
Keep the registry at user-intent granularity (“resets password”), not click-path granularity — paths churn, intents don’t.
This composes with demoting E2E tests: coverage stays visible while the proving test moves down the pyramid.