Skip to main content

Pyrefly Coverage

pyrefly coverage measures how much of your code is annotated with types. It has two subcommands:

How coverage is measured​

Coverage is counted over typables: function return types, function parameters, module-level variables, and class attributes. Each typable is either typed (annotated, no Any in the resolved type), any (annotated, but the resolved type contains Any), or untyped (no annotation).

  • Coverage is the percentage of typables that are annotated; Any counts as covered.
  • Strict coverage is the percentage of typables that are typed; Any doesn't count.

Only public symbols count: names without a leading underscore, plus anything in __all__. self/cls are not typable, overloads are merged, and .py files are skipped when their .pyi stub is also analyzed (--prefer-stubs=false disables this). With --public-only, only symbols reachable from public modules via re-export chains count: the library's public API.

A simple unannotated assignment to a module-level variable only counts as a typable if the value is a call, since a call hides the type at the assignment site. Four kinds of call are exempt:

  • Constructors of non-generic classes that return an instance of that class, like Path(".").
  • Constructors of some generic builtins, like list().
  • Explicitly specialized classes, like Spam[int]().
  • Anything that returns a logging.Logger or re.Pattern, since those are conventionally left unannotated.

A variable assigned in both branches of an if/else counts whatever the values.

Cython​

In Cython pure Python mode, C-level declarations don't exist at runtime, so they aren't counted as typables:

  • @cython.cfunc functions and methods.
  • Module-level names assigned from cython.declare, cython.typedef, cython.struct, cython.union, or cython.fused_type, as in x = cython.declare(cython.int).
  • @cython.internal classes.
  • Fields of extension types (@cython.cclass, or @cython.total_ordering, which also creates one), except dynamic attributes of a type that declares or inherits __dict__.
  • The C-only special methods __dealloc__, __richcmp__, __getbuffer__, and __releasebuffer__.

An extension type's __cinit__ counts as its __new__, so a stub must declare __new__ (not __init__) to cover it.

Limitations
  • .pxd files aren't read, so public and readonly fields aren't counted.
  • Keyword declarations such as cython.declare(x=cython.int) aren't recognized.

Checking coverage against a threshold​

pyrefly coverage check robot/ --fail-under 60

It reports every offending symbol in the same style as pyrefly check, as coverage-missing (no annotations at all) or coverage-partial (only some), followed by a one-line summary:

 WARN `walk` is not fully typed [coverage-partial]
 --> robot/legs.py:1:1
  |
1 | / def walk(speed, direction: str) -> None:
2 | |     print(f"walking {speed} {direction}")
  | |_________________________________________-
  |
 WARN `stop` is untyped [coverage-missing]
 --> robot/legs.py:5:1
  |
5 | / def stop():
6 | |     print("stopping")
  | |_____________________-
  |
ERROR type coverage 66.67% (4 of 6 typable) is below the 100.00% threshold

When coverage meets the threshold, the findings are printed but the summary is INFO and the exit code is 0:

 INFO type coverage 66.67% (4 of 6 typable)

Findings go to stdout, the summary to stderr. With --strict, annotations that resolve to Any count as untyped. --output-format controls the findings format. See pyrefly coverage check --help for all flags.

Pre-commit​

To check coverage on every commit with prek or pre-commit, use the pyrefly-coverage-check hook from meta/pyrefly-pre-commit.

Generating a JSON report​

pyrefly coverage report path/to/directory/ > coverage.json

The report contains a "major.minor" schema_version, a summary, and one entry per module in module_reports, with:

  • per-symbol typed/any/untyped counts and locations (symbol_reports)
  • the module's public names (names)
  • error-suppression comments (type_ignores)
  • typable totals, coverage and strict_coverage percentages, and counts of functions, methods, parameters, classes, attributes, and properties

See pyrefly coverage report --help for all flags. The output is easy to consume with e.g. jq:

pyrefly coverage report path/to/directory/ | jq .summary.strict_coverage