Skip to main content

What's type-coverage, and why should I care?

· 5 min read
Marco Gorelli, Quansight Labs

Pyrefly is one of the major new Python type-checkers taking the world by storm. It's often touted for its speed, usability, and typing-spec-conformance. But it also offers an extra nifty little feature you may never have heard of: type-coverage.

Let's learn about what it is, why you might want to care about it, and what it can do for you.

TL;DR:

  • Type coverage measures what percentage of your project's typables are type-annotated.
  • Optionally, you can choose to only include publicly-exported typables.
  • You can use it similarly to how you would use test coverage.

What's type coverage?​

Similarly to how test coverage measures how much of a library's source code is hit by its tests, type coverage measures what percentage of a library's typables have type annotations. By typable, we mean something which can be meaningfully annotated with a type, such as function arguments and return types, class variables, and constants. Local variables in function bodies, on the other hand, are excluded, as users would never interact with them anyway.

If this sounds familiar to you, it may be because we've previously written about the topic in typing pandas and typing numpy. In those posts, we were using Pyright's verify-types feature. Now, however, Pyrefly ships its own type coverage tool (pyrefly coverage), which we're now recommending you use instead.

Concrete example​

Say we have a Python file which defines the following functions:

  • def one(a: int) -> int
  • def two(b: int) -> None
  • def three() -> None

and a stub file (one which ends with .pyi) which contains:

  • def one(a: int) -> int
  • def two(c) -> None

Note how there are two issues with the stub file:

  • The c parameter in two is misnamed (it should be b). It's also unannotated.
  • The three function is missing entirely.

Note that if a stub file is present, then that's all the type-checker looks at. The typing spec enforces this: "If a stub file is found for a module, the type checker should not read the corresponding “real” module". So, if stub files are present, they could be both accurate and complete.

Here, however, the stub is incomplete. How could we have been alerted about this? Just running a type-checker isn't enough, pyrefly check would return zero errors in this code alone. Just running ruff check with ANN enabled isn't enough either, as that would only tell us that the c argument is unannotated (it wouldn't tell us anything about the three function missing from the stub). Fortunately, there is indeed a tool which can tell us that three is missing from the stubs: pyrefly coverage. In this case, running pyrefly coverage check tells us:

$ pyrefly coverage check
 WARN `foo.three` is untyped [coverage-missing]
 --> src/foo/__init__.py:7:1
  |
7 | / def three() -> None:
8 | |     return None
  | |_______________-
  |
 WARN `foo.two` is not fully typed [coverage-partial]
 --> src/foo/__init__.pyi:2:1
  |
2 | def two(c) -> None: ...
  | -----------------------
  |
ERROR type coverage 60.00% (3 of 5 typable) is below the 100.00% threshold

Great! Now we can add three to the stubs and our users can safely use it with type-checking.

Can't I just use Ruff's ANN rules?​

The Python linter ruff has a suite of typing-related rules grouped together under the ANN prefix. It's very useful for catching missing type annotations, and because Ruff analyses files statically, the implementation is blazingly fast. So what does Pyrefly's type-coverage offer on top of that? We present three benefits.

First, class variables. If you have a class which does any non-trivial logic in its __init__ method, then those class variables might not get inferred uniformly across type-checkers (if at all!). pyrefly coverage enforces that you explicitly specify types for your class variables. This PR which typed the GroupBy attributes is a good example of this, where offset, period, and closed all go through non-trivial transformations in the __init__ method.

Second, you might not currently have type annotations everywhere. By giving you a report, a coverage score, and the option to only include public typables (--public-only), pyrefly coverage allows you to focus your efforts on the highest-yield parts of your codebase and tell you how far along with your effort you are. We used this to prioritise typing efforts in NumPy.

Finally, stub files. pyrefly coverage checks that you don't forget to include annotations for anything in your Python files, if you have stub files. stubtest can also do this, but that also checks for runtime behaviour and so is a more intensive check, whereas pyrefly coverage is fast enough that you could easily include it in a pre-commit configuration file without it impacting developer productivity.

Who's using it?​

Pyrefly coverage is a new-ish feature, yet it's already being used by a few major projects:

If you would like to use it in your project and would like any assistance, feel free to reach out, and if you encounter any issues, please do report them!