3 posts tagged with "language-server"

Jupyter notebooks have become an essential tool for Python developers. Their interactive, cell-based workflow makes them ideal for rapid prototyping, data exploration, and scientific computing: areas where you want to tweak a small part of the code and see the updated results inline, without waiting for the whole program to run. Notebooks are the primary way many data scientists and ML engineers write Python, and interactive workflows are highlighted in new data science oriented IDEs like Positron.

But notebooks have historically been second-class citizens when it comes to IDE features. Language servers, which implement the Language Server Protocol (LSP) to provide features like go-to-definition, hover, and diagnostics across editors, were designed with regular source files in mind. The language server protocol did not include notebook synchronization methods until five years after it was created, and the default Jupyter Notebook experience is missing many of the aforementioned IDE features.

In this post, we'll discuss how language servers have been adapted to work with notebooks, how the LSP spec evolved to support them natively, and how we implemented notebook support in Pyrefly.

As we move closer to a stable release of Pyrefly, our efforts have moved from expanding the language server’s capabilities to tackling performance edge cases. Recently, our friends at Astral alerted us to a specific edge case that is a great example of the kind of issues we’ve been aiming to uncover. The specific example Astral highlighted showed that in some edge cases Pyrefly could take multiple seconds to update diagnostics after editing. This is much slower than expected (used across Meta’s codebases, Pyrefly usually takes less than 10 milliseconds to recheck files after saving them) and prompted us to investigate further.

Now we’d like to share the story of how that edge case led us to rethink our underlying approach to diagnostics and dependency tracking, improving the speed at which Pyrefly's type errors update by 18x in the process.

The challenges of managing ever-growing codebases are hardly new. As far back as 1995, Niklaus Wirth, (creator of programming language Pascal) already emphasized the importance of keeping software lean in his essay A Plea for Lean Software. Fast forward to today: many programmers still face the reality that, despite their best coding intentions, as projects grow in size and complexity, so do their codebases. Even when a project scales to millions of lines of code, developers still expect their IDE to be fast, accurate, and efficient. And with increasingly large and interconnected (dare I say, spaghetti?) code, you probably rely on your IDE even more to help you navigate the chaos (ahem I mean complexity).