Netflix has announced that it is open sourcing Polynote, which, as the name implies, is a polyglot notebook. Polynote provides Scala support, Apache Spark integration, as-you-type autocomplete, and multi-language interoperability with Scala, Python, and SQL.
According to Netflix, Polynote will allow data scientists to integrate its JVM-based machine learning platform with Python’s ecosystem of machine learning and visualization libraries.
Polynote was created due to an internal frustration with existing notebooks. Netflix developers felt that current notebooks lacked things like support of Scala and had a bad code editing experience. And in the machine learning community, there is a need for polyglot support since machine learning researchers are often working with multiple programming languages, the company explained.
Two of the guiding principles of the notebook are reproducibility and visibility. The team achieved these goals by building a code interpretation from scratch rather than relying on a REPL (read-eval-print loop) like most traditional notebooks.
In addition, it added some IDE-like features like auto-complete, error highlighting, and a rich text editor that allows scientists to insert equations from LaTeX. “Let’s face it — for someone used to IDEs, writing a nontrivial amount of code in a notebook can feel like going back in time a few decades. We’ve seen users who prefer to write code in an IDE instead, and paste it into the notebook to run. While it’s not our goal to provide all the features of a full-fledged modern IDE, there are a few quality-of-life code editing enhancements that go a long way toward improving usability,” Netflix wrote in a post.
Polynote also stores its configuration and dependency information directly in the notebook itself, which means scientists won’t have to rely on external files or a cluster/server level configuration.
Finally, it integrates with two popular open-source visualization libraries: Vega and Matplotlib.
The notebook has been adopted heavily within Netflix’s personalization and recommendation teams, and will now be integrated with the rest of their research platform.
