Observability has really taken off in the past few years, and while in some ways observability has become a bit of a marketing buzzword, one of the main ways companies are implementing observability is not with any particular company’s solution, but with an open-source project: OpenTelemetry.

Since 2019, it has been incubating at the Cloud Native Computing Foundation, but the project has its origins in two different open-source projects: OpenCensus and OpenTracing, which were merged into one to form OpenTelemetry. 

“It has become now the de facto in terms of how companies are willing to instrument their applications and collect data because it gives them flexibility back and there’s nothing proprietary, so it helps them move away from data silos, and also helps connect the data end to end to offer more effective observability,” said Spiros Xanthos, SVP and general manager of observability at Splunk. 

OpenTelemetry is one of the most successful open-source projects, depending on what you measure by. According to Austin Parker, head of DevRel at Lightstep and maintainer of OpenTelemetry, it is the second highest velocity project within the CNCF, only behind Kubernetes, in terms of contributions and improvements.

According to Parker, one of the reasons why OpenTelemetry has just exploded in use is that cloud native development and distributed systems have “eaten the world.” This in turn leads to increased complexity. And what do you need when complexity increases? Observability, visibility, a way to understand what is actually going on in your systems. 

RELATED ARTICLE: How to ensure open-source longevity 

Parker feels that for the past few decades, a real struggle companies have run into is that everyone has a different tool for each part of observability. They have a tool for tracing, something for handling logs, something to track metrics, etc. 

“There’s scaling issues, lack of data portability, lack of vendor agnosticism, and a lack of ability to easily correlate these things across different dimensions and across different signal types,” said Parker. “OpenTelemetry is a project whose time has come in terms of providing a single, well-supported, vendor-agnostic solution for making telemetry a built-in part of cloud native systems.” 

Morgan McLean, director of product management at Splunk and co-founder of OpenTelemetry,  has seen first-hand how the project has exploded in use as it becomes more mature. He explained that a year ago, he was having conversations with prospective users who at the time felt like OpenTelemetry didn’t meet all of their needs. Now with a more complete feature set, “it’s become a thing that organizations are now much more comfortable and confident using,” Morgan explained. 

Today when he meets with someone to tell them about OpenTelemetry, often they will say they’re already using it. 

“OpenTelemetry is maybe the best starting point in that it has universal support from all vendors,” said Xanthos. “It’s a very robust set of, let’s say, standards and open source implementation. So first of all, I know that it will be something that will be around for a while. It is, let’s say, the state of the art on how to instrument applications and collect data. And it’s supported universally. So essentially, I’m betting on something that is a standard accepted across the industry, that is probably going to be around for a while, and gives me control over the data.”

It’s not just the enterprise that has jumped on board with OpenTelemetry; the open-source community as a whole has also embraced it. 

Now there are a number of web frameworks, programming languages, and libraries stating their support for OpenTelemetry. For example, OpenTelemetry is now integrated into .NET, Parker explained. 

Having a healthy open-source ecosystem crucial to success

There are a lot of vendors in the observability space, and OpenTelemetry “threatens the moat around most of the existing vendors in the space,” said Parker. It has taken a lot of work to build a community that brings in people that work for those companies and have them say “hey, here’s what we’re going to do together to make this a better experience for our end users, regardless of which commercial solution they might pick, or which open-source project they’re using,” said Parker. 

According to Xanthos, the reason an open-source standard has become the de facto and not something from a vendor is because of demand from end users. 

“End users essentially are asking vendors to have open-source standards-based data collection, so that they can have more effective observability tools, and they can have control over the data,” said Xanthos. “So because of this demand from end users, essentially all vendors either decided or were forced to support OpenTelemetry. So essentially, there is no major vendor and observability that doesn’t support it today.”

OpenTelemetry’s governance committee seats are tied to people, not companies, which is the case for some other open-source projects as well. 

“We try to be cognizant of the fact that we all work for people that have commercial interests here, but at the end of the day, we’re people and we are not avatars of our corporate overlords,” said Parker. 

For example, Morgan and Parker work for two separate companies which are direct competitors to each other, but in the OpenTelemetry space they come together to do things for the project like form end-user working groups or running events. 

“It doesn’t matter who signs the paycheck,” Parker said. “We are all in this space for a reason. It’s because we believe that by enabling observability for our end users through OpenTelemetry, we are going to make their professional lives better, we’re going to help them work better, and make that world of work better.”

What’s next?

OpenTelemetry has a lot planned for the future, and recently published an official project roadmap. 

The original promise of OpenTelemetry back when it was first announced was to deliver capabilities to allow people to capture distributed traces and metrics from applications and infrastructure, then send that data to a backend analytics system for processing. 

The project has largely achieved that, which presents the opportunity to sit down and ask what comes next. 

For example, logging is something important to a large portion of the community so that is one focus. “We want to be able to capture logs as an adjacent signal type to distributed traces and to metrics,” said Morgan.

Another long-term focus will be capturing profiles from applications so that developers can delve into the performance of their code.

The maintainers are also working on client instrumentation. They want OpenTelemetry to be able to extract data from web, mobile, and desktop applications. 

“OpenTelemetry is very focused on back end infrastructure, back end services, the stuff that people run inside of AWS or Azure or GCP,” Morgan explained. “There’s also a need to monitor the performance and get crash reports from their client applications, like front end websites or mobile applications or desktop applications, so they can judge the true end to end performance of everything that they’ve built, not just the parts that are running in various data centers.”

The promise of unified telemetry

At the end of the day, it’s important to remember the main goal of the project, which is to unify telemetry. Developers and operators are dealing with increasing amounts of data, and OpenTelemetry’s purpose is to unify those streams of data and be able to do something with it. 

Parker noted the importance of using this data to deliver great user experiences. Customers don’t care whether you’re using Kubernetes or OpenTelemetry, he said. 

“Am I able to buy this PS5? Am I able to really easily put my shopping list into this app and order my groceries for the week?” According to Parker this is what really matters to customers, not what technology is making this happen. 

“OpenTelemetry is a foundational component of tying together application and system performance with end user experiences,” said Parker. “That is going to be the next generation of performance monitoring for everyone. This isn’t focused on just the enterprise; this isn’t a particular vertical. This, to me, is going to be a 30 year project almost, in terms of the horizon, where you can definitely see OpenTelemetry being part of how we think about these questions for many years to come.”