State of Observability Report: 2023 Key Findings

We’re all trying to keep our finger on the pulse of the industry, and we at Observe are no different. In our effort to continue to assess our users’ most pressing needs we enter the third year of the longest running State of Observability Report! You can download the full report here for more data and analysis, but below are some highlights from this year’s survey of IT decision makers. 

We were surprised by the extent of data growth that organizations are anticipating, and at a time when an unfavorable market is providing IT with more challenges. Couple that with tool sprawl and unfavorable market conditions, and you have quite the set of challenges. If these pain points sound familiar, or your organization is struggling on its observability journey, read on because there is a path to better observability without the tool sprawl and high costs.

Market Conditions Make Observability Crucial

Unfavorable market conditions have been an unavoidable topic this year and they’ve made day-to-day work for IT teams all that more challenging. The report shows 61% or organizations have experienced reduced budgets, and 55% have had to reduce headcounts. Those impacts would be bad enough on their own but 48% of organizations say that layoffs have affected their ability to troubleshoot incidents. Despite the macroeconomic conditions that have impacted organizations of all sizes, observability remains a high priority for 63% of companies. And even though budget limitations is the second most cited challenge to observability, 81% of organizations are planning to adopt a new observability tool in the next year (showing no decrease since 2022). This indicates that even though tool sprawl is a pain point for many organizations, legacy tools are creating insurmountable challenges or failing to fulfill current observability needs.

Observability Maturity Is A Work In Progress

We often characterize observability as a journey, and in the past we’ve seen a high number of organizations saying they are “doing observability”, yet they are still suffering from slow troubleshooting and siloed tooling. This year a staggering 91% of organizations say they currently practice observability. However, only 11% of organizations think their entire environment is currently observable, while 51% feel that ¾ or less is observable. This indicates that maturity is still highly variable as organizations look to reign in their growing, and increasingly distributed, environments. 

Complexity of the environment was considered the biggest challenge to observability for the second year in a row. However, some challenges have decreased dramatically, for example 45% of companies cited organizational buy-in as one of the top challenges in 2021, but it dropped to 24% in 2023 as the awareness and importance of observability has grown considerably. While organizational dynamics are more favorable (68% of orgs are planning to hire for SRE roles in the coming year!) the issues of limited visibility, siloed tooling, and data that can’t be correlated will continue to slow maturity.

Tool Consolidation Moves To The Front Burner

Tool sprawl, is it an actual challenge or just a boogeyman observability vendors like to harp on? Well 65% consider observability tool sprawl a significant pain point and even more, 86%, say consolidation is a priority in the coming year. Just how bad is the situation? On the more extreme end of the spectrum 37% of orgs cite having 11 or more observability and monitoring tools in their organization today. Increased costs, difficulty correlating data and operational complexity are the leading challenges that result from it. Even 45% cited tool sprawl resulting in slower troubleshooting, meaning their numerous tools are actually having a negative impact instead of providing value. 

While having many observability and monitoring tools has become the norm, the reality is it does create challenges for organizations and there are tangible benefits to reducing the number of siloed tools. Cost concerns (I harken back to aforementioned market conditions) may be the driver for many orgs to prioritize tool consolidation, but ditching legacy tooling to adopt a modern architecture can simultaneously result in better observability, faster troubleshooting and longer data retention.

Data Growth Is On Track For New Highs

As the saying goes, “data is the new oil” so having lots of it should be a good thing right? Not if you can’t actually use what’s being generated. Currently 51% of organizations expect their organization’s data to grow by more than 75% in the next 12 months. That includes the 19% that expect data growth of 100% or more. While Observe has previously cited past third-party research expecting data growth of 40% year-over-year there is indication that growth rates will increase for many organizations. The volume of data being ingested by observability tools also continues to trend upwards with 19% of orgs taking in more than a Terabyte a day, compared to 10% last year.

Cost continues to be the enemy of growth for many organizations and 64% say they have sampled or reduced incoming observability data to bring down costs. While this may be a functional stop gap for some companies, it means sacrificing potentially crucial data and limiting the observability of one’s environment. Retention is also impacted by costs as only 41% are able to keep log data for longer than a few weeks, something that can be a risk depending on security and compliance requirements.

Incident Response Times Show Little Progress

The average time it takes to investigate an incident remains high, 64% say a day or longer is the norm at their organization. Considering that 58% or organizations are citing 6+ incidents a month on average, that’s a lot of time spent troubleshooting and having customers suffering with impacted service. While our data shows the number of incidents has started trending downward over the past three years, the average time to investigate has not seen a similar decrease. We’ve experienced with our own customers that some come to us “stuck” at a certain MTTR and needing a new approach to break through that barrier.

You Need Better Observability

We can expect that most organizations still have serious work to do to achieve full observability, and it will take concerted effort. Multicloud and microservices have made incident response even more complicated and there’s no shortage of emerging tech that needs to be observed. Faster troubleshooting and being able to easily pinpoint the cause of an incident is invaluable, and currently many IT teams are being asked to do this with sub par tools and fewer resources. Considering the data growth that most organizations are projecting, it is essential that you get ahead of the problem now and not wait a year or two. With Observe we can get you on the path to solving your observability data problem regardless of what stage in your journey you’re at. With the unique architecture of The Observability Cloud we can help you solve the observability data problem and get better observability while still saving money over legacy tools that aren’t architected for data growth.

---

If you want to dig deeper into the data, you can find the full State of Observability Report here. And if your organization is one of the ones aiming to consolidate tools, keep more of their data, and achieve better observability then get access to Observe here.