Observe Direct Fall 2022: Welcome To The Observability Cloud

Welcome back to Observe Direct! If you didn’t catch our announcement video you can find it here. In this blog post, we’ll be summarizing the announcements. This includes the culmination of our work over the past five years which has resulted in The Observability Cloud. This is Observe’s entirely new approach is focused on eliminating data silos and being able to quickly access relevant context while troubleshooting.

We’re also announcing new product features including: distributed tracing, a slew of cost management capabilities, and Observe Apps for GCP, GitHub, and OpenTelemetry.

Business Update

We are pleased to report that Observe continues to grow! Observe now has more than 60 customers and over 800 monthly active users. This year alone our users have pulled tens of petabytes of data into Observe, and are performing 40 million Snowflake queries a day (almost 2% of Snowflake’s daily query volume)! That’s up from 25 million since the previous Observe Direct in May. We’ve also set a new milestone for our team as the headcount is over 100 as we continue to expand.

Introducing The Observability Cloud

The Observability Cloud is the evolution of Observe and it is about having all your data in one place, not having a mishmash of backends and silos for different data types, and not having to pay for multiple tools to accommodate for various use cases. The Observability Cloud is comprised of the Data Lake, Data Graph, and Data Apps.

Data Lake: Observe has one destination for all of your observability data – logs, metrics, traces, and everything else all go to Snowflake. By keeping data together in one data lake we can correlate your data with context. This also allows us to separate storage and compute to power our usage-based pricing. 

Data Graph: Remember the “Data Universe” from the previous Observe Direct? That’s the Dataset Graph being used to navigate your world of data. Datasets are a core element in Observe and are the “things” you care about as a user – like Pods, Load Balancers, or Customer Tickets – and the Data Graph is those Datasets linked together. Linked Datasets enable rapid navigation to more context during investigations.

Data Apps: Apps are an easy, self-service, way to get data into Observe, but they offer a lot more than just integrations. Our Apps package integrations, Datasets, Dashboards, monitors, and more to get you up and running in minutes with observability data from your favorite services. Custom applications can also be created for customer-specific use cases. More on Apps further down this page.

Product Updates

Today Observe released several new features to The Observability Cloud supporting a broader range of use cases, simplifying usage, and enabling users to better manage their observability costs. This includes new distributed tracing visualizations and workflows along with OpenTelemetry instrumentation. There are several entirely new Apps along with expansions to some of our previously released Apps. We’ve also added three cost management features to give users full transparency into their usage as well as guard rails they can set to reduce unintended spend.

Tracing For All Your Workflows

Tracing is not entirely new to Observe, we’ve long been able to ingest tracing data and link it with other data types such as logs, and many of our customers have been actively doing this. However, you had to use OPAL (Observe Processing and Analysis Language). While that worked for power users we wanted to democratize the tracing experience and meet user expectations for typical tracing workflows. Now there’s no OPAL necessary to do things like visualize spans and that functionality is ready to go out of the box.

With Observe, it’s easy to use tracing for common use cases such as seeking out a particular error to find a root cause swiftly. However, because Observe has 13-month data retention by default, SREs can view historical data to really understand the health of a service over time. Whether it’s an engineer using trace Dashboards to filter to a specific error, or SREs using the service graph to get a broader picture and assess reliability, Observe supports your tracing-related workflows.

Apps: GCP, OpenTelemetry, and More!

We introduced Observe Apps earlier this year to provide users with an App Store-like experience and an intuitive UI for installing and configuring out-of-the-box integrations. We first launched Kubernetes, AWS, and Linux Host Monitoring Apps that include various Datasets, Dashboards, and monitors out-of-the-box. 

Apps are more than just mere integrations. They streamline the process of collecting data, and also get users up and running quickly with relevant insights via curated selections of Dashboards and monitors that cover commonly used and relevant data. Our Apps for public cloud providers – like the AWS and GCP App – contain bundles of integrations so you can start ingesting data from a multitude of services from those providers, and we are constantly expanding the reach of those Apps.

We are committed to constantly expanding our range of Apps and we’ve now added OpenTelemetry, GCP, Prometheus Node Exporter, Jenkins, GitHub, and GitLab. You can read more about the GCP App here, and head to our docs for a full list of integrations. More info to come on the OpenTelemetry app!

Cost Management Controls: Transparency And User Consent

Does cost management sound foreboding? Fear not because, Observe can and does impart significant cost savings to our customers. However, with many observability and monitoring products, users have little to no control over cost. Typically, your best recourse against high costs is often limiting data ingestion or the number of users. However, that can reduce the efficacy of the very tool you’re paying for! Because Observe utilizes usage-based pricing, the variable that users must consider is any usage that consumes compute, which puts control directly in their hands.

Your team is probably adhering to a set budget and you want to be sure you stick to it. Since usage-based pricing can be a new experience for many users we want to ensure customers don’t encounter any cost-related surprises. To achieve this we’ve engineered several methods to help you keep what you pay in line with your own expectations.

Usage Dashboard: All customers are able to use Observe Dashboards to track their own detailed usage data. This lets you know which users, and which actions (monitors, queries, etc…) used however many credits. This can help you review past consumption and keep track of any unexpected spikes to zero in on why they happened and how often they have happened.

On-Demand Acceleration: Usage-based pricing isn’t about getting users to run up their own bill haphazardly just for using the product, that’s why we’ve introduced the concept of user consent to on-demand acceleration. You can think of this as a guardrail for certain queries. Is a new user querying a year’s worth of data by mistake? On-demand acceleration will let them know that might take a while and cost more than the average query. They could instead opt to just query more recent already accelerated data than that full year’s worth, the user is given the choice. If they still want to proceed with the initial query they can do so, but we want them to know when something might have a pronounced impact on credit burn.

Credit Manager: The Credit Manager is not the first resort, but if you want to be absolutely certain you can stay within a certain budget then you can use the Credit Manager to impose a hard limit on queries and transforms. It can be overridden at any time so if you’re thinking “the last thing I need is something to stop my queries when I’m in the middle of a high-stakes high-pressure situation” then don’t worry.

Metric Expression Builder Comes To Dashboards

Our Metric Expression Builder makes it easy to set up alerts and monitors based on multi-metric expressions via a simple GUI. When we announced the Metric Expression Builder earlier this year we mentioned it would eventually be coming to the Dashboard UI, and it’s here. Now you can use the Expression Builder as part of your Dashboarding workflow as another way to craft useful alerts and monitors.

Looking Ahead

Observability is all about the data, and the Observability Cloud is about taking an entirely new approach to your data. The Observability Cloud is based on a modern architecture consisting of inexpensive cloud storage and elastic compute. Event data is compressed an average of 10x and stored for 13 months for no additional charge above AWS S3 costs. Compute is charged based on usage (to the nearest millisecond) when, for example, a user searches, an alert is fired or new event data is accelerated. The Observability Cloud architecture changes the economics of Observability – freeing users from having to sample, filter or eliminate event data in order to manage costs.

As for additional new releases, we’ve got many more Observe Apps on the way. We’ll then be rounding out the public cloud trifecta with an App for Microsoft Azure. There will also be a slew of new service integrations added to our GCP App. Stay tuned for more App updates as we get into 2023! In the meantime, you can check out this post from Observe CEO Jeremy Burton as he reflects on our path to The Observability Cloud.