What’s New: AWS Integration

AWS integration has been one of the most popular features among our customers, enabling seamless collection of logs, metrics, and events from AWS accounts for enhanced service monitoring within Observe. Over time, many customers have shared their experiences, use cases, and challenges with the existing integration.

Some of the most commonly reported issues include:

• Data sent via Firehose/Lambda traversed the public Internet, raising security and cost concerns.
• Limited filtering options for collecting CloudWatch metrics.
• Lambda timeouts when capturing snapshots of numerous AWS resources (e.g., 100,000+ SNS topics).

We’re excited to announce a significant update to the AWS integration, addressing these concerns and offering a more efficient, secure, and user-friendly experience.

Key Enhancements

Cost-Effective and Secure Data Transfer Over AWS’s Internal Network

Customers with strict compliance requirements, such as “no data transfer over the public Internet,” or those seeking to reduce data transfer (egress) costs will benefit from our updated integration.

The new workflow accumulates CloudWatch Logs, CloudWatch Metrics, and AWS Config snapshots in a customer-managed S3 bucket. A Lambda function (CopyObject) then sends these objects securely to Observe’s S3 bucket.

When data is transferred between Amazon S3 buckets, it typically uses AWS’s internal network, avoiding the public Internet. This approach not only enhances security and reduces latency but also lowers data transfer costs:

• Same region: No data transfer fees if both the customer’s S3 bucket and the Observe tenant are in the same region.
• Cross-region: Charges decrease significantly—from $0.05–$0.09 per GB to as low as $0.02 per GB, if the customer’s S3 bucket is in us-east-1 and the Observe tenant exists in us-west-2

Enhanced Filtering for CloudWatch Metrics

Customers have requested more precise control over which CloudWatch metrics to collect. Previously, all metrics from a region and AWS account were pushed to Observe indiscriminately. The new integration provides granular filtering options, allowing customers to:

• Near real-time metrics collection: Use custom filters to specify which AWS services (e.g., AWS/EC2, AWS/RDS, AWS/Lambda) and metric names to include or exclude. For instance, the following filter excludes the TruncateFinishedPoint metric from RDS instances:
  

  ExcludeFilters:
    - Namespace: AWS/RDS
      MetricNames:
        - TruncateFinishedPoint

• Tag-based filtering: For customers using the CloudWatch Metrics poller, metrics can now be filtered by tags. For example, the following configuration collects metrics from AWS/EC2, AWS/Lambda, AWS/RDS, AWS/SNS, and AWS/SQS namespaces where the Environment=Prod tag is present:
  

 

Efficient Resource Data Collection with AWS Config

In the past, Observe used a Lambda function to capture the state of AWS resources during integration setup and periodic updates. While this worked for most cases, it occasionally failed with large-scale resources (e.g., hundreds of thousands of SNS topic subscriptions) due to timeouts.

The updated integration now relies on AWS Config, a native AWS service, to collect resource-related data. AWS Config delivers configuration snapshots directly to an S3 bucket, which are then securely transferred to Observe. This approach eliminates Lambda timeouts and scales effortlessly with large AWS environments.

Experience the New AWS Integration

These enhancements reflect our commitment to addressing customer feedback and delivering a more robust AWS integration experience. Want to learn more? Visit the Observe booth (#459) at AWS re:Invent 2024, or come take Observe for a spin yourself with a free trial!