Observe’s Response to CVE-2024-4323

Fluent-Bit Memory Corruption

We are aware of CVE-2024-4323, a vulnerability for fluent-bit. While details are still emerging, Observe uses open source agents including fluent-bit to collect endpoint data and we take security issues seriously. Please be aware that there are public exploit POCs now for this CVE.

Fluent-Bit is used for log collection in Observe’s Host Monitoring and Kubernetes (Kustomize Manifests and Helm Charts) integrations. The fluent-bit package includes a built in HTTP server. It supports an /api/v1/trace endpoint, used for monitoring fluent-bit’s own health. This endpoint in has a memory corruption vulnerability. In order to be exploited, the built-in HTTP server must be on; by default, it is off. See the fluent-bit configuration reference for more detail. The server typically listens on TCP port 2020 (but can be configured for other ports using the http_port configuration variable). 

• In Observe’s usage of Fluent-Bit for Host Monitoring, the HTTP Server is off, so CVE-2024-4323 is not exploitable.
• In Observe’s usage of Fluent-Bit for Kubernetes, the HTTP Server is on, as it is used to collect runtime diagnostic metrics. However, because this port is only accessible internally within the Kubernetes cluster, exploitation of CVE-2024-4323 would require the ability to initiate malicious traffic from within the Kubernetes cluster.

Regardless of known exploitability, we recommend that customers update any vulnerable installs to the latest version of fluent-bit, as follows:

Host Monitoring App Update

CVE-2024-4323 is not exploitable by default in Observe’s logs collection for Host Monitoring, as the affected HTTP Server is configured to be off. As of May 21, 2024, the installer script for host monitoring uses the latest version of fluent-bit (v3.0.4), which is patched against this vulnerability. For customers who want to upgrade fluent-bit to v3.0.4, please follow Observe’ documentation to use the installer script to install the latest version of fluent-bit (v3.0.4). If desired, follow these instructions to uninstall fluent-bit, telegraf, and/or osquery.

Kubernetes App Update

If you use Observe’s Kustomize manifests for Kubernetes, we have updated the manifests (v1.4.0) to use fluent-bit (v3.0.4). New installations will be safe against this vulnerability as of May 21, 2024. For customers who want to upgrade fluent-bit to v3.0.4, please follow Observe’ documentation to use the updated manifests (v1.4.0).

kubectl apply -k 'https://github.com/observeinc/manifests/stack?ref=v1.4.0'

If you use Observe Helm Charts for Kubernetes, we have updated the helm charts (v0.4.28) to use fluent-bit (v3.0.4). New installations will be safe against this vulnerability as of May 21, 2024. For customers who want to upgrade fluent-bit to v3.0.4, please follow Observe’ documentation to use the updated helm charts (v0.4.28).

helm repo update
helm upgrade --namespace=observe observe-stack observe/stack --reuse-values

Fluent-bit Detection

Determining if you have vulnerable fluent-bit binaries present in your environment depends on the type of data that you’re collecting into Observe. If you have a default install of the Observe Host Monitoring app, you can use the Server/Process Hashes dataset to find the SHA256 hashes from any running fluent-bit binaries (including binaries that are not sending data to Observe). 

A similar approach can be applied to logs from EDR or MDR products such as Cybereason or Crowdstrike. For example, you could use the following OPAL to list the host details and fluent-bit SHA256 values that are detected from a raw datastream containing OSQuery snapshots:

filter name = "process_hash_snapshot"
drop_col name
make_col host:string(tags.host),
         ec2_instance_id:string(tags.ec2_instance_id),
         ec2_az:string(tags.ec2_az),
         snapshot:log.snapshot
drop_col tags, log
make_col pivot_array(array(snapshot), "name", "sha256")
drop_col snapshot
make_col "fluent-bit_hash":string(pivot_array_array_snapshot['fluent-bit'])
drop_col pivot_array_array_snapshot
filter is_null(@."fluent-bit_hash") or (@."fluent-bit_hash" != "")

You can also use the Host Monitoring app’s Network Connections dataset to find unexpected use of port 2020, perhaps in conjunction with the Basic Threat Intelligence App’s listings of non-public or IaaS IP space.

Please use the in-app contact tool or the Observe Slack Community if you would like help with detection.