Keeping an eBPF sensor under 1% CPU, and what we gave up to get there

Every conversation about a runtime security sensor eventually arrives at the same question, usually from a platform engineer with an SLO: what is this going to cost me?

Our answer is under 1% of one core and roughly 80 MiB resident on a busy production node. Here is how, and — more usefully — what we traded away.

Where the time actually goes#

An eBPF sensor has four cost centres, and they are not equally sized:

1. The probe itself — code running in kernel context on the hot path
2. The transport — getting events from kernel to userspace
3. Userspace processing — parsing, enriching, serialising
4. Network — batching and shipping

Most naive implementations spend their budget in (3). Ours spends it in (1), deliberately, because kernel-side work is the only place where a filtering decision can prevent all the downstream cost.

Filter in the kernel or pay four times#

The single largest lever. An event that is filtered in the probe costs a few pointer dereferences and a comparison. The same event filtered in userspace costs a ring buffer slot, a copy, a parse, a serialisation and a network byte.

Concretely, on a node running a CI workload:

Filtering point	CPU	Events shipped
None	4.8%	41,000/s
Userspace only	3.1%	2,900/s
In-kernel	0.6%	2,900/s

Same events reach core. Five times the cost to get there.

This is why exclude_path_prefixes and exclude_comms are evaluated inside the probe against a BPF hash map rather than applied after the fact — and why always_include_path_prefixes is checked first, so a broad exclusion can never accidentally suppress /etc/shadow.

Ring buffer, not perf buffer#

BPF_MAP_TYPE_RINGBUF (kernel 5.8+) gives a single shared buffer with global ordering and a reserve/commit API that avoids a copy. The older perf buffer is per-CPU, which means N buffers to poll, N wakeups, and reassembly in userspace to recover ordering.

Switching cost us a week and bought roughly 25% on high event rates, plus correct global ordering, which made several correlation rules simpler.

On kernels below 5.8 we fall back to the perf buffer and the sensor reports the degradation rather than hiding it.

Never block the kernel path#

This is a hard rule, not a tuning parameter. The probe reserves a slot, writes, commits, returns. If the ring buffer is full, the reserve fails and the event is dropped and counted.

The alternative — applying back-pressure into the kernel path — would mean a security sensor that can stall a payment service. That is a worse outage than the one we were installed to prevent.

Drops are reported on every heartbeat as events_dropped_ringbuf, and the console marks a sensor degraded when they are sustained. A visible drop is recoverable; a silent stall is not.

What the userspace does not do#

The Rust userspace is deliberately thin:

• It does not parse the full command line in the probe. It copies argv bytes and parses lazily, in userspace, only for events that survive filtering.
• It does not resolve container identity per event. Cgroup id to container is a cache lookup, refreshed from the CRI on cgroup creation.
• It does not hash executables inline. Hashing is done asynchronously with an inode-keyed cache, so a repeatedly-executed binary is hashed once.
• It does not allocate per event. Events are written into a pre-allocated arena and the arena is reset per batch.

That last one is the difference between an 80 MiB steady state and a sawtooth that trips a memory limit at 3am.

Sampling, and what it costs you#

Per-class sampling is available and we are careful about recommending it.

Safe to sample: 1001 File System Activity, if always_include_path_prefixes covers what matters. The sample applies only to the residue.

Sample with care: 4001 Network Activity. Volume-based detections tolerate it; statistical analytics like beaconing detection degrade in accuracy proportionally.

Never sample: 1007 Process Activity, 1003 Kernel Activity, 1005 Module Activity. These are low-volume and high-value, and sampling them means a correlation silently misses its other half. A rule that requires an exec and a connect, with the exec sampled at 60%, fires 60% of the time — and you will not know which 40% you missed.

The numbers, in context#

Measured on a 16-core m6i.4xlarge running a mixed Kubernetes workload, default policy, over 24 hours:

Metric	p50	p99
Sensor CPU (% of one core)	0.41	0.87
Resident memory	78 MiB	84 MiB
Events emitted	3,100/s	9,400/s
Ring buffer drops	0	0
End-to-end latency (exec → core ack)	340 ms	890 ms

Build fleets are the pathological case: 3–5× the file event volume, almost all of it build churn with no security signal. That is what the low-overhead policy exists for — include_only on file_open plus prefix exclusions for /builds and /nix/store brings a build runner back into the same envelope.

What we gave up#

Three things, honestly.

Full argv on every exec. We cap the copied argv at a configurable length. A pathologically long command line is truncated, and the event says so. We have not yet met a detection that needed more than the default.

Environment variables, by default. Collecting them is a policy option, off by default. They are frequently large, frequently sensitive, and rarely the thing that catches an intrusion.

Per-event container resolution. The cgroup-to-container cache can be briefly stale for a container in its first few milliseconds. In practice this affects the container's own init process and nothing else, and the event carries the cgroup path so core can backfill.

Each of those is a deliberate trade, written down, with a policy switch where the trade is reasonable to make differently. That is the general shape of how we think about sensor overhead: not "how fast can we make it" but "what exactly are we choosing not to see, and does anyone know we chose it".