Architecture
How falak-neo-minion, falak-neo-core and the analyst console fit together, and the architectural laws that keep them honest.
Falak Neo has three components and one wire contract. Every decision below exists to reduce the number of things that can be misconfigured between a syscall and an alert.
The shape of it
Linux kernel falak-neo-core consumers
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ eBPF probes │ │ ingest │ │ console │
│ exec │ mTLS gRPC │ enrichment │ REST/JSON │ SIEM / SOAR │
│ file_open │───────────▶│ detection │──────────▶│ data lake │
│ connect │◀───────────│ control │ │ your clients │
│ bpf/ptrace │ signed └──────┬───────┘ └──────────────┘
└──────┬───────┘ config │
│ ┌──────┴───────┐
┌──────▼───────┐ │ ClickHouse │ OCSF events
│ minion (Rust)│ │ PostgreSQL │ application state
└──────────────┘ └──────────────┘Architectural laws
These are numbered in the repository because they get quoted in code review.
Law 1 — OCSF first
Every event a sensor emits must be constructible from the OCSF object definitions before core accepts it. Field names mirror OCSF JSON attribute names 1:1, so protojson output is a valid OCSF document with no translation layer.
The practical consequence: there is no internal schema to drift from the external one, because there is only one schema.
Law 2 — Direct sensor-to-core streaming
There is no relay, no collector tier and no message bus between the sensor and core. One long-lived bidirectional mTLS gRPC stream per sensor carries EventBatch up and BatchAck down.
A Push fallback exists for environments where long-lived HTTP/2 streams do not survive an egress proxy, with identical semantics and one round trip per batch.
Law 3 — The sensor never blocks the workload
eBPF programs write to a ring buffer and return. Userspace drains it. If userspace falls behind, the ring buffer overflows and the sensor reports the drop count — it does not apply back-pressure into the kernel path, because a security tool that can stall a payment service is a worse outage than the one it was installed to prevent.
Law 4 — Tenancy is server-authoritative
tenant_id rides on the envelope, not inside the OCSF payload, because it is an authorisation fact established by the mTLS peer identity or the signed enrollment token. Core overwrites whatever a sensor sends before the event reaches storage.
Law 5 — No blind detection
A finding without at least one MITRE ATT&CK technique is rejected by the pipeline. This is what makes the coverage matrix a measurement rather than a marketing asset.
falak-neo-minion — the sensor
Rust, one per host, deployed as a DaemonSet or a systemd unit.
| Responsibility | Detail |
|---|---|
| Probe management | Attaches CO-RE eBPF programs, reports per-probe state on every heartbeat |
| Event construction | Builds OCSF v1.3 documents on the host, including K8s and container context |
| Classification | Resolves path_class for file events in-kernel so sensitive-path rules stay cheap |
| Batching | Closes a batch on count, age or byte limit — whichever trips first |
| Spooling | Disk-backed queue survives a core outage; the cursor advances only on BatchAck |
| Fast-path rules | Evaluates compiled LocalRule expressions so a kill decision skips a network round trip |
Steady state on a busy production node: under 1% of one core, roughly 80 MiB resident.
falak-neo-core — the brain
A Go monolith. Deliberately not microservices: the components below share a process because the coupling between ingest, enrichment and detection is genuine, and distributing it would buy operational complexity rather than scalability.
| Stage | What happens |
|---|---|
| Ingest | Schema validation, idempotent dedup over a sliding window, back-pressure hints |
| Enrichment | Kubernetes informer, cloud metadata, threat intel, asset criticality |
| Detection | Sigma-family rules plus behavioural and statistical correlation |
| Storage | OCSF events to ClickHouse (tenant-partitioned); state to PostgreSQL |
| Control plane | Enrollment, signed SensorConfig distribution, signed response actions |
| API | REST/JSON for the console and for your own clients |
The console
Next.js, multi-tenant, described in its own repository. Notably: it never queries ClickHouse directly. Every event read is brokered by core, so the tenant predicate is applied by the component that owns the data rather than by the component rendering it.
Data placement
| Data | Store | Why |
|---|---|---|
| OCSF event bodies | ClickHouse | Volume, immutability, time partitioning |
| Finding triage state | PostgreSQL | Mutable, low-volume, relational |
| Sensor policy | PostgreSQL, signed on publish | Auditable and versioned |
| Secrets | External secret store | Never in the application database |
A note on retention
Hot retention is what hunt and the alert feed query. Cold archive is OCSF parquet in object storage — queryable, but not interactively. Sizing the hot window is the single decision that most affects both cost and how far back an investigation can reach.
Failure modes, and what happens
| Failure | Behaviour |
|---|---|
| Core unreachable | Sensor spools to disk up to spool_max_bytes, then drops oldest and counts it |
| Ring buffer overflow | Events dropped, counter reported, console surfaces the sensor as degraded |
| Probe fails to attach | Sensor reports failed, continues with the rest, coverage matrix reflects the gap |
| Sensor killed | Heartbeat gap detected by core, which raises a finding — tampering is itself a detection |
| Clock skew | BatchAck carries core's time; sensors that drift beyond tolerance are flagged |
Something wrong or missing? Edit this page