For several weeks now I've been working with a small team on Synthaea, a fully open-source detection and response platform for workstations and servers (an EDR, with the ambition of growing into an XDR), released under the Apache-2.0 license. The project's website sums up the idea in one sentence: endpoint security that behaves like a living organism. Kernel-level sensors (eBPF on Linux, ETW on Windows), rules mapped to ATT&CK techniques, machine learning running on the machine itself, and an agent that defends itself. The project is upfront about where it stands: it's an implementation in progress, not a finished product.
My part is the Linux side, and above all validation on real systems: Debian, Ubuntu, Fedora. This post isn't an architecture tour, the project site does that. It's an account of four moments where something I thought I knew didn't survive a measurement.
Project website — synthaea.com
GitHub repository — synthaea-lab/edr
A green build that contains nothing
The first moment is the silliest and the most instructive. To validate the agent on an Ubuntu virtual machine, I run the test script: build, install, start. The build says "finished", everything is green. Yet at startup the agent announces that eBPF is unavailable and falls back to a degraded mode, which in turn fails on a permission error.
The real cause had nothing to do with permissions. The VM's disk was too small to install the toolchain, and one component (bpf-linker, which builds the eBPF programs) had never been installed. The project then compiles without a single error, but without embedding any probe: an agent that runs, sees nothing, and complains about the wrong thing. A whole series of measurements went in the bin before I went back to the first error message instead of stopping at the last one.
The test script now refuses to go on without that component. The rule I take from it: for software that depends on the machine it runs on, "it compiles" tells you almost nothing. You need a test that proves the sensor produces an event, not just that it starts.
Debian loops, Ubuntu doesn't: my assumption was wrong
Another surprise, on the systemd service shipped with the agent. It runs as an unprivileged user with a kernel capability set cut down to the strict minimum, which is exactly what you want from a security agent. On Debian 13, that service restarts in a loop: attaching the probes fails.
After a few measurements, the culprit is a kernel setting, perf_event_paranoid, which Debian sets by default to a stricter level than the service accepts with its current capabilities (that's my interpretation; what I measured is that lowering it one notch makes the service start and stay up). I was then convinced Ubuntu, whose default is even stricter, would have the same defect. I measured: no. On Ubuntu 24.04, with its default setting, the same service starts, attaches everything and sees events. The two distributions evidently don't treat this setting the same way, and my "stricter breaks more" intuition was worthless.
The options going forward aren't free: lowering the setting protects the whole machine less, giving the service a very broad capability looks a lot like running it as root, and rewriting the attach step to avoid that kernel feature would mean converting around fifty programs. So it isn't mine to decide alone; the problem is written up, numbers included, in an open issue so the team can choose.
Being wrong in public, then fixing it
On Fedora, which enables SELinux in enforcing mode by default, I wanted to prove that the agent was blocked by no rule of the security policy. My script counted denials with the usual tool and printed zero. I wrote "0 denials" in several code reviews.
While rereading the lab documentation for something else entirely, I hit this sentence: on this machine, the audit search tool is blind, because the audit daemon isn't running. It reads a file that doesn't exist and answers "no matches" whatever happens. My zero proved nothing. I redid the measurement at the right source, the system journal: sixteen denials since the machine booted, the most recent dated September 30 (they came from earlier packaging defects, since fixed), and none on the day of my tests. So the number was right, but I had no right to state it back then.
I posted the correction on the reviews concerned, with the method, and the script now counts from the journal. Nothing heroic: it's the kind of mistake you avoid by asking yourself, before writing down a negative result, whether the measuring tool could have said anything else.
Reading a web server's logs without drowning the analyst
The workstream where I spend the most time: having the agent read a web server's access logs (nginx, Apache) and a MySQL or MariaDB error log, to detect attacks the kernel sensor can't see, such as a control file dropped on a web server and then executed. The engineering problem is volume: a busy site writes thousands of lines a minute, and sending every request to the detection server would be both costly and pointless.
The approach: only emit an event for a request that looks like an attack (SQL injection, path traversal, a "web shell"-style request, a scanner's signature), and summarize everything else in a single event every sixty seconds (request count, 4xx and 5xx errors, distinct clients, the clients failing the most). A cap also limits the number of events per signature type and per window. It's deliberately per signature: a scanner sending a thousand injection attempts must not be able to use up the quota of the single web shell request that really matters.
From log to alert: only the request that precedes a shell spawned by the web server is surfaced as evidence.
The real test is a real nginx. I send one request from each family, then copy a command interpreter under the web server's name so that it spawns a shell. The agent produced the four expected events, the one-minute summary and the web shell alert. But it named the wrong request: the SQL injection sent first, not the web shell request that came right before the shell.
The explanation: the agent timestamps a request at the moment it reads the line, not when it was received. When several lines are read in one go, they share the same timestamp, and the correlation rule picked "the closest in time", that is, the first of a series of ties. The fix fits in one sentence (on a tie, keep the last one read, the one that comes last in the log's order) and the unit test that goes with it reproduces exactly that case. No test on simulated data would have found it: it took the burst of requests from a real server to surface it.
What I take away
The common thread in these four stories isn't technical difficulty. It's the gap between "it should work" and "I saw it work, and I know what my measuring tool would have said if it hadn't". For security software, that gap is the one between a promise and a protection. The project's test tooling goes in that direction: one script per scenario, a PASS or FAIL per check, a log kept so someone else can reread it.
Synthaea is still early in its life: the sensors and the rules engine exist, while the layer that gathers cases across several machines and the engine that makes each installation unique remain to be built. If the subject speaks to you, the code and the discussions are open, and field feedback, bad surprises included, is exactly what a project like this needs. As with the rest of this blog, these techniques are for your own systems or an authorized setting.