My Master's capstone had a simple brief: detect hostile traffic on a network before it does damage. The result was a hybrid intrusion detection system that scores 0.95 F1 and 0.98 ROC AUC on CICIDS2017, a benchmark dataset built from real attack traffic. This is the story of why it took three models to get there.
One model kept lying to me
The first version was a single Random Forest, and on paper it looked brilliant. The problem with intrusion detection is that the interesting traffic is rare. A model can score high accuracy by waving almost everything through, and the one attack it misses is the one that matters. Precision and recall told a much uglier story than accuracy did.
The hybrid idea
The fix was to stop asking one model to do two jobs. The final system splits the work three ways:
An Isolation Forest goes first. It knows nothing about attack types; its only job is to flag traffic that looks unusual. Because it trains on the shape of normal traffic, it catches oddities that no labelled dataset could have taught it.
A Random Forest handles the known threats. Trained on labelled attack classes, it is the workhorse that says "this is a port scan" or "this is a brute force attempt" with high confidence.
An autoencoder watches for the quiet stuff. It learns to compress and rebuild normal traffic, so anything it reconstructs badly is by definition something it has never seen. That reconstruction error turned out to be the best early signal for novel attack patterns.
The insight that made it work: the three models disagree in useful ways. Where they agree, you get confidence. Where they split, you get a shortlist worth a human's attention.
What I would tell past me
Spend the first week on the data, not the models. CICIDS2017 has quirks, duplicated flows and class imbalance that quietly poison results, and every hour spent cleaning paid back ten in training. And put the dashboard up early: the Streamlit front end turned a pile of metrics into something people could poke at, and half my findings came from watching it with real traffic patterns.
The system now lives as a password-gated live demo, and the same instincts drive ThreatLens, my live vulnerability feed. Detection work rewards the same habit as data engineering: distrust anything that looks too clean.