01 / THE PROBLEM
Finding useful signals
in network traffic.
A packet is a small piece of a larger conversation. This project explores botnet detection by grouping packet records into flows and extracting features that describe how those conversations behave.
The README frames the project around botnet and IRC traffic classification. The published source focuses on the data pipeline: labelling packet records, identifying bidirectional flows and calculating inputs for a machine-learning stage.
02 / THE DATA PIPELINE
Give each conversation
a consistent identity.
- Load and label packet recordsRead a CSV into a GraphLab SFrame, convert hexadecimal TCP flags to integers, and assign bot labels from a list of known endpoints and endpoint pairs.
- Build a bidirectional flow keyCombine the two IP addresses, their ports and the protocol. Order the endpoints consistently so packets travelling in opposite directions can share an identifier.
- Sort and segment the trafficSort by flow key and timestamp, then assign sequential flow numbers. The loop uses key changes, TCP FIN flags and a 3,600-second elapsed-time threshold as flow boundaries.
- Aggregate and retain the resultGroup packet records by flow, join computed statistics back to the data and persist the flow assignments for subsequent processing.
Keeping both directions together makes it possible to describe a conversation rather than treating each packet as an isolated observation.
03 / FEATURE ENGINEERING
From packet fields
to behavioural features.
The visible code includes several complementary ways of describing traffic. Each captures a different property of a flow rather than relying only on a raw packet record.
- Directional packet ratio
- Count forward and reverse packets using the canonical endpoint ordering. Compute reverse-to-forward volume, with a sentinel value of −1 when there are no forward packets.
- First packet length
- Find the earliest timestamp in each flow and join it back to the packet records. Average the lengths if more than one record shares that earliest time.
- Average packet rate
- Divide the packet count by the flow duration, with a zero-duration guard. The excerpt references a duration table whose construction is not included.
- Protocol and payload clues
- Flag UDP traffic and define a helper for recognising packets with no payload from segment lengths and protocol headers.
Other selected columns include duration, exchanged bytes, packet-length variation and equal-length packet ratios. Their complete derivations are outside the published sample.
04 / SYSTEM DESIGN
The feature pipeline
supports the classifier.
The README diagrams connect the flow generator to a separate machine-learning module. The first view establishes the detection task; the second shows how extracted features feed training, testing and optimisation.
This separation gives the feature pipeline a clear role: produce a consistent representation of each traffic flow so that modelling approaches can be evaluated on the same inputs.
05 / WHAT THE REPOSITORY SHOWS
A feature-engineering
implementation to inspect.
The public main2.01.py is an exported notebook-style source sample. It shows the labelling rules, flow-building loop, aggregation logic and feature calculations. The referenced traffic CSV and complete modelling code are not included.
The README describes deep learning and comparison across classification algorithms. The available source does not include the fitted classifiers, train/test split or recorded metrics needed to reproduce those results. This case study therefore presents the demonstrated data-engineering work without an accuracy claim.
06 / LOOKING BACK
The representation
sets up the result.
The engineering work sits in the transition from packet rows to structured flows: making identifiers consistent, preserving direction, handling flow boundaries and deciding how to aggregate observations.
A stronger follow-up would test reverse-direction matching and boundary cases explicitly, publish the feature schema, and evaluate held-out captures or hosts. Because the initial labels are based on known endpoints, the evaluation should check whether a model generalises to new traffic rather than simply recognising those identities.
Botnet recall, precision and the false-positive rate on benign traffic would make a classifier comparison more useful than a single overall accuracy score.