Here is a model with 99.9% accuracy: it predicts that no transaction is ever fraudulent. If fraud is one transaction in a thousand, that model is almost always right – and completely useless. This little trap sits at the heart of fraud detection with machine learning. The problem isn't that fraud is hard to model; it's that fraud is rare, adaptive and expensive to get wrong in both directions, and the usual ways of measuring a model fall apart under those conditions.
Why fraud detection is a special machine learning problem
Four properties shape every design decision:
- Extreme class imbalance. Fraudulent cases are a tiny fraction of the total.
- An adversary. Fraudsters change tactics when they get blocked. Patterns that worked last quarter fade.
- Delayed and noisy labels. A chargeback can arrive weeks after the transaction; some fraud is never reported; some "fraud" reports are buyer's remorse.
- Asymmetric, real costs. Missing fraud loses money. Blocking a good customer loses the sale, annoys them and may lose them for good.
A good system is designed around these facts, not just around an algorithm.
Handling imbalanced data
The first instinct is to "balance" the dataset. Some techniques help; none are magic.
- Class weights. Tell the algorithm that errors on the fraud class cost more. Simple, and usually the first thing to try with gradient boosting or logistic regression.
- Undersampling the majority class. Train on all fraud cases and a sample of legitimate ones. Faster training, with some information lost.
- Oversampling / SMOTE. Duplicate or synthesise minority examples. Can help, but synthetic fraud cases may not resemble real fraud, and it's easy to leak synthetic points into validation data.
One thing that matters more than any of these: if you resample, the model's raw scores no longer reflect real-world probabilities. Recalibrate, or at least choose thresholds on data with the true class balance.
Features matter more than algorithms
Gradient boosting on well-engineered features is the standard approach for transaction fraud, and the features carry most of the weight. The strongest ones usually describe behaviour relative to a baseline:
- Velocity: number of transactions, cards or failed attempts per account, device or IP address in the last minutes, hours or days.
- Deviation: this amount compared with the customer's typical amount; this merchant category compared with their usual ones.
- Identity and device signals: new device, mismatch between billing and shipping country, account age, email age.
- Network features: how many accounts share this device, card or address. Fraud rings reuse infrastructure.
- Time: unusual hour for this customer, a sudden burst after a long dormant period.
Building these reliably in real time – often within a strict latency budget at checkout – is usually the hardest engineering part of the system.
Precision vs recall: choosing the trade-off
Accuracy is meaningless here. The metrics that matter are:
- Precision: of the transactions we flagged, how many were actually fraud?
- Recall: of all the fraud that happened, how much did we catch?
They pull against each other. Lower the threshold and you catch more fraud (higher recall) while flagging more good customers (lower precision). Use the precision-recall curve and its area (PR AUC) to compare models; ROC AUC tends to look flattering on heavily imbalanced data.
Then pick the operating point with money, not intuition. A worked example with hypothetical numbers: suppose an average fraudulent order costs you the goods plus a chargeback fee, a falsely declined order costs you the margin on that sale plus some chance of losing the customer, and a manual review costs an analyst's time. Put those costs against the confusion matrix at each threshold and choose the one with the lowest total cost.
Many teams use more than one threshold:
| Score band | Action |
|---|---|
| Low risk | Approve automatically |
| Medium risk | Step-up check (e.g. 3-D Secure, SMS code) or manual review |
| High risk | Decline or hold |
The middle band is where most of the value lives: it lets you be strict without punishing good customers outright.
Supervised models and anomaly detection together
Supervised models learn from labelled past fraud, so they're strong on known patterns and weak on genuinely new attacks. Anomaly detection – isolation forests, autoencoders, simple statistical outlier rules – flags behaviour that is unusual regardless of labels. In practice, production systems layer them:
- Hard rules for clear-cut cases and regulatory requirements (blocked countries, known bad cards).
- A supervised model producing a risk score.
- Anomaly signals as extra features or as a separate alert stream for analysts.
- Human review for the uncertain middle, whose decisions become new labels.
That last loop matters. Analysts' decisions are training data; a clean case-management process makes the model better every month.
Keeping a fraud model current
Because fraudsters adapt, a fraud model goes stale faster than most. Plan for:
- Frequent retraining, with labels matured long enough that chargebacks have arrived.
- Monitoring of score distributions, decline rates and fraud rates per segment, with alerts on sudden changes.
- Champion/challenger testing, so a new model proves itself on live traffic before taking over.
- Explanations for every decision (reason codes, SHAP values), for analysts, for customer support and, in many jurisdictions, for regulators.
Mraviko builds this kind of scoring pipeline into payment and marketplace platforms, and the monitoring is always the part we scope first.
Checklist for a fraud detection project
- Clear definition of the fraud label and how long labels take to mature
- Time-based train/test split, never random
- Velocity, deviation and network features computed identically offline and in real time
- Models compared with PR AUC and cost at the chosen threshold
- Thresholds set from real costs, with a review band
- Feedback loop from analysts back into training data
- Drift monitoring and a retraining schedule
Closing thought
Fraud detection with machine learning works when it is treated as a living system: rare-event modelling, features that capture behaviour, thresholds set from real costs, and a feedback loop that keeps up with an opponent who is also learning. The algorithm is the easy part. Everything around it is what stops the fraud.
