Ask a data scientist which machine learning algorithm they reach for on a spreadsheet-shaped problem – customers in rows, attributes in columns – and the honest answer is usually boring: start with a linear model, then try gradient boosting. Not a neural network. Not whatever was announced last month. That habit exists for good reasons, and understanding them is the fastest way to make sense of the machine learning algorithms you keep hearing about.
This guide walks through the classic family tree, from linear regression to gradient boosting, explaining what each algorithm actually does, where it shines and where it breaks.
Linear regression: the baseline you should always run
Linear regression predicts a number as a weighted sum of inputs. A house price might be modelled as a base value, plus a certain amount per square metre, plus a premium for a garden, minus a discount for distance to the city centre. Training means finding the weights that minimise the squared difference between predictions and real prices.
Why it still matters:
- It trains almost instantly, even on large datasets.
- Every weight is readable. You can tell a stakeholder exactly why the model predicted what it did.
- It gives you a baseline. If a fancy model can't beat it by a clear margin, you don't need the fancy model.
Where it breaks: relationships that aren't straight lines, and interactions between variables (a garden adds more value in the suburbs than downtown). You can engineer features to capture those, but that gets laborious.
Logistic regression: the classification workhorse
Despite the name, logistic regression is used for classification. It computes a weighted sum like linear regression, then squeezes it through the logistic (sigmoid) function so the output lands between 0 and 1 – a probability. "This customer has a 0.72 probability of churning."
It is the default choice for credit scoring, click prediction and many medical risk scores, precisely because it is well understood, produces calibrated probabilities with little effort, and is easy to audit. Regularisation (L1 or L2 penalties on the weights) keeps it from overfitting when you have many features; L1 also tends to push useless features' weights to zero, which acts as automatic feature selection.
Decision trees: rules a human can read
A decision tree splits the data with a series of yes/no questions. Is the contract monthly? If yes, has the customer contacted support more than twice? If yes, predict "high churn risk". The algorithm picks each split to make the resulting groups as pure as possible, using measures like Gini impurity or entropy for classification and variance reduction for regression.
Trees handle non-linear relationships and interactions naturally, need little preprocessing and don't care about feature scale. Their weakness is serious, though: a single deep tree overfits. It memorises quirks of the training data and changes dramatically if you add a few rows. On their own, trees are better for explaining than for predicting.
Random forests: many trees, averaged
A random forest fixes the instability by training hundreds of trees, each on a random bootstrap sample of the rows and considering only a random subset of features at each split. Their predictions are averaged (regression) or voted (classification). This technique – bagging plus feature randomness – cancels out much of each tree's individual noise.
Random forests are hard to break. They work reasonably well with default settings, tolerate messy data and rarely produce embarrassing results. The trade-offs: larger models, slower predictions than a linear model, and less interpretability – though feature-importance measures and SHAP values help.
Gradient boosting: the tabular-data champion
Gradient boosting also combines many trees, but differently. Instead of training them independently, it trains them in sequence: each new, shallow tree is fitted to the errors the ensemble so far is still making. Technically, each tree follows the gradient of the loss function – hence the name. A small learning rate scales each tree's contribution, so the model improves in many careful steps rather than a few big jumps.
Implementations such as XGBoost, LightGBM and CatBoost made this approach fast and practical, and gradient boosting has become the method to beat on structured, tabular data – fraud scoring, pricing, demand forecasting features, ranking.
What you pay for that accuracy:
- More hyperparameters to tune (number of trees, depth, learning rate, regularisation).
- Real overfitting risk if you skip early stopping on a validation set.
- Less transparency than a linear model; plan on SHAP or similar tools for explanations.
Random forest vs gradient boosting vs the rest
| Algorithm | Best for | Interpretability | Tuning effort | Typical weakness |
|---|---|---|---|---|
| Linear regression | Numeric targets, baselines | Very high | Minimal | Misses non-linear patterns |
| Logistic regression | Probabilities, regulated decisions | Very high | Low | Needs feature engineering |
| Decision tree | Explaining rules to humans | High | Low | Overfits, unstable |
| Random forest | Solid results with little tuning | Medium | Low | Large, slower to predict |
| Gradient boosting | Top accuracy on tabular data | Medium–low | Medium–high | Overfits without care |
A few other machine learning algorithms are worth recognising: k-nearest neighbours (predict from the most similar past examples), support vector machines (find the widest boundary between classes), naive Bayes (fast, surprisingly decent for text) and k-means (the standard clustering algorithm). And for images, audio and free text, neural networks take over – but that is a different conversation.
How to choose a machine learning algorithm
A practical order of work that holds up across most projects:
- Get the problem framing right. Classification or regression? What metric reflects the business cost of errors?
- Build a linear or logistic baseline with sensible features. Record its score on a held-out test set.
- Try a random forest for a low-effort non-linear comparison.
- Try gradient boosting with early stopping and a modest search over depth and learning rate.
- Compare on the same split, with the same metric. If the gain from a complex model is small, ship the simple one.
- Check what the model relies on. Feature importance often exposes leakage – a column that secretly contains the answer.
Most of the improvement in real projects comes from better features and cleaner data, not from switching algorithms. A thoughtful feature like "days since last order" routinely beats an hour of hyperparameter tuning.
Closing thought
The classic machine learning algorithms form a ladder: linear models for clarity, trees for rules, forests for stability, gradient boosting for accuracy. Climb it one step at a time, measure honestly at each step, and stop as soon as the next step stops paying for its complexity.
