ML algorithms

Quick comparison

Algorithm	Task	Key idea	Main hyperparameters
Linear regression	Regression	Least-squares line	Regularization λ
Logistic regression	Classification	Sigmoid of a linear score	Regularization λ, threshold
KNN	Both	Vote of nearest points	$k$, distance
Naive Bayes	Classification	Bayes' rule, independent features	Smoothing
Decision tree	Both	Purest splits	Max depth, min samples per leaf
Random forest	Both	Average of de-correlated trees	Number of trees, features per split
AdaBoost	Classification	Re-weight mistakes	Rounds, learner depth
Gradient boosting / XGBoost	Both	Trees fit residuals	Rounds, learning rate, depth, λ
SVM	Classification	Maximum margin	C, kernel, gamma
k-means	Clustering	Nearest centroid	$k$
Hierarchical	Clustering	Merge closest clusters	Linkage, cut height
GMM	Clustering / density	Soft Gaussian components (EM)	$K$, covariance type
DBSCAN	Clustering	Dense regions + noise	ε, minPts
PCA	Dimensionality reduction	Directions of max variance	Number of components

Which one to pick?

flowchart TD
  A{"Do you have labels?"} -->|"no"| U{"Goal?"}
  U -->|"fewer features"| PCA["PCA"]
  U -->|"groups, k known, round"| KM["k-means"]
  U -->|"groups of any shape + outliers"| DB["DBSCAN"]
  U -->|"soft, overlapping groups"| GMM["GMM"]
  U -->|"structure at every scale"| HC["Hierarchical"]
  A -->|"yes"| T{"Number or category?"}
  T -->|number| R{"Linear enough?"}
  R -->|yes| LR["Linear regression"]
  R -->|no| GB["Gradient boosting / random forest"]
  T -->|category| C{"What matters most?"}
  C -->|"probabilities, interpretability"| LOG["Logistic regression"]
  C -->|"text, many sparse features"| NB["Naive Bayes"]
  C -->|"top accuracy on tables"| GB2["Gradient boosting / XGBoost"]
  C -->|"small data, clear margin"| SVM["SVM"]
  C -->|"readable rules"| DT["Decision tree"]

Key points

Linear models are fast and interpretable because they are weighted sums of the features.
Distance-based methods need scaling because large-valued features dominate distances.
Forests reduce variance and boosting reduces bias because one averages parallel trees and the other adds trees that fix errors.
Clustering methods differ in shape assumptions because k-means uses centres, GMM Gaussians, DBSCAN density and hierarchical a linkage rule.
PCA compresses features because correlated features share directions of variance.

External resources

scikit-learn: Supervised learning: every supervised algorithm here, with code.
scikit-learn: Unsupervised learning: clustering and decomposition.
An Introduction to Statistical Learning: free textbook covering most of these models.
scikit-learn: Choosing the right estimator: a flowchart for picking a model.

Videos

YouTube search: StatQuest machine learning: a video for nearly every algorithm on this hub.

Mixed review

Quick comparison

The fourteen algorithms

Linear regression

Logistic regression

KNN

Naive Bayes

Decision tree

Random forest

AdaBoost

Gradient boosting / XGBoost

SVM

k-means

Hierarchical clustering

Gaussian mixture model

DBSCAN

PCA

Which one to pick?

Key points

External resources

Review schedule

Feynman prompt

My notes