ML Approach Selection Rules

Rules for choosing an AI/ML approach: start from the problem type, weigh constraints, prefer the simplest model that clears the bar, prove it with a baseline, and budget for the model's production life. See also decision patterns.

Rule 0: Start from the problem type, not the model

Map your task to a problem class first; it narrows the tool list far more than the hype cycle. GenAI is one row below, not the default:

Rule: if the output is a label, number, cluster, box, or rank, a specialised model is usually cheaper, faster, and more interpretable than an LLM.

Rule 1: Score your constraints before you pick

The "best" model clears every hard constraint, not the one topping a leaderboard. Let the strict ones eliminate options:

Rule of thumb: an LLM costs far more per call and adds large latency versus a small classifier. For high-throughput, low-margin decisions (every transaction, every log line), it rarely pays off.

Rule 2: Prefer the simplest model that meets the bar

Overfitting = the model memorised the training data and fails on new data; underfitting = it is too simple to capture the pattern. Simpler models overfit less and debug easier. Climb only when the rung below misses the bar:

1. Heuristic / rule  ("flag amount > 10k from a new account")
2. Logistic / linear regression
3. Tree ensemble (RandomForest, XGBoost, LightGBM)
4. Small neural net
5. Transfer learning from a pretrained model (vision: YOLO; NLP: BERT)
6. Large foundation model / LLM

Transfer learning = adapt a model already trained on millions of examples to your small dataset -- the default for vision and NLP with hundreds of labels.

Rule 3: Establish a baseline before going deep

A dumb baseline tells you whether the problem is learnable and what "good" looks like. If a transformer barely beats the majority class, the data is the issue.

from sklearn.dummy import DummyClassifier
from sklearn.model_selection import cross_val_score
from sklearn.datasets import load_breast_cancer

X, y = load_breast_cancer(return_X_y=True)
dummy = DummyClassifier(strategy="most_frequent")   # always the common class
print("dummy:", cross_val_score(dummy, X, y, cv=5, scoring="f1").mean())

Cross-validation (cv=5) = split into 5 folds, train on 4 and test on 1, rotate, average -- more honest than one split. Rule: report the baseline in every model review.

Rule 4: Validate honestly -- held-out test set, no leakage

Data leakage = test-set (or future) information sneaks into training, giving a fantasy score that collapses in production. The classic mistake: fitting a scaler on all data before splitting. Fit transforms inside the pipeline, after the split.

from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report

X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, stratify=y)
pipe = make_pipeline(StandardScaler(), LogisticRegression(max_iter=5000))
pipe.fit(X_tr, y_tr)   # scaler fits only on the training fold -> no leakage
print(classification_report(y_te, pipe.predict(X_te)))

For time series, never shuffle -- split by time (TimeSeriesSplit: train on the past, test on the future) or you leak the future.

Rule 5: Pick metrics that match the cost of being wrong

Accuracy lies on imbalanced data: calling everything "not fraud" scores near-perfect while catching zero fraud. Choose the metric that reflects what failure costs:

Rule: on imbalanced problems report precision and recall, not accuracy. Pick the threshold from the business cost, not the default.

Rule 6: Use pretrained models for vision and language

You rarely train from scratch. Pretrained models carry general knowledge you fine-tune with a fraction of the data:

# Vision: detection via transfer learning from a pretrained model.
from ultralytics import YOLO
results = YOLO("yolo11n.pt")("factory_line.jpg")   # fine-tune for QC defects

# NLP: a focused classifier, faster and cheaper than an LLM call.
from transformers import pipeline
clf = pipeline("sentiment-analysis")   # BERT-family encoder, not a generator
print(clf("The replacement part arrived bent and late."))

Embeddings = a model turns text/images into a vector where "similar" things sit close together; feed those vectors into a cheap classifier -- rich features at classical-model speed and cost.

Rule 7: Budget for monitoring and retraining from day one

A model is a perishable asset: the world shifts and accuracy quietly rots -- drift (live data stops looking like training data). Plan for it up front:

Rules: log every prediction with its inputs to replay incidents; capture labels later to measure real performance; set a retraining trigger (schedule or metric threshold) and a rollback plan. See troubleshooting and the cheat sheet.

Back to the Beyond GenAI course.