Troubleshooting ML Pipelines

A classical ML pipeline fails differently from an LLM call: no prompt to tweak, failures hide in the numbers. Organised problem to cause to fix. See decision patterns and tools reference. First law: trust the metrics; split off a test set first.

Model Will Not Converge

"Converge" means the training loss stops dropping and settles low. Flat, NaN, or oscillating loss means the optimiser is not learning.

Scaling is the most common fix: linear models, SVMs and neural nets assume features share similar ranges, so wrap a StandardScaler before the estimator (see Data Leakage). In PyTorch, clip gradients with clip_grad_norm_ before optimizer.step().

Strong Train, Poor Test (Overfitting)

Overfitting = the model memorises the training set instead of the pattern, scoring high on seen data, low on unseen. The tell: a large train-test gap (e.g. 0.99 vs 0.71).

Cross-validation trains and tests on several splits and averages the scores, far more honest than one; a wide std signals an unstable estimate (too little data). A frozen LLM cannot overfit your data because it never trains on it, which is why classical ML wins when you have labels and need calibrated scores.

from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score

model = RandomForestClassifier(max_depth=8, n_estimators=200)
scores = cross_val_score(model, X, y, cv=5, scoring="f1")
print(f"F1: {scores.mean():.3f} +/- {scores.std():.3f}")

Data Leakage

Leakage = test-set (or future) info sneaks into training, so offline metrics look brilliant and production collapses. The most dangerous ML bug: it hides as success.

Wrap preprocessing in a Pipeline so transforms learn from training folds only. For time series, never shuffle; use TimeSeriesSplit.

from sklearn.model_selection import cross_val_score, TimeSeriesSplit
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC

pipe = make_pipeline(StandardScaler(), SVC())  # scaler re-fit per fold
cross_val_score(pipe, X, y, cv=TimeSeriesSplit(n_splits=5))

Severe Class Imbalance

When fraud is ~0.1% of transactions, a model that always predicts "not fraud" hits 99.9% accuracy catching none, so accuracy is the wrong metric. Precision = of cases flagged positive, how many were truly positive (false alarms). Recall = of all true positives, how many were caught (misses).

from sklearn.metrics import classification_report
from sklearn.linear_model import LogisticRegression

# class_weight rebalances the loss toward the rare class
clf = LogisticRegression(class_weight="balanced", max_iter=1000)
clf.fit(X_train, y_train)
print(classification_report(y_test, clf.predict(X_test)))

SMOTE (train only) synthesises minority rows.

Deployment & Serving Errors

The model trained fine but the API returns 500s or wrong shapes: usually environment or contract mismatches.

Persist the entire pipeline so preprocessing ships with the model; validate the contract first:

import joblib, pandas as pd

loaded = joblib.load("model.joblib")  # saved via joblib.dump(pipe, ...)
EXPECTED = ["amount", "merchant_id", "hour", "country"]

def predict(row: dict):
    missing = [c for c in EXPECTED if c not in row]
    if missing:
        raise ValueError(f"missing features: {missing}")
    return loaded.predict(pd.DataFrame([row])[EXPECTED])[0]

Drift-Detection Alerts

Drift = live data moves away from training data, so a once-accurate model quietly degrades (e.g. a forecaster after a product launch).

Compare recent inputs to a training baseline; small p-value = they differ:

from scipy.stats import ks_2samp

# Kolmogorov-Smirnov: same distribution as training?
stat, p = ks_2samp(train_feature, live_feature)
if p < 0.01:
    print("Drift: investigate, then consider retraining")

Do not auto-retrain on every alert: confirm the drift is real and you have fresh, trustworthy labels, or you retrain on noise.

Training-Serving Skew & Feature-Store Inconsistency

Skew = a feature computed one way in training (batch SQL over history) and another at serving (live code). Fine offline, underperforms live, no error.

The durable fix is a feature store: one feature definition served to both training and inference. Spot-check by logging served vectors and recomputing them:

import numpy as np

served  = np.load("served_features.npy")      # logged at inference
offline = np.load("recomputed_features.npy")  # recomputed offline
assert np.abs(served - offline).max() < 1e-6, "training-serving skew"

Debugging order: (1) confirm no leakage, (2) check scaling, (3) pick a metric matching the problem (precision/recall over accuracy for imbalance), (4) verify serving matches training. See the rules and cheat sheet, or the course home.