Machine Learning from Scratch
Classical machine learning in Python, from numpy and pandas to models you can train, evaluate, tune and explain.
Tests check every exercise: in your browser and in the cloud
Exercises run in the browser, so you need nothing installed to take the course. To run lesson code and exercises on your own computer, set it up in four steps.
Install Python 3.12
You need exactly 3.12: the library versions the site runs have ready-made builds for it. Download the installer for your system from python.org.
Install VS Code
Download VS Code and add the Python extension from Microsoft.
Create an environment and install the libraries
Open a folder for the course in VS Code and create a virtual environment in it. Install the versions the site runs, so your output matches the lessons: numpy 1.26, pandas 2.2, scikit-learn 1.4 and pytest.
Run lesson code and exercises
Save a lesson's code to a file and run it with the ▶ button. For an exercise, save both editor tabs under their names and run the tests from the Testing panel.
Syllabus
- Rules versus examplesTheory
- Features and targetTheory
- Quiz — features and targetQuiz
- Learning without answersTheory
- Quiz — supervised and unsupervisedQuiz
- Classification and regressionTheory
- Quiz — classification or regressionQuiz
- Training, testing and predictionTheory
- Quiz — fit, predict and the training scoreQuiz
- NumPy arraysTheory
- Quiz — a list and an array times 2Quiz
- Indexing and slicingTheory
- Quiz — sums over a boolean maskQuiz
- Axes and broadcastingTheory
- Quiz — axis and broadcasting shapesQuiz
- Exercise: values above the column meanExercise
- The pandas DataFrameTheory
- Quiz — counting missing valuesQuiz
- Grouping and the medianTheory
- Quiz — the median per groupQuiz
- Exercise: median per groupExercise
- Training and test setsTheory
- Quiz — rows after a splitQuiz
- Missing valuesTheory
- Quiz — filling a gapQuiz
- Encoding categoriesTheory
- Quiz — one-hot or integer codesQuiz
- Feature scalingTheory
- Quiz — a scaled test valueQuiz
- Exercise: standardize the test setExercise
- Data leakageTheory
- Quiz — where the test set leaksQuiz
- Exercise: prepare the featuresExercise
- Fitting a lineTheory
- Quiz — slope and interceptQuiz
- Exercise: fit a lineExercise
- Linear regression in scikit-learnTheory
- Quiz — the shape of XQuiz
- Quiz — reading a coefficientQuiz
- Regression metricsTheory
- Quiz — MAE and RMSEQuiz
- Quiz — negative R²Quiz
- Exercise: R² on the diabetes dataExercise
- Gradient descentTheory
- Quiz — one gradient stepQuiz
- Exercise: one gradient descent stepExercise
- Residual plotsTheory
- Quiz — reading residualsQuiz
- Polynomial featuresTheory
- Quiz — why the curve stays linearQuiz
- Quiz — how many columnsQuiz
- Exercise: polynomial featuresExercise
- Degree and overfittingTheory
- Quiz — degree and errorsQuiz
- Ridge and LassoTheory
- Quiz — Ridge or Lasso zerosQuiz
- Quiz — alpha and the slopeQuiz
- Exercise: Ridge on the diabetes dataExercise
- Support vector regressionTheory
- Quiz — the ε-tubeQuiz
- The sigmoidTheory
- Quiz — the sigmoid of a scoreQuiz
- Log-lossTheory
- Exercise: sigmoid and log-lossExercise
- Logistic regression in scikit-learnTheory
- Quiz — predict_proba and predictQuiz
- The confusion matrixTheory
- Quiz — the cells of a confusion matrixQuiz
- Precision and recallTheory
- Quiz — precision and recallQuiz
- Quiz — moving the thresholdQuiz
- ROC-AUCTheory
- Exercise: recall and ROC-AUC on tumor dataExercise
- Nearest neighborsTheory
- Quiz — the neighbors' voteQuiz
- Exercise: predict with k neighborsExercise
- k-NN in scikit-learnTheory
- Quiz — scaling and neighborsQuiz
- Choosing kTheory
- Quiz — small and large kQuiz
- Exercise: k-NN on the wine dataExercise
- Bayes' ruleTheory
- Quiz — probability of spamQuiz
- Naive BayesTheory
- Quiz — what makes it naiveQuiz
- Splits and Gini impurityTheory
- Quiz — Gini of a groupQuiz
- Exercise: the Gini of a splitExercise
- Decision trees in scikit-learnTheory
- Quiz — depth and training scoreQuiz
- Thresholds and leaf constantsTheory
- Quiz — a tree outside the dataQuiz
- Bagging and random forestsTheory
- Quiz — what bagging isQuiz
- Feature importanceTheory
- Quiz — importance of a row numberQuiz
- Exercise: a forest and its top featureExercise
- Bias and varianceTheory
- Quiz — bias and varianceQuiz
- Train, validation and test setsTheory
- Quiz — roles of the three setsQuiz
- Cross-validationTheory
- Quiz — what cross_val_score returnsQuiz
- Exercise: k-fold indicesExercise
- PipelinesTheory
- Quiz — naming a step's parameterQuiz
- Grid searchTheory
- Quiz — counting grid search fitsQuiz
- Exercise: tune C with a grid searchExercise
- Boosting on residualsTheory
- Quiz — one step on residualsQuiz
- Exercise: one boosting stepExercise
- Gradient boosting in scikit-learnTheory
- Quiz — boosting and missing valuesQuiz
- Exercise: boosting on a bundled datasetExercise
- Learning rate and number of treesTheory
- Quiz — learning rate and treesQuiz
- Early stoppingTheory
- Quiz — early stopping by defaultQuiz
- XGBoost, LightGBM and CatBoostTheory
- Quiz — how LightGBM grows a treeQuiz
- The k-means algorithmTheory
- Quiz — one k-means stepQuiz
- Exercise: one k-means stepExercise
- k-means in scikit-learnTheory
- Quiz — the value of inertia_Quiz
- Quiz — what k-means++ fixesQuiz
- Choosing the number of clustersTheory
- Quiz — elbow and silhouetteQuiz
- Exercise: the best k by silhouetteExercise
- Hierarchical clusteringTheory
- Quiz — predict on a new pointQuiz
- Principal componentsTheory
- Quiz — the share of varianceQuiz
- PCA in scikit-learnTheory
- Quiz — PCA output shapesQuiz
- Quiz — what PCA does and why scaleQuiz
- Exercise: project onto principal componentsExercise
- PCA inside a pipelineTheory
- Exercise: PCA before a classifierExercise
- Linear discriminant analysisTheory
- Quiz — LDA componentsQuiz
- Kernel PCATheory
- Quiz — kernel PCAQuiz
- The A/B testTheory
- Quiz — reading a p-valueQuiz
- Bandits and regretTheory
- Quiz — regret of a pull sequenceQuiz
- The epsilon-greedy strategyTheory
- Quiz — an arm's running meanQuiz
- Exercise: an epsilon-greedy runExercise
- UCB and Thompson samplingTheory
- Quiz — UCB score and plain meanQuiz
- Exercise: the UCB choiceExercise
- Bandit or A/B testTheory
- Quiz — bandit or A/B testQuiz
- Cleaning and splitting textTheory
- Quiz — counting tokensQuiz
- Bag of wordsTheory
- Quiz — a word outside the vocabularyQuiz
- Exercise: a bag of words by handExercise
- TF-IDFTheory
- Quiz — IDF of a common and a rare wordQuiz
- Naive Bayes for textTheory
- Quiz — why alpha is neededQuiz
- Exercise: a spam filterExercise
- Words as vectorsTheory
- Quiz — cosine or distanceQuiz
- Neurons and layersTheory
- Quiz — a ReLU layerQuiz
- BackpropagationTheory
- Quiz — the output-layer gradientQuiz
- Exercise: a tiny networkExercise
- MLPClassifier in scikit-learnTheory
- Quiz — counting a network's parametersQuiz
- Exercise: a network on the digits dataExercise
- Overfitting and early stoppingTheory
- Quiz — early stoppingQuiz
- Convolution and poolingTheory
- Quiz — why a CNN needs fewer parametersQuiz
- Saving a modelTheory
- Quiz — saving only the model stepQuiz
- Batch and online servingTheory
- Quiz — a scaler fitted on one rowQuiz
- Data driftTheory
- Quiz — mean shift in deviationsQuiz
- Exercise: a drift checkExercise
- Permutation importanceTheory
- Quiz — what permutation importance showsQuiz
- Exercise: the top feature by permutationExercise
- Bias through proxy featuresTheory
- Quiz — dropping a sensitive columnQuiz