Skip to content

IT_AbyssIT_AbyssSign in

Machine Learning from Scratch

Rules versus examples
Features and target
Quiz — features and target
Learning without answers
Quiz — supervised and unsupervised
Classification and regression
Quiz — classification or regression
Training, testing and prediction
Quiz — fit, predict and the training score

NumPy arrays
Quiz — a list and an array times 2
Indexing and slicing
Quiz — sums over a boolean mask
Axes and broadcasting
Quiz — axis and broadcasting shapes
Exercise: values above the column mean
The pandas DataFrame
Quiz — counting missing values
Grouping and the median
Quiz — the median per group
Exercise: median per group

Training and test sets
Quiz — rows after a split
Missing values
Quiz — filling a gap
Encoding categories
Quiz — one-hot or integer codes
Feature scaling
Quiz — a scaled test value
Exercise: standardize the test set
Data leakage
Quiz — where the test set leaks
Exercise: prepare the features

Fitting a line
Quiz — slope and intercept
Exercise: fit a line
Linear regression in scikit-learn
Quiz — the shape of X
Quiz — reading a coefficient
Regression metrics
Quiz — MAE and RMSE
Quiz — negative R²
Exercise: R² on the diabetes data
Gradient descent
Quiz — one gradient step
Exercise: one gradient descent step
Residual plots
Quiz — reading residuals

Polynomial features
Quiz — why the curve stays linear
Quiz — how many columns
Exercise: polynomial features
Degree and overfitting
Quiz — degree and errors
Ridge and Lasso
Quiz — Ridge or Lasso zeros
Quiz — alpha and the slope
Exercise: Ridge on the diabetes data
Support vector regression
Quiz — the ε-tube

The sigmoid
Quiz — the sigmoid of a score
Log-loss
Exercise: sigmoid and log-loss
Logistic regression in scikit-learn
Quiz — predict_proba and predict
The confusion matrix
Quiz — the cells of a confusion matrix
Precision and recall
Quiz — precision and recall
Quiz — moving the threshold
ROC-AUC
Exercise: recall and ROC-AUC on tumor data

Nearest neighbors
Quiz — the neighbors' vote
Exercise: predict with k neighbors
k-NN in scikit-learn
Quiz — scaling and neighbors
Choosing k
Quiz — small and large k
Exercise: k-NN on the wine data
Bayes' rule
Quiz — probability of spam
Naive Bayes
Quiz — what makes it naive

Maximum margin
Quiz — who violates the margin
Soft margin and C
Quiz — margin, support vectors and C
Exercise: the SVM objective
SVC in scikit-learn
Quiz — support vectors and C
Kernels and gamma
Quiz — gamma and scaling
Exercise: SVC on the wine data

Splits and Gini impurity
Quiz — Gini of a group
Exercise: the Gini of a split
Decision trees in scikit-learn
Quiz — depth and training score
Thresholds and leaf constants
Quiz — a tree outside the data
Bagging and random forests
Quiz — what bagging is
Feature importance
Quiz — importance of a row number
Exercise: a forest and its top feature

Bias and variance
Quiz — bias and variance
Train, validation and test sets
Quiz — roles of the three sets
Cross-validation
Quiz — what cross_val_score returns
Exercise: k-fold indices
Pipelines
Quiz — naming a step's parameter
Grid search
Quiz — counting grid search fits
Exercise: tune C with a grid search

Boosting on residuals
Quiz — one step on residuals
Exercise: one boosting step
Gradient boosting in scikit-learn
Quiz — boosting and missing values
Exercise: boosting on a bundled dataset
Learning rate and number of trees
Quiz — learning rate and trees
Early stopping
Quiz — early stopping by default
XGBoost, LightGBM and CatBoost
Quiz — how LightGBM grows a tree

The k-means algorithm
Quiz — one k-means step
Exercise: one k-means step
k-means in scikit-learn
Quiz — the value of inertia_
Quiz — what k-means++ fixes
Choosing the number of clusters
Quiz — elbow and silhouette
Exercise: the best k by silhouette
Hierarchical clustering
Quiz — predict on a new point

Principal components
Quiz — the share of variance
PCA in scikit-learn
Quiz — PCA output shapes
Quiz — what PCA does and why scale
Exercise: project onto principal components
PCA inside a pipeline
Exercise: PCA before a classifier
Linear discriminant analysis
Quiz — LDA components
Kernel PCA
Quiz — kernel PCA

Itemsets and support
Quiz — support of an itemset
Confidence and lift
Quiz — confidence of a rule
Quiz — confidence and lift
Exercise: support, confidence and lift
Apriori pruning
Quiz — Apriori pruning
Exercise: frequent pairs

The A/B test
Quiz — reading a p-value
Bandits and regret
Quiz — regret of a pull sequence
The epsilon-greedy strategy
Quiz — an arm's running mean
Exercise: an epsilon-greedy run
UCB and Thompson sampling
Quiz — UCB score and plain mean
Exercise: the UCB choice
Bandit or A/B test
Quiz — bandit or A/B test

Cleaning and splitting text
Quiz — counting tokens
Bag of words
Quiz — a word outside the vocabulary
Exercise: a bag of words by hand
TF-IDF
Quiz — IDF of a common and a rare word
Naive Bayes for text
Quiz — why alpha is needed
Exercise: a spam filter
Words as vectors
Quiz — cosine or distance

Neurons and layers
Quiz — a ReLU layer
Backpropagation
Quiz — the output-layer gradient
Exercise: a tiny network
MLPClassifier in scikit-learn
Quiz — counting a network's parameters
Exercise: a network on the digits data
Overfitting and early stopping
Quiz — early stopping
Convolution and pooling
Quiz — why a CNN needs fewer parameters

Saving a model
Quiz — saving only the model step
Batch and online serving
Quiz — a scaler fitted on one row
Data drift
Quiz — mean shift in deviations
Exercise: a drift check
Permutation importance
Quiz — what permutation importance shows
Exercise: the top feature by permutation
Bias through proxy features
Quiz — dropping a sensitive column

Project plan
Quiz — a baseline model's score
Exercise: a baseline and a first model
Exercise: compare two model families
The final report
Exercise: the project report

Courses/Machine Learning from Scratch

Theory · Gradient Boosting · lesson 11 of 12

XGBoost, LightGBM and CatBoost

← BackQuiz — early stopping by default Next →Quiz — how LightGBM grows a tree

IT Abyss v2026.10.2

Search Roadmap Interviews Blog Pricing Account

Privacy Policy Terms of Service Refund Policy© 2026 IT Abyss

Menu · sectionLearn

CoursesMy progress

RoadmapChoose a stack

Kotlin / Android

Machine Learning

System Analysis

Menu · sectionInterviews

SimulatorResume check

QuestionsChoose a stack

Kotlin / Android

Machine Learning

System Analysis

Menu · sectionFeed

BlogPosts by IT Abyss NewsIT news from outside sources