Welcome!

This is just a compilation of machine learning notes I wrote so I wont forget stuff.

Plus its nice revision 😊

 

Pages are sorted under major headings.

Everything is written in Word! (lol).

 

Optimizers

 

a)    Adam, AdamW Optimizers

b)    Weight Initialization

a.     Xavier Glorot, He Initializations

b.     LSUV Initialization

c)    Learning Rate Range Finder

d)    Bag of Tricks

a.     Loss Accumulation

b.     Bias Initialization

c.     Class Imbalance & Oversampling

d.     Learning Rate and Loss Plateaus

e.     Leslie Constant Relationship

f.       Standardization, Normalization

g.     Gradient Centralization

e)    RAdam Rectified Adam

f)      State of The Art Optimizers

a.     Lookahead Optimizer

b.     Ranger Optimizer

c.     LAMB, LARS Optimizer

d.     RangerLars Optimizer

e.     Paratrooper Optimizer

g)    Batch Normalization

h)    ►Cyclical Learning Rates

i)      Superconvergence and Flat Cosine Annealing

j)      Linear Regression & Least Squares

a.     Recursive Stable Cholesky Decomposition

b.     Jacobi Preconditioning & Standardization

c.     Ridge Regression & LOOCV

d.     Levenberg Marquardt Modified Correction

k)    ►LBFGS and Second Order Methods

 

 

Linear Models

 

a)    Generalized Linear Models

a.     Canonical Form and Derivatives

b.     Link Functions

c.     Exponential Family Distributions

d.     Iteratively Reweighted Least Squares

e.     Practical Models

b)    Statistical Inference for GLMs

a.     Dispersion Parameter Estimation

b.     Coefficient Significance

c.     Degrees of Freedom Estimation

d.     Variance Estimation

e.     Likelihood and Deviance

f.       AIC and BIC

c)    Diagnostics for GLMs

a.     Leverage Score Estimation

b.     Standardized, Studentized Residuals

c.     Cooks Distance, DFFITS

d.     Confidence, Prediction Intervals

d)    High Dimensional Optimization

a.     Underdetermined Systems

b.     Standardization Tricks

c.      

e)    Logistic Regression

Programming

Tricks

 

a)    ►Fast Mathematical Functions

a.     Bhaskara’s Sine, Cosine Approximation

b.      

b)    ►Random Number Generation

a.     Linear Congruential Generator

b.     XORSHIFT

c.     Gaussian RV - Box Mueller Transform

d.      

c)      

 

 

 

 

Copyright Daniel Han 2024. Check out Unsloth!