Welcome!

This is just a compilation of machine learning notes I wrote so I wont forget stuff.

Plus its nice revision 😊

Pages are sorted under major headings.

Everything is written in Word! (lol).

Optimizers
	a) ►Adam, AdamW Optimizers b) ►Weight Initialization a. Xavier Glorot, He Initializations b. LSUV Initialization c) ►Learning Rate Range Finder d) ►Bag of Tricks a. Loss Accumulation b. Bias Initialization c. Class Imbalance & Oversampling d. Learning Rate and Loss Plateaus e. Leslie Constant Relationship f. Standardization, Normalization g. Gradient Centralization e) ►RAdam Rectified Adam f) ►State of The Art Optimizers a. Lookahead Optimizer b. Ranger Optimizer c. LAMB, LARS Optimizer d. RangerLars Optimizer e. Paratrooper Optimizer g) ►Batch Normalization h) ►Cyclical Learning Rates i) ►Superconvergence and Flat Cosine Annealing j) ►Linear Regression & Least Squares a. Recursive Stable Cholesky Decomposition b. Jacobi Preconditioning & Standardization c. Ridge Regression & LOOCV d. Levenberg Marquardt Modified Correction k) ►LBFGS and Second Order Methods
Linear Models	a) ►Generalized Linear Models a. Canonical Form and Derivatives b. Link Functions c. Exponential Family Distributions d. Iteratively Reweighted Least Squares e. Practical Models b) ►Statistical Inference for GLMs a. Dispersion Parameter Estimation b. Coefficient Significance c. Degrees of Freedom Estimation d. Variance Estimation e. Likelihood and Deviance f. AIC and BIC c) ►Diagnostics for GLMs a. Leverage Score Estimation b. Standardized, Studentized Residuals c. Cooks Distance, DFFITS d. Confidence, Prediction Intervals d) ►High Dimensional Optimization a. Underdetermined Systems b. Standardization Tricks c. e) ►Logistic Regression
Programming Tricks	a) ►Fast Mathematical Functions a. Bhaskara’s Sine, Cosine Approximation b. b) ►Random Number Generation a. Linear Congruential Generator b. XORSHIFT c. Gaussian RV - Box Mueller Transform d. c)

Copyright Daniel Han 2024. Check out Unsloth!