XGBoost Algorithm
Gradient Boosting with Decision Trees
Initial Prediction
F₀(x) = Constant
Calculate Residuals
r = y - ŷ
Gradient of Loss
Train Decision Tree
on Residuals
Update Model
Fₜ(x) = Fₜ₋₁(x) + η·hₜ(x)
Learning Rate: η
Decision Tree Structure
x₁ > 5?
False
True
3 > x₂?
x₃ = 1?
-0.2
0.5
0.1
0.7
XGBoost Features
• Regularization
• Parallel Processing
• Handle Missing Values
• Tree Pruning