DSML / 01 / REGRESSION
How does a model learn a curve?
Move from a cloud of points to a prediction. Watch gradient descent adjust a model, then investigate what happens when its steps are too small—or too large.
Start here: A regression model predicts a number from an input. The blue line or curve is its prediction; the orange triangles are synthetic observations. Each dashed vertical line is a residual, the difference between an observation and its prediction.
Loading the interactive activity… JavaScript is needed for the controls; you can still read the explanations below.
Changing the model keeps the same observations. Resetting keeps both the observations and your chosen learning rate. New data changes the noise.
Explore the numerical data
Try three experiments
- Start with Linear and a learning rate of 0.1. Take a few steps, then press Play. What happens to the residuals and mean squared error?
- Choose Curved data. Train a linear model, note its error, then choose Quadratic and train again on the same observations. Which pattern can each model express?
- Keep the data fixed. Compare learning rates of 0.01, 0.1 and 2.5, resetting the coefficients each time. Does a bigger step always help?
What is being minimised?
Mean squared error (MSE) is the average of the squared residuals. Squaring stops positive and negative errors cancelling each other and gives larger errors more influence. A gradient-descent step changes each coefficient in the direction that locally reduces this loss.
This activity uses ordinary batch gradient descent: all observations contribute to every step. Inputs lie between −1 and 1, making it easier to compare step sizes. The quadratic model uses the features 1, x and x². A large learning rate can overshoot and make the loss grow; training stops if the loss becomes very large.
Check your reasoning
With a suitable step size, the fitted curve should settle and the residuals should become smaller overall. A straight line cannot reproduce a curved relationship. A quadratic model can bend, but a lower error on these training points does not establish that it will predict new observations better.
Connect it to movement
A regression model might estimate a continuous movement measure from sensor features. Real recordings need additional care: validation across people or sessions, measurement error and an appropriate baseline. The points here are invented and are not participant data.
Adapted from my regression-animation teaching tool. The website version keeps data fixed when comparing models and uses scaled inputs with ordinary gradient descent.