On Training and Building Machine Learning Models
By Scale Team, Barrett Williams and Marwan Mattar on September 15th, 2022
What distinguishes a ML model from simple heuristics or hard-coded feature detectors is a series of “weights,” typically floating point numbers, linked by functions. The system is trained through trial and error, attempting to minimize error (known as “loss” in the ML world) over time. There are too many of these weights to adjust them manually or selectively; they must be “trained” iteratively, in an automated fashion to be useful.
Training ML models shouldn’t be a daunting proposition given the smorgasbord of tools available today. It’s never been easier to get started.
That said, don’t let ML be a solution looking for a problem. Prove that simpler approaches aren’t more effective before you land on a CNN, DNN, or LSTM as the ideal solution.
Training models can feel like an art, but really it’s best to approach it as a science. Hyperparameters and quirks in your dataset may dictate outcomes, but dig in deeper: eventually you can prove or disprove hypotheses as to what’s causing better or worse outcomes.
It’s important to be systematic and methodical: it’s best to keep an engineer’s log or notebook. Tracking hyperparameters in GitHub is largely insufficient; you should ruthlessly test your models for regressions, track aggregate and specific performance metrics, and use tooling where possible to take the tediousness out of writing scripts.