The Learning Curve: Humans-in-the-Loop Don't Perform Gradient Descent

Join us for a Tech Talk!

Data labeling at scale is a complicated endeavor involving operations, machine learning and game theory. In this talk we’ll focus on the game theory, and introduce a model of human-in-the-loop labeling as an optimization problem. Based on our (Scale AI’s) experience of operating tens of thousands of labelers on high complexity tasks, we will discuss how various problems predicted by the model map to observed results.

We welcome all Product Managers, Machine Learning Engineers, Software Engineers, Technical Program Managers, Research Scientists, and Applied Scientists for this technical discussion about optimizing human-in-the-loop labelling.

Agenda

• Intro to Human-in-the-Loop Labeling
• Similarities to training an ML Model
• Data augmentations for training labelers
• The right loss function
• Q&A