image.jpg

Mark Pfeiffer

Engineering Manager, Validate

As ML applications grow in size and complexity, testing is essential to ensure integrity and reliability of the application at hand. Given the non-deterministic nature of machine learning (ML) models, testing and continuous evaluation of ML systems is even more important.&nbsp;
That&rsquo;s why today, we&rsquo;re thrilled to announce Early Access to <a href="https://scale.com/nucleus" target="_blank" rel="noopener">Scale Validate</a>, our testing solution for ML models. The predictive performance of ML components is essential to the overall success and adoption of whole products. Progressively developing a high-performing ML-based system is not trivial, because it typically <a href="https://exchange.scale.com/public/blogs/3-lessons-on-how-to-build-high-performance-ml-models" target="_blank" rel="noopener noreferrer">requires continuous iteration and experimentation</a>. During these iterations, engineers need to identify new edge cases and improve model performance on those. With every new model iteration, ML engineers want to be sure that they aren&rsquo;t introducing any performance regressions on mission-critical subsets of data.&nbsp;
Using Scale Validate, you can now define data-centric tests for your ML models to avoid performance regressions during development. This step helps ensure that only the best ML models make it into production.
<a href="https://scaleai.wistia.com/medias/ks6a02aqwq" target="_blank" rel="noopener noreferrer">Video</a>
For ML-based products, model performance should always directly correlate with positive business outcome.&nbsp;The performance of ML components is often crucial to the success and adoption of whole products. Getting to a high-performing ML-based system is not trivial but typically <a href="https://exchange.scale.com/public/blogs/3-lessons-on-how-to-build-high-performance-ml-models" target="_blank" rel="noopener noreferrer">requires many iterations</a> and will almost never be done. During these iterations, engineers need to identify new edge cases and improve the model performance on those. With every new model iteration, ML engineers want to be sure that no performance regressions on mission-critical subsets of data are introduced.&nbsp;
While ML model testing is key to fast and robust model iteration, there are a few common issues that we identified with existing testing and evaluation pipelines for ML models:
<h4>Aggregate evaluation metrics miss critical regressions</h4>
A common way of validating ML models is to compute aggregate scores and metrics (e.g. mAP, precision, recall, IoU, &hellip;) across fixed test/validation datasets (commonly understood as model evaluation). Evaluation with aggregate metrics is necessary but not sufficient for robust models because they often miss failure modes that are underrepresented due to data biases. Users won&rsquo;t be able to track regressions with explicit checks on such failure modes that were previously addressed but are failing now.
<h4>Coverage for ML model testing is not well defined</h4>
While we can analyze covered lines of code in software testing, there&rsquo;s no such number which informs about the coverage of tests for ML-models. Lines of code still need to be covered, but more importantly, tests need to cover the operational domain of the model input data. The high dimensionality of typical inputs (e.g. an image has more than a million pixels with 3 channels) does not allow for realistic or efficient input-fuzzing approaches.&nbsp;
Testing pipelines are static and don&rsquo;t evolve with the models
Building ML models is an iterative process. So is testing ML models. Not adapting testing and validation datasets results in vanity metrics which in turn oftentimes overestimate the confidence in the models. Continuously evolving test definitions and adding the most tricky examples is crucial to continuously improve the test coverage in the operational domain. This is the only certain path to confidently deploy robust models to production systems.
Accordingly, we built Scale Validate with the following capabilities:
<h3>Scenario Tests</h3>
Setting up unit and integration tests is the go-to practice to ensure correct behavior of codebases. For ML systems, unit tests are designed to test the correctness of individual model components, which is necessary for a reliable model performance. Scenario tests are designed to test the actual model performance and decision-making capabilities of ML models on mission-critical scenarios in the operational domain. Scale Validate allows users to easily define scenario tests on subsets of data (<a href="https://dashboard.scale.com/nucleus/ut_camxgk2yxqgg05rzc4bg" target="_blank" rel="noopener noreferrer">example</a>), using off-the-shelf or custom evaluation metrics. New models are automatically tested against defined scenario tests. Well-defined scenario tests allow ML engineers to catch regressions before deploying a model to production. When a test fails, users immediately see in which scenario the required performance is not achieved anymore (<a href="https://dashboard.scale.com/nucleus/prj_cabt0xkhtxbg048dvqn0?spoof=select%40scaleapi.com&amp;val_ev_gsp=2165&amp;from_unit_test_id=ut_camxgk2yxqgg05rzc4bg&amp;model_page_display=Evaluations&amp;val_model_gsp=1154" target="_blank" rel="noopener noreferrer">example</a>).
<img src="https://lh6.googleusercontent.com/lyc0KYEFc_1GRv-k5m7BPrtijOlmb4ef2xg6J_oVdWJFimn3rfwTjMkpaYs3aUu5PUJI5VjPExMjvC3tuK3V6LLEuAXPlZDRwoT1wfXaTfmFBeZxuq8ov_3OLitltffWIaU_-Q7RJ2y1XbZefMfzkXI">
<h3>Model Comparison</h3>
Model performance rarely boils down to a single metric. With Scale Validate, it&rsquo;s trivial to compare the performance of new models against baseline models (<a href="https://dashboard.scale.com/nucleus/analyze?analyze_compare_model_id=run_cabp73ckv9wg078pk440&amp;analyze_base_model_id=run_camszmp8nd200ch71170&amp;q=predictions.count+%3E+0" target="_blank" rel="noopener noreferrer">example</a>). You can use either our standard performance metric tracking across scenario tests or our side-by-side model comparison. All charts are fully interactive and users can start by informing themselves from higher-level metrics, then graduating into learning from item-based analysis.
<img src="https://lh5.googleusercontent.com/Ny6n_eMIjJ1kMGRDpWyp47In_FjhErjXbVUS1ou5VvweGqSo_oDjyd4ORb07DmiCrSBPIbkhxQz-v840Nlaj8DmKzFny66BCVsBh9b1z0gMw23Wgq86jkpasU8QdBzp90S9ER5BQBKAIuPAhodUT8e8">
<h3>Slice Discovery</h3>
Scale Validate enables users to automatically identify coherent and underperforming slices of data that do not yet have test coverage. Users can be sure they are always testing against a representative test set.
<img src="https://lh3.googleusercontent.com/tsjA4raZ7xVSuMM9FZR-gLVSn-pX_r99a1r-2hPCTyub75QNG5J6SSRXf2J5c7WiQtWZIHyEgkS78j-ST7u2qooBIDEXC7xZcGKbo6ojWQmdmhBkkLiFPkF7YS7gDw4j5-uvbhjsUXC67OuuxT451fU">
We built Validate as a developer-friendly solution both for API-first and UI-focused users. The Validate interfaces and endpoints make it as easy as possible to get started and integrated into existing pipelines. Today, Scale Validate is used for the testing pipelines of leading ML teams in areas including autonomous driving, surveillance, and industrial robotics.
As Pickle Robot&rsquo;s CTO, Ariana Eisenstein, explains:
<img src="https://lh6.googleusercontent.com/gb9ujMPiBfQOhkiz36npqFR8kY44KymYOC79ac0AQaL4UAK9H2Nm7w3O5bLE7V0xF6w3tOe2E1eRrl57hDs_FMOaZCm9HbzFutL6UmX_s2mPuJ_psw-Aoub5rNo9jynrD_q0yxbtDG8o-QFEwHEkHxQ">
Getting started with Validate only takes a few steps!&nbsp;
<ol>
<li>Sign up / log into the <a href="https://dashboard.scale.com/corp/login?redirect_url=%2F&amp;clear=true" target="_blank" rel="noopener noreferrer">Scale dashboard</a></li>
<li><a href="https://dashboard.scale.com/nucleus/scenario-tests" target="_blank" rel="noopener noreferrer">Explore Validate with public datasets</a></li>
<li><a href="https://nucleus.scale.com/docs/datasets-in-nucleus#uploading-your-data" target="_blank" rel="noopener noreferrer">Upload your data</a></li>
<li>Upload <a href="https://nucleus.scale.com/docs/model-predictions-in-nucleus" target="_blank" rel="noopener noreferrer">models</a> and <a href="https://nucleus.scale.com/docs/ground-truth-annotations-in-nucleus" target="_blank" rel="noopener noreferrer">annotations</a></li>
<li>Compare your models in our <a href="https://dashboard.scale.com/nucleus/analyze" target="_blank" rel="noopener noreferrer">dashboard</a></li>
<li>Select interesting slices of data and <a href="https://dashboard.scale.com/nucleus/scenario-tests" target="_blank" rel="noopener noreferrer">define tests</a></li>
</ol>
Without a detailed model evaluation and testing pipeline, you&rsquo;ll never know which performance points you&rsquo;re leaving on the table and which performance regressions might sneak into your models. Instead, ensure that your model quality only increases with every model and dataset iteration. Test-driven modeling will ensure that you train a production-grade model faster! If you&rsquo;d like more information about Scale Validate, reach out to us for a detailed demo and discussion <a href="https://scale.com/validate" target="_blank" rel="noopener noreferrer">here</a>.

Introducing Scale Validate for Model Testing