Will LeVine

<h2 dir="ltr">Background</h2>
<p dir="ltr">Object tracking in aerial imagery is largely an unsolved problem in the AI overmatch future powered by Generative AI and platforms like <a href="https://scale.com/donovan" target="_blank" rel="noopener">Scale Donovan</a>; expert algorithms ATR and chain-of-custody play a crucial role as tools enabling the models to expand out from a pure text and structured data domain.</p>
<p dir="ltr">Although there exists extensive research on object tracking in the academic realm, those solutions are largely targeted at accurate assignment of track ID&rsquo;s on top of fixed object detector outputs. These outputs come from a nearly perfect detector on common commercial footage. This footage typically comes from a camera that is mounted front-facing on the ground in a fixed position with fixed zoom levels.&nbsp;</p>
<p dir="ltr">However, the needs of many Federal customers do not align with these cookie-cutter solutions. They often require aerial imagery and full motion video capabilities, which present additional challenges, on top of accurate assignment of track ID&rsquo;s as in the commercial realm:&nbsp;</p>
<ol>
<li dir="ltr" aria-level="1">
<p dir="ltr" role="presentation">The base object detectors are less performant, meaning there is significant room for the tracker to use temporal information to reduce false positives and false negatives.</p>
</li>
<li dir="ltr" aria-level="1">
<p dir="ltr" role="presentation">The camera moves quite a bit in terms of position (i.e., panning) and zoom levels.</p>
</li>
<li dir="ltr" aria-level="1">
<p dir="ltr" role="presentation">Long-term re-identification of targets is non-trivial and mission-critical.&nbsp;</p>
</li>
</ol>
<h2 dir="ltr">Scale&rsquo;s Solution: A Suite of Tracker Post-Processors</h2>
<p dir="ltr">To address these problems, Scale offers a suite of tracker post-processors, each of which can be turned on or off.&nbsp; We optionally package these on top of our own custom, base tracker which achieves state-of-the-art balance between precision and recall when tracking in the presence of camera movement.&nbsp;</p>
<p dir="ltr">These post-processors can be used alone to enhance an existing tracker, or used as an end-to-end tracking package on top of an existing detector. This package can be called in one line of code per frame after each object detector inference (for users using the package) or base tracker inference (for users using just the post-processors).&nbsp;</p>
<h2 dir="ltr">Sample Demo MP4 Results</h2>
<p dir="ltr">Below, we compare our Scale Tracker packaged with our custom base tracker (25FPS on the left) with two of the best publicly available trackers - one that is faster (BYTETrack at 30FPS in the middle) which has comparatively poor track ID permanence, and one that is slower with slightly better track ID permanence than BYTETrack (SmileTrack at 10FPS on the right), although still not as good as Scale Tracker&rsquo;s track ID performance:</p>
<p><img title="image1.gif" src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/01b73d81-ae6e-4a6d-b744-8cdcc98aedce" alt="" width="1200" height="300"></p>
<p><img title="image2.gif" src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/3945304f-e9e7-4aaf-a166-ff9f2ab89934" alt="" width="1200" height="300"></p>
<p><img title="image3.gif" src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/8dd6967b-f7bb-414d-aa5d-58952facd8d8" alt="" width="1200" height="300"></p>
<p dir="ltr">The Scale tracker:</p>
<ol>
<li dir="ltr" aria-level="1">
<p dir="ltr" role="presentation">Flickers less -- this creates more visually persistent tracks thereby creating a better user experience</p>
</li>
<li dir="ltr" aria-level="1">
<p dir="ltr" role="presentation">Is able to maintain track ID&rsquo;s through periods of erroneous track dropping -- this creates tracks that exhibit more permanence</p>
</li>
</ol>
<h2 dir="ltr">Empirical Results</h2>
<h3 dir="ltr">Academic results</h3>
<p><img title="image4.png" src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/0691ca39-f375-405a-8081-c8cfaf0484ad" alt="" width="615" height="438"></p>
<p dir="ltr">Note: The metrics for the other methods in the above chart were taken directly from the <a href="https://arxiv.org/pdf/2202.13514.pdf">StrongSORT paper</a></p>
<p dir="ltr">Even though our tracker is aimed at Federal problems, it actually improves upon the performance of trackers in the commercial realm and achieves state-of-the-art trade-off between speed and performance when paired with BYTETrack, as evaluated by IDF1 (a mixture between precision and recall) on common commercial tracking dataset MOT17.&nbsp;</p>
<p dir="ltr">The chart above demonstrates that ScaleTrack (packaged as our post-processors paired with BYTETrack) achieves a better speed/performance tradeoff than all other methods. BYTETrack and OC-sort are slightly faster, but significantly less performant than ours.&nbsp;</p>
<p dir="ltr">Why the emphasis on the comparison to BYTETrack (and not StrongSORT or SmileTrack)? Because any tracker that cannot be run at near-real time is nearly useless for Full Motion Video (FMV) workflows because most FMV users want algorithms that can be run as they review mission videos. And most cameras collect around 20FPS, meaning only the algorithms to the right of the red line can be of high value to FMV workflows.</p>
<h3 dir="ltr">Federal results</h3>
<p dir="ltr">The above graph is from evaluating an academic dataset. On mission-specific datasets, we outperform even more. As visual evidence of our outperformance over state-of-the-art in Federal datasets, please see the above MP4 sample results.&nbsp;</p>
<h2 dir="ltr">How Did We Achieve These Results? (Technical Deep Dive)</h2>
<h3 dir="ltr">Reduction of flickering</h3>
<p dir="ltr">Detectors basically never have 100% recall in the Federal domain. They often miss objects. Most trackers halt operations when this happens and kill the track. Certain trackers keep the object around in a state that is half dead/half alive for about 5 frames, and provide them the option of being revived if the object appears again.&nbsp;</p>
<p dir="ltr">We went with a different approach. We become skeptical of the detector when a track isn&rsquo;t propagated to the next frame and we try to persist the track by querying the image directly through a two-step process. This two-step process performs bounding box location prediction functions for a given track. The first prediction function is based on kalman information.&nbsp;</p>
<p dir="ltr">However, kalman filters fail sometimes. As a result, Scale did a thorough, detailed study of these failure cases and created a second, custom prediction function in bounding box search space. Given these bounding box hypotheses (from the kalman filter prediction function and our custom prediction function), we use a highly specialized visual feature function (which is not customer-locked or security-locked and therefore can be used by any customer at any security level) to select the bounding box which is most visually consistent with that track&rsquo;s visual history. Based on the level of visual consistency, we either persist the track to the next frame at that selected bounding box&rsquo;s location, or enter it into a state that is not yet dead but not entirely alive (i.e., a bank of tracks from which ReID can draw).</p>
<h3 dir="ltr">Ability to maintain track ID's through short periods of erroneous track dropping</h3>
<p>Sometimes, the flickering reduction mechanisms fail. But it&rsquo;s important that when the track shows up again, it maintains the same ID. So, for a set amount of frames after a track is last seen, we use its kalman filter to project the track over every frame. If there ever exists a new track that is visually consistent with the dead track and that new track exists in a location that is highly likely under the dead track&rsquo;s motion prediction distribution, we assign the dead track&rsquo;s ID to the new track. Thereby we would &ldquo;revive&rdquo; the old track in the new track&rsquo;s location, which allows us to maintain the track ID even though the object was missing for a short while. We refer to this as &ldquo;Local ReID.&rdquo;<br><br>Outside of handling algorithmic false negatives, this also addresses short-term occlusions (e.g. a vehicle driving underneath an occluding tree).</p>
<h3 dir="ltr">Ability to maintain track ID's through long periods in which the track wasn't present</h3>
<p dir="ltr">This addresses the following use cases:</p>
<ol>
<li dir="ltr" aria-level="1">
<p dir="ltr" role="presentation">Objects coming in and out of the frame, potentially at an entry location significantly different from its exit location</p>
</li>
<li dir="ltr" aria-level="1">
<p dir="ltr" role="presentation">Objects that are occluded for long periods of time (e.g., a person coming in and out of a building)</p>
</li>
<li dir="ltr" aria-level="1">
<p dir="ltr" role="presentation">Camera zoom ins, zoom outs, and pans</p>
</li>
</ol>
<p dir="ltr">In these cases, traditional motion prediction breaks down, meaning traditional motion information prediction algorithms cannot be used to link new tracks with dead tracks. However, the objects rarely significantly change their looks within the span of a video. So, we introduce &ldquo;Global ReID&rdquo; which attempts to link dead tracks to new tracks strictly through a harsh visual filter.&nbsp;</p>
<h3 dir="ltr">Ability to maintain track ID's through medium-length periods in which the track wasn't present</h3>
<p dir="ltr">We further address use-cases 1 and 2 in the above section using a ReID function that is based on custom motion functions mixed with the above mentioned visual filter with time between dead and new tracks as a salient input to the mixture function. We do so in such a way that the larger the gap between these tracks, the less the linking function relies on spatial information and the more it relies on visual information, as there is increased uncertainty in motion as the time gap becomes larger.</p>

Assign accurate track ID’s and improve performance on top of aerial object detector outputs with video inputs

Scale's Solution for Aerial Object Tracking | Scale AI

Scale's Solution for Aerial Object Tracking

<h3>Background</h3><p><br></p><p>The rise of social media platforms has driven the popularity of user-generated content (UGC) and it is changing from the bottom up at a pace faster than ever. Platforms are competing for user attention and engagement - some are winning, but some are falling behind. How can a social media platform stay ahead of the game?</p><p><br></p><p>One of the most effective ways to incentivize users to create and engage with content on the platform is by curating and promoting trends. A trend is a type of content repeated and shared amongst creators -&nbsp;a specific subject matter, action, or format that rises in popularity and eventually becomes viral. Common types of trends include dance challenges, transitions, and comedy skits. While videos in a trend follow a similar format, creators usually express their creativity and individuality by adding a personal touch to the videos. For that reason, the variety of the videos within the same trend can be wide and unpredictable.</p><p><br></p><p>The key elements of these video trends include the video’s image, motion, audio, and text overlay. For example, a transition trend is a trend where the participants show a sudden change in their appearance, often signified by a dramatic sound effect in the background music and sometimes accompanied by dance moves, lip syncing, and text overlay. The lifecycle of a trend can vary from days to months depending on various factors. To remain competitive, social media platforms need to detect the trends while they are still emerging and put them to use immediately by matching them to the right creators and promoting them to the suitable audience.&nbsp;</p><p><br></p><p><a href="https://scale.com/content-understanding" rel="noopener noreferrer" target="_blank">Scale Content Understanding</a> improves platform experiences by enriching content metadata, discovering trend insights, and flagging sensitive content. One feature of Content Understanding is content data enrichment for better trend detection. To facilitate this process, we developed proprietary algorithms to detect trends early, quickly, and thoroughly.&nbsp;</p><p><br></p><h3>ML Clustering Overview</h3><p><br></p><p><img src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/449d8647-5509-470c-a6cf-ea38c14a62f3"></p><p><br></p><p>We took a human-in-the-loop (HitL) machine learning approach to detect trends. We represent trends as clusters of videos estimated from video-extracted embeddings. Once formed, these clusters are sent to a human pipeline (which we call “Cluster Labeling”) where taskers answer a series of questions about the videos in a given cluster - namely, if a sample of the videos corresponds to a trend and the potential name of the trend. The following are the phases of the ML Clustering process:</p><p><br></p><h3>Phase 1: Embedding Extraction</h3><p><br></p><p>To extract embeddings, we use a variety of multi-modal information extraction machine learning models, each of which is trained on large-scale, common datasets with high distribution overlap to the social media domain.</p><p><br></p><h3>Phase 2: Clustering</h3><p><br></p><p>Once the embeddings are extracted, we perform a clustering pipeline to take in these embeddings and output clusters.</p><p><br></p><p><strong>Overview</strong></p><p><br></p><p>Given the embeddings, we first perform <a href="https://arxiv.org/pdf/1802.03426.pdf" rel="noopener noreferrer" target="_blank">Uniform Manifold Approximation and Projection (UMAP)</a> dimensionality reduction (inspired by <a href="https://arxiv.org/abs/2203.05794" rel="noopener noreferrer" target="_blank">BERTopic</a>). UMAP takes in high-dimensional data and reduces it down to a lower dimensional space scalably (e.g. UMAP performs computation on tens of thousands of points on the order of tens of seconds). It reduces dimensionality in a way that preserves global Riemannian geometric structure and pairwise distances. On these low-dimensional embeddings, we perform an iterative hierarchical clustering process. Each individual point in embedding space is initially represented as a cluster.&nbsp;We then calculate the proximity between all pairs of clusters. We calculate the proximity of two clusters in a given pair as the variance in embedding space (i.e. the average sum of squares from the joined cluster mean embedding vector) of the points corresponding to if these clusters were to be joined. The two clusters that are closest to each other are joined given the above definition of proximity. This cluster-joining process is performed iteratively until all clusters are above a specified variance that is provided as a hyperparameter.</p><p><br></p><p><strong><img src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/24053e18-8fbc-46a8-91ba-4ea274599636"></strong></p><p><br></p><p><strong>Stop Condition Variance Hyperparameter</strong></p><p><br></p><p>We carefully tune the variance hyperparameter in the stop condition to balance between making the clusters too coarse or too granular. For large variance hyperparameters, the final clusters are allowed to span a wide region in embedding space, meaning they are not semantically similar enough to represent trends (which tend to be extremely specific). Diametrically opposed, for small variance hyperparameters, the clusters end up with too few videos for users in the cluster labeling pipeline to recognize the trend. We therefore carefully tune the stop condition variance hyperparameter. This provides a balance between the clusters being fine enough that they represent groups of videos that are semantically similar enough to correspond to trends, and coarse enough that users from the cluster labeling pipeline can recognize the trend. This hyperparameter is normalized in order to reduce the amount of per-dataset manual tuning.</p><p><br></p><h3>Phase 3: Cluster Post-Processing</h3><p><br></p><p>To improve the quality and throughput of the taskers, we prune out clusters that are composed of a number of videos that is smaller than our definition of a trend. We also add a post-processing density score to help taskers focus on the clusters which are tightest in embedding space - i.e. the ones that are most likely to be semantically similar. This density score is calculated as the intra-cluster variance of each cluster in the dimensionality-reduced embedding space.</p><p><br></p><h3>Conclusion</h3><p><br></p><p>Our trend detection product consistently performs at a higher than 90% recall and 90% precision in production. Our HitL ML approach enables us to stay flexible with customization towards specific content niches while maintaining a high throughput. We are able to process tens of millions of videos with a few hours of turnaround. Our priority is to not only detect trends promptly, but also equip our customers with the data and insights on these trends to take actions immediately.&nbsp;</p><p><br></p><p>At Scale, we believe investing in data is the key to building better platform experiences and discovering trend insights. If you’re interested in learning more about <a href="https://scale.com/content-understanding" rel="noopener noreferrer" target="_blank">Scale Content Understanding</a>, speak to our team today.&nbsp;</p>

Discovering short-form video trends is challenging. In this blog post, we explain how we use an ML + HiTL approach for faster and higher-quality trend detection.

Will LeVine

Scale's Solution for Aerial Object Tracking

Discovering Emerging Short Form Video Trends with Clustering and HitL