Neil Sobin

Lead - Defense Deployment Strategy

<p dir="ltr">In order to facilitate the automation and speed of maintaining surface to air missiles (SAM) systems chain of custody, Scale developed an AI powered system to track movements at these sites using EO satellite imagery. This blog post explores some of the implementation and novel concepts around the problem.</p>
<p dir="ltr">SAM systems are guided missiles that are typically launched from a mobile launch vehicle or a stationary launcher and used to defend against air attacks. Two common stationary SAM systems are the SA-2 and SA-3, often used in former Soviet satellite states.&nbsp;</p>
<p dir="ltr">From overhead imagery, these systems are typically identified by hexagonal patterns in the ground with launchers placed around the hexagon (with or without missiles), a radar placed near the center (typically accompanied by a communication vehicle), and missile transloader vehicles dispersed throughout the site.</p>
<p><img src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/1efbb734-34e8-484f-8d41-f0858d34f69a" alt="" width="1200" height="720"></p>
<h6 dir="ltr" style="text-align: right;"><code class="c-mrkdwn__code" data-stringify-type="code">Image Source: Skyfi</code>&nbsp;</h6>
<p dir="ltr">A SAM system's perceived threat level is defined by the activity over time at a given site and the objects of interest at the site. For example, a site with ground vehicles and launchers present may be considered occupied but not operationally functional and therefore less of a threat. However, if a radar were to show up at that site it could then become operational, which is a potential cause for alert.&nbsp;</p>
<h2 dir="ltr">Machine Learning Approach</h2>
<h3 dir="ltr">Narrowing the problem set</h3>
<p dir="ltr">Tracking the status of SAM systems requires maintaining custody of 4 object classes: radars, ground vehicles, launchers, and missiles. However, there exists nuance about how these objects are detected: SA-2/SA-3 sites can be located in both dense urban and rural settings. Training an object detection model on a general region with a SAM system will lead to perceived annotation errors and false positives.&nbsp;</p>
<p dir="ltr">If, for example, there is a parking lot nearby (as there is in the image above) that does not actually contain vehicles relevant to the status of the SAM site, feeding this parking lot into the neural network without any annotations will likely confuse training and lead to declined model performance. Similarly, during inference time, the network is likely to output a number of false positive predictions around the vehicles in this parking lot when we are only interested in military vehicles relevant and directly within the SAM site.&nbsp;&nbsp;</p>
<p dir="ltr">To counteract this, Scale built a general SAM site object detector which places a focused bounding box around the site itself (see algorithm A in the figure below). This allows preprocessing for cropping out the specific site itself to feed into the final object detection network (see algorithm B in the figure below). Additionally, it provides a stand alone algorithm that can be used for broad area searches of new SA-2/SA-3 sites around the world.</p>
<p dir="ltr"><img src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/0c0b67e2-15fb-4f41-a1c0-c6b0fdb0d5f4" alt="" width="1200" height="430"></p>
<h3 dir="ltr">Including Apriori Knowledge</h3>
<p dir="ltr">Custody tracking in satellite images provides a unique training advantage over a static, stand alone object detection network that intakes a raw image and outputs detection predictions. In the case of custody tracking, there is almost always a prior timepoint of that Area of Interest that has been reviewed by an intel analyst and &ldquo;labeled&rdquo; with points. The work of the intel analyst can provide incredibly valuable information to the neural network learning to detect these objects on its own. To take advantage of this, instead of just feeding in an image of a SAM site to the network, we feed in both the image of the SAM site and a heatmap of the location of the objects of interest from the prior timepoint.&nbsp;</p>
<p><img src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/46203af2-8fdd-4bbd-b10a-7a3c07c10497" alt="" width="1200" height="888"></p>
<p dir="ltr">The addition of these prior heatmaps improved algorithm detection performance across all 4 classes of interest. Object detection performance on these classes is measured using mean average precision at 50% IoU (mAP50).</p>
<div dir="ltr" align="center">
<table><colgroup><col width="218"><col width="277"></colgroup>
<tbody>
<tr>
<td>
<p dir="ltr">Class of interest</p>
</td>
<td>
<p dir="ltr">mAP50% improvement w/ Prior Heatmaps</p>
</td>
</tr>
<tr>
<td>
<p dir="ltr">SAM specific ground vehicles</p>
</td>
<td>
<p dir="ltr">4.2%</p>
</td>
</tr>
<tr>
<td>
<p dir="ltr">Launchers</p>
</td>
<td>
<p dir="ltr">17%</p>
</td>
</tr>
<tr>
<td>
<p dir="ltr">Radars</p>
</td>
<td>
<p dir="ltr">20.9%</p>
</td>
</tr>
<tr>
<td>
<p dir="ltr">Missiles</p>
</td>
<td>
<p dir="ltr">13%</p>
</td>
</tr>
</tbody>
</table>
</div>
<h2 dir="ltr">Conclusion</h2>
<p dir="ltr">The simultaneous transition to great power competition and explosion of imagery data creates an urgent need for imagery analysts to exploit imagery faster than ever before. High performance SAM detection algorithms allow analysts to receive automated disposition of forces insights at machine-speed, freeing analysts to spend their time on more complex analyses. Because the algorithm&rsquo;s outputs are structured data, analysts can further automate downstream processes such as setting up alerts for anomalous behavior or maintaining chain of custody.&nbsp;</p>
<p dir="ltr">At Scale, we believe technology should be used to empower (not replace) the analyst in crafting their narratives to deliver decision advantage.&nbsp;</p>
<p dir="ltr">NOTE:</p>
<p dir="ltr">This material is based upon work supported by the USAF AFRL/SBRK under Contract No. FA864921P0899. Any opinions, findings and conclusions or recommendations expressed in this material are those of&nbsp; the author(s) and do not necessarily reflect the views of the USAF AFRL/SBRK.&nbsp;</p>
<p>&nbsp;</p>

High performance SAM detection algorithms allow analysts to receive automated disposition of forces insights at machine-speed

Neil Sobin

SAM Introduction