BlogCategoryTag

Header

Footer

PreFooter

ProseBlock

Blog

SocialLinks

Prose

AnnouncementContextProvider

UnnamedGlobalGroupOfVariants2

Screen

Segment

Experiment

Experiment2

scale.com (Production)

plasmic-nav

plasmic-embed-css

plasmic-basic-components

plasmic-tabs

plasmic-cms

react-chartjs-2

react-slick

react-quill

Calvin Huang

Engineering

Here at Scale, one of the image annotation services we offer is&nbsp;<a href="https://scale.com/3d-cuboids" rel="noopener noreferrer" target="_blank">Cuboid Annotation</a>, which annotates your two-dimensional images with projections of cuboids enclosing objects such as cars, trucks, pedestrians, traffic cones, you name it. With some additional information, we can turn those two-dimensional box annotations into full, three-dimensional boxes, with height, width, depth, rotation, and relative positioning info. <h2>Manual cuboid annotation</h2> With manual cuboid adjustment, scalers simply draw a 2D box representing one side of the cuboid and an additional side of the cuboid: <img src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/80ca67fd-00d1-4e55-b7a0-e1d22c826c88" alt="cuboid selection" height="186" width="240">ordinary cuboid This often does not result in a "true" cuboid as it is mathematically imprecise. The front face of a "true" cuboid will likely not be a perfect 90 degree rectangle, especially if it isn't facing the camera head-on. Furthermore, edges of a cuboid parallel to the ground should converge to the horizon, whereas the top/bottom edges of the right side of the above annotation are parallel. In our task response, we provide the image coordinates of the points and edges: <pre class="ql-syntax" spellcheck="false">{
 "edges": [
 {"description": "face-top", "x1": 236, "y1": 308, "x2": 328, "y2": 308, "type": "edge"},
 {"description": "face-right", ...},
 {"description": "face-bottom", ...},
 {"description": "face-left", ...},
 {"description": "side-bottom", ...},
 {"description": "side-top", ...}
 ],
 "vertices": [
 {"description": "face-topleft", "x": 236, "y": 308, "type": "vertex"},
 {"description": "face-bottomleft", ...},
 {"description": "face-topright", ...},
 {"description": "face-bottomright", ...},
 {"description": "side-topcorner", ...},
 {"description": "side-bottomcorner", ...}
 ],
 "label": "car"
}
</pre><h2>Adjusted cuboid</h2> Given the cuboid annotation above, and some additional information (namely, camera parameters and orientation), we automatically generate a more accurate annotation: The front face is no longer perfectly rectangular, but trapezoidal with the left edge slightly smaller, better reflecting the relative orientation of the car to the camera. The top/bottom edges of the right side now converge to a point on the horizon, as they should. <img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/767c2267-2c47-46e3-91ab-be1281020400/public" alt="adjusted cuboid selection">adjusted cuboid After cuboid adjustment, we add two new fields&nbsp;<code>points_3d</code> and <code>points_2d</code> to the original annotation: <pre class="ql-syntax" spellcheck="false">{
 "edges": {...},
 "vertices": {...},
 "label": "car",
 "points_3d": [
 {"x": 14.903, "y": 4.117, "z": -0.0928},
 ... // 7 more points
 ],
 "points_2d": [
 {"x": 235.39, "y": 308.05},
 ... // 7 more points
 ]
}
</pre>which represent the 3D coordinates of the vertices1 of the cuboid in space (relative to the position of the camera, in meters) and the pixel coordinates of the resulting 2D projection. From these eight points in 3D-space it's easy to infer the position, dimensions, and orientation of the resulting cuboid. With these attributes, one can train models such as <a href="https://arxiv.org/pdf/1611.10010.pdf" rel="noopener noreferrer" target="_blank">Magic Leap's Deep Cuboid Detection</a>, which predicts 3d coordinates of cuboid-like objects from a single image. In this way, one can create a system to identify the locations of cars in the world, using only camera images and annotations generated by Scale. <img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/e66af6d5-7c8c-4568-a6cd-846a247edd00/public" alt="cuboid annotations">Deep Cuboid Detection Pipeline <h2>Further examples</h2> Initial AnnotationAfter Adjustment <img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/2dd29cfd-8211-41d1-e0a0-258189a9db00/public"><img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/f2006f44-0454-466a-f24e-8a9076476c00/public"> <img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/5cca4474-7fce-4132-3e36-fa409ae90200/public"><img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/c6463b92-2b20-459e-f506-e2aa665efc00/public"> <img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/50106285-eed1-4ef8-0a8a-f3f263730600/public"><img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/68665344-da48-47a0-7c9e-eae355031100/public"> Because it's impossible to visually distinguish between one object, and another object twice the size and twice as far away, it's only possible to calculate 3D coordinates up to a scaling factor (i.e. you can multiply the points_3d of the resulting cuboid by an arbitrary value, and it would be just as valid). With additional data, such as depth data from LIDAR or from stereo imaging, one knows the exact distance from the front face of the cuboid and can scale the points accordingly. Alternately, if the elevation of the camera is provided, we scale the resulting cuboids so that the bottom face of each cuboid touches the ground.

Annotating 3d Objects with 2D Data - Scale

Adding a Dimension: Annotating 3d Objects with 2d Data

open-dataset-nucleus

slugs must be all lowercase. ie: ‘my-slug’

events-banner

This entry represents both interview quote highlight and interview chat highlight because of Builder limitations. Choose the fields that matches either of them.

interview-highlight

If it's selected, the author and quote fields will be taken, otherwise the chat fields. The default is for chat.

resources-download-model

transform-session

CTA to open the session. Leave blank if you don't want a cta for that session

Link for the CTA. Usually the link to the exchange session

blog-filter-type

event-transform-x-ticket

solution

homepage-quotes

logos

homepage-content

builder-debug

transfrom-x-in-person

event-model

Toggle if the event is featured. Featured events will be moved to the top of the list

guides-cta

summit-2023-partners

sales

z[deprecated] page

quotes

z[deprecated] compare

product

blog-category-model

announcement-banner

accelerate

Vercel Powered and Edge Computed OG Image. Preferred over og img file upload. If this field is non empty, it will take precedent over the uploaded file

summit-2023-speakers

homepage-resources

customers

Display order on the page /customers. Higher values show first

page-cross-product-webinar

interview

event-transform

Event model for new marketing events page

event-2-0

homepage-event

custom-fonts

transform-x-speaker

Be careful to not add any trailing white spaces to the slug. Slugs must be all lowercase. ie: ‘my-slug’

Order of the featured speakers displayed on the home page for transformX

z[deprecated] document-ai

summit-2023-agenda

event-converge

hosts

author

transform-x

Page: /In-Person. Section: Hero. Text: Title Text

Page: /In-Person. Section: Slider Card. Text: Title Text

Page: /In-Person. Section: Slider Card. Text: Card Body

Page: /In-Person. Section: Hero. Text: Tagline

open-dataset

z[deprecated] customers

URL redirections for Website. Disclaimer: takes 5~ min for each added or edited.

url-redirects

Use relative path. use `/blog/foo` not `https://scale.com/blog/foo`

legal

event

Optional second image intended to show on the grid view, should be a more squared image closer to 1:1 aspect ratio than the one used in the Cover field.

guides

z[deprecated] guides-old

z[deprecated] guides-lp

partners

blog-model

Option to list or not list this post in the Blog index. Enabled = not list in the index, Disabled = list in the index

Products

Government

Solutions

Resources

Customers

Pricing →

Adding a Dimension: Annotating 3d Objects with 2d Data

The future of your industry starts here.