experiment

Header

Footer

PreFooter

SectionTitle

PaginationButton

BlogPage

Trusted by world class companies, Scale delivers high quality training data for AI applications such as self-driving cars, mapping, AR/VR, robotics, and more.

AI & ML Blog | Scale AI

NextImage

AnnouncementContextProvider

Screen

plasmic-graphcms

scale.com (Production)

lottie-react

plasmic-nav

react-scroll-parallax-global

plasmic-embed-css

plasmic-query

plasmic-basic-components

plasmic-tabs

react-aria

plasmic-cms

react-chartjs-2

plasmic-strapi

react-slick

radix-ui

react-quill

Customers

Product

Company

Engineering

Scale relies on Mode Analytics for internal SQL and Python based reporting and analytics, helping power our operations, insights and business decisions. Many of these reports we set to run on a schedule, so that we can always go to a report and see data that’s no more than fifteen minutes to an hour out of date. However, Mode doesn’t natively support a method of notifying people when something needs attention, e.g. when our margins on a project take a sharp dive or many new tasks are suddenly created. We’ve set up a system to utilize these reports to easily alert us when something’s unusual, so that someone can take action.<h2>Alert generation</h2> <img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2Ff0583e4d10994410bb5db605266bf655" alt="Scale and Slack Dashboards"> The flow for alerting goes as follows: when a report is run (either manually or on a schedule), it sends a webhook to an AWS Lambda function, which then pulls the report data from the Mode API, and retrieves a list of alert definitions from our database. It then looks through the alert definitions and sees if any are applicable to the current report being run, and if so, creates an alert for each row in the report that satisfies conditions set out in the matching alert definition. These alerts are then sent to a Slack webhook, which generates an alert in a channel specified in the alert definition.<img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/2b8f800f-0c2f-4a52-53d5-d2a3205d8100/public" alt="Slack Dashboard"> <h2>Defining alerts</h2> <img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/8a14c54f-48ce-48b0-0d62-dbe006402700/public" alt="Alerts Dashboard"> Alerts are defined in our database, and are easily viewable/creatable from our corp dashboard.Alerts are simply defined with a handful of fields: <ul><li>Report URL: the URL of the Mode report that this alert can</li><li> fire for.</li><li> </li><li>Alert Title</li><li>Alert Condition: Python expression that is evaluated with</li><li> every row of a report to determine if that row is alert-worthy.</li><li> </li><li>Filter-max Column: name of a column to only take rows where</li><li> the value of that column is maximal. Useful for reports with time-series</li><li> data, usually you only want to alert on rows with the most recent data.</li><li> </li><li>Alert Text Format String: Python format string, used to</li><li> generate the message text in the Slack alert.</li><li> </li><li>Color: sidebar color of the alert message</li><li>Alert Channel: Slack channel to post to</li><li>Enabled</li><li> </li></ul><img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2F54c463b0ae5740d5af829e3d57c46a96" alt="Report Dashboard"> The combination of the easy-to-use dashboards, and the simple definitions of the alerts, makes creating alerts based on reports very easy to non-technical people—all that’s required to make fully-featured custom alerts is very basic Python knowledge—essentially basic math operations—and SQL if one is creating their own reports from scratch.

Scale relies on Mode Analytics for internal SQL and Python based reporting and analytics, helping power our operations, insights and business decisions.

How to Trigger Alerts With Mode Analytics - Scale

How We Made It Easy for Anyone at Our Company to Trigger Alerts

We have exciting news to share today! Scale raised $18M in Series B funding led by Index Ventures with Accel, Y Combinator, Drew Houston, and Justin Kan joining. Mike Volpi of Index Ventures will be joining our board. Scale has come a long way from the dorms at MIT, and we still have a long way to go. Our mission at Scale is to accelerate the development of AI applications. We are strong believers in the transformational impact of AI and deep learning across every piece of the global economy—from manufacturing and agriculture to autonomous vehicles and security. Deep learning is eating the world in computer vision—models that outperform humans have become standard. It is an important and exciting transition for us—the future will happen very quickly. The bottleneck that becomes apparent to every practitioner is labeled data. The most significant bit to the performance of your AI software is no longer the code you write, but rather the data you are training on. In fact, your code becomes a servant to your ever-growing data corpus—you build hacks to maintain the key invariant of data integrity. The way you improve your software is not to write more code, but to label more data. So when we founded Scale, we devoted ourselves to solving the problem of data labeling. We saw parallels between building web software before cloud computing and AI development today. Before cloud computing, a significant hurdle to building on the internet was building and operating large server farms to power your software. Ultimately, that activation energy prevented many applications from being built. In the same way, a significant hurdle to building machine learning is building the capability to label large datasets. We’re aiming to accomplish exactly what cloud computing did—offer reliable, cost-effective, scalable infrastructure to make it easier to build incredible applications. Scale turned 2 years old last month, and we are well on our way. In autonomous driving, one of the most prominent applications of deep learning today, Scale has become the industry standard for labeling data. We’ve partnered with many industry leaders such as GM Cruise, Lyft, Zoox, and nuTonomy. We’ve labeled more than 200,000 miles of self-driving data (about the distance to the moon). Scale is also used by tech giants like Airbnb and Pinterest, and Fortune 500 companies like P&amp;G and Liberty Mutual. We are proud that our partners have made Scale an integral and trusted part of their pioneering work. Our Series B funding reflects the next stage in our journey, and we are not slowing down. We have a lot left to build, from supporting new applications to creating the best experience around training on labeled datasets. We strive to significantly accelerate the development of AI in every application that makes the future happen faster. To that end, one of our primary goals with the new funding will be growing <a href="https://scale.com/blog/scale-engineering-team" rel="noopener noreferrer" target="_blank">our team</a>. If you want to make a difference in the world of AI, come <a href="https://scale.com/about#jobs" rel="noopener noreferrer" target="_blank">join us</a> on our mission. AI is just taking off. We’re excited to be fueling it.

Scale raised $18M in Series B funding led by Index Ventures with Accel, Y Combinator, Drew Houston, and Justin Kan joining.

Scale raised $18M in Series B with Index Ventures and Accel

Announcing Our Series B with Index Ventures and Accel

Here at Scale, one of the image annotation services we offer is&nbsp;<a href="https://scale.com/3d-cuboids" rel="noopener noreferrer" target="_blank">Cuboid Annotation</a>, which annotates your two-dimensional images with projections of cuboids enclosing objects such as cars, trucks, pedestrians, traffic cones, you name it. With some additional information, we can turn those two-dimensional box annotations into full, three-dimensional boxes, with height, width, depth, rotation, and relative positioning info. <h2>Manual cuboid annotation</h2> With manual cuboid adjustment, scalers simply draw a 2D box representing one side of the cuboid and an additional side of the cuboid: <img src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/80ca67fd-00d1-4e55-b7a0-e1d22c826c88" alt="cuboid selection" height="186" width="240">ordinary cuboid This often does not result in a "true" cuboid as it is mathematically imprecise. The front face of a "true" cuboid will likely not be a perfect 90 degree rectangle, especially if it isn't facing the camera head-on. Furthermore, edges of a cuboid parallel to the ground should converge to the horizon, whereas the top/bottom edges of the right side of the above annotation are parallel. In our task response, we provide the image coordinates of the points and edges: <pre class="ql-syntax" spellcheck="false">{
 "edges": [
 {"description": "face-top", "x1": 236, "y1": 308, "x2": 328, "y2": 308, "type": "edge"},
 {"description": "face-right", ...},
 {"description": "face-bottom", ...},
 {"description": "face-left", ...},
 {"description": "side-bottom", ...},
 {"description": "side-top", ...}
 ],
 "vertices": [
 {"description": "face-topleft", "x": 236, "y": 308, "type": "vertex"},
 {"description": "face-bottomleft", ...},
 {"description": "face-topright", ...},
 {"description": "face-bottomright", ...},
 {"description": "side-topcorner", ...},
 {"description": "side-bottomcorner", ...}
 ],
 "label": "car"
}
</pre><h2>Adjusted cuboid</h2> Given the cuboid annotation above, and some additional information (namely, camera parameters and orientation), we automatically generate a more accurate annotation: The front face is no longer perfectly rectangular, but trapezoidal with the left edge slightly smaller, better reflecting the relative orientation of the car to the camera. The top/bottom edges of the right side now converge to a point on the horizon, as they should. <img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/767c2267-2c47-46e3-91ab-be1281020400/public" alt="adjusted cuboid selection">adjusted cuboid After cuboid adjustment, we add two new fields&nbsp;<code>points_3d</code> and <code>points_2d</code> to the original annotation: <pre class="ql-syntax" spellcheck="false">{
 "edges": {...},
 "vertices": {...},
 "label": "car",
 "points_3d": [
 {"x": 14.903, "y": 4.117, "z": -0.0928},
 ... // 7 more points
 ],
 "points_2d": [
 {"x": 235.39, "y": 308.05},
 ... // 7 more points
 ]
}
</pre>which represent the 3D coordinates of the vertices1 of the cuboid in space (relative to the position of the camera, in meters) and the pixel coordinates of the resulting 2D projection. From these eight points in 3D-space it's easy to infer the position, dimensions, and orientation of the resulting cuboid. With these attributes, one can train models such as <a href="https://arxiv.org/pdf/1611.10010.pdf" rel="noopener noreferrer" target="_blank">Magic Leap's Deep Cuboid Detection</a>, which predicts 3d coordinates of cuboid-like objects from a single image. In this way, one can create a system to identify the locations of cars in the world, using only camera images and annotations generated by Scale. <img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/e66af6d5-7c8c-4568-a6cd-846a247edd00/public" alt="cuboid annotations">Deep Cuboid Detection Pipeline <h2>Further examples</h2> Initial AnnotationAfter Adjustment <img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/2dd29cfd-8211-41d1-e0a0-258189a9db00/public"><img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/f2006f44-0454-466a-f24e-8a9076476c00/public"> <img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/5cca4474-7fce-4132-3e36-fa409ae90200/public"><img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/c6463b92-2b20-459e-f506-e2aa665efc00/public"> <img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/50106285-eed1-4ef8-0a8a-f3f263730600/public"><img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/68665344-da48-47a0-7c9e-eae355031100/public"> Because it's impossible to visually distinguish between one object, and another object twice the size and twice as far away, it's only possible to calculate 3D coordinates up to a scaling factor (i.e. you can multiply the points_3d of the resulting cuboid by an arbitrary value, and it would be just as valid). With additional data, such as depth data from LIDAR or from stereo imaging, one knows the exact distance from the front face of the cuboid and can scale the points accordingly. Alternately, if the elevation of the camera is provided, we scale the resulting cuboids so that the bottom face of each cuboid touches the ground.

Annotating 3d Objects with 2D Data - Scale

Adding a Dimension: Annotating 3d Objects with 2d Data

TLDR: Row stores are fast to write but slow to read. Column stores are fast to read but slow to write. Load data from Mongo into Parquet files for fast querying using AWS Athena. MongoDB is a schema-less NoSQL document store that uses a JSON-like format for each document. Each document in Mongo WiredTiger is stored as a contiguous binary blob, which makes our MongoDB instance a row store. AWS Athena is a SaaS offering by Amazon that queries files on S3 using Presto, a distributed SQL query engine that can query high-performance columnar formats like Parquet and ORC. Here at Scale we use MongoDB backed by WiredTiger as our primary data store. This blog post describes why and how we load data from MongoDB into AWS Athena using Parquet files in order to get fast ad-hoc analytics queries on our data.<h2>Row stores and Column stores</h2> Suppose you want to get the sum of a single column(field) over all the rows(documents) in a table(collection). With a Row Store, without an index, to get a field you need to fetch the document, and to get the same field for all documents you would have to essentially load the entire table from disk. All the columns in your table (even columns that do not appear in your query) get fetched as well, which is slow:<img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2Ff746257aeb8d477d9d6d912f1a328ce7" alt="Data Fields"> You can speed up this query by building a covering index. However, each index you add increases disk usage and slows down inserts. Also, when doing data exploration it can be difficult to predict what indices you need ahead of time. With a Column Store, the query is much more efficient. A column store puts data from the same column together. This would allow us to just retrieve the columns we are filtering / aggregating over when doing queries. This allows us to get good performance on ad-hoc queries, without indices:<img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2Fae046f529b8a44e79aa710d117685d41" alt="Column Fields"> Column stores, however, make it very difficult to update a single record - due to the way data is arranged you have to move all of the data to change one record. Also, column stores require us to commit to a schema, to define the columns ahead of time.<h2> </h2><h2> Trading off read and write performance</h2> The slowness of column-store writes is related to the size of the table, because you need to rewrite the whole table to change a single record. By representing each row-store table as multiple column-store tables, we can get the column store to have faster writes at the expense of slowing down its reads. For our implementation, this means we map each Mongo collection to multiple Parquet files (each file actually contains multiple row groups, with each row group being column-major, but we use files as a unit for convenience), and refresh all the documents in a Parquet file whenever any one of them changes. We use smaller file sizes than what would be optimal for reads, in order to make writes faster and reduce replication lag.<h2>Cache Invalidation</h2> When updating the column store from reading the row store, we need to <ol><li>Figure out which records are stale</li><li>Update those records efficiently</li><li> </li></ol> To determine staleness, we need to know the identity of the document and the time of modification. This data is obtained by reading the Mongo oplog, a data structure that represents all inserts/updates/deletes to the database chronologically, available on Mongo replica sets for internal use in replication from primary to secondary nodes within the cluster. For efficiency, we want to ensure that documents to be modified belong in as few files as possible. We achieve this by grouping records by their Mongo <a href="https://docs.mongodb.com/manual/reference/method/ObjectId/" rel="noopener noreferrer" target="_blank">ObjectIDs</a>. We are making use of the fact that, like in <a href="http://wiki.c2.com/?GenerationalGarbageCollection" rel="noopener noreferrer" target="_blank">Generational Garbage Collection</a>, young documents tend to get modified more than old ones. We maintain an index recording the ObjectID ranges of each file, starting a new file whenever the newest bucket is filled with more than N documents.<pre class="ql-syntax" spellcheck="false">class Buckets(object):
 def __init__(self, size_limit):
 self.size_limit = size_limit
 self.buckets = [ObjectId("0" * 24)]
 self.max_bucket = set()

 def find_le(self, x):
 'Find rightmost value less than x'
 i = bisect.bisect_left(self.buckets, x)
 if i:
 return i - 1
 raise ValueError

 def get_bucket(self, oid):
 i = self.find_le(oid)

 # oid is in one of lower buckets
 if i &amp;lt; len(self.buckets) - 1:
 return self.buckets[i], self.buckets[i + 1]

 # oid is in maximum bucket
 self.max_bucket.add(oid)

 if len(self.max_bucket) &amp;lt; self.size_limit:
 return self.buckets[i], max(self.max_bucket)

 # maximum bucket is oversized, start a new bucket
 self.buckets.append(max(self.max_bucket))
 self.max_bucket = set()
 return self.buckets[i], self.buckets[i + 1]
</pre><h2>Implementation Details</h2> <ol><li> </li><li> The scripts that read from mongo and create parquet files are written in</li><li> Python and use the</li><li> <a href="https://pypi.org/project/pyarrow" rel="noopener noreferrer" target="_blank">pyarrow library</a> to write</li><li> Parquet files. The first version implemented a filter-and-append strategy</li><li> for updating Parquet files, which works faster than overwriting the entire</li><li> file. This proved to be a mistake as pyarrow (8.9.0) support for reading is</li><li> much less mature than that for writing. We would occasionally lose data when</li><li> reading rows for rewriting into the new parquet file, and eventually gave up</li><li> and went to an implementation that never read from the Parquet files.</li><li> </li><li> </li><li> Each Parquet file is made of row groups, and there is a size limit for each</li><li> row group. We batch documents by size to avoid this problem.</li><li> </li><li> </li></ol><pre class="ql-syntax" spellcheck="false">def islice_docs(docs, size):
 tot = 0
 for doc in docs:
 tot += len(doc.raw)
 yield doc
 if tot &gt; size * 1024**2:
 return

def batch_docs(docs, size):
 sourceiter = iter(docs)
 while True:
 batchiter = islice_docs(sourceiter, size)
 try:
 yield itertools.chain([next(batchiter)], batchiter)
 except StopIteration:
 return
</pre><ol><li> </li><li> When using pymongo, use RawBSONDocument to defer parsing. In the case of</li><li> multiple processes writing different Parquet files, this means you avoid</li><li> having parsing happen in the serial part of your program. You can</li><li> selectively parse sub-documents using bson.BSON</li><li> </li><li> </li></ol><pre class="ql-syntax" spellcheck="false"> def realize_raw_bson(d):
 if isinstance(d, RawBSONDocument):
 return bson.BSON(d.raw).decode()
 if isinstance(d, list):
 return [realize_raw_bson(v) for v in d]
 return d
</pre><ol><li> </li><li> Use <a href="https://pypi.org/project/atomicwrites/" rel="noopener noreferrer" target="_blank">atomicwrites</a> to</li><li> make writing a new Parquet file all-or-nothing</li><li> </li><li> </li></ol><pre class="ql-syntax" spellcheck="false">@contextmanager
 def atomic_parquet_writer(output, *args):
 os.makedirs(os.path.dirname(output), exist_ok=True)
 with atomic_write(output, mode="wb", overwrite=True) as f:
 with pq.ParquetWriter(f, *args) as writer:
 yield writer
</pre><ol><li> </li><li> On top of extracting explicitly enumerated fields, put the whole json in a</li><li> column by itself for querying fields not in the schema</li><li> </li><li> </li><li> For reading the Parquet files locally using SQL,</li><li> <a href="https://drill.apache.org/" rel="noopener noreferrer" target="_blank">Apache Drill</a> is very convenient -</li><li> just untar and run drill-embedded</li><li> </li><li> </li><li> After updating a parquet file, it has to be copied to S3 (using sync, not</li><li> cp). Using EC2 is essential here because we are constantly copying files</li><li> from the server to S3, and the bandwidth costs would be prohibitive</li><li> otherwise - traffic between EC2 and S3 in the same region is free. After</li><li> copying the file to S3, use AWS Glue to discover the schema from the files -</li><li> call Glue only when the schema changes, AWS charges you each time you call</li><li> it.</li><li> </li><li> </li><li> The oplog contains all the information you need to update the Parquet files.</li><li> We run an instance of mongo on the same machine, and maintain a one-node</li><li> slave as it were by using the applyOps command on oplog entries read from</li><li> the main cluster. Beware, however - Mongo developers might</li><li> <a href="https://jira.mongodb.org/browse/SERVER-33182" rel="noopener noreferrer" target="_blank">remove the docs</a></li><li> for applyOps completely in the future.</li><li> </li><li> </li></ol><img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2Fdf9dcbefbd7f43ce97d5b6780e60ac25" alt="Scale's flow">

How to make MongoDB not suck for analytics - Scale

How to make MongoDB not suck for analytics

We’ve had an incredible group of people join Scale—and we’re still <a href="https://scale.ai/about#jobs" rel="noopener noreferrer" target="_blank">hiring</a>! We wanted to take a moment to call out the amazing engineering team at Scale: <ul><li>Ricky Rauch / <a href="https://twitter.com/rickyrauch" rel="noopener noreferrer" target="_blank">Ricky</a> was the first designer and later Head of Design at <a href="https://auth0.com/" rel="noopener noreferrer" target="_blank">Auth0</a>. He has designed many developer-focused products including <a href="https://github.com/gitlabhq/gitlabhq" rel="noopener noreferrer" target="_blank">GitLab</a>, <a href="https://jwt.io/" rel="noopener noreferrer" target="_blank">JWT.IO</a>, and <a href="http://www.passportjs.org/" rel="noopener noreferrer" target="_blank">Passport.js</a>.</li></ul> <ul><li>Nate Herman / Nate most recently worked on infrastructure engineering at Dropbox. In the past he built an <a href="https://www.wired.com/2012/09/gameception-doom-iphone/" rel="noopener noreferrer" target="_blank">iOS port of Doom</a> and its associated mods, as well as a popular <a href="https://venturebeat.com/2016/08/18/new-pokemon-go-bot-launches-as-niantic-starts-permanently-banning-cheaters/" rel="noopener noreferrer" target="_blank">Pokemon Go</a> automator. He studied computer science at Harvard with a research focus on <a href="http://www.read.seas.harvard.edu/~kohler/pubs/herman16type-aware.pdf" rel="noopener noreferrer" target="_blank">concurrent computing</a>.</li><li> </li><li> </li><li>Steven Hao / Steven graduated from MIT and previously had stints at Jane Street, DE Shaw, and Quora. He received a gold medal at <a href="http://stats.ioinformatics.org/people/3113" rel="noopener noreferrer" target="_blank">IOI</a> and top honors at programming competitions including Codeforces and ACM-ICPC. His personal projects include the collaborative crossword site <a href="https://downforacross.com/" rel="noopener noreferrer" target="_blank">Down For A Cross</a> and a web implementation of <a href="http://set.stevenhao.com/" rel="noopener noreferrer" target="_blank">Set</a>.</li><li> </li><li> </li><li>Leigh Marie Braswell / Leigh Marie has previously worked at Google, Jane Street, and Blend with roles in software engineering, machine learning research, and product management. She was recognized as one of the top 10 competitive female mathematicians in the United States by the Math Prize for Girls Competition. She studied computer science at MIT.</li><li> </li><li> </li><li>Calvin Huang / <a href="https://calvinhuang.me/" rel="noopener noreferrer" target="_blank">Calvin</a> graduated from MIT with a research focus in <a href="https://dreal.github.io/" rel="noopener noreferrer" target="_blank">theorem solving</a> and natural language processing, and with experience at Quora and Google prior to joining Scale. He received two gold medals at <a href="http://ipho2013.dk/ipho2013-results-gold.htm" rel="noopener noreferrer" target="_blank">IPhO</a> as well as the award for Best Physics Experiment, and was a USAJMO winner. He has been known to build robots.</li><li> </li><li> </li><li>Guido Maliandi / Guido previously founded a virtual reality startup for collaborative environments in VR using Unity and three.js. He has led engineering teams at startups for the past nine years and was a major contributor to Microsoft’s <a href="https://prismlibrary.github.io/docs/" rel="noopener noreferrer" target="_blank">Prism library</a>.</li><li> </li><li> </li><li>Zhichun Li / Zhichun has previously worked at Lightspeed China Partners, Facebook, Microsoft and Airbnb with roles in investment and software engineering. She is a producer of <a href="https://mp.weixin.qq.com/s/Kph9YZ1Nvwf4pCt9inT0-A" rel="noopener noreferrer" target="_blank">VC Pulse</a>, a podcast spotlighting venture capitalists in China. Zhichun was the youngest ever admit to the Yale MBA program, and studied computer science at CMU. She has notably been featured on Eminem’s Snapchat story.</li><li> </li><li> </li><li>Rilka / <a href="http://rilka.li/" rel="noopener noreferrer" target="_blank">Rilka</a> previously led design and frontend engineering at a startup and worked as a full-stack engineer at Dropbox. Along with Nate she built a popular <a href="https://venturebeat.com/2016/08/18/new-pokemon-go-bot-launches-as-niantic-starts-permanently-banning-cheaters/" rel="noopener noreferrer" target="_blank">Pokémon Go</a> automator; other past projects include desktop Threes, Facebook Unfriend Finder, and the Quantified Piñata. She studied computer science at Harvard and (briefly) architecture at Parsons.</li><li> </li><li> </li><li>Sam Clearman / Sam received a PhD in mathematics studying <a href="https://vimeo.com/123640110" rel="noopener noreferrer" target="_blank">combinatorial interpretations of quantum immanants</a>. He was previously an engineer at Google working on Google Play Services. He also built a variant of Tetris called <a href="http://www.samclearman.com/2etris/" rel="noopener noreferrer" target="_blank">2etris</a>.</li><li> </li><li> </li><li>Chiao-Lun Cheng / Chiao received a PhD in computational chemistry from MIT, and has a diverse background in management consulting, quantitative trading, backend engineering, and data science. He has built models for supermarket revenue prediction, bookmaking sports bets, and PCB defect detection, and is intimately familiar with bugs in the Interactive Brokers API.</li><li> </li><li> </li><li>Akshat Bubna / Akshat was the first competitor from India to receive a gold medal at the <a href="http://stats.ioinformatics.org/people/3063" rel="noopener noreferrer" target="_blank">IOI</a>. He previously has worked at DE Shaw and database and fintech startups, and studied computer science and mathematics at MIT.</li><li> </li></ul><h2>Contractors</h2> <ul><li> </li><li>Albert Sheu / Albert was the second engineer at Quora and built a Python framework to render webpages on multiple cores, which sped up Quora by <a href="https://blog.quora.com/Improving-Site-Speed" rel="noopener noreferrer" target="_blank">up to 10x</a>. He also was the Head of Engineering at Patreon and an early engineer at both Facebook and Twitter.</li><li> </li></ul>

Scale Engineering Team - Meet our Team - Scale

Scale Engineering Team++

You can now be a part of a team on Scale! Until recently, only the original creator of a task could access its history, visually inspect responses, and provide direct feedback on quality. It was a hurdle to our users getting the fullest possible value out of our platform. That's why we're excited to announce Teams, which will allow multiple users to have access to the same set of tasks.<h2>Inviting team members</h2> To take advantage of Teams, head over to “Your Team” in your settings menu. As a team administrator, you can invite new team members, manage existing teammates, and update your team name and payment information.<h2>Shared task history</h2> All members in a team will have access to the team's full Task History on the Scale dashboard. Team members can also quickly access team API keys and create new tasks as part of the team.<img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2F84b4897a723f43fb97a56b46f2b51335" alt="Scale's Dashboard"> <h2>Quality assurance audit</h2> All team members have access to our Quality Assurance tool, where you can grab a random sample of completed tasks (filterable by task type, completion date, and more) and audit task responses using the same interface that our workers use to complete tasks.<img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2Fd04705d0f82d49ca9a220a15ee97866d" alt="Scale's Quality Assurance"> We hope that this improves your Scale experience! If you have questions or other feedback, <a href="https://twitter.com/scale_ai" rel="noopener noreferrer" target="_blank">we'd love to hear from you</a>.

Introducing Teams - Complete Tasks on the Fly - Scale

Introducing Teams

Today we are happy to announce a new version of our customer dashboard! We’ve cleaned up the dashboard so it’s faster and more efficient for you to use. Our mission is to help you become more productive and successful using Scale. In order to achieve this, we’ve <a href="https://scale.com/dashboard" rel="noopener noreferrer" target="_blank">redesigned our dashboard</a> to better reflect how tasks are being completed with an overview of your project’s progress.<h2>New Layout</h2> Immediately, you’ll notice the new layout. The header has a new search input that allows you to search for tasks by ID on any page. Beside the header, you can manage your API keys (via Manage API Keys), making it easier to access your keys and quickly integrate Scale in your code.<img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2Fd6176a7e06694a669b105644d2b1870c" alt="Scale's dashboard"> The sidebar navigation has direct access to the main sections (Overview, Tasks, and Payments) and Support (Chat &amp; Documentation).<h2>Overview</h2> We distributed our page’s elements more cleanly, and we updated the chart and the date filters as seen in the Overview.<img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2F2ffab995d1c84b1d9b785410e57c3616" alt="Scale's dashboard"> <h2>Tasks</h2> This is one of the most important features of our application, as this is where the detailed information about your tasks are shown.<img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2Fe7863fde39604fbb9a90255fa4ed4c27" alt="Scale's dashboard"> Here you will find the status and details of your tasks. You are also able to export the dataset and view the final result of a completed task.<h2> </h2><h2> Technical Aspects of Building the Dashboard</h2> A year ago, we decided to move all of our frontend to <a href="https://github.com/zeit/next.js/" rel="noopener noreferrer" target="_blank">Next.JS</a>, making it possible to have our website, blog, scaler tooling, and dashboard all in the same stack. This helped us move faster while sharing and extending react components.<img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/ee1030dc-13c2-47a4-4cdc-7020e6c1c800/public" alt="Scale's dashboard"> In order to create more consistency, we have a <a href="https://scale.ai/ui-library" rel="noopener noreferrer" target="_blank">UI library where all of our components are stored</a>. At the time of starting the redesign of the dashboard, we had most of the components designed and implemented, so the process of creating the updated UI was smooth and fast. All of our CSS is written in styled-jsx with PostCSS plugin, meaning we only have a few lines of Global CSS. Our next steps in terms of frontend, are to implement Redux, to better handle different states of the UI, and continue updating our UI Library with more definitions in terms of: Typography, Backgrounds and Branding.<h2>Our Future Roadmap</h2> <ol><li>Onboarding: By having our</li><li> Onboarding tool be the first thing you see as you enter our</li><li> site, we further our goal of helping developers quickly jump in and start</li><li> using Scale. With this new feature, we show the decision makers at your</li><li> company how our services can quickly and seamlessly begin improving your</li><li> productivity. We are in the process of implementing this into our website as</li><li> the final touches are being placed into our dashboard.</li><li> </li><li>Responsive Design We know it can be frustrating to check</li><li> the status of your tasks without having the ability to on your handheld</li><li> devices. While it wasn't included in this release,</li><li> we will support mobile devices in the very near future.</li><li>Payment Model We are soon introducing a better way to</li><li> experience Scale with a simpler pricing model that makes our rates clear to</li><li> the user without any ambiguity or uncertainty. Stay Tuned.</li><li> </li><li> </li></ol><h2>Conclusion</h2> We know having a quick feedback loop is very important in order to achieve our mission of making our customers more produc__tive and comfortable using our product. Our improvements to the dashboard are meant to help our users quickly see how their projects and tasks are developing. We hope you enjoy our dashboard’s new and improved developments as much as we do! 😃 As always, we’d <a href="https://twitter.com/scale_ai" rel="noopener noreferrer" target="_blank">love to hear your comments and opinions</a>.

New Scale Dashboard - Easily Managing Your API Keys - Scale

Introducing Our New Dashboard

One of the most obvious signs that a site is made with <a href="https://github.com/zeit/next.js/" rel="noopener noreferrer" target="_blank">Next.JS</a> is that you load the page, you click a link to another section and it loads instantly. That’s because when a page is loaded, <a href="https://github.com/zeit/next.js/" rel="noopener noreferrer" target="_blank">Next.JS</a> will download in the background all the other pages linked with tags. This works great for static pages, but as soon as you have to fetch some data in the getInitialProps, loading that page will take time even if it the page code is prefetched. And what is worse — unless you explicitly add a loading indicator, it will usually take time with no indication in the UI! Prefetching and caching data is not trivial — for example, it could be wrong to serve a slightly outdated version of certain information. Also, different data could have different caching rules. Here at Scale we decided to implement a lightweight caching and data prefetching strategy, which allows having this instant navigation experience without compromising data freshness. <h2>Caching in getInitialProps</h2> On certain parts of our site, we know it is OK to show data that is a few minutes old. For those cases, we implemented a simple caching strategy that stores data with a 5 minute TTL. The implementation uses an abstraction around the fetch api that we called <code>cached-json-fetch</code>: <pre class="ql-syntax" spellcheck="false">import lscache from 'lscache';
import fetch from 'isomorphic-fetch';
const TTL_MINUTES = 5;
export default async function(url, options) {
// We don't cache anything when server-side rendering.
// That way if users refresh the page they always get fresh data.
if (typeof window === 'undefined') {
return fetch(url, options).then(response =&gt; response.json());
}
let cachedResponse = lscache.get(url);
// If there is no cached response,
// do the actual call and store the response
if (cachedResponse === null) {
cachedResponse = await fetch(url, options)
.then(response =&gt; response.json());
lscache.set(url, cachedResponse, TTL_MINUTES);
}
return cachedResponse;
}
export function overrideCache(key, val) {
lscache.set(key, val, TTL_MINUTES);
}
</pre>We then use this in the <code>getInitialProps</code> of our pages: <pre class="ql-syntax" spellcheck="false">import React from 'react';
import cachedFetch, { overrideCache } from 'lib/cached-json-fetch';
const SOME_DATA_URL = '/some_data';
export default class SomePage extends React.Component {
static async getInitialProps(ctx) {
const someData = await cachedFetch(SOME_DATA_URL);
const isServerRendered = !!ctx.req;
return { someData, isServerRendered };
}
componentDidMount() {
// When the page is server-rendered,
// we override the value in the client cache
if (this.props.isServerRendered) {
overrideCache(SOME_DATA_URL, this.props.someData);
}
}
}
</pre>Now, when we client-navigate to this page, it will check the cache first before fetching the data from the server. Alternatively, on a full page reload, it will always fetch the latest data from the server and reset the TTL. <h2>Prefetching data</h2> With a caching layer like this in place, it is extremely straightforward to add prefetching. In the example above, all we need to do is call <code>getInitialProps</code> so the cache is populated with all the necessary data to load the page. Now, if this page is client-navigated to before the TTL expires, it will load instantly just like a static page! To achieve this, we can create a simple abstraction over Link, which not only downloads the page structure in the background but also calls its <code>getInitialProps</code> to populate the cache: <pre class="ql-syntax" spellcheck="false">import React from 'react';
import Link from 'next/link';
import { resolve, parse } from 'url';
import Router from 'next/router';
export default class DataPrefetchLink extends Link {
async prefetch() {
if (typeof window === 'undefined') {
return;
}
const { pathname } = window.location;
const href = resolve(pathname, this.props.href);
const { query } = parse(this.props.href, true);
const Component = await Router.prefetch(href);
if (this.props.withData &amp;&amp; Component) {
const ctx = {pathname: href, query, isVirtualCall: true};
await Component.getInitialProps(ctx);
}
}
}
</pre>Using this in our pages is as simple as: <code>&lt;Link prefetch withData href="…"&gt;</code>. In case you have some calls in your page’s <code>getInitialProps</code> that do not use a cache, you can use the <code>isVirtualCall</code> flag in the context to avoid making them when the method is called for caching purposes only. For example: <pre class="ql-syntax" spellcheck="false">import React from 'react';
import cachedFetch, { overrideCache } from 'lib/cached-json-fetch';
import fetch from 'isomorphic-fetch';
const SOME_DATA_URL = '/some_data';
export default class SomePage extends React.Component {
static async getInitialProps(ctx) {
const someData = await cachedFetch(SOME_DATA_URL);
const isServerRendered = !!ctx.req;
const isVirtualCall = ctx.isVirtualCall;
// No need to call this when prefetching the page,
// since this data won’t be cached
let someNonCachedData;
if (!isVirtualCall) {
someNonCachedData = await fetch('/some_non_cached_data')
.then(response =&gt; response.json());
}
return { someData, someNonCachedData, isServerRendered };
}
componentDidMount() {
// When the page is server-rendered,
// we override the value in the client cache
if (this.props.isServerRendered) {
overrideCache(SOME_DATA_URL, this.props.someData);
}
}
}
</pre>The key element that makes this possible is that the programmatic prefetch API through <code>Router.prefetch</code> returns the page’s constructor, so we can call the <code>getInitialProps</code> directly on that object! You can find this extended Link component in npm as <a href="https://www.npmjs.com/package/data-prefetch-link" rel="noopener noreferrer" target="_blank">data-prefetch-link</a>. In conclusion, we learned that with some fine-tuning we can make our dynamic next.js pages load as fast as static pages, without sacrificing the freshness of the data or the flexibility to choose which data we want to cache. We hope this helps you make your web platforms faster!

Increase the Performance of Dynamic Next.JS Websites - Scale

Increasing the Performance of Dynamic Next.JS Websites

We recently worked with the team at Cloudinary to help build and evaluate a better image compression and quality measurements. The results of Cloudinary’s work on Scale were very insightful and we wanted to share it broadly to demonstrate how more companies can leverage human judgments to build high quality features. Without further ado, <a href="https://cloudinary.com/blog/author/tal_lev_ami" rel="noopener noreferrer" target="_blank">Tal Lev Ami</a>, CTO of Cloudinary, wanted to write about how they used Scale: <blockquote> Here at Cloudinary, we provide a cloud-based tool that enables our users to</blockquote><blockquote> compress images and video for their websites and apps. Our goal is to</blockquote><blockquote> preserve the visual integrity of the content, but deliver the smallest file</blockquote><blockquote> size to any device or browser to ultimately optimize website performance and</blockquote><blockquote> end user satisfaction.</blockquote> <blockquote> One of the hallmarks of the Cloudinary solution is the ability to automate</blockquote><blockquote> many functions of image compression, so that developers don’t have to spend</blockquote><blockquote> time tweaking each photo and making multiple copies of different sizes and</blockquote><blockquote> resolutions to fit every possible scenario. Compression algorithms can be</blockquote><blockquote> tricky because they’re trying to make changes that have the smallest visual</blockquote><blockquote> impact, but different images can react differently to compression.</blockquote> <blockquote> As we were developing the algorithm for our “q_auto” capabilities — which</blockquote><blockquote> strikes a balance between visual quality and file size — we needed to test</blockquote><blockquote> it to understand how the resulting images compared to the human eye. Enter</blockquote><blockquote> Scale.</blockquote> <blockquote> Many image compression formats — like JPEG 2000 and JPEG XR — have been</blockquote><blockquote> tweaked to score well on particular metrics, such as peak signal-to-noise</blockquote><blockquote> ratio (PSNR). But these don’t always correlate with human perception on</blockquote><blockquote> image quality.</blockquote> <blockquote> We leveraged Scale to compare pairs of images and give us perspective on</blockquote><blockquote> which image was liked most by humans. With Scale, we did a variety of tests,</blockquote><blockquote> comparing several formats, including WebP, JPEG 2000, JPEG XR (lossy) Lepton</blockquote><blockquote> (MozJPEG, recompressed with Lepton), FLIF, BPG, Daala, and PNG8</blockquote><blockquote> (pngquant+optipng). We also were able to get input on the difference between</blockquote><blockquote> the uncompressed original image vs. a compressed version.</blockquote> <blockquote> Scale enabled us to create A/B comparisons that were viewed by human</blockquote><blockquote> observers. We submitted over 4,000 image comparisons to Scale, sending at</blockquote><blockquote> least four independent Scale requests for each pair of image. This resulted</blockquote><blockquote> in at least eight actual human comparisons for each pair of images. The</blockquote><blockquote> outcome of these comparisons were evaluated beside other perceptual metrics</blockquote><blockquote> such as PSNR, Google’s Butteraugli, DSSIM (Structural (Dis)Similarity) and a</blockquote><blockquote> new metric Cloudinary developed called</blockquote><blockquote> <a href="https://cloudinary.com/blog/detecting_the_psychovisual_impact_of_compression_related_artifacts_using_ssimulacra" rel="noopener noreferrer" target="_blank">SSIMULACRA (Structural SIMilarity Unveiling Local And Compression Related</a></blockquote><blockquote><a href="https://cloudinary.com/blog/detecting_the_psychovisual_impact_of_compression_related_artifacts_using_ssimulacra" rel="noopener noreferrer" target="_blank"> Artifacts)</a>.</blockquote> <blockquote> The results showed that overall, PSNR is “correct” in only 67 percent of the</blockquote><blockquote> cases. Butteraugli gets it right in 80 percent of the cases, and DSSIM in 82</blockquote><blockquote> percent of the cases. Our new metric, SSIMULACRA, agrees with human</blockquote><blockquote> judgments in 87 percent of the cases. Looking just at the high-confidence</blockquote><blockquote> human judgments, we found about 78 percent agreement for PSNR, 91 percent</blockquote><blockquote> for both Butteraugli and DSSIM, and almost 98 percent agreement for</blockquote><blockquote> SSIMULACRA. You can</blockquote><blockquote> <a href="https://cloudinary.com/blog/detecting_the_psychovisual_impact_of_compression_related_artifacts_using_ssimulacra" rel="noopener noreferrer" target="_blank">read more about SSIMULACRA and these results on the Cloudinary blog</a>. Or if you want to give it a try:</blockquote><blockquote> <a href="https://github.com/cloudinary/ssimulacra" rel="noopener noreferrer" target="_blank">SSIMULACRA is free and open-source software!</a></blockquote> <blockquote> The results of Scale comparisons gave us useful data points to validate our</blockquote><blockquote> metrics and provided more insights into the compression benchmarks we are</blockquote><blockquote> running and the comparison of various image formats. And from these insights</blockquote><blockquote> we were able to improve our visual perception metrics and fine-tune our</blockquote><blockquote> “q_auto” functionality so we know how aggressively we can compress images.</blockquote> <blockquote> Through this process we were impressed not only by the useful data points</blockquote><blockquote> derived from the Scale, but also the great support we got from the company</blockquote><blockquote> and the product’s ease-of-use, all which came at a reasonable price.</blockquote>

We recently worked with the team at Cloudinary to help build and evaluate a better image compression and quality measurements.

Image Compression: Cloudinary Relies on Scale Technology

Guest Post: Cloudinary Uses Scale to Focus on Image Compression

We are excited to&nbsp;<a href="https://www.wsj.com/articles/scale-helps-engineers-outsource-microtasks-1495452603" rel="noopener noreferrer" target="_blank">announce</a>&nbsp;that we have raised a $4.5 million Series A round of funding led by Accel. Along with this funding, Accel’s Daniel Levine has joined Scale’s board. The funding will be used to invest in our rapid growth, expand our offerings, and grow our team. When we started Scale ten months ago, our goal was to build the simplest API for training data. Since then we have released six APIs (<a href="https://scale.com/image-annotation" rel="noopener noreferrer" target="_blank">Image Annotation</a>,&nbsp;<a href="https://scale.com/ocr-transcription" rel="noopener noreferrer" target="_blank">OCR Transcription</a>,&nbsp;<a href="https://scale.com/categorization" rel="noopener noreferrer" target="_blank">Categorization</a>,&nbsp;<a href="https://scale.com/comparison" rel="noopener noreferrer" target="_blank">Comparison</a>, and&nbsp;<a href="https://scale.com/data-collection" rel="noopener noreferrer" target="_blank">Data Collection</a>), and we process millions of API requests each month. This success has confirmed our conviction that there is no better time than now to build Scale. We are currently at an inflection point in the age of artificial intelligence (AI). Just like software before it, AI will have an enormous effect on productivity and enable incredible new technologies. Scale works with many companies and technologies at the forefront of this shift, including self-driving cars, intelligent assistants, and voice analytics. Our customers agree that integrating AI with accurate human intelligence is crucial to building reliable AI technology. As a result, we believe Scale will be a foundation for the next wave of development in AI. This financing will allow us to grow our engineering, operations, support, marketing, and sales teams to continue on our journey to make our customers more productive and efficient. We’re committed to continuously improving our costs, speed, and quality and to providing a quality customer experience. We’d also like to thank our colleagues, customers, partners, and investors for supporting on this journey to build APIs for human intelligence. We look forward to continuing to support companies and developers in creating innovative software in the age of AI. This is just the beginning. We are actively hiring exceptional engineers, designers, AI researchers, and more. If you are interested in joining Scale, check out our&nbsp;<a href="https://scale.com/about#jobs" rel="noopener noreferrer" target="_blank">careers page</a>.

Announcing Our Series A With Accel

URL redirections for Website. Disclaimer: takes 5~ min for each added or edited.

url-redirects

Use relative path. use `/blog/foo` not `https://scale.com/blog/foo`

blog-filter-type

homepage-quotes

logos

homepage-content

quotes

blog-category-model

announcement-banner

homepage-resources

custom-fonts

homepage-event

author

Event model for new marketing events page

event

Optional second image intended to show on the grid view, should be a more squared image closer to 1:1 aspect ratio than the one used in the Cover field.

guides

announcement-news

homepage-research

legal

slugs must be all lowercase. ie: ‘my-slug’

eval

datastream

summit-2023-speakers

customers

Display order on the page /customers. Higher values show first

summit-2023-agenda

blog-model

Option to list or not list this post in the Blog index. Enabled = not list in the index, Disabled = list in the index

summit-2023-partners

enterprise-use-cases-categories

enterprise-use-cases-categories-items

partners

data-samples

enterprise-use-cases

Use only one image or video option. Add another item to add another source.

collapsible-sections

research

basecamp-speakers

Example: https://www.twitter.com/username

Example: https://www.linekin.com/in/username

leaderboards

Text for a short description like: Deprecated (as of January 2025)

Keyword for search. Add multiple keyword with comma. Example: hle, hleto, vista, enigma

Blog

How We Made It Easy for Anyone at Our Company to Trigger Alerts

Announcing Our Series B with Index Ventures and Accel

Adding a Dimension: Annotating 3d Objects with 2d Data

How to make MongoDB not suck for analytics

Scale Engineering Team++

Introducing Teams

Introducing Our New Dashboard

Increasing the Performance of Dynamic Next.JS Websites

Guest Post: Cloudinary Uses Scale to Focus on Image Compression

Announcing Our Series A With Accel

The future of your industry starts here

Blog

How We Made It Easy for Anyone at Our Company to Trigger Alerts

Announcing Our Series B with Index Ventures and Accel

Adding a Dimension: Annotating 3d Objects with 2d Data

How to make MongoDB not suck for analytics

Scale Engineering Team++

Introducing Teams

Introducing Our New Dashboard

Increasing the Performance of Dynamic Next.JS Websites

Guest Post: Cloudinary Uses Scale to Focus on Image Compression

Announcing Our Series A With Accel

The future of your industry starts here