BlogCategoryTag

Header

Footer

PreFooter

BlogPage

Trusted by world class companies, Scale delivers high quality training data for AI applications such as self-driving cars, mapping, AR/VR, robotics, and more.

AI & ML Blog | Scale AI

PaginationButton

AnnouncementContextProvider

UnnamedGlobalGroupOfVariants2

Screen

Segment

Experiment

Experiment2

scale.com (Production)

plasmic-nav

plasmic-embed-css

plasmic-basic-components

plasmic-tabs

plasmic-cms

react-chartjs-2

react-slick

react-quill

Product

Company

General

People

Engineering

Willow leads the Customer Operations team at Scale. She is a former director of logistics at Amazon.com, a graduate of MIT’s Leaders for Global Operations program, and a former captain in the U.S. Army’s Military Police Corps. I am a transgender pioneer within the military and corporate world, with a career spanning 12 years in the U.S. Army, MIT, Amazon, and Scale. Here are some of the things I learned along the way. Why I decided to change my gender I had decided to change my gender while in a concrete shelter from rocket attacks right before returning from deployment in Afghanistan. Rockets in this context were not particularly threatening, but sitting in a concrete shelter at night waiting for the sirens to go off can naturally lead to introspection. I had no intention of being a pioneer. It was a simple search for my own personal happiness. I felt like any other choice would be the wrong one. It was too much of a big and scary decision to think at all about how my career would shape out. There were deliberate risks I took and risks that I didn’t expect and needed help with. Back in the states, I was the first officer in my military unit to be out as bisexual after the repeal of “Don’t Ask, Don’t Tell” (DADT). This was a deliberate risk: my personal and professional lives didn’t intersect, but I thought that someone should be out as an example if other soldiers wanted to come out. I still felt like I had a vulnerable secret, as being transgender was still prohibited, and my plans to transition after departure were carefully guarded even as I changed my legal name and started living as female while on leave. My commanding officer was overtly hostile towards the repeal of DADT. After one staff call in which he used “faggoty” as a term of derision, I grew concerned about whether or not I would be able to leave the Army on my own terms. I need not have been concerned, however. Senior members of the staff met with the commander over his comments, and he decided to leave the Army over the pace of change. Like me, but for different reasons, he was voting with his feet. Returning to graduate school, I was the first openly transgender person in my program. I was one of two transgender students to tell my story to the school’s leaders in an ultimately successful bid to change insurance coverage policies for all students. Not all was respect and smooth sailing; in one emotionally crushing case, two teachers complained to my program admin about me using the restrooms that matched my gender identity (it seems strange to type this - much has changed in eight years). My friends calmed me down, and school administrators used the complaint as a teaching moment for the faculty. The only [other] in the room? As my civilian career advanced at Amazon and at Scale, I became more aware of how being transgender would shape my approach to work. No matter the realities of my professional role, I always feel like a bit of an outsider. It has been argued by folks more articulate and demonstrated in studies that diverse perspectives lead to better decisions, and clearer identification of risks. This is a good thing. The downside to feeling like an outsider is always feeling the need to push myself harder, to prove myself better, and to hold myself to an extremely high internal standard - and to always fear slipping. This approach to work has helped me advance quickly, and made me bold, but it is exhausting at times and ultimately rooted in fear. One of the factors in my recent decision to join Scale was that many company leaders identify with the LGBTQ community. I want to help build the future without worrying so much about how I fit in it. Taking a role at a smaller, fast-moving startup with less formal structure has been awesome. There’s a sense of purpose and mission and a rapid flow of ideas that I feel I can really be part of. Transgender people have always existed in one way or another, before society’s own peculiarities and social constructs started calling it a “new thing.” I wrote earlier that I was the first openly transgender applicant at my grad school program. It is highly likely that there were transgender people there before, and it is likely that some of my classmates were transgender and simply hadn’t decided to transition yet. It is important to remember that an individual story cannot represent a community, especially with big variation in opportunity and privilege shaping peoples’ lives. Advice for folks feeling like outsiders There are a few things that I wish I’d known earlier in my career. One piece of advice I got from a friend has stuck with me: “You will always think about yourself a lot more than others think about you.” That feeling of awkwardness, that one thing you did - most people simply won’t notice or remember. Our memories are inherently selective. Learn from your mistakes but choose to remember the good parts. The other thing I wish I’d understood better earlier is to not be afraid to pick a mission with purpose. Early on, I picked assignments that seemed like the best value for the company’s ability to save money or expand programs and to advance my career. I left jobs and teams that I enjoyed to seek out the external trappings of success. Many people do this, but the feeling of validation seemed so necessary when I was uncertain of my place in society. Finding a team and a voice that makes me feel at home and a mission that serves a purpose I believe in has ultimately been more gratifying. Now, what the people around me think matters more than what the world thinks. Scale believes there is tremendous potential for AI technology to drive a revolution in true efficiency and productivity, across industries - this is a change for which I want a front row seat. I am glad I am able to take that seat feeling like I am no longer an outsider.

Scale Operations leader Willow Primack shares her personal experience for International Transgender Day of Visibility

Willow Primack shares her experience for International Transgender Day of Visibility

Trans Day of Visibility: My Experience at Scale

<h1>Project Ideas and Inspiration</h1> Scale recently sponsored Treehacks, Stanford University’s annual hackathon. During this fully virtual hackathon, nearly 1300 active hackers and 114 teams submitted projects for judging. Out of all teams, 21 submitted projects using Scale’s API. The winning project for Best ML Hack was <a href="https://devpost.com/software/fine-tuned-heads" rel="noopener noreferrer" target="_blank">Fine Tuned Heads</a>, which proposed a new workflow to assist with the implementation of Computer Vision. Rather than training a single model from scratch, the encoding and fine-tuning aspects of a task were separated. The Fine Tuned Heads team created an automatic data collection pipeline using Flask to automatically execute scripts to grab .m3u8 URLs from Youtube, then used Scale Rapid to automate uploading and generating batches of ground truth labels for these images. According to the Fine Tuned Heads team, “this approach can be used to improve the efficiency of inference in ML tasks, and by engineering our self-supervised model with an efficient architecture and deploying on efficient hardware, we can reduce the carbon footprint by over 100x.'' An additional winner of the API Challenge was <a href="https://devpost.com/software/accecity" rel="noopener noreferrer" target="_blank">AcceCity</a>, which provides an ML-powered mapping platform that enables cities, urban planners, neighborhood associations, disability activists, and more to identify key areas to prioritize investment in. For their project, the AcceCity team first used the Google Maps API to send images of street views to their backend. They looked for handicapped parking, sidewalks, disability ramps, and crosswalks and used computer vision, by custom-fitting a zero shot learning model called CLIP from OpenAI, to automatically detect those objects from the images. They then tested the model using labeled data from the Scale Rapid API. Other interesting projects include using Scale Rapid to label map data to promote sustainable living, and classify patient medical image data. Beyond the examples from the Hackathon, one can use Scale Rapid for a variety of ML projects, including to <a href="https://scale.com/blog/rapid-cyclegan" rel="noopener noreferrer" target="_blank">anime-fy images</a> and to train a <a href="https://scale.com/blog/imdb-rapid" rel="noopener noreferrer" target="_blank">sentiment analysis model for movie reviews</a>. Use our <a href="https://scale.com/open-datasets" rel="noopener noreferrer" target="_blank">open datasets</a> to draw further inspiration for your projects.<h1> </h1><h1> How can you use Scale to build a Successful ML Hackathon Project?</h1> One of the most challenging problems preventing teams from creating novel ML projects in a short timespan is the “Cold Start” problem, which is when an ML model cannot draw inferences for users or items about which it has not yet gathered sufficient information. Especially during the compressed timespan of a hackathon, obtaining high-quality training data for models is nearly impossible. Scale Rapid is the fastest way to obtain production-level quality labels, with no data minimums. Using Scale Rapid, your hackathon team can experiment quickly by:<ol><li> </li><li> Setting up projects in minutes and receiving initial data within hours</li><li> </li><li> </li><li> Iterating over edge cases and instructions by getting real-time feedback</li><li> </li><li>Scale to production-level pipelines with precision quality control</li><li> </li></ol> To get started, sign up for an account with your university email on our <a href="https://scale.com/rapid/university" rel="noopener noreferrer" target="_blank">University Program Page</a> to receive $250 in Rapid Credits, or check out our <a href="https://colab.research.google.com/drive/1XZjbVtA2oGOdMp0-v0E80ujMB8tbeh7x?usp=sharing" rel="noopener noreferrer" target="_blank">quick start python notebook</a>.Rapid currently supports the following annotation types: <ul><li>General Image Annotation</li><li>2D Semantic Segmentation</li><li>Text Collection/Categorization</li><li>Document Transcription</li><li>Named Entity Recognition</li><li>Video Playback Annotation</li><li>Lidar Annotation</li><li> </li></ul><img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/2ac2bcbb-1e7d-4268-2c77-bf6d2623c400/public" alt="Annotation Types Supported in Scale Rapid"> If you have your own unlabeled data, add it to your project using local file or csv upload, import it from previous projects, or upload it from the cloud (S3, GCP, and Azure). Otherwise, check out our <a href="https://scale.com/open-datasets" rel="noopener noreferrer" target="_blank">open datasets</a>. Once you receive task responses, visualize your data in Scale’s dataset management platform, Scale Nucleus, by navigating to the data tab under your project and clicking “Visualize in Nucleus”. Nucleus helps you understand the strengths and weaknesses in your dataset, analyze the long tail of your dataset, collaboratively edit labels, and explore metrics, failure cases, and confusion matrices linked to your training data. <img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/899fda4e-d4ab-4ed4-7ef4-5fa12e396200/public" alt="Scale Nucleus <> Scale Rapid Integration"> Quickly find similar images using the “Autotag” feature in Nucleus, and further refine your set of images by providing a few examples of positive and negative examples. After identifying this subset of your dataset, you can send it back for labeling through Scale Rapid. Scale AI’s products help enable and strengthen a team’s ML development cycle immensely by delivering high quality data for any project. As you or your team continues to maintain and improve your project post-hackathon, use Scale AI for consistent, reliable data in order to create, validate, and maintain production for your high-performing ML models.

Scale AI recently sponsored Treehacks, Stanford University’s annual hackathon. Explore how to use Scale Rapid for a successful ML hackathon.

Using Scale Rapid for Hackathon Projects

How to Use Scale for Your ML Hackathon Project

<h1>Background</h1> At Scale AI, we’re always looking for ways to apply machine learning (ML) in our internal systems to deliver the best results for our customers. We’ve previously written about Scale’s <a href="https://scale.com/blog/how-we-scale-machine-learning" rel="noopener noreferrer" target="_blank">ML-powered pre-labeling</a> as well as our <a href="https://scale.com/blog/autosegment" rel="noopener noreferrer" target="_blank">active tooling, like Autosegment</a>, which helps our Taskers label faster and at higher quality. In this piece, we discuss how we think about and incorporate some basic tenets of design thinking to develop ML systems that make a difference for our labeling pipeline. <a href="http://web.mit.edu/jrankin/www/engin_as_lib_art/Design_thinking.pdf" rel="noopener noreferrer" target="_blank">Design thinking</a> is a mindset and set of practices which foster the collaboration required to solve problems in human-centered ways and was notably popularized by organizations such as IDEO. It’s useful for machine learning engineers who need to understand the full scope of how the system they are building will be used and are responsible for both business and operational outcomes. We’ll describe how design thinking techniques can be applied to scoping machine learning problems and introduce some concepts to help business stakeholders communicate more effectively with ML engineers.<img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/8a41ed7a-4287-491b-6202-1f80a4c29a00/public" alt="Phases of Developing an ML System">Idealized phases of developing an ML system. In practice, it’s not this linear.In our experience, the different phases of developing an ML system are: <ol><li>Discovery - Locate a business or operational pain point</li><li> </li><li> Design - Map the different ways a new or existing model could fit into the</li><li> extant pipeline</li><li> </li><li> </li><li> Decision &amp; Alignment - Align with operational and engineering</li><li> stakeholders on scope and resourcing</li><li> </li><li> </li><li> Development, Integration, Deployment - Deliver the system. Build data</li><li> pipelines, train models, integrate into the pipeline, etc.</li><li> </li><li> </li><li> Measurement &amp; Experimentation - Identify the effect of the ML system as</li><li> an intervention on the existing pipeline.</li><li> </li><li> </li><li> Continuous improvement - Monitor and retrain to account for distribution</li><li> shifts and new edge cases</li><li> </li><li> </li></ol> This is a descriptive mental model, not a prescriptive blueprint. In reality, a project is always moving between different phases and can often be in multiple phases at the same time. For example, measurement and development form a tight iteration loop. But, it’s still useful to think about these phases in isolation. Concepts from design thinking are mostly employed in the Discovery and Design phases when the problem statements are the most inchoate and ill-formed. In these phases, the need or specific outcome is communicated to an ML engineer and an initial strategy to address that need or deliver that outcome is designed. We focus on the Discovery and Design phases in this blog and cover the remaining phases briefly.<h1>Phase 1: Discovery</h1> An ML system gains life when a need is discovered. By need, we distinguish between a request for a specific feature, such as a linter (a tool we use to ensure data quality), and the underlying operational or business pain point or problem - such as a new labelling capability requiring ML support. It’s not uncommon for operational or business teams to stumble into the <a href="https://en.wikipedia.org/wiki/XY_problem" rel="noopener noreferrer" target="_blank">XY problem</a> by pattern matching and requesting a clone of an existing feature, without realizing that their underlying problem is actually quite different. Or, they might ask for something which is incredibly difficult, unaware that a slightly modified version would save literally months of development time. So an ML engineer should always hunt down the true problem that needs to be solved. Pain points and problems can be project-specific or exist horizontally across business teams or units. Historically, many of the problems that the ML team at Scale have tackled are project-specific and brought to us by our cross-functional stakeholders. For example, a project manager running into issues with a labeling project or a product manager looking to launch a new feature will surface these project-specific issues to our team. Tackling these isolated problems can result in marooned solutions which aren’t readily reusable. Solving horizontal issues, in comparison, maximizes our leverage, since the ML systems we build will then scale with our business. However, cross-business issues that can be solved with ML are often harder to identify because of the sheer operational complexity. These issues require a more active discovery process and take more time and deeper alignment with partners. The problems with the highest stakes are not obvious to ML engineers, who don’t get day to day exposure to them, even as these frictions impose significant pain on the operations or business side. To get a sense for this, imagine explaining the pain of managing multiple python environments and package versions to an operational lead - it can be incredibly painful for you, yet esoteric, even cryptic, to them. As another example, while working on an image semantic segmentation project, we learned that pre-labels generated by a model specific to that project were sometimes ignored by taskers, but the root causes were unclear. A close investigation identified a number of culprits, ranging from the inflexibility of pre-labels to having optimized for the wrong metric during development. <a href="https://scale.com/blog/autosegment" rel="noopener noreferrer" target="_blank">Autosegment</a>, developed by Sean Li, Saleh Hamadeh, and the two of us, aims to tackle these problems. Autosegment solves a challenge for all image segmentation projects and isn’t restricted to a particular project. Learning about this challenge and building the appropriate solution required input from numerous stakeholders - from the project leads, who wanted to accelerate taskers; the taskers themselves, who used our tools and knew what worked and what didn’t; to engineers, who were familiar with the implementation of image segmentation in our interface and understood the kinds of tradeoffs that different tools would be need to make.<img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/7866cd69-bd81-40f7-4e44-6094a84f2c00/public" alt="Stakeholder concerns">Example overlap of concerns between different stakeholders for a new ML-assisted tool. Identifying the pain points of the end user always requires talking to the end user. Instead of theorizing what could be a problem for them, it’s always more effective to speak to users directly and observe them as they encounter the problem. When building systems to accelerate labeling operations, ML engineers need to build user empathy. A simple way to build user empathy is to spend time tasking and dogfooding our own platform. There is no substitute for first hand experience for understanding what makes tasking hard.<h1>Phase II: Design</h1> After Discovery, the project enters a design phase where an initial approach is sketched out - what kind of model to build, what sort of data is needed, the timelines, and so on. Designing effective ML systems at Scale requires two things: a clear need, which we’ve discussed above, and knowing the many different ways a model can fit into the labelling pipeline. This last point is worth further elaboration. The same machine learning model can be used for a variety of operational purposes. For example, a model that predicts bounding boxes for cars can be used to:<ol><li>Estimate the number of boxes in a task (task difficulty estimation)</li><li>Prepopulate the scene with bounding boxes (pre-labelling)</li><li> </li><li> Suggest fixes to bounding boxes made in real-time by a tasker (active</li><li> tooling)</li><li> </li><li> </li><li> Identify missing boxes or unexpected boxes in a completed scene (linting)</li><li> For each of these, the inference API could have the exact same request and</li><li> return formats. However, each of the four approaches above require different</li><li> operational adjustments, not all of which may be feasible. Task difficulty</li><li> estimation makes assumptions about what makes a task difficult or easy for</li><li> humans and implicitly maps these to legible subpopulations of taskers.</li><li> Active tooling can require nontrivial operational investment in teaching</li><li> taskers how to use the new tool and engineering lift to build new frontend</li><li> components. Building a useful ML system means designing how an ML model will</li><li> interface with the rest of the operational and engineering systems, in</li><li> conjunction with actually training the ML model.</li><li> </li><li> </li></ol><img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/8e6d0b0b-5185-4a7c-227e-03350b1bdd00/public" alt="One model, many uses">The same model may be applied in different ways to an existing pipeline, requiring different accommodations by engineering or operational teams. We’ve noticed two particular traps for ML engineers which can lead them on wild goose chases during the design phase. The first is misalignment between the machine learning problem and the business problem. The second is confusing the customer and the end user of a system. ML engineers are trained to use models to solve machine learning tasks, like box detection or text classification. For a box detection dataset, you’d use a box detection model. For a text classification dataset, a text classification model. When the business problem looks like an image classification problem, you might reach for an image classification model, only to discover that what you really need to solve is tabular regression. Faced with using ML to assist, say, labelling semantic segmentation training data, a common reflex is to train a semantic segmentation model to do the labelling. But if you already had a capable model, you wouldn’t need to collect training data for it! The mistake here is to apply the familiar task paradigm - semantic segmentation - to solve the business problem - accelerating semantic segmentation. Many operational questions take the form “Is something X or not X?” To an ML engineer, this sounds like a classification problem, and it’s easy to reach for a classification model which directly answers the question. Putting aside whether modeling X directly is feasible, it’s often the case that we already have an operational or engineering system which could answer this question more effectively with some additional information. In this situation, it may be faster to figure out how to extract the augmenting information rather than replicate the existing system altogether; a strong reason to apply a modeling scheme that looks very different from the modeling question.<img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/3505a43e-5c45-4274-cdd3-2e27242e0600/public" alt="Leveraging existing systems">Modeling the business problem to solve doesn’t always mean reaching for the most obvious machine learning task paradigm. The second trap is mistaking the end user of a system with the people who request or manage the system. When we build a tool like <a href="https://scale.com/blog/autosegment" rel="noopener noreferrer" target="_blank">Autosegment</a>, even though the end user of the active tooling is the Tasker, the internal customer is actually the Delivery Operations team. What we build has to meet the customer’s expectations, so even if Taskers enjoy using our tool, we need to make sure to move the needle for the Delivery Operations team. But the expectations and needs of the end user may differ or conflict from the internal customer (e.g. latency tradeoff with cost). In order to avoid falling into these two traps, ML engineers have to be deliberate in how they transport the business problem into a machine learning context; and always keep in mind the different stakeholders.<h1>Phases III - VI: Decision and Alignment Through Continuous Improvement</h1> After the initial Discovery and Design phases, the project needs a go / no-go decision by both the ML team as well as cross-functional teams. In this phase, all teams must decide whether to allocate the necessary resources to the project. If the ML team decides to dedicate resources but the cross-functional team cannot commit the time or spare the bandwidth, the effort will ultimately fall short. Some factors teams must consider when making this decision include: prioritization, level of effort needed, and available bandwidth. Once all teams commit to the project, we can develop the ML-specific components of the ML system. This is when we build data pipelines, train models, and ship an endpoint. Once the model is developed, the project nearly always needs engineering and operational integration. For example, if we are developing a new pre-labeling model, the model requires integration with our labeling pipelines to attach the pre-label to the task. Now the project is actually deployed. Deployment should also follow best practices: meaning the feature should only be enabled for a small percent of tasks or users at the beginning, making it easier to revert if there are any issues. The system should then be constantly measured. Especially in the first few weeks, close attention should be paid to ensure no issues arise. We then use the metrics and qualitative feedback gathered in the measurement step to continuously improve the model. Models often need to be retrained to account for new edge cases. As we maintain the production model, we often encounter data distribution shifts or taxonomic changes which must be addressed.<h1>Conclusion:</h1> We hope this post provided some insight on how we design our ML systems and how you and your teams can effectively partner with horizontal teams to deploy models that serve your business needs. Throughout the phases of ML development, we strive to ensure alignment with our cross-functional stakeholders. While gaining and maintaining alignment takes time and deliberate effort, it allows us to unlock capabilities on our platform that wouldn’t otherwise be possible.

Understanding operational and business context is critical to shipping ML systems that work. 

How we leverage concepts from design thinking to design ML systems in our internal processes to deliver results for customers.

Design Thinking for ML Systems at Scale

It’s a story that you’ve heard before: the biggest and best machine learning models need enormous datasets to get good performance on complicated tasks. When you are just getting started with deep learning, the sheer volume of data that you hear the state-of-the-art models need can be pretty discouraging. However, today I’m going to show you how to stand on the shoulders of giants to supercharge your next ML project on a tough task like object detection. Do you want quality results without having to train on the same mass of data? Enter: transfer learning.(For those of you who are already familiar with transfer learning, feel free to skip ahead to the problem setup and results)“Transfer learning” refers to when we take a model that was trained on a related task and retrain it on a new task of interest. We can then piggyback off of the hundreds/thousands of computing hours spent training the existing model on some other task and train on a much smaller dataset and just fine-tune the model weights to be accurate for our use case.<img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/b82a3a9f-ac11-45c4-af64-005e9115d900/public" alt="Transfer Learning Diagram">We effectively leverage all the expertise and hard work leading companies and research groups have already invested to create a good model then take out the last layer and swap it to match our application. Intuitively, since the model was trained on a related task, the weights and internal representations that it already learned should be useful for the new use case. All the normal model training tips and tricks apply, except for the fact that we initialize the model with the pre-trained weights. Transfer learning is a really important technique for making many production-ready ML systems work well.With the explanation out of the way, let’s see it in action!<h2>The Scenario</h2>Let’s say that I’m an ML-savvy rancher trying to improve the automated monitoring of my horses with a state-of-the-art object detection system (a setting I’m sure you’ve all encountered before). Some initial research suggests that there aren’t any models out there that are pre-trained on horse detection. Unlucky! Good thing we have Rapid and transfer learning.<h2>Data</h2>For my data, I found an open source collection of animal images. With Rapid, I uploaded the images and instructed taskers to draw a tight bounding box around any horses present in the image. I then submitted a small calibration batch to make sure the labelers understood the task at hand and to spot check my own data. Once that was done, I was galloping along! I had 147 images labeled with bounding boxes around the horses back to me within 2 hours, all for around $20.<img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/d68acc6a-0a3c-499e-7f89-b29128955900/public" alt="Images of horses found in Scale Nucleus"><h2>Model</h2>My pre-trained model that I’m transferring on is the <a href="https://arxiv.org/abs/1506.01497" rel="noopener noreferrer" target="_blank">Faster R-CNN</a> model with <a href="https://arxiv.org/abs/1512.03385" rel="noopener noreferrer" target="_blank">ResNet 50</a> backbone.<pre class="ql-syntax" spellcheck="false">def get_pretrained_model(): # load model pre-trained on COCO model =
 torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True)
 num_classes = 2 # 1 class (horse) + background input_features =
 model.roi_heads.box_predictor.cls_score.in_features # the key part: swap out
 the last layer for a predictor that matches our needs
 model.roi_heads.box_predictor = FastRCNNPredictor(input_features, num_classes)
 return model
</pre>This model is already trained on the famous <a href="https://cocodataset.org/#home" rel="noopener noreferrer" target="_blank">COCO (Common Objects in COntext) object detection dataset</a>.<h2>Training</h2>Now that I have a model and data, I just need to train. After slicing my data as I desire in Nucleus, it’s easy to keep my images and files bundled together neatly:<pre class="ql-syntax" spellcheck="false">import nucleus SLICE_ID = 'slc_c8c7418zpj7g0csycj5g' # a slice of my animal
 dataset client = nucleus.NucleusClient(API_KEY) data =
 client.get_slice(SLICE_ID) data_list = data.items_and_annotations()
</pre>With all the data horseplay out of the way, we just have to load our model and data into the processor and let it do its thing.<h2>Results</h2>I trained the model for 10 epochs, which took about 15 minutes on GPU (or around 4-5ish hours on CPU if you’re not pressed for time). We achieved an F1 score of .517 (higher is better; when Faster R-CNN was initially released, its F1 score on COCO was .507). Way to go transfer learning! Here are some of the examples from the evaluation set:<img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/9b2da0d8-160b-490a-7146-8104f29d6500/public" alt="Examples from the evaluation set ">Not bad, but we could do better. At this point I could spend more time tuning hyperparameters or training for more epochs. However, I’m too busy running the ranch to spend time doing hyperparameter sweeps or ensembling models.Normally a bottleneck for model improvement is a lack of quality <a href="https://scale.com/guides/data-labeling-annotation-guide" rel="noopener noreferrer" target="_blank">labeled data</a>. Thankfully with Rapid, there’s no such bottleneck. I still have unlabeled images in my dataset – let’s see what happens when I change nothing else but add more labeled data!<img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/a275634f-e792-4be1-469d-c9ab175fe800/public" alt="Additional images labeled in Rapid">The newly labeled data brings my total to 351 annotated images, which only cost me around $45 on Rapid. After 10 more epochs with the new data incorporated, the model improves our F1 score to .655.Much better, and I didn’t even need to do any hyperparameter tuning! Let’s compare the new and old model predictions:<img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/e29a9815-ef1f-45f6-f82d-260121577700/public" alt="Examples from the evaluation set">With more data, it seems that the model gets a little less confused with crowds of horses and captures more of the visible edges of the horse.<h2>Conclusion</h2>We’ve seen how to apply transfer learning to a custom use case. Now all I need to do is productionize and deploy my custom horse detection model, and I’ll be sure to take my ranching (is that a word?) to the next level. I hope you’ve been inspired to try out Rapid for your next transfer learning project! You can <a href="https://dashboard.scale.com/login?redirect_url=rapid" rel="noopener noreferrer" target="_blank">get started with Scale Rapid today here</a>. Want to get a jump start for your own experiments? Access the code from my experiment on <a href="https://github.com/scaleapi/dev-blog/tree/main/nucleus-rapid-blog/transfer_learning" rel="noopener noreferrer" target="_blank">Scale's Github</a>.

In our latest blog, we explore how to implement transfer learning using Scale Rapid.

How to Implement Transfer Learning for Object Detection with Scale Rapid

Easy Transfer Learning with Scale Rapid

As a member of the U.S. Air Force Rescue and #AFWERX, I have been working to modernize the tools used by Search and Rescue to increase the odds of survival of those lost at sea. Every year, hundreds of individuals find themselves isolated and at the mercy of the ocean, creating a dangerous scenario which is impossible to navigate without the best technological tools and deep expertise. Being in the ocean is one of the more extreme environments to be isolated. The conditions that make it hard to survive also make it difficult to locate and rescue the isolated; rescue is rarely a simple mission. Attempting to modernize the Search and Rescue model is arguably just as difficult. Systems are out of date: the “how” hasn’t changed in over 20 years. And changes that have been adopted, have not been implemented at the speed of relevancy. Five years ago, I began exploring different approaches to solving Search and Rescue’s toughest problem statements. During this discovery process, two significant experiences led me to become a Fellow at Scale. In my first experience, I was deployed as a United States Special Operations Command (SOCOM) Tech Advisor as a Personnel Recovery Subject Matter Expert (SME). I had the opportunity to share these problem statements with Josh Wolfe, co-founder of Lux Capital, and the rest of the SOCOM Tech Advisory Panel team. Josh shared his opinion on how the DoD could use a page out of the venture capital playbook: to move faster with sustainable speed in developing solutions. Seeing how Josh was investing in startups that invent the future inspired me. I began to engage with early-stage startups to find new solutions that would save lives. The second instance was at a Hacking for Defense session at Stanford University. This time, I was working with young college students learning the lean startup methodology to solve DoD problems. Steve Newell, the founder and CEO of BMNT, introduced the Search and Rescue problem statement to the class. He presented it in such an elegant way—I could physically see his inspirational words produce passion on the young data scientists’ faces. At this moment, I realized that AI developed to save lives would be the catalyst for large-scale DoD adoption. A few years later, and several AFVentures Small business Innovation Research contracts later, Forrest Underwood introduced me to Shift.org. This was the opportunity I couldn't pass up: the chance to work inside a startup for 8 weeks. Shift was able to recognize a critical flaw in my strategy to solve problems. How can you get the DoD to adopt industry AI solutions without possessing a well-developed ML pipeline? For this reason, I think Shift matched me with Scale AI. Scale AI is the best-in-class at providing the entire ML lifestyle to its customers so they can focus on the real, pressing problems at hand. This is exactly what's needed when a DoD customer asks for an AI solution. It is a misunderstanding of the requirements surrounding ML that can be difficult when attempting field AI solutions. Without a mature ML pipeline, it’s very difficult to have your model perform correctly and continue to improve as the extremely dynamic world evolves. On the first day on the job, I attended #TransformX, a virtual conference with the CEO Alex Wang, a community of leaders, visionaries, practitioners, and researchers across industries. This was probably one of the best onboarding or familiarization phases that I've experienced in 20 years. Spending 2 days listening in on thought leaders discussing the art of what's possible in a long forum was extremely insightful. Throughout the entire fellowship experience, I compared how Scale operates against Air Force Institutions of which I’ve been a part. One observation that stood out to me was the holistically flat organizational rank structure of Scale. After seeing the level of engagement that Alex Wang had during TransformX and the scope of his responsibilities, it was easy to draw parallels between Alex and a Geographic Combatant Commander. One welcoming attribute of Scale was the inclusion of ambitious ideas. It is understood that the company is at the edge of the AI frontier, and new ideas are required to move forward quickly. This is what has allowed Scale to grow at an incredible rate; it’s also the attribute that the Air Force is trying to re-install into its ranks. Inside Scale, I had the opportunity to work for the Federal division. The level of talent and prior experience of the team was truly awe-inspiring. I had a false assumption that I would be working with individuals without experience in the Armed Forces. I was surprised when I learned that there was a rich mix of individuals from both the DoD and Silicon Valley. At first, it felt like I walked into a Special Operations Group with Senior Instructor Pilots, Chief of Intel, and the Ground Forces Commander discussing an Operations Order. Except, everyone was in business casual and there were lead engineers in the conversation. For Scale to recognize the talent pool coming out of the DoD was exciting to see. The method of mixing operators, strategists, and engineers inside is not common practice for the Air Force. The standard Air Force approach is to silo these different functions on a day-to-day basis. Even my two cents were considered as the new guy. That says a lot about an organization. The art of creating what's possible is something that Scale was always discussing with potential customers. This wasn't a "fake it until you make it” model. The conversation was followed up with things that were deployed in that space to demonstrate their success. My gained understanding of what's possible is what I will take back to the Air Force. Thank you Scale.

c63a1be0661f1594644b740c08bede0e.jpeg

Bon Strout details his experience as a Scale Fellow

Scale Fellow Spotlight on Bon Strout

Shift Ventures Fellow Spotlight: Bon Strout

Scale board member William Hockey details why he believes AI/ML has the fundamental ability to restructure the way that systems and businesses are built.

Scale welcomes William Hockey as a board member

Why I Joined Scale

<h1>Update:</h1><h1> </h1>We have received an overwhelming response to our announcement of free, AI-ready datasets in support of Ukraine. Nearly a hundred AI companies, researchers, developers, and GIS practitioners have offered to support the initiative with Scale – an inspiring global community of AI Developers for good. Given the high demand, we will provide the datasets in phases starting with requestors with the greatest need, followed by a broader community. At this time, our focus is on providing Structure Damage Assessments in Kyiv, Kharkiv, and Dnipro. The path forward is now two-pronged – our top priority is the delivery of damage assessments to those with urgent need. The second prong of our effort is to provide AI-ready training datasets for Damage Assessment to the broader AI community, starting with a structure recognition training dataset. Today, we are sharing an initial dataset with the broader AI community: <a href="https://drive.google.com/uc?id=17lpMFngiqeegIm1U0RzUYfB-9GjmI7fk&amp;export=download" rel="noopener noreferrer" target="_blank">link to download dataset</a>. Note: the corresponding ZIP file's SHA256 Checksum is: 379c394dab6a4d7f1358625d086ebea4b39aaa124548e8c079bee16fb574bc53. We appreciate your patience as we assess and prioritize requests for data and data development. We look forward to continuing to support Ukraine alongside other AI Developers leveraging critically important open-source and commercially available data for this important work. Sincerely,Scale AI Team ——————————————————————————————————————————— At Scale, we believe in using AI technology in support of democratic values. That’s why starting today, Scale will be providing a series of AI-Ready datasets that algorithm developers can use to rapidly train and deploy AI in support of Ukrainian operations. By providing these datasets at no cost to national security practitioners, we hope to support a diplomatic solution and swift end to this conflict. Diplomacy is fueled by insights; foundational to that is quality, trusted data. We have recently created strategic partnerships with commercial satellite imagery providers in order to develop these datasets, which include Aircraft Detections, Structure Damage Assessments, Change Detection, and the JSON and GeoJSON annotations. We encourage AI Developers to contact detections@scale.com to receive this data free of charge. To better understand how these datasets can be used, Scale presents a contemporary use case of our partnerships with national security leaders. We deliver an AI-powered change detection solution capable of identifying and tracking military formations across the globe. The video below demonstrates continuous monitoring and improvements by Scale’s ML engineers and deployment strategists. We provide timely and accurate insights through our AI Data Readiness platform, our Test and Evaluation platform and our continuous ML Ops tooling. Leveraging Scale's AI/ML ops platform, we continuously develop and hone deep learning models to describe scenes and detect objects across a range of inputs from commercial satellite imagery providers. <iframe class="ql-video" frameborder="0" allowfullscreen="true" src="https://scaleai.wistia.com/embed/medias/hphfibwbn6"></iframe> We are now delivering insights to U.S. national security customers from this full-stack, multi-intelligence, human-teaming product, including on Russian formations across Europe and Eurasia. Here, on Russian air and ground order of battle (OOB) data, we successively identified OOB changes across the EUCOM area of responsibility (AOR) over the past several months. We used high-resolution synthetic aperture RADAR(SAR) imagery from Capella Space and high-resolution optical imagery to deliver high-accuracy, time-sensitive insights. “In these intricate and time-sensitive scenarios, it is critical that we deliver insights from multiple, disparate datasets at the speed of relevance,” said Mark Valentine, Head of Federal at Scale. “We are proud to demonstrate the power of our platform, and our approach to human-machine teaming to support national security leaders and practitioners.” Monitoring the Russian OOB on the border of Ukraine using commercial satellite imagery, and other non-traditional Intelligence, Surveillance and Reconnaissance (ISR) sources requires adaptation to a variety of challenges in a constantly-evolving field. The challenges in using AI to analyze OOB changes include: exigent scenarios presented by weather, disparate image varieties and quality, and the inclusion of relatively rare military objects such as transporter erector launchers (TELs), for which there are limited real-world observations. The depth of ontology can also present complications: the presence of multi-generational aircraft, ground vehicles, RADAR and TELs mean that models need to be trained against tens of thousands of observations in a range of environmental conditions. If you are an AI Developer who would like to receive this data free of charge, please contact detections@scale.com.

ae82457827ce45793eefc6460623a063.jpeg

Scale AI provides free datasets for national security partners

Scale AI providing free datasets to help national security partners gain current insights into Russia-Ukraine conflict

Scale Rapid is a quick and easy way to get clean, high-quality labeled data for your next ML project. We used it to train a <a href="https://scale.com/blog/imdb-rapid" rel="noopener noreferrer" target="_blank">sentiment analysis model</a> in a previous post - and we’re back now to show you how to use Scale Rapid to train an unsupervised model! In this blog, we are going to use <a href="https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix" rel="noopener noreferrer" target="_blank">CycleGAN</a> (<a href="https://arxiv.org/pdf/1703.10593.pdf" rel="noopener noreferrer" target="_blank">paper</a>) to anime-fy images with Scale Rapid. CycleGAN is trained in an unsupervised manner, meaning each X input does not need to correspond 1:1 to a y output in order to train the net. Instead of using Scale Rapid to label training data, we used it to clean data. We separated a group of images scraped from the internet into categories. This classification allows us to make the two inputs necessary for CycleGAN - a class A (real-life images) and class B (Japanese anime-style images). To start, we downloaded 10k images from Google images from a variety of different searches. At a glance, the corpus had all sorts of images. <img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/27b06dda-1661-4805-c721-0ec70ee20300/public" alt="Sample images from Google">Class A contains real-life images and Class B contains anime-style images. Image 3 belongs to Class A, and Image 2 to Class B. Images 1 and 4 do not belong in the dataset as they are neither real-life nor anime-style images. The ask for Scale Rapid is to discriminate between these three types of images: a real-world, realistically edited photograph, a Japanese-style anime scene, or neither. Rapid responses are then used as ground truth to establish which images should and shouldn’t be in the dataset we train CycleGAN on. Rapid’s categorization endpoint supports image categorization - and it's only 8 cents per image! Because CycleGAN is trained on unsupervised data, we don’t need to pair class A and class B images - we can simply separate them from one another and train. <h2>Using Scale Rapid</h2> Scale Rapid works in two phases- first, the user sets up the project &amp; instructions, and runs small calibration batches to ensure the initial results are as expected. Once quality on those looks good, larger production batches may be uploaded for labeling. For this application, we set up a categorization project. The taxonomy is just one multiple-choice question: What type of image is this? <ul><li> </li><li>Anime (japanese animation) image</li><li> </li><li> </li><li>Photorealistic (real-life) image</li><li> </li><li> </li><li>Neither</li><li> </li></ul> Scale Rapid also has a feature called long hints, which allows us to prompt labelers with specific things to think about upon choice selection via a tooltip. For the first choice (Anime (japanese animation) image) I clarified with the following long hint: "Japanese animation refers specifically to anime, as in the instructions. Other cartoon-style or generated images belong in the neither category." We begin by uploading a calibration batch, and follow up with production batches. The labeling results are easily parseable via JSON. <h2>Scale Rapid Results:</h2> A random audit of tasks found just one mistake out of 200 randomly selected labeled images. A few examples of correctly labeled images are below, organized by class.<img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/d6e522b3-b00e-4e91-47f5-3f709d144300/public" alt="Examples of correctly labeled images by class">With the help of Scale Rapid, raw data from the web was transformed into cleaned and confirmed data and separated into class A, class B, and neither. <h2>Model</h2> CycleGAN produces mappings from class A to class B and vice versa using two different types of losses- adversarial loss and cycle consistency loss. Adversarial loss incentivizes the mapping to generate images that look similar to the target set. We achieve this by optimizing against an adversary D–a discriminator that tries to separate the generated images for the target class from actual data from the target class. Cycle consistency loss incentivizes the mapping to produce images that look like they fit the distribution of the target class.This occurs by minimizing the difference between original data and that same data transformed into the same target class.<img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/cdcd9177-a511-4863-70fa-6a9c1194dd00/public" alt="Cycle consistency loss">Cycle consistency loss is calculated by the difference between the untransformed class A image and the double-transformed class A image. The mapping aims to minimize this.We trained CycleGAN with our Scale Rapid sorted dataset.<img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/3c9b78d0-1aaa-4d49-ad1d-e7d91e37ca00/public" alt="Loss Function Graph"><h2>Model Results:</h2> We are now able to anime-fy images! A few examples:<img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/5a438327-e4b7-41e5-a19b-2cd457c94d00/public" alt="CycleGAN Images"><a href="https://scale.com/rapid" rel="noopener noreferrer" target="_blank">Try Scale Rapid</a> for your own project today!

In this blog, we explore combining Scale Rapid Labeling with CycleGAN to anime-fy images.

Using Scale Rapid to train an unsupervised model.

Using CycleGAN to Train Unsupervised ML Models

Using Rapid and Nucleus together will get you better results than using Rapid alone.

 Using Scale Nucleus with Rapid to Unearth Errors—Label Smart, not Hard

Peer-Reviewed Doesn’t Mean Perfect Data

Mission critical industries including logistics, financial services, government, and healthcare rely on high quality document data extraction in order to optimize business workflows, save money, and improve customer experience. In this article, we explain the conventional approaches for document processing, deep dive into drawbacks with these solutions, and introduce a new approach that enables higher accuracy, even across the most challenging document types. <h3>What is OCR?</h3> Optical character recognition (OCR) is a class of document processing technologies that, as the name suggests, reads characters from text on a page. Originating in the 1920s, OCR has since evolved to support use cases involving data extraction from digitized documents. There are two kinds of OCR technologies: First, generic OCR, which pulls all of the text on a digitized document (image, PDF, etc.) and outputs a raw list of the words on a page. With a number of improvements in character recognition technology, generic OCR has become suitable for distinguishing words and reading complex font types. <img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/98229f9a-1e6c-4a80-7416-5a38dfbd1d00/public" alt="Generic OCR: extraction/digitization but no field classification"> Second, intelligent OCR, which has become a popular alternative since these systems can automate data extraction for key fields of interest rather than just receiving a blob of all the text on a page. Intelligent OCR is built on generic OCR, but enables a user to define a specific template for a document, manually specify all the fields they need extracted, and automatically get only the structured data they need for all similar-looking documents. <h3>Problems with OCR</h3> There are clear challenges with both of these conventional OCR approaches that reduce the quality of data extraction. Generic OCR simply outputs all the text on a page and doesn’t extract only the key information you might need for downstream workflows. These solutions are typically paired with complex regex or hard-coded heuristics that could require a team of nearly 4-5 engineers just to support parsing out required fields and maintaining the infrastructure. Even then, the variability in text layout and document format ultimately results in poor data extraction accuracy (~50-70% for simple documents, 10%-60% for complex unstructured documents). Intelligent OCR can extract fields you need in a document, but requires manually creating templates and rules to define the location of those fields for every document layout. This approach can quickly become painstaking and incredibly difficult to scale as organizations expand use cases. These solutions also fail when it doesn’t find a matching template on new layouts or when you need to add new fields to extract—common in real-world scenarios that have high volumes or complex, unstructured document types. While intelligent OCR approaches might get up to 70-80% data extraction accuracy with heuristics, it still requires significant engineering and manual operations support to ensure errors don’t fall through the cracks. <img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/6eb13fad-3577-422f-8cc9-51fc0a1eb100/public" alt="Intelligent OCR: extraction and digitization, classification from templates only"> <h3>To summarize:</h3> <ul><li> </li><li>Generic OCR simply extracts text but doesn’t identify the specific fields you need extracted. This approach requires significant engineering effort to maintain the infrastructure and yields low accuracy data.</li><li> </li><li> </li><li>Intelligent OCR can extract your desired fields but requires manually defining templates and hard-coded rules. This approach can get to higher accuracy levels than generic OCR but still results in many errors and doesn’t scale to large volumes and high variability use cases. </li><li> </li></ul><h3>Why use Adaptive ML Document Processing instead?</h3> Running OCR is the first of many steps needed to have a robust document processing solution that can extract data at the highest quality. While these solutions may get you part of the way there, they don’t deliver a complete solution because OCR still requires significant engineering and operations support in order to pull out key fields and fix errors. While we hear about machine learning frequently in moonshot applications like autonomous driving and Artificial General Intelligence, the technology can be significantly useful today in enterprise use cases. Leveraging machine learning is key to getting a high quality and truly automated document processing solution even across highly variable and challenging document types. Rather than relying on off-the-shelf methods or ones that need manually defined templates, an end-to-end machine learning based approach can enable a robust solution with higher accuracy results for highly variable unstructured and structured documents. <img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/8d041be1-a4c2-4e5b-7641-785b4c115000/public" alt="Scale Document AI Adaptive Document Processing: the Best of All Worlds"> However, setting up these systems in-house requires significant investment across the machine learning stack in order to deploy a production system, including data labeling, model training, and continuous monitoring. Choosing the right partner to do this at scale can help you get higher accuracy, lower latency, and save time and money to spend on your company’s core competencies. <h3>Why Scale?</h3> Scale Document AI supports customers with quickly deploying a complete document processing solution with guaranteed quality SLAs for any of their document types. We use our own in-house OCR engine for identifying the text on a page, but don’t stop there. What’s unique about our approach is that we leverage base models built on state of the art Computer Vision and Natural Language Processing research. These models are trained on millions of data points and we further refine them for each customer use case. This fine-tuning uses Scale’s proprietary data labeling and ML training infrastructure that we’ve developed over the past few years to deliver a solution that consistently performs at high accuracy – even across unique edge cases, customer requirements, or variable document types. Scale provides Adaptive ML document processing for many enterprises and organizations across financial services, logistics, manufacturing, healthcare, real estate and government. For example, Scale delivers fast extraction from messy invoices <a href="http://scale.com/customers/brex" rel="noopener noreferrer" target="_blank">for Brex</a>, accurate and timely data from complicated shipping paperwork <a href="http://scale.com/customers/flexport" rel="noopener noreferrer" target="_blank">for Flexport</a>, and essential data out of titles, mortgage applications, and deeds <a href="https://scale.com/customers/states-title" rel="noopener noreferrer" target="_blank">for Doma</a> in real estate transaction processing. If you’re looking for a solution that’s higher quality, lower latency, and free of the hassle of creating and maintaining templates, <a href="https://scale.com/document-ai" rel="noopener noreferrer" target="_blank">learn more and contact us</a>.

OCR and even Intelligent Document Processing fall short, while AI-based extraction delivers.

With adaptive ML models performing extraction, Scale Document AI goes far beyond OCR.

Comparing Scale Document AI to Traditional OCR

open-dataset-nucleus

slugs must be all lowercase. ie: ‘my-slug’

events-banner

This entry represents both interview quote highlight and interview chat highlight because of Builder limitations. Choose the fields that matches either of them.

interview-highlight

If it's selected, the author and quote fields will be taken, otherwise the chat fields. The default is for chat.

resources-download-model

transform-session

CTA to open the session. Leave blank if you don't want a cta for that session

Link for the CTA. Usually the link to the exchange session

blog-filter-type

event-transform-x-ticket

solution

homepage-quotes

logos

homepage-content

builder-debug

transfrom-x-in-person

event-model

Toggle if the event is featured. Featured events will be moved to the top of the list

guides-cta

summit-2023-partners

sales

z[deprecated] page

quotes

z[deprecated] compare

product

blog-category-model

announcement-banner

accelerate

Vercel Powered and Edge Computed OG Image. Preferred over og img file upload. If this field is non empty, it will take precedent over the uploaded file

summit-2023-speakers

homepage-resources

customers

Display order on the page /customers. Higher values show first

page-cross-product-webinar

interview

event-transform

Event model for new marketing events page

event-2-0

homepage-event

custom-fonts

transform-x-speaker

Be careful to not add any trailing white spaces to the slug. Slugs must be all lowercase. ie: ‘my-slug’

Order of the featured speakers displayed on the home page for transformX

z[deprecated] document-ai

summit-2023-agenda

event-converge

hosts

author

transform-x

Page: /In-Person. Section: Hero. Text: Title Text

Page: /In-Person. Section: Slider Card. Text: Title Text

Page: /In-Person. Section: Slider Card. Text: Card Body

Page: /In-Person. Section: Hero. Text: Tagline

open-dataset

z[deprecated] customers

URL redirections for Website. Disclaimer: takes 5~ min for each added or edited.

url-redirects

Use relative path. use `/blog/foo` not `https://scale.com/blog/foo`

legal

event

Optional second image intended to show on the grid view, should be a more squared image closer to 1:1 aspect ratio than the one used in the Cover field.

guides

z[deprecated] guides-old

z[deprecated] guides-lp

partners

blog-model

Option to list or not list this post in the Blog index. Enabled = not list in the index, Disabled = list in the index

Products

Government

Solutions

Resources

Customers

Pricing →

Blog

The future of your industry starts here.