Prachi Bodas

At first glance, human-in-the-loop processes seem like a relic of older times. Before technological advancements, humans effectively performed functions completed by computers today. For example, NASA hired large teams of arithmetic processors- men and women whose only job was to evaluate numerical expressions quickly and accurately. These calculators were pivotal to enabling space exploration.At Scale, our labeling teams are just as critical to achieving our mission. We’ve found we’re able to effectively apply the principles behind generative adversarial networks to our human-in-the-loop processes to deliver the highest quality annotations to customers.<h2>How does this work, exactly?</h2>A GAN, or generative adversarial network, is given an unlabeled dataset and learns to produce new examples that are not already in the dataset but are indistinguishable from the items in the dataset. For example, a net meant to produce new images of flowers would be fed as input images of real flowers. To achieve this goal, the net has to achieve two things:<ol><li>The generated flowers must not be distinguishable from real flowers. That is to say, if given a set of both generated and real flower images, one should not be able to tell which is which.</li><li>In achieving the above, the GAN must not simply produce copies of the real flower images.</li></ol>Therefore, the net must be trained to generate new flowers, discriminate between real and fake flowers, and avoid simply replicating training data. The training loop jointly optimizes the generator (which produces new images that are not copies of the training set), and the discriminator (which attempts to discriminate between training data and generated images). The net is successful when the generator becomes accurate enough to fool the discriminator, i.e. the discriminator cannot outperform random choice when classifying generated vs non-generated examples.<img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2F018ccca0549b448bb3686f7d5703244e">In the context of human-in-the loop processes, we can apply this same principle to produce quality labels on subjective tasks. Let’s say the pipeline we’ve chosen has a minimum of two annotators seeing each task, one attempter and one reviewer.Example Task:<img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2F4a26d7e4818f49dfa8acd7616a2bf50c">Instructions: Please describe the image as accurately and descriptively as possible with a short phrase.Possible attempter responses: <ul><li>Attempter 1: ‘asdf’</li><li>Attempter 2: ‘trees’</li><li>Attempter 3: ‘green hills’</li><li>Attempter 4: ‘grassy hills with mountains, trees, houses’</li><li>Attempter 5: ‘grassy hills with trees and 3 houses, mountains and clouds in background’</li><li>Attempter 6: ‘tall trees on green grassy hill, small red and gray houses, blue mountains in background’</li></ul>The attempters act as generators, and can generate any one of the above responses for this task. Attempters 1-3 have written low-quality responses. On a regular rapid project, these attempters would be disabled via evaluation tasks. However, on a generative task like the above, exact-match evaluation tasks would not guarantee quality. For this image, both Attempters 5 and 6 provided accurate and descriptive responses. Therefore, we turn instead to the discriminator to filter out the better responses.Our discriminators are reviewers. A reviewer is the final person to look at a task and is able to see the attempters’ responses. Reviewers are the final stage before tasks are returned to the customer If we allow reviewers to edit an attempter’s response directly and write a caption themselves, we run into the same problem as with attempter evaluation tasks.&nbsp;However, once we move more into thinking of the reviewer as a GAN discriminator, the solution becomes clear. Reviewers must discriminate between good and bad responses, but they need not write any captions themselves. Although there is no 1:1 mapping between a task and a certain correct answer, we do know upon encountering a caption whether it is acceptable or unacceptable. So we give the reviewers the choice of either accepting or rejecting an attempter’s response, but we do not give them the option to change it. Now the process of evaluating their quality is simple- create evaluation tasks with pre-generated responses that are expected to be accepted or rejected appropriately.<img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2Fb8be702f8ef547cba93bf016ee11d500">The two parts of a GAN, generator and discriminator, work together to pass information between them. In our human network, reviewers may provide feedback upon rejecting an attempt, to help the generator learn. The discriminator, composed of both the reviewers and the reviewer benchmarks managed by the customer, learns in turn from the generator. From a variety of generated data, the customer and in turn the reviewers are able to converge on a similar understanding of the desired result.Now that we are sure the discriminator is accurately discriminating between low and high-quality responses, we are able to train the generator. What this means is that attempters are now disabled based on how many of their responses are rejected. Over time, this leads to a state where the attempters are effective, high-quality generators- essentially a trained GAN.This pipeline is available via Rapid’s <a href="https://docs.scale.com/reference/text-collection" rel="noopener noreferrer" target="_blank">text collection</a> and <a href="https://docs.scale.com/reference/categorization" rel="noopener noreferrer" target="_blank">categorization</a> projects. <a href="https://scale.com/rapid" rel="noopener noreferrer" target="_blank">Get started with Scale Rapid</a> today!

Applying the principles behind generative adversarial networks (GANs) to our human-in-the-loop data annotation processes.

Using generative adversarial networks (GANs) to align human metrics

Using GANs to Align Human Metrics

Scale Rapid is a quick and easy way to get clean, high-quality labeled data for your next ML project. We used it to train a <a href="https://scale.com/blog/imdb-rapid" rel="noopener noreferrer" target="_blank">sentiment analysis model</a> in a previous post - and we’re back now to show you how to use Scale Rapid to train an unsupervised model! In this blog, we are going to use <a href="https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix" rel="noopener noreferrer" target="_blank">CycleGAN</a> (<a href="https://arxiv.org/pdf/1703.10593.pdf" rel="noopener noreferrer" target="_blank">paper</a>) to anime-fy images with Scale Rapid. CycleGAN is trained in an unsupervised manner, meaning each X input does not need to correspond 1:1 to a y output in order to train the net. Instead of using Scale Rapid to label training data, we used it to clean data. We separated a group of images scraped from the internet into categories. This classification allows us to make the two inputs necessary for CycleGAN - a class A (real-life images) and class B (Japanese anime-style images). To start, we downloaded 10k images from Google images from a variety of different searches. At a glance, the corpus had all sorts of images. <img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/27b06dda-1661-4805-c721-0ec70ee20300/public" alt="Sample images from Google">Class A contains real-life images and Class B contains anime-style images. Image 3 belongs to Class A, and Image 2 to Class B. Images 1 and 4 do not belong in the dataset as they are neither real-life nor anime-style images. The ask for Scale Rapid is to discriminate between these three types of images: a real-world, realistically edited photograph, a Japanese-style anime scene, or neither. Rapid responses are then used as ground truth to establish which images should and shouldn’t be in the dataset we train CycleGAN on. Rapid’s categorization endpoint supports image categorization - and it's only 8 cents per image! Because CycleGAN is trained on unsupervised data, we don’t need to pair class A and class B images - we can simply separate them from one another and train. <h2>Using Scale Rapid</h2> Scale Rapid works in two phases- first, the user sets up the project &amp; instructions, and runs small calibration batches to ensure the initial results are as expected. Once quality on those looks good, larger production batches may be uploaded for labeling. For this application, we set up a categorization project. The taxonomy is just one multiple-choice question: What type of image is this? <ul><li> </li><li>Anime (japanese animation) image</li><li> </li><li> </li><li>Photorealistic (real-life) image</li><li> </li><li> </li><li>Neither</li><li> </li></ul> Scale Rapid also has a feature called long hints, which allows us to prompt labelers with specific things to think about upon choice selection via a tooltip. For the first choice (Anime (japanese animation) image) I clarified with the following long hint: "Japanese animation refers specifically to anime, as in the instructions. Other cartoon-style or generated images belong in the neither category." We begin by uploading a calibration batch, and follow up with production batches. The labeling results are easily parseable via JSON. <h2>Scale Rapid Results:</h2> A random audit of tasks found just one mistake out of 200 randomly selected labeled images. A few examples of correctly labeled images are below, organized by class.<img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/d6e522b3-b00e-4e91-47f5-3f709d144300/public" alt="Examples of correctly labeled images by class">With the help of Scale Rapid, raw data from the web was transformed into cleaned and confirmed data and separated into class A, class B, and neither. <h2>Model</h2> CycleGAN produces mappings from class A to class B and vice versa using two different types of losses- adversarial loss and cycle consistency loss. Adversarial loss incentivizes the mapping to generate images that look similar to the target set. We achieve this by optimizing against an adversary D–a discriminator that tries to separate the generated images for the target class from actual data from the target class. Cycle consistency loss incentivizes the mapping to produce images that look like they fit the distribution of the target class.This occurs by minimizing the difference between original data and that same data transformed into the same target class.<img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/cdcd9177-a511-4863-70fa-6a9c1194dd00/public" alt="Cycle consistency loss">Cycle consistency loss is calculated by the difference between the untransformed class A image and the double-transformed class A image. The mapping aims to minimize this.We trained CycleGAN with our Scale Rapid sorted dataset.<img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/3c9b78d0-1aaa-4d49-ad1d-e7d91e37ca00/public" alt="Loss Function Graph"><h2>Model Results:</h2> We are now able to anime-fy images! A few examples:<img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/5a438327-e4b7-41e5-a19b-2cd457c94d00/public" alt="CycleGAN Images"><a href="https://scale.com/rapid" rel="noopener noreferrer" target="_blank">Try Scale Rapid</a> for your own project today!

In this blog, we explore combining Scale Rapid Labeling with CycleGAN to anime-fy images.

Prachi Bodas

Using GANs to Align Human Metrics

Using CycleGAN to Train Unsupervised ML Models