Nishant Subramani

<h2>Introduction</h2> The ML Team at Scale hosts a weekly reading groups where members choose papers from the broad AI/ML community and discuss them ranging from topics in Computer Vision to NLP to Active Learning. Here, we describe a brief summary of some of the insights we gained from various papers and how we aim to use some of that knowledge in future research projects and applications to Scale AI’s business.<h3><a href="https://arxiv.org/abs/2008.09049" rel="noopener noreferrer" target="_blank">Discovering Useful Sentence Representations from Large Pretrained Language</a></h3><h3><a href="https://arxiv.org/abs/2008.09049" rel="noopener noreferrer" target="_blank"> Models</a></h3>Presenter:<a href="https://www.linkedin.com/in/nishantsubramani/" rel="noopener noreferrer" target="_blank">Nishant Subramani</a> This is Scale’s first research paper and we wrote a <a href="https://scale.com/blog/discovering-useful-sentence-representations" rel="noopener noreferrer" target="_blank">blog post summarizing the paper</a>. This paper focuses on whether we can adapt pre-trained language models off-the-shelf as universal decoders. To be considered “universal,” a decoder must have an implicit representation for any target sentence such that it can recover that sentence exactly when conditioned on its representation. We investigated whether such representations exist and whether they can be easily discovered. Experiments show that not only do these representations exist for sentences from a variety of different genres, but also that our methods can recover these sentences almost perfectly without fine-tuning the underlying language model at all.<img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/733c8e61-0b1b-4a08-9092-43c494a9fc00/public" alt="Scale blog ml papers B_01"> <h3><a href="https://arxiv.org/abs/1910.04126" rel="noopener noreferrer" target="_blank">Scalable Nearest Neighbor Search for Optimal Transport</a></h3>Presenter:<a href="https://www.linkedin.com/in/anastasiia-alokhina-534b1014a/" rel="noopener noreferrer" target="_blank">Anastasiia Alokhina</a> The authors of this paper look at the problem of nearest neighbor search with respect to Wasserstein distance. As Wasserstein distance has become popular in high-resource data environments and domains (images, text, etc.), search becomes prohibitively slow. As a result, approximation methods are important. The authors introduce a Quadtree variant called a Flowtree that formally shows better asymptotic accuracy and empirically show that it beats existing methods on a variety of different real-world datasets on either running time or accuracy. Flowtree is a fast nearest neighbor search algorithm for Optimal Transport (Wasserstein-1 distance), which has a linear running time and speeds up SOTA performance by up to 7x. They do this by evaluating the optimal tree flow in the original metric space. <img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/2f1c10db-58ad-4966-b9aa-a69c64080500/public" alt="scale blog ml papers B_02"> At Scale AI, we work with lots of high dimensional data such as documents and images for a variety of tasks. For these, we often need to find good representation spaces for these data points such that we can do fast and highly accurate similarity search in order to find outliers in datasets in order to find errors in labeling and find examples to label to improve our models using active learning. We think the flowtree method here could be invaluable in giving us the ability to use a different distance for this end and is something we will explore going forward. <h3><a href="https://demuc.de/papers/schoenberger2016sfm.pdf" rel="noopener noreferrer" target="_blank">Structure-from-Motion Revisited</a></h3>Presenter:<a href="https://www.linkedin.com/in/ppyht2/" rel="noopener noreferrer" target="_blank">Hongbo Tian</a> Structure-from-Motion(SfM) is a field of computer vision techniques which aim to reconstruct full 3-dimensional scenes from only a series of images. Accurate SfM algorithm can reproduce colored point clouds similar to LiDAR. The authors of this paper along with <a href="https://colmap.github.io/bibliography.html" rel="noopener noreferrer" target="_blank">Schönberger’s thesis</a> introduced a robust computational pipeline for accurate dense 3D reconstruction. The authors also introduced <a href="https://colmap.github.io/index.html" rel="noopener noreferrer" target="_blank">COLMAP</a> - an open-sourced CUDA accelerated software to reproduce the results. <img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/62e7cdd5-e0b4-4fc0-7dee-619984c25800/public" alt="Scale blog ml papers B_03"> At Scale AI, our labelling pipeline is not subjected to the same online computational constraints compared to our customers. This allows us to pursue batch methods to enrich our 2D dataset, enabling our human labelers to capture 3D contexts from only images. <h3><a href="https://storage.googleapis.com/pub-tools-public-publication-data/pdf/59f3bb33216eae711b36f3d8b3ee3cc67058803f.pdf" rel="noopener noreferrer" target="_blank">Representation Learning for Information Extraction from Form-like</a></h3><h3><a href="https://storage.googleapis.com/pub-tools-public-publication-data/pdf/59f3bb33216eae711b36f3d8b3ee3cc67058803f.pdf" rel="noopener noreferrer" target="_blank"> Documents</a></h3><h3><a href="https://storage.googleapis.com/pub-tools-public-publication-data/pdf/59f3bb33216eae711b36f3d8b3ee3cc67058803f.pdf" rel="noopener noreferrer" target="_blank"> </a></h3>Presenter: <a href="https://www.linkedin.com/in/nishantsubramani/" rel="noopener noreferrer" target="_blank">Nishant Subramani</a> The authors aim to extract structured information from form-like documents. They observe that forms have fields that often correspond to well-understood types and are often associated with a specific key phrase that bears a visual relationship with it. They also observe that key phrases in documents are drawn from a small vocabulary. Due to these observations, the authors build a system with two parts: candidate generation and scoring &amp; assignment. In the candidate generation phase each field type is associated with a candidate generator based on a cloud-based entity extraction service. Spans are then detected. In an invoice there are multiple dates. Every date in an invoice becomes a candidate for every date field in the target schema. This process is repeated for every target field. In the scoring and assignment phase, the goal is to find the correct extraction candidate for each field. To do this, a score is computed for each candidate independently using a neural model. Next, the most likely scored candidate is assigned for each field. This process yields an independent learned representation based only on the candidates’ neighborhood (this is entirely independent of other candidates of fields). The neural scoring model is below. <img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/18c49c25-c170-4dcd-e6da-51144e415600/public" alt="Scale blog ml papers B_04"> Our major takeaways are that structured extraction is not super well-studied by the academic community, but is relatively simple. The extraction system presented has promising accuracy and generalizes well to different domains. The representations have some interpretability as well. Ultimately some of the learnings from this paper could influence some of our work on <a href="https://scale.com/document" rel="noopener noreferrer" target="_blank">Scale Document</a>. <h3><a href="https://arxiv.org/abs/1912.11370" rel="noopener noreferrer" target="_blank">Big Transfer (BiT): General Visual Representation Learning</a></h3>Presenter: <a href="https://www.linkedin.com/in/felix-lau-0386641a/" rel="noopener noreferrer" target="_blank">Felix Lau</a> This paper focuses on how to transfer pre-trained representations to other tasks in the visual space. They propose a recipe called Big Transfer (BiT) and achieve very strong performance on ILSVRC-2012 (87.5% top-1 accuracy), CIFAR-10 (99.4%), and the Visual Task Adaptation Benchmark (VTAB; 76.3%). <img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/e2a0b8c1-282d-4511-835f-a183b368ad00/public" alt="Scale blog ml papers B_05"> BiT has two phases: upstream and downstream. In upstream pre-training, the authors first investigate the scale of computational budget and how that affects performance. Secondly, they look at ground normalization and weight standardization, which helps significantly with small-batch and large-batch training alike. In downstream transfer, the authors propose a cheap fine-tuning methodology by using the BiT-HyperRule to select the most important hyperparameters for tuning as a function of the task’s intrinsic image resolution and number of datapoints. The hyperparameters they thought were important were training schedule length, resolution, and whether to use MixUp regularization. There are a few takeaways from this method we found interesting. There’s a lot of value in balancing the computational budget and simplifying the experimental pipeline when needed. The BiT-HyperRule focusing on only a few hyperparameters was illuminating. We were interested in the dynamics of how large batches, group normalization, and weight standardization interplayed and were surprised at how poorly batch normalization performed relative to group normalization and weight standardization for large batches. The empirical strength of group normalization and weight standardization surprised us as well and we will start exploring these techniques in some of our computer vision models with lots of data available. However, the most impressive part was the empirical few-shot learning performance relative to the baseline. We are trying to build models that quickly adapt to new customers and domains and often struggle with fine-tuning, so these ideas are very top of mind. <h3><a href="https://arxiv.org/abs/1905.03677" rel="noopener noreferrer" target="_blank">Learning Loss for Active Learning</a></h3>Presenter: <a href="https://www.linkedin.com/in/rishgoyell/" rel="noopener noreferrer" target="_blank">Rishab Goyal</a> The authors present an active learning method that is a task-agnostic by attaching a loss prediction module to a target network and train it to be able to predict target losses of unlabeled inputs. The module can then provide insight into data that the model could predict incorrectly and thus improve the overall model by choosing just those examples to go and label. <img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/105e736f-2bce-4ad5-6e99-7279ab964900/public" alt="Scale blog ml papers B_06"> Their general process is that for every labeled example you predict the target and predicted loss for your prediction. Next, you take the unlabeled pool and pass them through the model to get predicted losses. Lastly you annotate the top-k data points and add them to the labeled training set. The loss prediction module uses a ranking loss. As a data labeling company, we are very interested in active learning and how we can leverage our own strengths to better build ML models. We really liked the approach because it is widely applicable, but were concerned that there could be many degenerate cases. We weren’t convinced that the loss prediction module would be strong enough to offer significant improvements over uncertainty or coverage based methods such as entropy, query-by-committee, and coreset based methods. <h3><a href="https://arxiv.org/pdf/1912.13318v5.pdf" rel="noopener noreferrer" target="_blank">LayoutLM: Pre-training of Text and Layout for Document Image</a></h3><h3><a href="https://arxiv.org/pdf/1912.13318v5.pdf" rel="noopener noreferrer" target="_blank"> Understanding</a></h3>Presenter: <a href="https://www.linkedin.com/in/malcolm-greaves/" rel="noopener noreferrer" target="_blank">Malcolm Greaves</a> The authors try and solve the problem of automated understanding from scanned business documents. They use both computer vision and natural language processing techniques via textual and visual layout information for pre-training their system. Their system achieves SOTA on a set of layout analysis, receipt understanding, and document image classification tasks. They claim that their empirical performance is from their novel loss functions: a masked visual-language model (MVLM) loss and a multi-label document classification (MDC) loss. <img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/756680db-57a2-4448-2f6f-3f0976fa5500/public" alt="Scale blog ml papers B_07"> MVLM extends the masked language model loss from BERT by randomly choosing some words to mask. They then keep the masked token’s 2D positional embedding and forces the model to predict the word’s text. MVLM forces the model to understand language context and 2D relationships between words in the document. MDC assumes that documents have possibly several labels and is a standard cross-entropy loss. This is optional and seems to help on the document image classification tasks. We found this paper interesting, but computationally demanding: alongside their loss functions, a key contribution of this paper is their pre-training routine. Without significant pre-training, these models do not seem to work well. This work requires off-the-shelf OCR and treats that as ground truth, which simplifies the work, but makes it less practical for document analysis practitioners. For the business document tasks that our ML team takes on, we often like to have greater control on the OCR task. It's sometimes advantageous to be able to adjust OCR to provide better downstream task-dependent results. We do a lot of document analysis at Scale, but can’t obfuscate or explain away the OCR component, which do produce some errors (e.g. with handwriting). Although this approach needs word-level annotations, Scale is in a unique position to get large amounts of these labor-intensive labels using our labeling platform. <h3><a href="https://arxiv.org/abs/2006.16205v1" rel="noopener noreferrer" target="_blank">Simplifying Models with Unlabeled Output Data</a></h3>Presenter: <a href="https://www.linkedin.com/in/alexandre-matton-73042b170/" rel="noopener noreferrer" target="_blank">Alexandre Matton</a> This paper focuses on tasks whose outputs have to obey some constraints. This happens in pseudo-code to code transcription for example. What we ultimately want is a code that compiles and runs without error. The usual way to address these tasks is to build a single end-to-end model and hope that it will learn the input-output mapping and understand the output constraints. The authors provide a new way to tackle these tasks by replacing the original model by two sub-models: a base predictor and a denoiser. The goal of the base predictor is to learn the mapping between the input and the output. The goal of the denoiser is to learn the constraints of the output space, in order to fix the base predictor’s output. <img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/9578c3e1-2f67-408f-2bd1-a2972c25ee00/public" alt="Scale blog ml papers B_08"> There is a reason why this decomposition works well. As the denoiser simplifies the job of the base predictor, it is now possible to increase the regularization of the base predictor without affecting the overall model performance. This in turn leads to solutions that generalize better on unseen data. The big win of this architecture is that the denoiser can be trained on unlabelled output data, which is often abundant (e.g. there is a huge amount of compiling code pieces freely available on github). A big number of tasks actually impose constraints on the output (such as molecule generation, or translation from one language to another). At Scale, our ML team also tackles such problems. This paper provides a very general way to increase our downstream scores on them. <h3>Join our Team</h3> If you’re interested in joining our growing engineering team, take a look at our <a href="https://scale.com/careers" rel="noopener noreferrer" target="_blank">careers page</a> for open positions.

The machine learning research papers the Scale AI team read and discussed in Q3 2020.

Research Papers the Scale AI Team has Read in Q3 2020

Scale AI Machine Learning Digest - Q3 2020

<h2>AI/ML Research at Scale AI:</h2>
&nbsp;
Scale AI is dedicated to accelerating the development of AI applications. As
part of this mission, Scale AI&rsquo;s growing AI/ML Team aims to meaningfully
contribute to the AI ecosystem with research and resources that can bring
significant and practical benefits to our customers and the broader research
community.
&nbsp;
Research and experimentation happens throughout Scale. In some cases, our
contribution is in areas where the bar has already been set high. Some
examples include the open-source datasets we have released with various
partners such as <a href="https://pandaset.org/" target="_blank" rel="noopener noreferrer">PandaSet</a>,
<a href="https://www.nuscenes.org/" target="_blank" rel="noopener noreferrer">nuScenes</a>, and
CoNLL Balanced. As our AI/ML team continues to grow, we aim to contribute in entirely new
spaces by publishing cutting edge research papers. In this blog post, we
spotlight a research paper we recently submitted.
&nbsp;
The research paper was a collaboration with
<a href="https://www.linkedin.com/in/nivedita-suresh/" target="_blank" rel="noopener noreferrer">Nivedita Suresh</a> of
Arrive. <a href="https://arriveorigin.com/" target="_blank" rel="noopener noreferrer">Arrive</a> links multi-omics and
AI cell characterization to discover hidden patterns in data and improve
outcomes in precision medicine.
<h2>&nbsp;</h2>
<h2>Conditioning Unconditional Language Models to Recover Sentences without</h2>
<h2>Fine-Tuning:</h2>
&nbsp;
Pretrained language models are used as encoders with widespread success for a
wide variety of NLP tasks. These models learn useful representations that can
be directly applied to certain tasks. Despite the success of these models on
the encoder, there isn't much research out today on whether these large
pretrained language models can be used as universal decoders. To be considered
"universal," a decoder must have an implicit representation for any sentence,
such that it can recover that sentence exactly when conditioned on its
representation.
&nbsp;
Task-specific decoders are typically trained from scratch for each task. In
this research, we propose a method to try and learn useful representations of
sentences using large, pretrained, transformer-based decoders. Our work
investigates whether a universal decoder is even possible. If we had access to
such a decoder, we could use it for a wide variety of sequence generation
tasks without having to retrain or rebuild the decoder on new data. This could
open up a new paradigm for low-resource machine translation, summarization,
image captioning, and dialog generation.
<h2>&nbsp;</h2>
<h2>The Sentence Space: How Language Models Represent Sentences</h2>
&nbsp;
Transformer-based language models like ELMo, BERT, and GPT-2 have replaced
word vectors as off-the-shelf general-purpose encoders for natural language
understanding. These models are trained on large amounts of text data and
represent a sentence as a sequence of hidden states that come from a final
layer of the transformer model. Representations in this sentence space are
sequence length dependent, making comparisons between sentences with differing
lengths inequitable and measuring the efficacy of using an unconditional
language model as a universal decoder impossible.
&nbsp;
To resolve these issues and to make analysis easier, we propose to
reparametrize the original sentence space into a lower-dimensional and
sentence length agnostic vector space. We do this by adding a bias term to the
fixed language model and finding the representation that minimizes the cross
entropy loss of the sentence. This reparameterization now gives us the ability
to project sentence tokens to representations (sentence encoding) and recover
sentences from the representation (sentence recovery) via the fixed language
model. We develop three representation injection mechanisms and inject biases
at three different locations.
<img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2F7b93f3caad104979b5aa2042a6c7a7ff">
We add a bias Z&rsquo; to the embedding, transformer layers, and before the language modeling head in GPT-2. &rsquo;SA&rsquo; refers to selfattention, &rsquo;LN&rsquo; to layer normalization, &rsquo;FFN&rsquo; to a fully-connected layer, and &rsquo;LM Head&rsquo; to the last fully-connected layer.
&nbsp;
&nbsp;
<h2>Experiments:</h2>
&nbsp;
Since we are testing the efficacy of a large pretrained language model as a
universal decoder, we measure how well a representation can recover a
sentence when injected into a language model. We measure performance across
sentences from 4 different genres: books, news articles, Wikipedia, and
movie dialogs. We conduct controlled experiments, varying the representation
injection mechanism, representation injection location, initialization, and
dimensionality of the representation.
&nbsp;
<h2>Results:</h2>
&nbsp;
We find that our representations recover sentences nearly perfectly even
across genres -- achieving a BLEU score of over 98.
&nbsp;
&nbsp;
<img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/696b5ccc-0d02-4fea-4baf-16390e159600/public" alt="BLEU Scores">
BLEU score performance stratified by genre for different dimensionalities of
Z.
&nbsp;
&nbsp;
&nbsp;
In addition, we observe that the intrinsic dimension of the representation
space is the model&rsquo;s latent dimension, which indicates that the language
model uses its entire capacity to represent sentences.
&nbsp;
&nbsp;
<img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/0e7fa636-8519-4ff8-c7ef-019293850b00/public" alt="Sentence length v. BLEU score">
Plot of sentence length vs. BLEU score on the dataset.
&nbsp;
&nbsp;
&nbsp;
Additionally, our interpolation experiments reveal that the representation
space has some human-understandable meaning. Our learned representations
seem to have some synonym awareness. In the first sentence pair example
below, the word &ldquo;tale&rdquo; transforms to the word &ldquo;story&rdquo; and the word &ldquo;long&rdquo;
transforms to &ldquo;long-running&rdquo; when referring to a war. In the second example,
we observe some syntactic awareness at the 0.7 mixture level. The syntax of
the first sentence is retained with mostly words from the second sentence.
&nbsp;
&nbsp;
<img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/6de41532-7363-4b27-5524-0bb6e8e94800/public" alt="Linear interpolations between perfectly recovered pairs.">
Two linear interpolations between perfectly recovered pairs of
representations. Pink indicates token overlap to the first sentence, while
blue indicates token overlap to the second sentence.
&nbsp;
&nbsp;
<h2>Broader Impact:</h2>
&nbsp;
This work informs us that we can discover meaningful, sentence-length
agnostic sentence representations, hinting at the possibility of a
&ldquo;universal&rdquo; decoder. Such a decoder would improve low-resource sequence
generation task performance and allow for considerable parameter sharing in
memory and data-limited environments.
&nbsp;
<h2>Conclusion:</h2>
&nbsp;
For further details on how we set up the experiment, results, and analysis,
you can find the full paper on
<a href="https://arxiv.org/abs/2008.09049" target="_blank" rel="noopener noreferrer">arXiv</a>. If you&rsquo;re interested
in joining our growing machine learning team, you can find all of the open
positions on our
<a href="https://scale.com/careers?location=All&amp;team=Engineering" target="_blank" rel="noopener noreferrer">careers page</a>.
&nbsp;

Learning useful sentence representations from large transformer-based language models.

Nishant Subramani

Scale AI Machine Learning Digest - Q3 2020

Discovering Useful Sentence Representations from Large Pre-Trained Language Models