Agreement / Disagreement Dataset

reddit interactions

Original Reddit Interactions

The original comments and subsequent interactions scraped from Reddit with no assigned labels.

brexit subreddit

Labeled Dataset: Full Agreement

Dataset where all annotators achieved full agreement and pairs have been checked by authors of the paper.

brexit subreddit

Labeled Dataset: 2/3 Agreement

Dataset where there is 2/3 inter-annotator agreement or above.

brexit subreddit

Labeled Dataset: Challenge

Dataset where consensus was not reached among annotators, or pairs were labeled as “unsure.”





data collection

The data was collected from Reddit using pushshift

scale redit

The pushshift.io Reddit API was designed and created by the /r/datasets mod team to help provide enhanced functionalityand search capabilities for searching Reddit comments and submissions. This RESTful API gives full functionality for searching Reddit data and also includes the capability of creating powerfuldata aggregations. With this API, you can quickly find the datathat you are interested in and find fascinating correlations.

data annotation

Labeling Taxonomies



The reply shows approval towards the initial statement or initial author.

Some examples of phrases that express agreement are: “I agree”, “That’s absolutely right”, “That’s so true”, “Exactly”, “That’s how I feel”, etc.



There is a topical exchange between authors but agreement / disagreement is not expressed.



The reply shows disapproval or dislike towards the initial statement or initial author.

Some examples of phrases that express disagreement are: “No way”, “I don’t think so”, “Not necessarily”, “That’s not always true”, etc.



It is not possible to make a decision based on the information at hand.

oxford dataset cover

dev kit

Get Started with Oxford Dataset

Read the full Research Paper. If you use the dataset, please make sure to cite our paper.