Methodology

Over our $M$ models, we let $A=\{(m,m^{'}): m<m^{'}, \text{ and } m, m^{'} \in [M]\}$ denote our comparative data set.

At time $t \in \mathbb{N}$, we serve the human a pair of models $A_{t}\in A$ and we have our evaluator’s response $H_{t}\in [0, 0.5, 1]$. A 1 means that model $m$ is preferred over model $m^{'}$ and a 0.5 means that the models were equally preferred.

$P(H_{t}=1)=\cfrac{1}{1+e^{\xi m^{'}-\xi m}}$

Where $\xi$ is an M-length vector of "BT" coefficients. We then want to estimate the BT coefficients by minimizing the binary cross-entropy loss:

$s(\hat{P}) = \mathop{\operatorname{arg\,min}}\limits_{\xi}\mathbb{E}^{\mathbb{P}}_{A,H}\bigg[l\bigg(H, \cfrac{1}{1+e^{\xi A_{2}-\xi A_{1}}} \bigg)\bigg]$

Where $l$ is the binary cross-entropy loss,

$l(h,p)=-(h\log(p)+(1-h)\log(1-p))$

Additionally, we’ll minimize this loss while using inverse weighting by $P(A_{t})$ to target a score with a uniform distribution over $A$. This inverse weighting isn’t strictly necessary, however, as our pairwise comparisons between models are very close to equal. We perform the below formula to get our final BT-score.

$s(\hat{P}) = \mathop{\operatorname{arg\,min}}\limits_{\xi}\sum^{T}_{t=1}\cfrac{1}{P(A_{t})}l\bigg(H_{t},\cfrac{1}{1+e^{\xi_{A_{t,2}}-\xi_{A_{t,1}}}}\bigg)$

where $A_{t} \text{\textasciitilde} P$. This score is converted to an Elo-scale with the simple conversion $1000 + s(\hat{P})\times 400$ and is sorted to get our final ranking.