We recently worked with the team at Cloudinary to help build and evaluate a better image compression and quality measurements. The results of Cloudinary’s work on Scale were very insightful and we wanted to share it broadly to demonstrate how more companies can leverage human judgments to build high quality features. Without further ado, <a href="https://cloudinary.com/blog/author/tal_lev_ami" rel="noopener noreferrer" target="_blank">Tal Lev Ami</a>, CTO of Cloudinary, wanted to write about how they used Scale: <blockquote> Here at Cloudinary, we provide a cloud-based tool that enables our users to</blockquote><blockquote> compress images and video for their websites and apps. Our goal is to</blockquote><blockquote> preserve the visual integrity of the content, but deliver the smallest file</blockquote><blockquote> size to any device or browser to ultimately optimize website performance and</blockquote><blockquote> end user satisfaction.</blockquote> <blockquote> One of the hallmarks of the Cloudinary solution is the ability to automate</blockquote><blockquote> many functions of image compression, so that developers don’t have to spend</blockquote><blockquote> time tweaking each photo and making multiple copies of different sizes and</blockquote><blockquote> resolutions to fit every possible scenario. Compression algorithms can be</blockquote><blockquote> tricky because they’re trying to make changes that have the smallest visual</blockquote><blockquote> impact, but different images can react differently to compression.</blockquote> <blockquote> As we were developing the algorithm for our “q_auto” capabilities — which</blockquote><blockquote> strikes a balance between visual quality and file size — we needed to test</blockquote><blockquote> it to understand how the resulting images compared to the human eye. Enter</blockquote><blockquote> Scale.</blockquote> <blockquote> Many image compression formats — like JPEG 2000 and JPEG XR — have been</blockquote><blockquote> tweaked to score well on particular metrics, such as peak signal-to-noise</blockquote><blockquote> ratio (PSNR). But these don’t always correlate with human perception on</blockquote><blockquote> image quality.</blockquote> <blockquote> We leveraged Scale to compare pairs of images and give us perspective on</blockquote><blockquote> which image was liked most by humans. With Scale, we did a variety of tests,</blockquote><blockquote> comparing several formats, including WebP, JPEG 2000, JPEG XR (lossy) Lepton</blockquote><blockquote> (MozJPEG, recompressed with Lepton), FLIF, BPG, Daala, and PNG8</blockquote><blockquote> (pngquant+optipng). We also were able to get input on the difference between</blockquote><blockquote> the uncompressed original image vs. a compressed version.</blockquote> <blockquote> Scale enabled us to create A/B comparisons that were viewed by human</blockquote><blockquote> observers. We submitted over 4,000 image comparisons to Scale, sending at</blockquote><blockquote> least four independent Scale requests for each pair of image. This resulted</blockquote><blockquote> in at least eight actual human comparisons for each pair of images. The</blockquote><blockquote> outcome of these comparisons were evaluated beside other perceptual metrics</blockquote><blockquote> such as PSNR, Google’s Butteraugli, DSSIM (Structural (Dis)Similarity) and a</blockquote><blockquote> new metric Cloudinary developed called</blockquote><blockquote> <a href="https://cloudinary.com/blog/detecting_the_psychovisual_impact_of_compression_related_artifacts_using_ssimulacra" rel="noopener noreferrer" target="_blank">SSIMULACRA (Structural SIMilarity Unveiling Local And Compression Related</a></blockquote><blockquote><a href="https://cloudinary.com/blog/detecting_the_psychovisual_impact_of_compression_related_artifacts_using_ssimulacra" rel="noopener noreferrer" target="_blank"> Artifacts)</a>.</blockquote> <blockquote> The results showed that overall, PSNR is “correct” in only 67 percent of the</blockquote><blockquote> cases. Butteraugli gets it right in 80 percent of the cases, and DSSIM in 82</blockquote><blockquote> percent of the cases. Our new metric, SSIMULACRA, agrees with human</blockquote><blockquote> judgments in 87 percent of the cases. Looking just at the high-confidence</blockquote><blockquote> human judgments, we found about 78 percent agreement for PSNR, 91 percent</blockquote><blockquote> for both Butteraugli and DSSIM, and almost 98 percent agreement for</blockquote><blockquote> SSIMULACRA. You can</blockquote><blockquote> <a href="https://cloudinary.com/blog/detecting_the_psychovisual_impact_of_compression_related_artifacts_using_ssimulacra" rel="noopener noreferrer" target="_blank">read more about SSIMULACRA and these results on the Cloudinary blog</a>. Or if you want to give it a try:</blockquote><blockquote> <a href="https://github.com/cloudinary/ssimulacra" rel="noopener noreferrer" target="_blank">SSIMULACRA is free and open-source software!</a></blockquote> <blockquote> The results of Scale comparisons gave us useful data points to validate our</blockquote><blockquote> metrics and provided more insights into the compression benchmarks we are</blockquote><blockquote> running and the comparison of various image formats. And from these insights</blockquote><blockquote> we were able to improve our visual perception metrics and fine-tune our</blockquote><blockquote> “q_auto” functionality so we know how aggressively we can compress images.</blockquote> <blockquote> Through this process we were impressed not only by the useful data points</blockquote><blockquote> derived from the Scale, but also the great support we got from the company</blockquote><blockquote> and the product’s ease-of-use, all which came at a reasonable price.</blockquote>

We recently worked with the team at Cloudinary to help build and evaluate a better image compression and quality measurements.