Goldilocks and the Three Experiments

Once upon a time, during a usability test in a low and Nether-land, a seemingly meaningless pinch of the fingers by a middle-aged Dutch man named Geert sparked a series of three experiments by a girl with golden hair.

While looking through the search results on the tablet website of Booking.com, Geert decided he wanted to see the hotel photos in a little more detail, but when he zoomed in all he got was a pixelated mess of colors! This observation lit a fire in Goldilocks, and she enthusiastically thought,

“Geert and other customers definitely want high-quality hotel photos that are beautiful and sharp!”

Viewing photos on the iPad website

My mission became as clear as an SVG icon on a retina-quality screen. I would fight for Geert and his desire to zoom. And so, during the team’s next planning meeting, I included a task to improve the experience of how the photos on the search results and landing pages were displayed.

Three of the different size property photographs

I began by digging through the treasure chest of goodies we refer to as the “tag dump” to see what sizes of images were available to use.

At the time, on both the search results and the landing pages, the website displayed performance-optimized 150 by 150 pixel hotel photos with a padding of ten pixels on each side used at their original dimensions.

The 150 pixel square photograph shown in search results

I discovered that we also had images in other dimensions such as a maximum height and width of 500 pixels, a maximum height and width of 300 pixels, and a 200 pixel square were all available.

First, I coded the square 200 pixel images in place of the square 150 pixel versions and reviewed the quality of these slightly larger images on a variety of tablet devices at different resolutions.

“Square200 photos are still much, much too blurry,” said Goldilocks. “That doesn’t solve the problem at all!”

So I moved on to the 300 pixel tall/wide images. They looked fine and appeared to be more nicely cropped, but they still weren't amazing.

The 300 pixel photographs shown in a search result

At this point, my designer-know-it-all sensibilities wanted something really, really perfect looking. After all, Geert and our other customers deserved the best that was available.

When I coded up the 500 pixel photos and refreshed the page, I was blown away. I was certain that these photos were the best solution to the problem.

“Wow!” said Goldilocks. “500 pixel photos are cropped so nicely and are so sharp they will cut through the glass of the iPads and jump out at Geert and our other customers!”

Of course, I realized that 500 pixel photos have much larger filesizes than square 150 pixel photos and that this fact would inevitably increase the time it took for the page to load. So I decided to experiment with both the 500 pixel and the 300 pixel versions, and to keep a very close eye on how the page load time was affected.

I coded up two different experiments. One experiment I created on the search results page replacing the square 150 pixel images with 500 pixel ones. And I created another experiment on the landing pages replacing the square 150 pixel with 300 pixel images.

After running both experiments for a couple of weeks the results were finally in. Both of the experiments were absolutely, conclusively, and beautifully neutral. Our customers seemingly didn't mind.

I went straight to the analytics to understand what went wrong. Initially thinking that the page load time would be a key factor in these experiments that was the first thing I checked. I found that the page load time on search results using the 500 pixel photos increased by more than half a second.

I wasn’t horribly surprised. What actually surprised me was that the experiment still ended up being neutral despite the incredible performance hit. That discovery led me to wonder whether higher quality images could, at the very least, balance out the negative impact of the increased page load time.

I moved on to the landing pages to see how page load time was affected there. The increase in page load time was negligible, but the results were still neutral. This baffled me. I stepped back from it for a couple of days and let everything subconsciously marinate.

After a few days, I was working on something in the search results completely unrelated to images when it hit me. This ten pixel padding around these photos serves absolutely no purpose. I’m going to get rid of it to make the images a little larger and clearer. It was the epiphany I needed.

The search results photos with unnecessary padding indicated

Excitedly, I mentioned the idea to my design friends.

Phil Hammel, Senior Designer @ Booking.com

Phil suggested, “You could use ‘background-size: cover’ to make the images fit even better! They can get bigger or smaller depending on the size of the block.” Brilliant!

Catalin Bridinel, Senior Designer @ Booking.com

And Catalin said: “Decreasing the overall height of the content blocks is typically a good thing from what I’ve noticed.” Great insight!

Erin Weigel, Senior Designer @ Booking.com

Finally, Goldilocks said: “And if I use the 300 pixel tall/wide photos the pictures will be cropped more nicely, but page load time shouldn’t be a big issue!”

And that was it. I coded up the experiment with smaller content blocks and the higher-quality images. Then I released it to customers. After a couple of weeks the results were conclusive. It was positive!

The search results photos shown without padding

“150 pixel squares were much too blurry. 500 pixel wide photos were far too heavy. But maximum dimensions of 300 pixel was just right!” declared Goldilocks as she enabled the experiment for all customers.

Reflecting on the process that led Goldilocks and her friends to the point of solving Geert’s problem, I realized that the initial “ideal solution” to the problem of using the 500 pixel photos because I liked the way they looked best would have ultimately been useless or even actively harmful. Without measuring how much our customers approve or disapprove of our application by using A/B testing and without the collaboration of my peers, I might have easily just changed the image quality based on my instincts and opinion.

Good design at Booking.com isn’t just about making something that seems to be better. It’s about measuring results, understanding the impact on the full customer experience, and actually making something better.

Collaboration is a key part of that process. Measuring our customers' happiness via A/B testing data is another key part. And finally, having incredible colleagues that contribute by providing their own insights and potential solutions to the problems provides the last key part. The ability to abandon ego, truly listen, and process this feedback is all part of the fairy tale magic.

(Goldilocks would like to thank Phil Hammel and Catalin Bridinel for always humoring her ridiculous conversation topics about design-and-the-like and offering incredible feedback, solutions, and insights. She’d also like to thank “Geert” wherever he may be.)

comments powered by Disqus