tomhubbardgreen.co.uk Bad ads, good ads, design & technology

User engagement insights: a case study

This article was published in the August 2009 issue of Figaro Digital magazine.

Although tools such as heat maps and eye tracking provide in-depth analysis of our users' behaviour there is also a wealth of simple user engagement experiments that can be carried out to better understand your users and achieve actionable insights that you can apply to your site.

The first step to generating user engagement insights involves devising robust experiments that will give you worthwhile data to interpret. If you know your site you should have no problem coming up with changes to make based on your hunches about what you think would make your website better but there are other, equally good sources of experiments too. I was looking into increasing sign ups on our events pages so one place I turned to for inspiration were direct competitors: more images, testimonials, a more emphatic call to action - I noted these techniques down along with my own ideas and drew up a plan of action.

Measuring your experiments is relatively straightforward as long as you avoid the pitfalls of choosing the wrong metrics; you only want to deal in metrics that cannot be too easily affected by external factors. I settled on examining four main metrics to assess the success of my experiments: average time on page, bounce rate, exit rate and, most importantly, conversion rate.

Our events pages generally only featured one image related to the event so the first experiment I did was to add another picture to our Great Wall of China trek page. After gathering 3 weeks worth of data I compared my user engagement metrics to the previous incarnation of the page for the same time period. I expected the effect of my change to be positive but minimal so I was surprised at the results the test yielded: average time on page was up 28%, bounce rate was down 25%, exit rate was down 14% and, crucially, conversion rate was up by 56%. In order to make sure that my results weren't an anomaly I added an additional picture to a different event page and received the same gains: bounce rate and exit rate went down whereas average time on page and conversion rate increased. These experiments led me to the insight that multiple images seem to have a positive effect on user engagement and that our events pages should be changed to incorporate this learning.

Taking a look at some competitors' sites convinced me to experiment with a testimonial on one of our events pages so I gathered some feedback from previous event participants and set up the new page. I expected the testimonial to show results similar to my previous experiment but what the results actually showed was a little more complicated. In the first iteration of the experiment the average time on page went up by 37% but the bounce rate went up by 60% and the exit rate by 50%. This all looked very counter-intuitive until I noticed that the conversion rate had increased by nearly 20%. It occurred to me that what I had created was a page that made the event look like a more serious prospect and was therefore driving away the casual enquirer but more effectively converting those users that were strongly considering taking part in the event.

It is, of course, imperative to recognize the limitations of these kinds of experiments. Just because a testimonial or an extra image increased the conversion rate on a certain page it doesn't necessarily hold true that the technique will work for all similar pages. Likewise it may be the nature of the individual image or the compelling copy of the testimonial that made those particular elements so effective in increasing the conversion rate; a different image or testimonial would have certainly provided different, and possibly negative, results.

Provided that you bear this in mind and design your experiments carefully with enough time to gather accurate results you will see how small refinements to your site can result in big rewards.