JZ Jeff Zych

Testing Theory: Mo' Choices, Mo' Problems

When asked, most of us would say we’d prefer more options to choose from, rather than fewer. We want to make the best possible choice, so more options should increase the likelihood we’ll choose correctly. But in actuality, research shows that more choice usually leads to worse decisions, or the abandonment of choice altogether. In this post, I will describe how we can use this knowledge to generate A/B test ideas.

Cognitive Overload

More choices are more mentally taxing to compare and evaluate, leading to cognitive overload and a decrease in decision making skills. Anecdotally, it’s the experience of walking into a supermarket to buy toothpaste, only to be confronted by an endless wall of brands and specialized types that all seem roughly the same. You’re quickly overwhelmed, and with no distinguishing characteristics to help you choose, you just grab whatever you bought last time and get the hell out of there.

This common experience was formally studied by Iyengar and Lepper (pdf) (2000), who compared buying rates when shoppers were presented 24 jams to sample, versus just 6. They found when 24 jams were available, only 3% of people bought a jar. But when only 6 jams were available, 30% bought a jar. By providing fewer jams to sample, it was easier for shoppers to compare them to each other and make a decision.

Generating Test Ideas

From these findings you can apply a simple rule to your site or mobile app to generate test ideas: any time a user has to make a choice (e.g. deciding which product to buy; clicking a navigation link; etc.), reduce the number of available options. Here are some examples:

  • Have just one call-to-action. If you have “Sign Up” and “Learn More” buttons, for example, try removing the “Learn More” button. (See below for an example).
  • Remove navigation items. For example, Amazon has been continually simplifying its homepage by hiding its store categories in favor of search. Shoppers don’t need to think about which category might have their desired item; rather, they just search for it. (For help simplifying your navigation, check out this series of articles on Smashing Magazine).
  • Try offering fewer products. See if hiding unpopular or similar products increases purchases of the few that remain.
  • If removing products isn’t feasible, try asking people to make a series of simple choices to narrow down their options. Returning to the toothpaste example, you could ask people to choose a flavor, then a type (whitening, no-fluoride, baking soda, etc.), and present toothpastes that only match those choices. The key is to make sure your customers understand each facet, and the answers are distinct and not too numerous (i.e. less than 6).
  • Break up checkout forms into discrete steps.
  • Remove navigation from checkout funnels. Many eCommerce sites (like Crate&Barrel and Amazon) do this because it leaves one option to the user — completing the purchase (see below).

"Crate&Barrel checkout flow comparison"

By removing the main navigation from their checkout flow, Crate&Barrel increased average order value by 2% and revenue per visitor by 1%.

"SeeClickFix call-to-action comparison"

By removing extraneous calls-to-action (“Free Sign Up” and “Go Pro! Free Trial”), SeeClickFix (a service for reporting neighborhood issues) focused users’ attention on the search bar and increased engagement by 8%.

Know your audience

Of course, there are times when more choice is better. Broadly speaking, experts typically know what they’re looking for, and are able to evaluate many options because they understand all the distinguishing minutia. For example, professional tennis players can rapidly narrow down the choice of thousands of racquets to just a few because they understand the difference between different materials, weights, head sizes, lengths, and so on. If you don’t offer what they’re looking for, or make it easy to get to what they want, they’ll look elsewhere. For this reason, it’s important that you understand your audience and cater to their buying habits.

We’re trained from an early age to believe that more choice is always better. But in actuality, more choices are mentally taxing, and lead to poor decision making or the abandonment of choice altogether. By testing the removal or simplification of options, you can increase sales, conversions, and overall customer satisfaction.

Further Reading

Tags: testing theory A/B testing choice sets

How Tina Fey’s “Lessons From Late Night” Apply to Product Design

In “Lessons From Late Night”, Tina Fey discusses the lessons she learned from Lorne Michaels while working at SNL. Turns out most of them are lessons I’ve learned in product design while working at Optimizely.

“Producing is about discouraging creativity.”

She goes on to say, “A TV show comprises many departments — costumes, props, talent, graphics, set dressing, transportation. Everyone in every department wants to show off his or her skills and contribute creatively to the show, which is a blessing.” But this creative energy must be harnessed and directed in a way that contributes positively to the sketch.

Applied to design, this basically means you need to say “no” a lot. Everyone’s full of ideas and will suggest things like, “Why don’t we just add a toggle so the user can choose their preference? More choice is good, right?” And then you need to explain that actually, users typically stick with the defaults, and don’t spend time configuring their software, and letting them toggle this option has all kinds of other product implications, so, no, sorry, lets try this other solution instead.

“The show doesn’t go on because it’s ready; it goes on because it’s eleven-thirty.”

This is a lesson in perfection. She elaborates:

You have to try your hardest to be at the top of your game and improve every joke until the last possible second, but then you have to let it go. You can’t be that kid standing at the top of the waterslide, overthinking it. You have to go down the chute. […] You have to let people see what you wrote. It will never be perfect, but perfect is overrated. Perfect is boring on live television.

Just change a few words to “design” and “the web,” and this applies perfectly to product design. Designers can polish forever, but perfect is the enemy of done. But unlike SNL, Optimizely (and I imagine most startups) doesn’t often have hard 11:30 PM Saturday night deadlines, which means we have a tendency to let dates slip. I used to think that was great (“I can spend time polishing!”), but I’ve found that deadlines force you to make tough decisions in order to release product (such as cutting the scope of a feature). And that extra “polish” I think I’m adding is just me overthinking decisions I’ve already made and, oh, guess what, now that actual human beings are using it we need to cut or change those corners of the UI you’ve polished because actually they don’t matter at all now that it’s being used in the real world.

“When hiring, mix Harvard nerds with Chicago improvisers and stir.”

The gist of this lesson is that diversity of thought is important when hiring. Having a variety of designers with different backgrounds and skills results in a better product.

At Optimizely, we have a mix of visual-designers-turned-UX-designers, designers formally trained in human–computer interaction and psychology (the Harvard nerds of the group, such as yours truly), and developers turned designers. We all push and learn from each other. “The Harvard guys check the logic and grammatical construction of every joke, and the improvisers teach them how to be human. It’s Spock and Kirk.” In our case, the HCI folks make sure designs are usable and don’t violate established interaction patterns, and the visual designers make sure we aren’t shipping a bunch of gray boxes.

“Never cut to a closed door.”

As applied to user experience design, this is a way of saying, “don’t leave the user at a dead end”. If users get to a screen where they can’t do anything, then you’ve lost them. Products often dump users in empty screens that have no content (we’ve made this mistake plenty at Optimizely), which lowers engagement and increases churn. Marketing pages can lack a call to action, leading to a high bounce rate. You should always provide clear next steps.

“Don’t hire anyone you wouldn’t want to run into in the hallway at three in the morning.”

This is a way of saying hire people you enjoy working with. At Optimizely, culture is a huge part of the hiring process. Work is easier, more fun, and turns out better when you’re working with people you like and respect.

Writing comedy and designing product don’t sound related, but there’s a lot of overlap in the creative process. As Tina’s lessons show, they each have a lot they can learn from each other.

Tags: product design

1-year Retrospective

August 2nd marked the 1-year anniversary of my first post, so it seems appropriate to do a quick retrospective of my first year blogging on my personal site.

Writing Stats

I’ve written 22 posts in that time, which is a rate of 1.83 per month. My (unstated) goal was 2 per month, so I wasn’t far off. My most prolific month is a tie between September 2013 and May 2014, in which I wrote 4 articles each. But in September I re-used some posts I had written previously for Optimizely, so May wins for more original content.

Sadly, there were two months in which I didn’t write any articles: Dec 2013, and July 2014. In December I was in India, so that’s a pretty legitimate reason. July, however, has no good reason. It was generally a busy month, but I should have made time to write at least one post. And looking closer, just saying “July” makes it sound better than it actually was - I had a seven week stretch of no posts then!

My longest article was “Re-Designing Optimizely’s Preview Tool”, clocking in at 4,158 words!

Site Analytics

Diving into Google Analytics, I’ve had 3,092 page views, 2,158 sessions, and 1,778 users. I seem to get a steady trickle of traffic every day, with a few occasional spikes in activity (which are caused by retweets, Facebook posts, or sending posts to all of Optimizely). All of which I find pretty surprising since I don’t write very regularly, and I don’t do much to actively seek readers.

Google Analytics stats for the past year

So where do these visitors (i.e. you) come from? Google Analytics tells me that, even more surprisingly, the top two sources are organic search and direct, respectively. From looking through the search terms used to find me, they can be grouped into three categories:

  • My name: this is most likely people who are interviewing at Optimizely.
  • Cloud.typography and Typekit comparison: people are interested in a performance comparison of these two services. And in fact, I wrote this article precisely because I was searching for that information myself, but there weren’t any posts about it.
  • Framing messages: I wrote a post about the behavioral economics principle of framing, and how you can use it to generate A/B testing ideas. Apparently people want help writing headlines!

Top Posts

Continuing to dig into Google Analytics, these are my three most popular posts:

  1. “Extend – SASS’s Awkward Stepchild”, with 354 page views.
  2. “Re-Designing Optimizely’s Preview Tool”, with 306 page views.
  3. “Performance comparison of serving fonts through Typekit vs Cloud.typography”, with 302 page views.

They’re all pretty close in terms of traffic, but quite different in terms of content. So what does this tell me about what’s resonating with you, the reader, and what I should continue doing going forward? The main commonality is that all of those articles are original, in-depth content. In fact, this holds true past the top 10. My shorter posts that are responses to other people’s posts don’t receive as much mind share. I’ll have to think more about whether they’re worth doing at all anymore.


Overall I’m pretty satisfied with those numbers, and the content I’ve been able to produce. Going forward I hope I can write more in-depth content, especially about the design process of my projects (which are my favorite to write). Here’s to the upcoming year!

Tags: retrospective

DRY isn't the One True Principle of CSS

Ben Frain wrote a great article of recommendations for writing long-living CSS that’s authored by many people (e.g. CSS for a product). The whole thing is worth reading, but I especially agree with his argument against writing DRY (Don’t Repeat Yourself) code at the expense of maintainability. He says:

As a concrete example; being able to delete an entire Sass partial (say 9KB) in six months time with impunity (in that I know what will and won’t be affected by the removal) is far more valuable to me than a 1KB saving enjoyed because I re-used or extended some vague abstracted styles.

Essentially, by abstracting styles as much as possible and maximizing re-use, you’re creating an invisible web of dependencies. If you ever want to change or remove styles, you will cause a ripple effect throughout your site. Tasks that should have been 30 or 60 minutes balloon into multi-hour endeavors. As Ben says, being able to delete (or modify) with impunity is far more valuable than the small savings you get by abstracting your code.

I have made this mistake many times in my career, and it took me a long time to distinguish between good and bad code reuse. The trick I use is to ask, “If I were to later change the style of this component, would I want the others that depend on it to be updated, too?” More often than not, the answer is no, and the styles should be written separately. It took some time for me to be comfortable having duplicate CSS in order to increase maintainability.

Another way of thinking about this is to figure out why two components look the same. If it’s because one is a modification of a base module (e.g. a small version of a button), it’s a good candidate for code reuse. If not (e.g. a tab that looks similar to a button, but behaves differently), then you’re just setting yourself up for a maintainability nightmare.

As beneficial as DRY code is, it isn’t the One True Principle of CSS. At best, it saves some time and bytes, but at worst, it’s a hindrance to CSS maintainability.

View all posts »