JZ Jeff Zych

Behind the Design: Optimizely's Mobile Editor

On 11/18/14, Optimizely officially launched A/B testing for iOS apps. This was a big launch because our product had been in beta for months, but none of us felt proud to publicly launch it. To get us over the finish line, we focused our efforts on building out an MVPP — a Minimum Viable Product we’re Proud of (which I wrote about previously). A core part of the MVPP was redesigning our editing experience from scratch. In this post, I will walk you through the design process, show you the sketches and prototypes that led up to the final design, and the lessons learned along the way, told from my perspective as the Lead Designer.

A video of the final product

Starting Point

To provide context, our product enables mobile app developers to run A/B tests in their app, without needing to write any code or resubmit to the App Store for approval. By connecting your app to our editor, you can select elements, like buttons and headlines, and change their properties, like colors and text. Our beta product was functional in this regard, but not particularly easy or delightful to use. The biggest problem was that we didn’t show you your app, so you had to select elements by searching through a list of your app’s views (a process akin to navigating your computer’s folder hierarchy to find a file). This made the product cumbersome to use, and not visually engaging (see screenshot below).

Screenshot of Optimizely's original iOS editor

Optimizely’s original iOS editor.

Designing the WYSIWYG Editor

To make this a product we’re proud to launch, it was obvious we’d need to build a What-You-See-Is-What-You-Get (WYSIWYG) editor. This means we’d show the app in the browser, and let users directly select and edit their app’s content. This method is more visually engaging, faster, and easier to use (especially for non-developers). We’ve had great success with web A/B testing because of our WYSIWYG editor, and we wanted to replicate that success on mobile.

This is an easy design decision to make, but hard to actually build. For this to work, it had to be performant and reliable. A slow or buggy implementation would have been frustrating and a step backwards. So we locked a product designer and two engineers in a room to brainstorm ideas and build functional prototypes together. By the end of the week, they had a prototype that cleared the technical hurdles and proved we could build a delightful editing experience. This was a great accomplishment, and a reminder that any challenge can be solved by giving a group of smart, talented individuals space to work on a seemingly intractable problem.

Creating the Conceptual Model

With the app front and center, I needed an interface for how users change the properties of elements (text, color, position, etc.). Additionally, there are two other major features the editor needs to expose: Live Variables and Code Blocks. Live Variables are native Objective-C variables that can be changed on the fly through Optimizely (such as the price of items). Code Blocks let users choose code paths to execute (for example, a checkout flow that has 2 steps instead of 3).

Before jumping into sketches or anything visual, I had to get organized. What are all the features I need to expose in the UI? What types of elements can users edit? What properties can they change? Which of those are useful for A/B tests? I wrote down all the functionality I could think of. Additionally, I needed to make sure the UI would accommodate new features to prevent having to redesign the editor 3 months down the line, so I wrote out potential future functionality alongside current functionality.

I took all this functionality and clustered them into separate groups. This helped me form a sound conceptual model on which to build the UI. A good model makes it easier for users to form an accurate mental model of the product, thus making it easier to use (and more extensible for future features). This exercise made it clear to me that there are variation-level features, like Code Blocks and Live Variables, that should be separate from element-level features that act on specific elements (like changing a button’s text). This seems like an obvious organizing principle in retrospect, but at the time it was a big shift in thinking.

After forming the conceptual model, I curated the element properties we let users edit. The beta product exposed every property we could find, with no thought as to whether or not we should let users edit it. More properties sounds better and makes our product more powerful, but it comes at the cost of ease of use. Plus, a lot of the properties we let people change don’t make sense for our use case of creating A/B tests, and don’t make sense to non-developers (e.g. “Autoresizing mask” isn’t understandable to non-technical folks, or something that needs to be changed for an A/B test).

I was ruthless about cutting properties. I went through every single one and asked two questions: first, is this understandable to non-developers (my definition of “understandable” being would a person recognize it from common programs they use everyday, like MS Office or Gmail); and second, why is this necessary for creating an A/B test? If I was unsure about an attribute, I defaulted to cutting it. My reasoning was it’s easy to add features to a product, but hard to take them away. And if we’re missing any essential properties, we’ll hear about it from our customers and can add it back.

Screenshot of my Google Doc feature organization

My lo-fi Google Doc to organize features

Let the Sketching Begin!

With my thoughts organized, I finally started sketching a bunch of editor concepts (pictured below). I had two big questions to answer: after selecting an element, how does a user change its properties? And, how are variation-level features (such as Code Blocks) exposed? My top options were:

  • Use a context menu of options after selecting an element (like our web editor)
  • When an element is selected, pop up an inline property pane (ala Medium’s and Wordpress’s editors)
  • Have a toolbar of properties below the variation bar
  • Show the properties in a drawer next to the app

Picture of my toolbar sketch

A sketch of the toolbar concept

Picture of my inline formatting sketch

A messy sketch of inline formatting options (specifically text)

Picture of one of my drawer sketches

One of the many drawer sketches

Interactive Prototypes

Each approach had pros and cons, but organizing element properties in a drawer showed the most promise because it’s a common interaction paradigm, it fit easily into the editor, and was the most extensible for future features we might add. The other options were generally constraining and better suited to limited functionality (like simple text formatting).

Because I wanted to maximize space for showing the app, my original plan was to show variation-level features (e.g. Code Blocks; Live Variables) in the drawer when no element was selected, and then replace those with element-level features when an element was selected. Features at each level could be separated into their own panes (e.g. Code Blocks would have its own pane). Thus the drawer would be contextual, and all features would be in the same spot (though not at the same time). This left plenty of space for showing an app, and kept the editor uncluttered.

A sketch told me that layout-wise this plan was viable, but would it make sense to select an element one place, and edit its properties in another? Would it be jarring to see features come and go depending on whether an element was selected or not? How will you navigate between different panes in the drawer? To answer these questions, an interactive prototype was my best course of action (HTML/CSS/JS being my weapon of choice).

Screenshot of an early drawer prototype

An early drawer prototype. Pretend there’s an app in that big empty white space.

I prototyped dozens of versions of the drawer, and shopped them around to the team and fellow designers. Responses overall were very positive, but the main concern was that the tab buttons (“Text”, “Layout”, etc., in the image above) in the drawer won’t scale. Once there are more than about 4, the text gets really squeezed (especially in other languages), stunting our ability to add new features. One idea to alleviate this, suggested by another designer, was to use an accordion instead of tab buttons to reveal content. A long debate ensued about which approach was better. I felt the tab buttons were a more common approach (accordions were for static content, not interactive forms that users will be frequently interacting with), whereas he felt the accordion was more scalable by allowing room for adding more panes, and accommodates full text labels (see picture below).

Screenshot of the drawer with accordion prototype

Drawer with accordion prototype. Pretend that website is an iOS app.

To help break this tie, I built another prototype. After playing around with both for awhile, and gathering feedback from various members of the team, I realized we were both wrong.

Hitting reset

After weeks of prototyping and zeroing in on a solution, I realized it was the wrong solution. And the attempt to fix it (accordions), was in fact an iteration of the original concept that didn’t actually address the real problem. I needed a new idea that would be superior to all previous ideas. So I hit reset and went back to the drawing board (literally). I reviewed my initial organizing work and all required functionality. Clearly delineating variation-level properties from element-level properties was a sound organizing principle, but the drawer was getting overloaded by having everything in it. So I explored ways of more cleanly separating variation-level properties from element-level properties.

After reviewing my feature groupings, I realized there aren’t a lot of element properties. They can all be placed in one panel without needing to navigate between them with tabs or accordions at all (one problem solved!).

The variation properties were the real issue, and had the majority of potential new features to account for. Two new thoughts became apparent as I reviewed these properties: first, variation-level changes are typically quick and infrequent; and second, variation-level changes don’t typically visually affect the app content. Realizing this, I hit upon an idea to have a second drawer that would slide out over the app, and go away after you made your change.

To see how this would feel to use, I made yet another interactive prototype. This new UI was clean, obviated the need for tab buttons or accordions, was quick and easy to interact with, and put all features just a click or two away. In short, this new design direction was a lot better, and everyone quickly agreed it made more sense than my previous approach.

Reflecting back on this, I realize I had made design decisions based on edge cases, rather than focusing on the 80% use case. Starting the design process over from first principles helped me see this much more clearly. I only wish I would have caught it sooner!

Admitting this design was not the right solution, after a couple months of work, and after engineers already began building it, was difficult. The thought of going in front of everyone (engineers, managers, PMs, designers, etc.) and saying we needed to change direction was not something I was looking forward to. I was also worried about the amount of time it would take me to flesh out a completely new design. Not to mention that I needed to thoroughly vet it to make sure that it didn’t have any major drawbacks (I wouldn’t have another opportunity to start over).

Luckily, once I started fleshing out this new design, those fears mostly melted away. I could tell this new direction was stronger, which made me feel good about restarting, which made it easier to sell this idea to the whole team. I also learned that even though I was starting over from the beginning, I wasn’t starting with nothing. I had learned a lot from my previous iterations, which informed my decision making this second time through.

Build and Ship!

With a solid design direction finally in place, we were able to pour on the engineering resources to build out this new editor. Having put a lot of thought into both the UI and technical challenges before writing production code, we were able to rapidly build out the actual product, and ended up shipping a week ahead of our self-imposed deadline!

Screenshot of the finished mobile editor

The finished mobile editor

Lessons Learned

  • Create a clear conceptual model on which to build the UI. A UI that accurately represents the system’s conceptual model will make it easy for users to form a correct mental model of your product, thus making it easier to use. To create the system model, write down all the features, content, and use cases you need to design for before jumping into sketches or prototypes. Group them together and map out how they relate to each other. From this process, the conceptual model should become clear. Read more about mental models on UX Magazine.
  • Don’t be afraid to start over. It’s scary, and hard, and feels like you wasted a bunch of time, but the final design will come out better. And the time you spent on the earlier designs wasn’t wasted effort — it broadened your knowledge of both the problem and solution spaces, which will help you make better design decisions in your new designs.
  • Design for the core use case, not edge cases. Designing for edge cases can clutter a UI and get in the way of the core use case that people do 80% of the time. In the case of the drawer, it led to overloading it with functionality.
  • Any challenge can be solved by giving a group of smart, talented individuals space to work on seemingly intractable problems. We weren’t sure a WYSIWYG editor would be technically feasible, but we made a concerted effort to overcome the technical hurdles, and it payed off. I’ve experienced this time and time again, and this was yet another reminder of this lesson.

On 11/18/14, the team was proud to announce Optimizely’s mobile A/B testing product to the world. Week-over-week usage has been steadily rising, and customer feedback has been positive, with people saying the new editor is much easier and faster to use. This was a difficult product to design, for both technical and user experience reasons, but I had a great time doing it and learned a ton along the way. And this is only the beginning — we have a lot more work to do before we’re truly the best mobile A/B testing product on the planet.

Tags: design process Optimizely

Building an MVPP - A Minimum Viable Product we're Proud of

On November 18th, 2014, we publicly released Optimizely’s iOS editor. This was a big release for us because it marked the end of a months-long public beta in which we received a ton of customer feedback and built a lot of missing features. But before we launched, there was one problem the whole team rallied behind to fix: we weren’t proud of the product. To fix this issue, we went beyond a Minimum Viable Product (MVP) to an MVPP — the Minimum Viable Product we’re Proud of.

What follows is the story of how we pulled this off, what we learned along the way, and product development tips to help you ship great products, from the perspective of someone who just did it.

Finished iOS editor

The finished iOS editor.

Genesis of the MVPP

We released a public beta of Optimizely’s iOS editor in June 2014. At that time, the product wasn’t complete yet, but it was important for us to get real customer feedback to inform its growth and find bugs. So after months of incorporating user feedback, the beta product felt complete enough to publicly launch. There was just one problem: the entire team wasn’t proud of the product. It didn’t meet our quality bar; it felt like a bunch of features bolted together without a holistic vision. To fix this, we decided to overhaul the user experience, an ambiguous goal that could easily go on forever, never reaching a clear “done” state.

We did two things to be more directed in the overhaul. First, we committed to a deadline to prevent us from endlessly polishing the UI. Second, we took inspiration from the Lean Startup methodology and chose a set of features that made up a Minimum Viable Product (MVP). An MVP makes it clear that we’ll cut scope to make the deadline, but nothing about quality. So to make it explicit that we were focusing on quality and wanted the whole team to be proud of the final product, we added an extra “P” to MVP. And thus, the Minimum Viable Product we’re Proud of — our MVPP — was born.

Create the vision

Once we had agreed on a feature set for the MVPP, a fellow Product Designer and I locked ourselves in a war room for the better part of a week to flesh out the user experience. We mapped out user flows and created rough mock ups that we could use to communicate our vision to the larger development team. Fortunately, we had some pre-existing usability test findings to inform our design decisions.

Picture of our war room in action

Sketches, mockups, and user flows from our war room.

These mockups were immensely helpful in planning the engineering and design work ahead. Instead of talking about ideas in the abstract, we had concrete features and visuals to point to. For example, everyone knew what we meant when we said “Improved Onboarding Flow.” With mockups in hand, communication between team members became much more concrete and people felt inspired to work hard to achieve our vision.

Put 6 weeks on the clock… and go!

We had 3 sprints (6 weeks) to complete the MVPP (most teams at Optimizely work in 2 week cycles called “sprints”). It was an aggressive timeline, but it felt achievable — exactly where a good deadline should be.

In the first sprint, the team made amazing progress. All the major pieces had been built, without any major re-scoping or redesigns. There were still bugs to fix, polish to apply, and edge cases to consider, but the big pieces core to our vision were in place.

That momentum carried over into the second sprint, which we spent fixing the biggest bugs, filling functional holes, and polishing the UI.

For the third and final sprint, we gave ourselves a new goal: ship a week early. We were already focused on launching the MVPP, but at this point we became laser focused. During daily standups, we looked at our JIRA board and asked, “If we were launching tomorrow, what would we work on today?”

We were ruthless about prioritizing tasks and moved a lot of items that were important, but not launch-critical, to the backlog.

During the first week of sprint 3, we also did end-to-end product walkthroughs after every standup to ensure the team was proud of the new iOS editor. We all got to experience the product from the customer’s perspective, and caught user experience bugs that were degrading the quality of our work. We also found and fixed a lot of functional bugs during this time. By the end of the week, everyone was proud of the final product and felt confident launching.

The adrenaline rush & benefit of an early release

On 11/10, we quietly released our MVPP to the world — a full week early! Not only did shipping early feel great, it also gave us breathing room to further polish the design, fix bugs, and give the rest of the company time to organize all the components to launch the MVPP.

Product teams don’t launch products alone; it takes full collaboration between marketing, sales, and success to create materials to promote it, sell it, and enable our customers to use it. By the time the public announcement on 11/18 rolled around, the whole company was extremely proud of the final result.

Lessons learned

While writing this post and reflecting on the project as a whole, a number of techniques became clear to me that can help any team ensure a high quality, on-time launch:

  • Add a “P” to “MVP” to make quality a launch requirement: Referring to the project as the “Minimum Viable Product we’re Proud of” made sure everyone on the team approached the product with quality in mind. Every project has trade-offs between the ship date, quality, and scope. It’s very hard to do all three. Realistically, you can do two. By calling our project an MVPP, we were explicit that quality would not be sacrificed.
  • Set a deadline: Having a deadline focused everyone’s efforts, preventing designers from endlessly polishing interfaces and developers spinning their wheels imagining every possible edge case. Make it aggressive, yet realistic, to instill a sense of urgency in the team.
  • Focus on the smallest set of features that provide the largest customer impact: We were explicit about what features needed to be redesigned, and just as importantly, which were off limits. This prevented scope-creep, and increased the team’s focus.
  • Make mockups before starting development: This is well-known in the industry, but it’s worth repeating. Creating tangible user flows and mockups ahead of time keeps planning discussions on track, removes ambiguity, and quickly explains the product vision. It also inspires the team by rallying them to achieve a concrete goal.
  • Do daily product walkthroughs: Our product walkthroughs had two key benefits. First, numerous design and code bugs were discovered and fixed. And second, they ensured we lived up to the extra “P” in “MVPP.” Everyone had a place to verbally agree that they were proud of the final product and confident launching. Although these walkthroughs made our standups ~30 minutes longer, it was worth the cost.
  • Ask: “If we were shipping tomorrow, what would you work on today?”: When the launch date is approaching, asking this question separates the critical, pre-launch tasks from the post-launch tasks.

Lather, Rinse, and Repeat

By going beyond an MVP to a Minimum Viable Product we’re Proud of, we guaranteed that quality was the requirement for launching. And by using a deadline, we stayed focused only on the tasks that were absolutely critical to shipping. With a well-scoped vision, mockups, and a date not too far in the future, you too can rally teams to create product experiences they’re proud of. And then do it again.

Tags: design process

Why I Became a Design Manager

At the start of 2015, I officially became a Design Manager at Optimizely. I transitioned from an individual contributor, product design role into a leadership role. For the entirety of my career up to this point, I had never planned on going into management. Doing hands-on, nitty-gritty design and coding work seemed so much more exciting and fulfilling. Management sounded more like a necessary evil than an interesting challenge. But since then, my thinking has changed. So why did I become a Design Manager?

When I joined Optimizely in September of 2012, the design team was just 2 people. I made it 3 in early 2013 when my role moved from engineering to design. And at the end of 2014, we were 16. (As of this writing we’re at 21!). So we’ve seen tremendous growth, and I’ve been present for all of it. And throughout this time, there had only been 1 manager for the entire team, which was not healthy for my manager or the team. The possibility of managing had been floated by me in mid-2014, but I wasn’t interested. I had been a Product Designer for less than a year and wasn’t ready to move on yet. I felt like I had just started hitting my stride as a designer, and wanted to continue honing my craft. I recognized the need for another manager, but I didn’t want it to be me.

As more designers joined Optimizely, I began taking on more managerial tasks. I also saw more issues rising within the team that our manager didn’t always have time to address. So in short, the importance of this role became more apparent, and the day-to-day work of the role became more real to me.

But the real turning point came when my manager went on vacation. In his absence, I was the go-to person for all of the team’s needs. I suspended most of my design work for this period, and really got a taste of what it would be like to work as a full-time manager. I started asking myself, “What if this was my full-time role? Would I enjoy it?” I went back and forth in my head quite a bit. The idea of leaving behind design and code was both scary and saddening. I had so much I was still looking forward to building! Plus, as we all know, change is hard.

But the team has more needs than our lone manager can handle. And I care deeply about the team, so for the greater good I decided it was time to step up. I realized that by helping my team be as great as possible, I would have a bigger impact on the company. And by working closer with engineering managers and PMs, I would have a bigger impact on the product. I’d be getting out of the weeds of day-to-day design to work on the product from a higher perspective across individuals and teams. The impact is less direct, but broader. All of this sounded tremendously exciting to me, and more impactful than individual contributor work.

I also realized the things I love about design (problem solving, ideating, etc.) would still be present in my new role. But instead of applying those skills to concrete visual interfaces, I would apply them to abstract team and personnel issues. I’d be using the design process to solve a different set of problems.

So when my manager got back from vacation, I told him my decision and we started transitioning me into a managerial role. As of the start of 2015, I’ve been managing full time and loving it. I’m still a bit sad to leave design work behind, and worry about my skills atrophying, but I look forward to the new challenges that await. It was a difficult decision that took me a long time to come around to, but I’m excited to make the team, product, and company as great as possible.


Warm Gun 2014 Conference Notes

This year’s Warm Gun conference was great, just like last year’s. This year generally seemed to be about using design to generate and validate product insights, e.g. through MVPs, prototyping, researching, etc.

Kickoff (Jared Spool)

Jared Spool’s opening talk focused mostly on MVPs and designing delight into products. To achieve this, he recommended the Kano Model and Dana Chisnell’s Three Approaches to Delight (adding pleasure e.g. through humorous copy, improving the flow, and meaning e.g. believing in a company’s mission [hardest to achieve]).

You’re Hired! Strategies for Finding the Perfect Fit (Kim Goodwin)

This was a great talk about how to hire a great design team, which is certainly no easy task (as I’ve seen at Optimizely).

  • Hiring is like dating on a deadline – after a couple of dates, you have to decide whether or not to marry the person!
  • You should worry more about missing the right opportunity, rather than avoiding the wrong choice

5 Lessons She’s Learned

  1. Hire with a long-term growth plan in mind
    • Waiting until you need someone to start looking is too late (it takes months to find the right person)
    • What kind of will you need? Do you want generalists (can do everything, but not great at any one thing; typically needed by small startups) or specialists (great at one narrow thing, like visual design; typically needed by larger companies)
    • Grow (i.e. mentor junior people) or buy talent? Training junior people isn’t cheap - it takes a senior person’s time.
      • A healthy team has a mix of skills levels (she recommends a ratio of 1 senior:4 mid:2 junior). (Optimizely isn’t far off – we mostly lack a senior member!)
    • A big mistake she sees a lot of startups make is to hire too junior of a person too early
  2. Understand the Market
    • The market has 5 cohorts: low skill junior folks (think design is knowing tools); spotty/developing skills; skilled specialists; generalists; and team leads
    • Senior == able to teach others, NOT a title (there’s lots of title inflation in the startup industry). Years of experience does NOT make a person senior. Many people with “senior” in their title have holes in their skills (especially if they’ve only worked on small teams at startups)
    • 5 years experience only at a design consultancy == somewhat valuable (lots of mentorship opportunities from senior folks, but lack working continuously/iteratively on a product)
    • 5 years experience mixed at design consultancy and on an in-house team == best mix of skills (worked with senior folks, and on a single product continuously)
    • 5 years only on small startup teams == less valuable than the other two; is a red flag. There are often holes in the skills. They’re often “lone rangers” who haven’t worked with senior designers, or a team, and probably developed bad habits. Often have an inflated self-assessment of their skills and don’t know how to take feedback. (uh-oh! I’m kind of in this group)
    • It takes craft to assess craft
    • Alternate between hiring leads and those who can learn from the leads (i.e. a mix of skill levels)
    • Education matters - school can fill in gaps of knowledge. Schools have different types of people they product (HCI, visual, etc.). (yay for me!)
  3. Make 2 lists before posting the job
    • First, list the responsibilities a person will have (what will they do?)
    • Second, list the skills they need to achieve the first list.
    • Turn these 2 lists into a job posting (avoid listing tools in the hiring criteria - that is dumb)
    • DON’T look for someone who has experience designing your exact product in your industry already. The best designers can quickly adapt to different contexts (better to hire a skilled designer with no mobile experience than a junior/mid designer with ONLY mobile experience - there’s ramp-up time, but that’s negligible for a skilled designer)
    • Junior to senior folks progress through this: Knows concepts -> Can design effectively w/coach -> Can design solo -> Can teach others
    • On small/inexperienced teams, watch out for the “Similar to me” effect. Designers new to hiring/interviewing will evaluate people against themselves, rather than evaluate their actual skills or potential. (Can ask, “Where were you relative to this person when you were hired?” to combat this).
  4. Evaluate Based on Skills You Need
    • Resumes == bozo filter
    • Look at the goals, process, role, results, lessons learned, things they’d do differently (we’re pretty good at this at Optimizely!)
    • Do “behavioral interviewing”, i.e. focus on specifics of actual behavior. Look at their actual work, do design exercises, ask “Tell me about a time when…”. It’s a conversation, not an interrogation. (Optimizely is also good at this!)
  5. Be Real to Close the Deal
    • Be honest about what you’re looking for
    • If you have doubts about a person, discuss them directly with the candidate to try overcome them (or confirm them). (We need to do this more at Optimizely)

Product Strategy in a Growing Company (Des Traynor)

This was one of my favorite talks. Product strategy is hard, and it’s really easy to say “Yes” to every idea and feature request. One of my favorite bits was you need to say “No” because somethings not in the product vision. If you don’t ever say this, then you have no product vision. (This has been a challenge at Optimizely at times).

  • Software is eating the world!
  • We’re the ones who control the software that’s available to everyone.
  • Niece asked, “Where do products come from?”. There are 5 reasons a product is built:
    1. Product visionary
    2. Customer-focused (built to solve pain point(s))
    3. Auter (art + business)
    4. Scratching own itch (you see a need in the marketplace)
    5. Copy a pattern (e.g. “Uber for X!”)
  • (Optimizely started as scratching own itch, but is adapting to customer-focused)
  • Scope: scalpel vs. swiss army knife
    • When first starting, build a scalpel (it’s the only way to achieve marketshare when starting)
    • Gall’s Law: complex systems evolve from simple ones (like WWW. Optimizely is also evolving towards complexity!). You can’t set out to design a complex system from scratch (think Google Wave)
  • A simple product !== making a product simple (i.e. an easy to use product isn’t necessarily simple [difference between designing a whole UX vs. polishing a specific UI]).
    • Simplify a product by removing steps. Watch out for Scopi-locks (i.e. scope the problem just right – not too big, not too small). You don’t want to solve steps of a flow that are already solved by a market leader, or when there are multiple solutions already in use by people (e.g. don’t rebuild email, there’s already Gmail and Outlook and Mailbox, etc.)
  • How to fight feature creep
    • Say “No” by default
    • Have a checklist new ideas must go through before you build them, e.g. (this is a subset):
      • Does it benefit all customers?
      • Is the value worth the effort?
      • Does it improve existing features? Does it increase engagement across the system, or divide it?
      • If a feature takes off, can we afford it? (E.g. if you have a contractor build an Android app, how will you respond to customer feedback and bugs?)
      • Is it low effort for the customer to use, and result in high value? (E.g. Circles in G+ fail this criteria - no one wants to manage friends like this)
      • It’s been talked about forever; it’s easy to build; we’ve put a lot of effort in already == all bad reasons to ship a new feature
    • (Optimizely has failed at this a couple of times. E.g. Preview As. On the other hand, Audiences met these criteria)
    • Once you ship something, it’s really hard to take back. Even if customers complain about it, there is always a minority that will be really angry if you take it away.

Hunches, Instincts, and Trusting Your Gut (Leah Buley)

This was probably my least favorite talk. The gist of it is that as a designer, there are times you need to be an expert and just make a call using your gut (colleagues and upper management need you to be this person sometimes). We have design processes to follow, but there are always points at which you need to make a leap of faith and trust your gut. I agree with those main points, but this talk lost me by focusing only on visual design. She barely mentioned anything about user goals or UX design.

Her talk was mainly about “The Gut Test”, which is a way of looking at a UI (or print piece, or anything that has visual design) and assessing your gut reaction to it. This is useful for visual design, but won’t find issues like, “Can the user accomplish their goal? Is the product/feature easy to use?” (Something can be easy to use, but fail her gut test). It’s fine that she didn’t tackle these issues, but I wish she would have acknowledged more explicitly that the talk was only addressing a thin slice of product design.

  • In the first 50ms of seeing something, we have an immediate visceral reaction to things
  • Exposure effect: the more we see something, the more we like it (we lose our gut reaction to it)
  • To combat this, close your eyes for 5 seconds, then look at a UI and ask these questions:
    • What do you notice first?
    • How does it make you feel (if anything)? What words would you use to describe it?
    • Is it prototypical? (i.e. does it conform to known patterns). Non-conformity == dislike and distrust
  • Then figure out what you can do to address any issues discovered.

Real Life Trust Online (Mara Zepeda)

This talk was interesting, but not directly applicable to my everyday job at Optimizely. The gist of it was how do we increase trust in the world, and not just in the specific product or company we’re using? For example, when you buy or sell something successfully on Craigslist, your faith in humanity increases a little bit. But reviews on Amazon, for example, increases your trust in that product and Amazon, but not necessarily in your fellow human beings.

  • Before trust is earned, there’s a moment of vulnerability and an optimism about the future.
  • Trust gets built up in specific companies (e.g. Craigslist - there’s no reason to trust the company or site, but trust in humans and universe increases when a transaction is successful).
  • Social networks don’t create networks of trust in the real world
  • Switchboard MVP was a phone hotline
    • LinkedIn: ask for job connections, no one responds. But if you call via Switchboard, people are more likely to respond (there’s a real human connection)
    • They’re trying to create a trust network online
  • To build trust:
    • Humanize the transaction (e.g. make it person to person)
    • Give a victory lap (i.e. celebrate when the system works)
    • Provide allies / mentors along the journey (i.e. people who are invested in the journey being successful, and can celebrate the win)
  • She brought up the USDA’s “Hay Net” as an example of this. It connects those who have Hay with those who need Hay (and vice versa). UI had two options: “Have Hay” and “Need Hay”, which I find hilarious and amazing.

Designing for Unmet Needs (Steve Portigal)

Steve Portigal’s talk was solid, but it didn’t really tell me anything I didn’t already know. The gist of it was there are different types of research (generative v. evaluative), you need to know which is appropriate for your goals (although it’s a spectrum, not a dichotomy), and there are ways around anything you think is stopping you (e.g. no resources; no users; no product; etc.). The two most interesting points to me were:

  • Create provocative examples/prototypes/mocks to show people and gather responses (he likened this to a scientist measuring reactions to new stimuli). Create a vision of the future and see what people think of it, find needs, iterate, adapt. Go beyond iterative improvements to an existing product or UI (we’re starting to explore this technique at Optimizely now).
  • Build an infrastructure for ongoing research. This is something that’s been on my mind for awhile, since we’re very reactive in our research needs at Optimizely. I’d like us to have more continual, ongoing research that’s constantly informing product decisions.

Redesigning with Confidence (Jessica Harllee)

This was a cool talk that described the process Etsy went through to redesign the seller onboarding experience, and how they used data to be confident in the final result. The primary goal was increasing the shop open rate, while maintaining the products listed per seller. They a/b tested a new design that increased the open rate, but had fewer products listed per seller. They made some tweaks, a/b tested again, and found a design that increased the shop open rate while maintaining the number of products listed per seller. Which means more products are listed on Etsy overall!

I didn’t learn much new from this talk, but enjoy hearing these stories. It also got me thinking about how we don’t a/b test much in the product at Optimizely. A big reason is because it takes too long to get significant results (as Jessica mentioned in her talk, they had to run both tests for months, and the overall project took over a year). Another reason is that when developing new features, there aren’t any direct metrics to compare. Since Jessica’s example was a redesign, they could directly compare behavior of the redesign to the original.

Designing for Startups Problems (Braden Kowitz)

Braden’s talk was solid, as usual, but since I’ve seen him talk before and read his blog I didn’t get much new out of it. His talk was about how Design (and therefore, designers) can help build a great company (beyond just UIs). Most companies think of design at the “surface level”, i.e. visual design, logos, etc., but at its core design is about product and process and problem solving. Design can help at the highest levels.

  • 4 Skills Designers Need:
    1. Know where to dig
      • Map the problem
      • Identify the riskiest part (e.g. does anyone need this product or feature at all?)
      • Design to answer that question. Find the cheapest/simplest/fastest thing you can create to answer this question (fake as much as you can to avoid building a fully working artifact)
    2. Get dirty
      • Prototype as quickly as possible (colors, polish, etc., aren’t important)
      • Focus on the most important part, e.g. the user flow, layout, copy, etc. Use templates/libraries to save time
      • Use deadlines (it’s really easy to polish a prototype forever)
    3. Pump in fresh data
      • Your brain fills in gaps in data, so have regular research and testing (reinforces Portigal’s points nicely)
    4. Take big leaps
      • Combine the above 3 steps to generate innovative solutions to problems

Accomplish Big Goals with Objective & Key Results (Christina Wodtke)

This was an illuminating talk about the right way to create and evaluate OKRs. I didn’t hear much I hadn’t already heard (we use OKRs at Optimizely and have discussed best practices). But to recap:

  • Objective == Your Dream, Your Goal. It’s hard. It’s qualitative.
  • Key Results == How you know you reached your goal. They’re quantitative. They’re measurable. They’re not tasks (it’s something you can put on a dashboard and measure over time, e.g. sales numbers, adoption, etc.).
  • Focus: only have ONE Objective at a time, and measure it with 3 Key Results. (She didn’t talk about how to scale this as companies get bigger. I wish she did).
  • Measure it throughout the quarter so you can know how you’re tracking. Don’t wait until the end of the quarter.

Thought Experiments for Data-Driven Design (Aviva Rosenstein)

This was an illuminating talk about the right way to incorporate data into the decision making process. You need to find a balance between researching/measuring to death, and making a decision. She used DocuSign’s feedback button as a good example of this.

  • Don’t research to death — try something and measure the result (but make an educated guess).
  • DocuSign tried to roll their own “Feedback” button (rather than using an existing service). They gave the user a text box to enter feedback, and submitting it sent it to an email alias (not stored anywhere; not categorized at all).
    • This approach became a data deluge
    • There was no owner of the feedback
    • Users entered all kinds of stuff in that box that shouldn’t have gone there (e.g. asking for product help). People use the path of least resistance to get what they want. (I experienced this with the feedback button in the Preview tool)
  • Data should lead to insight (via analysis and interpretation)
  • Collecting feedback by itself has no ROI (can be negative because if people feel their feedback is being ignored they get upset)
  • Aviva’s goal: find a feedback mechanism that’s actually useful
  • Other feedback approaches:
    • Phone/email conversation (inefficient, hard to track)
    • Social media (same as above; biased)
    • Ad hoc survey/focus groups (not systematic; creating good surveys is time consuming)
  • Feedback goals:
    1. Valid: trustworthy and unbiased
    2. Delivery: goes to the right person/people
    3. Retention: increase overall company knowledge; make it available when needed
    4. Usable: can’t pollute customer feedback
    5. Scalable: easy to implement
    6. Contextual: gather feedback in context of use
  • They changed their feedback mechanism slightly by asking users to bucket the feedback first (e.g. “Billing problems”, “Positive feedback”, etc.), then emailed it to different places. This made it more actionable.
  • Doesn’t need to be “Ready -> Fire -> Aim”: we can use data and the double diamond approach to inform the problem, and make our best guess.
    • This limits collateral damage from not aiming. A poorly aimed guess can mar the user experience, which users don’t easily forget.

Growing Your Userbase with Better Onboarding (Samuel Hulick)

This was one of my favorite talks of the day (and not only because Samuel gave Optimizely a sweet shout out). I didn’t learn a ton new from it, but Samuel is an entertaining speaker. His pitch is basically that the first run experience is important, and needs to be thought about at the start of developing a product (not tacked on right before launch).

  • “Onboarding” is often just overlaying an UI with coach’s marks. But there’s very little utility in this.
  • Product design tends to focus on the “flying” state, once someone is using a system. Empty states, and new user experiences, are often tacked on.
  • You need to start design with where the users start
  • Design Recommendations
    • Show a single, action-oriented tooltip at a time (Optimizely was his example of choice here!)
      • Ask for signup when there’s something to lose (e.g. after you’ve already created an experiment)
      • Assume guided tours will be skipped, i.e. don’t rely on them to make your product usable
    • Use completion meters to get people fully using a product
    • Keep in mind that users don’t buy products, they buy better versions of themselves (Mario + fire flower), and use this as the driving force to get people fully engaged with your product
    • Provide positive reinforcement when they complete steps! (Emails can help push them along)

Fostering Effective Collaboration in a Global Environment (PJ McCormick)

PJ’s talk was just as good this year as it was last year. He gave lots of great tips for increasing collaboration and trust among teams (especially the engineering and design teams), which is also a topic that has been on my mind recently.

  • His UX team designs retail pages (e.g. discover new music page). In one case, he presented work to the stakeholders and dev team, who then essentially killed the project. What went wrong? Essentially, it was a breakdown of communication and he didn’t include the dev team early enough.
  • Steps to increasing collaboration:
    1. Be accessible and transparent
      • Put work up on public walls so everyone can see progress (this is something I want to do more of)
      • Get comfortable showing work in progress
      • Demystify the black box of design
    2. Listen
      • Listen to stakeholders' and outside team members opinions and feedback (you don’t have to incorporate it, but make sure they know they’re being heard)
    3. Be a Person
      • On this project, the communication was primarily through email or bug tracking, which lacks tone of voice, body language, etc.
      • There was no real dialog. Talk more face to face, or over phone. (I have learned this again and again, and regularly walk over to someone to hash things out at the first hint of contention in an email chain. It’s both faster to resolve and maintains good relations among team members)
    4. Work with people, not at them
      • He should have included stakeholders and outside team members in the design process.
      • Show them the wall; show UX studies; show work in progress
      • Help teach people what design is (this is hard. I want to get better at this)

A question came up about distributed teams, since much of his advice hinges on face to face communication. I’ve been struggling with this (actually, the whole company has), and his recommendations are in line with what we’ve been trying: use a webcam + video chat to show walls (awkward; not as effective as in person), and take pictures/digitize artifacts to share with people (has the side effect of being archived for future, but introduces the problem of discoverability).

And that’s all! (Actually, I missed the last talk…). Overall, a great conference that I intend to go back to next year.

Tags: conference notes warm gun

Optimizely's iOS SDK Hits Version 1.0!

On Novemeber 18th, 2014, Optimizely officially released version 1.0 of our iOS SDK and a new mobile editing experience. As the lead designer of this project, I’m extremely proud of the progress we’ve made. This is just the beginning — there’s a lot more work to come! Check out the product video below:

Stay tuned for future posts about the design process.

Tags: Optimizely ios
View all posts »