Writing, like design, is a craft that can be practiced and improved. To do so, I’ve been compiling a collection of principles from books and articles. I decided to share them online to crystallize my own understanding of what I’ve learned so far, and to help others who want to improve at the craft of writing as well. I will continue to update the page as I learn more.
After jamming with a friend, she commented that she felt emotionally spent afterwards. Not quite sure what she meant, I asked her to elaborate. She said that improvising music makes you feel vulnerable. You’ve got to put yourself out there, which opens you up to judgement and criticism.
And she’s right. In that moment I realized that being a designer trained me to get over that fear. I know I have to start somewhere shitty before I can get somewhere good. Putting myself and my ideas out there is part of that process. My work only becomes good through feedback and iteration.
So my advice to you, young designer, is to accept the fact that before your work becomes great, it’s going to be shitty. This will be hard at first. You’ll feel vulnerable. You’ll fear judgement. You’ll worry about losing the respect of your colleagues.
But get over it. We’ve all felt this way before. Just remember that we’re all in this together. We all want to produce great work for our customers. We all want to make great music together.
So get comfortable sharing your shitty work. You’ll start off discordant, but through the process of iteration and refinement you’ll eventually hit your groove.
After launching a feature, coworkers often ask me, “Did you A/B test it?” While the question is well-meaning, A/B testing isn’t the only way, or even the best way, of making data-informed decisions in product development. In this post, I’ll explain why, and provide other ways of validating hypotheses to assure your coworkers that a feature was worth building.
Implied Development Process
My coworker’s simple question implies a development process that looks like this:
You have an idea for a new feature
You build the new feature
You A/B test it to prove its success
Profit! High fives! Release party!
While this looks reasonable on the surface, it has a few flaws.
Flaw 1: What metric are you measuring?
The A/B test in step 3 implies that you’re comparing a version of the product with the new feature to a version without the new feature. But a key part of running an A/B test is choosing a metric to call the winner, which is where things get tricky. Your instinct is probably to measure usage of the new feature. But this doesn’t work because the control lacks the feature, so it loses before the test even begins.
There are, however, higher-level metrics you care about. These could range from broad business metrics, like revenue or time in product, to more narrow metrics, like completing a specific task (such as successfully booking a place to stay in the case of AirBnB). Generally speaking, broader metrics are slower to move and influenced by more factors, so narrow metrics are better.
Even so, this type of experiment isn’t what A/B tests excels at. At its core, A/B testing is a hill climbing technique. This means it’s good at telling you if small, incremental changes are an improvement (in other words, each test is a step up a hill). Launching a feature is more like exploring a new hill. You’re giving users the ability to do something they couldn’t do before. A/B testing isn’t good at comparing hills to each other, nor will it help you find new hills.
Flaw 2: What if the new feature loses?
Let’s say you have good metrics to measure, and enough traffic to run the test in a reasonable timeframe. But the results come back, and the unthinkable happened: your new feature lost. There’s no profit, high fives, or launch party. Now what do you do?
Because of sunk costs, your instinct is going to be to try to improve the feature until it wins. But an A/B test doesn’t tell you why it lost. Maybe there was a minor usability problem, or maybe it’s fundamentally flawed. Whatever the problem may be, an A/B test won’t tell you what it is, which doesn’t help you improve it.
The worst-case scenario is that the feature doesn’t solve a real problem, in which case you should remove it. But this is an expensive option because you spent the time to design, build, and launch it before learning it wasn’t worth building. Ideally you’d discover this earlier.
Revised Development Process
When our well-meaning coworker asked if we A/B tested the new feature, what they really wanted to know is if we have data to back up that it was worth building. To them, an A/B test is the only way they know how to answer that question. But as user experience professionals, we know there are plenty of methods for gathering data to guide our designs. Let’s revise our product development process from above:
You have an idea for a new feature.
You scope the problem the feature is supposed to solve by interviewing users, sending out surveys, analyzing product usage, or using other research methods.
You create prototypes and show them to users.
You refine the design based on user feedback.
You repeat steps 3 and 4 until you’re confident the design solves the problem you set out to solve.
You build the feature.
You do user testing to find and fix usability flaws.
You release the feature via a phased rollout (or a private/public/opt-in beta) and measure your key metrics to make sure they’re within normal parameters.
This can be run as an A/B test, but doesn’t need to be.
Once you’re confident the feature is working as expected, fully launch it to everyone.
Profit! High fives! Release party!
Optimize the feature by A/B testing incremental changes.
In this revised development process (commonly called user-centered design), you’re gathering data every step of the way. Rather than building a feature and “validating” it at the end with an A/B test, you’re continually refining what you’re building based on user feedback. By the time you release it, you’ve iterated countless times and are confident it’s solving a real problem. And once it’s built, you can use A/B testing to do what A/B testing does best — optimization.
A longer process? Yes. A more confident, higher quality launch? Also yes.
Now when your coworkers ask if you A/B tested your feature, you can reply, “No, but we made data-informed decisions that told us users really want this feature. Let me show you all of our data!” By using research and A/B testing appropriately, you’ll build features that your users and your bottom line will love.
Further Reading
If you’d like to learn how other companies incorporate A/B testing into their development process, or about user-centered design in general, these articles are great resources:
“Building websites with science” by Peter Seibel — A broad overview with examples of how Etsy incorporates A/B testing into the development process.
“Experiments at Airbnb” by Jan Overgoor — A great overview of how AirBnB incorporates A/B testing into feature development, complete with best practices.
“How Not to Run an A/B Test” by Evan Miller — Best practices for running an A/B test, which are especially important to get right when testing in product.
Over on Design @ Optimizely (the Optimizely Design team’s Medium publication), I posted a photo essay documenting the unique sketching and note taking styles of our team members.
Google Ventures invited design leaders from Twitter, Uber, and GoPro to discuss the topic of hiring designers. What follows are my aggregated and summarized notes.
Finding Designers
Everyone agrees, finding designers is hard. They’re in high demand, and the best ones are never on the market for long (if at all). “If the job is good enough, everyone is available.” There are a few pieces of advice for finding them, though:
If you’re having trouble getting a full-time designer, start with contractors. If they’re good, you can try to woo them into joining full-time. Some designers like the freedom of contracting and don’t think they want to be full-time anywhere, but if you can show them how awesome your team and culture and product are, you can lure them over.
Look for people who are finishing up a big project, or have been at the same place for 2+ years. These people might be looking for a new challenge, and you can nab them before they’re officially on the market.
Dedicate hours each day to sourcing and recruiting. Work closely with your recruiters (if you have any) to train them on what to look for in portfolios and CVs. Include them in interview debriefs so they can understand what was good and bad about candidates, and tune who they reach out to accordingly. I.e. iterate on your hiring process. We’ve done this a lot of Optimizely.
Even better is to have dedicated design recruiter(s) who understand the market and designers.
If you have no recruiters, you could consider outsourcing recruiting to an agency.
When reaching out to designers, get creative. Use common connections, use info from their site or blog posts, follow people on Twitter, etc.
Typically you’ll have the highest chance for success if you, as the hiring manager, reach out, rather than a recruiter.
As a designer, this is what hiring managers will be looking for:
Have a high word-to-picture ratio. Product Design is all about communication, understanding the problem, solutions, and context. If you can’t clearly communicate that, you aren’t a good designer.
An exception is visual designers, who can get away with more visually-oriented portfolios.
What about your design is exceptional? Why should I care? Make sure to make this clear when writing about your work.
When looking at a portfolio, hiring managers will be wondering, “What’s the complexity of the problem being solved? Can they tell a story? Are they self critical? What would they do differently or what could be better?” Write about all of these things in your portfolio; don’t just have pictures of the final result.
An exception to the above is high demand designers, who don’t have time for a portfolio because they don’t need one to get work. Hiring these people is all based on reputation.
Don’t have spelling errors. Spelling errors are an automatic no-go. Designers need to be sticklers for details, and have “pride of ownership.”
One million percent agree
On Interviewing Designers
Pretty much everyone has a portfolio presentation, followed by 3–6 one-on-one interviews. Everyone must be a “Yes” for an offer to be made. (Optimizely is the same.)
Look for curiosity in designers. Designers should be motivated to learn, grow, read blogs/industry news, and use apps/products just to see what the UX and design is like. They should have a mental inventory of patterns and how they’re used.
In portfolio review, designers should NEVER play the victim. Don’t blame the PM, the organization, engineering, etc. (even if it’s true.) Don’t talk shit about the constraints. Design is all about constraints. Instead, talk about how you worked within those constraints (e.g. “there was limited budget, therefore…”)
On Design Exercises
People were pretty mixed about whether design exercises are useful during the interview process or not. Arguments against them include:
They can be ethically wrong if you’re having candidates do spec work for the company. You’re asking people to work for free, and you open yourself up to lawsuits.
I wholeheartedly agree
They don’t mimic the way people actually work. Designers aren’t usually at a board being forced to create UIs and make design decisions.
I disagree with this sentiment. A lot of work I do with our designers is at whiteboards. Decisions and final designs aren’t always being made, but we’re exploring ideas and thinking through our options. Doing this in an interview simulates what it’s like to work with someone, and how they approach design. It isn’t about the final whiteboarded designs, it’s about their process, questions they ask, solutions they propose, how they think about those solutions, etc. Plus, you get to experience what they’re like to interact with.
Take home exercises aren’t recommended. People are too busy for them, and senior candidates won’t do them.
The exception to this is junior designers who don’t have much of a portfolio yet so you can see how they actually design UIs
All of this has been true in my experience, as well.
Arguments for design exercises:
You get to see how candidates approach a problem and explore solutions
You get a sense of what it’s like to work with them
You hear them evaluate ideas, which tells you how self-critical they are and how well they know best practices
Personally, I find design exercises very useful. They tell me a lot about how a candidate thinks, and what they’re like to work with. The key is to find a good exercise that isn’t spec work. GV wrote a great article on this topic.
On Making a Hiring Decision
It’s easy when candidates are great or awful — the yes and no decisions are easy. The hard ones are when people are mixed. Typically this means you shouldn’t extend an offer, but there are reasons to give them a second chance:
They were nervous
English is their second language
They were stressed from interviewing
In these cases, try bringing the person back in a more relaxed environment; for example, have lunch or coffee together.
Some people have great work, but some sort of personality flaw (e.g. they don’t make eye contact with women). These people are a “no” — remember, “No assholes, no delicate geniuses”, and avoid drama at all costs.
When making an offer, you’ll sometimes have to sell them on the company, team, product, and challenges. One technique is to explain why they’ll be a great fit on the team (you’ll flatter them while simultaneously demonstrating the challenges they’ll face and impact they’ll have). If you have a big company and team, you can explain all the growth and learning opportunities a large team provides. And you don’t need to be small to move fast and make impactful decisions.
On Design Managers
Hiring design managers is hard. They’re hard to find, hard to attract, and most designers want to continue making cool shit rather than manage people. But if you’re searching for one, your best bet is to promote a senior designer to manager. They already understand the company, market, culture, and team, so they’re an easy fit. The art of management is often custom to the team and company.
If that isn’t an option, go through your network to find folks. You aren’t likely to have good luck from randos applying via the company website, or sourcing strangers.
Great managers are like great coaches — they’re ex-players who worked really hard to learn the game, and thus can teach it to others. Players that are naturally gifted, e.g. Michael Jordan, aren’t good coaches because they didn’t have to work hard to understand the game — it came naturally to them.
I feel like I fit this description. I worked hard to learn a lot of the skills that go into design. It took me a long time to feel comfortable calling myself a “designer”; it didn’t come naturally.
Management is a mix of creative direction, people management, and process. They should be able to partner with a senior designer to ship great product. Managers shouldn’t evaluate designers based on outcomes/impact. People can’t always control which project they’re on, some projects are cancelled, not all projects are equal, etc. Instead, reward behavior and process (e.g. “‘A’ for effort”.)
There are 4 things to look for in good managers:
They Get Shit Done
They improve the team, e.g. via recruiting, events, coaching/mentoring
They have, or can build, good relationships in the organization
They have hard design skills, empathy, and vision
On Generalists vs Specialists and Team Formation
The consensus is to hire 80/20 designers, i.e. generalists who have deep skills in one area (e.g. visual design, UX, etc.). They help teams move faster, and can work with specialists (e.g. content strategists) to ship high quality products quickly. Good ones will know what they don’t know, and seek help when they need it (e.g. getting input from visual designers if that isn’t their strength). “No assholes, no delicate geniuses”. Avoid drama at all costs.
This is the type of person we seek to hire as well. I’ve also seen firsthand that good designers are self-aware enough to know what their weaknesses are, and to seek help when necessary.
Cross-functional teams should be as small as possible while covering the breadth of skills needed to ship features. More people means more complexity and extra communication overhead. (I have certainly seen this mistake made at Optimizely.)
Having designers on a separate team (e.g. Comm/marketing designers on marketing) makes for sad designers. They become isolated, disgruntled, and unhappy. Ideally, they shouldn’t be on marketing. If they are separate, make bridges for the teams to communicate. Include them in larger design team meetings and crits and stuff so they feel included.
I totally agree. At Optimizely, we fought hard to keep our Communication Designers on the Design team for all the reasons listed here (Marketing wanted to hire their own designers). Our Marketing department ended up hiring their own developers to build and maintain our website, but earlier this year they moved over to the Design team so they could be closer to other developers and the Communication Designers working on the website. So far, they’re much happier on Design.
Should designers code?
People were somewhat mixed on this question. It was mostly agreed that it’s probably not a good use of their time, but it’s always a trade-off depending on what a specific team needs to launch high quality product. A potential danger is that they may only design what’s easy to code, or what they know they can build. That is, it’s a conflict of interest that leads to them artificially limiting themselves and the design.
As a designer who codes, I only partially agree with what was said here. It’s true that you can fall into the trap of designing what’s easy to build, but it doesn’t have to be that way. I overcame this by focusing on explicitly splitting out the ideation/exploration phase from the evaluation/convergence phase (something that good designers should be doing anyway). When designing, I explore as many ideas as I can without thinking at all about implementation, then I evaluate which idea(s) are best. One of those criteria (among many) is implementation cost and whether it used existing UI components we’ve already built. I’ve found this to be effective at not limiting myself to only what I know is easy to build, but it took a lot of work to compartmentalize my thinking this way.
Artificially constraining the solution space is also a trap any designer can fall into, regardless of whether or not you know how to code. I’ve heard designers object to ideas with, “But that will be hard to build!”, or, “This idea re-uses an existing frontend component!” Whenever I hear that, I always tell them that they’re in the ideation phase, and they shouldn’t limit their thinking. Any idea is a good idea at this point. Once you’ve explored enough ideas, then you can start evaluating them and thinking about implementation costs. And if you have a great idea that’s hard to implement, you can argue for why it’s worth building.
Design-to-Engineering Ratio
It depends on the work, and what the frontend or implementation challenges are. For example, apps with lots of complex interactions will need more engineers to build. A common ratio is about 1:10.
More important than the specific ratio is to not form teams without a designer. Those teams get into bad habits, won’t ship quality product, and will dig a hole of design debt that a future designer will have to climb out of. (I’ve been through this, and it takes a lot of time and effort to correct broken processes of teams that lack design resources).
One way of knowing if you don’t have enough designers is if engineering complains about design being a bottleneck, although this is typically a lagging indicator. A great response to this was that the phrase “Blocked on design” is terrible. Design is a necessary creative endeavor! Why don’t we say that engineering is blocking product from being released? (In fact, for the first time ever, we have been saying this at Optimizely, since we need more engineers to implement some finished designs. Interested in joining the Engineering team at Optimizely? Drop me a line @jlzych).
Another good quote: “There’s nothing more dangerous than an idle designer.” An idle designer can go off the deep end redesigning things, and eventually get frustrated when their work isn’t getting used. So there should always be a bit more work than available people to do it. True dat.
This was a great event with fun speakers, good attendees, and excellent advice. The most interesting discussion topic for me was on design managers, since we’re actively searching for a manager now (let me know if you’re interested!) Overall, Optimizely’s hiring practices are in line with the best practices recommended here, so it’s nice to know we’re in good company.