Finding the (Real) Truth in Product Validation with Rob Fitzpatrick

If market research returns feedback like, “I think it’s a good idea,” or “I’d buy that,” it’s NOT a good thing.

Chances are, you’re asking the wrong questions — especially because the responses are so hollowly encouraging. Remember, the respondent has no skin in the game, and talk is free.

Ask Mom what she thinks about your new product, and she’ll tell you it’s fabulous — even if nothing could be farther from the truth.

Everyone will naturally lie to you, just like Mom, because it feels helpful and is easy. No one wants to hurt your feelings. Trouble is, when your market survey respondents don’t give you the straight goods, it defeats the entire purpose of validation work, and bad data can bait you toward product failure.

When you’re launching an innovative product, product-market fit is essential. That means knowing a sufficient, reachable customer base will be more excited about the product than you are. Conducting validation research is crucial. But the questions you ask is where things often start to go wrong.

Author Rob Fitzpatrick has been a tinkerer and tech entrepreneur for over a decade. He believes you can avoid mistakes by asking questions that elicit the answers you NEED, not the ones that make you feel good. Complements, verbal support, and new ideas can feel helpful, but in conversation with Graphos Product CEO Laurier Mandin, Rob Fitzpatrick says it’s much better to ask questions about people’s problems or challenges — as well as current solutions and what they cost.

If you ask someone how often they go to the gym, they’ll probably inflate the number. If you ask “how many times did you go to the gym last week,” you’ll probably get a more realistic answer. Rob’s first book, “The Mom Test” is taught at Harvard, MIT, UCL and many other top universities. Buy The Mom Test on Amazon.

Episode transcript:

Andreas: Welcome to Product: Knowledge, the podcast about marketing products that improve people’s lives. I’m Andreas Schwabe, director of media services at Graphos Product.

Andreas: Mom is a liar. If you ask mom what she thinks about your new business or product, she’ll lie and tell you it’s fabulous. Even if it’s not. If you’re launching a new business or product, a million things can go wrong. It’s important to consider every step when you’re introducing something to market. Asking questions is obviously crucial, but it’s also where problems can start. The root of marketing is communication. If market research questions, return answers like “I think it’s a good idea” or “I’d buy that,” then you’re probably asking the wrong questions. People who know you and like you will lie to you because it seems helpful and kind. Just like mom.

Andreas: For example, if you ask someone how often they go to the gym, they’ll probably inflate the number, but if you ask how many times did you go to the gym last week, you’ll probably get a much more realistic or honest answer. Rob Fitzpatrick describes himself as a tinker and a tech entrepreneur for over a decade. He believes you can avoid mistakes by asking questions that elicit the answers you need, not the answers you want. Complements verbal support and new ideas can be helpful, but in conversation with Graphos CEO, Laurier Mandin, Rob Fitzpatrick says it’s better to ask questions about people’s problems or challenges. Laurier spoke with Rob Fitzpatrick on Zoom from his home in Barcelona.

Laurier: Can you tell us about the mom test and how you came up with it?

Rob: Yeah, I was a technical founder. I heard about Y Combinator back in 2008 and kind of pitched my idea and they accepted us. We were a very technical product oriented team. He said “Okay, you’ve got to go talk to your customers. You need to understand what the customers cared about.” I was like “All right, that’s a terrible job, but you know, I’ll go do it. I’ll take one for the team. I worked really hard. I spent so much time, so much energy, getting customer feedback and user feedback and asking what they thought before we built things, after we built things and then we still went out of business.

Rob: When I looked back on it, I was like so deeply misunderstanding what I was being told. It was because I was asked asking for opinions and so I was getting opinions. That’s not like the good data you want, that’s not what you’re looking for. You’re looking for real insights about how and why your customers do things. Later when I was starting my next company, I was like “Oh, it’s not their job to tell you the truth. It’s your job to find the truth.” You do that by asking questions that are so good that even a very biased person, like your mother, couldn’t lie to you about because it’s a well-structured question.

Laurier: I guess you kind of told us a little bit, but what more specifically is The Mom Test itself?

Rob: Oh, so the rules for good questions is you never want to ask for opinions about your idea. Instead you want to ask how they’re already dealing with that issue today or how they dealt with the last time. Concrete specifics in the past or the now. You don’t want to be the one talking, as much as possible. You want to ask them about their life or their problems and then let them go off about it.

Rob: If you’re talking most of the time it probably means you’re pitching, you’ve made the conversation about your product instead of your customer. Don’t take hypotheticals or guesses at what people will do in the future. Don’t go “Would you ever do this? Might you ever do this?” Of course they might, someday maybe. Instead ask them the specifics “How did you do it today? How did you do it last time?” No opinions, no hypotheticals, and as much as possible, shut up and let them talk.

Laurier: Yeah. And I can think of so many products where the research turned up 100% of people would buy this product and use it. When the product was on the market, zero of those people actually went out and paid the money and bought it and used it. So yeah, they were absolutely lying. They’re lying thinking they’re helping us. I think that’s the worst part of it, right? They think they’re doing us a favor by giving us bad data. The mom test tells us there are three types of bad data. I think each one of those is important for the listener to recognize because those are some of the things that when I was reading the book, I was nodding and saying “Oh yeah, absolutely. This is so true.”

Laurier: They can be so hard to detect if you don’t know what you’re looking for too. Those types of bad data are compliments, fluff, and ideas. I think it’s worth going through those three briefly, because I think they’re so on the money as far as what is what happens.

Laurier: One thing that you point out a few times in the book is how dangerous compliments are. We love to hear compliments, they make us feel good about our ideas, but you call complements the “fool’s gold” of customer learning. Tell us about that. Why are compliments so bad?

Rob: If someone’s obviously lying to you or if they’re giving you an obvi- it’s easy for you to discount it and disregard it. When the lie is in the shape of what you really want to hear and what you really want to believe, it’s your job, everything you want is to prove that there’s a market for this, to prove that you’re on the right track, to prove the customers want it. You’re searching through all these dead leads and disinterested customer segments to find that. When you think you’ve seen it, when someone’s giving you, they’re saying “Yeah, I love this. This is awesome. I’ve never seen anything like this before. It’s so innovative. It’s so well designed.” Your brain so desperately wants to read that as like the sign of a real customer, as a commitment. So the compliments are more dangerous just because we want to believe them so badly.

Laurier: So when you’re interviewing someone and they are paying you compliments, do you find it’s best to say thanks or just ignore the compliment altogether and move on?

Rob: I mean ideally you avoid getting into the situation because where does the compliment come from? It comes from you pitching your idea. When you pitch the idea, what you’re really doing is you’re putting your ego on the table. You’re saying “Look, this is my dream. I poured my heart and soul into this. What do you think? Be honest.” And they’re like “Aw man, I don’t want this person to cry in front of me.” They’re automatically forced to pull their punches a little bit. If you’re getting complimented, you already screwed up. So the best thing to do is catch yourself. You’re like “Whoops, I made a mistake. Disregard what I’m currently hearing.” And be like “Hey, you were telling me something about your life” or “Hey, how did you deal with this last time?” You try to get them back into the concrete facts about themselves and out of the land of feeling like they have to pay compliments to your idea.

Laurier: So the compliment—

Rob: Taking notes helps a lot, because you can review your notes later and you’re like “Wait, compliment, compliment and compliment. I see how I caused that.” It’s easier to strike them out of your notes than your memory.

Laurier: Yeah, that’s what I was recognizing here is that the compliment can actually be a flag that you can watch for. It can actually be kind of a method you use for self-correction, getting back on track and saying “Oh, oops, okay. I see by the fact that this is happening, that I’ve got to make some corrections if I want this data to be worth anything to me in the end.” I understand what you’re saying with that.

Laurier: The next kind of bad data you have in there is fluff. I love that the world’s most deadly fluff is the statement “I would definitely buy that,” because a lot of marketers, even researchers, might find it surprising that that’s bad because that’s what they think they’re looking for. They think that’s telling them the product is a slam dunk, at least with one customer. What’s wrong when someone says “I would definitely buy that”?

Rob: The problem is they’re not in a buying mindset, they’re not in a buying situation and they’re not actually giving anything up. That’s a really easy thing to say. When you’re pitching an idea or showing someone in early demo, what you’re doing is they’re imagining all the benefits without facing the realities of actually using it. When you’re like “Yeah, it’s an alternative to email. It’s so much better.” People are thinking “Oh.” They’re only thinking about the good things. They’re not thinking about how much they’d have to tear up their whole workflow and tool chain if they replaced email with something else. That’s part of it, they’re not in a buying mindset, they’re not in an installation mindset, they’re not dealing with the realities. Like “Where do I put this great new product on my shelf?” Like “Where does it fit into my kitchen?”

Rob: That’s part of it. Also, people are just so optimistic. How many new year’s resolutions have you broken to yourself? You make it “I am definitely going to go to the gym this year” and you don’t. We’re just, people are terrible about predicting future behavior. The way you solve that is you either try to judge their seriousness by how much effort they’re already spending on the problem. If there’s a problem at some business and they’ve hired a team of five interns to solve it, who they’re paying like “Whoa, they’re spending a lot of money to deal with this issue.” Whereas someone else is like “Eh, I just kind of ignore it.” You kind of have to learn about their life and make their own decision or get them to give you something they care about.

Laurier: Yeah, ask for money.

Rob: Compliments are free. Yeah, you can ask for money. You can ask for like big meaningful chunks of their time. You can ask for introductions to their boss or superiors, basically anything that they’d only give you if they really care. You can use that to immediately shift them back into a buying mindset. If you notice discomfort, it’s not about tricking them, it’s just about noticing that discomfort and be like “Oh, it seems like it’s not quite for you.” You’re just using the commitment as a way to be like “Is it an empty commitment? Are they imagining they’d want it or do they potentially actually want it?”

Laurier: Yeah, and you mentioned just a second ago that if they’ve dedicated resources or effort, have they Googled the problem, or are trying to find a solution because in many cases there might even be existing solutions but they haven’t even gone so far as to look for them.

Laurier: The third type of bad data is ideas, which again, on the surface it sounds like “Well, people are giving ideas and making suggestions to help improve this thing. That should be helpful. Let’s see how many good ideas we can get from this research.” But that’s, when people are telling you “Oh, it’s got to have X in order to be truly great” that obviously can be really bad too, because if you have an MVP you’re striving to make, now you’re going to bloat the product, you’re going to, you’re going to change the whole development process, you can get really distracted by this stuff. But, is there any good that can come from, from these ideas that come forward? Can they not be taking you in a direction of improving your product or maybe even getting you back on track to what the real need is in the market maybe if it’s not what you think? Can there be good ideas?

Rob: Yeah, so I love hearing ideas from customers. It’s just I don’t blindly do them. Where the real gold is is if you dig underneath that feature request, because you go “Hey, why do you want that? What are you doing today since you don’t have that feature? What are you really trying to accomplish? How would this change your life to have this?” Often the feature request is a sign that they have some pain or some unsolved goal or some frustration. Sometimes it’s important. Sometimes it’s just like a fun idea that they’re giving you because whatever. You’re trying to dig beneath this.

Rob: The feature request is the signal and you’re trying to dig below that to find the root cause and that’s the problem or the goal. Then, you come to your own decision as a product designer about like “Okay. 1) is that goal worth serving? I don’t have to say yes to every request, even if it’s valid. If you decide it is, what’s the best way to do it? It might end up looking very different than what the customer originally requested. They asked for a better way to attach files to emails, but what they really want is hands-off backup or secure backup or whatever. Who knows? I’m try to get to their motivations.

Laurier: Yeah. We work mostly with physical products, and it can happen there too, although I’ve been involved in software products, especially, where you’re working on an app and every single meeting brings somebody bringing up a new feature idea that has to be added to the scope and this is going to be the thing that’s going to make it wonderful. Here’s why everybody’s going to buy the product. You know exactly what that does, right? More than I see the value of ideas, I see the damage they cause and I see the risk potential for even entertaining them, especially at an early stage. Especially when you’re the marketer, because that’s the role I’m usually in, or are working on the development side. These ideas are popping up in derailing all the existing priorities because someone woke up with this Eureka moment in the middle of the night, right?

Rob: Yeah. I like to, for myself, I always try to think when I get excited about something, I’m like “Is that part of the core or is that an optimization?” And then I’m like “Oh, that’s a really cool, really clever optimization.” It’s like “Okay, I can do that after the business is already working and profitable and people like the product and I can return to that treasure trove of fun little add-ons.”

Laurier: Yeah. When you’re interviewing there’s something you call zooming and also another thing referred to as “premature zoom.” Tell us about that. How when you’re discussing a product with somebody and gauging interest and trying to find out if this is going to be viable, how you decide when to zoom.

Rob: Zooming is when you kind of shift to the next most… There’s multiple layers in the conversation. When I’m first talking to someone, my very first question is “Are they even one of my customers?” If I’m building a toy for babies, do they have babies? That’s like the first layer. Once you’ve confirmed that, you’re like “Cool.” Then you’re zooming in and then the next layer is maybe like “I want to understand if they care and all about the problem. Is this an issue to them?” If I built a cool new way for kids to learn how to create music, it’s like a simpler baby version of an instrument that lays good music fundamentals. In theory that’s good. In theory, everyone wants their kids to be musical, but this particular person I’m talking to, do they see music education as a problem worth solving? Are they going to be motivated to buy this thing at whatever?

Rob: Once you’ve confirmed like “Yeah, they care about the problem,” you can kind of zoom up into maybe talking about what the product is or the exact features or price point or some of the details. But, the point is with premature zooming, if you start a conversation by talking about product features, you will never learn if the problem matters. If you start by talking about the problem, you might not even realize that you’re not talking to someone who has it. So you want to start broadly and then you’re like you’re zooming in once you get the confidence that you understand that layer.

Laurier: When it comes to questions you talk about going into these interviews with your “big three questions” and I love that idea of having that kind of focus. How do you yourself determine what those “big three questions” should be so that you go in and and you’re at least on more or less on the right track, I know they can evolve.

Rob: If I’m thinking of it from like a whole business perspective, like early on it’s questions like “Do they care? Will someone pay for this?” Then, you’re getting into more concrete questions like “Okay. Have I chosen the right type of products? Does it have roughly the right features? Can I make it roughly available?” You’re going through the business model, but in any given individual conversation, it’s just what’s most immediately scary and what information I need to make the next set of decisions about my business or my product features.

Rob: An early one might be like “Is this legal?” Or like “Do they understand how to use, it?” It might be like a product focused one, some of them are about the customer’s lives, some are about the interaction with what you’re building. You kind of know and they just like shift. If you started like “What’s killing us this week?” Or like “What’s blocking us from making this big decision?” I think of them as areas I’m trying to learn about rather than exactly questions.

Rob: I might be trying to be like “Okay they say it’s a problem, but do they have a budget?” It might be like something I’m trying to learn about. So it’s just this area of budgets and willingness to pay. The reason I like three is I just found if I went in with 20 questions it would come off very boring. I’d be turning into an interviewing robot and the people I was talking to would stop enjoying it. Where if it was like three topics that really matter to me, it was structured enough for me to learn something of value but still loose enough that I could have a conversation that didn’t feel like hard work to the other person.

Laurier: Yeah, and you’d tend to get some consistency too, right? As far as apples to apples answers coming back in the data as opposed to having a list of 20 that you kind of pick and choose from and now a small percentage might answer some of the questions effectively and others are pushing your data all over the place. So you get more comparables that way I think.

Rob: You’ll shift your questions over time, but you’re usually… Like if I’m talking to five customers in a week, let’s say, or five users, I’m going to ask them kind of the same types of questions. So, I’m basically hearing the same things from everyone or I’ve heard it all before and I kind of understand it and then it’s like “Okay, answer that question then add a couple more to the group.”

Laurier: This has been really helpful, Rob. I really appreciate you taking the time to do this call with us. Is there anything else that you might add for our listener who typically has an idea in many cases and they are considering investing it and taking it to market and taking a chance on it. Is there anything else that you suggest that they should consider before they go into to doing these interviews and testing it? Testing the idea at least, testing what the idea is going to do for potential real world customers.

Rob: Yeah, a couple of comments. One is that you’re never going to get perfect certainty. It’s a bit of a trap to seek 100% safety or 100% certainty. What you’re trying to do is take the low hanging fruits, in terms of learning. At a certain point you need to put the thing in people’s hands and see if they use it and pay for it and whatnot.

Rob: Depending on the type of idea, you might be able to get an easy 20% certainty from having a few conversations. In some it’s much higher than that, you feel totally confident just off conversations. But , I’d say like “Take the easy learning. It’s crazy to not take the easy stuff, but don’t worry that you’re not getting up to 100% and recognize that your confidence is going to be different.” It’s always risky to make a toy than it is to make a problem solver because toys you can’t prove that people like them till they’re finished. Both are good types of products, right? It’s just easier to test the problem solver.

Rob: Start with the easy conversations first. Start with the friendly ones. Start with the non-stressful ones. It’s like it’s a skill that you’re going to build over time like skateboarding or pottery. If it’s new to you, you don’t start with the hard, scary people. Start with ones you can, not feel too bad about it if it goes totally sideways.

Andreas: That’s it for this episode of Product: Knowledge and our conversation with tech entrepreneur, Rob Fitzpatrick. You can find Rob’s website, including links to his books at robfitz.com.

Andreas: Visit graphosproduct.com where you can find out more about Graphos our services ideas or more podcasts and our blog. All our podcasts are transcribed for the deaf and hard of hearing. Reach out on twitter @graphosproduct or email us through the form on graphosproduct.com. Thanks for listening. I’m Andreas Schwabe.