I have been asked for advice on interviewing product managers. I started to write down some questions and ended up with this more comprehensive memo, so I decided to turn it into a post. This post builds upon Joel’s “Guerrilla guide to interviewing” which is definitely required reading. Everything from that post still applies but the detailed programming questions. Instead, here I propose questions and interview strategies specifically designed to find great product managers.
You can find some very good tips on how to avoid common mistakes during PM interviews here.
It has been a little while since I interviewed product managers, so I am a bit rusty ‘cos interviewing takes practice. I am planning on coming back to this post and add new things as they come back to mind.
The Product Manager
First things first. Let’s define what a product manager does. To me a product manager defines the “What and Why”.
The “What” covers the definition of the product, the set of features, what goes around it (packaging, pricing, support, documentation etc)
The “Why” covers the reasons why EVERY single thing that goes into the product is there.
I use the “what and why” extensively during the interview. I want to see if a PM is intimately familiar with every What of the product and very importantly for every what there is a clear Why.
Unlike Microsoft-like program managers, Product managers are not concerned with the “How” which is the realm of engineering but they should have enough technical knowlesge, experience and credibility to have good opinions about the how and ood judgment.
But it is important to keep responsibilities clearly separated. Engineer scan and should have opinions about the “what” and PMs may have input into the “How” but the ultimate responsibility to make calls stays within each organization.
If you are implementing the role of a Program manager as per the Microsoft school, then this is slightly different. In this memo, I will talk about the more traditional silicon valley product manager role.
This is one of the most important attributes for me. This is somewhat subjective as other people may value analytical skills over passion, this is just me. I look for passion in these three areas in priority order
Good product managers LOVE their customers. They understand their pain, their use cases, their roadmap an their business priorities. It is passion for customers that drives the definition of some of the best products. Moreover love for customers is the key ingredient for sustained success.
Interview Tactics and Questions
I throw them a soft ball to start. Talk to me about the product you built in the past and where you had the most fun and/or you had the most impact.
Here I am looking for the eyes to light up (passion) and I am listening for thing like “Customers were trying to do this…”, “Customers were struggling with…” you get the idea. I want to see if the candidate understands the customer pain and use cases.
- Then I probe to see if they measured the impact they had on their customers.
- What was the product ROI?
- What were the metrics impacted by a given features (Opex, Capex, ease of use etc)
- Talk to me about your most critical customer escalation and tell how you handled it.
- Again I am looking for his/her ability to address customer issues while balancing company priorities.
- Tell me about a time when you had to say no to a customer? Why? How did you handle it?
- You want to know that the candidate knows how to say no (a great treat for a PM) when it is the right answer.
- Why?? A great strategy when interviwing PMs is to ask “why” many, many times. Remember? Defining/knowing the “Why” is a big part of a PM’s job description so they better have great answer to that question.
- Why did you build that product?
- Why was that feature important?
- Why did you cut that other feature?
- Why did you move the release date etc.
Good PMs love great engineers. They know how to work with them, unleash them and get out of the way, standup to them when needed. A PM relationship with his/her engineering team is key to the success of the team and the product.
You really want to see a balance of power. Whenever you see one of the two teams pravaling on the other it is bad news.
Here a set of questions I use
- What is the worst thing you have done to an engineering team
- Every PM made mistakes in his past. Some of these mistake may have impacted engineers. For example, not cutting deep and early enough, or not having clearly defined and prioritize product features. You want to know that the candidate had thought about these issues.
- Describe to me the best engineer you ever worked with and why.
- What made him/her great?
- What did you achieve together?
- Every PM has great stories about great engineers. If they don’t they either don’t love working with engineers (very, very bad sign) or they have not worked with great engineers (another very bad sign). You want to know that your candidate has some good stories and tell them with passion
- In your opinion what is the different in productivity between a super star engineer and an average engineer?
- If they hesitate (which is kind of a bad sign), I make it easy for them and ask them “Is it 150% or 10 times?”. Of course there is no numeric answer, but if the candidate says 150% then they have never seen a super star engineer, clearly. Pass.
- Say you give your engineering team a list of 10 features for a product release, they come back and say that can implement 6 from your list and 3 that you did not think about, what do you do?
- Good question to see how well the candidate understands the give and take relationship with engineers. Also you want to see how s/he approaches the 3 features coming from engineers. Is s/he excited about what these ideas can be? Does it turn them down a priori ‘cos they were not on his list (clearly a deal breaker)
- Then I double down and I ask “say that one of the three features coming from the engineering team does now make sense to you, what do you do?”
- Again, you want to see if they know how to pick their battles and make the right tread offs.
I love when innovation is a company value. What in the world does it mean to have innovation as a value? You either have the right people or you don’t. Innovation does not happen because it is written in a company manifesto. Anyway, ranting apart, here I’m really trying to see if the candidate understands innovation, if s/he has lived it in the past. Can s/he think outside of the box, has he built products that were different, that broke with the past? Again, note that there are cases where you are looking for a farmer more than a hunter, in which case this attribute may not be as important. In any event, if you are recruiting for a farmer type of position you may be not able to attract innovator, so the point is moot.
What question do you ask to find this out?
- You can start with a soft ball and ask to describe a past project that was particularly innovative. From there you dig deeper and for every feature mentioned as differentiating, ask why 3 times. Again, always go back to customer pain addressed and how that particular feature helped customers get their job done better, faster, cheaper or/and easier.
- Ask about the competition. How did the product/feature help you competing? How was it different?
- If you have time (well, you should make time) ask them to define the requirements for a new version of an existing product. I use this in many different ways. In this case you would look for the candidate’s ability to look at an existing mainstream product and define new features or different integration points with other products based on latest technology available (e.g. instant messaging integrated with email in the browser) that display out of the box thinking.
- Don’t stop at technical features, sometimes the best innovation is around business model, sales channel, and so on.
I subscribe big time to Joel’s school of thought on hiring for talent and skills, NOT knowledge. Smart people will figure things out. Still, depending on how technical is your product, you want PM that can go deep on the technical side and have meaningful conversation with their engineering counterparts. These are the overall question you need to answer as the hiring manager
- Can they gain the respect of the engineering team?
- Do they have enough technical skills to make good judgment calls?
- Do they have enough technical chops to recognize great ideas coming form engineers?
- Do they have the credibility to have engineers trust their judgment calls?
Here are some point questions
- What was the theme of a given product release (from their resume)?
- Look to see if s/he understands release themes and more importantly how to use themes to focus the release and cut features
- What was the most painful cut you made in this or that release?
- They need to know this cold. PMing is more about what NOT to do than about what to do. If they cannot clearly articulate this, just pass. You are looking at a bad PM.
- If I called the lead engineer for that project what would s/he say about you?
- You are looking for clues of credibility, honesty and confidence here.
Whenever you see an opening to drill down on something technical do it.
- Why was that (____put some technical thingie that the candidate brought up here ___) important?
- You are not looking for the point knowledge here, you want to figure out if they understood the technical reason why this was important for customers.
Requirements and Priorities
- One way to see if they have a good sense of priorities is to ask them to write the list of top requirements for a well know product (same as above), but here you are really digging into the “Why”. For example
- Tell me an application that you are very familiar with and you use every day. Then ask them
- What drive you nuts about it and how would you fix it
- If you were the PM for this product, tell me what would be the theme of the next release. Why?
- What would be the top 5 features?
- For each feature then drill down into the ‘why” it should be there. He is the customer of the product, he should know the use cases and pain points.
- Tell me an application that you are very familiar with and you use every day. Then ask them
Out of context requirement question. I believe a good PM gets mad at bad products, bad processes and in general things that are poorly designed. PMs are professional problem solvers, always looking to improve things around them. I sometimes asked them to define the requirement for products that are completely outside of their specific domain.
- For example, I ask them if they travel for work, then ask them to define the requirements for a more efficient airport security procedure.
- I am looking for them to be mad at something “I hate the way they handle the security lines at airport” but I don’t stop there (I am not looking for a whiner, I am looking for a problem solver), so then I ask them how they would fix it if they were in charge.
- Look for something that they are very familiar with, not something you like or understand.
- You do want your PM to be analytical but I find that when people who are very passionate and innovative are sometimes not super analytical. That’s Ok as long as the team overall compensate for this.
- I am not sure how to assess this in an interview. I am open for suggestions.
Not just Features
It is not just about features and functions. Drill into other things what makes a product. Pricing, market, support, release cycle etc.
If they built a new product see how they went about it, how they validated the idea, the market opportunity, how/if they did early customer testing.
If they built mature products, see if they were formal about analyzing market, customer, support, field and revenue data.
This is important. Do they understand the challenges in selling products? Did they take the time in their past career to go out in the field and feel sales people’s pain? Do they know how to establish a great working relationship with the field?
Some example questions:
- Do you have an example where you were instrumental in closing a deal? What was your role? What did you do?
- Was there anything specific you did to make it easier for the sales guys to sell the product? Explain.
- How did you prospect customers for that particular product?
- An easy question to see if they even tried to put themslef in the sales people’s shoes
- What sales model did that company implemented?
- See if they were intimate with the sale model, direct, indirect, overlay etc.
- If you had a chance to change the go to market approach for a previou product, what would you do? Why?
Everybody makes mistakes. Ask them to talk about one and see what they learned from it. You want people who are honest and open about mistakes. If they get defensive, it is a bad sign
These are tough to evaluate. If you are the hiring manager, I do recommend to meet with a candidate three times.
- First for an introductory meeting get to know each other, assessthe chemistry
- Then for the formal interview. Make sure every key person that will work with the candidate has a chance to interview him/her. Not just in the PM team but across team: Engineering, Marketing, PM, ideally a System Engineer or sales person.
- Finally for the closing event and presentation of the offer. I pulled out of giving a candidate an offer at this stage. What happens after you see a person for the third time is that some of the main characteristics of their personality will become more evident. For example, if you felt a little arrogance the first time you meet them but you were not sure how bad it was, by the third time meeting if they really are arrogant they will start bragging and you will know. I gave up on a great candidate because I got a hunch that he was a whiner and by the third time we met he could not stop going on and on about how everything that ever went wrong with in his career was somebody elase’s fault. And I thought “Really??”
Let’s get back to soft skills. What I generally look for is the following:
PM is a lot about listening. Listening to customers to gather requirements and feel the pain, listening to sales people to learn how to help them sell the product, listening to executives to understand the company priorities, listening to engineers for great ideas and even listening to angry customers who are having a problem.
Are they good listener? Do they listen carefully and ask good questions before giving their answer?
PMs sits between an engineering team and everybody else. They interact with multiple organizations multiple times a day. They get interrupted often and they context switch a lot.
During the interview, whenever I see that the candidate is on the right track, I move to the next question, and then at the first chance I go back to a previous topic and see if they can take it from when we left it. I do this multi times during the interview. It allows me to see how well the context switch.
Communication skills are very important for product managers. You can assess this throughout the interview but you may want to ask them a couple of point questions here to see if they can be crisp and compelling in their communication. For example
- What was the elevator pitch for that product?
- How was it different from the competition?
If a PM is to be able to lead great engineers, make tough calls, handle customer escalation, they need to be confident. I don’t mind calling them wrong on something they are absolutely right on to see how they react. Do they stand for their opinionor do they quickly back down?
I generally agree with Joel on using puzzles in interviews. I have seen SUPER smart engineers (let alone PMS) miserably fail on relatively simple puzzles. Still, sometimes I do for PMs as it I allows me to see how they think on their feet, if they think out of the box, if they engage. This is tricky thou. You need to make sure they understand this is a way to engage together and they are not going to be judge solely on the puzzle.
A lot have been written about this topic but I do want to bring this up because this is indeed a key treat of a good PM. Product management is a lot about leverage, working with multiple people AND multiple teams to get stuff done.
You can tell a lot about team playing when you ask them about engineers and customers above, but overall you want to heard a lot of “we” and very little “I” in the candidate’s answers. “We really nailed that product release”, “we pulled it off by doing this and that…”. You want hear enough “I” though that gives you comfort that the candidate is confident and impactful. Tough balance.
Great PMs are very hard to find. Hiring a bad PM and empowering him/her is one of the worst things you can do to an engineering team. Hire a good one and see them argue passionately about customers, great technology, great people and thrive together.