He did a very thorough test using Google to drive traffic to his test website along with various A/B tests on pricing.
His conclusion was there wasn't a viable market. Tell that to SendGrid and their various competitors ;<). I've done the very same thing with several sites and was always left wondering was it the test or my idea?
My conclusion is that to do lean successfully you also need to get off the web and get out and talk with potential customers face to face. Sometimes if you listen carefully they will hand you an even better idea.
Yes! That's part of why "get out of the building" is such a popular Lean Startup mantra. More generally, I tell novices to cycle between qualitative and quantitative learning.
For example, I recently had an idea, so I did a bunch of interviews with potential users. That gave me enough information to form a quantitative hypothesis: if I offer users app X, then Y% will use it regularly over a two week period.
Whether that fails or succeeds, my next step will be to talk with users. If it succeeds, the interviews will tell me where to take the product next. If it fails, interviews will help me distinguish between a fatal failure (e.g., nobody will ever use this) or a fixable one (e.g., minor UI issues make friction too high for frequent repeat usage).
>My conclusion is that to do lean successfully you also need to get off the web and get out and talk with potential customers face to face. Sometimes if you listen carefully they will hand you an even better idea.
Can't repeat this enough. TALK to your (potential) customers. There's just too much to learn from them to ignore this step. Not knocking on the mostly valid assumptions one can make off a/b experiments, but to tweak your product to its best potential, you really should hear and see your customers react, interact, and critique your product.
Agree completely. We did run a number of phone and face to face interviews, but in hindsight I feel we didnt reach out to the right demographic. This is something we could have done more of. Thanks for the feedback.
Saw a video last night by Jeff Lawson of Twilio about a lean test he conducted on an email sending API. It's starts at the 9:30 mark:
http://vimeo.com/16306094
He did a very thorough test using Google to drive traffic to his test website along with various A/B tests on pricing.
His conclusion was there wasn't a viable market. Tell that to SendGrid and their various competitors ;<). I've done the very same thing with several sites and was always left wondering was it the test or my idea?
My conclusion is that to do lean successfully you also need to get off the web and get out and talk with potential customers face to face. Sometimes if you listen carefully they will hand you an even better idea.