Lean Machine
Chris Carter
Aug. 6, 2014
Lean Enterprise
Over the last 8 weeks, Tom D and I (Chris) have been taking a break from our regular design and development day jobs. We joined forces with Toby, the product owner from Naturejobs, and Paul at Lean Mammoth to identify potentially worthwhile features for the users of the popular science jobs platform using Lean principles and methods.
New features
The process was initiated by the compilation of a list of possible improvements, additional features and completely new innovative ideas that had been amassed over the previous months within the company. In order to make the most of our time, we picked two potential improvements on which to concentrate. One was an evolutionary feature to be added to their existing site and the other an innovative idea which would allow Naturejobs to diversify their offering.
Initially we concentrated solely on the evolutionary stream, focusing our customer segment on the employers who post jobs on the Naturejobs platform. We generated a few of the possible “problems" that we thought they might have been facing and undertook an "experiment” involving numerous phone interviews with existing Naturejobs users exploring a particular problem topic. The experiments were structured in the form of a specific hypothesis with one vital assumption and the point of the experiment was to test the hypothesis by proving or disproving that assumption. In order to keep track of our experiments we used the Javelin Experiment Board.
After many hours on the phone, 4 full experiments and 32 interviews, we hit a wall. The targeted customer segment simply didn't have a problem with the experience or the service provided by Naturejobs! Apart from the very occasional gripe with minor interface issues, users were generally very happy with the response to their jobs posts and the speed in which they managed to fill their roles. At this point we decided to take a step back and look more at the business as we were struggling to identify employer problems to solve.
The site provides a number of paid upgrades to job posters which aim to improve the visibility of their posts in various ways. Looking back at the copious notes which we had taken down from each individual interview it seemed that upgrades were being overlooked. Uptake of these upgrades was very low and we decided that it was time to change our problem focus from users to the business model.
Business model
The next phase of the process involved some investigation into how we could monetise the platform better. We came up with a number of new payment models and picked one which we thought would have the most traction with users. We boldly knocked up a fake paywall in a matter of hours using A/Bingo which we put in front of 50% of users over the course of a week. This would allow us to gauge users' natural interest in paying for the service.
Of course it was possible for us to have called up more employers instead, making no changes to the site. However, we decided that testing their behaviour when faced with the real deal was a far more accurate representation than testing whether or not they were willing to pay for a service on the phone. Getting feedback from them was key to whether to not we would continue developing and refining our business model ideas.
During the latter half of the project, whilst waiting for results from our live-on-site testing, we approached the other main user base for the site: jobseekers. What products and features could we offer them to make their life better? We journeyed down an innovative path with an idea that we hoped would provide specialised careers advice to PhD and Postdoc students looking to move into industry. Interviews were done with students from a number of cities (and countries!), and we built a base on which to conduct further experiments.
Fake it 'til you make it
After a surprisingly high click-through rate we spoke to a subset of the users who had been exposed to the paywall. This information was easily accessible thanks to the dashboard which had been provided as part of A/Bingo, which we had augmented with some details of our own. We called up to around 10% of the users who had viewed the test: those who had shown an interest in spending money, those who had instantly been turned-off at the idea of paying for the service, and even those who thought that there had been some sort of mistake! Many of those we spoke to were very surprised to suddenly presented with payment options, with a few saying that they would no longer use the site if it moved to a more premium service. We also learned from customers that we needed to be more rigorous at changing wording across the whole site if running a similar test again; many users saw conflicting information on various pages which caused some confusion.
It's quite an incredible achievement to have prepared, launched and analysed the results of a live feature-fake test on a long-running website, with senior executive backing in under 7 days. When we started, we had no idea we'd be questioning something as important as the business model!
So what this project teach me? Nature Publishing Group (the organisation behind Naturejobs) started the project with a vision to create a specific new feature for employers posting jobs on the site and within a few days - and only tens of interviews - it was clear that this new feature simply was not something users were interested in and that if it were provided, they certainly wouldn't pay for it. They say "time is money" but in this case we have managed to save thousands in development costs in so little time, avoiding building a feature that interviews revealed would have been massively under-utilised. It almost seems like common sense when you look back and think about it; validating that your customer wants and will pay for your product is a crucial step that should be taken before spending any sum of money on development time.
Lean Startup
During the course of the Naturejobs project, I also attended the Lean Startup Machine event over the weekend of 18-20 July, along with Tom S, Claire, Richard S, Martyn, and Paul (again!) from Lean Mammoth. With a strap-line of "Learn to build a successful business in three days", the workshop is designed to take a simple idea through to a business in an incredibly short amount of time. It consisted of a number of workshops to help us validate customer problems and their potential solutions, improve our customer development skills, and produce low-fidelity prototypes to put in front of users.
The weekend started with around 20 pitches from those who had come with a pre-existing idea in mind. Participants (around 60 in total) then split up into about 12 groups, with each taking on one of the pitched products. Most of us who went from Unboxed ended up in different teams, which added a great sense of competition! We spent the rest of the Friday evening getting to know our teams, listening to a number of case studies and informative workshops, and designing our first experiment. This would allow us to get out of the building and out onto the streets of East London first thing on the Saturday morning.
The next day we spent a number of hours out around Whitechapel, Bethnal Green and the City, chatting to users out in public. I had joined an idea that had come from our team leader Luke (@Vitamonth), which was aimed at trying to guilt people into avoiding their junk food vices. Trust me, it's not easy trying to talk to unprepared members of the public about their dietary habits!
By the end of Saturday, we'd conducted somewhere in the region of 50 interviews, and narrowed down our target market from those actively eating junk food, through those who are much more health-conscious, and eventually settled on people who have specifically had problems with dieting in the past. As we felt that we've found a fit between our problem and our customer, we switched into looking for solutions. Using POP, we managed to create a very basic MVP on our personal mobiles within half an hour, so that we could go and test it with users out on the street again. We also got a landing page up really quickly using QuickMVP, and used our own social networks to expose it to as many people online as possible.
Sunday morning had us back out on the streets again, showing our mobile app prototype to anyone we could, getting their feedback on how it made them feel and what could be done to improve it. Once we'd spoken to enough people, we returned to our designated room to create another experiment to test on some more unsuspecting participants.
At the end of the weekend, each team was required to present their journey and reflect on what we had achieved. In the space of 48 hours, we had converted a simple idea, a Javelin board, and some post-it notes, into: - around 70 interviews, all face-to-face on the street - 29 email addresses for people we could contact if the app was developed - one landing page, and two versions of an interactive MVP
The high point of the weekend however, was actually convincing two members of the public to part with real money, in exchange for a free beta version of the app!
Although it was an exhausting weekend, it was a great experience. We'd taken an idea, refined our customer down to a specific group of people, and even shown very early versions of it to real people. We validated that people would be willing to pay for the product, and avoided adding credence to the statistic that "up to 98% startups fail". A great video highlighting the event's activities can be found on YouTube, and a big thank you to @koomerang for organising such an insightful weekend!
Conclusions
I thoroughly enjoyed both experiences, and it was a great opportunity for me to develop some skills that I previously lacked - mainly those involving speaking to real people! As a developer during other projects, it was certainly a big change of pace and refreshing to be part of a small team without the siloing effect of having certain team members with fixed roles based on their traditional skill sets.