You think you have a great product. Your product manager thinks you have a great product. Your developers think they have created a great product. The question is – how do you prove this before you send it out to your alpha and beta testers for real-world feedback? 

Therefore, we recommend the multistage hackathon approach to ensure product-market fit and usability. With multistage hackathons, you can start them earlier than the “final product” stage to get more useful feedback. While the “final product” stage is not as well defined in our days of agile development and CI/CD, we’re defining “final product” as something that is generally agreed upon to be ready for market launch.

RELATED CONTENT: How to coordinate an exciting and productive hackathon

Nevertheless, using a series of hackathons can make it easier to verify that you are solving the customers’ problem that you intended to solve. What you think you accomplished in the lab isn’t always the case in the real world. Use hackathons to inject a bit of the “real world” in the development process.

You want to have at least three hackathons for three main reasons: 1) You won’t catch everyone in a given day. 2) You won’t catch everything in a given day. 3) You need time to iterate and incorporate feedback. 

Individual preparation

Hackathon #1 needs to focus on the use-case level. For example, you want someone to test a car by driving to a specific location. During hackathon #1, you give them GPS and detailed instructions.

For Hackathon #2, the task is the same, but instead of GPS and instructions, you give them a road atlas and some verbal directions. Hackathon #2 is more of a guided, end-to-end test.

Hackathon #3 is a true, open-ended usability test. Hand them the car keys and tell them to get to the destination. The goal of hackathon #3 is to determine whether, without any specific guidance, the user can easily achieve the objective using the product. This allows them to spend more time exploring and comprehensively stress-testing the application.

Tasks for all hackathons

The hackathon management team needs to have real-time visibility into what people are doing – either recording the sessions or via “feet on the ground” – when in-person hackathons come back. The managers should anticipate and prepare for questions related to the hackathon tasks but should also “hold back” guidance to make sure they don’t interfere with the process they are trying to test.

For all hackathons, prepare a way to measure results. Results come in two flavors, supervised and unsupervised metrics. Unsupervised metrics include basic system metrics, such as request latency, error rates, etc. Supervised metrics include data collected from the participants as well as more qualitative feedback, such as time to complete each step, individual videos of use-case execution, comments, complaints, and exit interviews.

Hackathon #1

The first hackathon should be small. Consider hackathon #1 to be your initial product focus group. You may have the most amazing back-end technology, but that’s pretty useless if no one can leverage it. The focus of the first hackathon should be usability.

The task should provide a “sample” of what the participants should expect to accomplish at the end. Can they get there? Is the product easy to use? Difficult? Was a user able to achieve what the UX manager set out to do?

Hackathon #2

The second hackathon needs to consist of a large crowd, the bigger the better. Again, make it simple by asking them to accomplish a specific task, but more complex than the first one. One goal of the second hackathon is to test performance. If it slows down when only an internal group is using it, degradation of performance will be an even greater issue when it’s being used by the “general” public.

Hackathon #3

Outcomes are tested during hackathon #3. Instead of assigning a single task, the hackathon manager needs to provide a series of objectives, without going into detail what the end products should look like. The results then need to be examined to make sure the teams could accomplish the individual objectives.

Post-hackathon analyses

While the hackathon easily allowed for supervised metrics, the real metrics come after the hackathon is over.

How useful is the product for the long-term? While some software is completely unique, with no other options on the market, most applications have alternatives. Once the hackathon is over, the product development team needs to track usage. Did the participants continue to use the product once the hackathon is over? Is it delivering results for them? Or did they use it for the hackathon and never log in again?

Each member of the development team is working on a specific task, in a silo, during the product development process. With the hackathon, they get the opportunity to see what their peers have accomplished and get introduced to the big picture, the full end result of their work.

While the hackathons help drive success for individual products, having hackathons as a regular part of the product stress testing reinforces the big picture to the entire team.