The hackathon management team needs to have real-time visibility into what people are doing – either recording the sessions or via “feet on the ground” – when in-person hackathons come back. The managers should anticipate and prepare for questions related to the hackathon tasks but should also “hold back” guidance to make sure they don’t interfere with the process they are trying to test.
For all hackathons, prepare a way to measure results. Results come in two flavors, supervised and unsupervised metrics. Unsupervised metrics include basic system metrics, such as request latency, error rates, etc. Supervised metrics include data collected from the participants as well as more qualitative feedback, such as time to complete each step, individual videos of use-case execution, comments, complaints, and exit interviews.
The first hackathon should be small. Consider hackathon #1 to be your initial product focus group. You may have the most amazing back-end technology, but that’s pretty useless if no one can leverage it. The focus of the first hackathon should be usability.
The task should provide a “sample” of what the participants should expect to accomplish at the end. Can they get there? Is the product easy to use? Difficult? Was a user able to achieve what the UX manager set out to do?
The second hackathon needs to consist of a large crowd, the bigger the better. Again, make it simple by asking them to accomplish a specific task, but more complex than the first one. One goal of the second hackathon is to test performance. If it slows down when only an internal group is using it, degradation of performance will be an even greater issue when it’s being used by the “general” public.
Outcomes are tested during hackathon #3. Instead of assigning a single task, the hackathon manager needs to provide a series of objectives, without going into detail what the end products should look like. The results then need to be examined to make sure the teams could accomplish the individual objectives.