“Quality is never an accident; it is always the result of intelligent effort.”
— John Ruskin
As creators of software, we strive to eliminate bugs to create the best experience for our users. But we can only fix problems we know about, and the more we know, the faster it goes.
All third-party mobile bug reporting platforms seek to provide timely data on the sources of crashes and bugs, but they differ in their approaches and capabilities.
Finding the right platform will depend on the project’s needs, but here is a high-level comparison of some of the best tools for iOS and Android: Appsee, Bugsee, Bugsnag, Crashlytics, HockeyApp and Instabug.
A crash report is useful insofar as it reveals what happened, where and why. In this, the major offerings differ in quality of their symbolication — properly identifying the line of code that caused the crash.
While each provides insight into which class or method experienced an error, not all with the same level of accuracy.
CrashProbe’s robust tests reveal the strength of Bugsee and HockeyApp in this area. Important to note here, that Bugsnag (ranked 3rd) data is about a year old, so update is due.
Leading up to the crash
So we see where the app crashed, but often enough, we lack the key data about how we got there in the first place. What exactly caused our code to execute that particular flow?
That’s where some of the tools provide important historical data about what preceded the crash.
Bugsee and Bugsnag also provide a readout of an app’s console log immediately before the crash. This empowers a developer to understand more about the state of the application at crash time.
Beyond mere crash reports, Appsee and Bugsee will play back an app’s interface in the moments preceding the error. While both match the event timeline to a video step-by-step, Bugsee’s videos are of higher fidelity, whereas Appsee’s seem more like a series of screenshots.
There is no substitute for the ability to witness the user’s actions that led to a crash. It can significantly reduce the time required to reproduce and diagnose a bug. Especially, if it is a rare and intermittent problem.
Both Appsee and Bugsee overlay circles indicating the user’s taps onto the video. This makes it even easier to retrace the steps that caused the error, separating them from the others.
Bugsee and Instabug characterize the network activity preceding a crash, something their competitors should strive to include. This helps to understand whether it was the server failure that might have caused the problem.
In-app bug reporting
Many bugs do not cause crashes, but diminish the experience of an app nonetheless. In addition to receiving reports and videos of crashes, Bugsee and Instabug provide users and testers the ability to report bugs manually from within the application.
In both cases, the user may upload an annotated screenshot and description of the problem. This can be triggered from a gesture or from explicit UI within the application, and is particularly useful for beta testers who are motivated to provide feedback.
It should be as easy as possible for a tester to report problems. Bugsee and Instabug excel in this regard, delivering the necessary information directly to the developer’s inbox.
These top platforms strive to make installation simple, and they mostly succeed, but differ in small ways.
Most of the solutions require to run scripts for the build phase of an application, and require the upload of a dSYM file to symbolicate crashes. This is a simple step, but can be a hassle when using dynamic frameworks, as most crash reporters only handle the main app’s dSYM, and anything further requires additional legwork.
Bugsee’s upload agent, however, tackles this in one step. Crashlytics and HockeyApp, on the other hand, provide Mac apps which walk developers through SDK integration, which can be effective to verify that installation was done correctly, but these helper apps are ultimately not necessary.
Typically, the online documentation is straightforward enough to integrate any of these frameworks into a project. Documentation for these reporting providers is generally top-notch, particularly when it comes to the basics.
Each has a website with hierarchical navigation that makes it easy to find relevant information. Where this fails, a simple Google search will do. Chances are, that a problem encountered with one of these providers will be discussed on Stack Overflow.
There can be some quirks with error reporting frameworks, however. For example, utilizing multiple offerings at once may result in one “catching” the crash before another.
Upon encountering problems, the documentation should be your first stop. Paying careful attention to the details upfront can avoid many hassles later, so review the documentation carefully before deploying.
Each of these crash reporting platforms has a dashboard where bugs are listed and described. As bugs begin to roll in for an app, its developers will need to determine which are the most important.
Some platforms, such as Crashlytics and Instabug, automatically prioritize particular crashes and problems based on their frequency and the number of users affected. The others generally allow you to filter or sort by these metrics, but do not do so automatically.
While each of the reporting platforms will integrate with other services, Bugsnag and Crashlyitcs’ offerings are the most robust.
Bugsnag offers clever, two-way data sharing with some popular project management tools. Conversely, HockeyApp’s integrations require a bit more effort, and Appsee is a bit behind the rest, but most provide a way to communicate with BitBucket, Github, Jira, Slack, or Trello.
Be sure to check the reporting platform’s documentation to ensure your desired integration is supported.
Any of the options can be tried for free. With the exception of Crashlytics, which is always free, and Appsee, which offers a 14-day trial.
The top bug reporting platforms begin to charge for their services as the number of account logins or published apps/users increase. All of the providers publish their pricing schemes on their websites, with the exception of Appsee, which requires a call to their sales department.
All services structure their pricing slightly differently, making it a bit difficult to compare. Some charge per team members, some — per number of apps, or events, or per app users.
So while your mileage may vary, typically, for a small company with a handful of apps, the monthly price won’t rise above a few hundred dollars, but larger enterprises can expect to pay upwards of $1,000.
One may wonder why Crashlytics is free, given that its competitors all charge for their services. Crashlytics was acquired by Google in early 2017, with plans to integrate it into Firebase. It is not clear whether Google intends to use the free pricing to encourage adoption of other developer-related services from Google, or perhaps its ad offerings.
Nevertheless, developers should consider whether sharing competitively useful information with a third party is worth the risk, regardless of the price.
So how do you choose the best solution? It depends on your needs.
If cost is an issue, then Crashlytics is hard to beat. However, free does not always mean the best.
Not only other tools do not compromise your usage data, they also offer a way more comprehensive suite of features. These features are quite helpful in chasing those pesky bugs in mobile apps. However, those are paid services.
Below is a handy comparison table of the aforementioned bug and crash reporting tools.