When it comes to business intelligence and reporting—as with all other development resources—the challenge is to balance “buy” against “build.” The reasons for building are obvious: It’s your code, and nobody knows your business requirements and software resources better than you do.
Why buy, then? We talked to several providers of business intelligence solutions, and their arguments can be summarized as saving time, saving money, creating a scalable system, and leveraging existing business intelligence/reporting resources.
The costs involved don’t all have to do with licensing costs, either. The licensing cost is actually very minor regardless of which solution you choose, compared to the actual development and resource cost that it’s going to take to put it all together and maintain it. “There are many scenarios that need to be thought through when it comes to presenting and visualizing data,” said David Abramson, director of product management at LogiXML. “It’s one thing to be able to connect to data and know what’s in your data. It’s another thing to know the best ways to visualize and report on that data, how to optimize that experience for the widest variety of end users, from your C-level folks down to your managers down to your sales force or other operations people that need to get this information to make better business decisions.”
Organizations need a business intelligence reporting tool that can be widely adopted across many offices, time zones and devices. To create a robust solution, developers often need more than just programming experience.
“I think it takes an extraordinary amount of domain expertise and not just in software development,” said Nelson Ng, principal consultant at Dundas Data Visualization. “I think that’s an assumption that needs to be debunked. It’s not just about software development. Beyond making data available and accessible, data is only valuable once people understand it. To help people understand the enormous amount of data that you’ve conceivably spent an enormous amount of money collecting and warehousing, you’ve got to know how to correctly present that data.”
This is increasingly important, Ng said, because the expectations of business users today are extraordinarily high. “There is a whole notion of consumerized business intelligence, making business applications that were historically clunky and difficult to use and bringing them up to the same caliber, polish and great design as you get from, say, iOS or Android apps,” he said. “There’s an extraordinary amount of thought that goes into doing this for BI, and it’s a very difficult standard to meet if you’re not an expert in this space.”
The only way that human beings can make sense of the extraordinary amount of data that confronts them is to actually visualize it, Ng said. And that’s why he thinks that data visualization is so important, not just for the future of business, but also for society. “There’s so much data. The only way the human brain can process all of this, all these numbers and statistics, is to actually see it laid out visually,” he said. “It’s how our brains work, and it’s been proven to accelerate our ability to understand information dramatically.”
The growing Big Data problem
A lot of the challenges that developers face when they’re trying to build something on their own are related to trying to handle the increasingly larger amounts of data generated in the world today. “Data is growing at just an astronomical rate,” said David Garnett, VP of products at Actuate. “The forecasts are something about 30% per year of data growth. We’re talking megabytes 10 years ago to gigabytes seven years ago. Now it’s not uncommon for people to have petabytes of data that they need to find insight on.”
“Lots of data back in 1993 meant newfangled relational databases and data warehouses and so on,” said Nobby Akiha, senior VP of marketing at Actuate. “It’s evolved through to data on the Web to now, with the whole phenomenon of Big Data where now we’re talking about not just structured data, but unstructured information.”
When it comes to handling Big Data within an organization, business intelligence and reporting solutions from vendors can work with Big Data. “We can leverage high-performance data engines, and we also have a huge number of features and elements built into our architecture that support working with large volumes of data,” Abramson said. “We do this through various optimized reporting techniques like data caching, data scheduling, and all the different things that really make it easy to work with accelerated volumes of data that are coming into your systems.”
Moreover, with Big Data, as well as any data really, you need to be able to present data visually to your users. “You want to have some advanced visualizations for your users because they can’t consume or make a lot of sense of all this data by putting it into grids or simple graphs,” said Richard Daley, CEO of Pentaho. “You have to have more advanced techniques.”
“You have got to be able to easily get the data that you’re trying to visualize and put onto dashboards. So, what we provide are prebuilt connectors to obviously lots of the relational databases, but now also a lot of the Big Data sources that are coming online,” Ahika said. “These Big Data sources include such things as Hadoop, Cassandra, Mongo and so on. We give you connectors that are already built, so you don’t have to build those. We make it very easy to go get the data.”
Another reason why developers should buy instead of build is that there is just so much fundamental business intelligence technology available, there’s no need to build from scratch anymore; it’s more about assembly. “I always talk about how it’s not that you’re going to develop, it’s that you’re going to assemble a solution for a specific business purpose,” said Harriet Fryman, program director of business analytics market strategy at IBM. “There are components already available that you can use as your building blocks so that you’re not worrying about things like how to graphically lay out a table with columns, for example. All of that technology already exists.”
This practice is referred to as embedded business intelligence, where developers can take parts of business intelligence vendor solutions and embed them into the software that they’re developing. “We become the BI inside of all the other software that, if those companies had the expertise, they might build their own BI,” said Brian Gentile, CEO of Jaspersoft. “But it doesn’t make sense for them to build their own because they can license it from vendors and customize it into their own software.”
Providers of business intelligence and reporting solutions put a lot of effort into making sure their products perform well when dealing with different types and amounts of data. They wring out as much performance as they can, on large or even moderate-size datasets. “We’re going through and generating the reports and dealing with memory constraints and everything else in a way that’s really efficient for the user,” said James Johnson, product manager of ActiveReports at ComponentOne. “Those are things that are often afterthoughts if you were to build reports from scratch. If a developer were to run into a problem, it would be really hard for them to overcome a lot of these issues. Real hardcore programming is involved; it’s a different type of environment from business programming that most developers are dealing with.”
Developers also have to have certain skillsets to successfully build their own business intelligence and reporting solutions, and they don’t always have the necessary expertise to do so. Also, developers who are thinking about building their own should realize that there are actually a lot of different levels of what it means to build your own. “When developers think about BI, one thing that’s commonly said is that Microsoft Excel is the most commonly used BI tool,” said LogiXML’s Abramson. “So you do have a lot people that would simply use Excel to manage and manipulate their data. You find that a lot of people who build their own BI end up doing a lot of work in Excel.”
But the challenge with Excel, he said, is that it starts to break down once you get beyond the single user, or in use cases where you need to work against more robust or sophisticated applications and data sources. That’s really when building your own involves much more diverse sets of skills, according to him. The necessary skills to know include things like working with databases and proficiency in SQL programming.
“And once you get into the actual presentation of the data, now you’re talking about more programming skills, whether it is using products like Microsoft Visual Studio and .NET to develop applications that present the data,” Abramson said. “Or if you’re working the Java world and developing in Eclipse or some other IDE and connecting to a variety of data sources. So, again, there’s certainly different levels of building your own.”
Getting the right report
Another challenge developers face comes from creating business intelligence reports. Developers typically deal with three types of reports. The first type is transactional reports. An example of this is an invoice or purchase order; that is, a report that many applications need. The second is an analytical report, which displays data such as the top ten sales people at a business, or quarterly sales reports. The third type is a page report or form report, such as tax and insurance forms.
“Why should developers buy and not build?” said Issam Elbaytam, CTO of ComponentOne. “I think that, other than simple transactional reports like an invoice, the rest of these items are simply impossible for a developer to build on his own within the scope of a business application. You have to buy these tools really; there is no other way around it. Reports are not just a simple listing of data; they involve aggregation, they involve analysis, they involve a huge function library that only exists with a commercial product.”
Another consideration is simply lack of time. From a business standpoint, it doesn’t make sense to spend the time creating and maintaining your own business intelligence and reporting solution. “If the development team is looking to build their own infrastructure to deliver content, the real thing that we want them to recognize is that maintaining IT infrastructure doesn’t really offer terribly much value to the organization in terms of business value,” said Jeff Morris, VP of product marketing at Actuate. “What are you doing with the information that your business is creating, generating or needing? When you’re babysitting servers and Web applications and Web portals, you’re burning daylight on actually finding good, intelligent, insightful kinds of information for your users. There’s a tradeoff in their time between adding value to their business or simply maintaining an infrastructure.”
