The democratization of business intelligence

Waiting three months for a custom business intelligence report used to be business as usual, but today it’s risky business. The real-time nature of business coupled with shifts in end-user expectations mean that business intelligence solutions are being aggregated out to more departments and users via more device types and technologies than ever before. Words like “agile” and “self-service” are gaining popularity given the glut of data and the need to understand and apply it.

As a result, developers are tasked with building solutions that not only solve business problems, but are also are capable of adapting to rapidly changing needs.

“The industry is moving away from complicated slice-and-dice tooling to just-in-time business intelligence applications that allow a broader set of users to understand and analyze data,” said Jake Freivald, VP of corporate marketing at Information Builders. “If you’re creating business intelligence applications, you need to make them simple enough for anyone to use.”

Business intelligence was originally adopted by very large, sophisticated companies where only a handful of people had access to the tools or outcomes. According to Tobin Gilman, VP of product marketing for business intelligence and enterprise project management at Oracle, a number of surveys have indicated that fewer than 20% of workers who could benefit from business intelligence actually use it.

“Today that is changing rapidly,” he said. “Business intelligence is being embraced more widely as enabled by innovations such as interactive visualization, mobile access, integrated search and packaged analytic applications that make it easier to use and more useful to casual users.”

Traditional tools were also too difficult for the average worker to use, which meant non-technical workers had to request report builds that took weeks or months to deliver.

“Before, IT had the data so it had to run reports,” said Lisa Pappas, business intelligence strategist at SAS. “Now there’s demand for on-demand reports and self-service applications.”

Although business intelligence has been around for 20 to 25 years and has evolved rapidly it still has a long way to go, according to Ian Fyfe, chief technical evangelist at Pentaho.

“Companies have lots more data [but] much less information [because] the enterprise application data is mostly ignored,” he said. “Businesses of all sizes rank business intelligence as a No. 1 or No. 2 because they’re not leveraging their data assets.”

Culturally, the democratization of business intelligence is being driven top-down and bottom-up. Because the application of information has become critical to business competitiveness, senior executives are motivated to enable informed decision-making across more roles. At the same time, users want tools that will help optimize outcomes.

“Because business intelligence is being democratized, there is more emphasis on ease of use,” said Pappas. “The average user is used to working with Microsoft Office and intuitive UIs, so developers need to consider the end users’ goals.”

Great expectations
While the Baby Boomers had had no choice but to accept the business applications they were given, those born in the digital age expect them to behave as intuitively as Facebook or Google.

“The new generation of employees can do things their peers could not do five or 10 years ago,” said Nobby Akiha, senior VP of marketing at Actuate. “This is the era of app stores and manipulating data. If you’re smart, you want them to contribute ideas because they can do clever things.”

Younger workers are not shy about voicing their expectations, and they may even attempt to craft their own solutions. JackBe CTO John Crupi said that because Generation Y’ers are script savvy, they will do whatever they need to do. Jaspersoft’s CEO Brian Gentile considers them a threat to IT. According to Pentaho’s Fyfe, people are doing their own evaluations and proofs of concept because they’re tired of waiting for IT.

That’s why self-service has become so important. Rather than waiting weeks or months to get a report that’s now out of date, users want the ability to interact with real-time information in an ad hoc manner.

“Today every department is using business intelligence as a critical part of what they do,” said Stefan Andreasen, CTO at Kapow Software. “Business intelligence is moving to self-service because if a knowledge worker gets an idea about data, they want to try it out right away. Success is about time to market. If you’re late to market, you lose.”

Akiha warns that self-service should align with user expectations. “People expect to manipulate information and content by themselves,” he said.

“Giving users a pivot table or allowing them to toggle from a table to a chart is not enough. They want to be able to do better things with it, like annotate and mobilize. Interactivity is huge.”

The trick is pleasing a broader base of users without overburdening IT or developers, which is why platforms, components, SOA, data integration engines and other efficient approaches have become so popular.

“There is a true democratization of business intelligence happening,” said Dave Weisbeck, senior vice president of solution management, business intelligence and enterprise information management at SAP. “People expect instant gratification, so you have to make it easy to get an instant answer.”

Business intelligence solutions are also finding their way downstream to small and medium-sized businesses (SMBs). While traditional business intelligence solutions have been too sophisticated and costly to be practical, SMBs nevertheless need the ability to understand and apply data. As a result, there is a growing demand for simpler and lower-cost solutions.

“SMBs have to have some sort of business intelligence solution for competitive reasons, but their adoption is based on affordability,” said Pentaho’s Fyfe.

Regardless of which solution or tools a company adopts, providing instant answers to pressing questions requires fast access to data as well as the ability to extract what is meaningful. Of course, what is considered meaningful can change, which is one reason why it is no longer practical to take a waterfall approach to business intelligence.

“Traditional business intelligence products were built on a centralized data warehouse or cube and delivered pre-aggregated answers. If the answer aligned with the question, that was good,” said Mark Lorion, VP of product management at TIBCO.

“The problem was by the time the report was delivered, the question had changed or there were follow-up questions, so the process would start again.”

To address the issue, Bobby Hull, a corporate systems analyst at technical fabric manufacturer BGF Industries, is allowing engineers to build their own applications using SAS Enterprise Guide, a visual programming tool.

“We have a home page for each location with applications [that can be accessed] and shared,” he said. “When our VP of engineering wanted data collected, it took two weeks to collect the data, put it into Excel and generate a chart. Now that he’s learned to generate an application, [he] can click and get what he wants. Engineers are now able to focus on engineering rather than being clerks putting together reports.”

The effect of expanding data sources
Business intelligence has typically contemplated data inside organizations but there is a growing need to understand and apply additional data. According to Julian Bucknall, CTO of Developer Express, because the data to be analyzed now resides inside and outside organizations, the cost of business intelligence is being driven down by more open, agile and affordable models.

“Data is cheaper to analyze because there are websites providing data about certain things like stock trading,” he said. “It’s also cheaper because we’re getting to a state where people can analyze data more easily.”

Jaspersoft‘s Gentile said the lower costs and complexity are great for smaller businesses that want the benefit of business intelligence without the traditionally high price tag.

“Open-source products cost 80% less than proprietary products,” he said. “They’re perfect for SMBs that want a Web-based consumer-like experience.”

The Web’s effect on data-driven applications is the motivation for a number of solution providers—such as Actuate, JackBe, Jaspersoft and Kapow Software—that offer business intelligence solutions and complementary products. The Web has also affected the solutions offered by enterprise applications providers, like SAP.

“Today’s world is browser-centric,” said Kapow’s Andreasen. “Knowledge workers are getting information from inside the company, subsidiaries, business partners, competitors, and also using a browser to discover things like government requirements.”

Kapow integrates data from traditional business intelligence systems and other sources, then presents them in a browser. Andreasen said one financial services customer had accounts with 300 banks. Although the company could log into the individual accounts, it wanted to integrate its account information with its business intelligence system so it could have a real-time view of its net worth.

Its original solution was to call individual banks to request an API; however, only 20 banks out of 300 responded. Using Kapow technologies, the company was able to build a programmatic interface in three months that provides a dashboard view of its net worth based on the data from the 300 banks.

JackBe is also integrating data from disparate sources. It has a real-time intelligence platform that plugs into SAP, Excel, news feeds, databases and Web services.

“The 80/20 rule applies to traditional business intelligence,” said JackBe’s Crupi. “IT does 80% and business does 20%, taking what IT gave them and dumping it into Excel. We’re flipping the model so that IT focuses on security and governance and business handles the other 80%.”

Quick application development and data integration are in line with what some are calling “agile business intelligence,” as is quick prototyping.

“Developers are looking for platforms that allow them to quickly prototype applications,” said TIBCO’s Lorion. “Some customer allows users to prototype their applications, which allows the developers to focus on the most relevant applications.”

Staying nimble is the mark of a smart company, according to Actuate’s Akiha. To keep up with the dizzying pace of business, agility and adaptability are tantamount.

“Software solutions are always adding features, and companies are continually adding users,” he said. “Because you have to be able to expand on both axes, you need agile technology.”

Speed also applies to the immediacy of data and the flexibility of the system. TIBCO provides in-memory tools that define cubes on the fly. SAP recently added real-time in-memory capabilities to the 4.0 releases of its Unified Business Intelligence and Enterprise Information Management solutions.

Whether data resides inside or outside the organization, experts say that data quality is critical since bad data leads to bad decisions.

“The old axiom, ‘garbage in, garbage out’ is very applicable to business intelligence,” said Oracle’s Gilman. “If your users don’t trust the data they won’t use the business intelligence capabilities you are providing. It is critical that the data behind your reports and dashboards is accurate, consistent and auditable.”

Beyond rows and columns
Organizations have been applying data visualization to structured data for years. Because not all data fits neatly into rows and columns, though, the problem of semi-structured and unstructured data has been getting increasing attention.

Metadata has helped add some structure; however, it has limitations.

“Metadata management may improve the quality of metadata, but if you have no corporate taxonomy, you’ll end up with competing metadata,” said SAS’s Pappas. “When 1-800-Flowers, a large customer of ours, decided it was going to sell chocolate gift baskets, it had to map the metadata [across the business units].”

One of the problems with taxonomies is inflexibility, according to Akiha. If the taxonomies are too rigid, it is hard to define new structures. Actuate acquired Xenos (a document and data transformation solution provider) in 2010 with the goal of tackling semi-structured data beyond databases and reporting.

TIBCO Spotfire derives metadata from semi-structured data, and it is able to consume the metadata generated by other providers who extract metadata from unstructured data.

Pentaho’s Fyfe said that there’s a huge push to understand the unstructured data in blogs, gaming logs and sub-transactions, such as clicks or clickpaths.

“In the past, it was impossible to do because information had to be stored in a table,” he said. “Hadoop was built by Yahoo and Google to index Web data. It enables you to store data and get intelligence out of it. The calls are raw now and you have to write Java code, [but] we have native steps that work through the Hadoop file system. You can analyze Hadoop data without going to the data mart.”

One SAS customer had a spike in returns and knew there was a lot of unstructured data stored in forms that might be useful. Using text analytics, the company was able to identify a specific piece of equipment in the warehouse that was causing the problem.

Pappas said that sentiment and social media analysis are also areas of interest because they contain untapped wisdom. At the present time companies are paying more attention to blogs, but the interest in sentiment and social media has been ramping up in recent quarters. Several other analytic solution providers concur.

SAP recently announced deeper text analysis integration in the 4.0 releases of its Unified Business Intelligence and Enterprise Information Management solutions, so businesses can monitor sentiments expressed in social media streams, blogs and e-mail. The solution uses a new multi-source, multi-dimensional semantic layer.

“Today people need to realize data sources are structured and unstructured,” said Kapow’s Andreasen. “You need structure because it’s cleaner. Structured data has been fed into business intelligence systems to get better answers. Now businesses like hotels want to understand their ratings. There’s raw text but around that is structure, like the number of stars, the time, and whether or not there are multiple [reviews]. The numeric information around the answers is important; otherwise you have abstract information.”

Jaspersoft’s Gentile said business intelligence solution providers have been focused on metadata because most digital assets have some information about them. At the same time, there is a need for linguistics tools because the need to blend structured and unstructured data will become more common.

Another method of dealing with unstructured data is search.

“The problem is diversity at scale: There is diversity in the questions being asked and diversity in the data itself,” said Paul Sonderegger, chief strategist at Endeca. “The massive proliferation of the volume and variety of data means you have applications inside and outside the firewall, third-party data and Internet sources, which are unbridled but valuable.”

Endeca has been helping online merchants deal with disparate data stored in content management systems, catalogs, user reviews and transactions, although its solutions are being applied across more verticals now. The main benefit of search is ease of use.

According to Sonderegger, search facilitates the exploration and discovery of data, which contrasts with the formal approach to business intelligence. Business intelligence uses data models and places, conforming data into those models, which requires knowledge of the data that matters.

“The difference is structured drill-down vs. unstructured discovering without lots of effort,” he said. “If there’s a high cost to getting answers to questions, then key questions won’t be answered.”

Oracle also has business intelligence search capabilities that allow users to search, retrieve and even calculate results from unstructured data sources using keyword terms. According to Oracle’s Gilman, not all business intelligence tools address search adequately and security is often missing.

“The proliferation of social networks and business collaboration capabilities will drive more applications for analysis of unstructured data in the future, especially for gaining customer and market insights,” he said.

Business intelligence and analytics: Friends or foes?
“Business intelligence” is sometimes used as umbrella term for reporting, analytics, dashboards, knowledge management, performance management, and other things. Alternatively, some make a distinction between business intelligence and analytics, particularly if they consider business intelligence synonymous with static or historical reporting.

“In the early days, business intelligence included reporting and dashboards, but it now includes data analysis,” said Pentaho’s Fyfe.

“We now have slice-and-dice OLAP, data mining and predictive analysis based on historical data. There is also data integration for ETL. ETL is part of mainstream business intelligence performance optimization.”

Business intelligence and analytics both provide historical views of data; the difference is that analytics applies algorithms that can be used for historical or predictive purposes.

“Before we were trying to understand what happened so we could figure out where we were. Now there’s a shift to forecasting based on current trends,” said SAS’s Pappas. “If I change a variable such as adding more people, I want to see what effect will it have and how can I optimize the outcome.”

Some companies are adding analytics to their existing business intelligence querying and reporting capabilities. To ease the pain of transition, SAS developed a couple of solutions, including a Microsoft Office predictive model plugin that can be used to define analytical models. It also has a rapid predictive modeler that analyzes metadata and then proposes a model.

According to SAP’s Weisbeck, U.K. grocery retailer Tesco is using analytics to understand customers’ calorie intakes.

“Analytics is one of the greatest competitive [tools],” he said. “In the modern world, we want to minimize costs, figure out how to finance a venture, and understand why and how things are best accomplished. Competitive analytics is an untapped market.”

Dealer Services Corp., an Information Builders customer, is using analytics to minimize risk. The company provides financing to auto dealerships that borrow money so they can stock their lots with cars. Dealer Services is using predictive analytics to determine which vehicles are the least risky and thus the best investments.

The City of Richmond, Virginia, another Information Builders customer, is using predictive analytics for crime abatement purposes.

“The problem with business intelligence is it doesn’t tell where the issues are. If you want to figure out the highest rate of something in the U.S. community, it provides an archeological approach so you can drill down from the U/S. to states, to counties, to towns, and then to demographics,” said Dana Rafiee, U.S. director and IT principal at Destiny Corp.

“Where I think people are going is allowing the data to tell people what’s coming into play and what’s meaningful.”

Of course, data visualization has been used to provide visual clues as to what matters or needs attention for some time. DevExpress’ Bucknall said that the sheer demand for business intelligence applications has created an opportunity for companies like his.

“Once users get used to visualizing and manipulating data, it spreads across the enterprise, which is why more people are in tune with things like the red and green zones in gauges,” he said. “Data visualization gives you a great way of looking at data like sales and visualizing how well you’re doing.”

Data visualization is also beneficial for global collaboration because workers can view the same image rather than worry about translating images or text. Although data visualization is useful and pervasive, like business intelligence it is criticized for being able to answer the “what” questions but not the “why” questions, which is where analytics comes in.

“There has been a real explosion of data visualization methods and technologies that allow people to view image maps, trees and scatter plots,” said Jeff Erhardt, COO of Revolution Analytics. “Before, you’d get average straight lines, but now people want to understand why things are moving.”

According to him, analytics provides greater insight into complex data than is possible to achieve using graphics and reporting, which is why any business intelligence initiative should include analytics.

Analytics uses the data in business intelligence systems and other sources, which is one reason it has become an element of some business intelligence “suites,” and also the reason why business intelligence and analytics solution providers are teaming up as partners or merging.

“We’re still aggregating data via dashboards and reports, but in the last couple of years, business users have been voting with their own departmental budgets because they’re not getting enough of the information they need,” TIBCO’s Lorion. “Reports and dashboards are not the problem; the problem is why the KPI is moving.”

Noah Meister, VP of IT at Boyne Resorts is using Pentaho to centralize all business intelligence and analytics across the various resorts and to optimize business processes.

“Running a four-season resort is like managing a small city, because you have the resort, retail, real estate, food and beverage, and other things to consider,” he said. “Each of the resorts is unique, and by monitoring what happens in individual locations, we can improve what happens there as well as other locations.”

Meister and his team have been able to reduce labor costs because they can more accurately plan for a season. In addition, they have optimized ticket pricing by tracking the number of tickets sold, the ticket prices, weather and holidays.

One of TIBCO’s customers, a consumer goods company, is using multiple analytical plots to determine product pricing and placement. Without the underlying algorithms, the company would simply be aggregating historical data that is displayed as an average line with an arrow, said Lorion.

“Some of the most ingenious companies deploy our dashboard and analytics but are also using predictive analytics,” he said. “By integrating predictive capabilities that correlate trends, they are able to take advantage of smarter projections that are informed by more than just a simple data aggregation.”

Business intelligence and analytics are complementary regardless of how they are categorized, so companies should have both. Reports and dashboards remain valuable, but the application of algorithms to data can yield additional insights that result in improved decision making.

Business intelligence platforms, tools and complementary solutions
Developer Express provides .NET controls, IDE productivity tools, business application frameworks, and a reporting platform. Within a few months, the company will introduce DXperience Analytics, which will include a combination of new and existing controls for building dashboard-type applications.

Eclipse BIRT was founded by Actuate, which also co-leads this open-source project. Business Intelligence and Reporting Tools (BIRT), which runs on the Eclipse IDE, provides a set of reporting and data visualization tools that are used to build information applications.

Endeca Information Access Platform enables rapid configuration and deployment of intuitive, interactive search applications that can access large volumes of data regardless of the source or location. The platform includes the semi-structured MDEX Engine, data integration and enrichment tools, and application development toolkits.

Information Builders WebFOCUS is a comprehensive business intelligence platform that combines business intelligence, performance management, advanced analytics and data integrity. It also provides a unifying integration infrastructure that accesses, reconciles, cleanses and prepares data for business intelligence use as well as SOA support for the creation, consumption and publishing of Web services.

JackBe Presto is a platform that allows power users to rapidly assemble real-time intelligence solutions in minutes or hours from any source, so workers can make better decisions from live information. Applications built with Presto can be used anywhere, including portals, SharePoint, desktops, spreadsheets, websites and mobile devices.

Jaspersoft 4 is a business intelligence suite that allows developers to quickly add business intelligence functionality to any Web application. Jaspersoft 4 introduces a new modular UI architecture as well as a new GUI for visual dashboarding and self-service features.

Kapow Katalyst Platform is a data integration platform that extracts, transforms, integrates and migrates data from virtually any presentation layer, application layer or database layer source to virtually any other source without requiring APIs.

Oracle Business Intelligence Foundation Suite is an integrated standards-based platform that speeds up and simplifies business intelligence application development. Reporting, dashboards and scorecards, ad hoc analysis, OLAP, mobile, search, and collaboration are integrated via a Common Enterprise Information Model that centralizes facts, assumptions, definitions and KPIs.

Pentaho Business Intelligence Suite is an open-source solution that includes extract, transform and load, online analytical processing, reporting, data analysis, dashboards, data mining, and a business intelligence platform that integrates with the Apache Hadoop software framework. It allows organizations to rapidly build business intelligence applications and adapt more easily to business changes.

RevoDeployR is a new Web services framework from Revolution Analytics that integrates dynamic R-based computations into Web applications. Data analysts working in R can publish R scripts to Revolution R Enterprise production-grade analytics software. Developers can then use the RevoDeployR Web Services API to integrate the results of those scripts into any application without having to learn R.

SAP Business Intelligence and Enterprise Information Management 4.0 are based on a common infrastructure, and these latest releases provide real-time analytics, in-memory computing, expanded mobile device support, and the ability to combine business and social data.

SAS provides a broad array of business analytics software, services, and tools. Originally a programming language provider, SAS also offers UIs and applications built on its fourth-generation programming language that are designed to solve business problems.

TIBCO Spotfire is an analytics platform that can be used for rapid ad hoc analysis as well as building custom analytic applications. It combines in-memory analysis, predictive modeling and an intuitive UI, so business and technical professionals can rapidly apply data-driven insights for competitive advantage.