Business Intelligence in an era of Cloud Platforms, Data Science Algorithms, and Multiple Integration Patterns

Right now, enterprises are all about data. The companies that can leverage data effectively are having a winning edge. Data is a strategic asset. Data drives which products to choose, which business models to pursue, and which customer experiences to create. With ever increasing complex business requirements with a mushroom of tools, frameworks and platforms, the universe of business intelligence is no longer restricted to ERP and CRM insights coming out of simple data warehouse.

Common challenges often heard from C suite and similar echo the similar theme
  • I spend hours to consolidate and integrate data with common definition
  • Despite integrated view of data, our reporting is not accurate
  • With ever increasing complex choices, we are lost which tools and technologies to use
  • Am I really getting returns with my investments in newer technologies?
  • Am I right candidate for ML, AI and data science to be used in our journey of Business Intelligence?

Well, the answer which is not very palatable is – IT DEPENDS… Every problem is Unique.
Before we talk about the future of data intelligence, let us agree on goals and objectives of Business Intelligence

What are we trying to achieve out of Business Intelligence?
  • Accurate and trusted reporting so that I can be confident in my decision making
  • Business users to have self-service reporting capabilities.
  • My reports to have flexibility of rolling up (aggregations) and drill down capabilities so that I can slice and dice the data per my requirement
  • Interactive Business dashboarding capabilities for executives to have business KPIs at their finger tips
Let us compare 7 BI Patterns – Old vs New
  • Structured Data TO Structured + Semi Structured + Unstructured
  • Data Warehouse TO Data Warehouse, Distributed data, Data Lakes
  • Transactional Data TO Transactional + In Memory Data + Big Data
  • Monolithic One Time Planning TO Real time and Continuous Planning
  • Rule Based Forecasting and Budgeting TO Algorithm based Forecasting
  • ETL Patterns TO ELT + ETL
  • Point to Point Integration TO API, Microservices, and Point to Point Integration
Structured Data TO Structured + Unstructured Data

The data journey is becoming more complex as

Structured Data is data in a row and column format that would reside inside a database or data management application like Oracle, SQL Server, MYSQL, SAP HANA etc. These databases are capturing enterprise data via their ERP, CRM, HRMS and host of other applications. For web-based portals, this data can also be captured via user interactions.

Unstructured Data, now a days, comprise the majority of the data. This data generally cannot be stored in usual database like application. This data comes in different formats including audio, video, files, text, posts, PDFs. Getting this data, storing it and hence mining it to provide any meaningful insights is a huge and mammoth task.

Semi-structured Data, true to its description, is a mix of structured and unstructured data. The perfect example of semi structured data includes JSON files, XML files, HTML files etc.

In order to truly define insights out of the data, todays business intelligence demands deriving insights not just by data from transactional applications providing data in rows and columns but also mining the data which are collected outside conventional applications.

Traditional Data Warehouse TO Distributed Databases, Cloud based Data Warehouses

Data Warehouse – it is consolidated view of data with OLAP functionality designed on STAR or SNOWFLAKE schema.

The on-premise data warehouse space is gradual moving towards extinction. Most new customer data warehouses being built today are being built in the cloud (commonly called as Snowflake, AWS Redshift, Azure SQL data warehouse or Google BigQuery). Putting data repository in the cloud is simply better. It is faster, more scalable, with zero install time, you can go live in minutes and start producing insights immediately than having to wait for 6 months to provision and install data warehouses.