THE EVOLUTION OF
ANALYTICS - Part 1
An analysis of the Business Intelligence Industry; past, present and predictions for the future.
History of analytics
There are many examples throughout history that showcase how
others have presented their data and findings to the world. The old adage that a picture is worth a thousand words rings
true. For as long as people have been analysing data, they have been using visualisations
to share their findings. Before the dawn of the computer age, these hand-drawn
graphs were highly influential in the political and national arenas of their
time. Florence Nightingale’s graphical illustration of the key
causes of mortality during war, for example, showed at a glance the deaths of
soldiers from preventable causes and led directly to improvements in military
hospitals:
Charles Joseph Mindard’s graphic depicting the Russian
campaign of 1812 showed the relationship between number of soldiers, falling
temperatures and distance by soldiers, allowing for better planning on the part
of military advisers:
William Playfair’s chart of 1821 compared weekly wages of a
good mechanic with the price of a quarter of wheat, showing the decline in buying
power of the labour force as part of his book showcasing the causes of the fall
of powerful and wealthy nations:
With the onset of the industrial age, data analytics became a
vital tool for business. From the first time management exercises conducted by
Frederick Winslow Taylor in the late 19th century to the analytics
utilised by Henry Ford’s assembly line to measure pacing of production, this
field began to command more and more attention.
As computers became more prevalent in business, further
developments here led to the creation of systems that would capture and make
use of business data such as Enterprise Resource Planning Systems, Customer
Relationship Management systems, data warehouses and a variety of hardware and
software tools to further aid the cause.[i]
The Evolution of Analytics
Analytics has since grown more and more prominent. Today
nearly all organisations have some sort of methodology to track and utilise
their data as well as dedicated roles responsible for sustaining and managing
this growth.
Analytics 1.0: Highly Scientific and only for the
larger players
As more and more businesses embraced the power and
competitive advantages that analytics could bring, it became obvious that a
deep understanding of important business phenomena gave management a better
ability to make decisions about the various processes in the organisation. It
was during this era that the Enterprise Data Warehouse began to be used to
capture information, with BI software developed to query and report on it.
BI has been the mainstream word used to describe the
organisational intelligence software packages that are used by many companies
to connect to their data. These packages come in all sorts of varieties from
the simple to configure and implement to the more complicated and powerful. Out
of this space a few market leaders emerged including IBM, SAS, SAP, Cognos and
Qlikview.
Traditionally, the characteristics of BI environments in
organisations were such that:
- The software was configured, maintained, and administered by IT
- Few users had broad flexibility to customize or create their own reports (most users were generally limited to pre-defined reports and prompts)
- The vast majority of the reports generated contained some combination of grid-style data points and basic visuals such as line graphs, bar charts, and pie charts
- The logic behind the reports was limited to what could be generated through standard SQL programming language constructs
Analytics 2.0: The Social Media revolution and more
break-throughs
The next evolution came with the advent of what is commonly referred
to as “Big Data”. Firms began to amass large amounts of internet-based social
media information in addition to their own internal data,[iii]
giving them further insight into their customers. For the organisations that
were equipped to properly analyse this data, it proved to be a valuable source
of competitive advantage. During this time innovative technologies were
created, acquired and mastered and revolutionary ways to handle the data
volumes came about in both hardware and software technology.[iv]
This includes the creation of Hadoop open software framework, cloud-based
software environments, in-memory engines and NoSQL databases to name a few.
Today, the BI system has changed and is no longer governed
by IT. Now there is more flexibility, and more users than ever are able to
explore the data, discover new insights and share the results.
·
New,
powerful BI tools breaking the restriction that users could only access the
reports they were given
o
In terms of being able to experiment with new
metrics and views of data, users are no longer constrained as they once were. Users
now have the ability to explore their data (the term “data discovery” was
coined to describe this process). All the while there are controllable limits
on what data is accessible due to hardware and security limitations.
·
Advanced
visualisations and interactive dashboards
o
No longer just bar and pie charts with static
drill-downs – charts, graphs and maps are now explorable and can be linked to
real-time data updates enabling faster insights
·
BI tools
no longer constrained to just standard SQL programming language logic
o
More and more tools now have advanced analytic
techniques including predictive analysis and are not restricted to standard SQL
logic. The use platforms like Hadoop and
Teradata expanded the types of processing that can be applied and utilised to
expand the avenues that can be explored with data.
This means the modern BI environment is no longer limited to
just standard reports provided by a small team (usually connected to an IT
department) but rather BI is becoming more of a self-service space, where
visuals and interactivity are the norm. It leads to a blurring of the lines
between IT and business users.[v]
Analytics 3.0: Empowering more users
Analytics 3.0 is seen as the next stage of the evolutionary
chart for this industry and comes about when analytics becomes ingrained in almost
all of an organisation’s actions. Regardless of whether that organisation makes
or moves or consumes things, or produces or provides services, it will have access
to information and data to report and analyse. With Analytics 3.0, the
organisation can use the power of data analytics to create more valuable
products and services.[vi]
In this era, the concepts of data discovery and exploration
become even more important factors for organisational success, leading to
greater empowerment for both internal and external users who can now “quickly
plug-in, model, and analyse new data sources while still leveraging enterprise
metadata and data”.[vii]
BI tools in general have improved and now feature self-service
capabilities. The potential benefits of a self-service model include:
- Analysts have more time to concentrate on analysing reports as opposed to preparing them
- Users are empowered to discover data themselves, rather than relying on a reports team who might not have full understanding of the data
- Usability of reports has improved, especially for non-traditional BI users
- IT workload is reduced so they can concentrate on addressing any data requests more quickly and efficiently
Data volumes increase dramatically
Recently there has been not only a staggering uptick in the
volume of data produced and collected by businesses, but also a steady increase
in the awareness of the power of data analytics.
The combined effect is that users are increasingly
dissatisfied with prescriptive reports and dashboards that are handed down to
them and that do not evolve. This is especially evident where the reports raise
new questions, and users are then unable to obtain the answers quickly enough to
take advantage of a market condition or situation.[ix]
Self-service analytics can thrive if it can keep up with
user demand and is the key towards changing the perception of software from
being a cost centre to being a fundamental underpinning to organisational
success.
Lessons from the National Statistics Offices and
the Open Data Movement
Before moving on it is important to take a look at how the
providers of the biggest self-service platforms in the world have handled the
growing demand for data and the lessons that can be learned. Governments around the world have been leading
the way in self-service analytics for some time, with the data they release via
their National Statistical Organisations (NSOs).
Many of these NSOs have been using some sort of self-service
portal to serve up the statistics they collect. This includes the Australian
Bureau of Statistics, US Census, Office of National Statistics (UK) and many
others. These portals mostly provide large amounts of data for dissemination
purposes and use by researchers, statisticians and the public, who can access
datasets if they have the appropriate accreditation.
Lessons can be learned from the way these organisations
organise the protection of private information and balance that responsibility with
their mandate to ensure that enough data is released so that valuable insights
can be gained by researchers and the like.
A newcomer to the discussion is the Open Data Movement which,
at its very essence, is about the release and dissemination of government
datasets. It stems from the ideas of “Government 2.0”, which is defined as “the
use of technology to encourage a more open, transparent and engaging form of
government, where the public has a greater role in forming policy and has
improved access to government information.”[x]
The Open Data Movement encourages
the notion that government data should be freely available to everyone to use and republish as
they wish, without restrictions from copyright, patents or other mechanisms of control. The goals of the Open Data Movement are
similar to those of other "open" movements such as open source, open hardware, open content, and open
access.
As John Wilbanks, VP Science at Creative Commons, says “numerous
scientists have pointed out the irony that right at the historical moment when
we have the technologies to permit worldwide availability and distributed
process of scientific data, broadening collaboration and accelerating the pace
and depth of discovery, we are busy locking up that data and preventing the use
of correspondingly advanced technologies on knowledge.”[xi]
Open data sites are beginning to become more and more
prominent around the world. They include the Data.Gov websites seen in many
countries including the USA, Australia and UK.
There are great benefits from this model of openness for
society and government. These include economic benefits from innovations made
using the data as well as social benefits from more transparent governments.
The ability for data to help empower users is important but
the potential for inappropriate use must be also considered. This is especially true where the release of
confidential data can lead to severe repercussions.
To combat this, companies
will usually prepare aggregated data to hide any confidential data but often
this aggregated data cannot deliver the required abundance of information
needed to gain insights.
In most environments, the time taken to protect and
confidentialise data is time consuming, so a balance must be struck between
what can safely be released and the time and costs required to prepare it.
Back to Analytics: What does all this mean?
Bearing
in mind the trend towards self-service analytics, and the lessons learned from
the Open Data Movement the next question we might ask is “where do we go from
here?”
To attempt to answer this, we might ask what are the concerns
of those involved in deploying analytical or BI solutions, and what limitations
still exist with current generation software? How would these need to be addressed
before it really does become the norm in all businesses and all industries?
MORE TO COME IN PART 2..... STAY TUNED
References mentioned above will all be added in PART 2
No comments:
Post a Comment