History of Predictive Analytics
The field of Predictive Analytics is one that is seen as a next step in the evolution of Business Intelligence software and capabilities. However, it is not a completely new idea as it was seen as a sub component of the Expert Systems arena of the 70's, 80's and 90's. Let's have a look at that history for a moment.
According to Wikipedia an Expert System is an "a computer system that emulates the decision-making ability of a human expert. Expert systems are designed to solve complex problems by reasoning about knowledge, represented primarily as if–then rules rather than through conventional procedural code....Expert systems were among the first truly successful forms of AI software" http://en.wikipedia.org/wiki/Expert_system
Some examples of expert systems and the problems they addressed are shown here
| Category | Problem Addressed | Examples |
|---|---|---|
| Interpretation | Inferring situation descriptions from sensor data | Hearsay (Speech Recognition), PROSPECTOR |
| Prediction | Inferring likely consequences of given situations | Pretirm Birth Risk Assessment[34] |
| Diagnosis | Inferring system malfunctions from observables | CADUCEUS, MYCIN, PUFF, Mistral[35] |
| Design | Configuring objects under constraints | Dendral, Mortgage Loan Advisor, R1 (Dec Vax Configuration) |
| Planning | Designing actions | Mission Planning for Autonomous Underwater Vehicle[36] |
| Monitoring | Comparing observations to plan vulnerabilities | REACTOR[37] |
| Debugging | Providing incremental solutions for complex problems | SAINT, MATHLAB, MACSYMA |
| Repair | Executing a plan to administer a prescribed remedy | Toxic Spill Crisis Management |
| Instruction | Diagnosing, assessing, and repairing student behavior | SMH.PAL, Intelligent Clinical Training,[38] STEAMER[39] |
| Control | Interpreting, predicting, repairing, and monitoring system behaviors | Real Time Process Control,[40] Space Shuttle Mission Control[41] |
So what happened to these systems? In as much as they provided huge amounts of competitive advantage they also had disadvantages at the time. One of the most common problems known as the of knowledge engineering problem. Expert systems, especially the ones solving complex problems such as for credit card fraud detection required huge investments in the form of experts and their time. These experts would build the required rules so that these expert systems could be effective. Not every organisation could afford to maintain such a resource. Additionally, the ability to maintain and integrate such large systems along with limited technologies proved to be the limiting factor on these systems that were really ahead of their time.
In terms of making a comeback though, the expert systems can assist in a variety of areas especially with networked systems in place that can help overcome the limitations of the past.
Predictive Analytics Providers
There are a number of Predictive Analytics tools available coming from all sorts of IT software vendors large and small including those from SAS, SAP, IBM to Azavea and even open-source software R.
More here - http://en.wikipedia.org/wiki/Predictive_analytics
What can Predictive Analytics Solve?
There are a number of areas where Predictive Analytics is already being used or could prove to be quite useful. A few examples of where this can help various industries are shown below:
Law Enforcement
Companies such as IBM or Azavea (www.azavea.com) have created tools that process historical crime information along with geographic, demographic and other data such as weather information to build a forecast of criminal patterns into the future. In practice this would mean a system that tells officers to be at a certain location at a specific time frame as there is a high statistical probability of criminal activity during that time. This helps better prioritize the missions of Police Officers on the front line and has been proven to reduce crime rates in cities they've been deployed in.
Emergency Services
Similarly, Emergency Services can use models much like in Law Enforcement to provider better ability to plan for where their assets need to be located and at what times. Load forecasting capabilities in Azavea allow these services to be forecast to show volumes of incidents expected in various areas of a jurisdiction. Early Warning and Risk Forecasting can also be achieved in models such as from Azavea.
Education
In the Education sector, universities can use predictive analytics to give them forward forecasting capabilities over targeted markets such as international students. Feeding in information that can effect the choice of students from overseas to study here is important as it means that fluctuations in the Australian dollar versus foreign currencies today can have repercussions months into the future. Having tools that provide these capabilities mean better ability for universities to target various market segments and improve their load and retention capabilities.
Health
In the Health sector, Predictive Analytics can be used to improve patient care and reduce costs. Risks to patients or healthcare providers is better forecast and per-patient predictions can lead to better treatment decisions.
Furthermore, there are some more general applications of predictive analytics.
Predictive Search
Examples include personalisation of advertising to the point where it is based on your social media relationships or location or even your in-store or buying behaviour. Companies that are involved in this type of learning include the likes of Facebook, Evernote and Google. Google in this case, provides contextual information that relates to the way you interact online.
Transaction profiling
This sort of technique relies on compiled datasets of user behaviour over time and enables the software to accurately assess for fraud or credit risk within transaction systems. This is quite useful at large bank or lending organisations and is highly complex..
The goal of Predictive Analytics
The point of any report is what question it is answering and in the case of Predictive Analytics, that's all about answering 'What will happen next?'.
Vendors that have the capability to do this and more will ultimately create more value for clients than those who don't.
Furthermore, the following diagram shows the 5 stages of reporting capability. Whilst Predictive technologies are purported as the next wave of Business Intelligence features they are merely the next step in the evolution.
As shown above, the real goal is the activation and application of these predictions and embedding and employing these capabilities to the rest of the organisation. This ensures a greater ability to deliver decision
based actions thanks to better tools.
The combination of better systems and abundant data should lead to better ability for analysts to actually analyse the data than spend most of their time chasing it.
10 simple rules for getting the most out of Predictive Analytics
These have been built based on my own experience in the Business Intelligence community and from work and knowledge of very capable vendors like Azavea (www.azavea.com).
Some of the rules can relate to more mainstream Business Intelligence issues as well and a lot are common sense but as I've learned throughout my career, documentation is king!
1. Garbage in, garbage out
This is the starting point and the most important part of the whole idea of Predictive Analytics, the data you will use.
It is not simply a matter of the more data you have the better. Just because you have a Big Data source does not mean it is useful.
The quality of the data is of utmost importance because with any system, the bad data you put in becomes the bad results you see at the end.
Systems that handle this well are the ones that have a good handle on their metadata
2. Control
"A good biz decision trumps a good algorithm" - http://abbottanalytics.blogspot.com.au/2013/11/a-good-business-objective-beats-good.html
The algorithm is only as good as the analyst controlling the software. The user cannot be taken out of the picture completely because no amount of algorithm complexity can account for the complexity of human interactions and for every 1,000 rules built into a system there are bound to be those cases that were not accounted for.
Human intervention must be possible in any system as a fail-safe of control.
3. Study the greats, and adapt
A great place to start for any firm looking to dive into Predictive Analytics is to look at how other companies and industries are solving problems using this capability. If you can find a common theme that their model addresses and relate it to yours it can save hours of time spent trying to build something from the ground up.
For example, the components that make up a Predictive solution for Law Enforcement can work equally well in an Emergency Services situation or even something else like Education. The tenet that holds them together is the idea of loading various factors of human behaviour into a solution that finds the required correlations.
Adaptive technologies are well suited to this as it would be quite restrictive to choose a model that is not configurable to work in other industries.
4. Pictures are worth a thousand words
It's very important to visualise the data and the innovations over doing Predictive Analytics coupled with geographic mapping is highly important.
Whilst trendlines on graphs also give key insights, mapping the forecasting outcomes can provide clearer pictures to those who need it such as officers in a Law Enforcement.
The predictions used in these areas whilst not necessarily needing mapping, could still be better served with other optimisation techniques such as enhancements that help provide better decision analysis -
5. Think long term fix, not short term band-aid
Organisations looking at this should have a long term goal in mind for their use of Predictive Analytics and tie the goal to their overall business strategy.
This helps gain buy-in from rest of the organisation as this field of work is done best drawing on the expertise of others both within and outside of the organisation.
6. Create a feedback loop
This is necessary as it allows for the system to continuously improve itself. The feedback loop from those who create gather the data and create the data outputs to those who use it is important. This is because these interactions highlight the importance of human behaviour which is essentially what a lot of predictive solutions rely upon. It's about using human behaviour to solve the puzzle that is human behaviour and no amount of system can do this without having a proper feedback system.
In the Law Enforcement example this works when Commanders, Analysts and Officers all work together towards a common goal on certain crime types and communicate regularly to update each other of new datasets to look out for or feedback when certain predictions are not effective.
Continuously performing these checks and balances is important to be able to assess the validity of the system.
7. Data, data everywhere, not a drop to drink
The more data that can be obtained the better the predictions and this improved accuracy is important.
Newer resources of information are being created everyday and some organisations are looking at the vast amounts of data provided by collecting information from the Internet of Things (http://en.wikipedia.org/wiki/Internet_of_Things)
As this becomes prevalent and more and more items are being measured and data collected this becomes a new source of insight into behaviour.
8. Measure, measure, measure... the data and the vendor
Measuring the data is important because it relates to how accurate the system is and can lead to being able to calculate the Return on Investment (ROI) of the solution in place. It can also provide a broader idea of the opportunity cost associated with not putting a Predictive Analytics capability in place.
Equally, the software vendors must be measured too. When talking to vendors, organisations should see if they have analysis examples that show the efficacy of their solution. This proves how serious a vendor is and also shows the motivation of the vendors.
Why is this important? Because it is best to be aligned to a vendor that has similar goals whether that be public safety, or another. This shows whether they are thinking of long term solutions or just short term fixes that will eventually prove costly.
9. Connections, abundant connections
The systems you work with must be able to take advantage of new forms of connections to databases and files. So architecture styles like RESTful and open standard formats like JSON are highly important because the more connectible the Predictive Analytics software is, the more data it will be able to connect to.
Furthermore, data from various custodians or providers such as ESRI for mapping, the ABS for statistical data or even the Bureau of Meteorology for weather data must be brought in to make some of these analysis useful and more accurate. Some of these agencies already provide their own types of connections so systems that can incorporate these will be able to help your organisation get to your solution faster than anyone that has to build things from the ground up.
10. Privacy, it matters
So the big elephant in the room is the idea of privacy and when does predictive get too much.
There is the story of Target in the US that sent out unsolicited advertisements to a woman they predicted was pregnant because of their system sighting a change in buying behaviours. http://www.nytimes.com/2012/02/19/magazine/shopping-habits.html
Privacy will always be a key theme and one that must be balanced delicately. The safety of a community and the safety of personal information must be of utmost importance to any firm that goes into this field of analysis and intelligence.
Proper procedures must be put in place to help avoid issues with Predictive Analytics and this will allow organisations
The evolution of analytics has certainly come a long way but it is from ideas of the past that a lot of these ideas stem. The restrictions on hardware and software may have held them back but the pioneers of the 70's - 90's certainly knew the potential for success that smarter systems and Predictive Analytics could offer.
So as technologies continue to evolve and become more aligned to human needs the ability to better network the various sources of information to provide faster and more accurate forecasts will help transform society.
I hope that organisations that do this, do it with the best of societal intentions in mind and maintain the long reaching goals that will help them and the community they serve.
The world is changing because of analytics and I can certainly see it's for the better.
Special thanks to Jeremy Heffner of Azavea for his help and insights into this article.
No comments:
Post a Comment