Art of Knowledge - Episode 4

The title of this video stood out to me as I did my daily trawl through YouTube. There's so much content out there in the online space that I have doubts that I will ever be able to learn all of it. Much like the knowledge that we are in a cosmos that is vast and infinite and we do indeed look insignificant but I think that it is this insignificance that is special and that must be embraced. If life really does look insignificant then take the good out of that. The limited nature of the world we occupy, the life we live and the time we have should push us to realise that a life spent worrying is a wasted one. Focusing on negatives is  a waste and what we thrive on should be the positive, even if they are just silver linings in a cloud of negativity.

Anyway, this little video I'd like to share is all about your life's purpose. It's an amazingly simple idea that uses questions as the tool by which you can understand who you are and where you're going.

Check the video out here:

So, if after watching that video there is a disconnect between where you think you should be heading and what your answers tell you, then maybe it's time for a re-think and re-focus on what will give you the most joy.

As a bit of fun I even did it and am more than happy to share:

Q: Who am I?
A: Mark Monfort

Q: What do you love to do?
A:  I love to teach people about analytics

Q: Who do you do this for?
A: Mainly businesses but also anyone involved in data and more importantly anyone willing to learn

Q: What do they want or need?
A:  This one is a bit tougher and I think the answer will change over time especially as I gain more experience and learn. For now I have a strong sense that the people I do this for want the ability to get better insights out of their data and find the meanings and patterns there that will deliver the most value to them.

Q: How do they change or transform as a result of what they get from you?
A: I believe they can take better actions, make better decisions, ask better questions when it comes to the decision to buy analytics tools, re-think processes they are doing now and kick-start the learning process that so many businesses need if they are to survive.



In doing something like this it all boils down to why, why, why. We become less and less enthused and curious with the world around us as we grow up and it's a shame. The status quo doesn't need to be if we can find a better way and most importantly data can teach us this.

This isn't to say that this is all about how we find life's purpose in the business world or in analytics particularly but more about how we find purpose in whatever it is that we do.

Not sure if people here are familiar with the term "Elevator pitch", but it's a useful thing to know and a hard thing to create. Whenever people ask you about what you do and you have to stop and think or even if you do say something it is long-winded (and boring), then the elevator pitch is something for you. Essentially it's a way of boiling down what you do into it's very essence and in that raw state it's quite clean, simple and surprisingly effective.

Videos like this help you create that pitch. So what's mine?

"I help people get the best insights from their data so that they can enrich their lives and the lives of others"

Something like this in your line of work, whatever it is, will definitely beg the asker to question some more. They ask you how do you do that. Here's what I say.

"I do this through helping them transform what they do from being people who work tirelessly on the data to being people who have the data work for them"

Again this begs a further question, perhaps one of why do you do this?

"I do this because I am passionate about solving problems and I believe that technology and a command over data through leading edge analytical software is the way forward"


Whether it's pitching your company, going to a job interview, or telling someone you've just met what you do, this video will definitely help.

The most important thing to do with all this though is to share, share with others and get them to share with others further. If we as a society can say what we do, and do it clearly and with passion then the world will indeed be a better place.

Art of Knowledge Series - Episode 3

In this video we see General Stanley McChrystal discuss the need for active sharing of knowledge and how it can lead to better outcomes than keeping this information behind closed doors.

It's a lesson that the US Military learned the hard way but is useful in many other facets of business especially as more and more, data becomes an asset for organisations.

General McChrystal shows how they had to change the culture, significantly and move from knowledge is power to where sharing is power. Instead of 'need to know' they moved to 'who needs to know and how do we get information to them'.

This is true for other organisations too and ultimately the benefits of sharing outweigh the danger of withholding information.

I see a huge need for more sharing of data in the private and public sector we have today and of course there is no blanket approach to doing this. Each organisation needs to work out their own system and their own audience. Information is only valuable when it gets to the people who can take the right course of action because of it.

Technology is making it easier to share knowledge and the ideas that come out of this sharing will be because organisations took those steps to make the data available.

Predictive Analytics

In this post I look at Predictive Analytics from it's history till now and discuss some of the rules that should be used when assessing potential solutions. This particular topic is important because more and more I am seeing opportunities for this type of analysis to help not just these firms but the community at large.


History of Predictive Analytics
The field of Predictive Analytics is one that is seen as a next step in the evolution of Business Intelligence software and capabilities. However, it is not a completely new idea as it was seen as a sub component of the Expert Systems arena of the 70's, 80's and 90's. Let's have a look at that history for a moment.

According to Wikipedia an Expert System is an "a computer system that emulates the decision-making ability of a human expert. Expert systems are designed to solve complex problems by reasoning about knowledge, represented primarily as if–then rules rather than through conventional procedural code....Expert systems were among the first truly successful forms of AI software" http://en.wikipedia.org/wiki/Expert_system

Some examples of expert systems and the problems they addressed are shown here

Category	Problem Addressed	Examples
Interpretation	Inferring situation descriptions from sensor data	Hearsay (Speech Recognition), PROSPECTOR
Prediction	Inferring likely consequences of given situations	Pretirm Birth Risk Assessment[34]
Diagnosis	Inferring system malfunctions from observables	CADUCEUS, MYCIN, PUFF, Mistral[35]
Design	Configuring objects under constraints	Dendral, Mortgage Loan Advisor, R1 (Dec Vax Configuration)
Planning	Designing actions	Mission Planning for Autonomous Underwater Vehicle[36]
Monitoring	Comparing observations to plan vulnerabilities	REACTOR[37]
Debugging	Providing incremental solutions for complex problems	SAINT, MATHLAB, MACSYMA
Repair	Executing a plan to administer a prescribed remedy	Toxic Spill Crisis Management
Instruction	Diagnosing, assessing, and repairing student behavior	SMH.PAL, Intelligent Clinical Training,[38] STEAMER[39]
Control	Interpreting, predicting, repairing, and monitoring system behaviors	Real Time Process Control,[40] Space Shuttle Mission Control[41]

http://en.wikipedia.org/wiki/Expert_system

So what happened to these systems? In as much as they provided huge amounts of competitive advantage they also had disadvantages at the time. One of the most common problems known as the of knowledge engineering problem. Expert systems, especially the ones solving complex problems such as for credit card fraud detection required huge investments in the form of experts and their time. These experts would build the required rules so that these expert systems could be effective. Not every organisation could afford to maintain such a resource. Additionally, the ability to maintain and integrate such large systems along with limited technologies proved to be the limiting factor on these systems that were really ahead of their time.

In terms of making a comeback though, the expert systems can assist in a variety of areas especially with networked systems in place that can help overcome the limitations of the past.


Predictive Analytics Providers
There are a number of Predictive Analytics tools available coming from all sorts of IT software vendors large and small including those from SAS, SAP, IBM to Azavea and even open-source software R.

More here - http://en.wikipedia.org/wiki/Predictive_analytics


What can Predictive Analytics Solve?
There are a number of areas where Predictive Analytics is already being used or could prove to be quite useful. A few examples of where this can help various industries are shown below:

Law Enforcement
Companies such as IBM or Azavea (www.azavea.com) have created tools that process historical crime information along with geographic, demographic and other data such as weather information to build a forecast of criminal patterns into the future. In practice this would mean a system that tells officers to be at a certain location at a specific time frame as there is a high statistical probability of criminal activity during that time. This helps better prioritize the missions of Police Officers on the front line and has been proven to reduce crime rates in cities they've been deployed in.

Emergency Services
Similarly, Emergency Services can use models much like in Law Enforcement to provider better ability to plan for where their assets need to be located and at what times. Load forecasting capabilities in Azavea allow these services to be forecast to show volumes of incidents expected in various areas of a jurisdiction. Early Warning and Risk Forecasting can also be achieved in models such as from Azavea.

Education
In the Education sector, universities can use predictive analytics to give them forward forecasting capabilities over targeted markets such as international students. Feeding in information that can effect the choice of students from overseas to study here is important as it means that fluctuations in the Australian dollar versus foreign currencies today can have repercussions months into the future. Having tools that provide these capabilities mean better ability for universities to target various market segments and improve their load and retention capabilities.

Health
In the Health sector, Predictive Analytics can be used to improve patient care and reduce costs. Risks to patients or healthcare providers is better forecast and per-patient predictions can lead to better treatment decisions.

Furthermore, there are some more general applications of predictive analytics.

Predictive Search
Examples include personalisation of advertising to the point where it is based on your social media relationships or location or even your in-store or buying behaviour. Companies that are involved in this type of learning include the likes of Facebook, Evernote and Google. Google in this case, provides contextual information that relates to the way you interact online.


Transaction profiling
This sort of technique relies on compiled datasets of user behaviour over time and enables the software to accurately assess for fraud or credit risk within transaction systems. This is quite useful at large bank or lending organisations and is highly complex..


The goal of Predictive Analytics
The point of any report is what question it is answering and in the case of Predictive Analytics, that's all about answering 'What will happen next?'.

Vendors that have the capability to do this and more will ultimately create more value for clients than those who don't.

Furthermore, the following diagram shows the 5 stages of reporting capability. Whilst Predictive technologies are purported as the next wave of Business Intelligence features they are merely the next step in the evolution.

As shown above, the real goal is the activation and application of these predictions and embedding and employing these capabilities to the rest of the organisation. This ensures a greater ability to deliver decision
based actions thanks to better tools.

The combination of better systems and abundant data should lead to better ability for analysts to actually analyse the data than spend most of their time chasing it.



10 simple rules for getting the most out of Predictive Analytics
These have been built based on my own experience in the Business Intelligence community and from work and knowledge of very capable vendors like Azavea (www.azavea.com).

Some of the rules can relate to more mainstream Business Intelligence issues as well and a lot are common sense but as I've learned throughout my career, documentation is king!


1. Garbage in, garbage out
This is the starting point and the most important part of the whole idea of Predictive Analytics, the data you will use.

It is not simply a matter of the more data you have the better. Just because you have a Big Data source does not mean it is useful. 

The quality of the data is of utmost importance because with any system, the bad data you put in becomes the bad results you see at the end.

Systems that handle this well are the ones that have a good handle on their metadata

2. Control
"A good biz decision trumps a good algorithm" - http://abbottanalytics.blogspot.com.au/2013/11/a-good-business-objective-beats-good.html

The algorithm is only as good as the analyst controlling the software. The user cannot be taken out of the picture completely because no amount of algorithm complexity can account for the complexity of human interactions and for every 1,000 rules built into a system there are bound to be those cases that were not accounted for.

Human intervention must be possible in any system as a fail-safe of control.


3. Study the greats, and adapt
A great place to start for any firm looking to dive into Predictive Analytics is to look at how other companies and industries are solving problems using this capability. If you can find a common theme that their model addresses and relate it to yours it can save hours of time spent trying to build something from the ground up.

For example, the components that make up a Predictive solution for Law Enforcement can work equally well in an Emergency Services situation or even something else like Education. The tenet that holds them together is the idea of loading various factors of human behaviour into a solution that finds the required correlations.

Adaptive technologies are well suited to this as it would be quite restrictive to choose a model that is not configurable to work in other industries. 


4. Pictures are worth a thousand words 
It's very important to visualise the data and the innovations over doing Predictive Analytics coupled with geographic mapping is highly important. 

Whilst trendlines on graphs also give key insights, mapping the forecasting outcomes can provide clearer pictures to those who need it such as officers in a Law Enforcement.

The predictions used in these areas whilst not necessarily needing mapping, could still be better served with other optimisation techniques such as enhancements that help provide better decision analysis -


5. Think long term fix, not short term band-aid
Organisations looking at this should have a long term goal in mind for their use of Predictive Analytics and tie the goal to their overall business strategy.

This helps gain buy-in from rest of the organisation as this field of work is done best drawing on the expertise of others both within and outside of the organisation.


6. Create a feedback loop
This is necessary as it allows for the system to continuously improve itself. The feedback loop from those who create gather the data and create the data outputs to those who use it is important. This is because these interactions highlight the importance of human behaviour which is essentially what a lot of predictive solutions rely upon. It's about using human behaviour to solve the puzzle that is human behaviour and no amount of system can do this without having a proper feedback system.

In the Law Enforcement example this works when Commanders, Analysts and Officers all work together towards a common goal on certain crime types and communicate regularly to update each other of new datasets to look out for or feedback when certain predictions are not effective.

Continuously performing these checks and balances is important to be able to assess the validity of the system.


7. Data, data everywhere, not a drop to drink
The more data that can be obtained the better the predictions and this improved accuracy is important.

Newer resources of information are being created everyday and some organisations are looking at the vast amounts of data provided by collecting information from the Internet of Things (http://en.wikipedia.org/wiki/Internet_of_Things)

As this becomes prevalent and more and more items are being measured and data collected this becomes a new source of insight into behaviour.


8. Measure, measure, measure... the data and the vendor
Measuring the data is important because it relates to how accurate the system is and can lead to being able to calculate the Return on Investment (ROI) of the solution in place. It can also provide a broader idea of the opportunity cost associated with not putting a Predictive Analytics capability in place.


Equally, the software vendors must be measured too. When talking to vendors, organisations should see if they have analysis examples that show the efficacy of their solution. This proves how serious a vendor is and also shows the motivation of the vendors.

Why is this important? Because it is best to be aligned to a vendor that has similar goals whether that be public safety, or another. This shows whether they are thinking of long term solutions or just short term fixes that will eventually prove costly.


9. Connections, abundant connections
The systems you work with must be able to take advantage of new forms of connections to databases and files. So architecture styles like RESTful and open standard formats like JSON are highly important because the more connectible the Predictive Analytics software is, the more data it will be able to connect to.

Furthermore, data from various custodians or providers such as ESRI for mapping, the ABS for statistical data or even the Bureau of Meteorology for weather data must be brought in to make some of these analysis useful and more accurate. Some of these agencies already provide their own types of connections so systems that can incorporate these will be able to help your organisation get to your solution faster than anyone that has to build things from the ground up.

10. Privacy, it matters
So the big elephant in the room is the idea of privacy and when does predictive get too much. 

There is the story of Target in the US that sent out unsolicited advertisements to a woman they predicted was pregnant because of their system sighting a change in buying behaviours. http://www.nytimes.com/2012/02/19/magazine/shopping-habits.html

Privacy will always be a key theme and one that must be balanced delicately. The safety of a community and the safety of personal information must be of utmost importance to any firm that goes into this field of analysis and intelligence.

Proper procedures must be put in place to help avoid issues with Predictive Analytics and this will allow organisations 

Conclusion
The evolution of analytics has certainly come a long way but it is from ideas of the past that a lot of these ideas stem. The restrictions on hardware and software may have held them back but the pioneers of the 70's - 90's certainly knew the potential for success that smarter systems and Predictive Analytics could offer.

So as technologies continue to evolve and become more aligned to human needs the ability to better network the various sources of information to provide faster and more accurate forecasts will help transform society. 

I hope that organisations that do this, do it with the best of societal intentions in mind and maintain the long reaching goals that will help them and the community they serve. 

The world is changing because of analytics and I can certainly see it's for the better.

Special thanks to Jeremy Heffner of Azavea for his help and insights into this article.

Crunching the Budget

So the guys over at the Crawford School of Public Policy have been crunching the budget numbers and it's quite interesting to see how various family's will be affected (Click here). It definitely looks like the higher income earners aren't going to be affected all that much.

For more on the story have a look at their report in detail: Click here

Also have a look at this interactive graph of some of the data they collected:


I've also attached some more information from the ABS showing the number of households on various income brackets ($pw). Click the drop down on the top left hand side to see how various brackets have changed over time:

Statistics for a better world - Part 1

So my company Space-Time Research specialises in analytics tools for large data sets. We've partnered with many government agencies over the past few years including the Australian Bureau of Statistics (ABS).

One of our newer tools allows for this is called SuperDataHub and we've preloaded it with some ABS Statistics they've collected.

Have a look here:


Additionally, you can click on the "i" button on the top right hand side to see the meta data information about this chart.
It's interactive too. Click on each state name to turn it on or off. You can zoom into the data too by selecting specific sections.

NAPLAN.... we were warned!

I don't have kids. I do know some people who do have kids and I am sure that the subject of Naplan is a contentious issue.

I'll look at this from my point of view so please feel free to share your thoughts.

After reading something like this after seeing it on the 7:30 report it makes you wonder:  http://www.abc.net.au/news/2014-05-19/naplan-study-finds-school-testing-program-not-achieving-goals/5463004

Have we heard this before? Yes indeed, when the initial reforms were being brought in: http://www.smh.com.au/national/education/naplanstyle-testing-has-failed-us-schools-20110501-1e395.html

Anyway, I've had a focus on the education sector recently, and a great opinion that I find highly logical comes from Sir Ken Robinson (renowned educator from the UK/US)

I bet he'd have a lot to say about NAPLAN. Standardised testing is needed but when it becomes the only thing that kids (and their creativity) is measured by then we are failing our kids.

You'd probably think to yourself, why would I have any interest in this space especially coming from a data and analytics point of view. My interest stems from the learning perspective, the data helps create and ignite the connection to learning and we need to take advantage of that. It means not using the same yardsticks to measure our kids but using data to better identify ways that each can be successful.

We have to be ruthless about this and do something now, the future is on the line.

Blogs I follow

Been having issues getting these blogs I follow up on the right hand side panel so I'll post about them for now.

Below is a list of blogs and websites I highly recommend especially if you're interested in technology, government or both.

eGovAu Blog - Craig Thomler, well renowned voice in the government space is behind this blog and it's a great source of what's going on in and around technology and social media within government

Asian Efficiency - Aaron my good friend from university set this business up with a mate of his. Great at helping you get control of your time again.

Delimiter - Great blog on Government Technology news. Former ZDNet Editor, Renai Lemay is behind this one.

Lifehacker Australia - check these guys out as well as their sister sites for all things efficiency, gaming, technology etc