Wednesday, September 12, 2012

Get more value from Big Data technologies – Use it for Small Data Analytics

[ Note : I have published this blog originally at L&T Infotech blogsite - ].

Big data is often defined by three Vs. – Volume, Variety and Velocity. While this definition captures the essence of Big data, it is limiting when used to define technologies that support Big data. These technologies can do much more than just handling ‘Big data.’ In fact, most enterprises can derive more value by using these for ‘small data’ analytics.
Besides handling large variety of data, these technologies provide new analytical capabilities, including natural language processing, pattern recognition, machine learning and much more. You can use these capabilities effectively for small (or ‘not so large’ data) in non-traditional ways and get more value out of this data.
Here is how –
1. Create ‘Data labs’ rather than just a data warehouse
Big data technologies provide advanced analytical environment. The focus is on analyzing the data, rather than structuring and storing the data. Such environment gives a perfect sandbox for experts to ‘experiment’ with data and derive intelligence out of it. For example, Insurance actuaries can derive specific patterns out of claims history data by linking external factors with loss events and define rules for pricing and loss predictions.
2. Don’t just predict, but adapt continuously to changing realities
Big data technologies provide machine learning capabilities that allow calibrating predictive models continuously by comparing actual outcomes with predictions.
3. Change ‘Forecasting’ to ‘Now-casting’
Big data technologies can help in analyzing large stream of data at real-time, without hampering performance. This capability can be used effectively to provide ‘real-time’ analytics. For example, Insurers can define new products that charge premiums based on real-time risk data emitted by sensors or telematics instruments, rather than traditional approach of calculating premiums based on forecasting of risks.
4. Don’t get constrained by a Data model
Have you ever undergone the pain of living with a data model that no longer supports business requirements? Well, don’t worry anymore. Most Big data technologies support ‘Open format’ and dynamic changes to data records to suit analytical needs.
5. Forget Massive data movements
In big data platforms, the data is co-located with analytical processing involving minimal data movements. Forget about those large, multi-year ETL programs.
6. Save cost with low-cost commodity hardware
Large data warehousing and MDM programs often require expensive enterprise hardware and licensing to support desired level of performance. This expense can be as large as 50% of your total cost of ownership (TCO). The big data platforms are designed to work with low-cost commodity hardware (including bursting on cloud), and most are open-sourced. This can help you slash the hardware/licensing costs significantly.
So the moral of the story is – Big data technologies provide many capabilities that make them an attractive choice for ‘small data’ analytics as well. Be innovative in leveraging these capabilities to complement your current analytics world.

Friday, January 20, 2012

Big Data - A solution in search of a problem !

What does Big Data solve in Insurance, that cannot be really solved by traditional technologies? This one seemingly simple question generates a good deal of brainstorming. Let’s keep Health insurance aside (that’s easy) and think about P&C and Life insurance space.

Where is the big data ?

Number Uno is Social data, ever growing and less contextual, but BIG it is.
Then we have policy data over years. We, of course, have a loss history of several years.
We have external risk data sources.
Few companies may also have real-time data streams from Cars (PAYD or commercial fleets), factories etc..

What can we do with Big Data technologies?

Sure we have many problems.
First, we need to know customers better.
How many times we tell them that you can save $400 by switching and then when they ask for a quote, we provide a quote more than their current outgo.
Do we congratulate them when they have a new baby arrival at home?
Do we know that they are looking to buy a car?
With big data, we will be able to co-relate all the seemingly unrelated data sources and link them to derive the actionable intelligence.

Another application is Fraud detection. Some people are out of bars, just because we cannot practically spend time and energy to figure out their fraud. Big Data technology makes is possible and simpler.

What about Risk Analysis? Sure! More the data you have, more you know about your customers, you are likely to predict the risk better.

The real advantage
To some extent, we are doing all this with current traditional technologies as well. More the data, Merrier it is, so big data technologies will definitely help, but is that all?
In my opinion, the real advantage of Big Data is to find the problem (or opportunity) that you do not even know about. When a data scientist dives deep into data and finds patterns and co-relates, there will be an Eureka moment, that will provide you the real ‘intelligence’ hidden in this data. Indeed, Big Data is a solution in search of a problem!