Big Data: Are you still a fence sitter?

Get more value from Big Data technologies – Use it for Small Data Analytics

[ Note : I have published this blog originally at L&T Infotech blogsite - www.lntinfotechblogs.com/Lists/Posts/Post.aspx?ID=38 ].

Big data is often defined by three Vs. – Volume, Variety and Velocity. While this definition captures the essence of Big data, it is limiting when used to define technologies that support Big data. These technologies can do much more than just handling ‘Big data.’ In fact, most enterprises can derive more value by using these for ‘small data’ analytics.
Besides handling large variety of data, these technologies provide new analytical capabilities, including natural language processing, pattern recognition, machine learning and much more. You can use these capabilities effectively for small (or ‘not so large’ data) in non-traditional ways and get more value out of this data.
Here is how –
1. Create ‘Data labs’ rather than just a data warehouse
Big data technologies provide advanced analytical environment. The focus is on analyzing the data, rather than structuring and storing the data. Such environment gives a perfect sandbox for experts to ‘experiment’ with data and derive intelligence out of it. For example, Insurance actuaries can derive specific patterns out of claims history data by linking external factors with loss events and define rules for pricing and loss predictions.
2. Don’t just predict, but adapt continuously to changing realities
Big data technologies provide machine learning capabilities that allow calibrating predictive models continuously by comparing actual outcomes with predictions.
3. Change ‘Forecasting’ to ‘Now-casting’
Big data technologies can help in analyzing large stream of data at real-time, without hampering performance. This capability can be used effectively to provide ‘real-time’ analytics. For example, Insurers can define new products that charge premiums based on real-time risk data emitted by sensors or telematics instruments, rather than traditional approach of calculating premiums based on forecasting of risks.
4. Don’t get constrained by a Data model
Have you ever undergone the pain of living with a data model that no longer supports business requirements? Well, don’t worry anymore. Most Big data technologies support ‘Open format’ and dynamic changes to data records to suit analytical needs.
5. Forget Massive data movements
In big data platforms, the data is co-located with analytical processing involving minimal data movements. Forget about those large, multi-year ETL programs.
6. Save cost with low-cost commodity hardware
Large data warehousing and MDM programs often require expensive enterprise hardware and licensing to support desired level of performance. This expense can be as large as 50% of your total cost of ownership (TCO). The big data platforms are designed to work with low-cost commodity hardware (including bursting on cloud), and most are open-sourced. This can help you slash the hardware/licensing costs significantly.
So the moral of the story is – Big data technologies provide many capabilities that make them an attractive choice for ‘small data’ analytics as well. Be innovative in leveraging these capabilities to complement your current analytics world.

Translating Business Strategy to Enterprise Architecture

I recently concluded a consulting assignment to define Future (after M&A) enterprise Architecture for a health insurance company, who acquired another company with considerable overlap in business.

While it was ‘relatively’ easier to come up future IT architecture by analyzing future needs and system overlap, it was quite challenging to present to executive board (completely non-technical with attention span of max 5 mins) and explain how exactly it maps to their business strategy. We had generated loads of detailed EA artifacts, however, challenge was to put all this together in just couple of slides and create a strong business case to move forward.

I found TOGAF’s Content Metamodel very useful in creating this linkage. I identified strategy business themes and for each business theme, and developed a view similar to content metamodel to link business strategy to required business services first, and then further to changes required in business process & IT systems.

A quick glance at artifacts is as shown -

Business Architecture – Is it IT’s intrusion into Business?

OMG’s SOA Consortium working group recently published a paper on their perspective of Business Architecture.  The paper is clearly by the IT Practitioners.  The way they define Business Architecture is as follows –

We define business architecture as the formal representation and active management of business design. Expanding this definition, business architecture is a formalized collection of practices, information and tools for business professionals to assess and implement business design and business change.

The paper advocates the active management of business design, with the same focus as that of IT.  Though it sounds good on paper, is it really practical? -  Especially when the fact is Enterprise Architecture is driven by IT. Will the social and power structure present in Today’s organization allow this?

 No doubt that we need a clear understanding and formal representation of business design, however, the comprehensiveness should be limited to suit the need, which is typically an input to IT (and not management of business).  Also, the involvement of business in driving the IT solution is critical, however, the involvement should be periodic, although frequent, and every attempt should be made to keep the overhead on business  as minimum as possible.  Attempting active management of business may be considered as unnecessary intrusion of IT into business and often, it is counter-productive. Instead, a process for periodic review of business models and refresh should be institutionalized.  The EA-IT team should, however, do the active management of IT portfolio and ensure alignment of IT investments with business goals through active governance structure.

Having said this, the paper does provide some useful information and examples of artifacts. It will be good if OMG standardizes the business architecture models and encourage tool vendors to support it. The Enterprise Business Motivation Model (EBMM) will be a great start.

Cut your losses !

I have been working with customers to help them modernize their IT systems for few years now. I have seen that customers are more receptive, if we prescribe to build over the existing systems rather than rebuilding from scratch. Though it is comforting to see optimal reuse of existing, practically, sometimes it makes better sense to rewrite than reuse, especially if the current systems are architected in a fashion that will hinder achieving the desired target state for the business. Most often, it is a psychological block that makes it difficult to move away from something that we have cared for a long time. It is like an affinity of share pundit to his or her loosing stock. They buy every time stocks go below and average out its price, but they end up losing everything at the end. It sometimes makes sense to cut your losses and invest somewhere else that is likely to give you most returns.

Success secrets of Portfolio Management

Yesterday, I had an opportunity to talk with divisional CIOs of a large consulting firm about my experiences of implementing portfolio management and CIO dashboards. One question from them kind of intrigued me – What it the success secret of portfolio management? Is it a process, tool or people?
I reflected on my previous experiences and stories that I have heard from my colleagues. Many large organizations put in millions of dollars in setting up Portfolio management tool, only to abandon it later for want of value. Almost in every case, the root cause is – lack of quality data and inability to keep up with changes in the data. Garbage in, Garbage out !
So what could be secret for the success? I think, it is simplicity of the solution. Strive for it without compromising the analysis value. More often, the portfolio management exercise often starts with selection of a tool and implementation of that tool. The tool is often comprehensive and offers to capture a whole lot of data and provides numerous reports. The question one should ask – is it worth the value? The first time, you can capture the data, but can you keep with the changes in the data? IMHO, the first step should be to decide what you must see every day, every quarter and during your annual budget exercise. Keep it as simple as possible. Develop a data capture process, simpler tools and governance process around it. Try to integrate tools to your operational systems so as to keep the data entry minimum. For example, get the operational FTE cost by integrating your tools with timesheet systems. Enhance your process and tools over the period in agile way as you start using it.
After a year or so, once you are confident of the captured data and the process, you may want to look at professional portfolio management tools that can act as your repository. That is solely to keep your future maintenance costs minimum and get in-built integration capabilities. It may appear reversed approach as you will need to migrate data to the tools, but trust me, if you know what you want from tools, often it will be simpler exercise to select, setup and maintain tools. Otherwise, tools are sure to overwhelm you and set you in undesired direction.

Bogged down with EA frameworks?

If you are one of those overwhelmed with enormity of the popular EA frameworks, take my word – don’t sweat it too much. It is enough to look at 6X6 table of Zachman framework and understand it completely. You don’t have to go for Zachman conferences (unless you want to get thoroughly entertained. He is a great speaker!)

Don’t get me wrong. I like these frameworks as they give you idealistic view of what all you should consider for your EA exercise. Use these frameworks as reference, but never take on ‘all encompassing’ exercise as prescribed by these frameworks. It will only end when you run out of money.
Take a step-by-step approach. I would like to refer to the disciplines mentioned by Scott Ambler in his Enterprise Unified Process
Define what is important for your organization and only concentrate on those disciplines.

Typically, the EA exercise can be divided into 2 parts –
1) Define Enterprise Architecture and IT strategy roadmap
Here you concentrate on following disciplines –
· Enterprise Business Modeling ( Current state & desired state)
· Portfolio Management
· Enterprise Architecture ( Current state & desired state)
· Strategic Reuse
(Assuming that you have decent software development processes already in place. If not, you will have to include process definition discipline as well. )

Use these models to define the gaps / opportunities with your current IT and define the roadmap for implementation by dividing the work into manageable projects and also prioritize the work depending upon your budget and resource availability.

2) Implementation
Once you have identified smaller, manageable projects, apply the development and support disciplines (as defined in EUP) to each of this project and take it further.

I know, it is easier said than done. If you are doing it for the first time, make sure to have experienced resources on your side – especially those who have failed and learned.