Over recent years big data and its analysis has grown to become a major business resource. Many companies are involved in some form of big data initiative; in the 2013 Big Data Opportunities Survey it was reported that this applied to 43% of respondents across small, medium and large enterprises.
However what is important to enterprise isn’t the creation of in house big data expertise. What generates wealth is the application of the results. Here we will review big data analytics and make the case for outsourcing the activity which for most business is the most realistic and cost effective solution.
Big data is no longer the future; it is now a part of our everyday lives. Every year the amount of data, its variety and its velocity that organisations need to handle and deal with breaks new records. This huge growth in data has created many challenges in both understanding it and unlocking the potentially very valuable information that resides within it. The process of discovering that information and applying is called big data analytics, and its potential is huge.
Essentially big data analytics is the process that involves the discovery of the hidden patterns, correlations, and other information that resides in this huge quantity of data and applying it to businesses in order to provide better solutions.
The field of data analytics is vast and may involve many different disciplines. The reason is that big data is so big that the usual ways of analysing databases using such relational database management systems such as SQL are no longer possible. The problem is exacerbated by the fact that generally the data arrives in many different formats and stored in many different locations using different technologies.
For instance data might include the text from many web pages, blogs and social media which are updated on a daily basis; images that are generated from surveillance cameras; emails and messages; financial transactions and more. Much of this data, perhaps most of it, is unstructured and multidimensional. Without the use of high performance big data analytics it would be impossible to process.
One of the first challenges of big data analytics can be the choice of what data to retain and what data to discard. This is clearly exemplified at CERN (The European Centre for Nuclear Research) where the data generated by the LHC (Large Hadron Collider) generates far too much data to even store; most of it is discarded. The amount of data that is saved and made available to scientists around the world amounts to around 25 petabytes a year; and that is after 99.999% of the generated data is discarded.
It is likely that much of the data collected by CERN will take a decade or so to analyse, which is the way with big science, but isn’t relevant to business where big data needs to be analysed rapidly. This requires ultra-high performance data mining, predictive analysis, and many other beg data analytic techniques.
Part of the challenge involves discovering the most efficient ways of capturing and managing the increasing amount of data that is being generated, but many organisations already have huge amounts of it already stored on their computers that is not being analysed, or putting it another way, from which value is not being extracted. Many of these huge databases are potential goldmines however they are being ignored through the lack of access to big data analytics.
The two challenges of big data are easily stated:
• How it can be captured and stored.
• Analysing it to create value.
The first of these add costs to a business, while the second uses big data analytics to refine that raw material into useful information that creates value for the business.
Application of Big Data Analytics
Big data analytics is used on many different fields. Examples of these are:
• Marketing and Sales where the technology can be used to target the right customers using predictive modelling; discover patterns of customer behaviour to identify customer attitudes; and improve customer services.
• Finance functions where budgets and forecasts can be generated using predictive and content analytics.
• Creation of alternative business models and analyse data to create new products and services.
• Risk management can be taken to a new level and the cost of risk minimised by the application of predictive models.
• Utilise the data generated by the Internet of Things to drive innovation.
• Optimise supply chains and in time on site deliveries with real time big data analytics.
Technologies and tools
Big data analytics is a multidisciplinary field that relies on a large range of different technologies in order to provide results within reasonable time scales. New vendor applications and platforms are becoming available and applications are becoming more visual, easier to use and business friendly.
For instance Hadoop is an open source software library framework which enables the distributed processing of big data sets on computer clusters using basic programming models and can handle up to thousands of servers and the Hadoop toolset is growing rapidly.
Outsourcing Big Data Analytics
While businesses need the benefits that big data analytics are able to offer, they certainly don’t need the hassle involved in its implementation.
Few companies have the knowledge or experience that is need to put together a big data analytics team, and experts in the field are hard to come by; there is an industry wide shortage of data analysts with the required expertise and experience. Furthermore, undertaking this is not a core activity, what is core is the application of the results to driving the business.
Progressively more organisations are discovering that the solution is outsourcing the work overseas and in particular to India where the traditional skills in statistics and mathematics are ideal building blocks in the development of big data analytics expertise.