BIG DATA – what does the data say? Where did the data come from? What kind of analysis is used?

Collecting and Analysing data before taking decisions is not a new discovery. Business Intelligence gave organizations a new dimension that goes beyond intuition when making decisions. Data about product development, sales, business processes and customer experience were recorded, aggregated and analysed. Data warehouses were used to collect information and business intelligence software was used to query the data and report it.

Data volume grew rapidly which made it hard for Business intelligence companies to segregate it in warehouses and from here the concept of Big data and Analytics blew up, where new technologies were created. Big Data couldn’t fit or be analysed fast enough on a single server so it was processed with Hadoop, an open source software framework for large scale processing of data across several servers. The data itself was stored in public or private cloud and it was unstructured, so many analytic firms turned to NoSQL database.

The 2.0 version of analytics was perfect in terms of improving the internal business decisions but then it was realized that there is a big business opportunity behind that and we’re not talking only about improving the decision making process but also creating new products and services, and here came the 3.0 version where new agile analytical methods and machine learning techniques have been used to generate insights at a much faster rate.

A huge data is being processed every day on the aviation sector, around 16 Petabytes (16*10^15 Bytes) according to NASA Researches news. The importance of big data is not just a result of its volume and speed, but also the reality that data comes from trusted source. So, it’s not only having the big data that matters, its more about asking the right questions out of this big data: What does the data say? Where did the data come from? What kind of analysis is used?, etc.

18-December-2010, I would never forget this date where European Airports closed for 3 days due to heavy snow and freezing temperatures causing travel chaos across Europe; People slept for more than 48 hours in Airports. Seconds matter in airports, so using big data efficiently leads to better predictions, and better predictions yield to better decisions. What really matters is managing the right data from these Big data, so just imagine connecting all the airplanes in the global sky, feeding the data to the analytics system in the airports, and creating more valuable products and services by integrating this analytics platform with other data governance systems. If we had this system, do you think that the MH 370 flight would have been lost in the Indian ocean without any critical facts of what happened.