What's big data?
Big Data is the result of tracking and quantifying anything that can be recorded and stored. This can be everything from interactions on social media sites (Facebook ‘likes’ and Twitter posts, etc.) to mobile phone messages and financial transactions. An example of Big Data would be all of the NHS patient records for every person in Britain - that’s all the doctor’s appointments and prescriptions of 65 million people!
The challenges of Big Data include capture, curation, storage, search, sharing, transfer, analysis and visualisation. Scientists have to contend with the limitations of large datasets in areas such as meteorology, genomics and environmental research.
The limitations also affect the financial industries, the pharmaceutical sector and internet searches.
The concept of Big Data is constantly changing and evolving, which means it has to be updated all the time. It remains the driving force behind many ongoing waves of digital transformations, including artificial intelligence, data science and the Internet of Things.
Big Data can be defined through the 4 Vs:
-
In the past, the key challenge was around storage. Whilst this still continues to represent a challenge, decreasing storage costs mean there are now bigger challenges, such as how to find the relevant bits within the data (search) or how do we analyse it all, so it provides some valuable insight (analytics).
-
Data is requested and streamed at an ever increasing rate. Businesses must keep up with demand so that they can deliver data in a timely manner — customers don't like to wait for services and information!
-
Data today comes in all types of formats - pictures, text, and video to name a few. Organisations are constantly looking for new and efficient ways of managing, merging and governing different varieties of data.
-
The uncertainty of data presents challenges in maintaining good quality data. For example, if 100,000 people complete a survey how can you be sure of the honesty and accuracy of their responses?
How much is big, and why is it growing?
With computers and digital comes data, and lots of it. It started with the PC - desktop computers, and then, the internet. And suddenly, we began generating extreme amounts of data.
As technology advanced, we saw the creation of mobile phones, Artificial Intelligence (AI), and most recently voice data (Alexa, Siri and Google Assistant & Microsoft Cortana). This means, of course, that Big Data continues to grow at an exponential rate.
Even before computers, we had data that was generally kept on paper. Transactions, customer records, invoices, and library archives are among those. When computers arrived, we saw a way to store that data in an organised manner — meaning it was much quicker to catalogue and was available with a simple click of a mouse.
