The History of Big Data

18,000 BC

The earliest examples we have of humans storing and analysing data were called ‘tally sticks’. Palaeolithic tribes people used sticks, notched with marks, to keep track of trading activity or supplies. By comparing sticks and notches, they were able to predict how long their food supplies would last.

300 BC – 48 AD

The largest collection of data in the ancient world is thought to be the Library of Alexandria, located in Egypt. Approximately half a million scrolls were stored there, covering everything humankind had learned so far, on almost every topic imaginable.

Sadly, in 48AD it is thought to have been destroyed by the invading Romans, perhaps accidentally.

1926

In a magazine interview, Nikola Tesla states that when wireless technology is “perfectly applied, the whole Earth will be converted into a huge brain, which in fact it is, all things being particles of a real and rhythmic whole … and the instruments through which we shall be able to do this will be amazingly simple compared to our present telephone. A man will be able to carry one in his vest pocket”.

1928

Fritz Pfleumer, a German-Austrian engineer, invents a method of storing information magnetically on tape.

1944

Fremont Rider, Wesleyan University Librarian, publishes The Scholar and the Future of the Research Library. Based on the fact that university libraries were doubling in size every sixteen years, he estimates that in 2040 Yale library will have “approximately 200,000,000 volumes, which will occupy over 6,000 miles of shelves… [requiring] a cataloguing staff of over six thousand persons”.

1965

The world’s first data centre is planned by the US Government. The project aims to store 742 million tax returns and 175 million sets of fingerprints on magnetic tape.

1976

Material Requirements Planning (MRP) systems are becoming more commonly used across the business world. This is the first mainstream commercial use of computers to increase the efficiency of what had previously been manual processes.

1991

Computer scientist Tim Berners-Lee submits a post on Usenet group alt.hypertext, which describes the birth of what we now know as the web.

He says: “The original idea of the web was that it should be a collaborative space where you can communicate through sharing information.”

1996

Digital storage becomes more cost-effective for storing data than paper according to RJT Morris and BJ Truskowski, in “The Evolution of Storage Systems,” IBM Systems Journal, July 1, 2003.

1998

K.G. Coffman and Andrew Odlyzko publish “The Size and Growth Rate of the Internet.” They hypothesise that “the growth rate of traffic on the Internet, while lower than is often cited, is still about 100% per year, much higher than for traffic on other networks. Hence, if present growth trends continue, data traffic in the U. S. will overtake voice traffic around the year 2002 and will be dominated by the Internet”.

2010

Eric Schmidt, executive chairman of Google, tells a conference: “Every 2 days we create as much information as we did up to 2003”.

2014

For the first time, more people are using mobile devices to access digital data, than office or home computers. Companies across the globe, such as GE and Accenture agree that Big Data analytics is essential for their strategies.

2018

The number of companies with more than 100 terabytes of unstructured data has doubled since 2016.

The terabyte is a multiple of the unit byte for digital information. The prefix tera represents the fourth power of 1000, and means 10¹² in the International System of Units (SI), and therefore one terabyte is one trillion bytes.

2020

It is expected that by 2020, the accumulated volume of Big Data will increase from 4.4 zettabytes to roughly 44 zettabytes or 44 trillion GB.

2021

By 2021 it is expected that virtual digital assistants will overtake the world population with 7.5 billion active devices.