Big Data, What’s the Big Deal
Companies collect more data than ever before. Ever searched for a product and later seen an advertisement for that same product on your Facebook news feed? Congratulations, you have been retargeted. It probably didn’t come as a surprise because we know that companies collect this kind of data.
Big data isn’t a special data or more data, it is how machines learn from the type of data. I was listening to a fantastic podcast, I can’t remember the name, or I would share a link, but they were talking about how machine learning is both the cause and effect of big data.
Well, lots of people think that machine learning created big data. With smarter machines, companies can get more data, but big data is also how machines learn. It’s a chicken and egg situation.
We now have algorithms that can detect patterns, and the technology is better understood and more importantly cheaper than ever before.
Big Data will only get more important in time. Let’s look at 7 facts you should know about big data.
1) Every 2 days we create as much data as we did from the beginning of time until 2003
The full quote is:
There were 5 Exabytes of information created between the dawn of civilization through 2003, but that much information is now created every 2 days – Eric Schmidt, CEO of Google at Google’s 2010 Atmosphere Summit.
A lot of people disagree with this statement, but I think that we can all agree total human data is hard to approximate. RJMetrics’ Robert J Moore, wrote a great article that explores this quote.
2) Google now processes over 40,000 search queries every second on average which translates to over 3.5 billion searches per day and 1.2 trillion searches per year worldwide
“When Google was founded in September 1998, it was serving ten thousand search queries per day” since 2006 it has served more search queries than that every second. If you go to internetlivestats.com, you can see how Google searches have been performed today. Fun fact, after Googles initial 17,000% growth rate year after year, search query growth has leveled out to around 10% – 15% a year.
3) Big Data Could Add 6 Million Jobs to U.S. Economy
Back in 2012, Gartner predicted that Big Data would add 6 million jobs the U.S. Economy by 2015. What is kind of cool, is this study by Evans Data Corporation confirmed that approximately 6 million developers are working on big data worldwide, and it has created about 8 million jobs in the U.S. This is a massive industry. An estimated $57 billion dollars was spent just on the technology used to gather big data in 2017.
4) Poor data across businesses and the government costs the U.S. economy $3.1 trillion dollars a year
This gem came out of an IBM study. Ignoring the data is a costly mistake, one that costs the U.S almost 20% of the nation’s entire GDP. Specifically, the study said that this was the cost of inferior data. How about that? A pretty convincing case for improving and expanding data opportunity.
5) By the end of 2017, SNS Research estimates that as much as 30% of all Big Data workloads will be processed via cloud services as enterprises seek to avoid large-scale infrastructure investments and security issues associated with on-premise implementations
Scale-able big data and the increasing consumerization of enterprise software is driving a trend towards user-friendly, analytics software. And processing a massive amount of data on cloud-based servers just makes more sense. Businesses don’t need a room of supercomputers with 47 cores and a half dozen GPUs connected to a version of Linux that one guy understands. Everything can be done off premises and an organized reporting interface can display easy to understand graphs and tables.
6) Facebook users send on average 31.25 million messages and view 2.77 million videos every minute
The way that we use the internet has changed, and we must adapt to collect data from the platform and devices where people are using it.
Mobile devices have overtaken desktops, Facebook has taken a bite out of Google’s search queries, and people are just navigating to big online retailers now like Amazon and Walmart. These platforms adapted by becoming masters of big data themselves. Facebook uses advanced image recognition to recognizes people in photos. You know that annoying Google reCAPTCHA, prove you are not a robot by clicking the street signs? All that information is collected and used to teach Google’s Ai how to read street signs.
7) For a typical Fortune 1000 company, just a 10% increase in data accessibility will result in more than $65 million additional net income
Right here is why companies care about big data in 24 words. Sure, it is making Ai a reality, advancing medicine, and making governments more efficient, but the fact that better data = better return on investment proves that nothing drives progress like a bottom line.
The growing interest in big data and rapidly advancing technology, including Ai, suggest that big data still has a long way to go.