If you are connected to business or technology in any way, you have surely heard of big data. This article puts together the most important big data statistics you need to hone your own data strategy in 2019.
We begin with some stats on the current state of big data and analytics, follow it up with data on the growth projected for the future, and end with the most relevant statistics on the use of big data and analytics in enterprises.
As these statistics make it quite obvious, big data and analytics are not novelties anymore. They are hygiene factors that every business needs to make use of. Big data and analytics can derive insights that help cut costs, make operations more efficient, and target customers better.
Hopefully, this set of carefully compiled statistics will help you fine-tune your organization’s initiatives in this domain.
Before we continue, let us ensure we are clear on the basics. What is big data and analytics? Big data refers to the type of data sets that are too complex for traditional data processing applications.
Big data has one or more of the following characteristics: high volume, high velocity, or high variety. The increasing volume and complexity of data sets seen today are due to factors like fast-growing mobile data traffic, cloud computing traffic, as well as the rapid development of technologies such as AI and IoT.
Big data analytics is the use of advanced analytical tools, such as predictive analytics and data mining, to extract value from big data and generate insights for business. The insights gained from big data can drive intelligent decision-making in several ways.
1. Between 2012 and 2020, the digital universe will grow by two times every two years. (Source: IDC)
The digital universe is a measure of all the digital data created, replicated, and consumed in a single year. It is made up of videos uploaded on YouTube, digital movies seen on HDTVs, banking data swiped in an ATM security footage captured at airports, voice calls running through digital phone lines, and even the subatomic particles recorded by the LHC at CERN
2. How much data is generated every day? Over 2.5 quintillion bytes by the 2018 figures. (Source: Domo)
By 2020, it is estimated that 1.7 MB of data will be created every second for every person on earth. That’s over 1,000 quintillion bytes or more than 1 zettabyte of data every single day. This data is created by every swipe, click, like, purchase, search, and stream — actions that are going to grow more common as the spread of the internet widens across the world
3. In 2016, 90% of the world’s data had been created in the previous two years. (Source: IBM)
This answers the common questions like, “How fast is data growing?” A good part of this data can be used to capture insights. However, given the pace at which data is growing, it becomes critical for businesses to have a clear data strategy, which includes being clear on the tools needed to collect, manage, sort, and read useful data
4. In 2019, Twitter users send more than 500,000 tweets every minute. (Source: Domo)
Yes, that’s half a million new sets of data in a single minute! In the same time, Instagram users post over 250,000 stories, Twitch users view 1 million videos, and Tinder users swipe 1.4 million times. A quick look at the social media big data statistics shows the rate at which data is being generated by user activity. And this is not slowing down anytime soon.
5. 90% of enterprise analytics and business professionals currently say data and analytics are key to their organization’s digital transformation initiatives. (Source: MicroStrategy)
We will cover in detail the importance of big data and analytics in the context of businesses and enterprises in a later section. It is important to understand at this stage, however, that there is a growing realization that one of the surest ways to meet the business challenges of today is to harness the information hidden in the data generated all around us.
To say that the amount of data is increasing exponentially over time only means that the sooner a strategy to do this is put into place, the better it is for the business.
6. The number of firms investing more than $500 million annually in big data has grown from 12.7% in 2018 to 21.1% in 2019. (Source: NewVantage Partners)
Absolute dollar investment in big data and AI initiatives are also increasing every year. The survey findings indicate a growth in the number of firms investing between $50 million and $500 million in such initiatives — from 27% in 2018 to 33.9% in 2019. This is an indication of a significant increase in big data investments in just the past twelve months.
7. How much do companies spend on data analytics? About $187 billion in 2019. (Source: IDC)
According to the Worldwide Semiannual Big Data and Analytics Spending Guide released by IDC, of this overall worldwide spending on big data and analytics, services account for more than half, with IT services generating more than three times the annual revenues of business services. The software will account for more than $55 billion of the spending, while hardware spending will grow to nearly $28 billion in 2019.
8. Banking accounted for 13.6% of global big data and analytics revenues in 2018. (Source: IDC)
The industries accounting for the greatest share of the big data and analytics revenues worldwide in 2018 are banking, discrete manufacturing (11.7%), process manufacturing (8.7%), professional services (7.9%), and federal/central government (7.1%), with the five making up nearly half of the global revenue generation.
However, between 2018 and 2022, as per data growth projections, the fastest growth is expected to come from retail (13.5% CAGR), followed by banking (13.2% CAGR) and professional services (12.9% CAGR).
9. IBM is the largest big data and analytics vendor in terms of revenue, with $2.66 billion in 2017. (Source: Statista)
IBM’s big data and analytics income come from its presence in services, software, and hardware. The other leading vendors are HP, Dell, SAP, Teradata, Oracle, SAS Institute, Palantir, Accenture, PwC, and Deloitte
10. Data warehouse optimization remained the top big data use case in 2018. (Source: Dresner Advisory Services)
Ranking very close is forecasting, followed by customer/social analysis, predictive maintenance, fraud detection, clickstream analysis, and IoT, in that particular order. While the top two are marked important by more than 80% of the respondents, customer/social analysis and predictive maintenance have been marked important by more than 70% of the respondents.
According to current big data stats, the much-discussed IoT, a likely use case for big data, continues to be a comparatively low priority for the survey respondents.
11. Spark is the most-preferred among big data frameworks, databases, and related technologies, considered critical or important by 56% of respondents. (Source: Dresner Advisory Services)
The survey collects the respondents’ opinions on big data infrastructure awareness and adoption in 2018, with the finding that Spark tops the list of technologies. Following it are Kafka, Map/Reduce, Kubernetes, Yarn, and Google Dataflow, in that particular order. The perceived importance of these technologies varies across geographies, industries, functions, and organization sizes.
12. Amazon S3 is the most popular big data data-access method, with more than 50% of respondents considering it critical or very important. (Source: Dresner Advisory Services)
The survey includes a wide range of technologies, products, and services for indirect access to Hadoop and other related engines. Of these, big data statistics show the ubiquitous Amazon S3 scoring the highest, followed by Spark SQL, Hive/HiveQL, HDFS, MongoDB, Impala, ADLS, and Google BigQuery.
13. Among big data search facilities, Elasticsearch marks a slight lead, with more than 40% of respondents rating it critical or very important. (Source: Dresner Advisory Services)
In Hadoop, big data search facilities include indexing and natural language textual search. In the 2018 survey, Elasticsearch is closely followed by Apache Solr and Cloudera Search. There doesn’t seem to be a dominant choice in big data search, with all tools having at least some importance to the majority of the survey’s sample audience.
14. Spark MLib and Tensorflow are the top two big data analytics and machine-learning technologies, with more than 50% considering them at least important. (Source: Dresner Advisory Services)
Interestingly, data analytics statistics also show that across all categories of respondents, Spark MLib and Tensorflow score above others. Scikit-learn scores the third with at least 44% marking it as important. Following these three are H2O, Rhipe (R), Mahout, Oryx, and Myrrix.
15. Cloudera leads the pack among big data distributions, considered at least important by more than 50% respondents. (Source: Dresner Advisory Services)
The other prominent big data distributions are Amazon EMR, Hortonworks, MAP/R, and Microsoft HD Insights. Google Dataproc, IBM BigInsights, and Qubole are considered not important by more than 50% of respondents.
Big Data Growth Statistics
16. The Hadoop and Big Data market are projected to grow from $17.1 billion in 2017 to $99.3 billion in 2022. (Source: Statista)
Hadoop is an open-source software framework used for storing and processing big data in a distributed manner on clusters of commodity hardware. It enables the handling of many concurrent tasks. Its market growth at a CAGR of 28.5% in the above-mentioned period is a direct sign of the growing use of big data
17. The global big data market for software and services is estimated to be worth $49 billion in 2018. (Source: Statista, Statista)
How much is big data worth? It is forecasted to grow to more than twice this size at $103 billion by 2027. As per another set of figures from Statista, the global big data and business analytics market was valued at $168.8 billion in 2018. It was forecast to grow at a CAGR of 13.2% to $274.3 billion by 2022.
18. Big data applications and analytics segment are projected to grow from $5.3 billion in 2018 to $19.4 billion in 2026. (Source: Statista)
If considered segment-wise, the largest growth between 2018 and 2026 is expected in the apps and analytics segment of the big data market at a CAGR of 15.49%. The largest segment, however, in 2026 is expected to be professional services, which was worth $16.5 billion in 2018 and will grow to $21.3 billion in 2026. Other large segments in 2026 are expected to be storage, computation, and SQL.
19. The software segment of the big data market will be the fastest growing from 2019 ($17 billion) to 2027 ($46 billion). (Source: Statista)
According to big data facts and figures for 2019, the revenues from the global big data market, when divided by type, will see the fastest growth in the software segment — a CAGR of 13.25%. Revenues from the services segment will grow from $19 billion to $33 billion during the same period, while those from hardware will grow from $14 billion to $24 billion.
The dominance of the software and services segments over hardware in the big data market becomes clear from these figures.
20. The Chinese big data market is predicted to grow at a CAGR of 31.72% between 2014 and 2020. (Source: Statista, Beyond Summits)
In the forecast period, the Chinese big data market is projected to be one of the fastest-growing in the world. While its value stood at ¥8.4 billion in 2014, data growth statistics put its value in 2020 at ¥57.8 billion. Big data is witnessing particularly strong adoption in marketing in the Chinese e-commerce industry.
Companies like Taobao, JD, and Tencent are making use of marketing analytics for more efficient advertising and better segmentation and targeting.
21. Non-relational analytic data stores are projected to be the fastest-growing big data technology category, growing at a CAGR of 38.6% between 2015 and 2020. (Source: Statista)
The other top technology categories in terms of growth rates in the forecast period are expected to be cognitive software platforms (23.3% CAGR) and content analytics (17.3% CAGR). These are followed by search systems, IT services, and others.
22. The amount of global datasphere subject to data analysis will grow by a factor of 50 to 5.2 zettabytes by 2025. (Source: IDC)
Is big data the future? Well, big data and analytics will cease to be an important function only when humanity stops producing data. Far from this happening, considering that the data produced by humans continues to double every two years, the field of big data is only going to get more important. Even during downturns or market crashes, big data remains important to understand the underlying issues.
23. By 2020, there will be 2.7 million job postings for data science and analytics roles in the US alone. (Source: PwC)
These 2.7 million job postings represent a more than 35% growth from 2015, making this area one of the fastest growing job profiles in the US. Figures similar to these United States big data statistics are expected to hold true for other industrialized countries, thanks to the growing realization of the importance of big data analytics.
Big Data and Enterprises
24. 88% of data is ignored by companies. (Source: Forrester Research)
A widely-quoted figure from a 2012 paper from Forrester Research says that, on average, companies analyze only 12% of the available data. Reasons for this include a lack of analytics tools, repressive data silos, and the difficulty in knowing which information is valuable and which is worth leaving.
Of course, not all generated data is useful. According to an IDC report, only 22% of all data had the potential for analysis in 2012, with the figure expected to rise to 37% as per 2020 technology predictions.
25. Only 15% of organizations are currently very effective in delivering relevant and reliable customer experience. (Source: HBR)
In the HBR survey of approximately 700 business professionals, only 3% of the respondents said they were able to act on all of the customer data they collect, while 21% said they could act on very little of it. The need to close this worldwide gap in what businesses currently offer and what the customers really want is one of the key reasons behind companies making investments in big data and analytics.
26. Even in 2018, 58.5% of organizations worldwide plan to implement big data technology adoption after more than a year. (Source: Statista)
According to big data growth statistics for 2018, while the largest organizations, as surveyed by NewVantage Partners, show a greater willingness to make investments in big data initiatives, the same cannot be said for organizations of all sizes.
The global big data technology adoption rate is still quite low, with 30% of global organizations planning to do it in 2019 and only 12% in 2018, according to a 2018 survey.
The sense of immediacy had risen only slightly from 2017 and has, in fact, come down from 2016, when 61% of organizations expected to implement the plans between 2016 and 2017.
27. By 2021, insight-driven businesses are predicted to take $1.8 trillion annually from their less-informed peers. (Source: Forrester Research)
According to these data growth statistics, insight-driven businesses are customer-obsessed firms that systematically harness insights across their organization and implement them to create competitive advantage through software.
This goes beyond just being data-aware and requires CIOs to redirect the firm’s data ambitions toward effective insights and action. What’s more, such businesses are reported to be growing at an average of more than 30% annually.
28. Data-driven organizations are 23 times more likely to acquire customers than their peers. (Source McKinsey Global Institute)
Such organizations are also six times more likely to retain customers and 19 times more likely to be profitable as a result. There is mounting evidence that data-based decision-making makes strong commercial sense, providing a conclusive answer to the question, “Why is big data important?”
29. Nearly 97% of respondents in the 2018 Big Data Analytics Market Study rate big data between somewhat important and critical. (Source: Dresner Advisory Services)
Precisely 36.44% of organizations rate big data as a critical function, 28.89% as very important, 20% as important, and 11.56% as somewhat important. These attitudes vary according to geography, industry, function, and organization size.
The strongest favorable attitudes correspond to Latin America; telecommunications, advertising, insurance, and financial services; R&D and business intelligence; and organizations with 5000+ employees.
30. Nearly 60% of respondents globally claim to be using big data in 2018. (Source: Dresner Advisory Services)
The global adoption rate of big data seems to have improved consistently every year from 2015 to 2018. The adoption rates are the highest in Latin America; telecommunications, insurance, and advertising; R&D and operations; and organizations with 5000+ employees.
31. 91.6% of firms in the Big Data and AI Executive Survey 2019 confirm an increased pace in investment in big data. (Source: NewVantage Partners)
These artificial intelligence stats vary by sector. However, 95.2% of firms in financial services reported an increased pace of investments, compared to just 76.9% in healthcare. Furthermore, 87.8% of executives in the survey also report a greater urgency to invest in big data and AI initiatives, with the highest figure (91.7%) for financial services and the lowest (78.6%) for healthcare.
32. 91.7% of executives cite business transformation and agility as the primary driver for investing in big data and AI. (Source: NewVantage Partners)
Approximately 75% of executives also acknowledge the fear of disruptive forces and competitors as an important motivator for big data investments. Although firms have recognized measurable cost-savings as one of the benefits of big data, only 4.8% of executives viewed it as the driving factor for investments.
33. 96.4% of companies are investing in AI and machine learning capabilities in 2019 compared to 68.9% in 2017. (Source: NewVantage Partners)
This steady increase, as shown by data growth trends, reinforces the view that investment in AI has become nearly universal. Another technology that has seen consistent growth is cloud computing — 85.2% of companies had invested in it in 2017 and 90.5% in 2019.
This is followed by digital technologies (falling from 78.7% to 77.4%), fintech solutions (rising from 45.9% to 47.6%), and blockchain. Though blockchain investments have risen from 37.7% to 41.7%, and despite it being touted as a transformative technology, momentum is yet to pick up.
34. 98.6% of executives cite data privacy as their top data priority. (Source: NewVantage Partners)
When dealing with big data, its handling is also a concern faced by companies. With well-publicized data breaches getting common, it is understandable that data privacy is the top concern for most executives, along with cyber security being the priority for 94.3% of the executives surveyed. Data ethics has been cited by 55.7% of executives as an additional priority.
35. More than two-thirds of enterprises trying out big data initiatives have seen a decrease in expenses through operational cost efficiencies. (Source: NewVantage Partners)
What are the benefits of big data? Of the enterprises surveyed, 72.6% had begun using big data for operational cost efficiencies and 49.2% had already experienced decreased expenses as a result. Other key areas where successful results were seen included creation of new avenues for innovation and disruption, launch of new product and service offerings, and establishment of a data-driven culture in the enterprise.
36. 84.1% of enterprises have begun using big data initiatives to improve their decision-making. (Source: NewVantage Partners)
Approximately 69% of enterprises have also seen success in this area from their big data initiatives, while 36% say that this is the area of top priority when making investments in big data and analytics. As per big data statistics, decreased expenses and improved customer service round up the top-three areas seeing the greatest big data investments from enterprises.
37. Only 62.2% of executives surveyed report seeing measurable results from their big data and AI initiatives in 2019. (Source: NewVantage Partners)
This is a notable decrease from 2018 when 73.2% of executives reported measurable results. The number was 48.4% in 2017. The results of investments in big data have not kept up with the growth in investments. One of the reasons for this is that business adoption of big data initiatives and deriving of measurable business results from these investments is a multi-year journey.
38. Nearly 95% of executives cite a cultural or organizational issue as a challenge slowing down business adoption of big data. (Source: NewVantage Partners)
Even 77.1% of executives report that business adoption of big data initiatives remains a challenge. The reasons cited for these challenges include lack of organizational alignment, cultural resistance, understanding of data as an asset, and executive leadership.
As per big data stats for 2019, only 5% of executives cite inadequate technology solutions as a challenging issue.
39. The number of firms claiming to have created a data-driven organization has dropped from 32.4% in 2018 to 31% in 2019. (Source: NewVantage Partners)
Again, even as the investment has increased, the number of firms claiming to have created a data-driven organization has fallen from 2017 (37.1%) through 2019. Even 71.7% of firms report that they have yet to forge a data culture, while 53.1% state they are not yet treating data as a business asset.
40. 67.9% of firms report appointing a Chief Data Officer compared to just 12% in 2012. (Source: NewVantage Partners)
In the last few years, the CDO has emerged as the main executive in an organization with the responsibility for data-related initiatives. The 67.9% figure for 2019 is a notable jump over 62.5% in 2018 and 55.9% in 2017. Some firms are also going beyond this to establish an integrated Chief Data and Analytics Officer function.
Key Takeaways From Big Data Statistics 2019
- The amount of raw data generated around us is increasing exponentially.
- Big data becomes important when it is combined with effective big data management tools, so that insights can be derived from the mass of raw data.
- Business investments in big data and analytics are increasing steadily, and companies are also beginning to see positive results.
- However, there are certain challenges that can slow down and even nullify these initiatives, particularly those related to organizational culture.