Few things in technology outside of catching Pikachu on your iPhone have been hyped as much as big data. However, with any trending topic, new technology or invention that’s being called the “wave of the future,” it’s important to look past the hype and analyze what’s actually going on. Outside of widespread media attention and the increasing importance placed on collecting data, big data is rarely used.
According to MIT Technology Review, only 0.5% of digital data is ever analyzed. From a business perspective, this means that millions of companies are losing significant opportunities to increase efficiency, lower costs, and target new customers because they aren’t using the data they’re collecting.
While the rate of analyzed data is discouraging, new data is being created and collected at an exponential rate. By 2020, approximately 1.7 megabytes of new information will be created every second for every human.
When exploring why big data analysis is lagging and how to fix this problem, it’s important to consider where data is being stored and crunched. While some organizations have been storing their information in the cloud for years, sensitive data is typically kept on-premises. Although secure, building an infrastructure and a massive data processing center to keep up with the amount of gathered data is inconvenient due to the incurred costs and efforts.
Data analysis needs cost-efficient, easy-to-implement cloud technology
The cloud has automated some of the heavy lifting in technology today and is cost-effective, but it has not been perfected for big data analytics. Moving large amounts of data in and out of the cloud comes with security risks and performance fluctuations, especially when dealing with terabytes, petabytes, and even exabytes of digital content. Moreover, traditional cloud solutions have yet to meet the bare minimum requirements regarding big data application integration and software orchestration. The tedious jobs of designing, deploying, fine-tuning, and maintaining big data architectures still have to be done by the clients.
Google, for example, has over 105 million active users and collects 14 different types of data including ad clicks, browser information, search queries, and more. Storing and processing this massive amount of data demands a robust and powerful solution that’s uninterrupted and consistent. Such a solution was difficult to achieve with virtualization, where multiple workloads from different companies are running on the same server. A hypervisor, which allows virtualization, also impedes big data performance by limiting processing power when being spread thin among multiple hosts. This is known as the “noisy neighbor” effect, and it has limited the cloud’s potential to faithfully serve big data because a single architecture is serving multiple customers.
Up to this point, the two primary big data solutions are either too costly and time-consuming (on-premises data centers) or unreliable, insufficiently automated and insecure (the virtualized cloud). Without a clear solution, the default has been to do the bare minimum with the available data.
Three waves of technological innovation
Significant technology innovations, including big data, typically move in sets of three waves. The first wave is infrastructure, “the cornerstone of big data architecture.”
The second wave includes the tools created to help harness the power of the technology and the third and final wave is applications. With an infrastructure in place and available tools, big data applications are being optimized for cloud use with a blend of technologies such as:
- Programming Frameworks: Hadoop, Apache Spark
- SQL Databases: Oracle, MySQL
- NoSQL Databases: Datastax, Couchbase, MongoDB
- Analytics: Datameer, Platfora, Trifacta
- Visual Analytics: Tableau, Zoomdata
The cloud is getting lighter for big data. The bare-metal cloud, also described as a dedicated infrastructure, is brightening the future of cloud-based big data projects. While the traditional cloud has performance bottlenecks and security risks, a bare-metal or dedicated infrastructure removes the uncertainty, providing predictable performance and single-tenant isolation. This eliminates the noisy neighbors and allows companies to power their big data efforts with dedicated hardware.
With uncertainty removed, costs minimized and security ensured, the cloud is resurfacing as a viable big data solution. Moreover, next-generation big data clouds offer automation and orchestration at every layer in the technology stack, starting with the underlying bare-metal infrastructure and spanning everything from application configuration and tuning to dependency and software upgrade management. The time has come for big data architects to take another look at the cloud as the primary facilitator of big data, empowering companies to affordably analyze information faster and on a greater scale.
Like this article? Subscribe to our weekly newsletter to never miss out!
Image: Ulrika
