Ron founded Think Big to help companies realize measurable value from Big Data. Previously, Ron was VP Engineering at Quantcast where he led the data science and engineer teams that pioneered the use of Hadoop and NoSQL for batch and real-time decision making. Ron was also Co-Founder and CTO of B2B applications provider C-Bridge.
A common misconception about Big Data stems from its very name—“big” is not its key point of differentiation. When we think about the challenges surrounding managing and analyzing Big Data, it’s the variety and diversity of the information—not just its size—that present the greatest challenges to traditional data management and processing tools and techniques.. While Big Data has certainly entered the canon of business buzzwords, there is a considerable room for innovation in the months and years ahead. Here are some of the things we’re looking for in 2015.
1. Big Data goes mainstream in the Enterprise.
In 2014 one of the things that we noticed changing rapidly in Big Data was its increasing enterprise focus. Adoption of open source platforms like Hadoop was originally limited to specific applications within early adopters like ad-tech and global web properties. But today, more and more mainstream companies view Big Data as a must-have. Manufacturing companies, for example, are now able to combine reliability and performance data from the field with testing data from the factory to help design and build better and more profitable products. Expect to see Big Data make major impacts on the competitive landscape in 2015. Companies which effectively embrace and deploy these solutions will expand their market and profit shares at the expense of lagging competitors.
2. For consumers, Big Data will largely remain the “man behind the curtain.”
Big Data will continue to impact consumers, but the changes will be much more incremental. Consumers may notice that they’re getting better service and options—such as better suggestions for dining, hotels, and recreation based on your travel itinerary and purchasing habits—but they will have no insight into the technologies and techniques behind these improvements. We have been fortunate to work with a number of forward-thinking leaders to help engineer and build just these sorts of systems. But for most consumers, they’re happy to let the “man behind the curtain” improve their customer experiences, unnoticed.
3. Professional services will remain critical to successful adoption
The importance of Hadoop being open-source cannot be over-stated; the fact that anyone can download, experiment with, and even improve these tools has been essential to the technology’s advancement. And while companies like Cloudera and Hortonworks do a good job of providing enterprise support, no one is going to confuse Hadoop for a ready-to-run packaged application any time soon—let alone emerging systems like Storm or Spark. And even if the technology tools were ready to run out of the box, each company’s challenges, problems, and opportunities are too unique for out of the box solutions. Combining the right technologies with the right architecture to address the right business needs and opportunities is as much art as science, and there’s simply no substitute for experience. While some companies allow these new technologies to languish in one “proof of concept” after another, they risk being left behind by those who benefit from expert assistance to help accelerate them down the right path.
4. Full-stack knowledge becomes essential
Much has been written about the “skills gap” and other hiring challenges around Big Data in general and Data Science in particular. But we have found that our most successful team members are those who come to us with well-rounded engineering backgrounds—even if they lack substantive Big Data experience. Whether working with more mature platforms like Hadoop or emerging tools like Storm and Spark, full-stack experience is critical, from the hardware and the network to the data structures and algorithms. Peope with experience with a wide range technologies—and an ability and hunger to learn more—are those who thrive in this environment and with these tools.
We are beginning to see what shape the future may take when Big Data increasingly will play a role in improving our lives. For example, products like Nest already begun to make our devices smarter, and more efficient. But let’s face it – we’re a long way from having enough connected devices to point to an “internet of things”. Truth be told, we’re still just now unlocking Big Data’s potential. But these technologies are so powerful that even if its developments remain incremental, 2015 will still be Big Data’s most exciting year yet.
