Research and Markets recently published their report on the Global HDaaS (Hadoop-as-a-Service) Market 2015-2019. They market, they believe will continue to grow at a CAGR of 84.81 percent over the period 2015-2019.

Although many have been quick to claim the death of “big data” and the demise of Hadoop, adoption continues to rise. M&R analysts attribute this dissonance to one key factor- although the big players may be moving on, SMEs are flocking to data services and HDaaS in their droves.

“One key trend emerging in this market is the increased adoption of HDaaS by small and medium enterprises. The market has witnessed that SMEs are among the earliest adopters of cloud computing and big data technology. Because this end-user segment is already aware of the potential benefits of cloud computing, HDaaS vendors are looking to capitalize on the increase in demand for HDaaS solutions from this segment,” notes an analyst from the research team.

To calculate the market size, the report considers revenue generated from Hadoop analytics solutions, Hadoop software, applications, services, support, and maintenance.

Some of the key findings of the report point out:

1. The key vendors in the Global HDaaS Market are Amazon Web Services, EMC², IBM and Microsoft.

   Other Prominent Vendors in the market are: Altiscale, Cask Data, Cloudera, Google, Hortonworks, HP, Infochimps, Karmasphere, MapR Technologies, Mortar Data, Pentaho and Teradata.

2. Cost-effective solutions for big data management find growing demand and drive the market. Cost-effective cloud computing technology, with capabilities to manage and analyze large amounts of data,through Hadoop, enables enterprises to manage their big data of any size in a cost-effective manner, the report states.
3. On the other hand, “lack of awareness about or unfamiliarity with the Hadoop technology,” is a drawback affecting the market.

   “The lack of awareness about the Hadoop technology and the lack of trained professionals are two of the major issues that have prevented enterprises from investing in Hadoop-based big data solutions, which is hindering the growth of the market.”

The report contains a comprehensive market and vendor landscape in addition to a SWOT analysis of the key vendors. The detailed report is available here.

Photo credit: Udri / Foter / CC BY-NC-SA

Darwin never actually said the following quote, but it’s truthy so I’ll use it:

It is not the strongest of the species that survive, nor the most intelligent, but the one most responsive to change. —Darwin-ish

I was watching a talk by Josh Wills the other day. He was applying Lean engineering concepts to data science. To illustrate how important rapid learning is, he told a story about the team that built the Gossamer Condor and won the Kremer Prize for human-powered flight. They won because they failed more repeatedly than their competition. I’m sure their competition was brilliant, and they definitely had several years head start, but they lost because it took them maybe a year to iterate from design, to build, to test flight, to crash and destroy, and go back to design again. The Gossamer Condor team’s breakthrough insight was this: if the Condor could be repaired and improved in days, then they’d test 100 designs in the time their competition tested 1.

50 years on, and data science is following this same anti-pattern of the teams that didn’t get the Kremer Prize: come up with brilliant ideas, painstakingly move them out to the real world, watch them fail, and then slowly start the brittle process anew. This would be a problem for any profession—the faster you can iterate, the faster you can learn, and the more problems will be solved—but it is an especially pressing problem in data science because there is a huge shortage of data scientists, so inefficiencies mean that many critical problems are not getting solved.

What can we do about it? Luckily, software engineering, which is a sister to data science, has been working through these problems for the last two decades and has some pretty good patterns to build from. Devops is the area of software engineering concerned with moving software from development to real-world use quickly and safely. Devops lets software engineers try more things, and therefore learn faster. Here are the pieces I believe are necessary for data science:

1. Automated tests: these don’t have to be exhaustive, but there should be an automatic way to know that changes don’t horribly break your user-facing system
2. 1-Button Deploy: If releasing changes takes more than one step, it will break more frequently, and more importantly in the context of this article, releases will happen less frequently.
3. 1-Button Rollback: The counterpart to 1-Button Deploy, if an error is discovered in a user-facing system, reverting to a pre-error state must be swift and reliable.
4. Instrumented Infrastructure: Data science problems often require distributed architectures, non-obvious dependencies, and complex feedback loops. To successfully try many things quickly, it is necessary to spend a minimal amount of time understanding the infrastructure, tuning it, and correcting errors.

It’ll take some work, but I believe Devops is the next crucial frontier for Data Science—a massively underrated piece of this rapidly changing discipline.

Yes, I cofounded a company, Mortar, that amongst other things addresses these problems… because I think they are so important to solve.

Follow @DataconomyMedia

K Young has been CEO of Mortar Data since 2010. Mortar helps data scientists and data engineers spend 100% of their time on problems that are specific to their business—and not on time-wasters like babysitting infrastructure, managing complex deploys, and rebuilding common algorithms from scratch. Mortar’s platform runs pipelines of open technologies including Hadoop, Pig, Java, Python, and Luigi to provide out-of-the-box solutions that can be fully customized. Prior to founding Mortar Data, K built software that reaches one in ten public school students in the U.S. He holds a Computer Science degree from Rice University.