A recent survey of over 225 enterprise Data Scientists, AI technologists and business stakeholders involved in active AI and machine learning (ML) projects, suggests that for most organizations, it’s still early days for AI technology. 

The AI market is projected to become a $190 billion industry by 2025 ( according to Markets and Markets), and global spending on cognitive and AI systems is expected to reach $35.8 billion in 2029, an increase of 44.0% over the amount spent in 2018 (according to IDC). This research suggests AI is advanced and on the move, already being undertaken by large enterprises and ready to make an impact on how we live and work.   

But it is still early days for AI when it comes to the implementation of AI in organisations and there are reasons for that. An AI system requires meticulous training before it can perform its intended function. When that function involves something as complex as making human-like judgments about images or videos – “seeing,” in other words – the system must be exposed to enormous volumes of accurately labeled and annotated training data. With AI becoming a growing enterprise priority, data science teams are under tremendous pressure to deliver projects, but frequently are challenged to produce training data at the required scale and quality. 

Why do organizations face challenges in structuring data suitable for the AI strategy

The urgency of this challenge was one of the findings that emerged from the survey conducted by Dimensional Research and AIegion, the results of which are compiled in the report  Artificial Intelligence and Machine Learning Projects Obstructed by Data Issues. The survey respondents confirmed that enterprise machine learning is nascent, data science teams are still small, growing data science expertise is not yet matched with equally mature ML project expertise, and training data challenges pose broad challenges to project success. Graphic demonstration of this last observation is reflected in the 96% of respondents who reported that their lack of training data technology and skills has impeded their ability to train their ML algorithms and attain the confidence their model must provide. 

Today, large enterprises with more than 100,000 employees are most likely to have an AI strategy – but only 50% of them currently have one, according to MIT Sloan Management Review. The survey reinforces this finding that AI is still nascent in the enterprise: 

• 70% report that their first AI/ML investment was within last 24 months
• Over half of enterprises report they have undertaken fewer than four AI and ML projects
• Only half of enterprises have released AI/ML projects into production 

A little less than two-thirds of survey respondents indicated that their ML project has progressed to the point that it is being trained on labeled data, which is a relatively early phase in the ML project life cycle. And more revealing of the immaturity of ML in the enterprise, were reports of where teams struggle and why half of projects never get deployed.  

Survey respondents expressed:

• 78% of their AI/ML projects stall at some stage before deployment
• 81% admit the process of training AI with data is more difficult than they expected 
• 76% combat this challenge by attempting to label and annotate training data on their own
• 63% go so far as to try to build their own labeling and annotation automation technology

Nearly 40% of failed projects reportedly stalled during training data-intensive phases e.g., training data preparation, algorithm training, model validation and scoring, and post-deployment enhancement. 

When asked the reason for the failure, respondents cited:

• Lack of expertise (55%)
• Unexpected complication (55%)
• Data problems (36%)
• Lack of model confidence (29%)
• Budget (26%), and
• Not enough people (23%)

As already indicated, nearly two-thirds report that their ML project has progressed beyond proof of concept (POC) and algorithm development to the training data phase. For most, this phase is not going well; 80% report that training their algorithm has proved more challenging that they expected. 

The reasons why training algorithm data is challenging are numerous:

• Bias or errors in the data
• Not enough data
• Data not in a usable form
• Don’t have the people to label data
• Don’t have the tools to label the data

Less than 4% have reported that training data has presented no problems. These data-related problems could stem from how data is being produced and labeled internally. Nearly three-quarters of the survey group indicated that they’re attempting to label and annotate training data on their own. A little over 40% suggested that they’re relying in whole or in part on off-the-shelf, pre-labeled data. 

These problems led to 7 out of 10 companies utilizing external services for their AI or ML projects with many of them focusing on data collection, labeling and expertise. With AI/ML talent rare and expensive, this research suggests that enterprises should consider using external solution providers for critical activities like data labeling and model scoring. The data provides evidence that such outsourcing leads to improved outcomes. 

Conclusion
Enterprises assign strategic value to their machine learning initiatives and expect AI and ML to improve all aspects of their businesses, and potentially to be disruptive in their industry sectors. 

However, AI/ML projects are still early in their development within enterprises. Data science teams are relatively small and inexperienced, which impacts the efficacy and outcome of these projects. Securing and labeling the amount of data needed to support algorithm development and confidence levels continues to be a pain point for these organizations.

Note: Participants were from all 5 continents representing 15 industries (technology, healthcare, financial services, education, manufacturing, government, retail, services, telecommunications, food and beverage, media and advertising, energy and utilities, transportation, pharmaceutical and nonprofit). Sixty-three percent of respondents represented companies with more than 5,000 employees, and 37% of respondents represented organizations with 1,000 – 5,000 employees. 

To download the full survey report, clink the link here.

You know who you are. You’re a Data Scientist.

In interview rooms across the world, fat cat executives are pushing contracts under your noses with extortionately high numbers on them. What a time to be one of a few doing backstrokes at the world’s most exclusive pool party, right!?

What happens when other bright academics, who stand outside with noses pressed up against the glass, get bored, rush the gates and start doing huge running pool bombs? One only has to look at the statistics at the growing numbers of governments and businesses funding universities to teach Machine Learning or the accessibility of online courses and competitions to see how this is becoming a more accessible and inclusive market place.

Economist, Tyler Cowan once said; “Food is a product of supply and demand, so try to figure out where the supplies are fresh, the suppliers are creative, and the demanders are informed.”

The same can be said about Data Scientists. Businesses are becoming more informed about the potential of Machine Learning. When these businesses demand the skill, the supply must be met at any cost. Many businesses are smart and they’re flooding billions of dollars into closing the skills gap so it’s only a matter of time before the next generation of Data Scientists emerge.

So, how do you ensure you remain afloat on your inflatable crocodile with your Piña Colada in tact? Simple really, you have to diversify…or drown. Make yourself futureproof now. Continue pushing yourself in new ways, to ensure you’re not left behind.

Here are a few suggestions:

Become a He-Man/She-Ra Coder

One of the current trends we’re seeing is the growing demand for Data Scientists with production coding experience. For some businesses, proof of concept coding is great. For others, the ability to write the code that takes it into production, is even better! Some businesses that don’t have engineering teams tend to prefer this option. Other businesses just like to kill two birds with one stone made out of budget. It’s a great blend of experience to have and one that would see you fit into the “Unicorn” category in the eyes of some hiring managers.

Be a Business Brain

What we see time and time again, is Data Scientists who are employed to solve specific problems in the business, unbeknown to them (and their managers) that there are countless other problems they can solve too. The most successful commercial Data Scientists are the ones who can understand the rhythm of the business around them – how it works, why it isn’t working and taking ownership to solve the problem. You need to be able to proactively approach stakeholders within the business and confidently challenge them on the shortfalls of their department and how an application like Machine Learning can help the company be more successful. Engage yourself with the business. Have one eye on the project in front of you, but be able to identify where the next opportunity is coming from. Sell Machine Learning to everyone.

Become famous!

When we started Big Cloud, we were amazed at how online the Data Science community was. Compared to “old school” industries, it’s crazy how quickly you can make a name for yourself. We are constantly being asked by hiring managers to find them the best Kaggle Masters. Because of the accessibility of Kaggle, it has become a platform from which someone can gain notoriety very quickly. Another reason why managers like people who compete on Kaggle or partake in other extracurricular activities such as hackathons, is because it shows they have a genuine passion for the subject. How many hot shot lawyers finish a tough day in court, go home, log on and start solving cases in their spare time? How many Firemen go out looking for fires to put out when they’re not on watch? Data Science is an industry of discovery and people who are inquisitive and push themselves outside of work, in their own private research to better themselves, are being prioritised by more and more companies.

Look for the pool parties abroad

While there are many opportunities in the United States and the UK, as well as the rest of Western Europe, there are far more businesses in other parts of the world who are looking for Data Scientists. With lower tax brackets than the West, and a less saturated job market, you can expect to live very comfortably in say Bangkok or Kuala Lumpur for a fraction of the cost of London or San Francisco. If nothing else, it offers the opportunity to diversify your CV and solve problems that will benefit people in completely different parts of the world…how cool would that be! Just as important though, it’s also a chance to fulfill a desire of adventure.

Mo Money, Mo Problems

One of the biggest reasons we see offers rejected at final stage is because of a change in salary expectations. Sometimes it’s easy to increase your expectations at the last minute, particularly when the recruiting company have rolled out the red carpet and expedited their recruitment process from 4 weeks to 1 day, just to accommodate you. However, before you decide to hike your demands up at the last minute, consider your next move. It’s true, we see some candidates who not only price themselves out of the job, but sometimes the market entirely! It makes your job search that little bit trickier next time (who wants to earn less?), but it also means you’ll sometimes have to say goodbye to the businesses who are solving the most interesting problems. Strike while the iron is hot, but don’t get caught out in the abyss when equally skilled and cheaper rival applicants begin to emerge.

Get promoted

If there is an end point to your research days, a day when you’ve written your last line of code, perhaps a day when cleaning data is just too much of a pain , there is always a position upstairs. You will always find solace (and safety) in positions of management. Mentor and guide teams and become the parental figure. However, be prepared for the politics that can come along with this gig, as you will fight a daily battle with people in other areas of the business, who sometimes don’t have a clue what you do and why you’re telling them what they should be doing.

Start-up!

You don’t want to be replaced? Upstaged by a snotty nosed kid? Left to rot on the scrap heap? Simple, go into stealth mode and start up your own business! It seems like the most popular destination for most hardcore Machine Learning practitioners, who have a great idea and the balls to live it out. Build a proof of concept, showcase it and get a load of cash rich investors and don’t look back. How hard can it be?…

It’s critical that as Data Scientists who are always seeking to perfect and optimise your models and frameworks, you also need to take the same approach to who you are and what you’re offering to the world around you. When in the moment for here and now, it’s sometimes very easy to lose sight of what can and will be. One thing is for sure, with all the hype and attention Data Science is receiving in the press right now, this party is going to get a whole lot busier! Grab your shades, grip on tight and prepare for the swell!

image credit: Kajoaaa

Like this article? Subscribe to our weekly newsletter to never miss out!

Follow @DataconomyMedia

In Europe, startups and established corporations alike are tapping into the power of data science. However, putting the right team together to your business’ data sciences problems remains challenging, as the talent scarcity continues. This is, of course, good news for data scientists; those who combine expertise in machine learning & big data technologies with business acumen can expect to play a central role in an organisation’s decision-making process, and start an exciting career in a rapidly evolving industry.

Graduates of quantitative disciplines like engineering or the natural sciences will find alot of their current expertise overlap with the skills needed to become a data scientist. Blog posts, online-courses and tutorials can provide a strong foundation for transitioning to data science, though most people who choose this route reach a sticking point. To progress further, practical experience becomes crucial; it’s one thing to learn these new skills, but it’s quite another to apply them to unique problems and challenges as you would in the workplace. MOOCs have undoubtedly been instrumental in bringing education to the masses, but to fully master a field as multi-faceted and nuanced as data science, it helps to shed the anonymity. Hands-on experience, overseen by a mentor who can identify your particular strengths and weaknesses, offers huge benefits. Particularly in terms of time; with hard work, expert support and a solid background in mathematics, statistics and programming languages the transition from rocket science to recommendation engines can be a matter of just a few months.

2014 saw high-quality training programmes flourish, particularly in the US, but Europe was largely left playing catch-up. This is where Data Science Retreat comes in. Started by former Chief Data Scientist Jose Quesada, PhD, DSR is on a mission to train passionate data scientists from all over the world. Since the spring of 2014, he and his team have been working on a rigorous and intensive 3-month training program in Berlin, that recruits its participants from all around the world.

Each class involves a small number of select participants, who attend classes taught by chief data scientists, and work to solve real business challenges under expert mentorship. The curriculum spans everything from big data technologies to business communication, with the hope of setting up the pupils with a skill set covering every key aspect of work as data scientist. During the course, students also work on their own portfolio projects, which they present to a number of German and international companies at a hiring-day at the end of the program. Previous participants’ projects include a recommendation engine for flatshares & the Berlin real-estate market, a predictor for the outcome of art-auctions and the location of crime-scenes in urban areas.

The program has also been well-received by the hiring companies: more than 86% of the participants found the job they wanted within 3 months of their completion of the program, with 60% of participants getting multiple job offers, and having the freedom to pick.

The next 3-month DSR session will begin in early February 2015 and will be hosted at the German online retail giant Zalando, home to a team of more than 40 data scientists.

Are you a budding data scientist who is looking to specialize and excel in the data science field? Applications are still being accepted if you are interested in being a part of Data Science Retreat batch 03!

Follow @DataconomyMedia

(Image credit: Data Science Retreat)