The Tech Industry Is in the Middle of a Data Engineering Talent Shortage

The number of data engineers doubled from 2013-2015, but there is still a shortage. Click here for high-resolution version

PHILADELPHIA, PA--(Marketwired - Sep 12, 2016) - A new study, The State of Data Engineering released today by Stitch, an ETL service; and Galvanize, a learning community for technology, reveals a shortage in data engineering talent. There are only 6,500 self-reported data engineers on LinkedIn, but the San Francisco Bay area alone has 6,600 job listings for that same title.

"Companies are increasingly viewing data as a competitive advantage," said Jake Stein, CEO of Stitch. "The Ubers and Airbnbs of tech have mastered using data to build better and smarter technology products. Now other tech companies are looking to do the same, and this is causing a major talent shortage."

The report delves deep into three core trends around data engineering:

  • Growth in the data engineering discipline
    While there's a shortage of this talent today, the numbers of data engineers are growing rapidly. From 2013-2015, the number of data engineers increased 122%. For context, the number of data scientists increased 47% in that same time period. Jonathan Coveney, data engineer at Stripe, says this rise in data engineering talent reflects a new sophistication in how companies think about data and the people who manage it, "There's a growing sense today that data isn't just a byproduct, but rather it's a core part of what a company does."

  • Where data engineers are coming from
    42% of all data engineers were formerly working as a software engineer, making it above and beyond the most common background. Data engineers are coming from other disciplines as well, but in much smaller numbers. A few other backgrounds include analyst (7%), consultant (6%), business analyst (3%), and data architect (3%).

  • The skill sets of data engineers
    The top five skills of data engineer are: SQL, Java, python, Hadoop, and Linux. But these skills are notably different depending on the size of the company. For data engineers working at large companies (1000+ employees), top skills are: data warehousing, business intelligence, Oracle, ETL, and Unix. For data engineers working at small companies (1-200 employees), top skills are: python, Java, javascript, MySQL, and machine learning.

Working with massive data sets requires specialized data engineering talent, and the race is on to get it. "The need for data science skills has become dramatic because companies realize the value and growth potential in their data assets," said Jim Deters, co-founder and CEO of Galvanize. "That's why companies of all sizes are starting to send their software engineers to Galvanize to learn how to work with big data and even the companies that aren't re-training or re-investing in their talent are telling us they need more people who can do this work -- it's a great opportunity for those with aptitude and ambition to learn these skills and to take some of these jobs."

All this data, and more, can be found in The State of Data Engineering.

This report is based on self-reported information from Linkedin, including all publicly visible personal and company profiles, skills, and professional experiences. We identified data engineers based on their professional headline and current title, and only included data engineers associated with identifiable companies.

About Stitch

Stitch is a simple, powerful ETL service built for software developers. Clients include Invision, Charity:Water, 500px, and many more. For more information on Stitch, read the blog, visit the website, and follow them on Twitter.

About Galvanize, Inc.
Galvanize is a dynamic tech learning community that that offers education, workspace and networking for students, startups and large companies. Galvanize teaches web development, data science and data engineering to students, offers support and workspace to member companies and provides over 200 networking events across ninth urban campuses throughout the nation. Galvanize campuses bring together entrepreneurs, students, investors, mentors, and great people and companies to develop the skills, mindset and networks necessary to thrive in a technology driven world. To learn more about Galvanize, visit our website, like us on Facebook, or follow us on Twitter @Galvanize.

Image Available: