Data sophistication has forever and profoundly changed the way in which companies do business today. Data-driven innovation has led to better business decisions, enhanced customer engagement, and improved customer retention, all of which are essential to succeeding in today’s competitive market.
Data scientists and data engineers play the primary role in accelerating a company’s data sophistication, providing both the technology and domain expertise from a sea of zeros and ones into valuable data products. As we reflect on 2019 and look forward to the year and decade ahead, we will examine the evolving nature of these roles and teams.
From Citizen Data Scientist to Citizen Data Engineer
2019 was the year of the “citizen data scientist,” a term that refers to individuals who utilize data science practices and tools but whose training and primary role is not that of a data scientist. The explosion in data science initiatives across departments and job functions has stirred an increasing need for streamlined and scalable access to data, a function typically handled by data engineers.
Having a strong data engineering team is important in today’s world, and more companies are taking notice. Much of the work done by data scientists proves to be very difficult without the support of data engineers. The volume, velocity, and variety of data available to data scientists increases daily, and data engineers are fundamental to ensuring their success by creating scalable, reliable, and efficient systems for processing and delivering data. However, there are simply not enough data engineers to meet the demand. To fill this void, I predict 2020 will be the year of the “citizen data engineer.”
The emergence of the citizen data engineer highlights a trend in which those outside of the data engineering team will oversee and manage data pipelines, as well as the overall data lifecycle. Either by choice or by necessity, individuals will begin taking on these new responsibilities to keep up with the demand for data engineering. Citizen data engineers will help businesses fill the gap in 2020 and beyond; they will be brought in to build and manage pipelines, automate digital transformation projects, and see data-driven projects through to production.
Career Opportunities for Data Engineers
With an ever-increasing demand for data engineering, we expect to see a steep increase in compensation for skilled data engineers. According to LinkedIn’s 2020 Emerging Jobs Report, data engineering now joins machine learning and data science as one of the top-10 “jobs experiencing tremendous growth” in the U.S., with industries from retail to automotive taking notice and making this hard-to-hire talent a part of their teams. The most popular skill sets for data engineers include SQL, Python, ETL (extract, transform, load), data analysis, Apache Spark, and Amazon Web Services (AWS).
With continued and rapid growth in those roles that rely upon data engineering, however, there simply won’t be enough supply of data engineers to meet the growing demand. For data-driven businesses to fill the gap, citizen data engineers will become a crucial factor for success.
As an emerging profession, many data engineers have entered the field from backgrounds in software engineering, databases, infrastructure and tooling, or other related IT fields. Many organizations have already begun to identify and train employees within their company as a way to manage the dearth of data engineers, creating a growth opportunity for those interested in the field, and this trend will undoubtedly continue over the next year.
Sean Knapp is the founder and CEO of Ascend.io. Prior to Ascend.io, Sean was a co-founder, CTO, and Chief Product Officer at Ooyala.
