Much like data scientists, data engineer roles are among the fastest-growing in the U.S. Data engineers use skills such as programming, data modeling and knowledge of algorithms to help organizations build systems for collecting, managing and converting raw data into usable information. As more companies awaken to the possibilities of data analytics, they’re hiring more data engineers—generating great careers for those with the right skills.
But which skills? Whatever the organization and industry, data engineers need to design, build, optimize and maintain data infrastructure at scale. There’s also the need for “soft skills” such as empathy and communication, as data engineers must secure buy-in from other stakeholders throughout an organization.
Even though data engineering is a relatively young discipline that started gaining momentum in 2016, recent transitioners are already among the highest-paid tech talent, with average total pay of $114,026. Even better, their salaries show no signs of slowing down anytime soon.
However, because it's a fairly new field, professionals who transitioned into data engineering from database administration, data analytics, programming or software engineering face a unique challenge: an unclear career path. If you're wondering how to proceed, here are more than five possible career paths that allow you to use the knowledge you acquired as a data engineer.
Generalist to Specialist
As data teams continue to specialize and adapt their structures, new roles have emerged.
Data platform teams are now quite common, and are great places for data engineers to cut their teeth and move from being generalists to specialists, explained Shane Murray, field CTO for Monte Carlo and former SVP of Data at The New York Times.
For example, you can specialize in a specific domain of data that is central to business operations, such as customer data or product/behavioral data. In this role, you should aim to gain an understanding of the end-to-end problem, from source to the analytical use case, which will make you an asset to the team and broader business.
Alternatively, you might specialize in a specific capability of the data platform, such as reliability engineering, business intelligence, experimentation, or feature engineering. These types of roles typically give you a broader (but shallower) understanding of each business use case, but may allow you to expand your skills into software engineering and other disciplines.
Data Engineering Manager
After ascending from junior to mid to senior level, and assuming more responsibility for strategy, planning and architecture of the data pipeline, some data engineers transition to a managerial role, where they oversee a company’s data engineering department.
However, data engineering managers are responsible for more than just hiring and managing people; they wear many hats. It’s similar to a software engineering management role in that it’s part-developmental, part-managerial, explained Jay Feng, CEO of Interview Query and former data scientist.
Data managers provide leadership, coaching and direction to a team of data engineers. They also select the tools and data architecture, create the data stack and drive the vision of the department.
If you’re interested in moving from individual contributor to data engineering manager, see if you can prepare by taking on more responsibility for working with team members and assigning tasks.
Getting exposure across a range of teams or capabilities within a data platform can be a good springboard into leadership roles in data, as well. Data platform teams are very often led by someone from a data engineering background who has demonstrated a breadth of understanding across business and technical projects.
Data Architect
If you have a strong interest in design and want to focus on building advanced data models, frameworks and pipelines, another option is to become a data architect. Despite its title, data architect is a business-oriented role. In order to provide the roadmap for building advanced data models and pipelines that meet specific business needs and requirements, data architects need a strong grasp of the business’s direction and strategy.
To make a successful transition into data architecture, focus on the design elements in your data engineering role and become proficient with database management system software, especially SQL.
Also, because data architects are required to communicate with diverse stakeholders, Feng suggests getting more experience collaborating with your data team and taking the lead in driving team cohesiveness.
Data Product Manager
Another path for data engineers is data product manager. If you are working on a data engineering team, but particularly enjoy talking to end users, articulating the problems to be solved, and distilling the vision and roadmap for the team, then a product management role could be a great step in your career.
Data teams are beginning to invest in this skillset as we move to treating “data as a product,” ranging from critical dashboards and decision-support tools to machine learning applications that are critical to business operations or customer experience, Murray noted. Great data product managers understand how to build a reliable and scalable data product, but also apply product thinking to drive the vision, roadmap and adoption.
Back-end Engineer
Alternatively, you could find your way into back-end engineering, software engineering or machine learning engineering roles, which are often part of the data platform ecosystem.
If you’re looking for more task diversity, consider becoming a back-end engineer. You will still be involved with data; in fact, you may even be required to perform some data engineering in a more generalized back-end role. However, you may also work with databases, back-end logic, APIs, architecture, servers and more as a back-end engineer.
Given the rising demand for anyone with data science capabilities, it's clear that anyone looking to advance from data engineer will have plenty of options for doing so.
Related Data Engineer Jobs Resources:
