Every industry needs data engineers, which is why they are in big demand. In days ahead, their salaries are expected to increase
Right now, if there’s a job that is relatively recession-proof, it is that of a data engineer. Demand has dropped for many roles. But Sanju Ballurkar, president at Experis IT, part of HR solutions firm ManpowerGroup India, said demand for data engineers is more or less steady.
Data engineers are backroom people. They manage mostly the unstructured data that flows in from all over. Unstructured data is not organised in a pre-defined manner – data in Word documents, social media, email messages, sensors. Data engineers make sure that this data is channelled the proper way so that it is served properly to their consumer, the data scientist. Data scientists rely on this data to create visualisations or apply AI models to predict what the future holds for the business.
According to Gaurav Vohra, co-founder and CEO, Jigsaw Academy, if a job role has demand across all industries, then you usually don’t see the impact of a recession, such as data-related jobs.
Vohra said that in India, demand for data engineers has started outpacing the demand for data scientists by a ratio of 2:1. “Earlier, data scientists were paid higher salaries than data engineers. Even that trend is changing, with companies paying at least 20%-30% more to data engineers than to data scientists,” he said.
Sundar Vijaynagar, who worked as a data science consultant with the Karnataka government during the Covid crisis, said everybody is looking to train in data science and AI/ML, leading to a dearth of data engineers.
A major reason for the rise of the data engineer is the mass migration to cloud, said Anand Narayanan, chief product officer, Simplilearn. “What used to be on-premise data engineering has now become data engineering on the cloud. On-premise specialised roles like data administrator, database developer, data architect and Business Intelligence (BI) developer have now consolidated into one powerhouse of a role, that of the modern data engineer,” he said.
When a company decides to move into the cloud, while its developers’ SQL skills may serve them well, that’s not enough. They need to understand how cloud functions, understand the concept of data lake, the tools used to do real-time streaming of IoT data. “You will have to rapidly pick up this technology set in order to be able to create a new data engineering architecture,” said Prashant Momaya, director, solution engineering, Tableau Software.
A growing number of statistics and commerce graduates too have started flocking to data engineering classes. But it’s easier for those with a computer science degree to train for it, as it requires knowledge of scripting languages, and programming languages like Python and R. It also helps to work towards becoming a “full-stack” data engineer, who can play the role of a data scientist too. While large companies can afford to have separate armies of data engineers and data scientists, Vohra said smaller ones prefer data engineers who are comfortable with the data analysis part, or data scientists who can also do data engineering.
What is data engineering?
Data engineering is the aspect of data science that focuses on practical application of data collection and analysis. For all the work that data scientists do to answer questions using large sets of information, there have to be mechanisms for collecting and validating that information. For that work to ultimately have any value, there also have to be mechanisms for applying it to real-world operations in some way. Those are both engineering tasks: the application of science to practical, functioning systems.
