In recent years, Data Engineers have been in high demand and data engineering is one of the fastest-growing fields. There are several reasons for this development, for instance an increase in skill demands, technology advantages, and rising competition in the market. At Advian, we have been able to make several successful hires and attract top talent to join our team of experts. Now we are presenting our talented Data Engineer Sweeta, one of our recent additions to the team. In this blog, Sweeta gives a peek into her life as a Data Engineer and shares thoughts on her career and what awaits for her in the future.
According to Sweeta, being a Data Engineer is designing and building systems for collecting, storing, and analyzing large volumes of data.
For instance, by
Integrating various systems and managing their data in one single place, which can then be further used for different analytical purposes.
Building systems that collect, manage, and convert raw data into usable information for data scientists and business analysts to interpret.
Developing algorithms to transform data into useful and actionable information.
There are various skills a Data Engineer needs: Coding, data warehousing, knowledge of operating systems, data analysis and much more to succeed in their job. But besides the technical skills, there are a few fundamental practices in Sweeta's day-to-day life that strongly define her way of working.
Data Engineers are usually responsible of finding trends in data sets and developing algorithms to extract meaningful information from raw data. Therefore, knowledge of SQL database design and multiple programming languages is important for these types of roles. But more importantly, having good communication skills is necessary when collaborating with multiple teams and different stakeholders. Understanding the business's needs is essential in order to efficiently design and build the pipeline that best suits the requirements. To practice your communication skills, a Data Engineer not only has to listen, but also ask the right questions. Before starting a new project a Data Engineer needs to find answers to questions like:
What type of information needs to be captured and monitored for dashboarding?
What is the volume of the data being processed and what about the frequency of the processing?
What are the sources and targets?
What kind of transformations should be made?
How should the data be stored?
Finding answers to these questions is vital to fully understand the customers requirements - what should be done with the data and how it should be done.
Secondly, doing research & building solutions
The other main aspect of data engineering is building efficient ETL pipelines with the right set of tools. Again, it all comes down to communication, but this time with yourself. Ask yourself the following questions when you’re on the quest to finding the right set of tools:
What type of processing is required?
Does the tool have a reasonable learning curve?
Does it fit in your budget without compromising the quality?
Is the tool able to process massive quantities of data and can it also scale up or down depending on the business needs without compromising the performance?
Does it have enough customer support in case of a disaster?
Once you’ve found the tools that satisfy the requirements, you can either build a fully automated pipeline or a pipeline with minimal manual intervention and lastly, of course start building the actual ETL pipeline. In an industry where most businesses are fully moving to cloud providers, the need for proper research and identification of the best solutions is emphasized even further.
Sweeta’s final thoughts
During her career as a Data Engineer, Sweeta has had the opportunity to participate in several interesting projects. One example of her past work is a project where she identified and built a data pipeline that collects data from a mobile application, stores it, and converts it into a readable format. This data was then analyzed to build dashboards for monitoring users' health and to even take safety measures when needed. The tech stack used in this project was: Kinesis Firehose to collect the data, S3 as a data Lake, Athena for transforming the data and Splunk & Quicksight for dashboarding.
What do you think will be the next big technology you need to master?
As the technologies in the field continuously develop, a Data Engineer needs to keep learning new skills to stay on top of the trends. According to Sweeta Artificial Intelligence and Machine Learning will be the next big thing, as it can accelerate the big data process for example by using machine learning algorithms that can categorize the incoming data, recognize patterns and translate data into insights.
What have you learned the hard way during your career?
“Maintaining a work-life balance. Unfortunately, throughout my career, it was easy to feel like I shouldn’t take time off and that work should be my focus. Work is a fundamental part of the way we live, but there are a plethora of experiences waiting for us outside of work. Make sure to take time for family and friends, and get out of the house or office whenever you can,” Sweeta says.
Anything you would have done differently during your career?
“I would go back to college and add information technology to my bachelor's studies. Understanding more about IT would have helped me to progress faster in my career and made it easier for me to understand some aspects of my first few roles,” she comments.
What do you want to learn next?
“I’d want to develop better cultural understanding, I believe it can lead to more positive interactions, better collaboration, and truly diverse conversations. On the technical side, I would focus on multi-Cloud computing,” Sweeta concludes.
Interested in a career at Advian?
Perfect! Check out our career page and get to know us 👇