Become a Data Engineer in 2025 (Based on 100 jobs data!)

Happy New Year, everyone! Reposting a combination of 3 of my most upvoted posts last year at the start of the year for those looking to set ambitious career goals in 2025 assuming lot of new people are looking for this info now. After all, there’s no better time to plan your next big leap into Data Engineering!

**1. Top skills in demand -**

I analyzed 100 data engineering job descriptions from Fortune 500 companies to find the most frequently mentioned skills. Here are the top skills in demand:

|**Skill Group**|**Frequency**|**Constituents with Frequency**|
|:-|:-|:-|
|Programming Languages|196|SQL (85), Python (76), Scala (21), Java (14)|
|ETL and Data Pipeline|136|ETL (65), Pipeline (46), Integration (25)|
|Cloud Platforms|85|AWS (45), Azure (26), GCP (14)|
|Data Modeling and Warehousing|83|Data Modeling (40), Warehousing (22), Architecture (21)|
|Big Data Tools|67|Spark (40), Big Data Tools (19), Hadoop (8)|
|DevOps, Version Control and CI/CD|52|Git (14), CI/CD (13), Jenkins (7), Version Control (7), Terraform (6)|
|Data Quality and Governance|42|Data Quality (20), Data Governance (13), Data Validation (9)|
|Data Visualization|23|Data Visualization (11), Tableau (6), Power BI (6)|
|Collaboration and Communication|18|Communication (10), Collaboration (8)|
|API and Microservices|11|API (8), Microservices (3)|
|Machine Learning|10|Machine Learning (7), MLOps (2), AI/ML Model Development (1)|

**2. 4 Month Study Plan -**

**Month 1: Foundations**

* DBMS & SQL: Basics of database concepts, querying, and design.
* Python: Focus on Python essentials, including libraries like Pandas and NumPy.
* Linux: Basic commands and navigation.
* DSA: Data structures and algorithms, especially for big tech roles.

**Month 2: Key Concepts & Tools**

* Data Concepts: Topics such as Data Lake, Data Mart, Fabric, and Mesh.
* Data Governance: Management, security, and ethics in data.
* Spark: Introductory concepts with Apache Spark.
* Distributed Systems: Overview of Hadoop, Hive, and MPP systems.
* Cloud Services: Options such as AWS, GCP, or Azure.

**Month 3: Advanced Topics**

* Orchestration: Basics of workflow orchestration with tools like Apache Airflow.
* Compute: Databricks, Snowflake, or equivalents like AWS EMR.
* Containers: Introduction to Docker and Kubernetes.
* CI/CD: Tools such as Jenkins and SonarQube.
* Streaming: Fundamentals of Kafka.
* ETL/ELT: Tools like dbt and Talend, along with architecture basics.
* Terraform: Code-based infrastructure setup.

**Month 4: Projects & Portfolio**

* Build a project portfolio to showcase skills. Examples include:
* Bank Data Warehouse
* Fraud Detection ETL
* Reddit Review Tracker
* Retail Analytics
* Trip Data Transformation
* YouTube Clone

**3. Certifications**

Note - You don't have do all of these, do 1/2 of AWS or Azure, 1 of Datarbricks or Snowflake, and 1/2 of optional certifications based on your interests. Also I have mentioned resources only for the ones I know - for the ones I haven't attempted/know have left it empty - please add the same in the comments.

|**Certification**|**Coverage**|**Cost (USD)**|**Resource**|
|:-|:-|:-|:-|
|AWS Certified Cloud Practitioner|Basics of AWS Cloud concepts, services, and support.|$100|Stephane Maarek's Udemy courses|
|**AWS Certified Solutions Architect – Associate ⭐**|Designing and deploying scalable systems on AWS.|$150|Stephane Maarek's Udemy courses|
|**AWS Certified Data Engineer – Associate ⭐**|Managing data pipelines, analytics, and ETL workflows on AWS.|$150|Stephane Maarek's Udemy courses, AWS Builder Labs|
|Microsoft Azure Data Fundamentals (DP-900)|Core data concepts and implementation using Azure.|$99|Eshant Garg/Scott Duffy Udemy courses, Coursera prep courses|
|**Microsoft Azure Data Engineer Associate (DP-203) ⭐**|Integrating and transforming data for analytics on Azure.|$165|Eshant Garg/Scott Duffy Udemy courses, Coursera prep courses|
|Databricks Lakehouse Fundamentals|Basics of Databricks Lakehouse architecture and workflows.|Free||
|**Databricks Certified Data

Engineer Associate ⭐**|Building ETL pipelines and managing data workflows.|$200|Ankit Mistry's Udemy courses|
|Databricks Certified Data Engineer Professional|Advanced data engineering skills on Databricks platform.|$200||
|**SnowPro Core Certification ⭐**|Foundational knowledge of Snowflake architecture and operations.|$175||
|SnowPro Advanced Certification|Advanced expertise in complex Snowflake solutions and optimizations.|$375||
|SnowPro Advanced: Data Engineer|Data modeling, ETL, and tuning on Snowflake.|$375||
|Astronomer Certification for Apache Airflow Fundamentals|Core Apache Airflow concepts, including DAG authoring and scheduling.|$150|Mark Lamberti's Udemy course|
|Confluent Certified Developer for Apache Kafka|Developing applications with Kafka, architecture, and APIs.|$150||
|dbt Analytics Engineering Certification|Building and maintaining data workflows with dbt.|$200||
|HashiCorp Certified: Terraform Associate|Managing cloud resources using Terraform.|$70||
|Data Management Fundamentals Exam|Core principles: data architecture, governance, and quality.|$311||
|Data Governance Specialty|Best practices for governance, compliance, and data quality.|$311||

Tips to save money on these:

* AWS offers 50% discount on next exam: So after you give your first certification you can use a coupon code for the next ones.
* Azure - Coursera prep courses for Azure certifications offer 50% exam discount upon completion.
* For Airflow Fundamentals - Astronomer sometimes runs a promotion to get the certification for free. Follow them/Marc on LinkedIn to know the dates - I got mine in Jan last year.

**➡️Dive deeper! - Checkout my playlist "Data Engineering Career" with details of all of the above -** [https://www.youtube.com/watch?v=5b4CIon\_1pY&list=PLYAUClNVzmDN5D9IW-COX0xy\_8fz8r51k&ab\_channel=AnalyticsVector](https://www.youtube.com/watch?v=5b4CIon_1pY&list=PLYAUClNVzmDN5D9IW-COX0xy_8fz8r51k&ab_channel=AnalyticsVector)

Thanks, hope it added some value! All the best!

https://redd.it/1hzf2uf
@r_devops

Cross-platform alternative to npm scripts

I really enjoy the ability to create alias for a command using "package.json.scripts" in Node. Is there a lightweight tool that allow me to do that cross-platform, workspace-first and allow comments?

P/s: I've looked into just, make, mise but they don't seem to do what I want. Mise seems to be very close but it is also quite heavy to install.

https://redd.it/1hzlzgw
@r_devops

University and learning - devops

Hi, in just over a year, I'll finish my studies in telecommunications and IT. I have a broad foundation in various IT topics, a bit more knowledge in networking, but nothing particularly advanced apart from that. Based on my interests, I’d like to follow the DevOps path and learn more about it before starting an internship.

What learning path should I choose? What steps should I take to make it logical and effective? What should I focus on the most? Should I use platforms like Kode Kloud, for example?

And one more thing: what are the intermediate job positions for someone aiming to become a DevOps engineer in the future?

https://redd.it/1hzobzo
@r_devops

Best practice to filter out bot traffic and spammers?

I launched a site which gets some pretty heavy traffic currently. When i look into my logs it seems like a good chunk of the traffic is trying some wordpress vulnz and other stuff i dont want to have on my server.

Whats the best way to mitigate this traffic? I already have fail2ban configured for SSH. Would it be suitable to use some other tooling?

https://redd.it/1hzuyth
@r_devops

Display host cursor when recording a video on headless windows machine via RDP

I am connecting to a headless windows machine via RDP hosted on Google Cloud Platform via Remote Desktop. I need to record some videos on this windows machine. However, every time I try to record something, the mouse cursor does not appear. I am wondering how I can display the host's cursor on a headless windows machine? All of the solutions I've seen so far have been able to access the host device's hardware, but in my case, I cannot do that.

https://redd.it/1hzxrzs
@r_devops

Communities in Pittsburgh

Anyone know of any AWS communities in Pittsburgh. I’m new to learning cloud and would like to connect with others to learn and attend events etc.

https://redd.it/1i003rv
@r_devops

anyone has experience working at this place?

husband got offered devops position in Miami by millennium management, how is there tech work culture? high pressure like their finance division? crazy work hours? treatment ? thanks !

https://redd.it/1i02918
@r_devops

Next level after DevOps (what role is better paid: SRE, DevSecOps, MlOps, Platform Engineer, Cloud Engineer)

Currently DevOps, looking forward to reaching the next level and earn more.

What role is better paid and future proof: SRE, DevSecOps, MlOps, Platform Engineer or Cloud Engineer, etc.?

https://redd.it/1i02szp
@r_devops

How are you tracking changes in 3rd party tools that could disrupt your CI/CD pipelines?

I am curious on how you all in the devop world keep track of SaaS application updates to keep on top of potential breaking/high impact changes.

https://redd.it/1i04tby
@r_devops

Any 30+% discounts for Certified Kubernetes Administrator?

I'm an up-and-coming DevOps engineer and want to complete CKA certifications.
I couldn't procure enough funds during the Cyber Monday sale, hence missed the offer.
I want to do it now but I got only 50-60% of the full price. Hence looking for discounts of maybe 40-50%.
If anyone knows of any such coupons, please DM me. Would really appreciate it. Thanks

Apologies if this is not the right community to post in.

https://redd.it/1i064iw
@r_devops

Cross-platform vs. single platform—what’s better for early apps?

I’ve been debating whether to expand my app to iOS or focus solely on Android for now. Developing for both seems like a good idea, but it’s also a lot of work for a small team.

What’s your take? Did starting with one platform work better for you, or is it smarter to go cross-platform from the beginning?

https://redd.it/1i0bdsb
@r_devops

I automated myself out of my job. That's a first.

I expected it to happen at some point in my life, but not that early.

Worked at a smaller company (50 devs) - it was pure hell at the beginning.
Within about three years we fixed every problem and automated/standardized everything that might disturb the developers workflow. I tutored everyone and documented everything. We actually got the ball rolling to a really sweet spot.

The last few weeks were pure boredom. Since there were no legit projects left.

Well. Now they kicked me out of the company. Nothing left to do.
I'll get full salary for two months and don't have to work a second anymore.

WTF?

https://redd.it/1i0ctdt
@r_devops

How can I overwrite name and password during setup jenkin

Hi , I stuck with find way to set name and password jenkin during setup ,
Is there any way? Thanks

https://redd.it/1i0dteg
@r_devops

Database skills and daily tasks

Hello everyone,

How often do you get tasks or work on databases in your role? What kind of tasks are they typically?
Also, what is the expected level of database knowledge for a junior DevOps engineer? Do you have any recommendations for courses or books to learn database skills relevant to devops?

Thanks in advance!

https://redd.it/1i0fkky
@r_devops

New DevOps Manager Tips

I am a security engineer by trade. I have been working on a DevOps team for 1.5 years now but mainly act as a security SME. My manager might get promoted in the upcoming months and mentioned that I'll take over his old position.

While I know some of what my team does simply by answering questions and by osmosis, I am by no means an expert at DevOps.

1. What are some tips you have for me in this new role?

2. What would you wish your DevOps manager did? What do they do that you like?

3. What should I do to get up to speed and not act like I haven't been paying attention to everyone's work for the past 1.5 years?

4. What are some good ways to get caught up in all things DevOps while not getting too into the weeds? Just enough to be dangerous.

https://redd.it/1i0hc92
@r_devops

If not Jenkins then what?

I'm working at a place that's using BitBucket (on prem) with Bamboo Data Center (also on prem) and we are deploying .net applications on Windows VMs (drum roll also on prem). I know all of the above is not very popular as a setup in this subreddit, but it is what it is.

The problem is that I'm getting really sick and tired of Bamboo for the following reasons (not an exhaustive list):
- shitty documentation
- seems semi-abandoned, especially after Atlassian dropped support for Bamboo server
- It keeps bugging out in weird ways - certain deploy plans fail at random with a generic "Contact Atlassian support" errors, that disappear on retries (which really doesn't help in terms of automation and user experience)
- Certain plans just don't work properly - SPECs doesn't recognize the list of environments and nopes out 19 out of 20 times without any reason at all.
- The sever that hosts Bamboo needs to be restarted weekly for one reason or another (you would think we'd be used to it as a Windows shop, but we really aren't)
- Oh, and my favorite - the Bamboo plan is often telling me that the plan ran fine, a-okay, 10/10, never better... although the logs are full of nothing but errors.

We are evaluating a potential migration and although I was a bit sceptical at first, Jenkins seems to be a good fit - works on prem, plays well with Windows and is.. alive and free (which also helps in the current climate).

From the miriad of posts I read on here, it seems like you guys aren't really fond of it though, so... Why? Is Jenkins really that much of a pain to maintain and are there any (on prem) alternatives for (on prem) Windows workloads?

Sorry for the rant and for the overuse of (on prem). I'm just trying to get my point across.

https://redd.it/1i0i9s2
@r_devops

10 years of building Apache Kafka

Hey folks, I've started a Substack dedicated to the development of Apache Kafka. I've started off with some posts about our build infrastructure and I thought this community might find it interesting.

Here's a blurb:

> The Apache Kafka build system has evolved many times over the years. There has been a concerted effort to modernize the build in the past few months. After dozens of commits, many of conversations with the ASF Infrastructure team, and a lot of trial and error, Apache Kafka is now using GitHub Actions.

Read the full post on my free Substack: https://mumrah.substack.com/p/10-years-of-building-apache-kafka

https://redd.it/1i0iujk
@r_devops

Horror Story/Rant: Bad manager that just destroyed team work

My manager (lets call him Bob) is pretty good with human leadership skill. And it is good to have that kind of character in manager.

However, he refused to take engineers recommendation to resolve technical debts, operation challenges, stack complexity. For example:
- we have three different eks clusters in the same region because Bob thinks that increase reliability and HA. Mind you, those clusters also backed the same EC2 in the same region and AZs. If EKS and EC2 are down, 3 clusters are just down too. No matter how many clusters we have. We told him, we just need one. And the answer is no given the reason above. Now, eks is out of date and we are forced to upgrade 3 eks clusters. And surprisingly, we let go of our team EKS admin last month lol. The recommendation was made 6 months ago.
- have a release approval for any changes to Prod controlled by terraform. But Bob tends to make changes by hands without release approval and ask to do it in terraform with release approval. we told Bob we shouldn’t do this. Let’s follow the correct process. And we are violating company release approval chain. Again no. Bob does what Bod needs to.
- Bob thinks being DevOps is being able to be great SRE and developers at same time. Sure those fields are related. But one person can only do so much. If there are such people, they are unicorns and get paid way more than us.

I know the ship is going down. I am trying to save the ship but the captain is just bad.

Rant ends.

https://redd.it/1i0lfpl
@r_devops

Anyone regretted moving back to Engineering?

Has anyone successfully transitioned from Management back into Engineering and regretted it? If so, what did you regret and did you end up taking a pay cut? If not, are you happier now?

Edit: I am a Manager now with a decent salary, but I realized I don’t care about management at all and really miss hands-on work, so I’m considering transitioning back into Engineering, be that DevOps, Cloud, or something similar.

https://redd.it/1i0levs
@r_devops

Expensive logging

I work in a GCP environment and due to reportedly hideously expensive logging costs, I'm being told to cut down on logging. I believe in logging errors, but now we take a Java exception and report that XYZ exception occurred. No stack trace.

Tragically, this code will be deployed to production, leaving some poor support person the unenviable task of guessing where and why the exception occurred.

How are modern corporate apps doing logging given the unaffordable cost of logging? Please note, our current logging is going to GCP log explorer. The multi billion dollar corporation cannot afford to log, at least to gcp log explorer.

https://redd.it/1i0lumb
@r_devops