Reddit DevOps
270 subscribers
6 photos
31.1K links
Reddit DevOps. #devops
Thanks @reddit2telegram and @r_channels
Download Telegram
From idea to deployment in 4 hours

Hey guys,

yesterday i've accomplished something i've had in my head a few weeks already and only needed around 4 hours from idea to deployment. Check it out: https://og-img.com

It is completely free with no registration required.

The idea:

I wanted to have some dynamic Opengraph Images (those og:image metadata you see in blog & social media posts) on my own blog to look better when shared via social media.

The dynamic part is in the url, which means if i want to write a blogpost about devops salaries i can easily write my own OpenGraph Image with simply just changing the URL like this: https://og-img.com/Devops%20salaries/og.png

The "Devops%20salaries" part is the dynamic part which results in the final OpenGraph Image.

You can generate every Image you can image, as long as it fits in 100 characters, and embed this into your own blog or social media posts.

Tech stack:

The heavy lifting is done via node.js in the backend + vanilla javascript in the frontend

I decided to dockerize the whole application, especially because i wanted Caddy as a reverse proxy to handle the node.js container traffic.

The stack node.js + caddy is running in a docker compose setup on a mediocre VM from hetzner.

I announced the service on r/webdev and got some huge traffic yesterday, but the stack handled all of this perfecty.

If you have any feedback or questions let me know! <3




https://redd.it/1hywubd
@r_devops
Is there a good static analysis tool that's free and that's better than semgrep?

Is there a good static analysis tool that's free and that's better than semgrep?

https://redd.it/1hz3emt
@r_devops
Logging software recommendations

Hey! So I'm looking for a logging software that has a free plan for starting and cheap plans with good retention times. It's for Java apps (specifically Minecraft servers) where I run multiple instances per server type (for example, first server type would be running in 5 instances and second server type would be running in 10 instances). Any recommendations that you can give? I've seen Axiom, I'm currently thinking on that. Or should I use Sentry self-hosted or another product? Thanks in advance.

https://redd.it/1hz0tv6
@r_devops
How to pass environment variables to remote server using GitHub Actions

I have a flask app for which I want to setup CI/CD using GitHub Actions. I am not sure what the best way to pass environment variables for my Postgres and flask containers. Of course I cannot keep a `.env` file in version control.

My approach is to connect to the server through SSH and create a `.env` file and refer to it in my `docker-compose.yml`

Is this bad practice?

name: Deploy Flask App

on:
push:
branches:
- main

jobs:
deploy:
runs-on: ubuntu-latest
environment: production
steps:
- name: Deploy to production server
uses: appleboy/ssh-action@master
with:
host: ${{ secrets.SERVER_HOST }}
username: ${{ secrets.SERVER_USERNAME }}
key: ${{ secrets.SSH_PRIVATE_KEY }}
script: |
mkdir -p /app
git clone https://my_app /app
cd /app

# Create .env file from GitHub secrets
cat > .env.prod << EOL
REDDIT_USER=${{ secrets.REDDIT_USER }}
REDDIT_USER_PASSWORD=${{ secrets.REDDIT_USER_PASSWORD }}
REDDIT_CLIENT_ID=${{ secrets.REDDIT_CLIENT_ID }}
REDDIT_CLIENT_SECRET=${{ secrets.REDDIT_CLIENT_SECRET }}
USER_AGENT=${{ secrets.USER_AGENT }}
POSTGRES_DB=${{ secrets.POSTGRES_DB }}
POSTGRES_USER=${{ secrets.POSTGRES_USER }}
POSTGRES_PASSWORD=${{ secrets.POSTGRES_PASSWORD }}
POSTGRES_HOST=${{ secrets.POSTGRES_HOST }}
POSTGRES_PORT=${{ secrets.POSTGRES_PORT }}
OPENAI_API_KEY=${{ secrets.OPENAI_API_KEY }}
OPENAI_ENCODER=cl100k_base
OPENAI_EMBEDDING_MODEL=text-embedding-3-small
OPENAI_EMBEDDING_MODEL_TOKEN_LIMIT=8191
CHUNK_SIZE=1200
CHUNK_OVERLAP=120
EOL

https://redd.it/1hz6xv7
@r_devops
Free solution to having Nx Cloud on our bare metal server?

Hi everyone. I wanted to ask if there is a free solution to having Nx Remote Cache locally on our bare metal server? Basically a free local version of Nx Cloud.

I have a project in my team that uses an Nx monorepo, and we have own bare metal servers.

I want to be able to share the Nx cache with other team members, so it can speed up development and testing.

Is there any solution out there? Even if it doesn't have all the bells and whistles of the paid version?

https://redd.it/1hz8shb
@r_devops
OpenTofu 1.9 is here with foreach support for providers, here is how to use it.

Last week I decided to try out the for
each and there was not really any clear documentation on how to use it. So after looking through the PRs and some other mentions to figure out how to use it, I threw together a blog post with a complete example.


The TL;DR, so you don't even need to go to the blog is:



variable "aws_regions" {
type = map(string)
default = {
"global" : "us-east-1",
"backup" : "us-west-2"
}
}

terraform {
required_providers {
aws = {
source = "hashicorp/aws"
version = "~>5"
}
}
}

provider "aws" {
for_each = var.aws_regions
alias = "by_region"

region = each.value
}

data "aws_availability_zones" "this" {
for_each = var.aws_regions
provider = aws.by_region[each.key]
}

output "provider_regions" {
value = { for k, v in data.aws_availability_zones.this : k => tolist(v.group_names)[0] }
}

data "aws_availability_zones" "global" {
provider = aws.by_region["global"]
}

output "global_region" {
value = tolist(data.aws_availability_zones.global.group_names)[0]
}




Follow on blog post showing some higher complexity use cases: https://dwood.dev/posts/opentofuproviderforeachcomplexexample/

https://redd.it/1hzch9r
@r_devops
Become a Data Engineer in 2025 (Based on 100 jobs data!)

Happy New Year, everyone! Reposting a combination of 3 of my most upvoted posts last year at the start of the year for those looking to set ambitious career goals in 2025 assuming lot of new people are looking for this info now. After all, there’s no better time to plan your next big leap into Data Engineering!

**1. Top skills in demand -**

I analyzed 100 data engineering job descriptions from Fortune 500 companies to find the most frequently mentioned skills. Here are the top skills in demand:

|**Skill Group**|**Frequency**|**Constituents with Frequency**|
|:-|:-|:-|
|Programming Languages|196|SQL (85), Python (76), Scala (21), Java (14)|
|ETL and Data Pipeline|136|ETL (65), Pipeline (46), Integration (25)|
|Cloud Platforms|85|AWS (45), Azure (26), GCP (14)|
|Data Modeling and Warehousing|83|Data Modeling (40), Warehousing (22), Architecture (21)|
|Big Data Tools|67|Spark (40), Big Data Tools (19), Hadoop (8)|
|DevOps, Version Control and CI/CD|52|Git (14), CI/CD (13), Jenkins (7), Version Control (7), Terraform (6)|
|Data Quality and Governance|42|Data Quality (20), Data Governance (13), Data Validation (9)|
|Data Visualization|23|Data Visualization (11), Tableau (6), Power BI (6)|
|Collaboration and Communication|18|Communication (10), Collaboration (8)|
|API and Microservices|11|API (8), Microservices (3)|
|Machine Learning|10|Machine Learning (7), MLOps (2), AI/ML Model Development (1)|

**2. 4 Month Study Plan -**

**Month 1: Foundations**

* DBMS & SQL: Basics of database concepts, querying, and design.
* Python: Focus on Python essentials, including libraries like Pandas and NumPy.
* Linux: Basic commands and navigation.
* DSA: Data structures and algorithms, especially for big tech roles.

**Month 2: Key Concepts & Tools**

* Data Concepts: Topics such as Data Lake, Data Mart, Fabric, and Mesh.
* Data Governance: Management, security, and ethics in data.
* Spark: Introductory concepts with Apache Spark.
* Distributed Systems: Overview of Hadoop, Hive, and MPP systems.
* Cloud Services: Options such as AWS, GCP, or Azure.

**Month 3: Advanced Topics**

* Orchestration: Basics of workflow orchestration with tools like Apache Airflow.
* Compute: Databricks, Snowflake, or equivalents like AWS EMR.
* Containers: Introduction to Docker and Kubernetes.
* CI/CD: Tools such as Jenkins and SonarQube.
* Streaming: Fundamentals of Kafka.
* ETL/ELT: Tools like dbt and Talend, along with architecture basics.
* Terraform: Code-based infrastructure setup.

**Month 4: Projects & Portfolio**

* Build a project portfolio to showcase skills. Examples include:
* Bank Data Warehouse
* Fraud Detection ETL
* Reddit Review Tracker
* Retail Analytics
* Trip Data Transformation
* YouTube Clone

**3. Certifications**

Note - You don't have do all of these, do 1/2 of AWS or Azure, 1 of Datarbricks or Snowflake, and 1/2 of optional certifications based on your interests. Also I have mentioned resources only for the ones I know - for the ones I haven't attempted/know have left it empty - please add the same in the comments.

|**Certification**|**Coverage**|**Cost (USD)**|**Resource**|
|:-|:-|:-|:-|
|AWS Certified Cloud Practitioner|Basics of AWS Cloud concepts, services, and support.|$100|Stephane Maarek's Udemy courses|
|**AWS Certified Solutions Architect – Associate **|Designing and deploying scalable systems on AWS.|$150|Stephane Maarek's Udemy courses|
|**AWS Certified Data Engineer – Associate **|Managing data pipelines, analytics, and ETL workflows on AWS.|$150|Stephane Maarek's Udemy courses, AWS Builder Labs|
|Microsoft Azure Data Fundamentals (DP-900)|Core data concepts and implementation using Azure.|$99|Eshant Garg/Scott Duffy Udemy courses, Coursera prep courses|
|**Microsoft Azure Data Engineer Associate (DP-203) **|Integrating and transforming data for analytics on Azure.|$165|Eshant Garg/Scott Duffy Udemy courses, Coursera prep courses|
|Databricks Lakehouse Fundamentals|Basics of Databricks Lakehouse architecture and workflows.|Free||
|**Databricks Certified Data
Engineer Associate **|Building ETL pipelines and managing data workflows.|$200|Ankit Mistry's Udemy courses|
|Databricks Certified Data Engineer Professional|Advanced data engineering skills on Databricks platform.|$200||
|**SnowPro Core Certification **|Foundational knowledge of Snowflake architecture and operations.|$175||
|SnowPro Advanced Certification|Advanced expertise in complex Snowflake solutions and optimizations.|$375||
|SnowPro Advanced: Data Engineer|Data modeling, ETL, and tuning on Snowflake.|$375||
|Astronomer Certification for Apache Airflow Fundamentals|Core Apache Airflow concepts, including DAG authoring and scheduling.|$150|Mark Lamberti's Udemy course|
|Confluent Certified Developer for Apache Kafka|Developing applications with Kafka, architecture, and APIs.|$150||
|dbt Analytics Engineering Certification|Building and maintaining data workflows with dbt.|$200||
|HashiCorp Certified: Terraform Associate|Managing cloud resources using Terraform.|$70||
|Data Management Fundamentals Exam|Core principles: data architecture, governance, and quality.|$311||
|Data Governance Specialty|Best practices for governance, compliance, and data quality.|$311||

Tips to save money on these:

* AWS offers 50% discount on next exam: So after you give your first certification you can use a coupon code for the next ones.
* Azure - Coursera prep courses for Azure certifications offer 50% exam discount upon completion.
* For Airflow Fundamentals - Astronomer sometimes runs a promotion to get the certification for free. Follow them/Marc on LinkedIn to know the dates - I got mine in Jan last year.

**➡️Dive deeper! - Checkout my playlist "Data Engineering Career" with details of all of the above -** [https://www.youtube.com/watch?v=5b4CIon\_1pY&list=PLYAUClNVzmDN5D9IW-COX0xy\_8fz8r51k&ab\_channel=AnalyticsVector](https://www.youtube.com/watch?v=5b4CIon_1pY&list=PLYAUClNVzmDN5D9IW-COX0xy_8fz8r51k&ab_channel=AnalyticsVector)

Thanks, hope it added some value! All the best!

https://redd.it/1hzf2uf
@r_devops
Cross-platform alternative to npm scripts

I really enjoy the ability to create alias for a command using "package.json.scripts" in Node. Is there a lightweight tool that allow me to do that cross-platform, workspace-first and allow comments?

P/s: I've looked into just, make, mise but they don't seem to do what I want. Mise seems to be very close but it is also quite heavy to install.

https://redd.it/1hzlzgw
@r_devops
University and learning - devops

Hi, in just over a year, I'll finish my studies in telecommunications and IT. I have a broad foundation in various IT topics, a bit more knowledge in networking, but nothing particularly advanced apart from that. Based on my interests, I’d like to follow the DevOps path and learn more about it before starting an internship.

What learning path should I choose? What steps should I take to make it logical and effective? What should I focus on the most? Should I use platforms like Kode Kloud, for example?

And one more thing: what are the intermediate job positions for someone aiming to become a DevOps engineer in the future?

https://redd.it/1hzobzo
@r_devops
Best practice to filter out bot traffic and spammers?

I launched a site which gets some pretty heavy traffic currently. When i look into my logs it seems like a good chunk of the traffic is trying some wordpress vulnz and other stuff i dont want to have on my server.

Whats the best way to mitigate this traffic? I already have fail2ban configured for SSH. Would it be suitable to use some other tooling?

https://redd.it/1hzuyth
@r_devops
Display host cursor when recording a video on headless windows machine via RDP

I am connecting to a headless windows machine via RDP hosted on Google Cloud Platform via Remote Desktop. I need to record some videos on this windows machine. However, every time I try to record something, the mouse cursor does not appear. I am wondering how I can display the host's cursor on a headless windows machine? All of the solutions I've seen so far have been able to access the host device's hardware, but in my case, I cannot do that.

https://redd.it/1hzxrzs
@r_devops
Communities in Pittsburgh

Anyone know of any AWS communities in Pittsburgh. I’m new to learning cloud and would like to connect with others to learn and attend events etc.

https://redd.it/1i003rv
@r_devops
anyone has experience working at this place?

husband got offered devops position in Miami by millennium management, how is there tech work culture? high pressure like their finance division? crazy work hours? treatment ? thanks !

https://redd.it/1i02918
@r_devops
Next level after DevOps (what role is better paid: SRE, DevSecOps, MlOps, Platform Engineer, Cloud Engineer)

Currently DevOps, looking forward to reaching the next level and earn more.

What role is better paid and future proof: SRE, DevSecOps, MlOps, Platform Engineer or Cloud Engineer, etc.?

https://redd.it/1i02szp
@r_devops
How are you tracking changes in 3rd party tools that could disrupt your CI/CD pipelines?

I am curious on how you all in the devop world keep track of SaaS application updates to keep on top of potential breaking/high impact changes.

https://redd.it/1i04tby
@r_devops
Any 30+% discounts for Certified Kubernetes Administrator?

I'm an up-and-coming DevOps engineer and want to complete CKA certifications.
I couldn't procure enough funds during the Cyber Monday sale, hence missed the offer.
I want to do it now but I got only 50-60% of the full price. Hence looking for discounts of maybe 40-50%.
If anyone knows of any such coupons, please DM me. Would really appreciate it. Thanks

Apologies if this is not the right community to post in.

https://redd.it/1i064iw
@r_devops
Cross-platform vs. single platform—what’s better for early apps?

I’ve been debating whether to expand my app to iOS or focus solely on Android for now. Developing for both seems like a good idea, but it’s also a lot of work for a small team.

What’s your take? Did starting with one platform work better for you, or is it smarter to go cross-platform from the beginning?

https://redd.it/1i0bdsb
@r_devops
I automated myself out of my job. That's a first.

I expected it to happen at some point in my life, but not that early.

Worked at a smaller company (50 devs) - it was pure hell at the beginning.
Within about three years we fixed every problem and automated/standardized everything that might disturb the developers workflow. I tutored everyone and documented everything. We actually got the ball rolling to a really sweet spot.

The last few weeks were pure boredom. Since there were no legit projects left.

Well. Now they kicked me out of the company. Nothing left to do.
I'll get full salary for two months and don't have to work a second anymore.

WTF?

https://redd.it/1i0ctdt
@r_devops
How can I overwrite name and password during setup jenkin

Hi , I stuck with find way to set name and password jenkin during setup ,
Is there any way? Thanks

https://redd.it/1i0dteg
@r_devops
Database skills and daily tasks

Hello everyone,

How often do you get tasks or work on databases in your role? What kind of tasks are they typically?
Also, what is the expected level of database knowledge for a junior DevOps engineer? Do you have any recommendations for courses or books to learn database skills relevant to devops?

Thanks in advance!

https://redd.it/1i0fkky
@r_devops