Please guide me in learning infrastructure automation

I currently manage a few servers running some ecommerce sites (WordPress) and some custom PHP based applications (Vanilla PHP, and Laravel) on DigitalOcean. My setup is pretty basic and consists of

* Fedora Cloud OS (I upgrade servers every 6 months for my sanity)
* Nginx, PHP-FPM (multiple pools), MariaDB, Valkey (Redis)
* Postfix (send-only mail server), OpenDKIM
* Logrotate (to rotate logs per user)
* Cron job for files and db backups to each user's directory, logrotate renames the backups and retains last x days of backups.

Earlier, when servers were few, I used to do setup and configure these manually. Server would be taken down a couple of hours for maintenance and upgrade every 6 months.

Then I did basic automation and configuration by writting custom bash scripts. The maintenance time reduced from hours to less than 30mins every 6 months. Downloading backups and restoring them is the only thing that consumes more time now as the data is huge.

I'm now at a stage where I need to fully automate it as the number of servers are growing each month. From what I've understood, I need to:

* Switch from Nginx, PHP-FPM to Caddy & FrankenPHP
* Containerize each application. We currently use docker-compose for development and testing. I guess we need to learn how to use that safely in production.
* Switch from raw logs to ELK stack.
* Switch from Postfix, OpenDKIM to Maddy/Haraka/Postal setup on a separate server and use SMTP from others server to this server.
* Switch from Fedora to some LTS OS like Ubuntu.
* Switch from bash scripts for setup and configuration to something like Ansible combined with Terraform and Nomad (not sure about these two).
* Add replication to MariaDB.
* Add CI/CD pipelines with Github Private repo.

I'm quite overwhelmed and it's taking a lot of time to wrap my head around these things. I know I have to take it slow and not do it all at once.

Have someone been through such manual to fully automated setup? How did you figure your way out? Please guide me if you have any experience with any of these.

Edit: List formatting.

https://redd.it/1kf0qts
@r_devops

What’s one cloud concept that took you way longer to understand than expected?

For me, it was IAM on AWS. At first, it seemed simple—just give users permissions, right? But once I got into roles, policies, trust relationships, and least privilege... it felt like falling down a rabbit hole.

I kept second-guessing myself every time I tried to troubleshoot access issues. Even now, I still double-check every policy I write like three times 😅

Curious—what was your “wait, why is this so complicated?” moment when learning cloud?

https://redd.it/1kf2vqj
@r_devops

LLMs ('AI') are coming for our jobs whether or not they work - Chris's Wiki

From here:

> In most non-tech organizations, both internal development and system administration is something similar to janitorial services; you have to have it because otherwise your organization falls over, but you don't like it and you're happy to spend as little on it as possible.

https://redd.it/1kf2kzc
@r_devops

Ibm Event notification question

Hello everyone,

I am having difficulties to configure my alerts with different templates.
Maybe can someone help me?

In Event-notifications i have created a Source.
In this sources i have 2 Topics.
I have 2 subscriptions and 2 templates.

But only one of the template is used to send the alerts to slack.

How can i change that?

Ideally would be to write the Template query to call the alert description on slack.
Is this possible?

https://redd.it/1kf8w1x
@r_devops

Passive FTP into Kubernetes ? Sounds cursed. Works great.

“talk about forcing some ancient tech into some very new tech wow... surely there's a better way” said a VMware admin watching my counter FTP strategy😅

Challenge accepted

I recently needed to run a passive-mode FTP server inside a Kubernetes cluster and quickly hit all the usual problems : random ports, sticky control sessions, health checks failing for no reason… you know the drill.

So i built a Helm chart that deploys vsftpd, exposes everything via stable NodePorts, and even generates a full haproxy.cfg based on your cluster’s node IPs, following the official HAProxy best practices for passive FTP.
You drop that file on your HAProxy box, restart the service, and FTP/FTPS just work.

https://github.com/adrghph/kubeftp-proxy-helm

Originally, this came out of a painful Tanzu/TKG setup (where the built-in HAProxy is locked down), but the chart is generic enough to be used in any Kubernetes cluster with a HAProxy VM in front.

Let me know if anyone else is fighting with FTP in modern infra. bye!

https://redd.it/1kfa7mz
@r_devops

Restart Operator: Schedule K8s Workload Restarts

github: [https://github.com/archsyscall/restart-operator](https://github.com/archsyscall/restart-operator)

Built a simple K8s operator that lets you schedule periodic restarts of Deployments, StatefulSets, and DaemonSets using cron expressions.

apiVersion: restart-operator.k8s/v1alpha1
kind: RestartSchedule
metadata:
name: nightly-restart
spec:
schedule: "0 3 * * *" # 3am daily
targetRef:
kind: Deployment
name: my-application

It works by adding an annotation to the pod template spec, triggering Kubernetes to perform a rolling restart. Useful for apps that need periodic restarts to clear memory, refresh connections, or apply config changes.

helm repo add archsyscall https://archsyscall.github.io/restart-operator
helm repo update
helm install restart-operator archsyscall/restart-operator

Look, we all know restarts aren't always the most elegant solution, but they're surprisingly effective at solving tricky problems in a pinch.

Thank you!

https://redd.it/1kfbkfl
@r_devops

Does anyone here use Humanitec? Feedback wanted!

I’ve been looking into Humanitec and I’m curious to hear from people who are actually using it.

What use case(s) you’re solving with it?
How it's integrated into your workflows?
Any wins or challenges you've encountered?
Would you recommend it to others building platform tooling?

I’m especially interested in any honest pros and cons.
Appreciate any insight you can share!

https://redd.it/1kfcpze
@r_devops

I got my first devops position

I'm really happy about this but I don't have a lot of experience. I'm Actually straight out of college. I studied what kubernetes and docker was and even went to linenode to create a kubernetes cluster to get some experience. After messing around a bit I realized I have no idea what to do with this stuff.

I start working a few weeks and I'm a little worried I'm going to go in just not knowing enough, which they probably know. I was wondering if anyone here had any advice on what I could maybe do in the meantime to get prepared. My current goal right now is to just get better with bash scripting because it seems like that's really important.

Thanks in advance!

https://redd.it/1kfce49
@r_devops

Any experience monitoring Redshift

Does anyone have experience monitoring Redshift? We've been having a series of data incidents and we're lacking visibility for what's happening with various jobs. The team usually resorts to tracking various sys_xxx tables to investigate failures. We're also using dbt, which writes some state to tables in Redshift as well. We're using Datadog and pulling in metrics for both Glue and Redshift, but none of those seem to be particularly helpful. I'm looking for any tips anyone has.

https://redd.it/1kfcfcy
@r_devops

Got a 3hr interview coming up. Tips/advice appreciated.

I got through the recruiter screening, a meeting with their main DevOps guy and CTO. I got notified that I'll be moving forward to the next round which is a 3 hour interview with other members of the team. I doubt it's going to be 3 straight hours and it'll probably be more like 3 1 hour blocks.

Anyways,
Any tips, advice, or suggestions? The interviews I already did were pretty chill and I think this might be the last round. The company is pretty cool and in a space where I have some expertise which I think gave me a leg up, I really want the job so help me get through the final push. A little background, I got about 10 years of full stack engineering experience and about the last 5ish years I've been exclusively doing DevOps

Oh edit to add: this is all completely remote

https://redd.it/1kffm0m
@r_devops

Best CI/CD tool

I love TeamCity, it looks great, it's easy to setup and it's easy to work with. The issue at hand tho, it is written in Java and requires over of 4GB free RAM which is just insane.

Is there a product that is as easy to deploy via Docker Compose, is as quality of a product and is more optimized?

https://redd.it/1kfi1d7
@r_devops

How do you inspect what actually changed in container images? (My Git-based approach)

Hey everyone,

When working with CI images or debugging build issues, I often need to understand *exactly* what changed in a container layer - not just which files were added or removed, but what was inside them.

[**Dive**](https://github.com/wagoodman/dive) is a great tool for exploring layers, but it mainly shows file names and status changes - not full file diffs. I wanted something more powerful and familiar.

So I built [**oci2git**](https://github.com/Virviil/oci2git), a tool that converts any OCI-compatible container image into a Git repo. Each image layer becomes a commit.

With it, you can:

* Run `git diff` between layers and see actual content changes, even better - use VSCode for ex, or [**lazygit**](https://github.com/jesseduffield/lazygit)
* Use `git blame` to find which layer added or modified a file
* Explore the entire filesystem history with regular Git commands

It’s been helpful for auditing, debugging, and understanding image composition more deeply. Would love feedback, and I’m curious how others inspect images: Dive? manual tarballing? something else?

https://redd.it/1kfkb6c
@r_devops

Personal Blog and Portfolio: Feedback?!

I have posted many blog articles on GitHub and other sites before and decided I want to have a personal homepage where they are all to find. I want to use this website as my portfolio as well.

It's fully open source if anyone is interested:

Repo: https://github.com/LukasNiessen/personal-website

Website: https://lukasniessen.com

Any feedback or thoughts are highly welcome :-)

https://redd.it/1kfmav7
@r_devops

Grafana Dashboard + Metrics For MCP Servers

I put together a Grafana Dashboard and metrics implementation for MCP servers. I thought some of you, might find it helpful. full post and code source here

https://redd.it/1kfnh0t
@r_devops

Any advice for fake it till you make it with AWS specifically?

Need some input on how to appear to know what I'm doing with AWS lol

https://redd.it/1kfrc3t
@r_devops

Snyk/Bitbucket?

Anyone here have practical experience using the Snyk integration on Bitbucket? We're pursuing SOC 2 compliance and one of the checks requires CVE scanning of code during CI/CD.

Other major CI/CD platforms offer free scanning like Dependabot, but sadly, we are on Bitbucket (constant irritation/constant disappointment), so we're looking at our options. They offer a Snyk integration, which (at our scale) will require a non-free Snyk plan.

Anyone gone through this? Happy to entertain alternatives, but we are likely to stay on BB because our company is all-in on Atlassian.

https://redd.it/1kftph6
@r_devops

Got ghosted after 3rd round

Hey everyone,

Just wanted to share my recent experience and see if others are going through the same thing.

I’ve been applying for DevOps roles for the past few months, and finally landed an interview. It started with a quick HR screen, followed by a technical round, which went well and I was immediately moved to the next stage.

The third round was a DevOps challenge, which I completed over my weekend. I presented it, answered all their technical questions, and felt the interview went smoothly.

I followed up with HR the next day — no response. I waited a week and followed up again — still nothing. Then I sent a message on LinkedIn just in case, and even followed up with the second HR contact mentioned in the original email — still complete silence.

At this point, I’m feeling pretty frustrated. It’s disappointing to invest so much time and effort, only to be met with no closure. Is this kind of ghosting becoming normal now?

Would appreciate hearing if others have gone through something similar, or any advice on how to deal with it.

https://redd.it/1kfuk0z
@r_devops

Anyone facing issue with Cloudflare recently of suddenly not honoring "Access-Control-Allow-Headers" set by origin?

Is anyone facing this recent issue lately where all the sudden, you're getting thrown Access-Control-Allow-Headers error across all proxied domains. Cloudflare proxy, out-of-the-blue, decided not to honor the Access-Control-Allow-Headers set by origin, and decided to block most headers, including "Authorization". This caused temporary downtime across all our services, totally unacceptable.

We had to remove proxy across multiple of our domains temporary and we can't find any changelogs, issues, etc. regarding any changes or reported issues to Cloudflare proxy anywhere (which is strange).

https://redd.it/1kfue56
@r_devops

Stategies for scaling out MySQL/MariaDB when database gets too large for a single host?

What are your preferred strategies when a MySQL/MariaDB database server grows to have too much traffic for a single host to handle, i.e. scaling CPU/RAM or using regular replication is not an option anymore? Do you deploy ProxySQL to start splitting the traffic according to some rule to two different hosts?

Has anyone migrated to TiDB? In that case, what was the strategy to detect if the SQL your app uses is fully compatible with TiDB?

https://redd.it/1kfwpne
@r_devops

LogWhisperer – AI-powered log summarizer that runs locally (no OpenAI keys, no cloud)

I built an open-source CLI tool called LogWhisperer that uses a local LLM to summarize Linux system logs into human-readable summaries. It’s useful for triaging noisy logs, quick postmortems, or just getting a sense of what the hell happened without manually parsing journalctl.

Key features:

Uses a local model (via [Ollama](https://ollama.com)) — supports `mistral`, `phi`, etc.
Parses logs from journalctl or file paths (e.g. /var/log/syslog)
CLI-friendly with flags for source, priority, model, entries
Outputs markdown reports for easy archiving
Includes a spinner so it doesn't feel frozen when summarizing large logs
100% offline (after install) — no OpenAI keys or cloud dependencies

Use case: you're SSH'd into a flaky VM, and you just want a summary of the last 500 err\-level logs without sifting through pages of noise.

Install it with a one-liner shell script — it sets up the Python env, installs Ollama, and pulls the model.

GitHub: https://github.com/binary-knight/logwhisperer

Would love feedback from fellow infra folks. I'm also thinking of extending this into scheduled cron-based summaries, Slack alerts, and anomaly tagging if anyone’s interested in contributing or ideas.

https://redd.it/1kfyv61
@r_devops

What really makes an Internal Developer Platform succeed?

Hey, I work at Pulumi as a community engineer and as we are doubling down on IDP features I’ve been looking around at various other platform tools and it's hard for me to tell which features are great for demos and which are really the important pieces of an ongoing platform effort.

so, in your experience what features are essential for a real world internal developer platform? and how are you handling infrastructure lifecycle management or how would you like to be handling it? I’m more interested in the day-2-and-beyond messy bits of a platform approach but if you are successfully using a 1-click to provision portals I'd love to hear about that as well.

https://redd.it/1kg3gj4
@r_devops