GitOps - The Healthy Work Process?

Hello everyone :)

I'm quite new to GitOps, so I appreciate any piece of advice.

In the company where I work, we have a system that is maintained by several different teams.

Our process looks like this:


1) A developer merges application code to master
2) The new container tag is pushed to the GitOps Manifest repo (branch per environment approach)
2) A CI job is triggered by the change in manifest that run deploys the charts using helm upgrade.


If the deployment fails to boot, we need to manually rollback the manifest to a prior version, while meanwhile other deployments occur at the same time.


We thought of integrating ArgoCD to use Auto-Rollbacks. But we encounter some issues:
1) If you use Auto-Rollbacks you can't use Auto-Sync.
2) The rollback only rollbacks the cluster state, and leave the GitOps state out of sync, meaning that a manual intervention have to take place. If in the meanwhile additional deployments are committed before someone fixed the bad deployment, the bad deployment will hit again.


Any solutions or thoughts?

https://redd.it/11clgl2
@r_devops

Anyone else feeling that AWS & Kubernetes is going downhill as a career path?

Lately its becoming increasingly hard to convince companies to hire an expert in the field - so a strong senior.

And it's not like it only started with the recession FUD.

Let's look at some stats to back this up.

For the past ~6 months, majority of cloud contracts posted across UK have been almost exclusively Azure & Google Cloud, even though they have much smaller market share than AWS.
Barely any AWS contracts.

Companies believe they can get a cheap "good enough" permie to do AWS work, company won't collapse, and upgrading the role to a senior level and senior pay, or a contract even, doesn't provide enough benefits anymore.

If that trend will continue, I think we're only going downhill from this point.

The same goes for Kubernetes.

Everyone is using it, but if you look at who is really using at scale, doing the most complex operations, it usually boils down to just a couple biggest banks and telecoms, and they sure need the experts.
And while I've worked for banks in the past, it's just too niche to make a living out of it long-term, especially in this economy.

The number of times I've been undercut out of senior, expert level job positions, just to see them downgrade the posted pay range and take someone with 2-3 YoE in DevOps has been extremely frustrating, and keeps happening time after time.

For context: I am a senior with ~10 YoE, trying to navigate a difficult remote market while living in Eastern Europe -
while geoblocking of job opportunities in IT keeps increasing to pre-covid levels

https://redd.it/11ci18h
@r_devops

figuring out the right ci/cd pipeline

I've read articles and posts that describe pipelines that are close but still don't click in my mind.

We develop services for kubernetes.

Each service with it's repository.

We're using bitbucket if that's relevant.

We have a few environments (let's say dev, staging, prod) and each one in a separate cloud project.

What I want to achieve is having each service repository have their own helm chart with all the templates required for the service to function and values for developers to be responsible for.

I also want to have argocd in each environment to keep syncing changes and preventing configuration drifts.

I can't figure out how the development and promotion of a new feature should flow to dev -> staging -> production from service repository.

Let's say I work on billing service and I branch out from master to a feature branch. I'm done tweaking the code and updated the helm chart and values file. What would be the process of pushing these changes so only dev will be updated? And then how do I promote staging?

Creating branches for each environment in a service repository is against trunk based development idea.

I can create a helm chart + values folder per environment in the main branch of the service repository and argo from each environment will look there but that means I need to constantly merge the main branch to do development tests.

Ideally after branching from main developer will be able to push and updated chart and values will go to dev so it can be tested by developer and once he is happy it should go to staging (maybe that's merge to main but than how do I auto pull request for prod deployment?).

So yeah... I kinda feel lost in the woods and I feel like the solution is much simpler than I think.

https://redd.it/11gxp02
@r_devops

How do you guys deal with Devops tools like docker, ansible, terraform and Kubernetes with all the different commands they individually have.

As the tittle says. I’ve been self studying devops tools n keeping track of all the different commands when switching different tools could be overwhelming. How do you guys deal with this. And in how many of these tools should one be really good at as a devops engineer?

https://redd.it/11bz4w2
@r_devops

SMS pumping attack

How to protect against SMS pumping attack for an app that is hosted on AWS

https://redd.it/11h07d6
@r_devops

Looking for teammate to join project (not a job posting)

Hi Sub,

I'd like to make an employee/dev onboarding system connected to azure ad/iam/or..whatever identity manager we think is fit. Make automated infrastructure/workspace deployment for That Dev/Ops/QA/.. or whatever employee is it. Have configurable groups/roles/policies/OUs(if aws)/Mgmt.Groups(if azure). Have VNC option too.. Onboarding user should give all necessary accesses and remote dev environment setup in minutes not days. There are solutions similar to this.. not gonna name them here, maybe not exactly like this.
Also this has to be zero trust network and access could be done with something like HC Boundary, obviously all resources needs to have proper tagging. Last thing is, I'd like to write boilerplate for this and keep it open so others can add modules for this. Say if someone wants to have DynamoDB/LakeFormation be part of the access grant during onboarding they can write their module for that. This all supposed to deploy with Terraform and could be run on vm,k8s,docker.. basically every time via CRI(container runtime interface) but on different machine setups.

If this makes sense to you Let do it together.

I'm looking for someone skilled or who'd like to put hours to improve.

I work as a DevOps, ex SWE., it's too much for 1 person to write, thus this post.. Thanks.

https://redd.it/11h3jyz
@r_devops

How to learn system performance as a beginner?

Is there any labs that I can built or do to learn it if I don’t have real life experience? I know Brendan Gregg has a great book but I can’t relate to it…

https://redd.it/11h343b
@r_devops

What's the nicest on-call scheduling and paging SaaS these days?

I've used both OpsGenie and PagerDuty in the past. I could just choose one of these, but I wondered, what's the new hotness for on-call scheduling and paging? The tool that everyone wishes their business was using?

I've seen LinkedIn Iris and LinkedIn OnCall, and I assume they're probably very good, however the team is tiny and we really don't want to run the software ourselves (and have to make sure it's dependably). So SaaS tools only, I suppose.

https://redd.it/11h2hi5
@r_devops

Is it possible to do infra as code without a virtual machine in windows?

Not a devops guy, so this question could be silly but looking to do this:

Terraform + Ansible to do auto provision
Gitlab
Where: Azure VM or bare metal running Windows Server
Someone said I need something like Oracle Virtualbox

https://redd.it/11h9bmh
@r_devops

Linode raising their prices by 20%, any good alternative?

I remember using Digital Ocean back in the days telling myself how cheap and nice they are (compared to AWS 🤡) until they raised their prices.

I then switched to Linode, that was well.. Not quite as pleasant to use, but hey what a great value for money..

..Until Akamai messed everything up again by bying Linode and raising the prices by 20%, without notice, 2 months later.


I am now looking for a cheap, good value for money cloud hosting, any idea?

https://redd.it/11hc8za
@r_devops

Devops and Platform Engineering

DevOps? Platform engineering? Has this been buzzing you for quite some time now? look no further and just watch the latest video that I just dropped on DevOps vs Platform engineering!
Remember Internal developers platforms are not built in air 😂 You need devops, sre, programmers to build that to be used by other teams. Everyone can have their own version of platforms like they have for DevOps Imo this is a self service tool for teams but IDP itself needs to be managed Watch the complete video And share your thoughts/opinions as well.

https://youtu.be/Jip3lBnxKXU

https://redd.it/11hbst9
@r_devops

What is the most tedious task?

We made so much progress in the last years, improvement all around that empower teams to execute their work with not much hazard.

But what is that thing that makes you miserable? That chore that pains you every day and, for some reason, no one has taken it away?

https://redd.it/11hbor9
@r_devops

Career Advice - focus on Leetcode or try to get my CKA

I'm at a crossroads in my career and I'm not sure what to focus on next. I kinda let my career take me where it wanted to take me for the last 5-6 years and I've somehow done a lot of DevOps for Security things, mostly in the Cloud Security and Infrastructure Security spaces.

However, I've completely missed the boat on K8s. Most of the last 6 years has focused on IAC for AWS Security related things (AWS IAM, AWS Firewalls, AWS security related to various primitives) or IAC for vendor appliances. None of this stuff has given me much exposure to K8s.


Looking for a new job recently, all the job posting I run into need me to be great at K8s and be great at Leetcode. I can coding basics, but I can't sail thru Leetcode easys - they take me a while and sometimes all I got is O(n2) loops. I haven't touched K8s much at all.


Since I seem to be stuck right now at my current job/level, which skill should I try to pick up first? Grind more at LC or try to get my CKA? What's more valuable in scaling up in my IC career in DevOps/Security?

https://redd.it/11hb96x
@r_devops

Im going crazy managing multiple services without docker. Could use some wisdom from others.

I’m a bit above average programmer with minimal devops experience and a big homelab who needs help keeping my sanity in check. Basically I think I’m making things harder then they need to be.

The problem:
I’m looking for solutions on the best way to manage (start/stop + view logs) for multiple services that can’t be run in docker. When I want to start my code on my main machine it requires staring two node.js apps (including the UI) and seven golang services; in addition to 5 docker containers. Without going into detail, just trust me when I say the go services can’t be dockerized without taking a sizeable performance hit. When I want to spin up the code on my second machine so it can run concurrently and pass data to the main machine I have to start the same services minus docker. Basically it’s 9 terminals running in the background each with a special command to get started which is driving me a bit crazy and makes me not want to write code because of having to restart them all.

The solution I know of:
The only thing that comes to mind for managing these is a bash script.

I’m hoping others with more experience can provide me with better options.



Other miscellaneous details which may/may not be relevant:
Epyc 32core with 256gb memory, 20tb ssd’s and 12gb GPU. Ryzen 12 core 128gb memory and 12gb GPU. Both run Ubuntu desktop on bare metal. Also unraid with 400tb of storage. When I run my code it will saturate my 1gb connection and all machines run at 75% load.

https://redd.it/11hi1ij
@r_devops

SonarQube is complete dog sh*t.

I have nowhere else to express my complete hatred of it. I honestly believe they roll out bugs on purpose to push people into the paid version.

https://redd.it/11hjzze
@r_devops

Any tool out there with dynamic parameters except Jenkins?

My team heavily uses the "Active Choice" Jenkins feature to allow developers to select from a dropdown generated from a script. This can be to choose from git branches, files in s3, keys in a database table, and so on..

This seemingly simple feature is apparently nowhere to be found on any open source automation server or CICD platform except for Jenkins. All of the others only allow for hard-coded parameters as simple text or choice inputs.

The lack of this ability is the main, if not the only blocker for my team to move on to another tool, basically the "self-service" capability it gives for developers to use the Jenkins UI.

So, as the title suggests, any other tool, free or not, that provides this capability or wraps another tool with it?

https://redd.it/11hl5y8
@r_devops

Release scheduling/calendar???

What do ya'll use for release scheduling/calendar? Currently, we're tracking development, smoke testing, QA validation, and deployments via Excel... I HATE it!

https://redd.it/11hjji0
@r_devops

What are other katakoda like free devops learning platform available online? From what I understand after O'Reilly bought them is not free

All in the title

https://redd.it/11hmj14
@r_devops

People with Linux work laptops: How does it work at your company?

To elaborate on the title, I'm curious to learn more about how other companies provide access to developers to run Linux as their main work OS. This is both from the perspective of how can I pitch this to my current employer, as well as how can I filter jobs in the future to find a place that won't force a Mac (or worse, Windows) on me, and avoid horribly locked down Linux (say, packages and DE enforced by IT)

Some example questions, please expand on it or share relevant things in any form that suits you!

- How big is your company?
- Does IT provide an install, or do you get to pick your distro / do the setup yourself?
- Do you get the declared holy config of the year? Pick your own flavor from a small set of machines? Pick your own device freely with a budget?
- What corporate spyware does your company use for Linux?
- Is this managed by central IT, or an exception special for your team?
- How did you dig for this, and more importantly the details of it, during the interview?
- What's your corp VPN, video chat, text chat, work calendar, office docs, project management + git repo stack like?

Thank you!

https://redd.it/11hpgij
@r_devops

How does this optimize the package managers?


RUN echo "https://dl-cdn.alpinelinux.org/alpine/edge/main" > /etc/apk/repositories && \
echo "https://dl-cdn.alpinelinux.org/alpine/edge/community" >> /etc/apk/repositories && \
echo "https://dl-cdn.alpinelinux.org/alpine/edge/testing" >> /etc/apk/repositories


We're just writing some url into a file. How is it going to optimize the package manager?

https://redd.it/11hojt0
@r_devops

How do you guys enjoy this?

Some context for y’all, i graduated in spring of 2022 with a degree in CS and immediately accepted an offer at a big bay area tech company as a devops engineer on the devops team. They promised me i’d learn soooo much and that’d it’d only require a little hard work at the start. Fast forward to a year later and all i’ve learned is how to navigate through the aws console, run jenkins jobs, install/update software on nodes, and change orgs passwords lol. They don’t let me or any of the other new grads touch the python automation scripts or do any developer related tasks. On top of that, there are usually 3 days of the week where i’m working 7:30 am to 7/8 pm and i don’t even have an organized routine at this work which seriously irritates me. I hate having to stare at my laptop all day waiting for a new issue to remediate. Id prefer to have a scheduled plan going into my workday. I feel burnt out as hell and with the shitty job market, i’m probably trapped at this company for another year. Don’t get me wrong tho, i really do want to enjoy my job and learn as much as possible but i feel so limited on this team and tbh deploying/monitoring services and fixing the bugs that appear in that process is just flat out boring to me. Is this how it is for everyone or am i just not at the right company?

https://redd.it/11hojst
@r_devops