What's at the top of the salary mountain?

Senior SRE here with 5 years in the industry. In the EU I'm making ~130,000 EUR a year. It doesn't look like I can go any higher in the "DevOps track" unless I move to the US. Is this the top of the mountain?

Where do Senior DevOps/SRE people tend to move from here?

https://redd.it/sp8u76
@r_devops

Developer access workflow

I work on a lot of projects that interface with the AWS API, getting info mostly on EC2 and S3, but random asks come up here and there which require me to get access to different API endpoints from various roles across disparate accounts.

The devops team at my company is very short staffed right now, so requesting aws api permissions takes longer than usual lately, and most of that time is just sitting in the queue rather than being worked on. It seems like a lower tier security person could approve these rather than having devops engineers have to approve everything.

What workflow do you have in your org for getting these permissions approved?

I thought a good tool would be something that converts a yaml or json document into a set of permissions (running the underlying API calls). The developer who WANTS the permissions could just create the document that describes the permissions needed, then either pushes to some git repo (which helps with accountability) or some other process to get approved. Then, the devops resource can simply approve/deny rather than having the ticket sit in Jira un-approved while no work is done on it and Devops works on my valuable uses of their time

https://redd.it/spbv5h
@r_devops

What is the role and function of a Infrastructure Automation Engineer?

I'm trying to transition from web development and someone offered I transition to INFRASTRUCTURE AUTOMATION WITH AZURE AND GITHUB. So for a noob, what's the this path about...

https://redd.it/spcgaz
@r_devops

Database Image as a service. What do you think?

Been thinking about it lately. What if there's lightweight portable database image like Docker image where you have datasets you want- queries and procedures already created/stored. This creates consistency in data state between two users- use case: testing/ doing demos with customers. What do you think?

https://redd.it/spddik
@r_devops

How do you manage parallel releases in one environment?

So I am currently working for a customer, they are setting up their contact centres. The contact centres consists of Amazon Connect and the configuration for its Lambdas and DynamoDB. They have parallel releases for deployment - like first release could include payments, another releases includes loans queues etc. Could anyone share any tips and tricks to manage parallel releases in one env(one AWS Account)

https://redd.it/spg6o3
@r_devops

packer build . fails with Ubuntu 20.04

I am trying to automate building a Linux ubuntu 20.04 server using Packer. Everything seems to be running well until I get to the point where it says "Waiting for SSH to become available...". An instance of ubuntu starts running, but while building I get the following subiquity error:

[https://imgur.com/a/wPno2o3](https://imgur.com/a/wPno2o3)

​

==> virtualbox-iso.ubuntu-20043-live-server: Retrieving Guest additions
==> virtualbox-iso.ubuntu-20043-live-server: Trying C:\Program Files\Oracle\VirtualBox/VBoxGuestAdditions.iso
==> virtualbox-iso.ubuntu-20043-live-server: Trying file://C:/Program%20Files/Oracle/VirtualBox/VBoxGuestAdditions.iso
==> virtualbox-iso.ubuntu-20043-live-server: file://C:/Program%20Files/Oracle/VirtualBox/VBoxGuestAdditions.iso => C:/Program Files/Oracle/VirtualBox/VBoxGuestAdditions.iso
==> virtualbox-iso.ubuntu-20043-live-server: Retrieving ISO
==> virtualbox-iso.ubuntu-20043-live-server: Trying https://mirrors.kernel.org/ubuntu-releases/20.04.3/ubuntu-20.04.3-live-server-amd64.iso
==> virtualbox-iso.ubuntu-20043-live-server: Trying https://mirrors.kernel.org/ubuntu-releases/20.04.3/ubuntu-20.04.3-live-server-amd64.iso?checksum=sha256%3Af8e3086f3cea0fb3fefb29937ab5ed9d19e767079633960ccb50e76153effc98
==> virtualbox-iso.ubuntu-20043-live-server: https://mirrors.kernel.org/ubuntu-releases/20.04.3/ubuntu-20.04.3-live-server-amd64.iso?checksum=sha256%3Af8e3086f3cea0fb3fefb29937ab5ed9d19e767079633960ccb50e76153effc98 => C:\Cache_ISO_ITMT_495\62850188884fff34d447798ebc9d9b22bf1f3f1f.iso
==> virtualbox-iso.ubuntu-20043-live-server: Starting HTTP server on port 9025
==> virtualbox-iso.ubuntu-20043-live-server: Creating virtual machine...
==> virtualbox-iso.ubuntu-20043-live-server: Creating hard drive output-ubuntu-20043-live-server\ubuntu-focal.vdi with size 15000 MiB...
==> virtualbox-iso.ubuntu-20043-live-server: Mounting ISOs...
virtualbox-iso.ubuntu-20043-live-server: Mounting boot ISO...
==> virtualbox-iso.ubuntu-20043-live-server: Creating forwarded port mapping for communicator (SSH, WinRM, etc) (host port 3249)
==> virtualbox-iso.ubuntu-20043-live-server: Executing custom VBoxManage commands...
virtualbox-iso.ubuntu-20043-live-server: Executing: modifyvm ubuntu-focal --memory 4096
==> virtualbox-iso.ubuntu-20043-live-server: Starting the virtual machine...
==> virtualbox-iso.ubuntu-20043-live-server: Waiting 5s for boot...
==> virtualbox-iso.ubuntu-20043-live-server: Typing the boot command...
==> virtualbox-iso.ubuntu-20043-live-server: Using SSH communicator to connect: 127.0.0.1
==> virtualbox-iso.ubuntu-20043-live-server: Waiting for SSH to become available...

Any idea as to what might be causing this issue?

https://redd.it/sphjt5
@r_devops

Observations on using PowerShell scripts in Azure pipelines

And, in today's "Nerds of the World, Unite!" corner: PowerShell syntax sucks. I can't think of enough bad things to say about it.

It's so bad, it makes Linux shell script look like plainspoken English.

https://redd.it/spit1x
@r_devops

Two new blogs that might spark the interest of the community here. ArgoCD with Portainer

https://www.portainer.io/blog/deploy-and-use-argocd-with-portainer

https://www.portainer.io/blog/deploy-and-use-argocd-with-portainer-part-2

https://redd.it/spkxf2
@r_devops

jenkins script does not run the nc properly with a parameter

I am trying to add a validation step in my script that will do nc and take hostname as a parameter. I added this remote ssh step

hostname=${hostname} echo $hostname nc -zv $hostname 10000 -w 5

but when running , I get failure with these logs.

SSH executing... usage: nc -46CDdFhklNnrStUuvZz -I length -i interval -M ttl -m minttl -O length -P proxy_username -p source_port -q seconds -s source -T keyword -V rtable -W recvlimit -w timeout -X proxy_protocol -x proxy_address[:port] destination port

What am I missing?

https://redd.it/spk8kr
@r_devops

Is there a job title more preferred over others?

Let me preface by saying I know job description matters way more than the job title. However I’m moving into a position at a small business that lives under both Devops/Infrastructure/Cloud Architect/Engineer.

When this happens I’ll be the first one in that position so it’s possible I’ll get to argue for a specific title. I know from living in the web development world, ideally you get titled “Software Engineer” over software/web dev. Just curious if you guys think there’s something similar in this realm as well when wearing multiple hats

https://redd.it/spoh9w
@r_devops

What have you learned in past 3 months?

Share your learning experience with others.

https://redd.it/sph86w
@r_devops

Tailwinds Workflows: Request your Inputs and feedback

At Tailwinds, we observe that there are many organizations/folks out there who need help in DevOps, Cloud-Native and Kubernetes. Also, they would like to pay for workflows that they would like to execute instead of complete products.

If you are one among them and like an a la carte model of workflows, would love to hear more from you. Pls provide your feedback by connecting to https://apps.tailwinds.ai and share your thoughts

We are providing the first Five workflows for FREE for folks who register early!!. Go grab them before they run out.

https://redd.it/spk2sz
@r_devops

What deficiencies or issues are "must fix ASAP"?

In DevOps, there is always a fine line to walk between striving for technical perfection and accomplishing business objectives.

View Poll

https://redd.it/sps7gw
@r_devops

I don’t know if we’re doing devops engineering right

I started working at this place a few months ago.

Their applications and databases are deployed across a fleet of virtual machines and a kubernetes cluster.

Now, the issue is - I do not know how these applications contribute to the company’s revenue, and I haven’t received any responses about what these applications do either.

I had setup monitoring and alerting for all these pieces of infrastructure, but I don’t see any team bothering to take action on the pieces of infrastructure they own.

With previous teams that i had worked, I was given a peak into their infrastructure and applications and was given full latitude on how to respond to infrastructure alerts.

My understanding of devops engineering over these past three years as a devops engineer have been that we liase with engineering, operations, and product groups at a company and provision infrastructure and deploy application updates (or architect entire infrastructure setups if needed).

But in our case, I haven’t interacted with the engineering groups or the product groups in a work context.

Which is kind of bothersome.

So, fellow DevOps folks - are we (me and my workplace) doing devops engineering right?

https://redd.it/spso5l
@r_devops

Head hunted opportunity

Fellow devops engineers, I’m turning to you for some advice because I’ve been going around in circles - ultimately I know it’s my choice but I want to gauge what you’d do.

Recently got offered a job from a guy I use to work with 7 months ago (we both left our previous employer at the same time). No interviews, just a friendly chat on his company. The company is a startup with about 40 employees with some decent benefits but no way near as good as my current ones. Here’s the mindset I’m in

New offer
- 13K pay rise
- I’d be the second devops engineer
- basically a greenfield, already moving to AKS for their applications
- propel my career a lot faster
- working with someone I have a great dynamic with
- shares / progression would be faster
- go into London once a week (1.5 hour journey each way)
- good people, good company, good value (it seems)
- 10K stock options
- This company is going places fast

Current company
- offered to give me 7K pay rise to stay
- much better benefits around holidays / pensions / budgets etc
- much better socials
- WFH whenever I want
- dogs allowed in the office
- good people, good company, good value
- a little bit behind the new company technology wise but not by far


I’ve been at my current company for 7 months now, and in that time I’ve had a couple of soul crushing moments / news and they’ve said just take a week off or whatever I need. It does feel like they have my back.

My mindset seems to be that if I stay, I’d have a better / easier time, more laid back and can take some time to build me

If I move, my career position in a few years could be insane, and would be working with someone I would really well with.

What’s your thoughts?

https://redd.it/spv983
@r_devops

How do I OpenTelemetry?

Very open ended question, just hoping for some anecdotal support from the community.

We're just starting with the Tracing part of observability now and it's probably the biggest challenge any of us in the devops team has faced so far.

Because it's so tightly bound into the code, that the developers handle.

I think our conclusion for now is that the devs should just go ahead and start implementing the opentelemetry SDKs in their code, small scale, and then we can have an easier time learning how to process the traces from their running code.

Instead of trying to find sample applications and learning on them.

https://redd.it/spx1p8
@r_devops

On-prem K8s guides

I'm being assigned on a project where I'll have to create and manage K8s on prem. There's a possibility of no internet access. Any guides or resources I can use to help me achieve this?

I've used K8s on AWS EKS, FWIW.

https://redd.it/spwra0
@r_devops

On-prem K8s guides

I'm being assigned on a project where I'll have to create and manage K8s on prem. There's a possibility of no internet access. Any guides or resources I can use to help me achieve this?

I've used K8s on AWS EKS, FWIW.

https://redd.it/spwra0
@r_devops

How to Handle Configuration Mismatch in GitOps

The past few days, I've been reading up on GitOps and while the community seems to really like the idea, specifics tend to be glimpsed over. Bottom line, I'd like to use ArgoCD but I have realized that there are a few rough edges that I can't seem to resolve by googling.

Up to now, our application repositories have included the following subtree:

deploy/
chart/
Chart.yaml
values.yaml
templates/
environments/
staging.yaml
production.yaml

Then, in our CI, we deploy to staging via helm upgrade --install (using the staging.yaml overrides for the Helm chart values) once we push to main ("GitLab Flow") and deploy to production (using production.yaml overrides) once main is tagged. On the helm upgrade call, we also set the appropriate image tag.

In most GitOps guides I've read, it is recommended that the deploy folder is put into a different "configuration repository" which I agree has some nice benefits. However, I'm not sure how to handle "configuration mismatches".

Imagine, I want to deploy a new version of my application. The new version now needs to interface with AWS S3 and, hence, needs some credentials attached as environment variables (assume that credential management is not a problem). In the "monolithic" repository with our push-based approach, we would just take care that by merging into main/tagging, the chart is updated appropriately. However, how would I handle that with a pull-based approach? If I use the configuration repository as a source for ArgoCD to pull the desired configuration, I'd need to update the manifests prior to deploying the new application. However, that already causes changes to the old application which is currently deployed. While adding a new environment variable doesn't cause any problems, I can imagine cases, where this is undesirable.

Also, since GitOps guides tend to ignore the CI steps, would the CI in the application repository always commit to the configuration repository to update container image tags?

I'd appreciate if any experienced GitOps engineers could provide some guidance!

https://redd.it/sq1m3k
@r_devops

What is a good way to automate the process of pushing code from the dev branch to the production branch for all of our repos?

As the comment above says any recommendations would be greatly appreciated.

https://redd.it/sq292i
@r_devops

Why do y'all hate Jenkins? (What pipeline tools do you use?)

Seen several posts/comments about people hating Jenkins but haven't really been able to discern why. We use Jenkins to a very basic level for our builds to environments, then self-hosted bitbucket pipelines for pre-merge unit testing. Going to be looking to soon reworking our use of Jenkins to actually write out pipeline steps to account for deploying the first microservice.

Before doing that I want to make sure I do my due diligence in seeing why people hate jenkins and debating on if we should move away from it. If you hate jenkins and have moved to something else, what was it and why?

https://redd.it/sq5c2o
@r_devops