How painful was Log4j for you?

My team, org, and probably company STRUGGLED with log4j remediation. Giant micro service architecture meant hundreds of apps needed their repo's updated, rebuilt, and redeployed. Worse, since log4j, the company has cracked down and implemented intense scanning and remediation requirements across all image repositories. An image in a repository with any CVEs now gets escalated up to war rooms with VPs involved if its not resolved in a few days... Our CI/CD was definitely not prepared for this, and have been struggling to stay on top of our hundreds of running applications as new vulnerabilities are discovered... And we're a BIG company (like, one of the biggest).

Just wondering what other devs experienced... Do you have 100's-1000's of apps and log4j was a walk in the park for your org? Huge impact that brought everything else to a stand still? Are you taking advantage of SCA like whitesource, snyk, etc? What tools do you use that make it so easy or hard to manage high volumes of code/repo level changes like this? Does gitlab just f*ckin do it all for you? And, if a you do have 100s-1000s of apps and the next log4j scenario comes around, are you setup to automatically fix it this time? How?

Any insight anyone can provide would be super valuable! And if you want to DM me and have a deep conversation about it, that's even better -- I have a decent amount of DevOps knowledge and expertise I'm happy to pass along (I'd pay you if I could, but that's the best I got).

Thanks in advance! Really appreciate this community :)

https://redd.it/tjqnij
@r_devops

Interview with Rona Hirsch, DevOps Engineer at Komodor on ValidKube, Female DevOps engineers and women in tech

Link to the interview: https://www.youtube.com/watch?v=bNG5nRXMCFc

I particularly enjoyed the segments about women in tech. I find that Devops is a very male dominated field and it would be fantastic to have more women in the field.

https://redd.it/tjk4jp
@r_devops

Has anyone left Devops to go back to just being a Dev?

I’m starting to get extremely bored and stressed with Devops. I notice alot of developer positions ask for Devops experience and was thinking of switching back. Has anyone done this? Did you have to take a pay cut?

https://redd.it/tk2z7u
@r_devops

Build vs Buy your development platform? Which option do you choose & why?

Some companies prefer to build their own internal development platforms while others prefer to buy something that's already on the market and focus on building their product instead. I'm looking for pros & cons for each option from your own experience. What worked, what didn't, what didn't you expect to happen, etc.

https://redd.it/tkru1n
@r_devops

Learning Python Boto3 and Terraform at Home?

I'm looking to move into devops from a software development role and wanted learn Boto3 and Terraform. What is the best way to practice using these to simulate real world experience/use cases? I can do simple scripts using the documentations, but can't think of "bigger"/challenging scripts to write. Any ideas or suggestions would be great!

https://redd.it/tnbwfd
@r_devops

Running Postgres 10 in production. Should I just upgrade to 11, or use the maintenance window to jump to 12/13/14?

Might be a dumb question but I'm wondering if it's ever common to jump multiple major versions?

Obviously we would test it thoroughly in our lower environments. We're using multi-region clusters in RDS.

https://redd.it/tn1tr9
@r_devops

is it just me or there is no sonarqube equivalent in azure DevOps?

was asked to implement sonarqube, saw how its a bit clunky to install and maintain. tried to look for something more seamless and integrated into azure DevOps pipelines (so no need to config/deploy our own sonarqube/code analysis tool) but couldn't find something that competes it. am i missing the mark or there is no such solution by azure and/or the tasks in azure devops? thanks.

https://redd.it/tna3bx
@r_devops

Kubernetes, Helm and automated deployments that read the kubernetes status

For my understanding of Helm so far, it seems that it is "write only (in etcd) or deploy only".

That is, a chart done with helm deploys something in kubernetes, it can have multiple phases (with pre hooks and post hooks and so on), but it doesn't really reuse the status of what is being deployed.

Trivial example: Helm cannot read the name of a pod that gets deployed during a deployment (that pod name will have an hash, so it is cannot be precomputed before).

I know that in the best case a chart deploys what is needed without much need to use the state of what is deployed, but unfortunately time constraints and other limits often prevent the best case (for the same reason Agile is mostly never applied, or the devops aproach is also mostly never applied).

Hence the question: is there a way in Helm to read the state of what was deployed during a deployment and use it for further actions? (I know one could wrap helm in scripts and make two deployments or the like, but that is clumsy)

If Helm cannot do it, is there any other deployment manager that can do it?

https://redd.it/tnh4fq
@r_devops

How do you manage your terraform library?

Just genuinely curious what your iac practice and pipeline is like. I support many teams and find myself writing terraform for them since they either don't know terraform or won't write it given my desired standard. I am contemplating publishing a library of modules I use and having my developers use them but I am not sure how I can enforce this or allow them to combine them in different combinations. Any ideas?

https://redd.it/tn2um5
@r_devops

Data center migration due to geopolitics

Some customers of ours are either upset about what's going on in Eastern Europe or they're anxious for some reason. Anyway, they want their data moved out of our Moscow data center. This would be a massive undertaking, as there are hundreds of "tenants" to deal with, all with differing configurations. Are any of you also working at companies who also have to deal with a similar scenario? How are you handling it?

https://redd.it/tniy2n
@r_devops

What is a good way to manage 100-300+ microservices on multiple environments?

I am trying to find a good way to manage 100+ or even 300+ microservices on-prem k8s clusters that consist of deployments, services, configmaps, secrets, databases, load balancers and other configs and for now some of the good options that I can find are to:

\- Create a Helm Charts and combine multiple chars under one big Umbrella Chart - searching for good articles or experience with this

\- Manage different environments with Kustomize

\- Use Terraform for the deployments - but for some reason this seems to be very complex for management

Any advice, articles, guides or shared knowledge in managing a lot of services would be very appreciated.

Thanks!

https://redd.it/tnkwtd
@r_devops

Podman 4 still unusable on macOS

I'm macOS user which means I'm forced to use docker (we can argue but there's nothing better than docker for running multiple apps locally). I was really excited when podman 3 got released but excitement passed once I tried it. Long story short volumes were less than usable. With the release of podman 4 I got excited again. And again I hit the volumes wall. Is it me doing something wrong or is podman still not usable on macOS?

https://redd.it/tnncir
@r_devops

I got tired with AWS status pages. So I built my own

Here's the link.

It's open source if you want to make any suggestions

https://redd.it/tnoln3
@r_devops

Discussion: Who Should write the ci/cd pipEline (e.g. jenkinsfile)

I had a debate with a development team at my organization recently. It was regarding who should be responsible for writing and maintaining the Jenkinsfile that handled testing and deployment of the application they develop to test and prod environments.

Their stance was, "Ops should be responsible. Deployment is an ops task, ops should be responsible for it or we should hire someone external to our team who would be completely responsible for it."

My stance "The Dev team should be responsible. Most of the complexity in this pipeline is in the portions for deploying to dev environments and automated testing. Your team is and will continue to be the experts in that so moving the responsibility for defining those processes to an external position or team will only cause frustration and delay to your team."

I lost, and I realize I'm biased because I've always worked at smaller orgs where their wasn't a "DevOps" team and everyone wanted as much access to do as many things as we could so we would never have to wait on someone or give up control.

Given the different structures, sizes, processes and backgrounds we all have here I'd love to posit this to the group:

Who do you think should be responsible for defining and maintaining the CI/CD Pipeline? In general what factors do you think are the most important in driving that decision?

https://redd.it/tnwh2k
@r_devops

How to pivot into DevOps?

Hello,

I've been thinking a lot lately about my career, and I think I would like to move into software engineering; I have non-professional experience in the past on low-level video game development, and I have lots of professional experience with PowerShell/automating things.

I feel like it might be easier for me to make this transition by first pivoting into a DevOps role, as I would have more relevant experience to transition with. But I think most DevOps positions are focused on infrastructure

My work experience is isolated to Windows environments, and while I have several years experience of managing desktops (mostly) and servers, the last few years has been focused around Azure AD & Identity Management.

I'm currently spending some time working on some personal projects to build a portfolio and learn things on the coding side, but I'm wondering how much I'll need to spend learning infrastructure operations? Are there DevOps positions around Azure AD/IAM security?

https://redd.it/tnzhey
@r_devops

How do you test your Terraform+Python Code?

Hi y’all i’m trying to learn about devops and was curious how do you go about testing your terraform/python code before actually putting it up in production? Is it by unit testing or deploying the infrastructure locally?

https://redd.it/toe5i5
@r_devops

How do you send secrets to other people

As a devops engineer we also are in charge of creating various user accounts for people in the org. What is the best and most secure method to to send the passwords and other secrets to people? We have tried slacking them the username and email the password with encryption but that seems kind a hack and not sure how secure that is.

https://redd.it/tofkfs
@r_devops

How do you send secrets to other people

As a devops engineer we also are in charge of creating various user accounts for people in the org. What is the best and most secure method to to send the passwords and other secrets to people? We have tried slacking them the username and email the password with encryption but that seems kind a hack and not sure how secure that is.

https://redd.it/tofkfs
@r_devops

AWS SES is not sending emails to one particular domain, what might be the issue and how to solve it?

AWS SES sending mails to all domains except one which has an - (hyphen) in the domain name.

experiencing a strange error.

[UPDATE\]: the mail is going to only one email ID ([email protected]) other then that id its not going to any other email id on that domain.

https://redd.it/toh2ma
@r_devops

Creating a Grafana dashboard for a PoC

Hi, I'm not sure how easy or hard this topic. I recently moved from Monitoring dashboards and analysing kibana logs for day to day tockets, to a DevOps postion and altough I'm very eager to learn, I'm very much lost at the very first task at the work and feel ashamed to ask them very basic stuff and I don't know where to start.
The first task I've been given is to monitor the avalibilty of the poc, I'm suppose to create sample grafana dashboard for a white lable proof of concept that I cloned company's repo.

In the readme file on the cloned repository it's mentioned to create .env file on a desired package to run it but I don't understand how that works since it's my first time doing such a task.

Please advise me how to run the project and where to get data source so I can create sample dashboard for it.

https://redd.it/tot028
@r_devops

Typical tasks you do at your job?

I’m looking to move into devops in the future. I’m curious what are some typical tasks you do during the week that are related to more of the development side of devops?


I’m assuming it’s mostly scripting of some sort, but what types of scripts do y’all write?

https://redd.it/tp2da1
@r_devops