Considering switch to Blue-Green deploy model, how to handle DB sync?

Hi all,

First of all - I'm not a devops, but a QA Lead, and one of my long term quality goals includes switching my company's deployment model to blue/green (now it's essentially roll out and pray for the best). However, my issue is that each production environment includes a orders database (we're an online retailer), and architects are concerned about discrepancies with orders stored in DB in case we rollback or switch productions. So in a nutshell - I'm looking for a solution, that would keep both blue and green DBs synced with actual information and prevent losing orders, duplication, etc.

My thought is - perhaps some sort of a broker between the two DBs, that would only store and cast this essential info?

Would greatly appreciate any advice.

https://redd.it/1c4g71a
@r_devops

What is OpenID Connect Authentication? A Practical Guide

Hello, devops community,
Today, I present to you a topic that is less discussed and often taken for granted in our daily jobs.
OpenID Connect is among our industry's most widely used and least discussed topics.
Yet, it is so crucial when it comes to granting third-party access to a service provider. Have you seen those "sign-in with Google" before!?
In this guide, I will explain the notion of OIDC using a practical, real-world example: granting GitHub Actions access to an AWS account.
Feel free to ask any questions that come up.
https://developer-friendly.blog/2024/04/14/what-is-openid-connect-authentication-a-practical-guide/


\#oauth2 #oidc #github #aws

https://redd.it/1c4fbhh
@r_devops

Deny rules in RBAC

Hi Guys,
Is it possible to deny a rule on all the RBAC roles?
I want to deny or exclude -
Microsoft.Storage/storageAccounts/generateUserDelegationKey.
By default Contributor, Storage Account Contributor and few others have them and I don't want these roles have this permission.
Is it possible in azure to deny? Kindly let me know, I have tried to check on the internet but not able to find much.

https://redd.it/1c4jfvk
@r_devops

Need advice on giving developers more information on their e2e runs

We're running e2e test suites using Argo Workflows. The flow is basically:

1. Setup the environment
2. Run tests
3. Upload results to our TestOps platform (RPortal)
4. Send Slack message to the team with links to Grafana, RPortal, etc.

Our e2e tests are running on spot nodes which are sometimes relcaimed during the run (they take about 2-3 hours). If the node was reclaimed during the test then there is no output from step 2, which means once the workflow reaches the part where it uploads the results to RPortal, the result is empty and the developers are not sure what happened.

I'm looking for advice on how to improve this flow. We do send Slack messages on failed runs, but they show as failed whether the node was reclaimed or the tests simply failed. In my case, the devs kind of rely on the results in Report Portal, which show as empty. I was thinking I might handle "empty" results by uploading a simple text file as the test results that would state that something went wrong during the workflow which is not related to the tests suite.

Any suggestions would be great :)

https://redd.it/1c4kdjj
@r_devops

DevOps Professionals: Are You Limited to Roles Specifically Labeled 'DevOps' or Exploring Other Titles Like 'Cloud Engineer'?

I have been navigating the job market and while I currently have a role in devops I have also looked at roles that have similar requirements to devops roles that are listed as cloud engineer or etc..

I am curious if anyone else has taken roles that are really devops but not titled as such.

https://redd.it/1c4lcy3
@r_devops

Forgotten Origins of LXC/LXD

I'm not sure if this is the correct place compared to /Kubernetes or /Docker. Maybe I'm also off base on this... Just wanted to ask about/share some thoughts/rant. I don't even have a problem just talking to the void.

Is LXC/LXD not the origin of both Kubernetes and Docker? Docker originally being released essentially as a wrapper of LXC. And LXD being a container orchestrator supporting standard APIs, networking, cluster support, etc. I'm generalizing a bit, but the concepts behind LXC and LXD were well ahead of later adaptations of similar projects. These are essentially primitives of our current abstractions, yet I never hear/read anything about LXC/LXD anymore? There is no drawing correlations or explaining the alternatives in recent blog posts, guides, etc.

Why is it? Both projects are still alive and supported, seemingly able to solve a lot of the same problems. Is it just not preferred anymore due to more features or usability of newer products (I assume)? Is it performance and I'm stupid to run containers in LXD cluster? There is a certain level of "pureness" that I like.

One last thing - is anybody using Incus or seeing more popularity? Actually reason for this post is due to seeing this project seems to be lead by previous LXD maintainers. Except it seems not tied to Canonical this time. I'm interested but don't see much traction yet.

https://redd.it/1c4kp5o
@r_devops

Question - How do you guys do Patching?

Question on how you all do Patching - both Windows and Linux? is it integrated in any way into your build process or do you use a dedicated patching software? If you use a dedicated software, what do you use?

Any thoughts, regrets, lessons learned that can be shared are helpful.

We still use WSUS with the old Solarwinds add on, but we want to move that into something modern and more effective. Ours is "ok" but needs improvement in all areas - Installation, Compliance, Reporting.

https://redd.it/1c4mayd
@r_devops

How do you define your SLA?

I'm trying to brush up on my basic SRE chops and was reading ye olde Google posts on calculating SLOs based on past performance, and I know that SLA's are supposed to just be an agreement to meet that SLO, but is this really how it works in your organization?

Back in the day the answer often boiled down to 'our biggest enterprise customer forced us to guarantee this SLA,' and since so many other decisions like the cadence of monitoring are based on your SLA, how does your team define the SLA you're trying to deliver?

https://redd.it/1c4j7da
@r_devops

Any ex-TAM(technical account manager) having/had issues to return into the hands on IT market?

The title is self explanatory.

https://redd.it/1c4rwyg
@r_devops

I’m conflicted

Last year I’d taken over the CI/CD for the entire UI team wrangling their massive web monorepo, porting iOS to Az and building everything from the command line, as well as Android. I was handling building and detecting all of the apps on the web, containerizing them, and generating reports for them. Backend was a different story as they do something completely different.

So everything was nice and consolidated into AZDO.

Well, the UI director recently decided that they were taking over CI and I would handle delivery of the artifacts. Now they’re using GitHub Actions.

Now I feel like we’re digging deeper into silos.

I often say that DevOps should be a culture not a title. And part of me is a little relieved not to have to worry about it, but also it’s just going to be further fragmented and harder to maintain.

Do y’all think this is a good change or stepping backwards? Does DevOps handle the entire lifecycle or just the logistics of getting packages from A to B?

https://redd.it/1c4swmo
@r_devops

About Fullstack Web App Deployment.

Hi team,

I have a frontend (react) , backend (spring) and mysql database, how can I deploy. Can I deploy all in any one compute engine like digital ocaen droplets or deploy the frontend in Netlify and backend and db together on Compute Engine. Which is the best approach, if I use netlify how it balances the load, where as in droplet containing all in one VM I use nginx as load balancer.
Kindly le me know which one should I go for. Which optimized and the best way to do deployment. Thanks

https://redd.it/1c4v1eb
@r_devops

Keeping Old Deployments

I don't have too much devOps experience, but I'm wondering what others do when it comes to old deployments on production, staging, and dev servers.


Is there any value in keeping all old deployments on any of these servers?
Is there any harm in deleting them?
What is best practice for "cleaning up" deployments on servers?


I'm starting to run out of hard drive space and deciding whether to cull come deployments or just add more gigs to the servers.
Thanks in advance for any thoughts.

https://redd.it/1c4vo4t
@r_devops

"You Build It, You Run It" is it a lie?

Yo, recently, we had a podcast guest on the show who talked about the evolution of DevOps, and he had a hot take of why the classic "you build it, you run it" approach is a lie and equated it to asking a mechanical engineer who builds a plane to also fly it.

Usually, the "You Build it, You Run it" is a traditional approach of separating development and operations. It has been challenged by the need for developers to take ownership of their code in production. This shift requires devs to not only build the software but also be responsible for its smooth operation and maintenance.
Devs, what are your thoughts? Do we hate it? If you're not a fan of 'you build it, you run it,' what's a better way to go about it? Or maybe that's full of crap, and you truly can both build it + run it.


You can listen to full episode here for context: https://soundcloud.com/ambassador-livinontheedge/s3-ep5-the-evolution-of-devops-ai-why-you-build-it-you-run-it-is-a-lie

https://redd.it/1c4xv6f
@r_devops

Minikube

Hi, so does minikube itself maintain any directory which has all the list of containers that ran on it?

I was just digging around on some of the directories, found out that while executing this command : docker exec -it minikube ls -1 /var/lib/docker/containers

It provided me with a subsequent list of directories which had information about the containers.

Can anyone provide more insights on this?

https://redd.it/1c4tp24
@r_devops

Why not shift to Nomad instead of K8s?

I've been trying to deploy my infrastructure both on K8s and Nomad (+ Consul).

The only benefit I see, that covers up for greater complexity of K8s, is the availability of managed service providers (GKE/EKS). Which makes things easier.

But Nomad seems to fit my use-case of 8-10 services deployed behind a proxy and load balancer, and is so much easier to setup and use.

I've seen a lot of people agreeing that Nomad is more straight-forward, at least for developers who don't want to invest too much time in infrastructure. But there is still hesitance in wider adoption of Nomad.

Is it largely the unavailability of a managed Nomad (or HashiStack) provider who can run the scripts on your behalf, or some other reason?

I want to understand the primary justifications behind why you as current K8 users will not adopt Nomad.

Thanks ❤️

https://redd.it/1c57oot
@r_devops

Theoretical underpinnings of IaC

Tools like Pulumi / Terraform have a very complex state management IMO and diffing algorithms as well as planning to ensure the desired state is met with as little obtrusive changes as possible. Are there any papers that discuss how is this done? Asking for a friend!

https://redd.it/1c5a2rh
@r_devops

Career/Resume Advice Graduated 3 years ago and can't find a job

My resume

---

So I graduated nearly 3 years ago and had to do mandatory military service straight after, which was a total waste of time. Despite listing it as experience on my resume, I was definitely not doing those things everyday and due to no internet and limited infrastructure, I wasn't able to learn new skills and improve myself either.

Anyways, once I finished service, my mom got diagnosed with cancer literally the following week. I was depressed about it and I was staying at home helping her out. She was weak, couldn't use her left arm, and occasionally would just fall due to dizziness from the therapy and whatnot. However, she got better eventually and she's doing fine now. So I feel like I can't use "caring for elderly parent" as an excuse for my huge employment gap because she got better after like 6 months.

Ever since then I've just been applying and trying to learn in-demand skills. However, I never get a call back; rather, it's always the default "we've decided to continue with another candidate" email. Getting a job in my country (Egypt) relies on nepotism for the most part, but no one in my family really knows any people working in tech. In the space of a year, I only got 2 offers, both through nepotism: one was for some Java developer role offering $60/month and an ERP consultant role for $200/month. Given these salaries were essentially unlivable wages and literally had nothing to do with the skills I know or related to the DevOps/Cloud career I'm trying to pursue, I declined those offers. I've been trying to learn new skills, but it seems to not amount to much as employers prioritize on-the-job experience with those skills, and not at-home projects.

So now I'm stuck with essentially a 3 year employment gap, unable to find any job. I'm sorry if this feels like a vent post, I'm just totally lost right now and feel like a burden to my parents living at home while they feed me on their retired pensions. I don't know what to do anymore.

https://redd.it/1c59p3p
@r_devops

Free Azure Devops

I need to teach myself Azure Devops and as part of that I tried https://azure.microsoft.com/en-us/products/devops/ (Start Free). My environment was setup in minutes. Is this totally free? I didnt enter any billing information. Just wanted to make sure because I had a bad experience with AWS.

https://redd.it/1c59uz7
@r_devops

What's the most significant pain you've faced when automating your deployments?

It's driven me nuts how much documentation I've had to read for minimum permission deployments on the three cloud providers within the command line.

A few weeks back, I finished 5 part-time days of constant back and forth between GCP (which I consider more friendly than AWS) to deploy a dockerized application to Cloud Run.

Note that I'm more of a developer who wants to "just run" the application without paying enormous fees to wrapper services like Vercel.

What's been the most considerable pain you've faced over the years?

​

https://redd.it/1c5fzxq
@r_devops

Transitioning to DevOps from Data/AI

Hi guys,

I am a mid-level engineer with about 6 years of experience in building Data and ML applications. In the early years of my career I spent a lot of time on the maths/modelling side (C++/Python) and later on found the systems and engineering (Cloud & devOps) side of building applications more enjoyable.

I haven’t had a formal training in computer science (my bachelors was in Robotics and Master’s in Applied Maths), so I’ve learnt most of what I know on the job. I’ve worked on the devOps side of things - CI/CD, Iac (Terraform), Azure, AKS- for the last couple of years in my roles as a ML/Data engineer.

Late last year I had to resign from this Data related role due to relocation & other reasons and I’m finding it quite challenging (unlike previous times in my career) to land a new role, preferably in DevOps. Having applied to many roles in the last couple of months, I am questioning why I am not even getting interviews, and would like some advice on what I should do to position myself better.

What I enjoyed the most about DevOps work was the satisfaction of helping all others devs work efficiently, and being able to work on system design to some extent.

I understand that DevOps is a somewhat senior role, for those who’ve had experience in other SW roles before, so I feel that I’ve been in the tech industry long enough. Then again, from the rejections I faced Im wondering are companies interested in folks who’ve had 5+ years pure of DevOps experience on a particular tech stack? Or do I apply to really Junior positions?

Thank you!

https://redd.it/1c5hdg7
@r_devops

Recommendation for running a few containers in a small environment

What is the best way to run a few internal (but important) containers in an environment with low/no DevOps experience?

We have a branch office with no DevOps people around. They want to run a few services which are delivered as containers. They have a few IT staff who take care of SaaS apps and on-prem infrastructure.

I'm wondering if a machine running Docker + some startup scripts might be the simplest way to go... Is this such a bad idea?

Kubernetes seems like it would be impossible for them to maintain. Even a managed Kubernetes service like GKE requires "someone" at the helm to operate it.

Is there a happy medium for this kind of situation, or is the general consensus "containers -> welp you better hire a team to run them!"

https://redd.it/1c5fw2q
@r_devops