Safe terraform apply in CI

I know that some shops do terraform apply as part of their CI.


How do ensure that your CI doesn't do something horrific in production? In terraform plan we trust?

Do you use GitHub Actions? What has been your experience?

https://redd.it/1c7ismb
@r_devops

Blue Green and CD on EKS

Hey guys, so at my current company - we are having some tension between product and qa/platform. Currently we are doing manual code freezes so qa can thoroughly test a bunch of things before we push to prod. This is causing tension as product engineering wants things out ASAP. Problem being is we’re getting tons of bugs due to this. We are severely lacking the CD portion of our pipeline, and one of my plans (in agreement with the VP of infra) is to get B/G deploys rolled out using something like Argo rollouts (paired with ArgoCD). This way we can push to “prod” and have QA run as many tests as possible without causing a code freeze. We use automated tests like cypress, but another step will be automating more manual QA processes down the line.

In the meantime is this a good idea, would love to hear pushback. Currently we only have a dev env, and while yes we can do a staging - my exp in past is that things can pass staging but break in prod due to minor discrepancies.

Would love to hear what you guys have been doing, especially revolving b/g and CD. We are GitHub actions shop and deploy with helm.

Also given I have a chance to really architect this however I’d like, let me know how you would do deployments if you had no limitations. My last job we did GitHub actions and argocd as well, but did not have any B/G as there were not frequent app breaking changes.

(Note I am new to this company so I did not set any of the current processes up).

https://redd.it/1c7nnhb
@r_devops

Seeking Advice from Respected Experienced DevOps Engineers.

I’m a level 1 seller on Fiverr, offering Linux system and server administration services. To be honest, I haven't completed any course fully yet; I've only done about 30 to 50 percent of each course I started, mainly due to procrastination. However, since I'm good at problem-solving and using Google searches, I accept orders from buyers for projects that I have never worked on and have no idea how to handle.

I’m very interested in DevOps, as I believe that to apply for a job in this field, you must have skills in CI/CD, IaC, etc., and I’ve noticed many Linux sysadmins are moving into DevOps. I've realized I need to learn things properly. I’m currently about 40% through the RHCSA and my goal is to complete it without leaving it incomplete. After that, I'm considering pursuing Docker and Kubernetes as they are critical for DevOps roles. I believe good Linux skills are a prerequisite for DevOps.

I'm also practicing networking fundamentals on EVE-NG. I would like to know if I’m on the right path now. Hearing about your success story, the path you followed, and any recommended courses would be much appreciated. Thank you so much!"

This revised message reflects your updated goals and emphasizes your commitment to completing the RHCSA before moving on to Docker and Kubernetes. It’s great to see that you're setting specific objectives to guide your learning path in DevOps.



https://redd.it/1c7qrsi
@r_devops

Running Kong on GKE

Hi, new to Kong, love the concepts. Experimented with Gateway APIs on KIND, worked great. Now need to move it to GKE. There are several things I would like to do that I'm not sure how:

1. How to make the kong-proxy service to use the reserved static ip in GKE? Before I would use the kubernetes.io/ingress.regional-static-ip-name annotation on Ingress, but not sure how to do that with Gateway APIs

2. What is the right way of running multiple replicas of kong? I saw there is replicaCount value in the chart, but there is also a cluster.enabled mode. Should it be enabled as well?

3. Related to 2, how to run those replicas so that they are spread across zones & nodes?

4. I'm installing using kong chart from https://charts.konghq.com and with pretty lean values below. What are other fields you would suggest to specify for a typical production use.

ingressController:
  enabled: true

5. Want to apply CRDs separately. What would be the best place to find them?

Thank you!

https://redd.it/1c7qoin
@r_devops

How can we update docker containers on edge device with no Internet or should we simply get rid of docker?

We need to deploy docker container on edge devices which wont be having Internet. These devices occasionally connect to a network and one of the device (lets call it H) on the network will have internet access. So, I want to know how we can update docker containers in such scenario. I imagine following two approaches:

1. Create tar of image. Copy it to edge device (say over USB) and then update the image on the edge device.
2. Create local registry on device H. Pull the updated image from remote registry to local registry on device H. Make edge device pull only updated layers from this local registry on H.

I feel option 2 is good given it only moves updated layers between devices making update size small, while tar contains all layers resulting in tar of size 300 MB. So, option 2 was good option till we thought device H will be x86 Windows device. But now we are told that it can be Android or iPad companion device for edge device. We cannot run docker registry on Android or iPad right? So what solution we have remained with for updating docker container on edge device?

We did various docker related POCs. But now after knowing the fact that device H can be android or iPad device, we may have to get rid of docker completely and deploy apps say through other non container approaches say snap etc.

Should we let edge device access Internet through device H say through tethering? We don't want Internet on edge device for security reason, but then should we restrict the Internet access on edge to only servers hosting docker registry? Or there can be any better solution without requiring Internet on edge device at all?

https://redd.it/1c7sb25
@r_devops

Salary Expectation for 8 yoe in India

Hi Everyone,

May I ask what should be the expected CTC for a Devops Engineer in Bengaluru (India).

Thanks



https://redd.it/1c7s4e0
@r_devops

Jenkins exodus? Is it really a thing?

I recently attended Kubecon CloudNative con, and I can't remember who, but someone mentioned a "Jenkins exodus" occurring in the industry.

Where I work we heavily use Jenkins, with Spinnaker sprinkled on top for cross CI communication and pipeline orchestration mainly.

Seeing all the "cool" tools out there, opensource or not, makes me think we need to evolve for the future. For example, Argo workflows, Tekton, Harness, Octopus Deploy, Gitlab, Azure DevOps, Github actions etc.

We have a mix of legacy and cloud native microservice products so any migration/change would be taken slowly.

I'm interested in what people are using but more importantly what their experience is like.

In the Jetbrains State of Developer Ecosystem report 2023 (https://blog.jetbrains.com/teamcity/2023/07/best-ci-tools/#the-best-ci-cd-tools-in-devops-according-to-our-state-of-developer-ecosystem-report), they polled over 26k developers worldwide and Jenkins, Github Actions and Gitlab CI were top of the pile when it comes to which tools are regularly used, but it doesn't give any context around whether the developers are happy with them!

Be great to hear what you're using, and what your experience is like! TIA

​

https://redd.it/1c7tvg4
@r_devops

Lots of work opportunities I see lately are looking for full-stack experience, including with application layer

I started my career as a developer and for various reasons, it wasn't for me. I found AWS, cloud infrastructure, CI/CD etc. came to me more naturally and when I finally made the transition, I was happy to be able to focus on that and let the developers take care of the application. At my first company there were a few people who were completely full-stack, but it was normal that platform engineers managed the infrastructure while the developers managed the application. I know that may raise questions about silos but that's another can of worms.

Lately I've been looking for a new job and my company is also looking for a project for me (we're a consultancy). One opportunity that got sent to me today asks for "a background in Enterprise Application Development" and "Proficient understanding of the entire technology stack of the service, from application to infrastructure". A few months ago I inquired to a friend about any work going at his company and he said that me not knowing much about application code might be a problem, as that's just not how they did it there.

The fact is that I really don't know much about application code and there's not much I feel I can offer for it. I get the impression that most of the folks here don't get involved with the application much either. Do you also see this in job advertisements like this lately? Is it ok to just want to keep the focus on infrastructure or is it limiting? I'm in the Netherlands, not sure how different it is between countries.

https://redd.it/1c7umuq
@r_devops

Just feeling a bit defeated.

Managed to make it to the last interview for the 4th time to see this again: “we decided to go with a candidate with more experience.”

I’m closing in on 6 years of experience. Feels like I’m constantly competing against people with 10+ years nowadays. This wasn’t even a senior role. I know there’s been a tech exodus, but damn.

Anyone else in the same boat?

https://redd.it/1c7ur1a
@r_devops

People who have personally migrated from Istio to Cilium, what was your experience like?

We’re pretty heavily committed to Istio, but I can’t deny that.. well almost any of the service mesh solutions are less complex and easier to maintain.

Those who have made the jump to Cilium specifically, what has your experience been and would you do it again? Thanks!

https://redd.it/1c7wxzd
@r_devops

decided for Flux against Argo now folks want a UI ...

well the title says it. I've worked with both Flux and Argo in the past and I decided for Flux. Now people are asking me for a UI. FML.

I've stumbled upon this: https://flux-subsystem-argo.github.io/website/

has any of you guys used this? It seem well maintained and even has Flux' Daniel Holbach on board.

https://redd.it/1c7xvmc
@r_devops

How do you deal with developers not paying attention to slack

Frustrated. I feel like I'm the only one paying attention to things like alerts, PR requests, pings, or even just conversation. Hours go by in the workday and people aren't responding to my pings or PRs. Builds are broken. I'm the only one reviewing PRs. I know it's a culture thing and an expectation that should be set by management but I don't really have any control or influence there.

Edit: just want to clarify my expectations. We all have work to do and are busy with tasks. I get it. I think it's pretty reasonable to watch for broken builds after a push. Or do respond to messages in a team slack channel within half a day. Or to acknowledge someone asked for you to look at a PR within a few hours (not saying you need to look at it then). If an alert in production fires yes IMO that justifies stopping your current task to investigate. If it's bad alert then fix it or turn it off.

Slack is literally the only way we can communicate with each other for the majority of us WFH. We need to acknowledge each other.

https://redd.it/1c7zbsu
@r_devops

Do most companies stick to the big 3 (AWS, Azure, GCP)? How come I never see Digital Ocean or Linode come up?

There are many online tutorials that teach people how to get set up on Digital Ocean/Linode but I noticed that I've never seen people discuss production issues on them. Most people only talk about the big 3. Even job listings only mention the big 3 and never about Digital Ocean.

Do big companies or even small tech companies not consider Digital Ocean and others? Are those renegated to hobby projects or small time customers?

https://redd.it/1c814pa
@r_devops

DevOps & Cloud Architecture Books

Hello DevOps folks,

I kinda find myself to struggle in understanding the big picture when taking on work tasks in my current role (Cloud Engineer - Azure). I feel like that I'm working only on small gears of a huge machine.

My company does consulting for big/medium businesses and I can't really understand how changing some IaC or YAML code provides value to the big picture or environments built with them. I mean, I kinda do, but i struggle to visualise the interconnection of the whole environment.

Based on this, can you recommend 1/2 books that would provide me the "big picture" of DevOps or Cloud Architecture solutions, especially in a consulting point of view? I want so bad to understand from 0 to 100 how to make up a digital DevOps/Cloud solution based on real life scenario but i miss the "knowledge components" (maybe something that is acquired after years of experience?).

Thanks in advance!

https://redd.it/1c86r1z
@r_devops

GitOps is matured. Right ?

Hi all,

I was talking to our company DevOps Director and was asking questions on why we don't use GitOps and instead use typical Azure Build and Deploy pipeline.

His answer was GitOps isn't matured and isn't good enough for Standardizing across organization ? (I wasn't given proper reason and don't plan on asking)

I personally don't feel that way. Is there any reason why a team wouldn't opt GitOps and keep it the old way ?

Also, if guys have some good learning material to understand these, it would be helpful.

https://redd.it/1c88thx
@r_devops

Artifactory: What am I missing here?

I’ve been using Artifactory for a few years at multiple different companies and on different projects and it never ceases to amaze me how useless it is for any non-trivial use case. Pretty much all my teams have used it for are pushes and pulls; I can see that there are other slightly more advanced features, but none that any project I’ve worked on has been willing to pay for. Is there something happening here that makes it more practical than a RAID Linux server hosting a shared filesystem? (Besides maybe the server admin side)

Not saying it’s universally useless, but for this particular type of use it doesn’t seem like the right tool.

https://redd.it/1c8af1q
@r_devops

How to deploy a multi-service application

I have a backend that looks like this

https://excalidraw.com/#json=HF2\_DBOgoti6C6tbVG\_6k,ZXerWMQn1sKcnySy3hUWFQ

I think a lot of people have similar architecture, but I have not seen a shared common solution online for such architecture

locally I use docker-compose, but from what I understand it's not recommended for prodcution(not sure of this)


How to deploy this app architecture to production(without kubernetes)?

What tools people use?



https://redd.it/1c8eglj
@r_devops

Tools to create diagrams/graphs with detail view function

I'm looking for a tool to create a network graph with a "detail view" function.
E.g. something like Python diagrams or Mermaid that allows you to define a diagram/graph in code (with custom SVG icons) but with the ability to output multiple SVGs (because of scalability) and a display tool with zoom in function which allows you to get a more detailed view of parts of the diagram/graph that can be embedded, e. g. in a Sphinx documentation.

The idea is to visualize an entire network in one place with details only being loaded in on demand if that makes sense.
An example of what I imagine: Starting out you see a cluster overview, with just load balancing, frontend, API, DB and GlusterFS data clusters. When you click on the data cluster you see the bricks. When you click on the bricks you see the individual servers. And when you click on the server you see its hardware details. And of course there needs to be a "back" button that zooms out again.

Does something like that exist?

https://redd.it/1c8m8b0
@r_devops

Senior Dev to DevOps transition

Wanted to ask what skills I should read up on before switching to DevOps. My current and prior companies has had massive problems finding DevOps people and I know the wages are higher. So been thinking about changing teams.

I think it's mainly imposter syndrom holding me back. I have 15 years of experience in software development. I have worked in both Azure and AWS cloud. Had hobby projects hosted in both.

I am currently hosting my own K8's cluster on Hetzner ARM instances for my private projects. Running postgres, REDIS and different WebApps / apis aswell.

So I would say I have done a wide range of DevOps tasks in prior jobs setting up and maintaining build pipelines in Azure and GitHub.

What I mainly lack is "real" production hosting of databases and backup strategies, since it's either been handled by others or not caring losing data in hobby projects.

I am Abit lost what to read up on before applying for team change. Any good advice?

https://redd.it/1c8of46
@r_devops

Gitlab vs GibHub vs Bamboo+Bitbucket

Okay guys, I need your group wisdom. And knowledge.

I have a group of developers I need to support. Some of them are currently using GitLab and others Bamboo/Bitbucket. They, of course, prefer their tool. Still others are lobbying for GitHub. Due to company policies, we can't use AI, and whichever we choose will be on prem.

We also have a requirement to use GitLab in one of our environments, so my thought is any not standardize on GitLab? Then we only have one too to support and one type of yaml to maintain. We also have a sister company using GitLab that could be useful as consultants for GitLab. The people championing the other two are quite loud though. Also, the quotes have come in where the Atlassian tools and GitHub are much cheaper and about the same price... But GitLab is much more expensive.

I really think it should be GitLab, then if we don't get that, then GitHub, with Atlassian tools being a distant third choice.

We do use Jira and Confluence, but I don't think the integration with those is worth it. Also, we aren't sure if we want Ultimate or Premium GitLab. Also not sure if the security scanning in Ultimate is a decider.

I need a "smoking gun" to prove my recommendations are correct, but I haven't a concrete reason for this, other than above. This is to convince the high up leadership hopefully to go GitLab and make it worth it for price.

https://redd.it/1c8tc1b
@r_devops

Overqualified and Underpaid

Last year, after 6 months of interviewing, I accepted the only job offer I received. It wasn’t a Senior-level position and I took a 35% pay cut. The expectation was that I would come in, show my worth, and be promoted to a Senior title and a decent pay raise.

After completing my onboarding in 2 months instead of 3, I’ve been given the responsibilities of a Senior DevOps Engineer and excelled. It’s been almost a year and I have exceeded everyone’s expectations.

Despite everything, management continues to drag their feet on promoting me and I’ve lost confidence that they will until maybe early next year.

I recently started looking for a new job with a Senior title and pay. The number of job postings seem slightly better than last year, but salaries are lower.

I’m curious if others are in the same boat and wondering how they are approaching their job search.

https://redd.it/1c8v0vx
@r_devops