How do I fix these deployment/git workflow issues?

We have a massive and very old .NET project at work. It's codebase is a mess, there's no test suite, it's running on very old tech, and we're startup so don't have the capacity to really change any of this. But, it's our most important service.

Some quick info on our infra, we have `prod`, `uat` and `dev` environments. QA testing is done manually on `uat`.

Our git branches are setup as so:

* `master` branch -> `prod`
* `dev` branch -> `uat`
* PR against `dev` branch -> `dev`

So when we're working on a feature branch, we can deploy to `dev` via a PR, once checks have passed and it's been approved it's merged into the `dev` branch and deployed to `uat`.

**First issue**: this deployment strategy means that if I'm working on feature 1 and deploy to dev, someone else can then work on feature 2 and deploy to `dev`, overriding my feature. And when deployments are 40 minutes, this can be a bit of a blocker (and it means we can't QA test on `dev`).

**Second issue**: if feature 1 and 2 pass code review and goes out to `uat` for QA testing. But feature 2 takes longer to test, or a bug is found, feature 1 is now blocked from going to `prod`. Our temp fix for this is to cherry pick feature 1 onto a "release" branch and merge into `master` (`prod` deployment) but this is causing merge conflicts and is a bit messy

I'm pretty new to DevOps and have been asked to solve these issues. Been reading up on GitFlow and trunk based solutions but can't seem to get around our blockers

https://redd.it/1c71mpm
@r_devops

Best way to store nested json data

What's a better way to store nested jaon data

Ive been using azure tables recently, they fit my needs of storing and recalling data when I need it in powershell/python/js.

I have a system that manages renewals for a suite of products, when a renewal is coming up, I store the renewal data as a Json in a single cell.

Later the rowkey is used as a query parameter and the json is retrieved and rendered.


Over simplification but that's the idea.

I know I could flatten the json and store the data flat, but that is so much manipulation and translation that it doesn't t seem worth it.

Without flattening it, I can't directly query it in the table.

Should I be using a no-sql database to store Json always?

I do not know any no-sql, but I'm willing to start learning if that's really the best path.



https://redd.it/1c72n45
@r_devops

Understanding AWS End of Service Life Is a Key FinOps Responsibility by Marry Henry

Mary Henry from Fairwinds wrote an article about AWS increasing their pricing on EKS extended support, meaning, running out of support Kubernetes versions on EKS.

Never had that on AWS myself, but we ran into it once or twice on Azure, simply because we had no time to upgrade 🫣

AWS wants $0.60 instead of $0.10 per hour in this case. I can't remember we paid anything extra on Azure. That is insane to increase to 6x the price 🤯

https://www.fairwinds.com/blog/understanding-aws-end-of-service-life-is-a-key-finops-responsibility

https://redd.it/1c72l1y
@r_devops

How Jersey Mike's Rebuilt their Infrastructure during COVID

During the COVID pandemic, Jersey Mike’s ran an advertising campaign offering 50% off all sandwiches.

The ad went viral, jumping from an average of 800 orders a day to 80,000+ orders within the first four hours.

Jersey Mike's rebuild their infrastructure to meet high user demand, optimize cloud spend, and reduce the operation workload for their team.

https://www.fullstackexpress.io/p/how-jersey-mikes-rebuilt-infrastructure-covid

https://redd.it/1c723dz
@r_devops

Prometheus instance with data

Hey everyone

Does anybody know a good option for creating a Prometheus instance loaded with sample data for testing?

Thanks!

https://redd.it/1c76hrb
@r_devops

Have we reached a stage of AI/ML models for DevOps related tasks?

Like as simple as providing a solution for certain use cases like provisioning storage capacity as needed or open up n/w ports for a new service that gets spinned up?


I want to see if there are any development out there around data science to deal with day-to-day tasks or even provisioning infrastructure when a use case is provided.


Cheers!

https://redd.it/1c789lc
@r_devops

How is Your Company's Use of AWS Evolving?

Are your companies increasing their use of AWS services, maintaining their current level of usage, or are there instances where projects are being moved back on-premises?

I'm interested in understanding the reasons behind these decisions as well. Whether it's due to cost, security concerns, performance issues, or any other factors....

If you're comfortable sharing, I'd appreciate if you could also mention the industry your company operates in and the scale of your AWS usage. :-)

https://redd.it/1c75bpv
@r_devops

Service Mesh Preferences- Istio, Linkerd, or Something else?

Ok so we're not a service mesh, nor do we have a stake in either of these products so don't come for us in the comments (though- if you want to talk Gateways, hit us up).

We recently received a guest blog that covered a full guide on traffic management in Istio: https://www.getambassador.io/blog/istio-kubernetes-traffic-management-guide

Now, we've played well with both Istio and Linkerd from an integration + open source perspective, but we're curious...do y'all have one you prefer? Or maybe a third service mesh option you prefer more? Any downsides to either that you've seen?

Give us the best and worst you've had from using either of these. :)

https://redd.it/1c7a2t6
@r_devops

Resume/CV Review

https://imgur.com/a/fdFWyDk

[Reposting because wrong link\]

Hello all, this is my resume (version 8). I have been approached by recruiters and applied to a lot of positions. However no one has come back with an interview for me, really unsure what it is. I am a DevOps Engineer of 3 years really hoping someone can shed some light weather I am trolling with my resume or improvements. Let me know. Many thanks.

https://redd.it/1c7elez
@r_devops

Best way to learn Windows IIS

Hey all! I need to learn Windows IIS and am trying to figure out the best way to do it. Is there a recommended book, tutorial, YouTube series I should be looking at?


Context. I've been hired as a lead software engineer for a brand new department. I need to be able to deploy legacy code (OLD asp) *and* modern django applications on an air gapped system that utilizes Windows Server and Windows IIS.


All my experience is Linux and Mac. I haven't touched any devops stuff in a *really* long time, and back then I did LAMP, mostly small scale and from the developer side.

We are building and supporting internal tools for support staff. We aren't expecting high traffic.

\-------------------------------------

Bonus question/thought. No one around me has any idea of how modern software engineering is supposed to work. The deployment process from the guy I replaced was to write code in production, in the production environment, and hit save (no, I'm not kidding). I'm trying to set up something resembling modern day development practices from scratch.

Is there any benefit to even using Windows Server/IIS? Or is the group I'm working for throwing away money?

https://redd.it/1c7drx5
@r_devops

Safe terraform apply in CI

I know that some shops do terraform apply as part of their CI.


How do ensure that your CI doesn't do something horrific in production? In terraform plan we trust?

Do you use GitHub Actions? What has been your experience?

https://redd.it/1c7ismb
@r_devops

Blue Green and CD on EKS

Hey guys, so at my current company - we are having some tension between product and qa/platform. Currently we are doing manual code freezes so qa can thoroughly test a bunch of things before we push to prod. This is causing tension as product engineering wants things out ASAP. Problem being is we’re getting tons of bugs due to this. We are severely lacking the CD portion of our pipeline, and one of my plans (in agreement with the VP of infra) is to get B/G deploys rolled out using something like Argo rollouts (paired with ArgoCD). This way we can push to “prod” and have QA run as many tests as possible without causing a code freeze. We use automated tests like cypress, but another step will be automating more manual QA processes down the line.

In the meantime is this a good idea, would love to hear pushback. Currently we only have a dev env, and while yes we can do a staging - my exp in past is that things can pass staging but break in prod due to minor discrepancies.

Would love to hear what you guys have been doing, especially revolving b/g and CD. We are GitHub actions shop and deploy with helm.

Also given I have a chance to really architect this however I’d like, let me know how you would do deployments if you had no limitations. My last job we did GitHub actions and argocd as well, but did not have any B/G as there were not frequent app breaking changes.

(Note I am new to this company so I did not set any of the current processes up).

https://redd.it/1c7nnhb
@r_devops

Seeking Advice from Respected Experienced DevOps Engineers.

I’m a level 1 seller on Fiverr, offering Linux system and server administration services. To be honest, I haven't completed any course fully yet; I've only done about 30 to 50 percent of each course I started, mainly due to procrastination. However, since I'm good at problem-solving and using Google searches, I accept orders from buyers for projects that I have never worked on and have no idea how to handle.

I’m very interested in DevOps, as I believe that to apply for a job in this field, you must have skills in CI/CD, IaC, etc., and I’ve noticed many Linux sysadmins are moving into DevOps. I've realized I need to learn things properly. I’m currently about 40% through the RHCSA and my goal is to complete it without leaving it incomplete. After that, I'm considering pursuing Docker and Kubernetes as they are critical for DevOps roles. I believe good Linux skills are a prerequisite for DevOps.

I'm also practicing networking fundamentals on EVE-NG. I would like to know if I’m on the right path now. Hearing about your success story, the path you followed, and any recommended courses would be much appreciated. Thank you so much!"

This revised message reflects your updated goals and emphasizes your commitment to completing the RHCSA before moving on to Docker and Kubernetes. It’s great to see that you're setting specific objectives to guide your learning path in DevOps.



https://redd.it/1c7qrsi
@r_devops

Running Kong on GKE

Hi, new to Kong, love the concepts. Experimented with Gateway APIs on KIND, worked great. Now need to move it to GKE. There are several things I would like to do that I'm not sure how:

1. How to make the kong-proxy service to use the reserved static ip in GKE? Before I would use the kubernetes.io/ingress.regional-static-ip-name annotation on Ingress, but not sure how to do that with Gateway APIs

2. What is the right way of running multiple replicas of kong? I saw there is replicaCount value in the chart, but there is also a cluster.enabled mode. Should it be enabled as well?

3. Related to 2, how to run those replicas so that they are spread across zones & nodes?

4. I'm installing using kong chart from https://charts.konghq.com and with pretty lean values below. What are other fields you would suggest to specify for a typical production use.

ingressController:
  enabled: true

5. Want to apply CRDs separately. What would be the best place to find them?

Thank you!

https://redd.it/1c7qoin
@r_devops

How can we update docker containers on edge device with no Internet or should we simply get rid of docker?

We need to deploy docker container on edge devices which wont be having Internet. These devices occasionally connect to a network and one of the device (lets call it H) on the network will have internet access. So, I want to know how we can update docker containers in such scenario. I imagine following two approaches:

1. Create tar of image. Copy it to edge device (say over USB) and then update the image on the edge device.
2. Create local registry on device H. Pull the updated image from remote registry to local registry on device H. Make edge device pull only updated layers from this local registry on H.

I feel option 2 is good given it only moves updated layers between devices making update size small, while tar contains all layers resulting in tar of size 300 MB. So, option 2 was good option till we thought device H will be x86 Windows device. But now we are told that it can be Android or iPad companion device for edge device. We cannot run docker registry on Android or iPad right? So what solution we have remained with for updating docker container on edge device?

We did various docker related POCs. But now after knowing the fact that device H can be android or iPad device, we may have to get rid of docker completely and deploy apps say through other non container approaches say snap etc.

Should we let edge device access Internet through device H say through tethering? We don't want Internet on edge device for security reason, but then should we restrict the Internet access on edge to only servers hosting docker registry? Or there can be any better solution without requiring Internet on edge device at all?

https://redd.it/1c7sb25
@r_devops

Salary Expectation for 8 yoe in India

Hi Everyone,

May I ask what should be the expected CTC for a Devops Engineer in Bengaluru (India).

Thanks



https://redd.it/1c7s4e0
@r_devops

Jenkins exodus? Is it really a thing?

I recently attended Kubecon CloudNative con, and I can't remember who, but someone mentioned a "Jenkins exodus" occurring in the industry.

Where I work we heavily use Jenkins, with Spinnaker sprinkled on top for cross CI communication and pipeline orchestration mainly.

Seeing all the "cool" tools out there, opensource or not, makes me think we need to evolve for the future. For example, Argo workflows, Tekton, Harness, Octopus Deploy, Gitlab, Azure DevOps, Github actions etc.

We have a mix of legacy and cloud native microservice products so any migration/change would be taken slowly.

I'm interested in what people are using but more importantly what their experience is like.

In the Jetbrains State of Developer Ecosystem report 2023 (https://blog.jetbrains.com/teamcity/2023/07/best-ci-tools/#the-best-ci-cd-tools-in-devops-according-to-our-state-of-developer-ecosystem-report), they polled over 26k developers worldwide and Jenkins, Github Actions and Gitlab CI were top of the pile when it comes to which tools are regularly used, but it doesn't give any context around whether the developers are happy with them!

Be great to hear what you're using, and what your experience is like! TIA

​

https://redd.it/1c7tvg4
@r_devops

Lots of work opportunities I see lately are looking for full-stack experience, including with application layer

I started my career as a developer and for various reasons, it wasn't for me. I found AWS, cloud infrastructure, CI/CD etc. came to me more naturally and when I finally made the transition, I was happy to be able to focus on that and let the developers take care of the application. At my first company there were a few people who were completely full-stack, but it was normal that platform engineers managed the infrastructure while the developers managed the application. I know that may raise questions about silos but that's another can of worms.

Lately I've been looking for a new job and my company is also looking for a project for me (we're a consultancy). One opportunity that got sent to me today asks for "a background in Enterprise Application Development" and "Proficient understanding of the entire technology stack of the service, from application to infrastructure". A few months ago I inquired to a friend about any work going at his company and he said that me not knowing much about application code might be a problem, as that's just not how they did it there.

The fact is that I really don't know much about application code and there's not much I feel I can offer for it. I get the impression that most of the folks here don't get involved with the application much either. Do you also see this in job advertisements like this lately? Is it ok to just want to keep the focus on infrastructure or is it limiting? I'm in the Netherlands, not sure how different it is between countries.

https://redd.it/1c7umuq
@r_devops

Just feeling a bit defeated.

Managed to make it to the last interview for the 4th time to see this again: “we decided to go with a candidate with more experience.”

I’m closing in on 6 years of experience. Feels like I’m constantly competing against people with 10+ years nowadays. This wasn’t even a senior role. I know there’s been a tech exodus, but damn.

Anyone else in the same boat?

https://redd.it/1c7ur1a
@r_devops

People who have personally migrated from Istio to Cilium, what was your experience like?

We’re pretty heavily committed to Istio, but I can’t deny that.. well almost any of the service mesh solutions are less complex and easier to maintain.

Those who have made the jump to Cilium specifically, what has your experience been and would you do it again? Thanks!

https://redd.it/1c7wxzd
@r_devops

decided for Flux against Argo now folks want a UI ...

well the title says it. I've worked with both Flux and Argo in the past and I decided for Flux. Now people are asking me for a UI. FML.

I've stumbled upon this: https://flux-subsystem-argo.github.io/website/

has any of you guys used this? It seem well maintained and even has Flux' Daniel Holbach on board.

https://redd.it/1c7xvmc
@r_devops