How Jersey Mike's Rebuilt their Infrastructure during COVID

During the COVID pandemic, Jersey Mike’s ran an advertising campaign offering 50% off all sandwiches.

The ad went viral, jumping from an average of 800 orders a day to 80,000+ orders within the first four hours.

Jersey Mike's rebuild their infrastructure to meet high user demand, optimize cloud spend, and reduce the operation workload for their team.

https://www.fullstackexpress.io/p/how-jersey-mikes-rebuilt-infrastructure-covid

https://redd.it/1c723dz
@r_devops

Prometheus instance with data

Hey everyone

Does anybody know a good option for creating a Prometheus instance loaded with sample data for testing?

Thanks!

https://redd.it/1c76hrb
@r_devops

Have we reached a stage of AI/ML models for DevOps related tasks?

Like as simple as providing a solution for certain use cases like provisioning storage capacity as needed or open up n/w ports for a new service that gets spinned up?


I want to see if there are any development out there around data science to deal with day-to-day tasks or even provisioning infrastructure when a use case is provided.


Cheers!

https://redd.it/1c789lc
@r_devops

How is Your Company's Use of AWS Evolving?

Are your companies increasing their use of AWS services, maintaining their current level of usage, or are there instances where projects are being moved back on-premises?

I'm interested in understanding the reasons behind these decisions as well. Whether it's due to cost, security concerns, performance issues, or any other factors....

If you're comfortable sharing, I'd appreciate if you could also mention the industry your company operates in and the scale of your AWS usage. :-)

https://redd.it/1c75bpv
@r_devops

Service Mesh Preferences- Istio, Linkerd, or Something else?

Ok so we're not a service mesh, nor do we have a stake in either of these products so don't come for us in the comments (though- if you want to talk Gateways, hit us up).

We recently received a guest blog that covered a full guide on traffic management in Istio: https://www.getambassador.io/blog/istio-kubernetes-traffic-management-guide

Now, we've played well with both Istio and Linkerd from an integration + open source perspective, but we're curious...do y'all have one you prefer? Or maybe a third service mesh option you prefer more? Any downsides to either that you've seen?

Give us the best and worst you've had from using either of these. :)

https://redd.it/1c7a2t6
@r_devops

Resume/CV Review

https://imgur.com/a/fdFWyDk

[Reposting because wrong link\]

Hello all, this is my resume (version 8). I have been approached by recruiters and applied to a lot of positions. However no one has come back with an interview for me, really unsure what it is. I am a DevOps Engineer of 3 years really hoping someone can shed some light weather I am trolling with my resume or improvements. Let me know. Many thanks.

https://redd.it/1c7elez
@r_devops

Best way to learn Windows IIS

Hey all! I need to learn Windows IIS and am trying to figure out the best way to do it. Is there a recommended book, tutorial, YouTube series I should be looking at?


Context. I've been hired as a lead software engineer for a brand new department. I need to be able to deploy legacy code (OLD asp) *and* modern django applications on an air gapped system that utilizes Windows Server and Windows IIS.


All my experience is Linux and Mac. I haven't touched any devops stuff in a *really* long time, and back then I did LAMP, mostly small scale and from the developer side.

We are building and supporting internal tools for support staff. We aren't expecting high traffic.

\-------------------------------------

Bonus question/thought. No one around me has any idea of how modern software engineering is supposed to work. The deployment process from the guy I replaced was to write code in production, in the production environment, and hit save (no, I'm not kidding). I'm trying to set up something resembling modern day development practices from scratch.

Is there any benefit to even using Windows Server/IIS? Or is the group I'm working for throwing away money?

https://redd.it/1c7drx5
@r_devops

Safe terraform apply in CI

I know that some shops do terraform apply as part of their CI.


How do ensure that your CI doesn't do something horrific in production? In terraform plan we trust?

Do you use GitHub Actions? What has been your experience?

https://redd.it/1c7ismb
@r_devops

Blue Green and CD on EKS

Hey guys, so at my current company - we are having some tension between product and qa/platform. Currently we are doing manual code freezes so qa can thoroughly test a bunch of things before we push to prod. This is causing tension as product engineering wants things out ASAP. Problem being is we’re getting tons of bugs due to this. We are severely lacking the CD portion of our pipeline, and one of my plans (in agreement with the VP of infra) is to get B/G deploys rolled out using something like Argo rollouts (paired with ArgoCD). This way we can push to “prod” and have QA run as many tests as possible without causing a code freeze. We use automated tests like cypress, but another step will be automating more manual QA processes down the line.

In the meantime is this a good idea, would love to hear pushback. Currently we only have a dev env, and while yes we can do a staging - my exp in past is that things can pass staging but break in prod due to minor discrepancies.

Would love to hear what you guys have been doing, especially revolving b/g and CD. We are GitHub actions shop and deploy with helm.

Also given I have a chance to really architect this however I’d like, let me know how you would do deployments if you had no limitations. My last job we did GitHub actions and argocd as well, but did not have any B/G as there were not frequent app breaking changes.

(Note I am new to this company so I did not set any of the current processes up).

https://redd.it/1c7nnhb
@r_devops

Seeking Advice from Respected Experienced DevOps Engineers.

I’m a level 1 seller on Fiverr, offering Linux system and server administration services. To be honest, I haven't completed any course fully yet; I've only done about 30 to 50 percent of each course I started, mainly due to procrastination. However, since I'm good at problem-solving and using Google searches, I accept orders from buyers for projects that I have never worked on and have no idea how to handle.

I’m very interested in DevOps, as I believe that to apply for a job in this field, you must have skills in CI/CD, IaC, etc., and I’ve noticed many Linux sysadmins are moving into DevOps. I've realized I need to learn things properly. I’m currently about 40% through the RHCSA and my goal is to complete it without leaving it incomplete. After that, I'm considering pursuing Docker and Kubernetes as they are critical for DevOps roles. I believe good Linux skills are a prerequisite for DevOps.

I'm also practicing networking fundamentals on EVE-NG. I would like to know if I’m on the right path now. Hearing about your success story, the path you followed, and any recommended courses would be much appreciated. Thank you so much!"

This revised message reflects your updated goals and emphasizes your commitment to completing the RHCSA before moving on to Docker and Kubernetes. It’s great to see that you're setting specific objectives to guide your learning path in DevOps.



https://redd.it/1c7qrsi
@r_devops

Running Kong on GKE

Hi, new to Kong, love the concepts. Experimented with Gateway APIs on KIND, worked great. Now need to move it to GKE. There are several things I would like to do that I'm not sure how:

1. How to make the kong-proxy service to use the reserved static ip in GKE? Before I would use the kubernetes.io/ingress.regional-static-ip-name annotation on Ingress, but not sure how to do that with Gateway APIs

2. What is the right way of running multiple replicas of kong? I saw there is replicaCount value in the chart, but there is also a cluster.enabled mode. Should it be enabled as well?

3. Related to 2, how to run those replicas so that they are spread across zones & nodes?

4. I'm installing using kong chart from https://charts.konghq.com and with pretty lean values below. What are other fields you would suggest to specify for a typical production use.

ingressController:
  enabled: true

5. Want to apply CRDs separately. What would be the best place to find them?

Thank you!

https://redd.it/1c7qoin
@r_devops

How can we update docker containers on edge device with no Internet or should we simply get rid of docker?

We need to deploy docker container on edge devices which wont be having Internet. These devices occasionally connect to a network and one of the device (lets call it H) on the network will have internet access. So, I want to know how we can update docker containers in such scenario. I imagine following two approaches:

1. Create tar of image. Copy it to edge device (say over USB) and then update the image on the edge device.
2. Create local registry on device H. Pull the updated image from remote registry to local registry on device H. Make edge device pull only updated layers from this local registry on H.

I feel option 2 is good given it only moves updated layers between devices making update size small, while tar contains all layers resulting in tar of size 300 MB. So, option 2 was good option till we thought device H will be x86 Windows device. But now we are told that it can be Android or iPad companion device for edge device. We cannot run docker registry on Android or iPad right? So what solution we have remained with for updating docker container on edge device?

We did various docker related POCs. But now after knowing the fact that device H can be android or iPad device, we may have to get rid of docker completely and deploy apps say through other non container approaches say snap etc.

Should we let edge device access Internet through device H say through tethering? We don't want Internet on edge device for security reason, but then should we restrict the Internet access on edge to only servers hosting docker registry? Or there can be any better solution without requiring Internet on edge device at all?

https://redd.it/1c7sb25
@r_devops

Salary Expectation for 8 yoe in India

Hi Everyone,

May I ask what should be the expected CTC for a Devops Engineer in Bengaluru (India).

Thanks



https://redd.it/1c7s4e0
@r_devops

Jenkins exodus? Is it really a thing?

I recently attended Kubecon CloudNative con, and I can't remember who, but someone mentioned a "Jenkins exodus" occurring in the industry.

Where I work we heavily use Jenkins, with Spinnaker sprinkled on top for cross CI communication and pipeline orchestration mainly.

Seeing all the "cool" tools out there, opensource or not, makes me think we need to evolve for the future. For example, Argo workflows, Tekton, Harness, Octopus Deploy, Gitlab, Azure DevOps, Github actions etc.

We have a mix of legacy and cloud native microservice products so any migration/change would be taken slowly.

I'm interested in what people are using but more importantly what their experience is like.

In the Jetbrains State of Developer Ecosystem report 2023 (https://blog.jetbrains.com/teamcity/2023/07/best-ci-tools/#the-best-ci-cd-tools-in-devops-according-to-our-state-of-developer-ecosystem-report), they polled over 26k developers worldwide and Jenkins, Github Actions and Gitlab CI were top of the pile when it comes to which tools are regularly used, but it doesn't give any context around whether the developers are happy with them!

Be great to hear what you're using, and what your experience is like! TIA

​

https://redd.it/1c7tvg4
@r_devops

Lots of work opportunities I see lately are looking for full-stack experience, including with application layer

I started my career as a developer and for various reasons, it wasn't for me. I found AWS, cloud infrastructure, CI/CD etc. came to me more naturally and when I finally made the transition, I was happy to be able to focus on that and let the developers take care of the application. At my first company there were a few people who were completely full-stack, but it was normal that platform engineers managed the infrastructure while the developers managed the application. I know that may raise questions about silos but that's another can of worms.

Lately I've been looking for a new job and my company is also looking for a project for me (we're a consultancy). One opportunity that got sent to me today asks for "a background in Enterprise Application Development" and "Proficient understanding of the entire technology stack of the service, from application to infrastructure". A few months ago I inquired to a friend about any work going at his company and he said that me not knowing much about application code might be a problem, as that's just not how they did it there.

The fact is that I really don't know much about application code and there's not much I feel I can offer for it. I get the impression that most of the folks here don't get involved with the application much either. Do you also see this in job advertisements like this lately? Is it ok to just want to keep the focus on infrastructure or is it limiting? I'm in the Netherlands, not sure how different it is between countries.

https://redd.it/1c7umuq
@r_devops

Just feeling a bit defeated.

Managed to make it to the last interview for the 4th time to see this again: “we decided to go with a candidate with more experience.”

I’m closing in on 6 years of experience. Feels like I’m constantly competing against people with 10+ years nowadays. This wasn’t even a senior role. I know there’s been a tech exodus, but damn.

Anyone else in the same boat?

https://redd.it/1c7ur1a
@r_devops

People who have personally migrated from Istio to Cilium, what was your experience like?

We’re pretty heavily committed to Istio, but I can’t deny that.. well almost any of the service mesh solutions are less complex and easier to maintain.

Those who have made the jump to Cilium specifically, what has your experience been and would you do it again? Thanks!

https://redd.it/1c7wxzd
@r_devops

decided for Flux against Argo now folks want a UI ...

well the title says it. I've worked with both Flux and Argo in the past and I decided for Flux. Now people are asking me for a UI. FML.

I've stumbled upon this: https://flux-subsystem-argo.github.io/website/

has any of you guys used this? It seem well maintained and even has Flux' Daniel Holbach on board.

https://redd.it/1c7xvmc
@r_devops

How do you deal with developers not paying attention to slack

Frustrated. I feel like I'm the only one paying attention to things like alerts, PR requests, pings, or even just conversation. Hours go by in the workday and people aren't responding to my pings or PRs. Builds are broken. I'm the only one reviewing PRs. I know it's a culture thing and an expectation that should be set by management but I don't really have any control or influence there.

Edit: just want to clarify my expectations. We all have work to do and are busy with tasks. I get it. I think it's pretty reasonable to watch for broken builds after a push. Or do respond to messages in a team slack channel within half a day. Or to acknowledge someone asked for you to look at a PR within a few hours (not saying you need to look at it then). If an alert in production fires yes IMO that justifies stopping your current task to investigate. If it's bad alert then fix it or turn it off.

Slack is literally the only way we can communicate with each other for the majority of us WFH. We need to acknowledge each other.

https://redd.it/1c7zbsu
@r_devops

Do most companies stick to the big 3 (AWS, Azure, GCP)? How come I never see Digital Ocean or Linode come up?

There are many online tutorials that teach people how to get set up on Digital Ocean/Linode but I noticed that I've never seen people discuss production issues on them. Most people only talk about the big 3. Even job listings only mention the big 3 and never about Digital Ocean.

Do big companies or even small tech companies not consider Digital Ocean and others? Are those renegated to hobby projects or small time customers?

https://redd.it/1c814pa
@r_devops

DevOps & Cloud Architecture Books

Hello DevOps folks,

I kinda find myself to struggle in understanding the big picture when taking on work tasks in my current role (Cloud Engineer - Azure). I feel like that I'm working only on small gears of a huge machine.

My company does consulting for big/medium businesses and I can't really understand how changing some IaC or YAML code provides value to the big picture or environments built with them. I mean, I kinda do, but i struggle to visualise the interconnection of the whole environment.

Based on this, can you recommend 1/2 books that would provide me the "big picture" of DevOps or Cloud Architecture solutions, especially in a consulting point of view? I want so bad to understand from 0 to 100 how to make up a digital DevOps/Cloud solution based on real life scenario but i miss the "knowledge components" (maybe something that is acquired after years of experience?).

Thanks in advance!

https://redd.it/1c86r1z
@r_devops