What technologies do you find the most challenging to use in your jobs?

I'm looking to find common challenges so I can improve in those areas and speak to them during interviews.

https://redd.it/t3ipt0
@r_devops

How do you use ArgoCD to deploy applications and infrastracture related components?

Hi there, how is everyone?

Lately I switched to GitOps and I decided to use ArgoCD for that.. I am a software engineer that really like DevOps and I am looking to find ways to impement GitOps in good practices.

Currently I have a single repository to hold all the components related to k8s and then other repos to hold the source code of the microservices.

In the repos that I have the source code I have a CI with github actions where I test, build the docker image, and then update the image tag (in the other repo) for that specific application in the repo that I hold the manifest for each application, then argo grabs the changes of the tags and stuff, and automatically updates my appication.

I use kustomize to separate the environments with a base and overlays pattern.

If you can please share your practices on how are you handling/implementing GitOps with Argo.

I am mostly interested in these:
- What do you store in the repo that you have for gitops?
- What are your practices to automate as much as possible
- How do you deploy other third party apps (charts, etc)
- Do you automate the provisioning of the ArgoCD with IAC tool like Teraform?

I am asking these questions as I want to
leave everything on that Git Repository, and to automate as much as possible.

Thank you very much in advance.

https://redd.it/t3ug45
@r_devops

Does any arbitrary Domain Name provider support round robin DNS load balancing, or do you need a specific provider to do that?

Does the DNS protocol do round-robin by default, or is it something you need to set up in a particular provider's workflow, like cloudflare? I.e: if I use some simple provider like namecheap or google domains and just create multiple A records, is it part of the protocol to do round-robin?

https://redd.it/t3td8f
@r_devops

Where would you host/store programs that are not actual web apps but some background jobs in linux server?

so typically on linux server if we host webapp we put into /var/www dir


but where should I place a program that is supposed to be run as background job. Is ok to put it in /var/www dir as well?

https://redd.it/t43uv7
@r_devops

Getting Started With Kubernetes Ingress Controllers

Do you want to learn about the Kubernetes Ingress Controller and how to use it? 🤔

Check out this excellent article by Farhan. It covers everything you need to know to get started with Kubernetes Ingress Controllers. 👇

https://blog.getambassador.io/getting-started-with-kubernetes-ingress-controllers-f3b669d19b31

https://redd.it/t47b88
@r_devops

Terraform recreating private endpoint for sql database

Hi all,

I have upgraded azurerm version to 2.91.0 and since then all my private endpoints are getting destroyed and recreated. The resource forcing replacement is not getting changed. I tried to look around for this issue but couldn't find anything. I also manually removed from state and imported it again just in case something on the state was messed up but the problem persists.

Any idea why this might happen? What could cause a resource to be recreated on each plan when it hasn't been modified.

​

Thanks!

https://redd.it/t48nhp
@r_devops

In E2EE, do all TLS certs from the edge to the LBs and origins need the same certificate?

We use CloudFlare, AWS ELB and EC2. At the edge we have a custom wildcard TLS cert for our domain. The same certificate is replicated to AWS ACM (for ELB) and EC2 servers. The origins have the private key as well to decrypt traffic. These certificates expire yearly and it’s a pain to replace them throughout our entire infrastructure.

Here’s where my E2EE understanding isn’t clear. I wondered if we actually needed the same certificate replicated like this and instead leverage AWS-issued ACM certs that are auto-renewed and Cloudflares Origin Certificates that have a longer expiry. I replaced the certs at the LB and origins to use these certs and we still have encryption from what I see at the edge.

Is everything still encrypted? Or is encryption only happening at the edge and not E2EE? Accessing the LB directly shows it’s not encrypted so I assume we’re not doing E2EE. How could we be more efficient with the renewals of TLS certs across our infrastructure?

https://redd.it/t4bpsv
@r_devops

Would you use Portainer as your gitops engine?

Portainer isnt a CI or CD tool, but instead a gitops tool. it means all deployment configuration manifests that describe the "desired state" are held in git, automation is triggered by changes that occur in git, and as a result, this dramatically reduces developer tool context switching as they can trigger deployment updates from within the tool they use every day.

benefits:

Portainer's capability exists for Docker, Docker Swarm, and Kubernetes, so if you run a hybrid environment, we are the obvious choice.
ZERO load on your clusters, and nothing to install or maintain.
"enforce" mode, which overrides the running deployment with what's defined in Git
"change window" option, which disables automation outside of a preset time window (which is brilliant if your apps are not fully capable of rolling updates, but you still want the benefit of Gitops).

Portainer as part of your CIDC Pipeline

​

For those who are interested in GitOPs with CICD, you use Portainer as your gitops engine?

https://redd.it/t4cp02
@r_devops

Monthly 'Shameless Self Promotion' thread - 2022/03

Feel free to post your personal projects here. Just keep it to one project per comment thread.

https://redd.it/t4fo08
@r_devops

Monthly 'Getting into DevOps' thread - 2022/03

**What is DevOps?**

* [AWS has a great article](https://aws.amazon.com/devops/what-is-devops/) that outlines DevOps as a work environment where development and operations teams are no longer "siloed", but instead work together across the entire application lifecycle -- from development and test to deployment to operations -- and automate processes that historically have been manual and slow.

**Books to Read**

* [The Phoenix Project](https://www.amazon.com/Phoenix-Project-DevOps-Helping-Business/dp/1942788290) - one of the original books to delve into DevOps culture, explained through the story of a fictional company on the brink of failure.
* [The DevOps Handbook](https://www.amazon.com/dp/1942788002) - a practical "sequel" to The Phoenix Project.
* [Google's Site Reliability Engineering](https://landing.google.com/sre/books/) - Google engineers explain how they build, deploy, monitor, and maintain their systems.
* [The Site Reliability Workbook](https://landing.google.com/sre/workbook/toc/) - The practical companion to the Google's Site Reliability Engineering Book
* [The Unicorn Project](https://www.amazon.com/Unicorn-Project-Developers-Disruption-Thriving-ebook/dp/B07QT9QR41) - the "sequel" to The Phoenix Project.
* [DevOps for Dummies](https://www.amazon.com/DevOps-Dummies-Computer-Tech-ebook/dp/B07VXMLK3J/) - don't let the name fool you.

**What Should I Learn?**

* [Emily Wood's essay](https://crate.io/a/infrastructure-as-code-part-one/) - why infrastructure as code is so important into today's world.
* [2019 DevOps Roadmap](https://github.com/kamranahmedse/developer-roadmap#devops-roadmap) - one developer's ideas for which skills are needed in the DevOps world. This roadmap is controversial, as it may be too use-case specific, but serves as a good starting point for what tools are currently in use by companies.
* [This comment by /u/mdaffin](https://www.reddit.com/r/devops/comments/abcyl2/sorry_having_a_midlife_tech_crisis/eczhsu1/) - just remember, DevOps is a mindset to solving problems. It's less about the specific tools you know or the certificates you have, as it is the way you approach problem solving.
* [This comment by /u/jpswade](https://gist.github.com/jpswade/4135841363e72ece8086146bd7bb5d91) - what is DevOps and associated terminology.
* [Roadmap.sh](https://roadmap.sh/devops) - Step by step guide for DevOps or any other Operations Role

Remember: DevOps as a term and as a practice is still in flux, and is more about culture change than it is specific tooling. As such, specific skills and tool-sets are not universal, and recommendations for them should be taken only as suggestions.

**Previous Threads**
https://www.reddit.com/r/devops/comments/ru3zhm/monthly_getting_into_devops_thread_202201/

https://www.reddit.com/r/devops/comments/r6myz4/monthly_getting_into_devops_thread_202112/

https://www.reddit.com/r/devops/comments/qkgv5r/monthly_getting_into_devops_thread_202111/

https://www.reddit.com/r/devops/comments/pza4yc/monthly_getting_into_devops_thread_2021010/

https://www.reddit.com/r/devops/comments/pfwn3g/monthly_getting_into_devops_thread_202109/

https://www.reddit.com/r/devops/comments/ow45jd/monthly_getting_into_devops_thread_202108/

https://www.reddit.com/r/devops/comments/obssx3/monthly_getting_into_devops_thread_202107/

https://www.reddit.com/r/devops/comments/npua0y/monthly_getting_into_devops_thread_202106/

https://www.reddit.com/r/devops/comments/n2n1jk/monthly_getting_into_devops_thread_202105/

https://www.reddit.com/r/devops/comments/mhx15t/monthly_getting_into_devops_thread_202104/

https://www.reddit.com/r/devops/comments/lvet1r/monthly_getting_into_devops_thread_202103/

**Please keep this on topic (as a reference for those new to devops).**

https://redd.it/t4fozq
@r_devops

Am I too old to get into devops

Just sharing my worries. I just got 38 and I have been doing a Sysadmin job for the past 11 years. That said, I'm done with Sysadmin and I want to get into devops. I am currently learning what I need to do this move. My goal of to do it before I reach 40. My worry tho is that I will get bypassed by younger people in their 20s and early 30s. I just feel so old right now and kind of in a middle life career crisis. Need some advice on this.

https://redd.it/t4m9is
@r_devops

What's your celebration ritual ( dance, sounds, whatever) that you do when you find the solution ?

As me ,right now, James brown dance after a huge terraform refactor that ran on the first run changing 79 resources perfectly.

​

So, What you do when you solve that nasty issue or your code works on the first try?

https://redd.it/t4n7er
@r_devops

Am I being unrealistic with my salary expectations?

I'm currently working as a Principal DevOps engineer in Austin, Tx making $145k/yr. My salary hasn't changed in a couple years, so I have been looking elsewhere. I started as a SysAdmin with a Linux background and moved into the DevOps field for about 8 years now. I've been looking for other jobs in my area and gotten offers at almost every place I've interviewed. However, the salary for these positions seems to be very low for the position being advertised. For instance, I applied for 3 senior roles asking for 7+ years experience and got contingent offers from all of them at $120-$135/k range, no other comp besides your normal benefits and remote 100%. Another was for principal role similar to mine but at $115k/ yr, to include total comp, it would be $165k. These are mid level to established fortune companies I've applied at. I'm planning to apply to other companies in different states as well due to the salaries not meeting my expectations. Am I being unrealistic here, or are these normal salaries in Texas for senior roles?

https://redd.it/t50wph
@r_devops

If you were an experienced DevOps engineer, joining a company with inexperienced people doing the devops, how would you go about "fixing" things?

We use AWS as our cloud provider, and Jenkins as our CI/CD tool.

Most of our infra is a mix of manually spun up stuff from the AWS console, and some stuff created using terraform.

Most of the stuff that was spun up, our devs learnt how to do on the fly. Then at some point we hired a devops contractor to help us transition to using IaC (terraform), but he left after a few months without completing it, so we only had some of the infra represented in terraform. Then after that, we spun up some more stuff manually because we needed to. The jenkins jobs follow a similar story.

Basically, our stuff is a mess. We get by, but we feel this is one area that could probably be optimized. We're looking at hiring an experienced Devops/sysops engineer next quarter, but naturally we're not entirely sure how to put them to work.

If you were the engineer joining such a company, how would you go about rectifying their state of affairs.

https://redd.it/t533gr
@r_devops

Puppet in AKS

Junior DevOps here but.. sadly the only DevOps for the company I work for.

I've been working on a POC for about 6 months and am completely self-taught. I was originally hired into the role with the idea that I would be groomed by my boss, who left for another company about two months into this position.

All this to say that I'm not the most knowledgeable DevOps Engineer out there :)

Couple questions about Puppet specifically:

1) My idea is to have 1 Puppet CA Container in AKS in it's own pod. This container will manage the certs that the Prod/Dev/QA containers(in their own respective pods) use. Agents will connect directly to the Prod/Dev/QA pods. I will use Hiera (haven't played with this yet) to direct to the environments/modules they need. Manage the modules with r10k.

Does this sound.. like good architecture? I'm struggling with the Architecture of this thing.

2) I cannot for the life of me get the Docker Puppet CA Container to share it's cert with the Puppet test container within AKS. I've created a service that has port/targetPort set to 8140. The containers and the service are all on the same label. I've exposed 8140 on both containers. I have no clue on this one. I can get them to talk in Container Instances just fine. The test container has CAENABLED=false and CAHOSTNAME=the hostname of the puppet server. No bueno.

I've set up a mock environment with my home pc's and inside container instances I'm running just fine. But I would love to get this into AKS and then start work on heira/r10k.

Puppet help is hard to find it seems.

https://redd.it/t5akwt
@r_devops

What can I run in a company environment to learn Kubernetes officially?

Let's say I did some initial learning of Kubernetes, build my home lab raspberry cluster etc.

And like to evolve.

Developers do not use Kubernetes at the moment.

​

Want to use Kubernetes for something real in my job.

The best ideas, for now, would be running SonarQube on Kubernetes or some sort of monitoring tool

We are mostly use azure and .net

https://redd.it/t5hcht
@r_devops

Recommended tutorial on setting up a generic web app so i can better understand cicd and app lifecycle

As I'm on my ever evolving devops journey, there are gaps in areas of my knowledge about the app dev process which is why there are some concepts that i dont get. Since i learn by action, can anyone recommend a simple tutorial on setting up a very generic app that has resources of a typical app (e.g., server, front end, back end, database, etc)? I dont want to be a dev but want to learn from going through the steps of setting up an app with typical components.

Im familiar with most of the concepts from ops and support but want to learn more about the dev perspective so i can better grasp cicd, load balancers, containers, etc.

https://redd.it/t5je44
@r_devops

ELI5: What is CI/CD and Why do we need them?

As the title says, as a wanna-be DevOps Engineer, I would like to understand What is CI/CD and Why do we need them?

A few months ago, I asked for GitLab CI/CD example projects and one user commented,

"CI/CD is glue code between a language/runtime (e.g. Java/OpenJDK, JS/Node, Ruby, something in a Docker Container) incl delivery (depending on your kind of artifact, eg. Docker Container, Maven Repository, npm compliant registry) and a deployment mechanism to the environment you want to run in (e.g. Docker, Docker with docker-compose, Ansible, Kubernetes, Kubernetes with Helm, Kubernetes with kustomize, etc.)"

I would like to understand a bit more than that and I appreciate you reading this post.

https://redd.it/t5nufe
@r_devops

Deployment patters in Kubernetes

I am new to Kubernetes and trying to deploy my services. I am curious as to what is the best practice to deploy your replica sets and containers in Kubernetes. Should I just create templates of deployments abs buildconfigs abs have the cicd pipeline inject the end variables in them? Or is there another pattern? Does anyone have any good opinion/reading material on this?

https://redd.it/t5q2xm
@r_devops

How is your approach on working with CI/CD pipelines in azure?

Hi guys,

I started a few months ago as a devops engineer. Everything is going well and there is still a lot to learn. I started with creating a pipeline in our sandbox environment, but I’m a bit stuck. I think that we work with YAML pipelines (I’m also quite handy with python)

I understand the basics in the Microsoft documentation, but what I’m missing is the way of thinking (if you understand what I mean). If you guys would get a task to design a pipeline which fetches files from a location, transfers it to another and you want to visualize the results on a Splunk dashboard; how would you handle this?

I’m not necessarily looking for the answer on that task, but I’m more interested in what would be your approach and where do you get your sources from?

I have some documentation and videos, but if someone could share a library or practice assignments, that would be great!

https://redd.it/t5zdic
@r_devops

Wanting to Pivot into DevOps

I'm looking for some advice as to how I can pivot into a Jr DevOps Engineer or DevOps Engineer role. I have a BS in IT Management & currently ~5yrs of general IT support & sys admin experience (networking & server/virtualization management), and for the past year have been pursuing DevOps technologies in my own time. I built a home-lab that I use to learn Docker & K8s as well as the networking & storage that goes along with those. I'm currently studying for the CKA exam (Online Courses & "Kubernetes the Hard Way" by KelseyHightower) planning to test in April or early May, and plan to at least get an AWS Cloud Practitioner Cert afterwards.

My primary question is at what point should I start looking for job openings, and what else do I need to focus on? I know I still have a lot to still learn and I want to at least take courses and practice more topics on my own and possibly integrate them into my current work (ansible, terraform, CI/CD pipelines, python, etc.) but I'm having a hard time figuring out where I should be spending my time, and what a realistic timeline is to make this transition.

Any guidance would be really appreciated.

https://redd.it/t620tu
@r_devops