Reddit DevOps
270 subscribers
11 photos
31.1K links
Reddit DevOps. #devops
Thanks @reddit2telegram and @r_channels
Download Telegram
Do you trust SAS Github?

EDIT: Title should be Gitlab, not Github. It's early.

We've got some multi-org collaborations and tenancy expansion to some of our platforms coming up, and our VPN'd private Gitlab instance isn't ideal. We're also looking to leverage cloud CI toolsets like NVIDIA Air CI in the future, which require a publically-available repo.

So we have two options... to harden our self-service Gitlab instance and stick our org's SSO in front of it, or move to cloud Gitlab.

We're not idiots and clearly secrets and private detail are already encrypted with a vault key living outside the repo, public access would ideally be source-ip scoped and/or behind MFA SSO etc, but if Gitlab cloud were compromised, even internal IP address ranges, server names, etc in our repos would be icky to have out in public (and burdensome to encrypt every little thing). And if they're compromised as a whole (rather than our specific repos/account), it might not matter what external security measures we have in place.

But securing our own might be more of a risk. 🤷‍♂️

Does your org trust its internal code in cloud repos like Gitlab, assuming sensible security measures are taken? Not picking on Gitlab here of course, as the question is broader-- they're just who we're looking at using.

Thanks in advance.

https://redd.it/t3jzpj
@r_devops
Learning about Azure DevOps, does it looks like a solid pipeline for AKS?

Does below look like a solid pipeline for Kubernetes deployment? QA and PROD environments are built using the same Terraform configs.

​

1. Build Docker Images on Agent
2. Integration Test with Minikube on Agent
3. Push images to ACR
4. Provision AKS QA environment with Terraform
5. Deploy to QA Environment
6. Smoke Test
7. Destroy QA Environment
8. Approval
9. Provision AKS PROD environment with Terraform (on subsequent runs nothing happens if no changes made to Terraform config, if there are changes these were already tested as part of the QA since it deploys from same config. This is a stage I'm not to sure about. Should PROD provisioning be part of CI/CD or is it risky?)
10. Deploy to PROD environment

https://redd.it/t3mfd4
@r_devops
What technologies do you find the most challenging to use in your jobs?

I'm looking to find common challenges so I can improve in those areas and speak to them during interviews.

https://redd.it/t3ipt0
@r_devops
How do you use ArgoCD to deploy applications and infrastracture related components?

Hi there, how is everyone?

Lately I switched to GitOps and I decided to use ArgoCD for that.. I am a software engineer that really like DevOps and I am looking to find ways to impement GitOps in good practices.

Currently I have a single repository to hold all the components related to k8s and then other repos to hold the source code of the microservices.

In the repos that I have the source code I have a CI with github actions where I test, build the docker image, and then update the image tag (in the other repo) for that specific application in the repo that I hold the manifest for each application, then argo grabs the changes of the tags and stuff, and automatically updates my appication.

I use kustomize to separate the environments with a base and overlays pattern.

If you can please share your practices on how are you handling/implementing GitOps with Argo.

I am mostly interested in these:
- What do you store in the repo that you have for gitops?
- What are your practices to automate as much as possible
- How do you deploy other third party apps (charts, etc)
- Do you automate the provisioning of the ArgoCD with IAC tool like Teraform?

I am asking these questions as I want to
leave everything on that Git Repository, and to automate as much as possible.

Thank you very much in advance.

https://redd.it/t3ug45
@r_devops
Does any arbitrary Domain Name provider support round robin DNS load balancing, or do you need a specific provider to do that?

Does the DNS protocol do round-robin by default, or is it something you need to set up in a particular provider's workflow, like cloudflare? I.e: if I use some simple provider like namecheap or google domains and just create multiple A records, is it part of the protocol to do round-robin?

https://redd.it/t3td8f
@r_devops
Where would you host/store programs that are not actual web apps but some background jobs in linux server?

so typically on linux server if we host webapp we put into /var/www dir


but where should I place a program that is supposed to be run as background job. Is ok to put it in /var/www dir as well?

https://redd.it/t43uv7
@r_devops
Getting Started With Kubernetes Ingress Controllers

Do you want to learn about the Kubernetes Ingress Controller and how to use it? 🤔

Check out this excellent article by Farhan. It covers everything you need to know to get started with Kubernetes Ingress Controllers. 👇

https://blog.getambassador.io/getting-started-with-kubernetes-ingress-controllers-f3b669d19b31

https://redd.it/t47b88
@r_devops
Terraform recreating private endpoint for sql database

Hi all,

I have upgraded azurerm version to 2.91.0 and since then all my private endpoints are getting destroyed and recreated. The resource forcing replacement is not getting changed. I tried to look around for this issue but couldn't find anything. I also manually removed from state and imported it again just in case something on the state was messed up but the problem persists.

Any idea why this might happen? What could cause a resource to be recreated on each plan when it hasn't been modified.

​

Thanks!

https://redd.it/t48nhp
@r_devops
In E2EE, do all TLS certs from the edge to the LBs and origins need the same certificate?

We use CloudFlare, AWS ELB and EC2. At the edge we have a custom wildcard TLS cert for our domain. The same certificate is replicated to AWS ACM (for ELB) and EC2 servers. The origins have the private key as well to decrypt traffic. These certificates expire yearly and it’s a pain to replace them throughout our entire infrastructure.

Here’s where my E2EE understanding isn’t clear. I wondered if we actually needed the same certificate replicated like this and instead leverage AWS-issued ACM certs that are auto-renewed and Cloudflares Origin Certificates that have a longer expiry. I replaced the certs at the LB and origins to use these certs and we still have encryption from what I see at the edge.

Is everything still encrypted? Or is encryption only happening at the edge and not E2EE? Accessing the LB directly shows it’s not encrypted so I assume we’re not doing E2EE. How could we be more efficient with the renewals of TLS certs across our infrastructure?

https://redd.it/t4bpsv
@r_devops
Would you use Portainer as your gitops engine?

Portainer isnt a CI or CD tool, but instead a gitops tool. it means all deployment configuration manifests that describe the "desired state" are held in git, automation is triggered by changes that occur in git, and as a result, this dramatically reduces developer tool context switching as they can trigger deployment updates from within the tool they use every day.

benefits:

Portainer's capability exists for Docker, Docker Swarm, and Kubernetes, so if you run a hybrid environment, we are the obvious choice.
ZERO load on your clusters, and nothing to install or maintain.
"enforce" mode, which overrides the running deployment with what's defined in Git
"change window" option, which disables automation outside of a preset time window (which is brilliant if your apps are not fully capable of rolling updates, but you still want the benefit of Gitops).

Portainer as part of your CIDC Pipeline

​

For those who are interested in GitOPs with CICD, you use Portainer as your gitops engine?

https://redd.it/t4cp02
@r_devops
Monthly 'Shameless Self Promotion' thread - 2022/03

Feel free to post your personal projects here. Just keep it to one project per comment thread.

https://redd.it/t4fo08
@r_devops
Monthly 'Getting into DevOps' thread - 2022/03

**What is DevOps?**

* [AWS has a great article](https://aws.amazon.com/devops/what-is-devops/) that outlines DevOps as a work environment where development and operations teams are no longer "siloed", but instead work together across the entire application lifecycle -- from development and test to deployment to operations -- and automate processes that historically have been manual and slow.

**Books to Read**

* [The Phoenix Project](https://www.amazon.com/Phoenix-Project-DevOps-Helping-Business/dp/1942788290) - one of the original books to delve into DevOps culture, explained through the story of a fictional company on the brink of failure.
* [The DevOps Handbook](https://www.amazon.com/dp/1942788002) - a practical "sequel" to The Phoenix Project.
* [Google's Site Reliability Engineering](https://landing.google.com/sre/books/) - Google engineers explain how they build, deploy, monitor, and maintain their systems.
* [The Site Reliability Workbook](https://landing.google.com/sre/workbook/toc/) - The practical companion to the Google's Site Reliability Engineering Book
* [The Unicorn Project](https://www.amazon.com/Unicorn-Project-Developers-Disruption-Thriving-ebook/dp/B07QT9QR41) - the "sequel" to The Phoenix Project.
* [DevOps for Dummies](https://www.amazon.com/DevOps-Dummies-Computer-Tech-ebook/dp/B07VXMLK3J/) - don't let the name fool you.

**What Should I Learn?**

* [Emily Wood's essay](https://crate.io/a/infrastructure-as-code-part-one/) - why infrastructure as code is so important into today's world.
* [2019 DevOps Roadmap](https://github.com/kamranahmedse/developer-roadmap#devops-roadmap) - one developer's ideas for which skills are needed in the DevOps world. This roadmap is controversial, as it may be too use-case specific, but serves as a good starting point for what tools are currently in use by companies.
* [This comment by /u/mdaffin](https://www.reddit.com/r/devops/comments/abcyl2/sorry_having_a_midlife_tech_crisis/eczhsu1/) - just remember, DevOps is a mindset to solving problems. It's less about the specific tools you know or the certificates you have, as it is the way you approach problem solving.
* [This comment by /u/jpswade](https://gist.github.com/jpswade/4135841363e72ece8086146bd7bb5d91) - what is DevOps and associated terminology.
* [Roadmap.sh](https://roadmap.sh/devops) - Step by step guide for DevOps or any other Operations Role

Remember: DevOps as a term and as a practice is still in flux, and is more about culture change than it is specific tooling. As such, specific skills and tool-sets are not universal, and recommendations for them should be taken only as suggestions.

**Previous Threads**
https://www.reddit.com/r/devops/comments/ru3zhm/monthly_getting_into_devops_thread_202201/

https://www.reddit.com/r/devops/comments/r6myz4/monthly_getting_into_devops_thread_202112/

https://www.reddit.com/r/devops/comments/qkgv5r/monthly_getting_into_devops_thread_202111/

https://www.reddit.com/r/devops/comments/pza4yc/monthly_getting_into_devops_thread_2021010/

https://www.reddit.com/r/devops/comments/pfwn3g/monthly_getting_into_devops_thread_202109/

https://www.reddit.com/r/devops/comments/ow45jd/monthly_getting_into_devops_thread_202108/

https://www.reddit.com/r/devops/comments/obssx3/monthly_getting_into_devops_thread_202107/

https://www.reddit.com/r/devops/comments/npua0y/monthly_getting_into_devops_thread_202106/

https://www.reddit.com/r/devops/comments/n2n1jk/monthly_getting_into_devops_thread_202105/

https://www.reddit.com/r/devops/comments/mhx15t/monthly_getting_into_devops_thread_202104/

https://www.reddit.com/r/devops/comments/lvet1r/monthly_getting_into_devops_thread_202103/

**Please keep this on topic (as a reference for those new to devops).**

https://redd.it/t4fozq
@r_devops
Am I too old to get into devops

Just sharing my worries. I just got 38 and I have been doing a Sysadmin job for the past 11 years. That said, I'm done with Sysadmin and I want to get into devops. I am currently learning what I need to do this move. My goal of to do it before I reach 40. My worry tho is that I will get bypassed by younger people in their 20s and early 30s. I just feel so old right now and kind of in a middle life career crisis. Need some advice on this.

https://redd.it/t4m9is
@r_devops
What's your celebration ritual ( dance, sounds, whatever) that you do when you find the solution ?

As me ,right now, James brown dance after a huge terraform refactor that ran on the first run changing 79 resources perfectly.

​

So, What you do when you solve that nasty issue or your code works on the first try?

https://redd.it/t4n7er
@r_devops
Am I being unrealistic with my salary expectations?

I'm currently working as a Principal DevOps engineer in Austin, Tx making $145k/yr. My salary hasn't changed in a couple years, so I have been looking elsewhere. I started as a SysAdmin with a Linux background and moved into the DevOps field for about 8 years now. I've been looking for other jobs in my area and gotten offers at almost every place I've interviewed. However, the salary for these positions seems to be very low for the position being advertised. For instance, I applied for 3 senior roles asking for 7+ years experience and got contingent offers from all of them at $120-$135/k range, no other comp besides your normal benefits and remote 100%. Another was for principal role similar to mine but at $115k/ yr, to include total comp, it would be $165k. These are mid level to established fortune companies I've applied at. I'm planning to apply to other companies in different states as well due to the salaries not meeting my expectations. Am I being unrealistic here, or are these normal salaries in Texas for senior roles?

https://redd.it/t50wph
@r_devops
If you were an experienced DevOps engineer, joining a company with inexperienced people doing the devops, how would you go about "fixing" things?

We use AWS as our cloud provider, and Jenkins as our CI/CD tool.

Most of our infra is a mix of manually spun up stuff from the AWS console, and some stuff created using terraform.

Most of the stuff that was spun up, our devs learnt how to do on the fly. Then at some point we hired a devops contractor to help us transition to using IaC (terraform), but he left after a few months without completing it, so we only had some of the infra represented in terraform. Then after that, we spun up some more stuff manually because we needed to. The jenkins jobs follow a similar story.

Basically, our stuff is a mess. We get by, but we feel this is one area that could probably be optimized. We're looking at hiring an experienced Devops/sysops engineer next quarter, but naturally we're not entirely sure how to put them to work.

If you were the engineer joining such a company, how would you go about rectifying their state of affairs.

https://redd.it/t533gr
@r_devops
Puppet in AKS

Junior DevOps here but.. sadly the only DevOps for the company I work for.

I've been working on a POC for about 6 months and am completely self-taught. I was originally hired into the role with the idea that I would be groomed by my boss, who left for another company about two months into this position.

All this to say that I'm not the most knowledgeable DevOps Engineer out there :)

Couple questions about Puppet specifically:

1) My idea is to have 1 Puppet CA Container in AKS in it's own pod. This container will manage the certs that the Prod/Dev/QA containers(in their own respective pods) use. Agents will connect directly to the Prod/Dev/QA pods. I will use Hiera (haven't played with this yet) to direct to the environments/modules they need. Manage the modules with r10k.

Does this sound.. like good architecture? I'm struggling with the Architecture of this thing.

2) I cannot for the life of me get the Docker Puppet CA Container to share it's cert with the Puppet test container within AKS. I've created a service that has port/targetPort set to 8140. The containers and the service are all on the same label. I've exposed 8140 on both containers. I have no clue on this one. I can get them to talk in Container Instances just fine. The test container has CAENABLED=false and CAHOSTNAME=the hostname of the puppet server. No bueno.

I've set up a mock environment with my home pc's and inside container instances I'm running just fine. But I would love to get this into AKS and then start work on heira/r10k.

Puppet help is hard to find it seems.

https://redd.it/t5akwt
@r_devops
What can I run in a company environment to learn Kubernetes officially?

Let's say I did some initial learning of Kubernetes, build my home lab raspberry cluster etc.

And like to evolve.

Developers do not use Kubernetes at the moment.

​

Want to use Kubernetes for something real in my job.

The best ideas, for now, would be running SonarQube on Kubernetes or some sort of monitoring tool

We are mostly use azure and .net

https://redd.it/t5hcht
@r_devops
Recommended tutorial on setting up a generic web app so i can better understand cicd and app lifecycle

As I'm on my ever evolving devops journey, there are gaps in areas of my knowledge about the app dev process which is why there are some concepts that i dont get. Since i learn by action, can anyone recommend a simple tutorial on setting up a very generic app that has resources of a typical app (e.g., server, front end, back end, database, etc)? I dont want to be a dev but want to learn from going through the steps of setting up an app with typical components.

Im familiar with most of the concepts from ops and support but want to learn more about the dev perspective so i can better grasp cicd, load balancers, containers, etc.

https://redd.it/t5je44
@r_devops
ELI5: What is CI/CD and Why do we need them?

As the title says, as a wanna-be DevOps Engineer, I would like to understand What is CI/CD and Why do we need them?

A few months ago, I asked for GitLab CI/CD example projects and one user commented,

"CI/CD is glue code between a language/runtime (e.g. Java/OpenJDK, JS/Node, Ruby, something in a Docker Container) incl delivery (depending on your kind of artifact, eg. Docker Container, Maven Repository, npm compliant registry) and a deployment mechanism to the environment you want to run in (e.g. Docker, Docker with docker-compose, Ansible, Kubernetes, Kubernetes with Helm, Kubernetes with kustomize, etc.)"

I would like to understand a bit more than that and I appreciate you reading this post.

https://redd.it/t5nufe
@r_devops
Deployment patters in Kubernetes

I am new to Kubernetes and trying to deploy my services. I am curious as to what is the best practice to deploy your replica sets and containers in Kubernetes. Should I just create templates of deployments abs buildconfigs abs have the cicd pipeline inject the end variables in them? Or is there another pattern? Does anyone have any good opinion/reading material on this?

https://redd.it/t5q2xm
@r_devops