of the env-values.yaml and env-secrets.yaml files that the environment needs into ==> helm/deploy/*account-name*/*account-region*/*environment-name*/
8. And terragrunt will also drop off a tenant-values.yaml and a tenant-secrets.yaml PER tenant, into: helm/deploy/*account-name*/*account-region*/*environment-name*/*tenant-name*/
9. Finally, terragrunt also generates the argo.json (or argo.yaml) that is used by the argoCD ApplicationSet's Git Generator, in each of the tenant folders.
10. Tenant folders receive a helmfile.yaml, and every tenant's helmfile consumes tenant-values.yaml, tenant-secrets.yaml, env-values.yaml, and env-secrets.yaml
11. Git add, git commit, git push (the new tenants and all the new values + secrets)
12. ArgoCD does its magic

Thanks for any help or suggestions!!

https://redd.it/zhs6ks
@r_devops

Implementing Task Control | Ansible Tutorial for Beginners

https://youtu.be/qagBSArXJno

https://redd.it/zhvyv7
@r_devops

Monitoring Network Performance on thousands of devices across WAN

We have thousands of devices (it's a LattePanda running Ubuntu 20.04) across the country that we need to monitor (specifically network performance like ping, wireless signal strength, speedtests). I've been playing around with Prometheus, Zabbix, Nagios and looking at a bunch of other tools but keep getting lost jumping into these rabbit holes. Can anyone recommend an open-source or proprietary solution for something like this?

https://redd.it/zhutr0
@r_devops

What do you think of “Platform Engineering”?

I have read a bit on Platform Engineering. I understand that a company whose focus is to move to the cloud would want to have some team to take care of the architecture (including tooling, automation and CI/CD processes) and that'd nowadays be a Platform Engineering team.

Those are all well-known topics to someone who calls himself a DevOps-Engineer. So, to build a Platform Engineering team, you'd basically hire DevOps-Engineers.

In the end, Platform-Engineering does seem to be DevOps practices (that have been done in the past years) focused on Cloud, automation and good integration.

Do I miss anything? Do you think there is more to it?

https://redd.it/zhoso6
@r_devops

Help.. 36hrs in...

All,

Going slowly insane here, one of my team has wiped out our GKE cluster with Gitlab on it. Our restore from back has 'ish' worked but we cant get logged into Gitlab as we are in an infinite loop of redirects for Oauth, its the last hurdle in our road to recover. Someone had pushed manual changes to the server / cluster at somepoint before my current team took over.

​

We have IAP that then points to Gitlab on GKE deployed via HELM which uses google Oauth for authentication. Ingress controller is kubes nginx.

Tested -> disabling Oauth by standing up freash Gitlab in new namespace -> result we get through IAP auth and can touch the empty Gitlab login page.

Tested-> Default deploy & restore -> we login via IAP again fine but then sit on the select google account page.

Looking at logs:

I can see we get authenticated BUT immediately get a 302 redirect to the call back..

I've seen some `Request origin not allowed: https://<myServer>/` errors too.

Hunch..

I think i need to make a cfgmap change to our nginx ingress for maybe proxy pass or prevent some caching of cookies or something that is causing us to immediately redirect to the auth page again.

&#x200B;

Any ideas are appreciated.

https://redd.it/zi3q3g
@r_devops

Where do you test the stuff you are about to implement?

I'm in my first DevOps role and currently we only have a single environment (will have 3 in the upcoming moths). If I want to try things out (e.g. play around with the Gateway, Ingress, change or adjust CI/CD etc.), it's just a matter of time until something brakes.

Running only a single env. means that the devs possibly can't work during this time (assuming I broke something)

But running 3 env. (dev, staging, prod) doesn't solve this problem either. If I test on the dev. environment and it breaks, the devs won't be able to work either, since they implement their stuff on dev env. as well and our pipelines for deployment rely on the dev env. as the initial instance.

Any recommendations?

https://redd.it/zhtqi6
@r_devops

Cachet: a little bit abandoned, but still a decent monitoring page

Just did a write up on Cachet (code) -- would love to know what other people are using

I find that today it still has the best combination of features for open source stuff -- there are other alternatives like Vigil which are good but too basic, and then there's stuff that's even less featureful.

Anything people are using for monitoring/basic uptime?

https://redd.it/zibbgh
@r_devops

Moving a firewall service to AWS.

Suppose I have a L4 firewall service, would it be logical/possible to move this service to AWS? I have some sort of design/implementation in mind but I would appreciate any feedback on whether there is a flaw in the architecture or a better alternative?

The use case I have in mind is that:

**1. The users would be able to use the firewall service without having the firewall consume the local devices hardware resources.**

The firewall would be deployed on EC2 instances on AWS and through a web-based interface, the user would be able to access the firewall console.

They would still need to setup and configure the firewall initially, but in this case, especially for corporate use cases, the firewall would use multiple EC2 instances based on the number of corporate users.

**2. Latency**

The users would have the ability to connect to different deployed instances in different regions, for example, if the user is at USA, they would connect to one of the EC2 instances in the same region and availability zone. If they are in Asia, they would choose their region and availability zone based on that. Of course the firewall would not have an instance in every single AZ, but it would still be much more accessible and reduce latency?

**3. Scalability and Cost-Efficiency**

Using AWS services such as EC2 Auto Scale and Load Balancer, would it be more cost-efficient.

&#x200B;

**Some AWS services I have in mind to use are:**

EC2 \- Hosts the firewall, storage, and the web-server for connection.

EBS - Storage that will be attached to each EC2 instance that hosts the firewall.

VPC - For configuring the private network that the system will be on along with security measures.

IAM Identity Center - To connect the existing credentials of the users for authentocation so that the users don't need to create AWS accounts.

EC2 Auto Scale - For scalibility of the EC2 instances in each AZ.

ELB - For load balancing between the EC2 instances in each AZ.

I also am looking into Elastic Container Service (ECS), Route 53, CloudWatch, CloudTrail and how they would be useful to this architecture.

* Is this something possible and beneficial to implement for a firewall?
* Are there any major flaws or anything missing in the implementation?
* Would this kind of implementation limit the usage of the firewall at all?


I'm a beginner and this is just a proposal based on preliminary research. I would appreciate any sort of feedback.

https://redd.it/zi76o5
@r_devops

How would you scale self hosted Azure DevOps agents or other CI runners? Would you use Kubernetes?

This question refers to Azure DevOps but can also work with other CIs.

My original and current approach to automatically scale Azure DevOps (ADO) agents is using KEDA with ADO but there is a problem with this solution: I cannot run docker containers while running my pipelines because it's not recommended to use DIND nor the docker socket is not available in k8s anymore, therefore I cannot run docker containers inside of the agent's pod.

I mainly run docker containers in my pipelines for unit testing, integration testing and some other offline jobs within the CI.

I don't have problems building images as I'm not dependent on docker anymore but instead I use buildx with buildkit which works great.

I tried using "podman run" instead of "docker run" but there's an obvious difference in terms of speed - docker is faster (I assume it's because of its daemon although not sure about it).

Using VMSS auto scaling might be an option but I rather scale based on the jobs queue and not based on resources i.e using a VM per job. Maybe that would be too expensive?

How would you do it/did?

https://redd.it/zhymhc
@r_devops

Is everyone technically a web/mobile developer?

Besides things like embedded programming and computer vision, when you look at software engineering jobs at places like Google, Amazon, Meta, Paypal, etc. are there all just web/mobile development jobs?

Technically, services like Gmail, Uber, and Netflix all have a client-facing frontend and a backend. And if you get a job there, you usually work on either the frontend or backend or something related, right? So doesn't that mean that pretty much everyone is a web/mobile developer?

https://redd.it/zii2j0
@r_devops

Ansible Variables | Ansible Tutorial for Beginners

https://youtu.be/X41WY4Fn5Yc

https://redd.it/zhvtm4
@r_devops

how to "bind" docker container to subdomain

Hello, I'm experimenting a little with devops stuff to not get out of touch too much with tech. I have IT/CS degree, but only work in project coordination and management since. So I know the theory and principles but I have not much hands-on experience.

So, what I have now is:

1. a small virtual server with a dedicated IP
1. I have root access via KVM and SSH, got SSH access with SSH-key.
2. in there I have a docker container with Jupyterlab, which is secured with login and password. I can reach this Jupyterlab via the IP:custom_port of the server.
2. a domain, I can create subdomains

What I would like to have is a secure Jupyterlab for my private use, always available from any device:

1. have the container continue running even if I logout of the session
2. "bind" the Jupyterlab container with a subdomain, like jupyter.mydomain.com.
3. secure the connection with a SSL certificate (letsencrypt?), and have "something" automatically renew the certificate.

What is the pragmatic way to achieve those 3 goals? Thanks in advance.

https://redd.it/zhtomo
@r_devops

Ansible Playbooks

https://youtu.be/SZZJ9vtzy-M

https://redd.it/zhvlsd
@r_devops

How can project management help you?

I'm between jobs IT project manager with a strong background in infrastructure and client services. I am in the interviewing process where the meat of my position would be organizing and submitting for a dev-ops team. The PM process will also be undergoing changes during this time. Sector is retail for a direct to retail producer. Professionally, outside of helping stand up facilities, I have had some minimal cross training on handling the application/digitilization/production operations side of things. Tomorrow is a panel interview with some of the dev-ops team members.

Personally, not professionally, and mostly beginner level stuff. My first computer was a TI99 4/a, qbasic i believe, made bojangles dance. No computer until early 2000s. I've been a Linux (gentoo mainly) user at home for around 20 years. Done cross dev android stuff. Arduino (esp specifically). Virtual desktop stuff for ages.

So how could I help your team? Please ask me some questions you would ask if you were on the hiring panel. How would you feel I may be stepping on your toes?

Lastly, something noticed missing is value calculations. If I land the job, any tips to get the dev-ops team into a habit of quantifiable testing so I can present more avenues of business cost saving back up the chain (thus making us look like less of a money dump).

https://redd.it/zir25s
@r_devops

I am building an open-source Self Service Infrastructure platform

I am currently building an open-source Self Service Infrastructure platform to help DevOps engineers to serve and keep control of resources created and used by engineering teams at a large scale. My goal is to help DevOps teams to respond to the increasing number of requests Engineering teams have for them. You know, those kinds of requests:

- "I need to have a testing environment to run X update of my application? Can you provide such environment?"
- "Can you create a new benchmark environment, I need to run some performance tests."
- "We are launching a new project, can you please deploy this application and provide a Postgres database?"

Instead of being in a "reactive mode", DevOps engineers can configure the platform to indicate what services Software Engineers can use and for how long. Then, no more wasted IT resources and $$ in the cloud. (Yes, this platform will mainly focus to run on AWS, GCP, Azure... But ultimately run anywhere)

I am building this platform upon the 6 following pillars:

## Speed

How fast DevOps engineers can provide a self-service infrastructure platform to their engineering team. They can install it on their existing infrastructure (Kubernetes needed) in a few minutes and provide services to their Engineering teams.

## Control

The main complexity of such a platform is to keep full control of what’s going on on the infrastructure and application side. Built with the “white box” philosophy. Everything should be auditable. Cost control is also a very important aspect.

## Security

Security is an important aspect of DevOps since they are responsible for everything that runs on the infrastructure. They need to know exactly what happens at any time (Audit/Accounting), and they need to manage access to resources.

## Governance

DevOps engineers need to control what engineering teams are doing and see at any point in time what was provisioned and by who.

## Flexibility

DevOps engineers need to be able to re-use what they have done so far with their scripts (Terraform, Bash, Python…). Integrations with the most popular CI/CD system are built-in. Plus, an API, Web UI, CLI, and a Terraform Provider are provided to control the solution.

## Developer Experience

Needless to say that the platform must be usable by any developer. The idea here is to provide a delightful experience with a "Blueprint" system configured and maintainable by the DevOps engineers. Then, developers can use those "Blueprints" and just configure what they need.

---

I expect to launch something in January 2023, and I am looking for some DevOps beta testers. Are you in? My ideal candidates would be DevOps engineers working in an environment where they are highly solicited by Engineering teams.

Feel free to ask all your questions if some parts are not clear.

https://redd.it/zisxr0
@r_devops

sysadmin to DevOps

I am currently a systems administrator, and I think I want to get into DevOps. I write a lot of PowerShell, and know some Python. I have strong networking experience, and I also know my way around Linux. Do I need to work on myself before I apply, or do you think I could get through an interview?

https://redd.it/zivl4x
@r_devops

Currently IT support, want to transition into devops (Azure)

Hi guys,

I am currently an IT support engineer and have been for over a year. We deal with Azure. So I deal with AD, Intune, office365 , the usual.. I want to get into devops as I have a few friends doing that. And it's something that interests me.

I am currently learning Python Fundamentals and studying for my AZ 900 exam. I know it may be useless but I wanted to be grounded in the foundations.

I was told to continue learning Python, learn Linux, continue with further Azure certs. Add my projects to github and learn Jenkins or terraform.

Am I missing anything else? I have tried to speak to my manager about whether I can shadow someone from the operations or development team with ANYTHING and I basically got empty promises. I also ask the operations manager for advice on what I'm learning and he's extremely helpful.

But as far as I know I don't see myself progressing in the same company. I'm extremely desired at IT support engineer and I'm making my managers job easier. I've tried before. So it seems that when I'm grounded in all of these i would have to apply elsewhere.


What would you guys advise me on what to learn and how to do it? Thank you!

https://redd.it/zizrmo
@r_devops

Well-architected framework. How well should a DevOps engineer know it?

Hi I’m a DevOps engineer that does pipeline, software and platform engineering. I’m studying for the AWS exams (all of them) but taking my time to learn as much as possible that actually makes me better practically at DevOps and AWS.

Does anyone have any opinions on how important/useful it is to be familiar with the well-architected framework for a senior engineer?

It all looks like good stuff to know but I’m trying to evaluate how much time to spend on it. I don’t plan on becoming an architect at the moment.

https://redd.it/zj3box
@r_devops

How would you define "DevOps"?

I'm interested in hearing what people think defines DevOps.

If this has been addressed before in here I'd appreciate a link to the post as opposed to "we've been over this before" or something similar

I've been doing this job for 22 years now and I think it's a giant Venn diagram that differs by industry and company, as well as who the DevOps engineers report into, but I want to hear others' take on it, please, and thank you

https://redd.it/zj63ge
@r_devops

Week 2 Devops

Hi guys it’s my second week as a junior devops engineer and this is what I did:

> more Prometheus/monitoring - this is nearly fully set up and this phase of the project is pretty much done.

> I’m learning a bit more about kubernetes, and learning some terraform type stuff. I’ve never done terraform/kubernetes before but it’s nice to learn since this project is currently assigned to the other devops engineer we’ve been sharing knowledge on my monitoring project and his kubernetes one

> I had quite a few HR type meetings this week and I will have more next week.

> With the monitoring stuff it’s mostly finding out who needs to be notified and when they need to be called in/ integrating opsgenie - if anyone knows any good alternatives to opsgenie I’d also be ears.

> We currently are using clickup over Jira but all our stuff is in bitbucket so we’re currently moving towards GitHub and using GitHub actions. So I’ve been looking at the smoothest way to do this.

> Overall I think it was a good week and I’m enjoying my time. I mentioned I was interested in learning networks and getting more into the networks side of stuff, so I’ll be getting involved in a few projects there in the near future. Overall I’ve been told to just enjoy these last little relaxed bits before Christmas as in the new year the heavy lifting starts!

https://redd.it/zj7zqp
@r_devops