Manage Multiple Kubernetes Clusters with ArgoCD

How to manage multiple Kubernetes clusters with Argo CD following the GitOps approach. We will also use vcluster for running multiple virtual clusters and Kyverno: https://piotrminkowski.com/2022/12/09/manage-multiple-kubernetes-clusters-with-argocd/

https://redd.it/zgvejc
@r_devops

Should I attempt this take home assignment or not worth spending an afternoon?

I applied for Devops engineer position and received an email with this assignment to be completed before the interview. Though I can do this in an afternoon, I am just thinking if this is worth it as I have not spoken to anyone in the company yet. What do you guys think?
Is this reasonable to ask candidate to complete all this in a screening round?

## Guidelines - Build a Web App in Azure with status page

1. Use IaC (prefered Terraform) to create the Azure components and database.
2. Deploy this application to the Azure infrastructure that you prefer (can be on Azure VMSS, AKS, Azure App Service, …).
3. Feel free to choose the Database setup that you feel comfortable to work with.
4. You can think about all possible solutions that support the scalability, resiliency and security of the system.
5. Configuration file(s) that define your resouces and network.
6. Two additional infrastructure configurations (in seperate commits) that add scalability and security
7. Add basic logging/monitoring capabilities (in a seperate commit)
8. Apply CI/CD solution (Prefered - Azure DevOps) to support your approach.
9. Basic Documentation (README.md) and architecture diagram
10. Commit often, we like to see small commits that build up to the end result of your test, instead of one final commit with all the code.
11. Do a pull request from the dev
branch to the master
branch.
12. In your pull request, make sure to write about your approach and ideally document your architected solution.
13. Reply to the email thread you are having with our Talent Acquisition department and inform them you have raised a PR so that we can start reviewing your code.
14. One or more of our engineers will then perform a review of the solution and, in eventual later steps of the interview process, there will be a deep dive on the code and the architected solution. This interaction with your peers should provide a better understanding of what working together might look like.

### Bonus Points

If you can provision Self Host agents and benefit of private networking.
If you can document all aspects of your code, in the README and within the code itself.
If you can make this run all in one playbook.
Directory/file structure and the naming convention is a plus

https://redd.it/zgzwhy
@r_devops

Monitor object created in cloud bucket in datadog?

currently trying to set up GCP integration with datadog and then may have to monitor logs to see if a bucket following a naming convention is created daily and alert if not. Would also like to set alert limits for sizes

What is an easy and cheap way to do this? just via parsin logs?

https://redd.it/zgypva
@r_devops

DevOps and DeskSide Support

I’ve been in a DevOps role for over 6 months now and I love it! Recently, my manager approached me saying that I’ll be included in the company’s new DeskSide Support Team to assist on various Tier 2 hardware/software support issues.In the job offer/contract I signed, there is no mention of any desk-side support related responsibilities. Since when a DevOps does desk-side support? Aren’t these two different roles ?!

EDIT: for clarity, the new responsibilities were somewhat imposed to me. No in-writing notifications or any formalities were done… just a casual video conference call to which I never gave my consent to take these responsibilities.

https://redd.it/zh715w
@r_devops

How often do you investigate infra "issues" that are actually developer errors?

Is it common to get asked by other teams to troubleshoot an issue that ends being their own typo or failure to read/comprehend documentation?

How many times do you get asked for xyz when it's already documented?

https://redd.it/zh6xhd
@r_devops

trying to run a simple python script from github in aws ec2

im new to aws but successfully cloned a github repo and installed python onto my ec2 instance.

I made a weatherbot i am trying to deploy that texts me the weather. it has some dependencies. i tried to set up docker but that got complicated. I don't know how to connect through ssh.

I am getting an initialize main error, but the only script I have has the right code init main code and then simply prints "hello world"

has anyone dealt with this error? is there an easier way to deploy a python script with some dependencies that through cloning a github?

https://redd.it/zhez7d
@r_devops

Terraform Deployment testing

What are the best practices in testing terraform code deployments?

Examples: terraform plan, apply, destroy?

I need to check all of them are working properly. And then the services that are created from those are working properly!

https://redd.it/zhgzm5
@r_devops

Is docker/k8s necessary for most situations?

I'm a software engineer working at a start-up now with experience mainly in developing MERN applications and little experience in DevOps. Recently, we've been ready to push our app into a private beta production for a selected group of users, and this is how I manage our infrastructure now,

\- using Terraform as an IaC tool to deploy the apps on AWS with Ansible as a provisioning tool,

\- a single VPC with 1 public subnet & 1 private subnet (not in use at the moment)

\- an EC2 provisioned with Nginx, node & PM2 to host the backend Express app

\- an S3 bucket with CloudFront to host the frontend

\- Github Actions as CI/CD tool to build the app and deploy on AWS

I understand that this is not ideal since there is no scaling solution/protection for EC2 like bastion/no monitoring tool/etc.., so I'm looking for ways to improve the infra and have a redesign since we have time 'till the actual launch, and I'm thinking about dockerizing the backend app and use EKS as a way to scale, but I'm quite lost as I'm trying to understand the needs for Docker + K8s if I can set up an auto-scaling group for the EC2s hosting the app, why would I need Docker. And if I do need these, I still have questions regarding the container image such as where do I install Nginx/where do I put the EKS (public/private subnet), and so on.

Can anyone give me a few suggestions/guidance on where I can learn the basics/more about the topic? Much appreciate everyone's help!

https://redd.it/zh6a8i
@r_devops

What open source tools do you wish existed?

I am doing some platform engineering in tasks where I can overdeliver on, and I realized I'm surprised why the stuff I'm doing doesn't exist in some open source constellation. What are stuff you had to invent at work that you wished already existed?

https://redd.it/zgru9m
@r_devops

How to evolve into/out of DevOps for (real!) dummies

Hi all.

​

I'm currently part of a devops team at my place of employment. I got there by "rising" through the ranks as a regular 1st line technical support person. My only IT credentials are a single sysadmin course (\~6 months fulltime training) which I completed fine, but didn't come with official cert/diploma.

I don't know coding nor security or profound networking. Before you tell me to just pick up these studies in weekends/evenings; I'm dumb as a post and have dyscalculia. I am by FAR not intelligent enough to ever be part of a dev team and I'm extremely aware of this. I honestly have no idea why they even placed me in this team considering I feel like a fish out of water and it shows.

​

I've always been incredibly insecure because all projects I've ever attempted have failed. I feel like I've been set up to fail right now as well, not deliberately mind you, but because of the Peter Principle.

​

Are there any low-level sources of information that do not already require a high level of understanding and are free or low cost?

I know this will never put me on remotely the same level but if it helps to mask my ineptitude for a bit longer, that at least puts a roof over my head and food in my family's mouths for as long as I manage to stretch it.

​

Thank you and all the best.

https://redd.it/zgrns1
@r_devops

need suggestion for certification in - ckad vs aws

Hi peers,

I have been doing java dev from last 16 years. Worked in many latest technologies in last 10 years. Implemented Microservice, event-driven, deployed in ansible, basic python dev, kafka, openshift, kubernetes, ECS, keycloak, s3 etc. From last 2 years I started taking more tasks in devops. Writing ansible, terraform scripts to spin up AKS and configuration done with ansible. Also wrote few automation scripts using python and shell.

All the above is to give intro about my experience. I will be moving to USA soon and thinking of getting into devops than java dev. I also want to get some certifications done before I move. So I need suggestion from who ever is in market which certifications gives better job opportunities.

https://redd.it/zhnx77
@r_devops

How do you manage Self-Service with Terraform?

Do you ever consider providing an interface to your developers to provide self-service infrastructure provisioning? How do you manage this, OR how would you manage to provide this?

One concrete use case for me would be providing an interface where our developers can spin up new environments. I need to keep control and governance of what they provision.

https://redd.it/zho25d
@r_devops

What opensource tools have changed your company culture?

Hi,

Over the last years I've succesfully implemented opensource products that bridged the gap between infra/ops and development.

Tools like jenkins for cicd, awx for cfgmgmt and deployment, netbox for datacenter management and cmdb, elasticsearch for logging and a bit apm, keycloak.

Not only did those tools help the company but it also set in motion a devops culture (not there yet but growing every year) and there is a lot of compatibility between the projects. (Jenkins does api calls to awx, awx uses netbox,....) And every one knows each other's tools (to a certain extent)

I want to expand this even further but am kind of out of ideas.

So what opensource project would you recommend to look into?

Next on my list is Prometheus and grafana. But our current monitoring is largely focused on infra (hp ilo, snmp , monitoring scripts,...) and it looks like Prometheus isn't the right fit here.

https://redd.it/zhplei
@r_devops

Sonarqube LTS upgrade

Hi, We are currently on 8.9.8 LTS sonarqube version and next year we will plan for next LTS upgrade once it is launched officially , my question is can we upgrade directly from 8.9.8 LTS --> to futured launched LTS or we need to follow 8.9.8 LTS--> 8.9.10---> and then futured launched LTS .

https://redd.it/zgopta
@r_devops

zone aware telemetry ingestion

Hello all, I'm currently evaluating replacing my traditional prometheus/jaeger/ELK telemetry stack with the Grafana stack to reduce TCO and simplify day-to-day operations by eliminating elastic.

As part of this, I'm taking a very detailed look at where exactly metrics are scraped and sent to in order to drastically reduce cross-AZ networking costs. A default deployment of the stack in a k8s cluster distributed across three AZs would cause scrapers to potentially reach cross-AZ to scrape metrics, and then again to remote write those metrics to the Grafana Mimir distributor.

Ideally, I'd like to have two Grafana Agents/Distributors (agent=scraper, distributor=receiver) per AZ (HA-per-AZ), and have those Agents only scrape endpoints in the same AZ and only remote write what it scrapes to the Distributors in the same AZ. I'd then like to have three Grafana Ingesters per AZ, and have the Distributors perform sharding/replication across the three Ingesters in its AZ. From there, the ingesters write out data to long-term storage in S3 and we're done.

Has anyone done anything like this before? I'm evaluating Grafana's zone-awareness and it seems to have little documentation about it. I also have zone-aware networking available via my service mesh (Istio) so I'm wondering if I could make this happen without even needing to let Grafana know about it.

https://redd.it/zhqi6m
@r_devops

Tekton + ArgoCD is a thing? If so, why?

Hey friends -

tl;dr - I wasn't aware that Tekton and ArgoCD are used together (but the internet says they can be). Is that an old pattern, or is that still a good idea? If so, why? Can't Tekton do everything ArgoCD also does? Should I use both, or only pick one? (my entire infra + platform is built using only 2 things: terragrunt & helmfile)

QUESTION:

When we started - 10 months ago - I picked ArgoCD for the continuous delivery/deployment of our PLATFORM (not our infra...although I tried out CrossPlane with Argo, and our CTO was too scared to commit).

For CI, and for CD of the infra (only), I thought I would use BitBucket pipelines and call it a day.

Then I did some Googling today, and I noticed that people mentioned Tekton as "the future," and how it's excellent --> and only getting better.

So then, I poked at Tekton's documentation, and some of their getting started content...and I was trying to understand whether Tekton's primary use-case is to replace my BitBucket pipelines, or if it's also supposed to replace ArgoCD as well.

Googling for that led me to a bunch of articles that discuss combining Tekton w/ ArgoCD, but those are typically a year old (or older). I understand that Tekton is composable, and you can CHOOSE to use it in a certain way...but that's not the answer I'm after.

I just want to understand if I should go into Tekton wholesale, or if - in practice - it is STILL better to combine Tekton w/ ArgoCD, or is it the common wisdom to replace ArgoCD with Tekton.

Technology team members outside of our Ops team (our ops team covers devops + platform eng + sre) are used to using (old) Jenkins. They're not going to learn anything on the CLI in order to "self-service" the infrastructure or platform...so Tekton and ArgoCD both having good/usable UI for them is a must.

Context:

I've been heads down for 10 months, rebuilding/re-platforming/rearchitecting a whole-ass production infra. We deal with some of the biggest names in the M&E vertical, and the stakes are high.

It's taken me a while, but I'm finally nearing the official certification/delivery of the entire project, and I'm buttoning up our brand-new CI/CD.

I planned to use BitBucket Pipelines to:
- lint and run tests
- apply terragrunt (to deploy + configure infra & generate dynamically templated output values for consumption by helm)

Then I would:
- use ArgoCD for "last-mile" continuous deployment of everything else on k8s.

Everything infra-related starts and ENDS with Terragrunt.

We work off of the standard concept of "environments" and "tenants." An environment is an EKS cluster + everything it requires to run our code...RDS, SES, ACM, S3, SQS, etc.). A tenant is a logical grouping of services that belong together, or belong to each other. Tenants can be different "customers" on a shared cluster, or they can be a "shared-services" tenant (running cert-manager, amazon load-balancer controller, external-dns, metrics-server, etc).

Some "environments" we have are also "single-tenant," so I had to think of a framework that allowed us to have/use both, generically.

Our terragrunt also has a special module responsible for generating values.yaml and secrets.yaml files that contain dynamic outputs from resources it creates (cluster endpoint, security-group IDs, role-arns, RDS username/password, etc).

Basic workflow is dead simple:
1. Copy + paste an infrastructure "template" (a folder with just terragrunt in it) to ==> infra/config/*account-name*/*account-region*/*environment-name*
2. Edit environment.hcl in the new folder and change just 1 variable env_name = my-new-env
3. ✅ terragrunt run-all --terragrunt-non-interactive validate
4. ✅ terragrunt run-all --terragrunt-non-interactive plan
5. ✅ terragrunt run-all --terragrunt-non-interactive apply
6. ...wait 35 minutes for all of the magic to happen (our environments are extensive and complex, so it takes a little while to bake and get set up)
7. When all the infra is baked, terragrunt also produces and drops off all

of the env-values.yaml and env-secrets.yaml files that the environment needs into ==> helm/deploy/*account-name*/*account-region*/*environment-name*/
8. And terragrunt will also drop off a tenant-values.yaml and a tenant-secrets.yaml PER tenant, into: helm/deploy/*account-name*/*account-region*/*environment-name*/*tenant-name*/
9. Finally, terragrunt also generates the argo.json (or argo.yaml) that is used by the argoCD ApplicationSet's Git Generator, in each of the tenant folders.
10. Tenant folders receive a helmfile.yaml, and every tenant's helmfile consumes tenant-values.yaml, tenant-secrets.yaml, env-values.yaml, and env-secrets.yaml
11. Git add, git commit, git push (the new tenants and all the new values + secrets)
12. ArgoCD does its magic

Thanks for any help or suggestions!!

https://redd.it/zhs6ks
@r_devops

Implementing Task Control | Ansible Tutorial for Beginners

https://youtu.be/qagBSArXJno

https://redd.it/zhvyv7
@r_devops

Monitoring Network Performance on thousands of devices across WAN

We have thousands of devices (it's a LattePanda running Ubuntu 20.04) across the country that we need to monitor (specifically network performance like ping, wireless signal strength, speedtests). I've been playing around with Prometheus, Zabbix, Nagios and looking at a bunch of other tools but keep getting lost jumping into these rabbit holes. Can anyone recommend an open-source or proprietary solution for something like this?

https://redd.it/zhutr0
@r_devops

What do you think of “Platform Engineering”?

I have read a bit on Platform Engineering. I understand that a company whose focus is to move to the cloud would want to have some team to take care of the architecture (including tooling, automation and CI/CD processes) and that'd nowadays be a Platform Engineering team.

Those are all well-known topics to someone who calls himself a DevOps-Engineer. So, to build a Platform Engineering team, you'd basically hire DevOps-Engineers.

In the end, Platform-Engineering does seem to be DevOps practices (that have been done in the past years) focused on Cloud, automation and good integration.

Do I miss anything? Do you think there is more to it?

https://redd.it/zhoso6
@r_devops