Is docker/k8s necessary for most situations?

I'm a software engineer working at a start-up now with experience mainly in developing MERN applications and little experience in DevOps. Recently, we've been ready to push our app into a private beta production for a selected group of users, and this is how I manage our infrastructure now,

\- using Terraform as an IaC tool to deploy the apps on AWS with Ansible as a provisioning tool,

\- a single VPC with 1 public subnet & 1 private subnet (not in use at the moment)

\- an EC2 provisioned with Nginx, node & PM2 to host the backend Express app

\- an S3 bucket with CloudFront to host the frontend

\- Github Actions as CI/CD tool to build the app and deploy on AWS

I understand that this is not ideal since there is no scaling solution/protection for EC2 like bastion/no monitoring tool/etc.., so I'm looking for ways to improve the infra and have a redesign since we have time 'till the actual launch, and I'm thinking about dockerizing the backend app and use EKS as a way to scale, but I'm quite lost as I'm trying to understand the needs for Docker + K8s if I can set up an auto-scaling group for the EC2s hosting the app, why would I need Docker. And if I do need these, I still have questions regarding the container image such as where do I install Nginx/where do I put the EKS (public/private subnet), and so on.

Can anyone give me a few suggestions/guidance on where I can learn the basics/more about the topic? Much appreciate everyone's help!

https://redd.it/zh6a8i
@r_devops

What open source tools do you wish existed?

I am doing some platform engineering in tasks where I can overdeliver on, and I realized I'm surprised why the stuff I'm doing doesn't exist in some open source constellation. What are stuff you had to invent at work that you wished already existed?

https://redd.it/zgru9m
@r_devops

How to evolve into/out of DevOps for (real!) dummies

Hi all.

​

I'm currently part of a devops team at my place of employment. I got there by "rising" through the ranks as a regular 1st line technical support person. My only IT credentials are a single sysadmin course (\~6 months fulltime training) which I completed fine, but didn't come with official cert/diploma.

I don't know coding nor security or profound networking. Before you tell me to just pick up these studies in weekends/evenings; I'm dumb as a post and have dyscalculia. I am by FAR not intelligent enough to ever be part of a dev team and I'm extremely aware of this. I honestly have no idea why they even placed me in this team considering I feel like a fish out of water and it shows.

​

I've always been incredibly insecure because all projects I've ever attempted have failed. I feel like I've been set up to fail right now as well, not deliberately mind you, but because of the Peter Principle.

​

Are there any low-level sources of information that do not already require a high level of understanding and are free or low cost?

I know this will never put me on remotely the same level but if it helps to mask my ineptitude for a bit longer, that at least puts a roof over my head and food in my family's mouths for as long as I manage to stretch it.

​

Thank you and all the best.

https://redd.it/zgrns1
@r_devops

need suggestion for certification in - ckad vs aws

Hi peers,

I have been doing java dev from last 16 years. Worked in many latest technologies in last 10 years. Implemented Microservice, event-driven, deployed in ansible, basic python dev, kafka, openshift, kubernetes, ECS, keycloak, s3 etc. From last 2 years I started taking more tasks in devops. Writing ansible, terraform scripts to spin up AKS and configuration done with ansible. Also wrote few automation scripts using python and shell.

All the above is to give intro about my experience. I will be moving to USA soon and thinking of getting into devops than java dev. I also want to get some certifications done before I move. So I need suggestion from who ever is in market which certifications gives better job opportunities.

https://redd.it/zhnx77
@r_devops

How do you manage Self-Service with Terraform?

Do you ever consider providing an interface to your developers to provide self-service infrastructure provisioning? How do you manage this, OR how would you manage to provide this?

One concrete use case for me would be providing an interface where our developers can spin up new environments. I need to keep control and governance of what they provision.

https://redd.it/zho25d
@r_devops

What opensource tools have changed your company culture?

Hi,

Over the last years I've succesfully implemented opensource products that bridged the gap between infra/ops and development.

Tools like jenkins for cicd, awx for cfgmgmt and deployment, netbox for datacenter management and cmdb, elasticsearch for logging and a bit apm, keycloak.

Not only did those tools help the company but it also set in motion a devops culture (not there yet but growing every year) and there is a lot of compatibility between the projects. (Jenkins does api calls to awx, awx uses netbox,....) And every one knows each other's tools (to a certain extent)

I want to expand this even further but am kind of out of ideas.

So what opensource project would you recommend to look into?

Next on my list is Prometheus and grafana. But our current monitoring is largely focused on infra (hp ilo, snmp , monitoring scripts,...) and it looks like Prometheus isn't the right fit here.

https://redd.it/zhplei
@r_devops

Sonarqube LTS upgrade

Hi, We are currently on 8.9.8 LTS sonarqube version and next year we will plan for next LTS upgrade once it is launched officially , my question is can we upgrade directly from 8.9.8 LTS --> to futured launched LTS or we need to follow 8.9.8 LTS--> 8.9.10---> and then futured launched LTS .

https://redd.it/zgopta
@r_devops

zone aware telemetry ingestion

Hello all, I'm currently evaluating replacing my traditional prometheus/jaeger/ELK telemetry stack with the Grafana stack to reduce TCO and simplify day-to-day operations by eliminating elastic.

As part of this, I'm taking a very detailed look at where exactly metrics are scraped and sent to in order to drastically reduce cross-AZ networking costs. A default deployment of the stack in a k8s cluster distributed across three AZs would cause scrapers to potentially reach cross-AZ to scrape metrics, and then again to remote write those metrics to the Grafana Mimir distributor.

Ideally, I'd like to have two Grafana Agents/Distributors (agent=scraper, distributor=receiver) per AZ (HA-per-AZ), and have those Agents only scrape endpoints in the same AZ and only remote write what it scrapes to the Distributors in the same AZ. I'd then like to have three Grafana Ingesters per AZ, and have the Distributors perform sharding/replication across the three Ingesters in its AZ. From there, the ingesters write out data to long-term storage in S3 and we're done.

Has anyone done anything like this before? I'm evaluating Grafana's zone-awareness and it seems to have little documentation about it. I also have zone-aware networking available via my service mesh (Istio) so I'm wondering if I could make this happen without even needing to let Grafana know about it.

https://redd.it/zhqi6m
@r_devops

Tekton + ArgoCD is a thing? If so, why?

Hey friends -

tl;dr - I wasn't aware that Tekton and ArgoCD are used together (but the internet says they can be). Is that an old pattern, or is that still a good idea? If so, why? Can't Tekton do everything ArgoCD also does? Should I use both, or only pick one? (my entire infra + platform is built using only 2 things: terragrunt & helmfile)

QUESTION:

When we started - 10 months ago - I picked ArgoCD for the continuous delivery/deployment of our PLATFORM (not our infra...although I tried out CrossPlane with Argo, and our CTO was too scared to commit).

For CI, and for CD of the infra (only), I thought I would use BitBucket pipelines and call it a day.

Then I did some Googling today, and I noticed that people mentioned Tekton as "the future," and how it's excellent --> and only getting better.

So then, I poked at Tekton's documentation, and some of their getting started content...and I was trying to understand whether Tekton's primary use-case is to replace my BitBucket pipelines, or if it's also supposed to replace ArgoCD as well.

Googling for that led me to a bunch of articles that discuss combining Tekton w/ ArgoCD, but those are typically a year old (or older). I understand that Tekton is composable, and you can CHOOSE to use it in a certain way...but that's not the answer I'm after.

I just want to understand if I should go into Tekton wholesale, or if - in practice - it is STILL better to combine Tekton w/ ArgoCD, or is it the common wisdom to replace ArgoCD with Tekton.

Technology team members outside of our Ops team (our ops team covers devops + platform eng + sre) are used to using (old) Jenkins. They're not going to learn anything on the CLI in order to "self-service" the infrastructure or platform...so Tekton and ArgoCD both having good/usable UI for them is a must.

Context:

I've been heads down for 10 months, rebuilding/re-platforming/rearchitecting a whole-ass production infra. We deal with some of the biggest names in the M&E vertical, and the stakes are high.

It's taken me a while, but I'm finally nearing the official certification/delivery of the entire project, and I'm buttoning up our brand-new CI/CD.

I planned to use BitBucket Pipelines to:
- lint and run tests
- apply terragrunt (to deploy + configure infra & generate dynamically templated output values for consumption by helm)

Then I would:
- use ArgoCD for "last-mile" continuous deployment of everything else on k8s.

Everything infra-related starts and ENDS with Terragrunt.

We work off of the standard concept of "environments" and "tenants." An environment is an EKS cluster + everything it requires to run our code...RDS, SES, ACM, S3, SQS, etc.). A tenant is a logical grouping of services that belong together, or belong to each other. Tenants can be different "customers" on a shared cluster, or they can be a "shared-services" tenant (running cert-manager, amazon load-balancer controller, external-dns, metrics-server, etc).

Some "environments" we have are also "single-tenant," so I had to think of a framework that allowed us to have/use both, generically.

Our terragrunt also has a special module responsible for generating values.yaml and secrets.yaml files that contain dynamic outputs from resources it creates (cluster endpoint, security-group IDs, role-arns, RDS username/password, etc).

Basic workflow is dead simple:
1. Copy + paste an infrastructure "template" (a folder with just terragrunt in it) to ==> infra/config/*account-name*/*account-region*/*environment-name*
2. Edit environment.hcl in the new folder and change just 1 variable env_name = my-new-env
3. ✅ terragrunt run-all --terragrunt-non-interactive validate
4. ✅ terragrunt run-all --terragrunt-non-interactive plan
5. ✅ terragrunt run-all --terragrunt-non-interactive apply
6. ...wait 35 minutes for all of the magic to happen (our environments are extensive and complex, so it takes a little while to bake and get set up)
7. When all the infra is baked, terragrunt also produces and drops off all

of the env-values.yaml and env-secrets.yaml files that the environment needs into ==> helm/deploy/*account-name*/*account-region*/*environment-name*/
8. And terragrunt will also drop off a tenant-values.yaml and a tenant-secrets.yaml PER tenant, into: helm/deploy/*account-name*/*account-region*/*environment-name*/*tenant-name*/
9. Finally, terragrunt also generates the argo.json (or argo.yaml) that is used by the argoCD ApplicationSet's Git Generator, in each of the tenant folders.
10. Tenant folders receive a helmfile.yaml, and every tenant's helmfile consumes tenant-values.yaml, tenant-secrets.yaml, env-values.yaml, and env-secrets.yaml
11. Git add, git commit, git push (the new tenants and all the new values + secrets)
12. ArgoCD does its magic

Thanks for any help or suggestions!!

https://redd.it/zhs6ks
@r_devops

Implementing Task Control | Ansible Tutorial for Beginners

https://youtu.be/qagBSArXJno

https://redd.it/zhvyv7
@r_devops

Monitoring Network Performance on thousands of devices across WAN

We have thousands of devices (it's a LattePanda running Ubuntu 20.04) across the country that we need to monitor (specifically network performance like ping, wireless signal strength, speedtests). I've been playing around with Prometheus, Zabbix, Nagios and looking at a bunch of other tools but keep getting lost jumping into these rabbit holes. Can anyone recommend an open-source or proprietary solution for something like this?

https://redd.it/zhutr0
@r_devops

What do you think of “Platform Engineering”?

I have read a bit on Platform Engineering. I understand that a company whose focus is to move to the cloud would want to have some team to take care of the architecture (including tooling, automation and CI/CD processes) and that'd nowadays be a Platform Engineering team.

Those are all well-known topics to someone who calls himself a DevOps-Engineer. So, to build a Platform Engineering team, you'd basically hire DevOps-Engineers.

In the end, Platform-Engineering does seem to be DevOps practices (that have been done in the past years) focused on Cloud, automation and good integration.

Do I miss anything? Do you think there is more to it?

https://redd.it/zhoso6
@r_devops

Help.. 36hrs in...

All,

Going slowly insane here, one of my team has wiped out our GKE cluster with Gitlab on it. Our restore from back has 'ish' worked but we cant get logged into Gitlab as we are in an infinite loop of redirects for Oauth, its the last hurdle in our road to recover. Someone had pushed manual changes to the server / cluster at somepoint before my current team took over.

​

We have IAP that then points to Gitlab on GKE deployed via HELM which uses google Oauth for authentication. Ingress controller is kubes nginx.

Tested -> disabling Oauth by standing up freash Gitlab in new namespace -> result we get through IAP auth and can touch the empty Gitlab login page.

Tested-> Default deploy & restore -> we login via IAP again fine but then sit on the select google account page.

Looking at logs:

I can see we get authenticated BUT immediately get a 302 redirect to the call back..

I've seen some `Request origin not allowed: https://<myServer>/` errors too.

Hunch..

I think i need to make a cfgmap change to our nginx ingress for maybe proxy pass or prevent some caching of cookies or something that is causing us to immediately redirect to the auth page again.

&#x200B;

Any ideas are appreciated.

https://redd.it/zi3q3g
@r_devops

Where do you test the stuff you are about to implement?

I'm in my first DevOps role and currently we only have a single environment (will have 3 in the upcoming moths). If I want to try things out (e.g. play around with the Gateway, Ingress, change or adjust CI/CD etc.), it's just a matter of time until something brakes.

Running only a single env. means that the devs possibly can't work during this time (assuming I broke something)

But running 3 env. (dev, staging, prod) doesn't solve this problem either. If I test on the dev. environment and it breaks, the devs won't be able to work either, since they implement their stuff on dev env. as well and our pipelines for deployment rely on the dev env. as the initial instance.

Any recommendations?

https://redd.it/zhtqi6
@r_devops

Cachet: a little bit abandoned, but still a decent monitoring page

Just did a write up on Cachet (code) -- would love to know what other people are using

I find that today it still has the best combination of features for open source stuff -- there are other alternatives like Vigil which are good but too basic, and then there's stuff that's even less featureful.

Anything people are using for monitoring/basic uptime?

https://redd.it/zibbgh
@r_devops

Moving a firewall service to AWS.

Suppose I have a L4 firewall service, would it be logical/possible to move this service to AWS? I have some sort of design/implementation in mind but I would appreciate any feedback on whether there is a flaw in the architecture or a better alternative?

The use case I have in mind is that:

**1. The users would be able to use the firewall service without having the firewall consume the local devices hardware resources.**

The firewall would be deployed on EC2 instances on AWS and through a web-based interface, the user would be able to access the firewall console.

They would still need to setup and configure the firewall initially, but in this case, especially for corporate use cases, the firewall would use multiple EC2 instances based on the number of corporate users.

**2. Latency**

The users would have the ability to connect to different deployed instances in different regions, for example, if the user is at USA, they would connect to one of the EC2 instances in the same region and availability zone. If they are in Asia, they would choose their region and availability zone based on that. Of course the firewall would not have an instance in every single AZ, but it would still be much more accessible and reduce latency?

**3. Scalability and Cost-Efficiency**

Using AWS services such as EC2 Auto Scale and Load Balancer, would it be more cost-efficient.

&#x200B;

**Some AWS services I have in mind to use are:**

EC2 \- Hosts the firewall, storage, and the web-server for connection.

EBS - Storage that will be attached to each EC2 instance that hosts the firewall.

VPC - For configuring the private network that the system will be on along with security measures.

IAM Identity Center - To connect the existing credentials of the users for authentocation so that the users don't need to create AWS accounts.

EC2 Auto Scale - For scalibility of the EC2 instances in each AZ.

ELB - For load balancing between the EC2 instances in each AZ.

I also am looking into Elastic Container Service (ECS), Route 53, CloudWatch, CloudTrail and how they would be useful to this architecture.

* Is this something possible and beneficial to implement for a firewall?
* Are there any major flaws or anything missing in the implementation?
* Would this kind of implementation limit the usage of the firewall at all?


I'm a beginner and this is just a proposal based on preliminary research. I would appreciate any sort of feedback.

https://redd.it/zi76o5
@r_devops

How would you scale self hosted Azure DevOps agents or other CI runners? Would you use Kubernetes?

This question refers to Azure DevOps but can also work with other CIs.

My original and current approach to automatically scale Azure DevOps (ADO) agents is using KEDA with ADO but there is a problem with this solution: I cannot run docker containers while running my pipelines because it's not recommended to use DIND nor the docker socket is not available in k8s anymore, therefore I cannot run docker containers inside of the agent's pod.

I mainly run docker containers in my pipelines for unit testing, integration testing and some other offline jobs within the CI.

I don't have problems building images as I'm not dependent on docker anymore but instead I use buildx with buildkit which works great.

I tried using "podman run" instead of "docker run" but there's an obvious difference in terms of speed - docker is faster (I assume it's because of its daemon although not sure about it).

Using VMSS auto scaling might be an option but I rather scale based on the jobs queue and not based on resources i.e using a VM per job. Maybe that would be too expensive?

How would you do it/did?

https://redd.it/zhymhc
@r_devops

Is everyone technically a web/mobile developer?

Besides things like embedded programming and computer vision, when you look at software engineering jobs at places like Google, Amazon, Meta, Paypal, etc. are there all just web/mobile development jobs?

Technically, services like Gmail, Uber, and Netflix all have a client-facing frontend and a backend. And if you get a job there, you usually work on either the frontend or backend or something related, right? So doesn't that mean that pretty much everyone is a web/mobile developer?

https://redd.it/zii2j0
@r_devops

Ansible Variables | Ansible Tutorial for Beginners

https://youtu.be/X41WY4Fn5Yc

https://redd.it/zhvtm4
@r_devops

how to "bind" docker container to subdomain

Hello, I'm experimenting a little with devops stuff to not get out of touch too much with tech. I have IT/CS degree, but only work in project coordination and management since. So I know the theory and principles but I have not much hands-on experience.

So, what I have now is:

1. a small virtual server with a dedicated IP
1. I have root access via KVM and SSH, got SSH access with SSH-key.
2. in there I have a docker container with Jupyterlab, which is secured with login and password. I can reach this Jupyterlab via the IP:custom_port of the server.
2. a domain, I can create subdomains

What I would like to have is a secure Jupyterlab for my private use, always available from any device:

1. have the container continue running even if I logout of the session
2. "bind" the Jupyterlab container with a subdomain, like jupyter.mydomain.com.
3. secure the connection with a SSL certificate (letsencrypt?), and have "something" automatically renew the certificate.

What is the pragmatic way to achieve those 3 goals? Thanks in advance.

https://redd.it/zhtomo
@r_devops