Monthly 'Shameless Self Promotion' thread - 2022/05

Feel free to post your personal projects here. Just keep it to one project per comment thread.

https://redd.it/ugqs3a
@r_devops

Which IDE/Editor is Your Daily driver?

In last few years I tried Vim with bunch of plugins, NeoVim, Emacs (Vanila, Spacemacs and Doom), VsCode (also with neovim), Acme (from Plan9), IntelliJ GoLand, Sublime Text... I'm curious, which IDE/editor with external tooling is Best for You.

View Poll

https://redd.it/ugstjr
@r_devops

Those of you using prometheus as part of your observability stack, what approach did you take to scaling to scrape 25+ clusters, and why? Is Thanos the answer to my problems?

Hi everybody,

First post, longtime lurker :) .

So recently i've been tasked with implementing prometheus/grafana in our org as part of our preparations to get good devops strategies in place, as we are now aiming to hit zero downtime deployments in production.

We've got almost 50 clusters to scrape, and I'm thinking about the most efficient way to scale prometheus to handle all this ingestion, while having some breathing space for future growth.

Particularly as there is a non-zero possibility of another \~100 clusters to be provisioned in less than 24 months.

My ideal outcome is that all metrics will be available via a single grafana instance, and we can manage alerting from there for all clusters, but I'm worried about if this will be doable at scale.

I understand that Thanos (https://github.com/thanos-io/thanos) was built with the idea of improving prom's scalability and availability , and will be testing this shortly, but would love to hear from others that have tried various approaches to try to solve this challenge.

At the bottom of the github link above there are two architecture diagrams for possible implementations, has anyone used it before, and how did it pan out?

Before learning of thanos, I was thinking of running prom on each cluster, and then having a "master" instance of grafana to offer us a centralised view in grafana, but i'm sure there's a better approach to this.

In terms of configuration, I'm installing the prometheus/grafana operator using the helm charts in the community repo: https://github.com/prometheus-community/helm-charts

https://redd.it/uhmk77
@r_devops

How do I find out which container is responsible for a specific docker overlay folder?

I'm inside a VM which runs multiple docker containers. We have inspected some logs and noticed that in the VM's /app/docker/overlay directory there's a bunch of folders (presumably from the containers). They have hashed names like "jsdnjenljf8239ujsdkaldkoksdjo". Anyway, how do I determine which container is responsible for a particular directory like "jsdnjenljf8239ujsdkaldkoksdjo"?

​

https://redd.it/uhcerd
@r_devops

For 2022 - Sites for learning Azure & DevOps Tools? Pluralsight vs aCloudGuru vs Youtube / Udemy vs Books?

I'm somewhat fucked - I actually got a job for a Cloud / DevOps position - without any experience in DevOps and only some experience with Azure.
They just like me a bit too much.
I'm a sysadmin with a knack for linux, automation, containers and monitoring. Nothing else.
I have a month and 10 days until it begins and I lost the last three weeks just understanding concepts and "playing" with the tools.

I have the problem that I'm tutorial hopping, switching between books, Youtube, aCloudGuru, blogposts, etc... without making real progress.
I never know if the course I start is even in-depth enough to be worth my time.
The hopping is mainly caused because I'm not able to decide if I really want to go deep and learn everything or if I should work on projects without understanding shit so I can show them what I "prepared".
But I need the basics. And the basics are worth years of content.

Everyone is saying something else... aCloudGuru is bad, it's good, it's not deep enough, they make awesome content, go back to reading books, find another job or do some goat farming.

It's just too much tbh. The breath of possibilities is immense.

I need some guide... some structure... I'm working on my own curriculum with questions, why's and projects right now, but it's permanent trial & error and I really don't have the time for this.

So, are there any good courses (or sites) on building solutions with Azure and CICD pipelines that stretch quite far and are deep... or should I go back to reading books and accept that this will take a long, long time?

https://redd.it/ui05m1
@r_devops

Deploying Kubernetes Clusters in Increasingly absurd languages

Pulumi announced YAML support today, and I hate writing YAML. So I did some horrible, horrible things. Behold:

https://leebriggs.co.uk/blog/2022/05/04/deploying-kubernetes-clusters-in-absurd-languages

https://redd.it/uiaaj3
@r_devops

GH actions - how to change trigger branch without a code change?

**Context**

We try to use git-flow.

During our sprint, we work on the develop branch and towards the end of the sprint, we cut to release branch and make any last min changes there. Meanwhile, the rest of the team keeps merging their PRs into develop. The GH action workflow watches the specified branch and deploys the code when we have a merge (into develop for example).

**Problem**

We want to change our trigger branch (branch to watch & deploy) when it comes towards the end of the sprint. We can do this by changing the branch specified in the yaml file but feels like an overkill. We would ideally like to do it from somewhere else like the workflow\_dispatch GUI or using a label or something that requires minimal effort for the team.

Here is our stripped down yaml:

name: Build & Deploy
on:
workflow_dispatch:
push:
branches:
- develop # I want to change this dynamically
jobs:
build:
runs-on: ubuntu-latest
- name: Install & build
run: |
echo "Deploy code..."

Is this a GH actions limitation or can our workflow be improved?

https://redd.it/ui2vk8
@r_devops

KUBECON EU 2022

KubeCon EU 2022 is just around the corner and LitmusChaos is all set for its Project Meeting on 16th May (Monday) at 13:00 to 17:00 hours CEST in Valencia, Spain.

Register now to book your seats NOW (limited seats available)!

https://linuxfoundation.surveymonkey.com/r/WCPMX6R

https://redd.it/uid5vm
@r_devops

Alert individuals when an item changes lanes in an ADO board

Is there a way to alert a group of people when a card is added to an ADO board? For example, if a card is moved from the Development lane to the Testing lane, the members of the Testers group is alerted? If so, where is the configured?

https://redd.it/uiczgq
@r_devops

Is there a good place to hear devops STARs stories, especially cloud ones?

Hi guys,

I've always learned best through example and I was wondering if there were any online resources to read or watch people explaining their devops STARs(situation, task, action, result) with good specifics? I'm just looking to see examples so that I know how to better talk about my own experiences. Thanks!

https://redd.it/uif51j
@r_devops

Using HaProxy on Nginx server. Not listening to port 80

Recently setup a server & using haproxy. Everything else runs smoothly but port 80 is not connecting. Here is the haproxy config file. Esp gives problem when certbot tries to renew. what am I missing here?

frontend backend.sample.com
bind *:80

# Test URI to see if its a letsencrypt request
acl letsencrypt-acl path_beg /.well-known/acme-challenge/
use_backend letsencrypt-backend if letsencrypt-acl
bind 64.123.456.124:6684 ssl crt /etc/haproxy/certs/backend.sample.com.pem
default_backend webapps

backend webapps
balance roundrobin
server app01 64.123.456.124:5684

backend letsencrypt-backend
server letsencrypt 127.0.0.1:54321

https://redd.it/uiqjq4
@r_devops

How do I get my newrelic app string code?

I created a Web app, and I got the long code for importing to the newrelic.JS file, BUT, I didn’t commit it, and I need the code to copy again....

Where do I find it at?

https://redd.it/uin9fg
@r_devops

When did it become standard for every job interview to require homework on top of 4-5 rounds of interviews?

I've been hearing the market is hot so I went to look for myself. I've gone through quite a few interviews and every one of them has required a take home skills assessment on Hacker Rank. I went through the first few but after that I just started declining the tests. I just don't have the time/capacity to dedicate 6 hours of interviews/testing per job right now. I can understand the appeal to weed out people but there has to be a better way. This used to be unique to FAANG companies but it seems to have caught on. Are there any DevOps/SRE jobs out there that don't structure their interviews like this? What has your experience been like?

https://redd.it/uix61t
@r_devops

GitHub Actions Security Best Practices cheat sheet

Been compiling security best practices when it comes to GHA and learned quite a bit in the process. Hope you find it useful!

https://blog.gitguardian.com/github-actions-security-cheat-sheet/

https://redd.it/uj13sq
@r_devops

How many (AWS) accounts do you have?

How many AWS accounts does your organisation have? Why that number, and what's the organisation strategy behind it? When do you add a new account? What are the best practices with accounts?

I'm curious because where I work we have, very roughly, one account per environment (dev, staging, prod) per repository. As a dev I find it hard to figure out where everything is, but then I'm not a devops person.

^(Assuming you're using AWS. Question is specific to AWS, but probably applies to other platforms too.)

https://redd.it/uj8yzi
@r_devops

Best CI/CD Practices on AWS and Kubernetes

We have a pretty standard NodeJS app. I am moving away from Elastic Beanstalk to K8s.

So far I have... Bitbucket Pipeline > Build Container > Push to ECR.

My question really is... whats the best practice for the next step? There are plenty of guides but many of them seem to assume you are using a myriad of other external tools, which I really dont see the necessity of.

My main issue is how to manage updates across our 3 envs, dev, uat, prod. Do I use different container repos for each, and have each of my 3 branches push to those?

How do I trigger the rollout of the new images, and specify which image should go to which, automatically?

I was thinking I can tag the images with {ENV}-{API_VERSION}-{BUILD_NUMBER} so that on any given image I can tell exactly what env + version it is, and how current it is, but how that gets executed in Kubectl is giving me issues.

Ive got dev and uat on one cluster and prod on another, all seems fine when I do it manually. I just need to piece this together in CI/CD now.

Any help is massively appreciated. Cheers!

https://redd.it/uiu0am
@r_devops

Role of AI/analytics in Devops/SRE scene ?

I hear all over the news that lots of non tech related industries are using data and analytics for decision making and improving performance — such as sports , advertising , sales.

Where does AI/analytics fit in with Devops-SRE Venn diagram wise?

I understand that some tech products are using ML/AI to find anamolies in logging data and alerts. Any other good use cases for AI that make sense? Can it help with CI-CD — or the process for building out infrastructure for the software to run on?

https://redd.it/ujhh9h
@r_devops

DevOps.com on why you need to back up your GitHub repositories

You’ve probably heard the joke that there are two types of people in IT: Those who do backups and those who will start. Though it’s still valid, this joke has become less relevant to businesses and professionals. The IT industry has been increasing expenditures on security for years, and backup is a critical area. However, despite the growing awareness of the need for backups and the wide availability of modern backup solutions, the problem still exists. The number of security breaches is growing, and the topic of data security looks like an endless arms race. So what can organizations do?

Find that out in this article: https://devops.com/why-do-you-need-github-backup/

https://redd.it/ujik66
@r_devops

DevOps.com on why you need to back up your GitHub repositories

You’ve probably heard the joke that there are two types of people in IT: Those who do backups and those who will start. Though it’s still valid, this joke has become less relevant to businesses and professionals. The IT industry has been increasing expenditures on security for years, and backup is a critical area. However, despite the growing awareness of the need for backups and the wide availability of modern backup solutions, the problem still exists. The number of security breaches is growing, and the topic of data security looks like an endless arms race. So what can organizations do?

Find that out in this article: https://devops.com/why-do-you-need-github-backup/

https://redd.it/ujik66
@r_devops

Current Windows / Azure Sysadmin interested in DevOps

### Background
I am a Sysadmin at a large MSP. I mostly deal with Windows and macOS. I am advanced level with PowerShell scripting, I have some limited experience with Python and Bash (just haven't had much need for either). I have a strong background with Azure / M365 (Enterprise admin certified) also VMWare and Hyper-V virtualization. I try to automate everything I can when I can in my job. I am getting a bit bored with it and I feel like DevOps or straight development might be a better fit for me.

If you were in my position, how would you start working towards a DevOps career?

https://redd.it/ujsa7v
@r_devops

Best and cheapest way to run lambda functions

Hello!I'm a JR Devops engineer tasked with reducing time and cost of some lambda functions we run on EC2 instances.

Currently we use hourly spot runs (to minimize cost), with most of the time being spent on installing requirements (python project, being cloned and set up from scratch every run).

How can I utilize a docker here?

I have read about Amazon ECS, serverless, and some other solutions, but i'm not sure which is the best case for our use.

Maybe utilize Docker? How do I go on about creating a docker image from this project and run it on an EC2 instance without having to set up the whole environment (especially requirements).

Edit:To be clear, I don't want hand to hand guiding, just a point out of best services to use.

Thanks!

https://redd.it/ujmbau
@r_devops