Process metrics of a workflow run of GitHub Action

Hi everyone,

We are highly using GitHub Actions for our operations. Last week, one of our workflows started running 2x slower than expected. Code hasn't changed, so as our workflow definition. We couldn't understand why by checking the metrics that GitHub Actions provide for us.

One approach is to catch the processes with their start-end times, arguments, and paths. I mean by the process is seeing every tiny operation done during the whole workflow execution.

Would it make sense to track the process metrics inside of the GitHub Actions?

https://redd.it/uemoia
@r_devops

Merge static application metrics in Grafana

Hi, I need some advice from the devops community.

I am mostly software developer with some DevOps tasks too and I am trying to find the best way to show application data metrics in Grafana.

We have some microservices/backends running in Kubernetes and I configured the have the default metrics in Grafana and it works just fine. The Prometheus framework we are using is providing them and Prometheus is scraping them automatically from the cluster.

Now I wanted to add some metrics of the application data (values from the database) to also show in Grafana. The nature of those metrics is a little bit different because they are not dependent on the application instance but they share the same value. Lets say for instance the amount of registered users or the amount of some interactions they do with the application that involves saved stuff in the database. I added a background worker that fetches those data from the database and provide it to the Prometheus integration framework we're using. Prometheus is scraping them but of course it shows them once per instance. I am struggling to find a way to merge those to a single value in the Grafana dashboards. I am currently using a workaround of adding the instance to the Grafana dashboard variables and then force the viewer to choose one but I wanted to know if there is a better way to do it.

As I see it I would have the following possibilities:

\- build an run an extra pod that only scrapes those database metrics and let Prometheus scrape it from there instead of the app instances. This could lead to duplicate code since it would use some of the business logic of the actual backend to interact with the database and would also be another place where database credentials must be given to (via Kubernetes secrets)

\- somehow try to prevent running the background worker in more than one pod. This could be complicated logic that would contain constantly checking if one of the pods is currently running the background service and starting stopping it automatically based on other pods

\- The approach I am currently using with only showing data for a single instance in the dashboard but the user must choose one where it doesn't really matter for the data

\- Kinda workaround using max() in Prometheus queries, this works for simple metrics but gets complicated

What would be the best way, is there some possibility to do this with a Prometheus query that I was missing when reading through the Prometheus docs?

https://redd.it/uem5p1
@r_devops

As a working professional, what certificates would you recommend to have?

I am a working professional, I would like to finish some certificate courses so that I can "Jump the Queue" if needed in the eyes of HR if I need to change jobs or ask for higher pay.

I live a modest, minimal life and happen to save a decent chunk to invest back into my career. I know that certificates may not be everything but I think having some may help my prospects for finding something better.

https://redd.it/uepjr9
@r_devops

Do you have personal wikis, websites or blogs full of your notes & documentation you like to share?

I'm kinda obsessed with documenting everything I do and use Zettelkasten to never ever find myself in the situation of "I knew this once, I also wrote it down, but I don't fucking know where to find it" again.
I also love to extract and combine the ideas of others (and shamelessly copy the knowledge of some people).

So, if you're new, experienced, or whatever, do any of you want to share something of that kind?
Like Github repos full of your projects and documentation of how you did it, like personal Wikis or Obsidian Publish Sites full of notes with detailed documentation, concepts or fresh ideas about DevOps?
Maybe you even wrote a book?

If you don't wanna share it with the people here you could also DM me.
I just don't want to miss the crispy experience of indulging in ideas and alien ways of thinking.

https://redd.it/ueswl1
@r_devops

Pipeline security question in interview

I recently had an interview, that I wasn’t successful with, I think partially because my response to a security question wasn’t good enough.

The question was to describe the security steps present in a CI pipeline.

I talked about SAST and DAST, and mentioned having policies in place to protect things in deployment (I realised I Probably went here because I didn’t feel confident in my response, and felt I was missing something)

I am now spending the weekend researching the nuances of the answer, but wondered what you fine people would respond to such a question

https://redd.it/ueyb3v
@r_devops

Use multiple cloud-init datasource in VMware

I have little issue understanding how cloud-init datasource work and hope for your advice.

I deploy a VM using terraform from the ubuntu cloud image Ova.
This image supports userdata via parameters which are picked up by cloud-init on boot. Works as expected and solves my problem bootstrapping a VM.

Cloud-init also has the VMware datasource which is provisioned by extra parameters on the VM itself. This datasource is behind the ovf datasource in cloud init by default. Side note, VMware datasource is available in cloud init by default. This was an extra step in earlier versions.

So now I'm curious how cloud-init should work when I provision userdata on both datasources. When I debug available datasources in the vm both are detected with Ovf in the first place, followed by VMware.

In the logs I can see Ovf datasource is executed. VMware seams to be ignored.

So should cloud-init even consider the VMware datasource if Ovf is present?

https://redd.it/uf7bvi
@r_devops

GitGoat - deliberately misconfigured GitHub org

Pretty cool way to generate dummy data on GitHub, such as invite members, add them to Teams, commit code and secrets, raise & review PRs, and configure different branch protection policies (such as CODEOWNERS).
Link: https://github.com/arnica-ext/GitGoat

https://redd.it/ufbrx4
@r_devops

Vault HA mode(OSS) vs Vault Enterprise

Hashcorp vault enterprise provides three main features, performance replica, disaster recovery, and namespace. Well my use case is not required to go with disaster recovery and for performance replica i can setup Vault OSS with consul backend and run many active cluster which will be equivalent of performance replica, Is my understanding s correct will that feasible to not to use license and still have the same what Vault Enterprise

https://redd.it/ufdrv4
@r_devops

Is my expectations for candidates too high?

So we’re taking interviews for a senior DevOps role.

Most of the candidates are good with just tools, they know how to make a declarative pipeline with Jenkins, kubernetes deployments, daemonsets etc., Prometheus, grafana.

When we start to talk about systems, most candidates have no idea. For example, when I ask candidate who possess a CKA certification about the function of kube proxy, they really have no idea. Just saying kube proxy takes care of networking is a good answer? Expecting a candidate to know what subsystem of Linux is used by kube proxy is too much?

Expecting a candidate to know what layer of OSI does SSH belong irrelevant to DevOps?

Some basics about SSH/DNS/HTTP/TLS are essentially for any DevOps/systems engineering role imho, but candidates pursuing DevOps as a career lacks these.

Sorry for the rant, what I’m coming to ask is, just knowing to operate tools without giving a damn about the internals good enough ? How do you select people?

Edit: I see lot of engineers don’t like the idea of asking a osi reference as a question, just to let you know, I don’t cling to the osi and reject them for not knowing it. All I’m talking about is, lacking in basics of systems like DNS, ports, HTTP and etc. I’m sorry if I didn’t make it clear at my initial attempt.

https://redd.it/uffq2u
@r_devops

Out the Womb, Straight into DevOps

I will be joining all you guys as an associate devops engineer after graduation. If there is 1 piece of advice you could give me, what would it be?

What tool or concepts do you wish you knew when starting out? Any advice is much appreciated!

UPDATE: Thank you to everyone who commented! Everyone gave good advice and I really appreciate it!

https://redd.it/uerrww
@r_devops

How important is Design Patterns in a non-SWE role?

I am a self taught sysadmin, worked decades in the field and trying to gain dev competency, although I dont intend to be a full blown SWE. I know there are massive gaps in my knowledge and I am looking into "Design Patterns" and woah... the rabbit hole is deep. I am starting to think I cant even say I know Python now.

My question is: how important is "Design Patterns" for a devops/SRE role? Would you expect someone in this role to know it? Would a job that requires Python knowledge also require a good appreciation of "Design Patterns"?

https://redd.it/uf1ufp
@r_devops

How does your dev team onboarding look like?

Do developers install all dependencies (go/node/python/php/etc) and versions locally? If so, how do they manage to use the exact same setup as the rest of the team?

How do they make sure that their precommit hooks are aligned with the exact same tooling as the rest of the team?

Have you considered using a remote development workspace solution?

I’m looking for a way to simplify my team’s onbording setup by using their laptop as a gateway to a remote workspace where they could clone repos, push docker images faster with a faster connection, run the exact same versions and tools with a click of a button. Is this a common problem?

https://redd.it/uex0vq
@r_devops

PCI evidences located on servers

Asking folks who are in the devops team. Do you folks download credit card evidences like application log files from your linux servers? Or is there another team like application developers that downloads it? I'm just quite annoyed today that I had to do it. I believe it should be done by developers. However, they keep on saying we don't have access to PCI environment. What is your process?

https://redd.it/uey24x
@r_devops

Releasing some free tools ⚒️ 🪛🔧

I made a free web app to perform functions like DNS lookups, port checks port scans, IP Information gathering, hash generation, blacklist checks, subnet calculation and more.. saves me some time in the infosec/net-sec space.

Hope you enjoy! 🎸🤘🔥

Net-Tools.io

https://redd.it/ufr7qz
@r_devops

react app deployment in the cloud

So I have to deploy a react app on cloud

Assume a million Hits in future ( 1-3 years ).

Think of it as q checkout plug-in. It is not a static site as it will take inputs and do things like retrieval., authentication etc and much more.

What's the best way to deploy this and its implications.

Currently deployed on a node based docker image on Ubuntu using npm start as the entry point. This is through azure app service

Is docker the right choice ? Can another server like be used instead of npm with better performance.

Can a smaller image be built without using the node image ?

Any other things that I should think about.

Thank you in advance

Does docker make sense here

https://redd.it/ufrwcd
@r_devops

Do you do Application support as DevOps engineer?

I am quite new to DevOps and have been working for a software company scale-up. We are not a big team and we take care of the builds, deployment, automation, collaboration with Devs etc, so the usual tasks required from a DevOps engineer. On top of that though my team takes care of the application support of the software we are selling, including being on-duty 1 week per month. While i enjoy some of the work, the on-duty is a bit tedious as it happens quite often that non-technical users would log silly cases and it feels a lot like 1st line support. Another aspect is that the company is developing around 20+ products meaning while on duty we often get cases about functionalities we did not know existed... As far as I know DevOps teams do need to carry the pager but I was under the impression that it is mostly about failing releases, problems with infra, services etc and not about application support..

https://redd.it/ufzap7
@r_devops

What are your strategies or advice on a new devops engineer learning to support an application?

I recently got a new role as a devops/sre engineer from a network engineer background and is wondering how do people "learn" applications to be able to provide operations support?

In networking, we engineers just have to have a solid understanding of standard networking protocols and we can start troubleshooting in most environments. However, in the app world, there is no "standard" protocols and we seem to need to troubleshoot incidents with little understanding of what is happening in the application. I'm kind of lost at the moment and would appreciate some advice on how to get started

https://redd.it/ug0ll3
@r_devops

ansible - Not sure where to begin

Undergrad student here -- I have a devops project I'm working on and need some guidance. My professors and TAs are confusing me even more so I've turned to reddit.

I'm building, testing (running project test cases), and deploying an open source project by having a specific job for each using ansible.

I'm starting off with the build job. The open source project repo defines steps in the readme on how to get it started but I'm confused on how to translate that into lines in ansible. Any help is appreciated.

https://redd.it/ug5ubs
@r_devops

For those looking to make a career switch to devops, did it ever feel overwhelming with all the things you need to know?

Right now I'm learning Python because I feel like many IT careers (not just devops) would benefit from learning it, but when I start to see everything else you need to know it feels like it's neverending.

How do you usually manage not jumping between things, I know you can't learn everything at once but at what point do you go from learning one thing to learning another thing simultaneously?

https://redd.it/ug939g
@r_devops

Ansible for windows?

Are there any programs I can use to configure \~40 computers? My company updates their software (.exe) once a week, and requires:

1. Updating \~5 softwares (.exe), which requires uninstalling and reinstalling
2. Pulling changes from 3 repositories on GitHub
3. Reinstalling the python virtual environment (lots of pip install commands)

I currently use Jenkins, but have been running into significant configuration drift because some pipelines may error out. I heard Ansible is a good program, but noticed it is mostly for linux. Are there any alternatives for Windows?

https://redd.it/ugfd8w
@r_devops

Some practical learnings while participating in on-call rotations

I wrote a short blog post on some practical learnings I gained while participating in on-call rotations. Read it here: https://ernestas.me/on-call-leave-it-better-than-you-found-it

https://redd.it/ugjs4r
@r_devops