Synthetic monitoring with distributed traces

Disclaimer: I run DevRel at Tracetest (open-source tool for trace-based testing).

I was wondering if the community has ever had the need to run scheduled trace-based tests either at intervals or based on event triggers?

My colleague wrote a tutorial on how to do this with GitHub Actions and Slack alerts: https://tracetest.io/blog/github-actions-observability-slack-synthetic-api-tests

Wanted to check if this makes sense? Or, if you just use Synthetics for heartbeats, healthchecks, etc...

https://redd.it/13xl4zl
@r_devops

Can recommend a dashboard/app to improve productivity when working with many repos in multiple CI/CD tools?

As the title says. I spend a significant amount of time in my day hopping between email notifications, slack notifications and browser tabs in different CI/CD tools, reviewing changes, PRs, etc. Wondering what others have found useful? Thanks.

https://redd.it/13xnjcg
@r_devops

Can anyone recommend a dashboard/app to improve productivity when working with many repos in multiple CI/CD tools?

As the title says... I spend a significant amount of time in my day hopping between email notifications, slack notifications and browser tabs in different CI/CD tools, reviewing changes, PRs, etc. Wondering what others have found useful? Thanks.

https://redd.it/13xo0vs
@r_devops

Monthly 'Shameless Self Promotion' thread - 2023/06

Feel free to post your personal projects here. Just keep it to one project per comment thread.

https://redd.it/13xq62t
@r_devops

Friends - needs help choosing solution for SBOM vulnerability

I am new to this community. We are getting onto "Shift left" band wagon. I am tasked with proposing a tool or platform for Software Supply Chain analysis for vulnerability. There are so many of them which is baffling.


Please share your experience in choosing a tool and what are the criteria you have used to evaluate them. I will deeply appreciate any feedback. My own criteria are
1. It should be able scan my code as well as container images for vulnerabilities
2. I should be able to see trends of vulnerabilities over time
3. I should be able to make the insights actionable - crate a JIRA ticket
4. We have 50 repos that produces 50 docker images. Price should be $50 per repo per month which is $2500 / month
I would love to have a conversation if you are kind enough to share your views. Thanks a lot in advance.

https://redd.it/13xrpmc
@r_devops

How does prometheus monitoring work?

This is a pretty n00b question even though I'd consider myself a pretty experienced DevOps and site reliability engineer. I've setup New Relic on our EC2 servers and have a lot of serverless metrics going to NR from AWS. I've also built some custom metrics for New Relic.

We recently started using a new tech called Rockset which is pretty cool. It has an endpoint for metrics collection. This endpoint spits out some metrics data in a portable known format - OpenMetrics/prometheus. Cool I get that so far. However I'm looking through the NR documentation and it's talking about setting up Kubernetes clusters? And/or a NR agent? The agent moves the data? Why would I need a huge cluster? For moving a ton of telemetry data? It's my goal to visualize this data in New Relic.

https://redd.it/13xr8dn
@r_devops

CD tool for many many deploy environment

I'm working in a software company which need to deploy remotely to many customer sides.
The deploy process is composed by a preprocessing process + actual deploy managed by Ansible . Preprocessing process creates/extracts from differ sources the many Ansible variables needed, so it's environment dependant and cannot be integrated in the CI steps.
I'm looking for a good CD tool that could manage many deploy environment ( talking in the order of at least 100-200).
Any thought?

https://redd.it/13xsmm0
@r_devops

Macbook air M1 8gb of ram worth it?

Hi everyone! im looking to buy my first macbook, the reason is that im starting to work as a contractor and many offers out there require me to have a macbook (do not know fucking why)

The thing is that is a super expensive machine for me since i can buy way better hardware for that price. Does it worth it for devops role? does the cheaper version of 8gb of ram and 256 gb of drive be enough? should i aim for a macbook pro insted?

​

Thanks.

https://redd.it/13xwfm8
@r_devops

How do your teams run DB migrations?

We're a new team and looking to standardise on an approach. Currently, devs run them manually from their machines (using flyway), migrations scripts are checked in to version control.

We'd like to try run them automatically on every merge to master across all envs.

This would require the application & DB evolution to be decoupled, which I think is a good practice anyway.

Wanted to canvas the community for any opinions, this is quite a common problem so I'm hoping there's a fair bit of experience I can leverage!

Thanks

https://redd.it/13xxovs
@r_devops

Ansible: Connecting to multiple jump hosts

So, I have a few groups of servers that are only accessible locally from nodes which I can reach from my controller. My controller needs to reach out to the nodes, and then to the servers, which I already have figured out.

The thing I'm working on now, is trying to iterate through those nodes, so that I can connect to node1, node2 and then ping all the servers beneath them.

Nodes
192.168.1.2
192.168.1.3

Servers
192.168.2.2
192.168.2.3

Servers.vars
ansiblesshcommonargs='-o ProxyCommands="ssh -W %h:%p -q 192.168.1.2"'

The way the vars is configured currently, it will connect to the node `192.168.1.2` and then ping the servers `192.168.2.2` and `192.168.2.3`, but I would like the then connect to node `192.168.1.3` and do the same.

I tried

[Servers.vars]
ansiblesshcommonargs='-o ProxyCommands="ssh -W %h:%p -q Nodes"'

etc, but with no luck.

https://redd.it/13xv8d2
@r_devops

Interviews - How do you choose to display your problem solving during questions you’re unsure of?

I’ve been interviewing for entry engineering roles recently and of course there are technical rounds where I’m given problems to solve. Sometimes, I don’t know the answer and can be at a loss of words.

What do you do to showcase the way you solve problems in those scenarios? For example, you’re given either a few lines of code to finish or an output you have to reach. I’ve hit these scenarios where, in my mind, I think, “oh, I know there’s a method for this I just don’t know what exactly it’s called” or “oh I think it’s this method but I’m unsure of the exact syntax”.

If I were on the job, I would just google it or look at the docs. But it feels odd to do that during an interview so I haven’t attempted other than joking that I would if I were able. It also makes me feel anxious to think about asking if I’m allowed to.

Anyway, what have you all found success in doing for these scenarios where you’re in a technical interview and don’t know the answer, but want to demonstrate your problem solving?

https://redd.it/13xyqjb
@r_devops

Dynamically generate Cloud System Architecture diagram

Hello friends - Our team is responsible for creating "System Architecture Diagram" showing how network topology looks like and how its connected to other resources such as applications, services, storage and databases. Currently we do it using Visio or other drawing tools. It takes very long time and often becomes stale.

Is there any tools that can generate the system architecture automatically? This will save our time as well as keep the architecture always up-to-date.

We do this primarily to

Provide training,
Understand and optimize architecture

Any help will be deeply appreciated. Thanks

https://redd.it/13y1vnj
@r_devops

Terraform Provisioners

Terraform provisioner is useful when you need to...

Intro blog to terraform provisioner.

https://faun.pub/demystify-terraform-provisioners-f75e738a824

https://redd.it/13y2zo6
@r_devops

How to conditionally run db seed in vercel-build script only for preview deployments on Vercel?

I need to set up a default test database for preview deployments with Prisma, and of course a safe way to NOT execute the seed in the production deployment. I can hack something with shell scripts and the APP_ENV env variable to set the condition, but I need a proper and standard way to do it, not to invent and hack, probably `vercel.json` can help here. It wouldn't be bad if I also check the branch to see if it's `main`, for additional insurance.

Has anyone done this before to describe the method or a point to useful link?

```
// package.json

// preview environment
vercel-build: npx prisma generate && npx prisma migrate deploy && npx prisma seed && next build

// production environment
vercel-build: npx prisma generate && next build
```

https://redd.it/13xodma
@r_devops

Seeking Advice: Gmail vs. Hotmail API Access and reliabe setup for a Python-based Email Automation

I recently set up an automation to check my Gmail inbox, but I'm facing many difficulties with Google's API. The setup was quite frustrating and convoluted, especially as an individual user (I don't have a corporate account which has some extra benefits).

After a few days of successful operation, the automation suddenly stopped working, and I'm now encountering an "access denied" issue while trying to renew my Gmail refresh token. There's no other info except "access denied".

Given these difficulties, I'm considering exploring alternatives to Gmail. Do providers like Hotmail (Outlook.com) or any other trusted email providers offer API access to their inbox? If yes, how complicated would you say their setup processes are compared to Google's. Also, how reliable have you found them to be?

Any advice or recommendations on reliable providers with user-friendly API access would be greatly appreciated. Thank you in advance for your help!

https://redd.it/13xp0d6
@r_devops

Wordpress in a HA k3s Cluster

Helllo Fellow Redditors,


Im currently working on a privat project for my club, we had our website hosted in Hostpoint but i want to migrate it into my HA kubernetes Cluster.


Im currently having 3 servers running on Hetzner ( 1 Controller/Master Node & 2 Worker Nodes)

Now my issues is regarding storage, i thought about creating a mysql pod ( basically a DB pod) and a seperate pod wp-content ( stores WP theme, images etc).

if i choose this route i would lose HA of my cluster ( and basically thats a nono for me)


So any one any experience how to implement a Wordpress page into a HA cluster ?

​

Cheers

:)

https://redd.it/13y5oie
@r_devops

Little rant about AWS

I am very disappointed in AWS especially ec2 instances we use for production. I honestly cannot accept the crap service they provide for THAT much money.


Problems I have regularly:

\- Price. Honestly, I have never found out the price per 1 instance. I can see price per month, price per ec2 instances alone, price per region, but the hell with that, it just takes so much time to find a simple things like how much does this particular instance cost. Its like they dont want you to know. Or they want you to make become a certified specialist to use their UI.

\- Hardware Degradation. This is happening too many times its just annoying. Your instance just stop working suddenly, then you get an email "oh sorry degradation blabla". Holy smokes just take care of it like any other provider.

\- 1/2 checks passed. What in the hell is this one. Randomly, some instances just stop working with a 1/2 checks passed result. No explanation nothing. I am sure its possible to dig the reasons somewhere in their damn UI but I just want my server to be up

.- Bandwidth. Small instances has small bandwidth. Cmon! It costs 7x more than equivalent server from Hetzner so just dont be assholes. Why would you think I need less speed when I need less ram and cpu? that doesnt make any sense.

​

Feel free to correct me, I am sure lots of these are coming from a lack of my knowledge regarding AWS. I, however, want to be efficient and feel like AWS is just way above our company needs. What we pay 700usd, we could pay 100 usd in hetzner.

I am thinking very hard about migrating.Any opinions, experience related are very welcome.

https://redd.it/13y6wgm
@r_devops

Azure + Cloudflare Workers or K8s on baremetal?

Azure is expensive on scale, but it’s easy. It’s possible to use Cloudflare Workers to cache Azure backend (with Cloudflare R2/KV/etc.), for authenticated/unauthenticated requests (this caching can also be done with Fastly, which actually Reddid uses, it just costs more than CF).

Managed K8s is expensive on scale in Azure, but cheap on baremetal (f.ex. Hetzner).

Which would you use to do cheap infra that can scale?

https://redd.it/13y6jpr
@r_devops

Would love some feedback on a little AI tool I tinkered with for answering questions about just Atlassian and helping coders get started with e.g. Jira

Demo link to try out the AI chat widget yourself is here: https://demo.kapa.ai/widget/atlassian

Please let me know what you think and whether or not you experience any bugs or issues. Thinking this could be a good way of using AI but still learning things more properly instead of just asking GPT to answer.

It does not work like GPT but more as an AI docs searcher and thus have very limited creativity ability. Feel free to shoot over questions if you are curious about how I built it or similar things

https://redd.it/13y8jlj
@r_devops

Need some infrastructure as code book recommendations for my bachelor thesis

I am currently studying Software Engineering in the fourth semester and preparing for my bachelor thesis. I am very interested in DevOps and probably want to study further in this area. Last week, we got a catalog for possible topics for a bachelor thesis, but none have interested me, so I thought writing one about DevOps. I talked with my professor, and he recommended me a thesis in which I cover what infrastructure as code is and comparing three common tools using a small example project. (I don't know which tools I will compare, but probably Terraform, Ansible, and another one. I am open for recommendations). For that I need some literature which I can use for references. I already searched for some, but I don't know which ones are good. Currently, is my safest bet, an O'Reilly.

I would love some good recommendations, as I am still very new in this area.

https://redd.it/13yad7e
@r_devops

To many pipelines in synapses and cant create ARM template as reaching max size

To many pipelines in synapses and can’t create ARM template as reaching max size.
We created PowerShell scripts to parametrize but this is causing issues between environments but causing many issues

Also looked a synapse extension which may help.

Any suggests on best practices or clean solutions?

https://redd.it/13yclrg
@r_devops