Monitoring using Serverless

Has anyone used serverless workers (Cloudflare Workers, AWS Lambda, GCP Cloud Functions) or CI/CD service (GitHub Actions, Jenkins) for monitoring?

I'd like to monitor a service, and normally I'd make an exporter for it and have a server for Prom/Grafana or InfluxDB or whatever.

But I'd like to not have to manage a server, since this is just a hobby project.

One option is paying for some Prom-aaS hosted service, but I think it'd be interesting to run something on GitHub Actions or Cloudflare Workers, since those are both essentially free for paid users.

I don't need Grafana levels of visualization, honestly I could generate hourly/daily/weekly/monthly/yearly graphs with rrdtool and be good with that.

Has anyone tried this before?

What did you use for a your (time-series) database? A file-based db like sqlite in git, or maybe in object storage? How was performance like?

How would you display your graphs? In git? How do you make sure your git fs doesn't become bloated?

https://redd.it/12200dv
@r_devops

How do you improve your speaking skills, especially when you're ruffling some feathers with a new process?

Need your help on something, and I'll create a hypothetical situation.

Let's say you're helping out a team at your organization (you're on team X, you're helping team Y). Team Y works slow, is backlogged, and doesn't understand how to use tech to its advantage. Let's say they do a process that takes them 1 hour, and you propose a solution where you can automate it where it takes 10 seconds.


You think to yourself: "Everyone is going to love this, this meeting is gonna be cake". So you get to the meeting, explain the process, and although half the team loves it, you have those people who hate change in regards to everything. Now I some of you are going to think "Well maybe they have a point", and you're right, they might...I'm ALWAYS open to suggestions, but for the sake of argument let's just assume these people argue against any change just because.

For instance, they might say "Well what if this thing fails and takes down the whole thing! We won't even be able to do it manually", they'll come up with this bizarre hypotheticals that have almost 0.0001% chance of happening, and like, I go blank....because the only responses in my head sound patronizing...like if I said doing it manually is prone to error, they might say "Well yes but we're in control", you know, that sorta thing.

Is this a skill that you developed yourself? Do some people just have it and just people dont?

https://redd.it/12239ji
@r_devops

You're messing up if you're not meditating like this

https://www.youtube.com/watch?v=WsVox90y6ow

https://redd.it/120t8vl
@r_devops

DevOps 101 for a Dev Who Doesn’t Like Ops

The basics of DevOps 

So, what is DevOps? At its core, DevOps is a culture and set of practices that aim to break down the barriers between development and operations teams to improve collaboration and efficiency. It involves automating and streamlining the software development process, from code creation to deployment and beyond.  DevOps is not just a set of tools or processes, but a way of thinking about software development. It’s about creating a culture of collaboration, communication, and continuous improvement. With DevOps, developers and operations teams work together to build, test, and deploy software faster and more reliably. 

Additionally, DevOps promotes collaboration and communication between different teams, which leads to a more efficient and streamlined development process. By breaking down the silos between development and operations teams, everyone is on the same page, working towards the same goal. This results in faster and more reliable releases, as well as overall better quality of the product. In short, DevOps is a time-saving and collaborative approach to software development that ultimately leads to better outcomes for everyone involved. 

Read more

https://redd.it/120kp5c
@r_devops

iniciante

Tenho interesse em entrar na faculdade porém gostaria de saber qual o melhor curso pra quem deseja aprender a programar do zero e sair no fim do curso com uma boa bagagem da área de tecnologia

https://redd.it/120hl1x
@r_devops

Need some advice on my infrastructure and ci/cd

First of all I am not an expert in devops, with that said this is the idea I have in mind for a b2b software I am involved in, the software has the following components

A sql database for each customer, giving a dedicated database (just database not server) makes it a lot easier to scale and the entire system won't slow down when one user abuses it, also gives me time to analyze bottlenecks when it happens since it won't impact everyone at once

About 3-7 web applications per client, some of them are ssr (sveltekit) and will run on either cloud run (gcp) or cloudflare workers, no big server side stuff just loading data and rendering pages

An api server that has an api, websocket server and probably a redis instance for some caching (stuff like logs go into redis before being flushed to sql, I can put it on the same server and not worry about persistence because I don't care about it since it's only for stuff like logs and gets flushed frequently)

All of that is per customer, there are some industry standards so giving everyone their own environment makes sense, also they will charge a couple hundred dollars a month so paying 20-30 for hosting is not a big deal

So when it comes to deploying I need to do the following steps

1. Generate the static web applications and deploy to the correct instances

2. Deploy the ssr web applications

3. Migrate the database

4. Deploy the api server

For testing (only when there are changes pushed) I have the following in mind

1. Deploy in a staging environment and create the database migrations on a populated database

2. Run all unit tests (could be performed in a simple github action)

3. Run all integration tests in the staging environment this includes using the client applications to interact with the api just like in prod, I am also including some tests to mock offline behavior

4. In this industry reliability is important so I am designing a stress test, it will consist of a staging environment simulating thousands of request basically mocking a days worth of activity (or maybe just a few hours) and log the results (speed, percentage of failed requests) and then evaluate based on the percentage, ideally I would automate this step but I think that it would be better if I could create an excel sheet or something and check the results before approving the commit, for example if all failed requests come from something trivial the team may choose to bypass it, in any case this is how I am planning it

One thing I haven't figured out yet is how to store sensitive info, each customer has a couple hundred variables like settings that should be stored in a central application, the problem is with stuff like database passwords I am not sure how to store it, I would just store it in a sql database (maybe encrypt it) and restrict access to that database but that doesn't seem right, I was looking into gcp secret manager but I don't wanna lock myself into an ecosystem

Additionally the only way I know how to trigger automatic builds is via a github webhook but I am sure it's possible to just forward the webhook to my own server that could in turn start the deploy process for each environment keeping in mind some restrictions that users might have enabled (like no changes during business hours) I also need to be able to do that from an internal admin tool

I have no idea if any of this makes sense or not, also I can write all of that in code but I am sure there are existing solutions, that could help me out I am just not aware of it

I know in the long run this project will require a full time devops person or team but we are not public yet and at this point I am just trying to get an mvp going without building uo technical debt

Any guidance will be appreciated, I am also willing to invest time to learn so if I am completely off with this whole approach please nudge me in the right direction

https://redd.it/1229frj
@r_devops

Meetup: ¿Todavía no has oido hablar de DevOps?

Es este MeetUp os ayudaremos a entender la cultura DevOps y su uso dentro de los proyectos.
enlace del meetup

https://redd.it/120hkwc
@r_devops

Is it necessary to maintain a logical layer on top of your codebase?

## Background

Although Git is a widely used platform for version control and collaboration, it does not have the capability to analyze and interpret the logic of code.

Assuming my program now wants to know what is on line 35 of the extract_test.go file. If using GitLab, it is possible to find the corresponding text information, which may look like this:

func TestExtractString(t testing.T) {
fileResult, err := ExtractFromString(javaCodeForExtract, &ExtractConfig{
LangType: core.LangJava,
ExtractType: extractor.TypeExtractFunction,
})
if err != nil {
panic(err)
}
for _, each := range fileResult.Units {
core.Log.Debugf("result: %s", each.GetDesc())
}
}

However, my program does not know what this text represents. This is even more difficult for programs written in other languages (such as Java or Python).

Becides, there are many tools require extracting information from source code for their secondary development. They have to re-implement the extractor once and once again. It's a huge cost.

## What we do

As a member of the infra team, we have decided to add an additional layer of logic to the code repository, so that all third-party tools can access the logic in the code repository using our unified API, reducing their costs in compatibility and language parsing.

​

With our API, they can easily extract metadata of different languages from code files without the need for any additional parsers to be written.

{
"repo_id": "sibyl2",
"rev_hash": "e995ef44372a93394199ea837b1e2eed375a71a0",
"path": "extract_test.go",
"signature": "sibyl2||TestExtractString|testing.T|",
"tags": ,
"func": {
"name": "TestExtractString",
"receiver": "",
"namespace": "sibyl2",
"parameters":
{
"type": "*testing.T",
"name": "t"
}
,
"returns": null,
"span": {
"start": {
"row": {
"$numberLong": "34"
},
"column": {
"$numberLong": "0"
}
},
"end": {
"row": {
"$numberLong": "45"
},
"column": {
"$numberLong": "1"
}
}
},
"extras": {},
"lang": "GOLANG"
}
}

By using MongoDB, we have opened up the data we collect to our data warehouse, which makes it possible to analyze data at the warehouse level. Users can easily analyze the repository situation of a specific version, or even between versions, based on this.

## Discussion

We have already given a speech on QCON 2023 in Beijing, China.

It is indeed able to meet the needs of some Chinese teams and has received some praise. But I'm not sure if this is a suitable and universal solution.

Feel free to leave your comments :)

## Others

This is not strictly speaking a promotion, but if you're interested, this is our GitHub link.

https://github.com/opensibyl/sibyl2

Thanks!

https://redd.it/122f8wu
@r_devops

Is there a tool tracking releases of infrastructures/cncf projects?

Recently, as the product grows, more infrastructures are being added to existed tool stacks, since we are a very small team and right now most of the infrastructures' version are not synchronized, I was quite scared of missing out big updates, then I suggested my team start having "release reviews" each quarter to see what have been updated or deprecated and discuss on whether to upgrade.
I know that to start implementing such routine, we'll need to start tracking releases of tools and compare with current version in used, luckily ArgoCD supports sourcing from different repos right now, so I put all the infrastructure together with application sets and wrote a tiny script that basically runs like this.


1. project repo, name, category, current version in used and latest stable release are manually listed inside a google sheet document
2. script pulls from sheets and getting information either from github or artifact hub
3. if versions are newer or different, it will then be updated back to the sheet and a slack notification will be sent to the channel tracking releases

​

I was wondering if there's any tool that works similarly to this, or is this even a good idea, because I cannot find tools that work like this, so I thought maybe people don't have this type of problem?


As I wish to contribute to the open source community, lots of projects that I'm familiar with are beyond my technical level since I don't have many years of experience in coding, so I guess this is a good place to start.


What do you think? How would you tackle this problem?

https://redd.it/122g5zs
@r_devops

Kubernetes Operations Survey

Hello, y'all!

I’ve started a new Kubernetes operations survey. The goal is to better understand so-called "2nd day operations" with Kubernetes clusters e.g. upgrades, maintenances, disaster recovery, etc.

I've already posted it on r/kubernetes, but I want to collect as many responses as possible.

You can check the results of the previous year’s survey here.

P.S. Last year, some folks suggested that Google forms have flaws, which is true. However, I don’t collect your email - I don’t care about that. This is purely my personal initiative. Also, it looks like Google Forms is the only free tool that is good enough for both form creation and some basic graph building, as well as it’s easy to use. Yet, if you have any suggestions on what can replace it - I would be happy to check those alternatives!

https://redd.it/122jopm
@r_devops

Redirect Ingress GKE trafic to an external domain

Hello,

We do have few endpoints that are too critical to be handled by our monolith app (even if it is replicated in k8s). we thought we could outsource them to cloud run and redirect traffic depending on the path to cloud run or the backed carried by k8s. like this

​

apiVersion: networking.k8s.io/v1
kind: Ingress
spec:
defaultBackend:
service:
name: monolith
port:
number: 80
rules:
- http:
paths:
- backend:
REDIRECTTO: "HOSTNAMECLOUDURUN.a.run.app"
path: /serverless
pathType: ImplementationSpecific

Each ingress rule is expecting a "service" backend.

The issue is a cloud run hostname is not a service, but a domain like https://serverless-xxxxx.a.run.app/

I just need to forward a certain path in my ingress to the hostname of our cloud run.

PS: I can do that via GCP console and update the load balancer, but this doesn't seem a good idea as it will be overwritten by k8s

Best regards

https://redd.it/122kfd4
@r_devops

Is it inevitable every DevOps job is high stress, over-worked?

Admittedly, my current company has a shit-ton of technical debts, no processes and crap management.

But how much better can it get? I'm wondering if a career in a company that doesn't have it's own app (cloud work, modern sysadmin if you like) might be a better bet (lower paid but less stress). I mean, there's a reason why the average tenure in a DevOps position is barely 1.5 years.

https://redd.it/122pq29
@r_devops

Learning about cloud security?

How do you learn about cloud security?

Are there some blogs or YouTube channels that are particularly helpful?

I tried searching but found that there are a lot of channels for cyber security but none for cloud security.

By cloud security, I mean Docker, k8s best practices, how to store passwords in a docker container, common pitfalls, etc.

https://redd.it/122p71b
@r_devops

Is it true that if you don't have proper grip over programming language you can't be successful in devops?

Is it a misconception or is it true that you don't need to know programming language to be successful in devops

https://redd.it/122sf0i
@r_devops

How do you get the list of successful builds in the "CD/Deploy" job from the "CI/Build" job in Jenkins Declarative Pipeline?

I am planning to keep the build multi-branch pipeline separate from the deployment pipelines (one pipeline each for pre-prod and production environments).

I wanted to list all the successful builds of the "main" branch of the build multi-branch pipeline in the deployment jobs so that the "Deployer" can select one of them (latest build during a "normal" deployment and one of the previous builds in case of a rollback).

In particular, the build pipeline pushes a docker image to JFrog artifactory. I want to list the image tags of successful builds that can be deployed. Deployment is done by using `helm upgrade --install` which accepts image tag as one of the options.

I am unable to find any resource that tells how this can be achieved in declarative pipeline.

Any suggestion would be extremely helpful!

https://redd.it/122scro
@r_devops

DevOps Junior

Hello everyone, I recently got a job as a junior devops, I had never worked in IT before and I would like to know as a senior devops what is expected of me in a period of six months.

context about me 2 aws certifications solutions architect and sysops certifications and some projects related to terraform, aws and python.

https://redd.it/122rnrf
@r_devops

My evaluation of the Scaleway Cloud provider

A couple of years ago, I developed an app that helped me manage my conference submission workflow. Since then, I have been a happy user of the free Heroku plan. Last summer, Heroku’s owner, Salesforce, announced that it would stop the free plan in November 2022. I searched for a new hosting provider and found Scaleway. In this post, I’d like to explain my requirement, why I chose them, and my experience using them.

Read more

https://redd.it/122vfb0
@r_devops

Implementing gitflow in github

Hi,

I recently learned about the Gitflow branching model. Now I'm trying to implement it in a github repo.
There seems to be two common methods of using the default branch (let's call it main) in a simplified repo with two long-lived branches.

1. Using it as the development branch from which we push to a stable/release branch.
2. Using it as the stable/release branch and having a designated development branch.

Since Github suggests PRs to the default branch on push to other other branches, each of the above methods come with their own issue:

1. Pushing to feature/bug branches makes Gihub suggest PRs to main - that's what I want in this case, since main is my development branch. However, when I merge main --> stable a commit is created. Github will then suggest this commit to be merged back into main and I have an infinite loop.
2. main is my production branch so I only want to allow merging to it from the development branch. It's annoying having to manually change the target branch for every PR from feature/bug branches.

Are these knows problems and are there some good/best practice solutions to them?

From a google search it seems like it's not an option to turn off Github's PR suggestions. Is it possible to trigger them programmatically and choose the target branch?

I don't mind having some workflows to make it work but I do prefer to keep them as simple as possible. Would it be possible to create workflow like this pseudo-code?

on: push

env:

branch-name: ${{ github.ref_name }}

if startsWith(branch-name, (feature || bug))

suggest PR targeting dev

if startsWith(branch-name, (dev || hotfix))

suggest PR targeting main

https://redd.it/122zh2c
@r_devops

Will AI Replace DevOps Engineers? Exploring the Possibility

Is it possible for AI to take over the role of a DevOps engineer in the future? If so, what would be the potential impact on the job market and how can current DevOps engineers prepare for this shift?

https://redd.it/1231h2p
@r_devops

Can someone experienced expand on some bullets from a job description.

So I have been reading a lot of job descriptions and I recently transitioned into devops. One of my weak points is networking and I am trying to gain experience and knowledge in it. I see this bullet point a lot in job descriptions that I was hoping someone could really expand upon it and how I can get that knowledge.

Knowledge of networking concepts (TCP/IP, DNS, Load Balancers)
Proficient with using and debugging: networks, DNS, HTTP, TLS, load-balancing,

I have taken some Udemy courses on networking but it seems like such a hard subject to learn if I didn't take the courses in college. I just don't know how to get good at this. My job mainly has me doing pipelines, so all I really do is Docker/Teamcity/Ansible/Artifactory/Bash. How do I get better at the above? I want to accelerate my networking knowledge fast but I am not sure how.

https://redd.it/1233bfd
@r_devops

Is a round-robin load balancer robust enough?

I was wondering what else I can do to make the load balance effective.

https://redd.it/122yow9
@r_devops