SLI/SLO calculation via vmalert

https://victoriametrics.com/blog/rules-replay/

https://redd.it/10qkc9p
@r_devops

securing github actions from the inside

Having trouble finding info about this from searching, I imagine it's because I don't have the right search strings...

I need to implement a github action on our development repo (where devs play) that will dynamically retrieve secrets from Hashicorp Vault and do a little work on AWS on our behalf. That's easy. I can do that.

The thing that's rubbing me the wrong way is that anyone (ie devs) can effectively push a new branch to this repo that says 'on push to my branch LOLOLOL run this arbitrary workflow that retrieves the secrets from Hashicorp Vault and outputs them in plaintext, or hijacks the AWS session to do silly things.'

Okay, yes, first line of defense is the principle of least privilege - reduce the blast radius by using Vault policies and AWS IAM. But, like, I don't want a blast at all.

Is there a clever way to prevent developers from running arbitrary workflows? I'm envisioning some sort of "only run github actions from the main branch" or something, so the developer would at least have to get their workflow past a PR.

https://redd.it/10ql5y8
@r_devops

Preparing for the interview about GCP(Google Cloud Platform )

Hello folks! I'm a beginner DevOps engineer. How I can prepare for the interview about GCP? For which part do I need to more focus? In GCP what kind of questions, they can give me. Thank you!!!

https://redd.it/10qkb6n
@r_devops

Help needed Improve CI for multi repo java application

Hi everyone,
in my company I was asked to contribute to our process for CI. But I don’t really have a good idea.

We have a monolithic main software and it is using multiple layers of dependency in a final build.

Monolith -uses-> Module -uses-> Library


Monolith -implements-> Library

We use Jenkins to build artefacts that will be provided with JFrog for other builds.

We have jobs in Jenkins that are used when a merge request is created that executes automated test and after merging a publish job will be started.

Currently we are going to publish each change in the master of a module or lib manually and do some manual testing with the monolith.

I don’t think that only automating the job start of the current ones will be enough and I hope you can give me a suggestion that might be more a best practice.

Edit: format & typos

https://redd.it/10qjl18
@r_devops

Looking for a standalone GitOps controller?

I'm looking for an open source component that can pull changes from a Git repository, can be configured to trigger scripts when arbitrary changes are detected (e.g. config files, Terraform, et c) and can report progress and perhaps metrics. Ideally, it should be able to run as a container in e.g. Fargate or Azure Container Apps. This is essentially the Flux source controller as a standalone component and with no Kubernetes dependency. Does anyone know of such a tool?

https://redd.it/10qo9mf
@r_devops

Terraform / ansible

Hello people :) , do you guys know any free training website to learn terraform and ansible ? I just got my azure fundementals certification and need to boost my skill with terraform specially . Thank you

https://redd.it/10qp99h
@r_devops

many services, single host: what tooling to use? Ansible, terraform, docker, bash scripts?

Hi everyone,

I'm posting here to ask some advice because I don't see my particular setup being discussed often (if ever) in DevOps forums or articles. Some background: I'm a freelance web developer that builds a lot of different web apps for different clients, but that are each small enough to be able to host on a single server, that is: I'm hosting all apps I ever made on a single DigitalOcean host. Each service is contained within a Docker container and all networks, env variables and secrets are defined in several Ansible playbooks.

The problem is, with every service/app I add, my collection of Ansible playbooks becomes more a mess. There's dependencies between playbooks, for example: all services rely on a single Traefik reverse proxy (or nginx, I'm thinking of switching back to that) being started and my vars-files are starting to become a big list of variables. Aside from that, writing an Ansible playbook where I'm just starting containers feels like a bit of a pain compared to writing a docker-compose.yml and having a quick look at Terraform makes me think that this declarative approach is much easier to manage and reason about than the way Ansible does things.

Is there a better approach to doing this? Should I seriously look into Terraform? Kubernetes seems way overkill as I don't need any orchestration as the trafic I deal with is so minimal. Would it be better to just manually copy docker-compose files and nginx configs to my host and keep each apps config in their own repo? Should I try to make better use of Ansible roles and split out my configs in that way, defining common components like firewall and reverse proxy in a common role? Curious to hear how other people would approach this!

https://redd.it/10qpri2
@r_devops

What are the best devops stacks these days?

I'm working at a small company and I've been recently upping my skills especially in the cloud department and I'm pushing to go serverless, probably by using Google Cloud Run and AlloyDB. This has led me to question our use of Azure DevOps since it was here before I got hired and I never got to see alternatives.

I've heard that nowadays the best integrated solutions are either Gitlab with its CI/CD or GitHub Actions but it's unclear whether it's fully mature. My main issue with ADO is obviously the cluttered UI and the convoluted pipelines but that may be the nature of it. Also Microsoft which makes me more interested in Gitlab as a default but I don't want to choose a lesser solution out of spite for MS

Our DevOps is relatively simple, we manage projects using Agile, backlogs and the usual stuff. We have a CI pipeline that compiles everything, runs unit tests, publishes artifacts and then deploys on our on-prem windows server IIS instance for testing. Once it's validated main gets PRed into a release branch and after manual approved everything gets deployed to our production VPS.

In the future we will move for serverless and thus the pipeline will also build docker images and upload to GCR for production. For testing we will either retain an on-prem solution or find something else.

So I guess I'm looking for the pros & cons of each from actual user perspective in the context of our workflow

https://redd.it/10qq6om
@r_devops

How do you cope with your employer forcing you to use their shit ass laptop ?

It’s frustrating limiting, I can’t connect multiple monitors to it, and I can’t install any basic tool on it that are from my point of view absolutely necessary.

https://redd.it/10qoukj
@r_devops

´include:´ keywords limit (250) in gitlab ci pipeline

Hello DevOps community,

I just switched to DevOps and trying to move ahead in this exciting field. As part of my new tasks, I must look after a HUGE pipeline that creates GCP projects for the whole company using terraform as IaC tool. A week ago, it reached the include: keyword limit, despite the CI files are already DRY'd as much as possible; mainly do to the fact that the company is huge and there is a lot of demand for new GCP projects everyday, and for every project we create an individual ci file that performs terraform init, plan and apply. An idea that came to my mind is, to write a simple code (maybe in python or go), that on every pipeline run checks wether a file has changed or not, and depending on the changes generate a new ci file that triggers the whole init, plan and apply flow again, but it may take me too long to develop that code. Any suggestion on how to approach this issue is welcome and really appreciated!

https://redd.it/10qoyvs
@r_devops

Best DevOps blogs to follow?

In an ever-changing industry, keeping your knowledge in point is very important. This Subreddit is a great way to keep up to date, but I was wondering what are the best DevOps blogs to follow?

Currently, I follow the AWS DevOps blog to keep with new services and architectures.

Please share your recommendations.

https://redd.it/10qvf4b
@r_devops

How to protect your database (from yourself)

Hi everyone,

My name is Rotem, I'm one of the creators of Atlas, an open-source database schema management tool that we're building at my company.

We created the project out of frustration from the amazing gap between the maturity of DevOps tooling in many layers of the software stack and what we could use for our databases.

One of the areas we focus on is increasing dev-velocity by making schema changes much much safer by automating the verification of schema changes. To do this, we've built a bunch of infrastructure for doing static code analysis of schema changes.

You can catch a quick demo of it in a video I uploaded today:

How to protect your database (from yourself) - YouTube

Since we've all experienced that database outages or failed migrations are a particularly painful subject, I wanted to ask, how does your team do CI for database schema changes?

https://redd.it/10qvd83
@r_devops

prometheus + grafana stack in k8s. is it a silver bullet?

I'm <3 years junior DevOps and I have experience with 2 companies that both have 20\~30 devs. Both companies use prometheus + grafana stack for their monitoring especially k8s nodes & pods. However having used these stack I got some curious that are other companies using same thing as me. Because these were not that bad in most cases though, but some trivial bugs or several-years-old issues gave me a hard time. (or maybe they were all my fault)

And it seems there are 4 major way to deploy prometheus based monitoring stack in k8s..
prometheus-operator/prometheus-operator,prometheus-operator/kube-prometheus,bitnami/charts/kube-prometheus,prometheus-community/helm-charts/kube-prometheus-stack,... they really confuse me

\---

So I wonder

\- Are you using prometheus + grafana for k8s monitoring?

\- If yes, are you using them without any problem?

\- If not, what are you using now?

&#x200B;

Thanks

https://redd.it/10qy7j9
@r_devops

What’s a project, personal or professional, that you can never find the time/resources for? Why?

For me it would be to update my personal website with a blog portion that contains well documented posts of my completed projects. I have the projects done. I have the knowledge and experience. I know what to do to get this all completed. I don’t really have any excuse, especially considering that if the big project was for work I’d already have the documentation completed.

https://redd.it/10r0i5w
@r_devops

Monthly 'Shameless Self Promotion' thread - 2023/02

Feel free to post your personal projects here. Just keep it to one project per comment thread.

https://redd.it/10r0ixm
@r_devops

Best Free Full-Stack monitoring Suite?

We currently have a hybrid Datacenter-AWS environment. We are currently using Dynatrace, but it is too expensive. What suite do you use to monitor Applications, Servers, Containers, networking, DB, logs, etc?

https://redd.it/10r1ca2
@r_devops

Test environment for branches?

Is there a standard procedure for manual testing different branches in a repo?

For example I have Azure DevOps server and I have an environment set up for my dev branch (site, db) so people can click stuff in the current general state of the app.

If I start implementing a feature on a separate branch, how would I handle the environment? Like say I want to show off the progress to see if this is functionally the right direction or whatever.

I would need to provision a new environment, frontend, backend, db, domains, etc. How would I achieve this in an automated manner?

https://redd.it/10r043l
@r_devops

looking for ways to learn AWS or other cloud service providers in a production environment

I have good experience managing infrastructure in a private cloud for more than 5 years, with strong k8s and docket fundamentals. However my job didn't involve me working on any public cloud providers. Every job I am looking for asks for strong experience in atleast one of the CSPs. So what is a good way to learn and use one of the CSPs at production level so that I can justify my experience. Don't want just introduction to AWS or something else.

https://redd.it/10r5tzt
@r_devops

Code Automation Help

Hi all,

I'm trying to figure out what the most efficient way is to automate my app deployment process. Currently using GitLab to store code for an app and using an AWS EC2 instance that it gets deployed on, I'm currently limited to EC2 only. My current process is, I have a Packer script that builds out the AMI image and includes a line to clone my git repo for the app code.

This is a manual process each time I update so I'd like to change it where whenever new code is committed in GitLab, it can automatically kick off the Packer script to create the newly updated AMI image. Is this possible to integrate with GitLab CI/CD or any other ways of doing it?

Thanks

https://redd.it/10r83zs
@r_devops

Frequent Targetgroupbinding reconciliation on an EKS Cluster

Hi Everyone,

Viewing my event logs, my EKS cluster tends to perform a lot of reconciling (less than 15mins continuously) by Targetgroupbinding with exposed service containers.

I understand it has to perform leader board election but checking if this is normal behavior

https://redd.it/10r9jst
@r_devops

What to expect for first DevOps interview? (with DoD-based contractor)

Hi everyone, while I think I have a federal "DevOps" role in the bag (seems to be taking forever for a solid offer letter though). I was invited by two DoD based companies to do an interview with them for mid-DevOps roles. I'm DevOps now, I think?

I use docker, Ansible, I create bash or python scripts to lockdown systems or to install sw prereqs, I do site reliability (but without monitoring software), I troubleshoot finicky service scripts, I ACAS scan systems for security vulnerabilities, I create and backup Linux systems, and of course more. So I do DevOps stuff but then I look at these job listings I applied for and I don't know cloud (@lat least no work experience), I don't know load balancers, I haven't produced code for a project, I don't know kubernetes, I don't practice leetCodes. Am I walking into a mind field for this interview? I have "PowerShell" scripting on my resume but now that I think of it, I have scripted anything in PowerShell since I made my few "Windows baseline configuration" scripts 2-3 years ago. 😶 I just use them quite often.

DevOps broadness is awesome when you're on the jobs but nerve-racking when you are trying to land a position. My current position was for Systems Administration but I kind of got thrown onto a very fast moving agile development project where I learned my above skills.

So what is expected of me if I don't know some of these technologies, what are some ways you guys navigate unknown technology questions in interviews?

https://redd.it/10ra89y
@r_devops