How to migrate off Hashicorp Vault (transit engine specifically)

Has anyone migrated out of Vault transit engine to other services like AWS KMS? I'd love to hear about your journey.

https://redd.it/12jr7hg
@r_devops

Building Unprivileged Multi-Arch Images Using Kaniko and Gitlab CI

Hey r/devops!

I know people aren't big fans of having work blogs posted here, but I recently tackled something that didn't have much documentation online and wanted to share in case this could help someone else!

Amongst other things, I used some pretty cool, as of yet, undocumented gitlab CI features that allow you to change your gitlab runner's node selectors through environment variables in jobs.

https://arborxr.com/blog/developers-journal-building-unprivileged-multi-arch-images-with-kaniko-and-gitlab-ci/

https://redd.it/12jstaj
@r_devops

Running SuperTokens self hosted on Kubernetes at scale

Hi, I've always written my own Auth services and ensured they were cloud-native and scalable (horizontally). I'm about to start a new project and I'm drawn to trying out SuperTokens.


Anyone here with experience deploying supertokens on kubernetes?
Is the image (registry.supertokens.io) designed to support multiple instances running in parallel, connecting to the same database?

https://redd.it/12jvq4i
@r_devops

The Case for Function-Level Metrics: An observability sweet spot that balances debuggability, cost, and ease of use

Hi all, I was inspired to write this up after reading a post from Cloudflare about how they run Prometheus at scale. They mentioned some of the engineering challenges around managing the full life cycle of metrics, and I think function-level metrics address those challenges in some kind of neat ways.

https://fiberplane.com/blog/the-case-for-function-level-metrics

I'd love to hear what you all think!

https://redd.it/12jpv5l
@r_devops

How do you read books to learn?

I'm really curious. I haven't learnt anything till date just by reading books(and practicing). I always required some form of videos to learn. Neither do tutorials websites work for me. How do you read books?

I'm having to read this book in short duration(as I'm in job) ""The Linux System Administrator's Guide"", how do I do it. I have at most 2 weeks to read that book. 2 weeks, 9hrs per day.

https://redd.it/12jp4wi
@r_devops

Terraform, AWS, and user management

Does anyone have a good way of notifying new employees about their IAM account and what their temporary password is?

I see that there is aws_iam_user_login_profile but it doesn't actually send the temp password anywhere. Should I use something like local-exec and just send an email template?

Any thoughts would be appreciated.

https://redd.it/12k0i4r
@r_devops

What is your methodology when dealing with IaC on a cloud service?

I’m using Terraform as my IaC tool, and find it very hard to setup the right configs, especially for cloud services I don’t use often.

Is there any proven methodology? I tried doing things on the console to get hands dirty on the services, however sometimes the Terraform parts are more granular and I can’t get a 1-to-1 equivalence between the console and TF parts.

What are your tips?

https://redd.it/12k0ygn
@r_devops

How can I know the total hours I worked on starting from a specific sprint

If I joined a start up as part time and I worked on some tasks for the last 2 months, how can I know the full hours Ive worked on since then till now. Thanks

https://redd.it/12jojte
@r_devops

How to improve negotiation skill as a DevOps Engineer / Consultant

As a DevOps consultant in an organization, what steps can be taken to improve skills in consulting and negotiation when discussing topics such as infrastructure deployment strategies, resource and manday allocation, and other related matters with developers or clients? This includes situations where miscommunication may occur, resulting in errors or misunderstandings.

Is there any books, video, or resource that are good for this kind of thing? Thank you, appreciate if you reply this thread

https://redd.it/12kgph7
@r_devops

Bind server in AWS?

We're starting the beginnings of a migration from our on-prem data center to AWS and for the initial testing I'm trying to replicate what our on-prem config and stack look like before I start tearing it apart and refactoring.

Has anyone tried something like this? My workflow now is that the bind server does the zone transfers from my AD and services different dev/team environments. teama.company.com teamb.company.com.

Any thouhgts or suggestions on this?

​

Thanks in advance.

https://redd.it/12klk46
@r_devops

You do not need yet another CI tool for your Terraform.

IaC is code. It may not be traditional product code that delivers features and functionality to end-users, but it is code nonetheless. It has its own syntax, structure, and logic that requires the same level of attention and care as product code. In fact, IaC is often more critical than product code since it manages the underlying infrastructure that your application runs on. That’s precisely why treating IaC and product code differently did not sit right with us. We feel that IaC should be treated like any other code that goes through your CI/CD pipeline. It should be version-controlled, tested, and deployed using the same tools and processes that you use for product code. This approach ensures that any changes to your infrastructure are properly reviewed, tested, and approved before they are deployed to production.

One of the main reasons why IaC has been treated differently is that it requires a different set of tools and processes. For example, tools like Terraform and CloudFormation are used to define infrastructure, and separate, IaC only CI/CD systems like Env0 and Spacelift are used to manage IaC deployments.

However, these tools and processes are not inherently different from those used for product code. In fact, many of the same tools used for product code can be used for IaC. For example: 1) Git can be used for version control, and 2) popular CI/CD systems like Github Actions, CircleCI or Jenkins can be used to manage deployments.

This is where Digger comes in. Digger is a tool that allows you to run Terraform jobs natively in your existing CI/CD pipeline, such as GitHub Actions or GitLab. It takes care of locks, state, and outputs, just like a standalone CI/CD system like Terraform Cloud or Spacelift. So you end up reusing your existing CI infrastructure instead of having 2 CI platforms in your stack.

Digger also provides other features that make it easy to manage IaC, such as code-level locks to avoid race conditions across multiple pull requests, multi-cloud support for AWS & GCP, along with Terragrunt & workspace support.

What do you think of this approach? Digger is fully Open Source - Feel free to check out the repo and contribute! (repo link - https://github.com/diggerhq/digger)

https://redd.it/12koqev
@r_devops

Good Certs for New Relic?

Hi y'all,

I have a quick question :

I see that NR University offers some certs, are they good? If not is there any out there?


In any cases, which one would you recommend?


Thank you in advance for your help!

https://redd.it/12kqrua
@r_devops

Welcome Kubernetes v1.27 release, a.k.a. Chill Vibes

Announcing the release of Kubernetes v1.27, the first release of 2023!

This release consist of 60 enhancements. 18 of those enhancements are entering Alpha, 29 are graduating to Beta, and 13 are graduating to Stable.

Here's everything you need to know about the latest K8s release:
https://kubernetes.io/blog/2023/04/11/kubernetes-v1-27-release/

https://redd.it/12ksmti
@r_devops

DevOps Consulting Interview

Current second year CS student.

What are things that should be known before going into an interview for DevOps/devops consulting position, anything that would impress them?

It’s my last opportunity before internship possibilities for me are done, I appreciate the help, I really want this position.

Any help is extremely appreciated.

https://redd.it/12kqe9d
@r_devops

Is it possible to access environment level secrets from within the context of a variable block when using GitHub environments?

I have the following code block:

on:
push:
branches: dev
...

env:
SUBSCRIPTIONID: ${{ secrets.SUBSCRIPTIONID }}
USERNAME: ${{ secrets.USERNAME }}

jobs:
terraform:
name: "Execute Terraform"
runs-on: ubuntu-latest
environment:
name: dev
steps:
...

I have both SUBSCRIPTION_ID & USERNAME created as GitHub environment secrets.

At what point do these env vars get loaded in? If I've not yet specified the environment
(as it is set during the job configuration), will the environment level secrets get loaded in, or will GitHub actions expect the secrets to exist at repository level?

https://redd.it/12kvolz
@r_devops

Ideas for over engineered k8s cluster with most popular tech stack and best practicies.

The idea here is to create a "hello world" project to learn and train most common used tech stack and scenarios according to best practicies. I dont want to develop app itself - im not a developer so its just app and database (no redis, rabbit, etc). Im focused on devops side of things.

Currently my project looks like this:
APPLICATION:
- python-django "hello world"
- k8s resources - (deployment, service, ingress) generated by kustomize
- k8s resources are placed in separate "gitops" repository
- docker-compose for local development includes postgresdb

CI (gitlab.com):
- builds docker image
- uploads image to gitlabs registry
- updates "gitops" repository with new version number

CD (argocd):
- argocd tracks "gitops" repository

INFRASTRUCTURE (AWS => terraform):
- VPC and EKS cluster are created with terraform
- RDS postgres (single node for now)
- ebs and efs storage drivers are included
- aws-loadbalancer-controller as service and ingress controller
- external-dns is implemented to update route53 records according to ingresses and alb urls.
- mydomain.lan was created as private domain
- AWS Client VPN is implemented for accessing local resources (private subnets) - app, grafana panel, argocd panel etc.

INFRASTRUCTURE (EKS => argocd):
- argocd installs prometheus + grafana + loki from helm charts (values files are placed in "gitops" repo)

TODO:
- split argocd apps to two argocd installations (django-only and the rest of k8s resources)
- implement terragrunt to manage different environments
- implement karpenter to implement dynamic cluster scaling (already had problems because of lack of it)
- move RDS to multi-AZ cluster
- securing all (investigate sec-groups, implement network policies, resources limits, pod security groups etc, hiding k8s-API from public?)
- implement some vulnerability scanner (armo? starboard?)

I chose AWS because its most popular cloud now and im most familiar with it. No secrets are included in repositories. At first monitoring stack was terraformed but i had a lot of problems destroying it and ive heard that managing k8s resources with terraform is overall bad idea (because of many problems similar to those i faced).


So what else i could add or what can i change to follow current trends and best practice?

https://redd.it/12kq5je
@r_devops

Notification solution for utilisized software updates.

Hi,


Recently I've been thinking I need to decide on how I'm gonna manage being notified for updates and their changelog to software we are using (libraries e.t.c) before finding out there's a package available in the package system (apt, e.t.c). Does artifactory help?


I'd love to ask the community on their opinion and if you could help me out with ideas. One org I was working for had actual mailing list followed for some of the software but not everything was available. I am trying to use Feedly for those that have rss feeds but I'm not satisfied.


How do you do it?

https://redd.it/12kot0d
@r_devops

progressive deployment in event based systems

Hi folks, I was wondering if anybody could recommend resources on what the current thinking is on how to achieve progressive deployments in event based systems, specifically in kubernetes?

There are a ton of resources on gRPC and REST services but I'm really struggling to work out how to achieve this with an events based model. We're using Kafka in case that helps/changes anything. I'm not very experienced with Kafka but am comfortable with pub/sub more generally, in case I'm overlooking anything

https://redd.it/12l1dj1
@r_devops

What comes after devops?

This question is more for the very experienced devops/SRE's.. do you feel like you've done it all? What is your next step in your career? Seems like some sort of management or starting your own business are the only two options for advancing with this skill set. Maybe it's the universe telling me to do something completely new...

https://redd.it/12krmxa
@r_devops

Homelab VMware + Cloudflare DNS automation

Does anyone have any suggestion for how to best deploy / destroy VM ware esx vm's , manage cloudflare dns records + ideally VM os config management?

My current manual method is as such.

Example: I need 5 VM's , DNS (cloudflare) and VM config

1. Create the VM template and save in VMware
2. Clone 5 VM's from template with customize OS. Here is where I have manually increment hostname ++ hostname+1 & and ip xxx.xxx.xxx.xxx+1
3. Log into the Cloudflare web console and do the same thing add dns records +1 5 times

​

Ideally I am looking for 1 solution to be able to do the above like below

run some thing that lets me say deploy X machines with name = name+1 X times, create dns record with name.domain ip address +1 X times

​

I am pretty sure this can be accomplished via API calls since API is a single form of communication to both VMware esx and Cloudflare

I think another way is to use terraform which I believe has hooks into both

https://redd.it/12l7eiv
@r_devops

How does GitLab protect itself from developers forking to get around license restrictions?

I’m exploring open sourcing a project and I want to learn more about how GitLab protects itself.

GitLab is open source, and though some features are exclusive to EE, their source is available for everyone to see/download in the same repository.

In theory, a developer could choose to ignore the license and fork the codebase to enable all features, right? How does GitLab protect themselves from this risk?

https://redd.it/12l6nkg
@r_devops