How to improve negotiation skill as a DevOps Engineer / Consultant

As a DevOps consultant in an organization, what steps can be taken to improve skills in consulting and negotiation when discussing topics such as infrastructure deployment strategies, resource and manday allocation, and other related matters with developers or clients? This includes situations where miscommunication may occur, resulting in errors or misunderstandings.

Is there any books, video, or resource that are good for this kind of thing? Thank you, appreciate if you reply this thread

https://redd.it/12kgph7
@r_devops

Bind server in AWS?

We're starting the beginnings of a migration from our on-prem data center to AWS and for the initial testing I'm trying to replicate what our on-prem config and stack look like before I start tearing it apart and refactoring.

Has anyone tried something like this? My workflow now is that the bind server does the zone transfers from my AD and services different dev/team environments. teama.company.com teamb.company.com.

Any thouhgts or suggestions on this?

​

Thanks in advance.

https://redd.it/12klk46
@r_devops

You do not need yet another CI tool for your Terraform.

IaC is code. It may not be traditional product code that delivers features and functionality to end-users, but it is code nonetheless. It has its own syntax, structure, and logic that requires the same level of attention and care as product code. In fact, IaC is often more critical than product code since it manages the underlying infrastructure that your application runs on. That’s precisely why treating IaC and product code differently did not sit right with us. We feel that IaC should be treated like any other code that goes through your CI/CD pipeline. It should be version-controlled, tested, and deployed using the same tools and processes that you use for product code. This approach ensures that any changes to your infrastructure are properly reviewed, tested, and approved before they are deployed to production.

One of the main reasons why IaC has been treated differently is that it requires a different set of tools and processes. For example, tools like Terraform and CloudFormation are used to define infrastructure, and separate, IaC only CI/CD systems like Env0 and Spacelift are used to manage IaC deployments.

However, these tools and processes are not inherently different from those used for product code. In fact, many of the same tools used for product code can be used for IaC. For example: 1) Git can be used for version control, and 2) popular CI/CD systems like Github Actions, CircleCI or Jenkins can be used to manage deployments.

This is where Digger comes in. Digger is a tool that allows you to run Terraform jobs natively in your existing CI/CD pipeline, such as GitHub Actions or GitLab. It takes care of locks, state, and outputs, just like a standalone CI/CD system like Terraform Cloud or Spacelift. So you end up reusing your existing CI infrastructure instead of having 2 CI platforms in your stack.

Digger also provides other features that make it easy to manage IaC, such as code-level locks to avoid race conditions across multiple pull requests, multi-cloud support for AWS & GCP, along with Terragrunt & workspace support.

What do you think of this approach? Digger is fully Open Source - Feel free to check out the repo and contribute! (repo link - https://github.com/diggerhq/digger)

https://redd.it/12koqev
@r_devops

Good Certs for New Relic?

Hi y'all,

I have a quick question :

I see that NR University offers some certs, are they good? If not is there any out there?


In any cases, which one would you recommend?


Thank you in advance for your help!

https://redd.it/12kqrua
@r_devops

Welcome Kubernetes v1.27 release, a.k.a. Chill Vibes

Announcing the release of Kubernetes v1.27, the first release of 2023!

This release consist of 60 enhancements. 18 of those enhancements are entering Alpha, 29 are graduating to Beta, and 13 are graduating to Stable.

Here's everything you need to know about the latest K8s release:
https://kubernetes.io/blog/2023/04/11/kubernetes-v1-27-release/

https://redd.it/12ksmti
@r_devops

DevOps Consulting Interview

Current second year CS student.

What are things that should be known before going into an interview for DevOps/devops consulting position, anything that would impress them?

It’s my last opportunity before internship possibilities for me are done, I appreciate the help, I really want this position.

Any help is extremely appreciated.

https://redd.it/12kqe9d
@r_devops

Is it possible to access environment level secrets from within the context of a variable block when using GitHub environments?

I have the following code block:

on:
push:
branches: dev
...

env:
SUBSCRIPTIONID: ${{ secrets.SUBSCRIPTIONID }}
USERNAME: ${{ secrets.USERNAME }}

jobs:
terraform:
name: "Execute Terraform"
runs-on: ubuntu-latest
environment:
name: dev
steps:
...

I have both SUBSCRIPTION_ID & USERNAME created as GitHub environment secrets.

At what point do these env vars get loaded in? If I've not yet specified the environment
(as it is set during the job configuration), will the environment level secrets get loaded in, or will GitHub actions expect the secrets to exist at repository level?

https://redd.it/12kvolz
@r_devops

Ideas for over engineered k8s cluster with most popular tech stack and best practicies.

The idea here is to create a "hello world" project to learn and train most common used tech stack and scenarios according to best practicies. I dont want to develop app itself - im not a developer so its just app and database (no redis, rabbit, etc). Im focused on devops side of things.

Currently my project looks like this:
APPLICATION:
- python-django "hello world"
- k8s resources - (deployment, service, ingress) generated by kustomize
- k8s resources are placed in separate "gitops" repository
- docker-compose for local development includes postgresdb

CI (gitlab.com):
- builds docker image
- uploads image to gitlabs registry
- updates "gitops" repository with new version number

CD (argocd):
- argocd tracks "gitops" repository

INFRASTRUCTURE (AWS => terraform):
- VPC and EKS cluster are created with terraform
- RDS postgres (single node for now)
- ebs and efs storage drivers are included
- aws-loadbalancer-controller as service and ingress controller
- external-dns is implemented to update route53 records according to ingresses and alb urls.
- mydomain.lan was created as private domain
- AWS Client VPN is implemented for accessing local resources (private subnets) - app, grafana panel, argocd panel etc.

INFRASTRUCTURE (EKS => argocd):
- argocd installs prometheus + grafana + loki from helm charts (values files are placed in "gitops" repo)

TODO:
- split argocd apps to two argocd installations (django-only and the rest of k8s resources)
- implement terragrunt to manage different environments
- implement karpenter to implement dynamic cluster scaling (already had problems because of lack of it)
- move RDS to multi-AZ cluster
- securing all (investigate sec-groups, implement network policies, resources limits, pod security groups etc, hiding k8s-API from public?)
- implement some vulnerability scanner (armo? starboard?)

I chose AWS because its most popular cloud now and im most familiar with it. No secrets are included in repositories. At first monitoring stack was terraformed but i had a lot of problems destroying it and ive heard that managing k8s resources with terraform is overall bad idea (because of many problems similar to those i faced).


So what else i could add or what can i change to follow current trends and best practice?

https://redd.it/12kq5je
@r_devops

Notification solution for utilisized software updates.

Hi,


Recently I've been thinking I need to decide on how I'm gonna manage being notified for updates and their changelog to software we are using (libraries e.t.c) before finding out there's a package available in the package system (apt, e.t.c). Does artifactory help?


I'd love to ask the community on their opinion and if you could help me out with ideas. One org I was working for had actual mailing list followed for some of the software but not everything was available. I am trying to use Feedly for those that have rss feeds but I'm not satisfied.


How do you do it?

https://redd.it/12kot0d
@r_devops

progressive deployment in event based systems

Hi folks, I was wondering if anybody could recommend resources on what the current thinking is on how to achieve progressive deployments in event based systems, specifically in kubernetes?

There are a ton of resources on gRPC and REST services but I'm really struggling to work out how to achieve this with an events based model. We're using Kafka in case that helps/changes anything. I'm not very experienced with Kafka but am comfortable with pub/sub more generally, in case I'm overlooking anything

https://redd.it/12l1dj1
@r_devops

What comes after devops?

This question is more for the very experienced devops/SRE's.. do you feel like you've done it all? What is your next step in your career? Seems like some sort of management or starting your own business are the only two options for advancing with this skill set. Maybe it's the universe telling me to do something completely new...

https://redd.it/12krmxa
@r_devops

Homelab VMware + Cloudflare DNS automation

Does anyone have any suggestion for how to best deploy / destroy VM ware esx vm's , manage cloudflare dns records + ideally VM os config management?

My current manual method is as such.

Example: I need 5 VM's , DNS (cloudflare) and VM config

1. Create the VM template and save in VMware
2. Clone 5 VM's from template with customize OS. Here is where I have manually increment hostname ++ hostname+1 & and ip xxx.xxx.xxx.xxx+1
3. Log into the Cloudflare web console and do the same thing add dns records +1 5 times

​

Ideally I am looking for 1 solution to be able to do the above like below

run some thing that lets me say deploy X machines with name = name+1 X times, create dns record with name.domain ip address +1 X times

​

I am pretty sure this can be accomplished via API calls since API is a single form of communication to both VMware esx and Cloudflare

I think another way is to use terraform which I believe has hooks into both

https://redd.it/12l7eiv
@r_devops

How does GitLab protect itself from developers forking to get around license restrictions?

I’m exploring open sourcing a project and I want to learn more about how GitLab protects itself.

GitLab is open source, and though some features are exclusive to EE, their source is available for everyone to see/download in the same repository.

In theory, a developer could choose to ignore the license and fork the codebase to enable all features, right? How does GitLab protect themselves from this risk?

https://redd.it/12l6nkg
@r_devops

Github Codespaces is blowing my mind

I can't believe this sub isn't filled with posts about this tool. Maybe I'm easily impressed, but I went down the codespaces / dev container rabbit hole last week and I haven't been able to stop talking about it.

If you haven't used it before, the TL:DR is that you can specify a dockerfile + some additional dependencies at the root of any repo on github, and, assuming you have codespaces enabled, you can spin up that container in seconds on Azure and develop in it with VSCode. 1 click dev environments, no local dependencies -- just a browser, a github account, and a few json/yaml files.

I've been at 5 places in the last 7 years (3x 2 year gigs, 2 very short gigs) and the onboarding / dev environment setup has always been the bane of my existence. I've done virtualbox, docker, wsl, podman, Windows Sandbox, EC2 / Workspaces, Azure Virtual Desktop, and on and on... Each of those had some sort of limitation that made them clunky to use.

Not codespaces. Nothing has ever felt this effortless.

I wrote a base config for AWS to launch a dev environment that authenticates to IAM Identity Center with minimal manual steps: https://github.com/robbycuenot/codespaces-aws-granted

Have any of you adopted Codespaces for your org?

https://redd.it/12j13we
@r_devops

Troubleshooting slow connections

Hey all,

I'm on a team without a ton of experience in how internet traffic is shaped. Ive been monitoring some synthetic traffic to our API gateway and have it return a 200. I chose this as it should have the least amount of variation from inside our application. We are seeing some pretty high latency from it, like 100ms, which seems crazy for just a 200.

How do we go about troubleshooting a problem like this?

https://redd.it/12lef97
@r_devops

Permissions on Argo Workflows

I’ve loved Argo CD and I’m experimenting with Argo Workflows. The specific use case I have is I want to enable developers to run specific jobs in production where they typically don’t have permissions. The problem is, the way I understand it, in order to be able to make jobs you need permission to make a workflow CRD but there are no controls as to what goes in the workflow. I’d like to open it up to run one or two specific jobs, not anything. Is there a way to close this off or am I misunderstanding something? I’m tempted to write a tool that sits in front of the Argo Workflows api that does have the permissions restricted the way I want, but that sounds like a pain.

https://redd.it/12le8h9
@r_devops

Use GitOps for Efficient DevOps

Hey everyone! We're a startup team focused on developing a cloud-native time series database. With no historical operational burdens, we eagerly adopted GitOps from day one to enhance our DevOps efficiency. We're excited to share our experiences and invite open discussions on this topic.

In this article, we'll explore how Greptime utilizes tools like Terraform, Kubernetes, and ArgoCD to establish an effective DevOps workflow. We'll dive into the connection between IaC and GitOps, explain the key components and benefits of GitOps, and discuss the critical technical decisions made throughout our implementation journey.

Full article here: https://greptime.com/blogs/2023-04-13-greptime-gitops

https://redd.it/12ljmaf
@r_devops

DevOps interview: Picking random facts from AWS docs and use them as interview question.

I think this is a bad technique. What is your opinion?

https://redd.it/12lkgwp
@r_devops

How do you solve multi-dev environment problem?

I have 4 environments: dev, test, staging and prod.

Things are working all good except one thing. i.e. developers waiting for dev environment while someone else is using it.

Tech stack for reference:
TeamCity for the CI
EKS for the backend
Cloudfront and S3 for the frontend

I’m willing to change anything of the above if required.

Please comment if you’re solving this issue. Itd be very useful.

https://redd.it/12lnuit
@r_devops

Best authentication mechanisms for different microservices communicating with each other intra-VPC and across VPCs.

We have a variety of different microservices, some within the same VPCs, and some across different VPCs.

Our go-to authentication mechanism has just been basic auth, and then making HTTPS calls across the public internet, but given that these are all our internal resources, we figure there should be a way of doing this that's rather more secure than that.

We've already decided that VPC peering will be used here to facilitate inter-VPC communication.

Since we're a bit new to having services in separate VPCs, we are wondering a couple of things:

1. One VPC peering is set up, what else needs to be changed to ensure the communication between the machines happens through the peering connection, as opposed to over the public internet.
2. Our current authentication method involves using Basic Auth. For REST API communication between our various services, what tends to be the most industry standard way to ensure authentication.

https://redd.it/12lrraw
@r_devops

Kubectl Cheat Sheet: Navigating Kubernetes like a Pro

Link to the article: https://devoriales.com/post/226

https://redd.it/12ltp3w
@r_devops