How do you manage your Helm packages for production?

I'm interested in how the community manages Helm for production, do you use Terraform and Helm provider? Do you use CI/CD with the Helm CLI as the deployment mechanism?

I've used both, generally prefer a pipelined cli driven approach, besides the obvious benefits of having Helm deployments stored in state files, I don't see a strong reason for deploying it in Terraform (same for native k8s).

https://redd.it/ukxt2v
@r_devops

Jira Integrations

My company like a lot of them out there use Jira. I have no ability to influence that in any way to get away from the entire Atlassian suite, so this isn't an option. We run in a federated environment which makes authentication .... tricky with some things. My end goal is to have a single application I can use to get status on JIRA things, interact with our JIRA tickets/issues/stories from that same tool, and then pop back over to keep coding without breaking my workflow.

I'm actively moving my workflows to use Dendron with vscode to keep track of my thoughts and meetings and asks and my own personal tips/tricks/discoveries. The vscode JIRA plugin doesn't quite fit the bill. I can leave comments, but I can't say resolve issues since we require a reason for resolution or closure and that breaks the plugin. I looked at maybe mirroring into Trello, and using a plugin to drive trello but that'll be prohibitively expensive.

Is my only option to roll my own? I've looked at kanban boards in vscode and i don't think it would be super terrible to convert to MD for those and then back to jira-ese. Anyone else built a single pane of glass like this before?

https://redd.it/uklag9
@r_devops

Wrote up a post on backup and disaster recovery planning

Hey folks,

I'm relatively new to writing, but I am really enjoying trying to document up some of the things I've learned from being in Dev Ops for 10-ish years. One of my favorite topics is backups/disaster recovery planning and testing. I think it's because I'm a fairly anxious person, and having a solid backup program has really helped me sleep at night.

Designing a Backup and Disaster Recovery Plan

If you have feedback, other perspectives, please hit me up. I'm still new to writing. I'm planning on going through each of the facets listed here: The Many Facets of Infrastructure.

https://redd.it/ul7ixk
@r_devops

How to value equity vs up front cash?

Hey all, I'm doing a negotiation for the next step of my career and I'm struggling to really grasp the value and reality of stock options and job titles and what direction to go in it.


One company is giving a massive raise (30-35%), title increase to management and very little stock. The other is giving a much smaller raise (10%) and monstrous pile of stock. Both numbers are life changing but I guess I don't understand why company A would give so little stock or why company B would give so much or even how to properly evaluate the differences.


They're both unicorns valued over $1B & etc.

​

Any help from veterans of the process would be appreciated.

https://redd.it/ulsyux
@r_devops

How I learned to stop worrying and love the YAML

Following on from my Deploying Kubernetes Cluster in absurd languages is my next blog post,

How I learned to stop worrying and love the YAML

https://redd.it/ulrlsq
@r_devops

Have there been any pipeline exploits that have been made public?

I've been putting a lot of time recently into getting a firm grasp on pipeline security. But I'm curious about how much of a threat this is.

If code is being pushed to a private Github (with in-team code review), then being built via Github actions / cloud provider pipeline, doesn't that make the whole thing pretty secure from bad actors.

Yes there could be dependancy issues, there could be an NPM that has bad code pushed to it / exploits found etc.

I'm not arguing against checking / securing these things, I'm trying to understand the actual risk involved.

So I'm curious whether there have been any exploits / hacks etc that have used the pipeline to get data / do bad things in prod environments?

Edit: just to add, the responses so far just show me that I know nothing at all!

https://redd.it/umc6o6
@r_devops

Leaving a high-tech company that you don't feel aligned with

Have you ever worked in a high-tech product company, but in a sector that is not "aligned" with your values or whatever you like?

Let's imagine that you're working on really cool stuff, but you're also increasingly feeling that technique isn't enough if you don't feel aligned with the product / industry you work for.

Thinking about changing company cause you (a bit of) a sense of failure because you feel like you're giving up on something very cool.

Has this ever happened to you? What would you do in such a situation?

https://redd.it/umf3v7
@r_devops

My DevOps Checklist

https://www.thecodedmessage.com/posts/process-checklist/ is what I personally look for in a new project I’m joining and what I strive for in my own projects. What do you think is important that I don’t mention? Anything you guys disagree with?

https://redd.it/umi62m
@r_devops

Tart – open source virtualization for Apple Silicon

Tart is a virtualization toolset to build, run and manage macOS virtual machines on Apple Silicon. Built by CI engineers for your automation needs.

One of the most interesting/unique features is the integration with OCI-compatible container registries. Tart can pull/push virtual machines from/to a registry. This feature was inspired by OCI Artifacts initiative.

https://redd.it/umlcnp
@r_devops

Portainer users can now provision Kubernetes environments on cloud providers directly from within Portainer. In the latest release, provisioning is supported on DigitalOcean, Linode and Civo.

With the launch of Portainer 2.13, you can now provision Kubernetes environments on cloud providers directly from within Portainer. In the latest release, provisioning is supported on DigitalOcean, Linode and Civo. Get the details here.

https://redd.it/umy43h
@r_devops

DevOps career stalled.

I have worked in Devops / System administration for 10 years. I make an above average salary at a midsized company. We do not use k8s or have a strong CI pipeline. The release processes is an over complicated mess the infastructure has largely been developer driven until last year when the CEO decided to build a "devops" team.

​

I am unsure what to do to make things better the company is very sucessful and growing but a release processes that seems like it should be very simple ( we have a monolith and some workers) has become increasingly complex eg. some of a terraform code is generated by python ( which is fine it just does not seem necessary)

​

Not sure what to ask to be honest . I am finally a little frustrated at not making any major improvements after a year and not seeing a path forward.

https://redd.it/unsp8r
@r_devops

Cheapest managed Kubernetes ?

Hi, I'm looking for a cheap managed Kubernetes cluster at any Cloud Provider. I want to host 2 websites with very low traffic, the Grafana/Prometheus/Loki stack to train monitoring, and maybe working later with telemetry and tracing tools. I'll also add a LoadBalancer with nginx-ingress. What are the options for a cheap cluster ? DigitalOcean starts at ~25$/month, can it be cheaper ?

https://redd.it/unlgco
@r_devops

Introduction to Test Driven Development course

I've just released another course: Introduction to Test Driven Development on Coursera in addition to my Introduction to DevOps and Introduction to Agile Development and Scrum courses. This course is an adaptation of part of the graduate class that I teach at NYU on DevOps and Agile Methodologies. The focus is on the practice of Test Driven Development (TDD). That is, writing tests first for the "code you wish you had" and then writing the code to make the tests pass. TDD keeps you focused on the behavior of the code and ensures that your code is always working properly.

Throughout this course, I'll teach the workflows and techniques that I use every day as a software engineer. You will learn how to write test assertions to check the behavior of your code. How to use test fixtures to establish an initial state so that tests run in isolation and you get repeatable results. I will teach you how to use factories and fakes to generate test data, and how to use mocking to make sure that your tests are isolated from external systems, and to simulate error conditions to test your exception handlers. Finally. I will take you through the red, green, refactor workflow with hands-on sessions.

We tied something different in this course. I recorded live demonstrations of me implementing the testing concepts that I just explained in the video lecture. So you get to watch me code "live" and then you get to try your hand at it in the hands-on labs. The idea is to present a concept, demonstrate the concept, and then have you perform the concept in the lab to reinforce your learning.

I'm looking for feedback on how well you think this works and if I should continue with this approach in future courses. Here is a link to the course introduction video of me giving a course overview: https://www.coursera.org/lecture/test-and-behavior-driven-development-tdd-bdd/course-introduction-0yZqX

https://redd.it/uo3ser
@r_devops

High Steal Percentage on One VM

Hi All,

I'm running into an issue where one of my VMs has around a 40-50 steal percentage. I checked the rest of the VMs on the host and no others are really hitting the CPUs that hard, one is almost at 75%. Also none of the other VMs have any steal percentage. The one thing is that due to licensing we have to run several of these on the same CPU eg. 0-3 and 25-28. I checked the host and both CPUs are showing. Does anyone else know what else I could check?

I am using OVirt as the Hypervisor.

Thanks,

https://redd.it/uo1xce
@r_devops

Secret detection needs to be free, even for private repositories

My name is Nir. I am one of the three co-founders of arnica.io.

I’ve worn many hats in cyber security over the years – sys admin, pen-tester, security architect, and Chief Information Security Officer (CISO). What really gets me excited about my work is making security easy and effective for developers and ops teams! In my last role, at one of the top 3 FinTechs, following the attack against Solarwinds, the CEO asked me what we are doing to secure our software supply chain. I met with 15+ vendors, did a few POCs, but unfortunately each solution either increased operational cost or was too narrow in scope. I really wanted to buy a solution instead building it, but even the ones that hit the short list were rejected by my team (thank you guys!).

I also found that many fellow CISOs faced the same problem. This is when I joined forces with my incredible co-founders - Diko and Eran. They were seeing the same pain in their worlds (engineering and ops) too! As a starting point for Arnica, we researched every software supply chain attack since 2018, and based on our research, we found two primary root causes:

1. Improper access management to developer tools
2. Inability to identify abnormal identity and code behavior

We studied the anatomy of each supply chain attack and designed a product to effectively secure developer tool stacks with a DevOps-first approach.

We decided to release a couple of features for free:

1. Identify excessive permissions to source code starting with GitHub and Azure DevOps repos
2. Automatically generate & modify a CODEOWNERS file via pull request, based on the contextual behavior of the pull request reviewers
3. Secret detection and validation without modifications of the build pipelines for all repositories, public and private without any user-count limitations
4. Map GitHub users to your SAML/SSO provider. Also free forever.

Why are we giving away so much functionality for free? I believe Arnica can do well by doing good in the DevSecOps community. Our mission is to be the easy button for DevOps security, and we are going to deliver it.

More info about why we released unlimited secrets scanning for free: https://www.arnica.io/blog/secret-detection-needs-to-be-free-even-for-private-repositories

https://redd.it/uo6jdk
@r_devops

Where are you finding new job postings these days?

I'm a build and release engineer with about 3 years of experience (and 2 more doing tech support before that). I manage releases at my company and build and support our CI/testing pipelines and related tools. Basically, most of the "dev" part of devops. Not as much ops, which is part of why I want to move (in addition to generally plateauing in my role). I'm in the Bay Area.

I've been applying to DevOps and other build/release engineer positions. I've been hearing that the market is hot right now but so far that hasn't been my experience. In fact, I've been getting less traction than I was getting in 2019 applying to very similar roles as I am now, with three years less actual experience.

I'm not ruling out that something about me is the problem, but given the variance from last time (and the fact I don't think I look terrible on paper), I want to address the approach first. Last time around I used google's job search aggregator to find postings to apply to. This time, I'm doing the same but the tool is just turning up fewer suitable reqs and I wind up applying to only 5-10 ones per week, of which maybe around 1 or 2 respond.

I know it's a numbers game but I'm not hitting sufficient volume at the moment to matter. My question is, for those of you who are looking for jobs, what are the resources you've had the most success with? And is there anything obvious I'm doing wrong?

https://redd.it/uo7t1h
@r_devops

Source of truth for sensitive kv pairs

I want to migrate away from storing our sensitive KV pairs in environment variables for task definitions and lambda functions, and instead grab them from SSM Parameter Store.

Normally, I set the environment variables as part of the terraform configuration for deploying the infrastructure (so Sensitive TF Var in TF Cloud, used in the resource definition which then sets the environment variable in the lambda function etc), but obviously if I want to pull these values from SSM, it doesn't necessarily make sense to seed them from the same terraform config in the first place (especially if the credentials are used in multiple configs, such as API keys for observation platforms)

How do I get away from manually defining secrets in some kind of secure "variable" system (like TF Cloud), and move into a more structured source of truth (like a file where I define all of my secrets that then runs a job to add them to SSM when I update the file).

My first thought is to host a JSON or YML file in a git repo, and then have a post-commit job that updates SSM, but we use Github pretty exclusively and I don't feel like that's a very good idea.

Are there any tools for doing this? (industry standard single sources of truth for KV pairs that are the definition, not the store itself)

https://redd.it/unwa2q
@r_devops

How to share infrastructure details with application Helm charts

Not sure if I worded this question in a way that makes sense, but our situation is as such: we create AWS resources such as security groups, ACMs, IAM resources etc. which we need to reference in application helm charts, for example ACM certificate ARN in the ALB annotations. The infrastructure is created via Terraform. Making manual changes to the values.yaml file of a Helm chart seems like a tedious and error prone practice. Deploying Helm charts via Terraform in the same state as the infra doesn’t seem like a proper practice as it couples the infra to the app too tightly.

What should be the approach here?

https://redd.it/uoaqii
@r_devops

Microservice Governance - Resilience Patterns

Microservice Governance - Resilience Patterns

https://redd.it/uoapus
@r_devops

Looking for contributors for a K8s related open-source project

Hi, I'm building a community driven K8s related open source project. We need help, lots of help, not only with coding. Thank you.

Project: [Kubevious](https://github.com/kubevious/kubevious)
Many ways you can help: Requirements, Coding, K8s Experience, Docs, Walkthroughs
Where do we want to be: [Roadmap](https://github.com/kubevious/kubevious/blob/master/ROADMAP.md)
How to contact: Slack or [email protected]

https://redd.it/uo6oep
@r_devops

Branch Deployments with IssueOps and GitHub Actions 🚀

# Branch Deployments with IssueOps and GitHub Actions 🚀

I developed my first ever GitHub Action and wanted to share it with the community as it is open source!

## What is this?

A fully package GitHub Action that can be used in any repo on GitHub with just a few lines of code

## What does it do?

Enables branch deployments so you can get far far away from the "merge -> deploy" model of the past

## Links please!

- Here is a Medium writeup about the project: link
- Here is a link to the full source code, Action, and documentation: link

♥ open source

https://redd.it/uoesik
@r_devops