Managing Ephemeral Storage in EKS with Bottlerocket Nodes

Hi everyone,

I have a question about ephemeral storage in EKS. I'm using version 1.29 and Bottlerocket images for the nodes. I deployed the cluster with Terraform. I noticed that the nodes have two volumes: a 2GB volume for the OS and a 20GB volume for data, which is the ephemeral volume.
Sometimes I encounter disk pressure issues, causing pods to be evicted. Before increasing the 20GB volume, which I understand is the default value, I want to understand how EKS manages this volume. Specifically, does EKS always maintain the volume at 80% usage? It seems to consistently hover around that capacity, leaving little room to handle these disk pressure incidents. I want to ensure that if I increase the volume to 40GB, it won't always remain at 80% usage.

Can anyone provide insights or share their experiences with managing ephemeral storage in EKS?Thanks!

https://redd.it/1d2oa31
@r_devops

Anyone has already read SonarQube code smell report to generate dashboard in Grafana OSS

I'd like to show the applications that had more issues in SonarQube. Is there a plugin for it? Or maybe a service for be used with Infinity Plugin on Grafana?

https://redd.it/1d2q1hf
@r_devops

Peer Code Review Tool

Hi,
We’re currently using GitLab with Atlassian Crucible for our peer code reviewing. GitLab doesn’t look like they’ll ever support a single code review for multiple projects.
Crucible is now in ‘basic maintenance’ mode therefore we’d ideally want to move to an alternative tool which is actively maintained.

Is there anything out there that does this as a standalone product or is the solution to find a tool like BitBucket which handles the repositories and the code reviewing?

Thanks

https://redd.it/1d2qqz7
@r_devops

What do you use at your org? Azure DevOps or GitHub Team?

I am working with a team of devs and we have been using GitHub teams for some time. In the beginning it was pretty good. But now it is not enough anymore.

The team is not happy with the project management tools provided by GitHub.
Resources for GitHub actions are not enough, we are exclusively using self-hosted runners.
We are worried about GitHub packages and the storage limit set for it.
Basically we are paying 4 USD per user just for version controlling.

To all the devs working in organizations, what has your experience been like? Would you prefer working with Azure DevOps instead?

Oh and if it helps, we use Azure for all our resource requirements.

Thanks for your time!

https://redd.it/1d2s336
@r_devops

Platform as a Product 101

So, we wrote up an intro/overview of what the platform as a product strategy looks like in practice since that concept has taken off in the past few months. But is anyone on here actually utilizing PaaP in their own organization that's not one of the big tech companies? Any examples you can share of how you're utilizing it in your own org?

Or is it all cap/irrelevant? Would love thoughts from both individual devs and those who maybe are in charge of leading full dev teams. :)

Check it out, thanks!
https://thenewstack.io/platform-as-a-product-101/



https://redd.it/1d2pr82
@r_devops

Elastic search question

Hi 👋 I am not a developer (or anything technical). I am a bootstrap founder looking for solutions for e-commerce and retail in my vertical using tech.

I discovered Elastic today. I uploaded a csv of products (skus) to test and can query the index with their example console to search for, as an example, a product with “cane sugar” in the ingredients. I’d like to know:

1) what code do I put in the php console to ask the index to show products that “do not contain cane sugar” for example (sort of an “if then” query)
2) how do I make a query that reads the ingredients or attributes in a product and then spits out (matches it to) their description (stored in the csv)? As in if a customer wanted to scan a product to read the ingredients attributes?
3) (I’m a total rookie I know) but how do I “export” what I’ve built and make it look pretty on the front end for customers to use on our website (integrate it) or app?

I know I’m totally out of my league here and feel quite dumb but this has been bothering me and I know there’s a simple solution, I just don’t have the skill yet would like to try. Also, if you don’t ask you’ll never know.

https://redd.it/1d2vhip
@r_devops

Terraform does not respect `depends_on`?

Hey Terraform gurus.

It seems that `depends_on` does not affect order in which Terraform creates resources. A minimal example:

resource "null_resource" "foo" {
provisioner "local-exec" {
command = "./foo"
}
}

resource "null_resource" "bar" {
depends_on = [null_resource.foo]
provisioner "local-exec" {
command = "./bar"
}
}

Output of `terraform plan`:

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
+ create

Terraform will perform the following actions:

# null_resource.bar will be created
+ resource "null_resource" "bar" {
+ id = (known after apply)
}

# null_resource.foo will be created
+ resource "null_resource" "foo" {
+ id = (known after apply)
}

Plan: 2 to add, 0 to change, 0 to destroy.

Why is `bar` goes before `foo`? Am I missing something?


Update: when I do `apply`, order is correct. So, I can't trust `plan`?

My version:

Terraform v1.8.3
on linux_386
+ provider registry.terraform.io/hashicorp/null v3.2.2

https://redd.it/1d2wknx
@r_devops

How are elevateduser and elevatedpassword used in a Packer template?

I had assumed that the user defined in the Packer `winrm_username` setting was logging in as an admin and could perform tasks as such. It seems there is some nuance to that. One of the things that it fails to do is run `Enable-WindowsOptionalFeature` in a Powershell provisioner with "Access is denied".

I'm concluding that the error is because the user isn't actually logged in as an admin and therefore must invoke a new Powershell environment with elevated privileges. This means using the `elevated_username` and `elevated_password` settings.

The Packer documentation doesn't often make it clear how to use a setting. For the `elevated_` settings, my guess is that I can use the automatically generated password used by the `winrm_username` user. I'm using HCL2 so this is `${build.Password}`. Does this mean `elevated_username` would be the same as `winrm_username` or would I use Administrator with it having the same password?

Do I need to do anything to inform Packer that a command should be executed in an elevated environment or does it automatically do that when it sees that the `elevated_` settings are configured?

https://redd.it/1d2mjeu
@r_devops

Article: Stop running your tests with you CI/CD tool

Would love to hear peoples thoughts on this - agree? disagree? how are you solving test execution in a complex CI/CD environment? what problems are you facing in this context? how have you solved them?

https://thenewstack.io/stop-running-tests-with-your-ci-cd-tool/

thanks in advance!

/Ole

https://redd.it/1d37wdk
@r_devops

Programming and the military.

So I’ve been thinking about how military ideas could work in software development, especially the commander's intent concept. And slowly incorporating it into my startup.

This concept sets clear goals, letting teams be flexible in how they achieve them, even if plans change. Unlike the strict, fear-based leadership seen in movies, the modern military focuses on trust and empowering subordinates to make decisions.

In software development, this means explaining the “Why” behind tasks so developers get the value, communicating the “What” for guidance, and treating your team like a sports franchise by investing in their growth and fostering a supportive culture. This approach is said to boost morale, productivity, and job satisfaction.

I made a project management tool called goleko and I'm thinking about changing things up in my team. It's been a bit hard for me. Maybe this will help, I hope.

Have any of you used these principles in your teams? What strategies have you found effective in motivating and leading your teams?

https://redd.it/1d38uuy
@r_devops

Can I switch from cloud support to devops?

I am working in VMware cloud support role for almost 1 year and this is my first job but I'm already sick of this support role. I'm really much interested in devops and I have started learning some tools related to devops. But the question that really stands up in my mind always is can I really switch to devops from my current role or am I wasting my time. Can I get some thoughts about this? so that I can choose one right path with confidence.

https://redd.it/1d38ng9
@r_devops

Dose devops engineer take responsibility for making Dockerfiles?

Does devops engineer take responsibility for making Dockerfiles?
or Frontend/Backend engineers take responsibility for making Dockerfiles?


When devops engineer configure CI/CD pipeline, does CI pipeline include pulling source codes from git and building Dockerfiles?
or dockerfiles are just built by developers, and CI pipeline just pulls the dockerfiles?

https://redd.it/1d38204
@r_devops

Introducing Beta9 - Open Source Serverless GPU Container Runtime

https://github.com/beam-cloud/beta9

Beta9 lets you run your code on cloud GPUs with simple Python decorators. Think of AWS Lambda, but with a Python-first developer experience.

We'll instantly containerize your code and run it in a serverless environment — your nodes will automatically spin down when they're not being used.

You'll also get cloud-native development experience in which your code is instantly containerized to a remote container with the logs shipped back to your terminal.

We designed this platform for AI/ML workloads — you can run this platform on an existing cluster with GPU nodes, or add GPU nodes from any external cloud provider.

https://redd.it/1d3ckub
@r_devops

Future tools for devops

What are tools should I
master for the next 5 years ?

https://redd.it/1d3e7gs
@r_devops

How to setup integration tests in the project

Hello everyone,

I'm investigating how to set up integration tests in our project, we do not have any tests in place at the moment. We are using NodeJS/NestJS on the backend side, and everything is deployed as AWS lambdas using a serverless framework. We use Postgres SQL database.

What I've discovered so far is that there are a couple of ways how can you implement integration tests but there are still some uncertainties.

First option: Running integration tests during CI/CD

- The issue here is that our build already takes \~10-15-20 minutes. So it would additionally slow it down even more. Also, we would need to spawn a new environment (not all services, part of the services that are needed for integration tests, database, etc.) before the PR gets merged (since you can not run integration tests with new code since the code is not yet deployed). The issue with spawning environments is the serverless framework which does not have an option to do it in regular way (at least I haven't found it). For these purposes, we could use Terraform or AWS SAM but it takes time to switch to these IAAC tools and we need to start using integration tests as soon as possible.



Second option: Running integration tests nightly

- This looks like a better option to me personally since we would use the existing test environment and every night with deployed code we would run integration tests and fix them tomorrow. The downside is that the code that does not work might be deployed anyway, but I don't see an issue with it since we have weekly releases and once code is deployed to a specific environment we leave it there for \~7 days.

I would appreciate your opinion or advice on which practices to implement and apply and if you think that my favorite option (second option) is not the best approach here.

Also, I assume for NodeJS/NestJS we should just Jest testing framework?

https://redd.it/1d3ee74
@r_devops

Want to get into DevOps as a career.

I have over a decade of software development experience as well as a bunch of years building, hosting, and managing deployments of projects. I also have work experience in automation (HVAC). For my most recent couple of personal projects I have set up deployment pipelines on GitHub and Azure. I fell in love. I love the automation of the process and I want to look into pursuing it as a career option.

My first big questions are:
What pitfalls should I watch out for that could sink me before I get started?
What kinds of experience will employers be looking for and how can I get them?
How "entry level" friendly is devops as far as the job market goes?
What courses would be most beneficial to starting me on the path to being an expert?

https://redd.it/1d3fu53
@r_devops

Identifying Container Image vulnerabilities with Docker Scout

We all know, that Docker technology is great and brings us many advantages, but also, unfortunately, Docker images include many attack surfaces on different layers.

Every day, there are new vulnerabilities discovered in open source projects and maintainers are tasked with patching their software. \~30k new vulnerabilities discovered in 2023 alone.

So how can we mitigate this risk? One solution is vulnerability scanning and its integration into your development lifecycle.

This video shows how to use Docker Scout to secure your Docker images

https://redd.it/1d3insy
@r_devops

Kubernetes is not DevOps.

See it time and time again here with people asking what to learn or creating roadmaps for DevOps saying you "need" Kubernetes (K8S) but that's such an incorrect statement.

Are there companies out there that use it and want you to have it? Sure. Many of those will never actually leverage it in the way it is constructed. Often times you will spend more time fixing issues with Kubernetes than you would other solutions. Kubernetes is container orchestration at scale, At Scale means "large volumes", which many companies don't have. I have worked for larger companies whose clusters almost never scaled and when they did they were minor. To the point that they spent more on K8S management, upkeep, and spending (servers on prem or cloud cost) than they made with systems on K8S.

Does this mean those companies built it incorrectly? Not all the time, no, it more often mean s that they were sold a solution for a problem they didn't have.

I want to be clear, you should ensure you learn Containerization. That is a modern framework for handling any application that's been made for or updated to handle it. Containers can be deployed on a variety of systems, both Kubernetes and not, and should be leveraged as such.

This ultimately comes down to a business case. Are you scaling, or planning to scale, at high levels? Say over 100 or 500 or greater? Yep Kubernetes is probably the solution you want at that point. Are you scaling less than 100, or 50, or 10? Other solutions are capable to assist you here for less cost and work.

From the azure end alone you have Container Instances and App Services which can host containers natively for both Windows and Linux. They have scale settings to allow up or out. They also have a much lower management overhead and cost overhead associated to them. AWS and GCP both have similar offerings that can be used instead.

Should you learn Version Control (git in most cases), a programming language (optional usually depending on the role you want), monitoring, log handling, IaC, Containerization, scripting, CI/CD? Yes to almost all. Should you learn Kubernetes? Only if the jobs you want need it.

https://redd.it/1d3kexn
@r_devops

Docker to kubernetes

Any advice here much welcome.
My company are transitioning from a monolithic architecture to breaking this up into microservices for future scaling and deployment of individual components.

We already have broken a few services out of the monolith into independant microservices, but there are now plans to do all. Docker was OK to manage 5 microservices or so but will be eventually running 20+ so moving to k8s makes sense.

I have a bit of previous k8s experience, but when it comes to moving a monolith architecture to separate services, I am wondering what the best approach is.
Is the best route to break up the monolith and after everything is running as its own docker container, port this to k8s, or is it better to transition the current microservices to k8s and the break up the monolith and add to the kubernetes cluster as I go?

I ask because both routes involve a lot of work, and anyone with a bit of experience of doing something similar in the past can share any knowledge or do's/don'ts when approaching this. Thanks

https://redd.it/1d3lrui
@r_devops

Is KodeKloud Standard sufficient enough to get you to feel confident about getting into DevOps or is KodeKloud Pro needed for that?

Nothing more than the above inquiry. Right now I’m learning on my own but could use some guidance. I was told to check out KodeKloud. I hear great things about it but noticed they have a tier subscription. 450 dollars a year for Pro is steep. I would have to cut my personal entertainment budget in half to afford that. Meanwhile, the Standard tier is cheaper but I’m not sure if it’s worth getting.

https://redd.it/1d3mfv1
@r_devops

Survey on the Best Reverse Proxy for Instant Rollback in Docker Deployments: Traefik vs. Nginx vs. OpenResty

Hello r/devops!

I am conducting a research study to determine the best reverse proxy solution for implementing an instant rollback feature in Docker deployments. If you have experience with Traefik, Nginx, or OpenResty, your insights would be incredibly valuable. The survey will take about 5-10 minutes to complete, and your responses will help identify the strengths and weaknesses of each reverse proxy in real-world scenarios.

Thank you in advance for your participation!

Link to Survey

https://redd.it/1d3jxoo
@r_devops