In the world on automation, which language gets requested the most between Python and Go? Do you think that will change?

Nothing to add to the title. It’s pretty straight forward.

https://redd.it/1g0tph0
@r_devops

The dillema: QA'd weekly releases or release ever hour with extensive integration tests?

I've run two startups for about 15 years, then joined 3 other startups that I wasn't a founder of.

When I ran my own startups I migrated to a system of extensive testing, and continuous releases.

The tests sort of made it impossible to push a broken build. When one snuck through, I'd do a post mortem analysis, and then shore up our tests to prevent that from ever happening again.

However, the startups I joined have had a terrible release process.

All of them had NO tests and just a "let's do everything very very very very carefully" approach to software engineering (which basically doesn't scale).

What ended up happening, is that once you pick this fragile deployment method, you're basically stuck.

It takes a ton of effort and changing the team to migrate BACK to continuous delivery.

The current startup I'm at has a weekly release schedule.

This costs us tons of time in lost productivity:

- more managers have to work around the timing of each release.

- engineers have to triage tickets making sure features and PRs are merged at the right time.

- it causes our PRs to be HUGE for various reasons which also snowballs into even bigger PRs because devs have to get everything into one big release.

- we don't invest in any testing so I can't personally to TDD and deliver high quality code. It makes dev a huge pain for me and it's not enjoyable.

- if we screw up, we have to wait until another push

- if an urgent bug fix happens, we have to cherrypick it and get it into prod.

Is continuous delivery accepted yet as best practice?

The issue I have now is that I'm going to try to convince our CEO that we have to make this change.

What I'm worried about here is that it's hard to migrate back to a continuous delivery practice and it will require time and effort.

If the project fails, it's going to be on my shoulders and the fingers will point towards me. Plus I'm going to have to allocate extra time and effort to make sure it works fine.

What I'd like to do is just tell him that this isn't a controversial suggestion - it's industry standard best practice.

But the question, is it? Do other startups take this approach or am I just biased because it's my personal preference?

Thanks guys!

https://redd.it/1g0vtys
@r_devops

how to manage secrets in gcp to have stateless projects?

Hi,

I have a shared project called tooling and several environment projects: dev, staging, prod

My idea is to have all dev,staging,prod stateless

But then how should I handle the secrets? we will be using mostly GKE for the apps.

I am using Terraform CDK with typescript.

My ideas are:

1. in Github actions store the secrets and in the app deployments create gke secrets replacing the map in every deployment so it does create/updates all the time and in one operation.
2. in Github actions store the secrets , create the secrets in Google secret manager at the app deployments, the problem is to handle the initial value and the updates, check if exists then create else create one version every time the app deploys? is not too many versions?
3. keep all secrets in the shared tooling project, for all the environments?

Any ideas? thanks

https://redd.it/1g0x658
@r_devops

How popular is this Wolfi base image as alternate to Alpine? Do you use it in production?

I am exploring ways to move away from Alpine as I encountered some DNS problem with it recently. Is Wolfi a good alternative base image? Please don't suggest bloated Debian and Ubuntu

https://redd.it/1g0wgky
@r_devops

Advice for new manager of a small team (3)

Hi - new manager here.

My background is SQL, python, powerBI automating data collection, creating reports and dashboards.

I used to work solo, on an island, with no real experience maintaining code for anyone but myself.

I’ve got my team a repo and we are working to deploy a pipeline and I need help with how we are going to manage our branch / branches…

Right now our plan is to spin off new branches for work items and push them into main when they’re ready.


We meet Monday + Thursday to chat formally.

I have set up coding guidelines and a repo.

What else can I do?

https://redd.it/1g0zrqo
@r_devops

Transition to Solution Architect?

I have a few years of experience in DevOps now.

I don't have a cloud cert under my belt just yet. Recently started working on getting AWS Solution Architect Associate cert.

I did a take home architecture exercise to get my current job. It was interesting and made me think, perhaps I would enjoy architecture more than DevOps. Maybe, I'm a big picture person?

DevOps is fine but I'm not sure I see myself doing this for the rest of my career.

How can I transition to a Solution Architect role? How would I know if being a Solution Architect is right for me?

Are there any Solution Architects out there that can tell me about their day-to-day?




https://redd.it/1g15168
@r_devops

A Self-Hosted Code Review and Analysis Server


We have built a self-hosted code review service, designed to be useful in the following scenarios:

* You have many repos but still want tight control over code quality
* Your repos are private, and commercial services seem overkill
* You want to continuously improve the process and rules, with full customization

We are open-sourcing it and hope it will be helpful.

[https://github.com/qiniu/reviewbot](https://github.com/qiniu/reviewbot)

Welcome feedback and suggestions. Thanks\~

https://redd.it/1g15h3s
@r_devops

After 3 Years on the Same Tech Stack, What Skills Should I Refresh for DevOps?

I've been on a consulting project with a bank for the past three years, but now that it's wrapping up, I'll be on the bench. My work has primarily involved GCP migration from on-prem using GitHub Actions for CI/CD and Terraform Enterprise for IaC and deployments. After three years of sticking with the same tech stack and mostly writing YAML, I feel like I’ve lost my edge and need to refresh my skills. Any suggestions on areas, tools, or skills I should focus on to get back up to speed?


TL;DR: Spent 3 years on GCP migration using GitHub Actions and Terraform. Project’s ending, and I feel rusty. What should I focus on to stay sharp in DevOps?

https://redd.it/1g16hkx
@r_devops

What project DevOps can build to make USA peoples say "Wow wow wee waa!"?

Jagshemash, DevOps neighbours!

It is I, Boyan, greatest DevOps in all Kazakhstan! I come to you with important question. I want to show my skill to USA companies—yes, land of McDonald’s, Pamela Anderson, and big monies! But how can I make them say, “Wow wow wee waa! This Boyan, we must hire him immediately!”?

What project can I make as DevOps engineer that is big and glorious? Something that will showcase all my big brain powers and make US and A recruiterka slide into my DM like smooth homemade rakiya.

Here is what I know to do very nice:

Make pipelines go fast, like rocket on cow’s milk.
I best snake handler in village: mostly pythons.
Automate things so I can rest and eat more cheese while servers run themselves.
Kubernetes? Yes, I can do! Even my neighbor Nursultan say, “Boyan, you are kuber-whatever genius!”
I also do monitoring, alerting, and can fix everything with only 3 lines of code—maximum!

So what can I build? Maybe I make:

Big project with CI/CD pipeline that deploy faster than gypsy stealing chicken?
Or I make kubernetes cluster that self-heal like strong Kazakh man?
Or maybe cloud infrastructure that so big and scalable, it can hold all of Kazakhstan’s goats?

What will make hot recruiter lady say, “This Boyan, we need him on remote contracts, fast!”? Please help me, friends! I want to bring my glorious DevOps talent to America!

Chenquieh!

Boyan Balgaran, soon-to-be American DevOps superstar

https://redd.it/1g173ww
@r_devops

Naming conventions for VMs?

Hello,

New to DevOps. Just started this role less than a month ago.

I am being tasked currently with writing up terraform for the existing infrastructure that was created through the cloud provider WebGUI, and with that I’m being tasked with coming up with a naming convention for these instances since there isn’t really any consistency between them. I have to account for environment, and scale.

So- I’m thinking most general -> least general, so these instances are grouped alphabetically by their env essentially.

For example- dev-app-01 or something.

Do you guys have any recommendations? Any tips or advice?

https://redd.it/1g17eco
@r_devops

Cloud and devops vs ml


Currently in 3rd sem been doing web dev for 7 months , I am not that good in web dev as of now but for long run I am thinking to do cloud and devops after web dev

don't have any prior knowledge of ml so it would be totally new as for future what should be my goal to learn after web dev should it be cloud or ml

I don't have a clear goal as to what to do I am just learning tech stacks and all and am bored doing web dev so thinking of switching to something else

https://redd.it/1g172ct
@r_devops

If you're struggling to learn, we have a bunch of projects!

Hey everyone, I'm Dan from roadmap.sh (which I know gets posted and mentioned all the time haha)!

We've been working hard on providing people with projects to help prove their knowledge, because as you know, the best way to really learn something is to build it!

We now have 21 DevOps projects that you can build with a good amount in Basic, Intermediate and Advanced!

https://roadmap.sh/projects?g=devops

If you want to see any other classic projects here then just submit an Issue on GitHub.

https://redd.it/1g19yc4
@r_devops

I am stuck in my job don't know whether to quit or go with the flow?

Please need genuine advice

Currently I work as the L1 NOC engineer and my work includes Linux OS, Networking, Putty, NS-OX, and communication with customers to resolve issue.

Now The scenario is earlier I was doing an internship in the startup based company and the role was Frontend dev. I left that internship because of this job due to higher package and the HR told me that they have various fields in the company so they will put me in web dev and I accepted the offer but later they put me in this NOC position and told me after 6 7 months I will get the domain of Devops, Cybersecurity, Cloud, Network, Database, and Backup. I don't trust them because there are many other people waiting for domain who are hired with me so it's gonna be in the randomised order.

Now my major concern is what to do here should I start studying for Devops and build projects in that to get a internship or entry level job which is quite difficult because no one hires a freshers devops engineer unless you are lucky. Or I should grind my Frontend skills and work on the js frameworks to get back in the web development field. Because I only Know HTML, CSS, JS and some react concept.

Currently its my fourth month here and there is nothing new to learn here and it's feel like this experience is nothing but just a waste of my time but the experience letter would say IT Operations Associate.


https://redd.it/1g1cble
@r_devops

Best way to do CI/CD on a self-hosted server running Proxmox for a small web app

Hello!
I'd like to add CI/CD to my small web app that's on GitHub and hosted and is self-hosted. I'm not looking for the easiest (but rather the best) solution as I'd like to learn something new that might be useful to me in the future. This app is literally used by me and my collegues, so there's basically no traffic on it.

The app uses:

Next.js - Frontend
Python with Sanic - Backend
Postgres
Redis

Right now all this is in 3 separate LXC containers (API and Web are in the same one as the API is exposed thru Next.js rewrites). I did my research and it seems like the way to go is Portainer and a GitHub Action that builds a container and then pushes it to Portainer to deploy (So this solves CI too!).

My questions:

1. Is this a good solution?
2. Does it make sense to run all services related to the app in 1 Portainer instance (So that is the whole web app in one LXC basically with Postgres and Redis alongside it)?
3. Related to 3., if there was another web app, would it make sense to have another separate Portainer instance for it in another LXC?

Thank you!

https://redd.it/1g1dtr9
@r_devops

Dashboard for Apache with Geo Location based IP address

Hi all,

Please suggest an dash board ( Prometheus + Grafana ) for Apache with Geo Location map based IP address.

https://redd.it/1g1eq15
@r_devops

GitOps - one deploy config per service version

I want to hear your thoughts on a problem statement that is not broadly discussed.

Let's say I've got a project in which I need to deploy multiple versions of the same service (different clients with different rollout schedules). Let's say each version of the service needs a different deployment config (env vars, secrets, whatever).

I'm using ArgoCD do deploy this services dynamically. I've got an abstract service helm chart that I use to deploy different services by feeding different deployment configurations.

Now I'm adding another layer to this, different configuration per service version. I've been thinking about the cleanest and most usable way of storing this configuration and I've come up with multiple possibilities:

# Option 1 - Big file per service with a block of configuration per version

# Option 2 - One file with base line configuration + one file per version for any version specific config. Periodically, we would promote those version specific config to the baseline.

# Option 3 - Store deployment configuration in the service repo. Helps a lot with organization but if you want to change deployment configuration you need to rollout a new version of the service, which doesn't make sense.

# Option 4 - GitOps repo would contain a folder per file, inside a deployment config file per deployed version of said service. This one is the most understandable but the number of files could be exponential. (let's say you have 40/50/100 clients, each using different versions).


What do you think? How do you handle this? Do you handle this at all?

Hopefully as part of our ArgoCD/GitOps initiative we will be able to reduce the time between deployments and minimising this issue

https://redd.it/1g1cs7m
@r_devops

Does it make sense to use trunk based development with canary deployments?

I've been reading a lot about deployment strategies recently to decide what to go with for my early stage startup. Stability is important as it's a website for lawyers to run their practices.

I want to do trunk based development, but wondering if it makes sense to pair it with canary deployments?

Say upon a release to production, for the next 24 hours, only 10% of users are routed to the new version. Given that no issues are caught over that period, all users are then routed to the new version.

The benefit is that any issues caught in production will affect only a small portion of users.

The drawback is some (minor?) complexity in setting this up with gcp cloud deploy, as well as monitoring canaries for every release.

Should I implement canary relases?

https://redd.it/1g1gr6s
@r_devops

podman upgrade possibly causing kube-apiserver high cpu?

I was working on some golang middleware tying into opentelemetry and saw the upgrade for podman and took it.

About 20 minutes later my laptop fan starts going like crazy. Come to find out kube-apiserver is maxing out a core on my laptop. I use kind, so I blow away the kind cluster and recreate and its good. Then 20 mins later it happens again.

I enabled audit logging on the apiserver and sure enough there are a ton of watch calls constantly. The sourceIP seems to be something within podman itself. I'll paste a sample line in the comments.

Anyone else seeing this? I'm wondering if podman is going wild on kube-apiserver and causing the spike.

So I spun up a vanilla control plane with kind with nothing loaded. Starts doing the same thing.


https://redd.it/1g1jxit
@r_devops

Learn AWS services for free as much as possible?

Hi everyone.

I am planning to learn aws services by building a simple app like a todo application with just an added feature of image upload to explore S3 as well.

Now, I have a basic plan in mind right now like use ec2, rds, s3 and codepipeline or github actions.

My question is, is it possible to learn all these for free in aws? Like a feee tier or something? Or maybe is it cheap to learn these services?

Please don't bash, I don't have background with DevOps and this will be may day 1 of learning.

Thank you for helping in advance. :)

https://redd.it/1g1l2ei
@r_devops

DevOps duties, but never job titles

Hi everyone! Is your current or past job title(s) super important when looking for new DevOps opportunities? For example, I have had roles where my responsibilities were DevOps but the title would be “Cloud Engineer” or “Systems Engineer”. Is it possible that the experience and skills that I gained from those roles could overshadow the official job titles when searching for a new role?

https://redd.it/1g1mays
@r_devops

What are the duties of devops role?

Need to know the duties of a devops engineer

https://redd.it/1g1nvlo
@r_devops