GitLab runner managing its own terraform code

Hi, I have encountered following problem.


I have a docker executor gitlab runner deployed in AWS with ASG using spot instances. There is a threshold which when gets crossed it creates a new EC2 instance and registers a new "instance" of a runner.

The runner is deployed through terraform and it manages its own terraform code. The problem is when you try to apply infra changes and for example, someone in other project is using the runner and new instance is spawned, terraform detects drift in the configuration as when the runner scales it changes the ASG policy, resulting in destroying the runners which don't fall in the min size parameter of the ASG. Also when making changes to the runner itself, terraform obviously detects the changes and while applying the runner destroys itself, but usually having enough time to also apply all other changes as well but the pipeline gets stuck and from my understanding there is no way to make other runner pick up the job to finish it.

Maybe my question is stupid and I don't see an obvious solution, but how would you mitigate these scenarios? My only idea which is related to runner destroying itself is to have a separate runner for the terraform code so the apply job can be finished without actually affecting the pipeline, but also I don't want to have manually turn on the runner and the turn it off when I'm not using it so it doesn't generate a pointless charge on the bill, also the changes to the runner very rare (maybe changing the instance size or the max size of ASG). I'm more worried on how can I make sure that if other people (including me) that are using the runner don't get their runner terminated in middle of the job besides applying the terraform changes when there is no other instance spawned? Or maybe it is the only way to do it and I'm overthinking it and I just need to deal with it and get used to it because there is no other way to do it.

Thank you for any ideas :)

https://redd.it/1enwwka
@r_devops

Looking for a E-Commerce Microservices Demo for Kubernetes Deployment

Hello all!

I'm looking for a demo application based on a microservices architecture that I can easily deploy on a Kubernetes cluster for demonstration purposes. Ideally, I'm interested in an e-commerce application written in Python, so I can make changes if needed.
If you find a suitable repository that fits this description, it would be really helpful!

Thanks,
grator57

https://redd.it/1eny3kx
@r_devops

An Ode to Logging

Oh, log, a nerdy scribe,
In you, all errors hide.
To write it well - not an easy quest,
Let's see how we can do it best!

True hackers always start with print()
Don't judge! They've got no time this sprint.
But push to prod - a fatal flaw.
Use proper logger - that's the law!

Distinguish noise from fatal crash -
Use Info, Error, Warn, and Trace.
Put a clear level in each line,
To sift through data, neat design!

You log for humans, this is true...
But can a machine read it too?
Structure is key, JSON, timestamp...
Grafana tells you: "You're the champ!"

Events, like books, have start and end.
Use Spans to group them all, my friend.
Then take these Spans and build a tree,
We call it Trace, it's cool agree?

Redact your logs: remove emails,
addresses, PII details.
Or data breach is soon to come,
and trust me, it's not fun :(

In modern distributed world,
Do centralize your logs, my Lord.
Retention policy in place?
Or cloud bill you will embrace!

(No LLMs have been used to write this)

https://redd.it/1eo0ih0
@r_devops

On-call alternative to Opsgenie?

Are there any free or open-source alternatives to Opsgenie for managing on-call alerts and incident responses? I'm looking for a solution that can handle alert governance and on-call management, would prefer if it was all in one but if you made any combo of tools work that’s fine as well. Any suggestions would be appreciated!

https://redd.it/1eo0vfm
@r_devops

Logging options


Anyone aware of any logging options that will allow me to use s3 as a data store backend ?


https://redd.it/1eo3xxo
@r_devops

Chatbot to ask about my infrastructure

I have my Ansible facts inventory in a mongodb. How can I create a chatbot to ask it questions like:

- How many Ubuntu servers do I have?
- Which devices are in ip range 10.4.5.0/24?
- Which servers have the package nginx installed?
- …


Thanks.

https://redd.it/1eo47cb
@r_devops

Started my career in devops but I have some concerns

I started as a DevOps intern during my CS bachelor's and transitioned to a full-time role at the same company. I love my job, and my boss is pleased with my work. I’ve always been interested in infrastructure/systems-level work, but I’m concerned that my development skills haven’t progressed much beyond some full-stack projects. I do hobby projects on weekends to keep my dev skills sharp and am also improving my computer networking knowledge through the CCNA. At work, I’ve learned EKS, Terraform, CI/CD, Golang, Kafka and Python, and I write code for internal tools(Though its nothing super fancy) . Could this lack of advanced development experience hurt my career in the long run, especially in DevOps?

https://redd.it/1eo6bx3
@r_devops

What are possible transitioning paths from Devops?

Hey there. I have 5 years of experience as Devops Engineer. My role mainly consists of tasks in Kubernetes clusters (8, each with like 300 namespaces), deploying services using Ansible and helm on on premise cloud and also Jenkins pipelines.

I am kind of getting tired cause of constant need of dealing with various problems which demand lots of research, learning, tweaking, debugging, almost everyday. I have practically no repetitive duties and it starts to stress me.

I was thinking about looking for some easier role - maybe being a sysadmin could be nice, however salaries are much lower (I am French living in France). I am not really fond of becoming developer. I used to learn a lot of security stuff - had pretty high ranking on try hack me and hack the box like two years ago but being a pentester might be even harder.

Anyone had the same experience and would like to share some tips?

https://redd.it/1eo5zzv
@r_devops

local hosting react+java+db+ci/cd

This is what I have right now:

laptop 1: docker with: react with nginx, java, mongodb, mysql, teamcity. + standalone teamcity agent

so when I need deploy, teamcity do 1. docker stop, docker rm, docker run with custom params.

I want to move teamcity and agent away from laptop1 to laptop2. I stuck with understanding how to deploy updated react and java to laptop 1. I can do push/pull to docker repositories (is it free?) but don't want to do it manually or via ssh from teamcity.

please point me in right direction. thanks

https://redd.it/1eo7w95
@r_devops

Are there any advantages in splitting single a Github Actions job into multiple ones except running in parallel?

Each of my Github workflows runs inside a single job. The pipeline will terminate if a single step fails, which is good because it doesn't need to run tests for the code if the code style ( linting / formatting ) is considered invalid.

But I see many people splitting tasks like testing, linting, formatting etc. into multiple jobs. They run in parallel which might be faster but if one job fails the other ones still run.

For now I'm thinking "they might be wasting CI minutes?"

Are there any advantages by doing so?

https://redd.it/1eoa3t1
@r_devops

Managing expectations

Came across this gem today, what are your yalls thoughts?

https://www.glassdoor.com/Community/bowl-B614ab98238f91800037bb4f2/post-P669492eb96a0694604086de6.htm?utm_source=post%20share&utm_medium=community%20share&utm_content=company%20bowl

I feel that people management is an interesting topic, especially the compensation associated to it.



https://redd.it/1eoffqj
@r_devops

Clean Slate at work and Building Tooling. Stuck on what language to plant our flag in.

I work at a smaller company/startup that's been around for about 9 years that has been on the "modernization" path from dotnet framework to core.

Long story short:
Our apps consist of legacy dotnet framework, dotnet core 6-8 for net new apps and Angular for the frontends. When it comes to our Operations and Infrastructure tooling, we have been basic(ish) up until recently. Number of projects are scaling and got a golden path for our application stack moving forward with our "common" patterns. We are serverless first (AWS and all the usual suspects, Lambda API GW, SQS) and ECS fargate for things that dont fit that paradigm very well.

Our deployment stacks:
1. aws sam for serverless stuff (moving to SST soon)
2. Terraform for everything else

Devops tooling:
1. alot of bash scripts
2. python for more complicated things
3. 1-2 go things which are mainly placeholder apps and ping/pong echo servers for testing connectivity.

Now our dilemma:
My team is just me(Lead) and a junior/mid engineer who is stellar and wish I could clone them. We have reached a point where we are trying to scale our services/tooling for more self-service such as repo generation/platform engineering influenced projects. Since we are using bitbucket (no option to move) we find that we need to do more custom stuff to make up for its shortcomings, template repos being one of them.

We are planning how we can keep our stacks across the engineering org between operations/development inline. I don't like having a million bash scripts, python, go, terraform, CloudFormation, yaml, c# dotnet, and context switching mentally is just annoying.

I been considering the following as any refactoring at this time wouldn't be too much of burden and would like to make the better decision towards consistency

1. Going all in on typescript: SST, pulumi/cdktf, amd ts-node for local scripts/npm.
2. SST, pulumi for apps/infra and then ALL of our tooling in golang

My goals are to just reduce the cognitive load of me and my team. My Leaders are supportive, and I have the rope to do anything I want along as I have sound justifications.

Now MY EGO wants me to do Golang for everything possible BUT I know the syntax is easy to pick up it seems the paradigms and COMPOSTION first approach is harder for other people to grasp.

While everything in Typescript would be "simpler" as it seems, anyone can do JavaScript now adays and our frontend devs obviously would have no issue groking our stuff while the backend c# devs are just a skip away.


Where would you plant your flag for setting your operations/infra and tooling for your org given the chance?

https://redd.it/1eog8xz
@r_devops

What is the best kubernetes Operator for GKE

Hi Folks,


Are there any operators for GKE, like Karpenter for EKS, or does GKE have its own way of scaling the node.

https://redd.it/1eoi5xa
@r_devops

Hiring Full Stack Developer - Web3 / Solana

Hi everyone, we have added a new job on our platform, if you are interested in this role please check out the job link below.



Role - Full Stack Developer - Web3 / Solana (Remote, Full-Time)



Job Link - https://devloprr.com/jobs#216

https://redd.it/1eol43j
@r_devops

I prepared an Ansible playbooks tutorial for DevOps Learners

Hello - I prepared a course from scratch that covers the core concepts of Playbooks, by showing a realistic example along the way. Would love to hear the feedback of experts and I also hope that it will be useful for anyone who has not used Ansible before for automation. Thanks :)
https://youtu.be/VDTfI7cB7YY

https://redd.it/1eonipk
@r_devops

Best way to introduce monitoring and observability?

Hey fellows,

I'm trying to make a series of guides about building monitoring and observability as a developer for gathering app and business insights. With the purpose of making the journey less complex. Just starting out, but want to make sure I’m building something that actually helps.

I'm having a hard time deciding what NOT to include and how to make it practical for the developers first time diving into this huge and complex topic. Your experiences and thoughts about the subject could really help me cut through the fog. 

Can you share any key topics, sub-topics or lessons learned from your experience? Anything that helped you can help others as well!

Thanks 🙏
Iliyan


https://redd.it/1eop4uq
@r_devops

Interviewing For On-Prem after being largely cloud focused

I have an opportunity to work for quite an awesome company but they are mostly on prem and my most recent experience has been cloud based

Does anyone know of any tools or processes I should look into that might be beneficial for an interview in a couple of weeks?

I am not too concerned about Linux management because I still do that in the cloud, but any advice would be greatly appreciated

https://redd.it/1eos1r2
@r_devops

CI/CD in the gaming sector?

Out of complete curiosity how does a cicd flow looks in the gaming side for both the client and server? What's commonly used out there for client-server games, standalones and even mobile?.


https://redd.it/1eot2qt
@r_devops

Anyone here familiar with setting up CI/CD for Clarion 4gl programming language?

I know it's a shot in the dark. I've managed to get a workflow that works however some bad dev practices make it very fragile.

Just wondering if anyone here has been down this dark and perilous path ;-)

edit: Clarion Wikipedia) entry.. for anyone interested..

https://redd.it/1eoy99f
@r_devops

Helpless

I started learning about devops a while back. I graduated with my bachelors degree in computer application this year and actively looking for job. I got just a month experience which was mandatory by my uni during my sophomore year.

The job posts for devops require over 3-5 years experience. I mean most do. It's honestly hard to find an entry level job, even if I do, it would've a twist.

I know doing projects have an upper hand but does it really? Even if I were to get a cert in Azure or AWS, it would get devalued since I got no experience to back it.

Oh the experienced professionals, guide me. Show me the path away from unemployment and broke.

https://redd.it/1eoxmrs
@r_devops

deploy MiniCPM-V / Florence-2 to ec2


How can I deploy MiniCPM-V 2.6 ( https://github.com/OpenBMB/MiniCPM-V ) and Florence-2 ( https://huggingface.co/microsoft/Florence-2-large ) on ec2?

Which instance should I pick?

https://redd.it/1ep1xxp
@r_devops