Order matters - making a compound index 50x faster

https://jaywhy13.hashnode.dev/order-matters-making-a-compound-index-50x-faster

https://redd.it/1g5gnai
@r_devops

Good devops oriented networking training for my team?

Hi guys,

I’m the lead of a team, and as it usually goes, it seems like most juniors and even more senior people, who are very qualified at cloud and whatnot, lack basic networking fundamentals.

They don’t truly understand what NAT is, what the difference between routing and bridging is, what peering is, route propagation, etc.

I’m looking for a certain course / training that I can give all of them so that at least everyone in the team has some basic foundational knowledge about networking. Eg if I ask them to set up a redundant wireguard tunnel for site-to-site peering between two different clouds, they know at least understand what they’re being asked to do.

There’s so much available online, but the Cisco etc types of trainings are way too much.

Does anyone in this community has any recommendations on a good ~ 3 day course that goes over the fundamentals?

https://redd.it/1g5jw9k
@r_devops

Is there an app or phone feature that would automatically notify clients of service delays after a certain time period has elapsed?

Hopefully, this is the correct reddit community to address this question. If not, please kindly direct me to a more appropriate one. I work for a home health care organization. The field staff often fail to notify clients via the company cellphone if and when they are running behind schedule. I'm under the impression this is due to the nature of their job. The job requires them to conduct hygiene and grooming services to vulnerable clients like the elderly or those with disabilities. There seems to be two main obstacles for the field staff. One, unexpected delays occur while staff are servicing clients, and due the clients risk for falls/ injury the staff members can not leave the client unattended and inform the office or other clients of the delay within a timely manner. Two, notifying a list of 10-15 or more clients each time a delay occurs due to unexpected circumstances isn't feasible due to the amount of time it takes to reach and communicate with the client as many of them have difficulty with mobility, hearing and the like.

Is there an app or phone feature that could send an automated message that notifies the next client and/or list of clients of a delay using a countdown? For example, x client has one hour time slot, and if the worker is still in the home past that time it sends an automated message to the next client of a possible delay.

https://redd.it/1g5l5sm
@r_devops

Pivoting towards being Network Focused DevOps Engineer

For context, I have a bachelor's degree in Computer Networks and have had a network engineer role for about 18 months before being let go, and I am currently searching for roles.

Would it be wise to pivot towards Network-Focused DevOps roles given the lack of computer networking jobs?

I know that lulls in opportunities can be temporary, I'm just considering my options.

I understand there's been a shift in the fact that there's a lot of network automation as well as a heavier focus on cybersecurity.

https://redd.it/1g5khgg
@r_devops

Self-hosted GitHub Actions runners on Kubernetes with MicroK8s

Got a powerful bare-metal server somewhere? Could be a good idea to setup self-hosted runners there with a simple Kubernetes distribution like MicroK8s.

In this article I setup ARC with the officially supporter runner-scale-set version. It's actually not that hard, although there are some limitations.

https://redd.it/1g5ml0h
@r_devops

How do you handle complex data wrangling in Google Cloud or Azure Data Factory?

Hey all,

I’m working with Google Cloud Dataflow and Azure Data Factory for complex data wrangling. Any tips for optimizing pipelines, especially with large datasets or intricate transformations? What strategies or tools have helped you improve performance or streamline processes?

Thanks!

https://redd.it/1g5o6j8
@r_devops

API logging recommendations

Hi, I'm working on a project for a foodservice company mainly focused on catering. We have a platform that allows customers to order catering from our in-house brands, but we need a logging mechanism for the site. The traffic will be relatively low especially given the nature of catering (only takes 1 user to order for many people), but as the company grows there's expected to be more traffic. I'm looking for a cost-effective solution for this. I've heard datadog is really good but I also hear it's expensive. Our software isn't extremely sophisticated and doesn't rely on "mission critical" data so I wasn't sure if that'd be overkill. New relic was suggested to me, but it's hard to get a feel on pricing giving the unknowns about data ingestion.

https://redd.it/1g5rb6i
@r_devops

Folder structure, Deployment best practices with bicep across multiple subscriptions with azure devops. Help

Hi Guys,

I'm kinda new with Bicep and I'm working on the folder structure, for Infra deployment with Azure DevOPS. This is my current folder structure. It seems I'm missing something that I cant point out . I tried googling it but I need some feedback. Anyone with experience? Thank you.

-infra
--->project1
-------main.bicep
-------params.json
--->project2
-------main.bicep
-------params.json
--->project3 (same subscription with project2)
-------main.bicep
-------params.json

-modules
-pipelines
-scripts

https://redd.it/1g5r4a9
@r_devops

Fraud detection data pipeline (ETL) on GCP

# How to create a scalable fraud detection steaming data pipeline

[https://medium.com/@rasvihostings/fraud-detection-data-pipeline-etl-on-gcp-2b15b8f3d65b](https://medium.com/@rasvihostings/fraud-detection-data-pipeline-etl-on-gcp-2b15b8f3d65b)

When we think about large volume streaming data pipeline three things come to our mind

* **Scalability and resilience**
* **Cost**
* **Infrastructure maintenance**

I designed a solution which can scale easily, use much as possible GCP managed services and finally reducing the cloud cost 😉

[read my blog post here](https://medium.com/@rasvihostings/fraud-detection-data-pipeline-etl-on-gcp-2b15b8f3d65b)

let me know what type of articles you guys are more interested on

https://redd.it/1g5t04r
@r_devops

Should I get into devops?

I am a frontend software engineer and I hate my life, I am sick and tired of putting buttons, input boxes, etc. on a web app... And I feel like frontend web development is a dead end job, or at-least it is not suitable for me, because all the frameworks do the same thing... I hate most of my coworkers (including developers, design, product managers, etc.) because they fight over trivial issues like, why use a forEach instead of reduce? Why is this button small, etc.

Should I switch to devops? I feel like there is much more variety of tasks available in this space, but I am afraid of AI, platforms like vercel, that abstracts all the complexity of managing infra, etc. Also currently my salary is quite high, so changing role will be extremely difficult...

Is Devops worth it? Or are all the IT jobs the same?

https://redd.it/1g5uf5x
@r_devops

Manage spot nodepool in GKE

Hi Everyone,

We run about 60-65% of our workloads on spot VMs, but during peak hours we usually hit stock out and new pods are usually in pending state for long hours waiting for a spot VM. So we have implemented 2 ways to improve this state.

1. deploy the same deployment on payg nodepool with a higher hpa threshold, so it scales when spot pods doesnt scale.

2. create a nodepool with different series of machines with same configurations, taints and labels, but at times one nodepool doesnt scale even if it isnt hitting stock out, whereas the other nodepool would have stocked out.

Are there any better ways you guys tackle the stock out situation ? Kindly advice.

Thanks !

https://redd.it/1g5wxze
@r_devops

How do you handle urgent communication when a team leader from another team doesn't respond?

Hey all,

I’m curious about how you handle situations where you need an urgent response from a team leader in another department or team, but they take hours to reply. For example, if you're waiting for critical input and it's been over two hours with no answer—how do you proceed?

Do you escalate the issue, follow up with other team members, or try to find alternative solutions? How do you balance urgency without coming across as too pushy?

https://redd.it/1g5z5y6
@r_devops

Move from Sprints to KanBan?

For a long time I have hated sprints and pretend agile for DevOps in our organisation. We plan, refine, retro, demo - blah blah blah - the whole thing in our org is pointless because we plan/refine but when the sprint starts we abandon the sprint for another team’s work because they didn’t think to involve us early enough or they say no when we asked if they had any requirements, and now we’ve blockers.

Instead, we’re going to use KanBan. Anything in to-do can be picked up by anyone at anytime. Items only move from backlog to to-do after it has been refined, backlog items don’t show on the board, and board is DevOps only work. We no longer include other teams work on our KanBan board.

I’m not saying we won’t do work for other team. Instead we rotate through other teams. Embedding in their process and our tasks in their epics belong on their board because they belong to that team.

My fear is - how will be prioritise our own board if we’re embedding in other teams. What happens to unfinished work after a rotation.

What are peoples thoughts? Is there a better approach?

https://redd.it/1g61gzq
@r_devops

What happens if I have multiple IP addresses in a single weighted routing record in route 53?

Basically the title.

I am in the process of migrating from simple routing to weighted routing and wanted to test using a few servers.

Currently, we have a single A record which is simple routing, it consists of all the server IPs.

I am trying to take out some servers and add some weighted routing entries for the same.

If I have 3 records,
Record A - weighted, 2 IPs, weight 50
Record B - weighted, 1 IP, weight 50

Will each of the IPs in record A get equal traffic, I.e 25%?

I was not able to replicate the above.

Please help.

Thanks in advance.

https://redd.it/1g67daj
@r_devops

First time creating a simple CI/CD pipeline help

Hey guys,

I'm a self learning developer trying to broaden my skills and learn more about cloud services and CI/CD pipelines and all that. To start simple I want to host my personal website on an EC2 instance and set up github actions to containerize and deploy to my instance such that when I push changes to my repository, that all happens automatically. My first basic question is how do you handle private environment variables? Do you use AWS parameter store? I'm using Go and the 'godotenv' package, but if I change my code around to use the AWS Parameter store, how would I test it locally? Since I wouldn't have access to the parameter store, right?

Thanks!

(you'll probably see me a lot around here in the near future 🙌)

https://redd.it/1g6auuk
@r_devops

Interview experience

I recently interviewed in mid level company where they rejected me in second round stating that I do have experience in CI/CD tool like Jenkins , harness but i don't have experience in azure cicd. Why can't they understand it's just a tool, logic and workflow is same everywhere.

https://redd.it/1g6b7yf
@r_devops

How Well Does "all in the same repo" CDK approach Scale?

I asked this in the r/aws (here) but will also post on here.


I am in the process of adopting and learning about CDK for our large-scale microservices architecture. What I want to know is how well does it scale when used in an environment with 100s+ of microservices and pipelines.

Has anyone got any recommendations on best practices in terms of structuring and managing CDK for scale? Does anyone have experience using CDK in environments with 100+ microservices?

I can see that the biggest shift with CDK is essentially coupling the CI/CD config, infra config and application code all in the same repo. How does this approach/recommendation scale?

Let's say I have 100s of microservices and I need to update CI/CD or some infra config across all. Every time you make a change to the pipeline config in the repo, you are potentially "touching" the app and making a release. I can accept the changes to the infra "close" to the app like Lambda config, SQS etc., but I'm not sure CI/CD config is the same.

How do others manage updates to shared infrastructure or CI/CD configurations across multiple services?

Also, regarding self-mutating pipelines: it's something I tried 5 years ago with raw CloudFormation but found that if there was a change to the CodePipeline executing the change to itself, the execution would instantly fail and I would need to rerun it. Has this been fixed?

Lastly, why would a developer want to see the "pipeline update" step execute and do nothing 99% of the time, just wasting time and slowing down the CI/CD cycle?

I'd love to hear about your experiences and best practices for using CDK at scale. Any insights would be greatly appreciated!



https://redd.it/1g6dfye
@r_devops

DevOps/Cloud Engineers: How Do You Manage and Visualize Your Infrastructure?

Hey everyone! 👋

I’m working on a new tool concept to help DevOps teams and cloud engineers better manage their infrastructure. Before diving deeper into development, I’d love to get some feedback from the experts—that’s you! 😊

If you have a few minutes, could you help me by answering some quick questions?

# 1. Visualizing Cloud Infrastructure

How do you currently visualize your infrastructure (e.g., cloud resources, clusters, VMs)?
Do you rely on built-in cloud tools, open-source solutions, or paid platforms?

# 2. Managing Costs and Outdated Components

How do you track cloud costs across different resources?
Do you encounter issues with outdated Terraform modules, providers, or infrastructure drift? If so, how do you handle them?

# 3. What’s Your Biggest Pain Point?

What’s your biggest challenge in managing infrastructure or optimizing cloud costs?
If a tool existed that could simplify visualization and provide actionable cost insights, would that be valuable to you?

Thanks a ton in advance! Your insights will help us make something that really solves problems for the community. 🙌

https://redd.it/1g6dazv
@r_devops

How can a devops engineer develop in machine learning?

Hi everyone, I'm a Devops engineer and I really love my job. I love doing linux, writing scripts, configuring networks and so on. But I also love math and algorithms and data structures. I want to participate in the development of artificial intelligence, apply my math knowledge, but still keep doing devops. Any advice?

https://redd.it/1g6gacx
@r_devops

My employer is offering me a 65% raise and a bonus in the next pay cycle if I rescind my 2 weeks notice.


In the past year working in a start up, I had made a transition working as a senior cloud infrastructure engineer to a junior and now mid level full stack engineer. 2 senior cloud guys and 1 senior full stack engineer decided to leave our company to take roles in FAANGs (who also happen to be our customers for our product) these last few months. Although we re’orgd and some duties got divvied out amongst us. I got bombarded doing my job and taking on cloud duties again. My mental health has been killing me with deadlines, and management asking us to push new releases on a Friday, which takes up some of my weekend. I’m just so done. I been offered employment elsewhere and put my notice in so I can take a month off for vacation and reset. Well I got a call almost instantly from the CTO, Product, and CEO about anything they can do to keep me including offering me a promotion to senior, a huge raise, focus on backend development only, and a $25k retention bonus on the next pay cycle. The raise is about 10% more than the new employee is offering.


They want to give me the weekend to think over it. I’m contemplating on whether I should take the offer or not.

https://redd.it/1g6he1w
@r_devops

Promoted to Manager

I've been in the sysadmin space for 7ish years, plus another 3 as DevOps engineer. The change into DevOps at first was a bit rocky and I think I suffered from imposter syndrome. I was the weakest one on the team (in terms of hard technical background and devops "years"). In recent months I realize I actually did beat out my coworkers at other skills: communication, organization/planning, enabling/empowering other engineers, not letting "great be the enemy of good", making actual progress. Recently we had a bit of a reorg and now I am managing the DevOps team. In a way it makes sense to me, I know the principles and our goals and the bite-sized chunks it takes to get there. I'll never be able to write slick bash one-liners on the fly with 5 ppl in a zoom meeting watching. Sorry for the rant and tamble. Long story short: any tips or suggestions for this transition from engineer to managers? Do you think I have enough background to succeed? Any suggested material or reading? (rn I've been reading Radical Candor)

https://redd.it/1g6hdmb
@r_devops