Elastic search DR

Just trying to set up DR for elastic cluster. 300gb+ data. DR site in another continent.

Does it make sense to do uni or bi directional CCR? So we can fail back and forth easily.

Anything you guys recommend?

The site to site vpn has good bandwidth.

https://redd.it/1cbdl55
@r_devops

How is the job market these days?

Sorry if these kind of posts aren't allowed, but I am genuinely curious about the job market for devops/sre. I used to work as a front-end web developer, and that field is very saturated imo among juniors and mid-level devs. Can the same be said for devops/sre? You will commonly see 1000+ applicants for SWE roles within a few days (granted most of them are probably unqualified but still daunting).

With software developers, obviously there was a lot of over-hiring and hoarding during the pandemic which can be said the same for other fields. But I think software developers have it especially tougher because offshoring is making a resurgence and AI is at least giving managers the impression that you don't need as many developers. I don't know if devops/sre is experiencing those kind of blowbacks.

Anyways I personally did not enjoy my career as a web developer. It got very stale solving similar CRUD problems and I don't have the energy to grind out LEETCODE problems. I want to go get a CS degree and transiton to devops but if it's very difficult to get into it, then I must re-assess. I guess I am looking for advice.

https://redd.it/1cbifqz
@r_devops

Segmenting VPC/vNet for a SaaS

Hey,

Title says it all, how do you guys segment your virtual networks for a SaaS? This includes static web apps for the frontend, databases, Redis caches etc.

How complicated is this, have any of you guys done it for a SaaS?

https://redd.it/1cblgcu
@r_devops

Backing up RDS instance to S3 bucket via Lambda

I have an EventBridge event triggering my Lambda function to backup my RDS (MySQL) database to an S3 function with an SNS topic emailing me the results as the destination.

I connected my RDS database to my Lambda function directly under the configuration tab and assumed this meant I wouldn’t have to account for the connection strings in my lambda function code body?

I Frankensteined some Python code from the interwebs but keep getting timeouts 😩

Can anyone point me in the right direction here?

https://redd.it/1cbnvd9
@r_devops

We switched our enterprise product to cater to young startups. Here's why (and we'd love to have you as beta testers)

[Disclaimer: I'm the founder of Facets.cloud, a DevOps solution built from my experiences as a CTO.\]

Hey everyone,

Rohit here. I want to share my experience from a few years ago and how that led us to pivot our strategy.

Here goes something...

I used to work at a late-stage startup where our cloud infrastructure had become a complex beast.

We faced many challenges with our infra, from launching in different regions to managing deployments across environments.

It was a constant struggle, and our tech debt kept growing.

To address these issues, we built an in-house "architecture-first" DevOps solution.

The idea was simple: make architectural documentation the single source of truth.

Any changes, whether in software or infrastructure, would be made at the architecture level and then cascade down to the environments.

But we didn't stop there.

We took it a step further by including alerts, observability, monitoring, CI/CD pipelines, and database schemas in the architectural model. This allowed us to manage critical operational concerns uniformly across the board.

The experiment was a success, so we turned it into a product called Facets.cloud.

We raised funding and built a comprehensive feature set for the DevOps space.

However, after a while, we identified two key problems that we'd overlooked:

1. Facets had become a complex enterprise product where we missed out on early user feedback.
2. Developers, especially those at early-stage startups, needed a more self-service and simple solution.

That's why we're releasing Facets 2.0 - focused on quick, clean cloud deployments optimized for early-stage startup developers.

We're still committed to our "architecture-first" approach, but we're simplifying the platform to make it accessible to any developer or DevOps engineer.

I don't have a trial version ready just yet, but I'd love to get your early feedback on the idea.

We've opened our Beta program, and I'm eager for you all to join us as beta testers: **https://www.facets.cloud/facets-for-startups-2**

Thanks for reading, and I look forward to your thoughts and suggestions!

https://redd.it/1cbs5sa
@r_devops

What do you think about support roles?

So far Ive had only devops / sre / cloud engineer roles but in my current unemployed job hunt I am often asked if I would be ok for a "support" role, like Cloud support engineer, DevOps support engineer and when I look into the description of these roles doesn't sound so different to what I used to do in the previous roles just that instead of the focus being the internal team now the one serving tickets is the client?


What I wanted to know is , taking one of those roles is considered a downgrade? a side change on career? or is just one more step in the line of these career.

https://redd.it/1cbsaub
@r_devops

We pivoted our startup from enterprise-only to SaaS. Here's why...

Hey everyone,

Rohit here, one of the founders of Facets.cloud.

I wanted to share my experience from a few years ago and how that experience led us to pivot Facets to a SaaS product from an enterprise-only product.

Here goes something...

I used to work at a late-stage startup where our cloud infrastructure had become a complex beast.

We faced many challenges with our infra, from launching in different regions to managing deployments across environments.

It was a constant struggle, and our tech debt kept growing.

To address these issues, we built an in-house "architecture-first" DevOps solution.

The idea was simple: make architectural documentation the single source of truth.

Any changes, whether in software or infrastructure, would be made at the architecture level and then cascade down to the environments.

But we didn't stop there.

We took it a step further by including alerts, observability, monitoring, CI/CD pipelines, and database schemas in the architectural model. This allowed us to manage critical operational concerns uniformly across the board.

The experiment was a success, so we turned it into a product called Facets.cloud.

We raised funding and built a comprehensive feature set for the DevOps space.

However, after a while, we identified two key problems that we'd overlooked:

Facets had become a complex enterprise product where we missed out on early user feedback.
Developers, especially those at early-stage startups, needed a more self-service and simple solution.

That's why we're releasing Facets 2.0 - focused on quick, clean cloud deployments optimized for early-stage startup developers.

We're still committed to our "architecture-first" approach, but we're simplifying the platform to make it accessible to any developer or DevOps engineer.

I don't have a trial version ready just yet, but I'd love to get your early feedback on the idea.

We've opened our Beta program, and I'm eager for you all to join us as beta testers:
https://www.facets.cloud/quick-cloud-deployments

Thanks for reading, and I look forward to your thoughts and suggestions!

>[Disclaimer: I'm the founder of Facets.cloud, a DevOps solution built from my work experience.\]

PS. I had to delete and repost since it didn't let me edit text. What am I doing wrong here.

https://redd.it/1cbtdoj
@r_devops

Do I have the chance as a junior devops engineer?

Do you think with this knowledge I can start at a Junior devops role?

So I have knowledge in Python, Git, docker, bash, SQL, Linux and a little powershell scripting.

what other skills do i need for a junior devops role?

https://redd.it/1cbub8j
@r_devops

Does anyone else suffer from "get across the line" syndrome?

I'm not sure what this behaviour is called, but I'm refering to the push by management to deliver tasks in such a way that they just about deliver what's needed in that instance, and then move onto the next deliverable ASAP.

No time for documentation, no time to plan out a body of work into the next few weeks, no time to consider futureproofing, keeping things DRY, reviewing old configuration to confirm it's valid, and so on.

I often hear that "if it's not a (paid for) feature, it's not getting done" is a mindset by product managers and the like, so "unsexy" things like security posture planning, technical debt and even documentation never gets addressed. How common would you find this style of work; this sort of relentless firefighting moving from task focusing on whats 3" ahead rather than 3ft or 3 miles?

https://redd.it/1cbu7sb
@r_devops

Looking for suggestions of books about corporate finances, and the economics of teams/departments

I’ve recently read my first few books on finance and economics, and they have been very enlightening. It has led to me reason about my work in terms of the ROI it generates to the company, at a deeper level than I previously considered. I work as a senior DevOps in an entertainment company, and am one of the few people in this role, for what is comparatively a very large company.

I wanted to go further, however, and learn about the economics of corporate environments, specifically the economics of managing a department or team. How can I deepen my insight on the impact of my work by analysing it through the financial lens? How can I better leverage the impact of my work to lobby for projects and efforts I think are important?

I was wondering whether anyone has suggestions for a read on this.

Thank you!

https://redd.it/1cbv30x
@r_devops

SAML -> Ldap

We're using Google Workspace as our main user directory, and we're also using Nexus OSS. Nexus only supports Ldap, and google's Ldap service is expensive.

Previous devops created a freeipa instance just for nexus but overtime, freeipa and Workspace became out of sync (there are many users in freeipa that have left the company but still have active accounts)

What would you do in this scenario?

My options are:

1. push the company to pay for nexus so we can get SSO
2. push the company to pay for google ldap
3. give HR access to freeipa and kick the ball to their court
4. use freeipa (or maybe okta?) as the main user directory and then sync to google workspace
5. make nexus and freeipa only accessible behind a ZeroTrust layer

Other options that I'm not sure if possible:

1. introduce a proxy layer that makes users sso and then generates username and password from their SAML object and authenticate with nexus using it. (are there tools that do that?)
2. something similar to https://github.com/hlavki/g-suite-identity-sync (the project is no longer maintained)

https://redd.it/1cbvojd
@r_devops

How can you stay safe and avoid scams when finding reliable and talented developers?

Back in the day, finding developers on platforms like Fiverr, Upwork, LinkedIn, and Freelancer used to be pretty straightforward. All you need to do is read reviews for each developer on the platforms, review their portfolios if available or request that they be sent to you, ask for one or two references from previous projects, and you are perfectly matched with a developer that fits your requirements.


But gone are those days, as lately most, if not all, of these platforms do not bother to vet the developers who are onboarded on their platforms. This has led to scams, false qualifications, the inability to meet timelines, and outrageous service charges for little or no work done. Lately, all of those things, such as reviews, portfolios, and references, which used to be ways to authenticate the originality of developers, can now be falsified, and this can oftentimes mislead individuals who are unable to detect such frauds.


In modern times, how can we mitigate this alarming issue(s)? Are there ways to detect these false qualifications? Are there tailor-made processes to filter the shaft from the grains? Are their platforms that help you navigate such important terrain?

https://redd.it/1cbz4ui
@r_devops

DevOps centralized labor pools vs. decentralized.

What labor models have you seen work well for DevOps?

Can shared labor pools work or is devops resourcing within product teams the only viable solution?

Are there hybrid approaches for teams who need less resourcing?


https://redd.it/1cc0za7
@r_devops

A Pastebin for Terminals

Runme Gist brings securely shareable GitHub Gists to your day-to-day DevOps workflows and documentation.

Learn more here: https://runme.dev/blog/runme-gist.

https://redd.it/1cc13h5
@r_devops

Udemy or KodeKloud

Hi, I am looking into starting my learning path into being a Cloud Engineer. I am considering two options: subscribe with KodeKloud and stick with its path, or buying individual separated courses on Udemy. Which choice is better?

https://redd.it/1cc0vge
@r_devops

Tips for dealing with alert fatigue?

Trying to put together some general advice for the team on the dreaded alert fatigue. I'm curious:
* How do you measure it?
* Best first steps?
* Are you using fancy tooling to get alerts under control, or just changing alert thresholds?

https://redd.it/1cbvf5w
@r_devops

How to prepare for certifications?

Hi guys!
I'm planning to appear for AWS certifications: Cloud Practitioner and SysOps Administrator.
Last year, I tried to pass Azure:104 but I failed by short margin(670), and because of that I lost some confidence. I, although, have a good experience with cloud fundamentals, but whenever I see the questions I very nervous, and get caught off guarded, despite prep.

So, I want your help and advices.....

Thanks!!

https://redd.it/1cc2vv2
@r_devops

HashiCorp joins IBM to accelerate multi-cloud automation

Today we announced that HashiCorp has signed an agreement to be acquired by IBM to accelerate the multi-cloud automation journey we started almost 12 years ago. I’m hugely excited by this announcement and believe this is an opportunity to further the HashiCorp mission and to expand to a much broader audience with the support of IBM.


https://www.hashicorp.com/blog/hashicorp-joins-ibm

https://redd.it/1cc99j9
@r_devops

“DevOps isn’t an entry level role” from an entry level DevOps

As one of the few people who actually did start out as an entry level DevOps Engineer for their first full time IT role out of college, I wanted to give my input on this.

I mostly agree that DevOps is not an entry level role. I wouldn’t recommend what I went through to anyone who could avoid it. Getting to even a minimum level of competence to be productive was filled with horrible growing pains that I didn’t see the entry level Devs come anywhere close to experiencing. Particularly the networking, infrastructure, and some of the containerization concepts were extremely hard to understand with no background. And I have a hard time believing that anyone “entry level” would know Linux to the level required, besides Linux just being boring to study. There was also tons of proprietary knowledge and business process stuff that I just didn’t know how to navigate the way someone with professional experience would have. Everything I mentioned so far is hard to practice or learn on your own compared to other roles, unlike making a simple portfolio website for example.

The other main problem with starting as a DevOps Engineer is that there’s not really a natural progression of tasks you can do as your knowledge increases, unlike developer and other IT roles, and the consequences for mistakes is typically an outage or some other critical.
Another Redditor u/MammothCache pointed out that there’s a very logical progression for how you grow as a SWE. You first start with bug fixes, then features of increasing scope, then to an entire application, API, or data model, ending at a more architect role. A developer can kind’ve just know a programming language decently and how to use google or ChatGPT to be given small tasks.

This doesn’t exist in DevOps. You can’t really just know a tool without understanding other IT concepts & tools with it. Even if you did know just Terraform or just Kubernetes or any DevOps tool really well in a vacuum somehow, you wouldn’t be able to do anything with it by just knowing the syntax and documentation. To make a CI/CD pipeline or troubleshoot an outage is basically already architect level knowledge. You need to know the software, admin/ops, and your DevOps tools to a decent level to be helpful. I would sometimes get jealous of the developers for having such an organic, painless progression compared to me.

I used to hear people say it takes about a year for most entry level/new grad developers to become useful to the business and feel somewhat confident in their skills. I think this is the case for most IT roles. Maybe it’s shorter now with ChatGPT and others massively increasing what Juniors can do, but it would still be completely unfair to give the same timeline to a truly entry level DevOps Engineer that you would an entry level data engineer, web dev, sysadmin, etc.

But it’s an over exaggeration to say that a smart person couldn’t provide more value than their salary after a slightly longer ramp up in the right scenario. I think this may be an ego thing of people trying to make their job sound harder than it is.

The SRE aspects are much easier to progress on from an entry level, so that’s how I started. A lot of monitoring, alerts, & logging. I was also allowed to do some cool Python coding for internal uses. That, plus writing tons of documentation and good ol’ trial by fire until eventually the dots started to connect around 9 or 10 months in. I didn’t study outside of work at all but I did put in long hours often. Through a path like this, entry level DevOps is possible.

Furthermore, a huge reason my ramp up was so rough is that I was at a toxic startup that didn’t train me, had no mentorship, had no documentation, no enforced standards or best practices, you name it. I was told that the Jr. DevOps I was brought on to replace was nearly useless in that same time frame. I pretty much only survived because I have more grit and talent than average.

Where I’m at now takes training juniors and documentation much more seriously,

and I’m really feeling the benefits. I could an entry level engineer having a much smoother time somewhere like here. But, even though it counters my own point, gone are the days when companies will truly train employees and people entering the workforce need to adapt. That’s perhaps the greatest lesson I’ve learned. In my new role, I was basically autonomous from the beginning and that didn’t seem unexpected. I’m effectively treated like a mid-level. That’s just the nature of DevOps in my opinion. You’re either able to do stuff without much hand holding or you’re not able to do anything at all.

I’ll end with a comment. There are some small advantages to starting out as DevOps. I agree that the DevOps ”philosophy” seems to be much rarer and less ingrained in people who switch later. Also, it was very humbling and made me emphasize working well with others, persistence, and doing good research. And we will see more of the business inefficiencies/bottlenecks with our fresher eyes, since new DevOps Engineers at your company will suffer the most from these. There’s more but nothing major. A good employee is a good employee.

Since people may ask, I graduated in 2022 as an Electrical Engineering major with two IT internships then worked as a DevOps engineer for a little under 1.5 years before being laid off in November 2023. The job hunt wasn’t bad for me. I put in ~125 job apps. I had 8 phone screenings, 4 interviews, and got 2 Jr. DevOps Engineer job offers (one remote, one hybrid, both contract-to-hire) at the end of February, plus a third offer for an Electrical Engineering position surprisingly. 5 of the phone screenings came from recruiters, so yeah my numbers from cold applying are a lot worse. I’m not a unicorn in any way(no prestigious university or big tech on my resume) but I do interview pretty well.

TL;DR: I agree that there’s no such thing as entry level DevOps, but it’s 100% possible to start out in DevOps and become useful in a similar timeframe to other IT roles if a company is willing to invest even a moderate amount into training you and by being smart about the task progression they’re given.

https://redd.it/1cc9qi6
@r_devops

Oncall rotation

My work decided today that our principle engineer isn’t going to be apart of our rotation anymore. We are going from being oncall once every 5 weeks to once every 4.

How has your work handled things similar situations?

https://redd.it/1cca5qe
@r_devops