Need to learn about cert (security)

Hi guys,

I am working for a while in devops and used to be developer. Certs are always scare me away all the time, so never involved in working with them. But recently most of the issues in our env is because of cert whether it is kubernetes or openshift or kafka.

We are having different types of issues and for me its very difficult to understand when our team discuss about it in meeting.
Can you guide me where should I start learning about it and also suggest me if any certification courses will help as well. But my main target is, I should be ready to solve security problems related to certs / keys.

​

Thanks

https://redd.it/10fy6ki
@r_devops

CORS issue after attaching AWS WAF to load balancer

Guys,
I am facing "Access to fetch at ' ' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled." after I have attached my load balancer to AWS WAF. Otherwise, it works fine, so what may trigger the issue or which rules are responsible for this scenario?

https://redd.it/10fw4wl
@r_devops

Beholder - Documentation search engine with K8S first approach

Hey everybody,

I just finalized the first version of my project: Beholder. When deployed to K8S it allows you to expose OpenAPI documentation for specifically labeled services.

It's the first version, I tested it as much as I could but there could be some lingering bugs. I would be more than grateful for the feedback.

https://github.com/gdulus/beholder

https://redd.it/10g1gxc
@r_devops

Should I continue my self-taught journey to become Remote worker?

Overthinker here...

There's a thing where it demotivate me to continue my self-taught journey to become DevOps Engineer. I'm from a third world country where there's barely any software job there, It's just web dev with pretty much bad salary..Currently learning sysadmin and "Golang" but it doesn't stop there I know there's much more for sure. I have already made my road map and path. also I'm on my third year in college "computer engineering"But the issue is I hear may people say that DevOps requires you to to work as a sysadmin or software engineer first and get hands-on experience then you move to DevOps, also it's so hard to get remote job outside US. My plan is to get as much knowledge and build up my github so i can land a "Junior remote job role" even if it pays much more below average (not necessarily a DevOps role.. sysadmin or Cloud Specialist first is fine)It's fine for me to work any of those jobs first but remotely? eh.. even for DevOpsWhat do you think guys? should I stay motivated and keep learning? I'm worried that all my studies will go to waste

I do have a carefully considered road map/path. Spent months doing researches and watching videos

Edit: I can't travel outside my country. I'm here taking care of my family alone :q

https://redd.it/10g4hmw
@r_devops

designing guide | DevOps

I have a homogeneous infrastructure on Cloud, I need to design the DevOps way of managing it.

My design should be capable of configuration management, security updates, scaling, automation and automation.

I have a very good knowledge on Linux, storage and operations, but i have no clue about DEVOPS ways of designing.

So any book, or website you could refer me? please

https://redd.it/10g4t22
@r_devops

Do you let devs deploy to production?

Just curious how other are doing. Here the devs need to open a Jira ticket requesting a specific build to be deployed to prod and then our team do the deployment with the cicd pipeline.

https://redd.it/10g3bcb
@r_devops

Azure Keyvault for multi-cloud use (AWS, Rancher onprem, and Azure)

Does anyone have experience utilizing Azure Keyvault outside of Azure? I've been tasked with identifying a multi-cloud solution for secrets management. We have an existing Hashicorp Vault setup, as well as an existing Azure Keyvault setup.

Is it possible to use Hashicorp vault as a secret store that pulls from Azure Keyvault? Alternatively, is it possible to use Azure Keyvault successfully in AWS kubernetes clusters or VMs, or Onprem kube clusters/VMs?

https://redd.it/10g9j9k
@r_devops

Hands-on examples of observability-driven development

https://tracetest.io/blog/observability-driven-development-with-go-and-tracetest

Based on one of my previous discussions about ODD, I wanted to go into more depth and explain how it works with a code demo using open-source tools like Go and Tracetest. The main point I think is that there are no mocks. Instead, you're running E2E and integration tests against real data. I think the biggest pain point in testing on the back end is the amount of coding you need to do to actually just make the test run. Mocking API responses, setting up credentials and env vars to access different services and databases. It's just a lot of hassle to run an integration test.

Disclosure: I am on the Tracetest team, so I'm passionately not disinterested in what you think about the whole ODD movement.

https://redd.it/10gab31
@r_devops

mOVING FROM Puppet To Ansible - A few questions around structure and config drift.

So we're on Puppet right now - it's old, out of date, but at the core of everything we do.

We'd like to move to Ansible, which a lot of us are familar with, and I think is the better path forward for us as we're moving a lot of things to the cloud.

Now I have a few thoughts/questions for which I don't have an exact answer for:


1: Configuration Drift

We can make a playbook, chuck it into gitlab, have a pipeline run it...but then what?
What if someone makes a config change on the box but not in git? (it WILL happen)
Puppet runs every 45 minutes or so, without using Ansible Tower, how are people doing this?
Something like Rundeck?
An "Ansible Master" server at each DC running cron jobs every hour?

2: Structure or hierarchy of our Playbooks/Roles, with multiple DCs

There will be quite a few common roles that ALL server will need:
NTP, Security/SSH settings, Log rotation, Log shipping etc etc
Do we just create a playbook for each server type/location, chuck in the "Common" roles and then the app/location specific role into that playbook?

Seems like #2 could get messy quick with lots of servers, doing the same thing over multiple DCs.

e.g. I might want to only affect the mail servers at DC1 today, and then DC2 tomorrow, and DC 3,4,5 & 6 later...but now that means I got 6 versions of the same role to maintain?


EDIT: Damn text editior FORCES YOU TO BE IN CAPS EVEN WHEN YOU'RE NOT SO THE TITLE LOOKS LIKESHIT..

https://redd.it/10gc90g
@r_devops

"I Know So Much Stuff I Learned Over The Years I Forgot Half Of That By Now?"

I feel like my brain has a limited capacity to remember stuff I dont repeat from time to time.

As a DevOps/SysOps/SysAdmin w/e I had so many tools I had to learn how they work over the years that I lost track of half of them..

Example 10 years ago was using puppet. Could write configurations 1b1, it was super easy to understand and now I would have to remind myself most of it.. coz Im using mostly GA..

Am I just a bad engineer or the tools change so often from company to company its just impossible to remember all of them ? Maybe some ppl can/ or most ?

Just curious whats the other ppl experience in this regard.

https://redd.it/10gfegd
@r_devops

Monitoring stack demo using Grafana, Loki & Mimir

Wanted to share a demo/tutorial with everyone on how get started with a monitoring stack using grafana, loki and mimir with prometheus metrics & promtail log sender:

[https://github.com/wick02/monitoring](https://github.com/wick02/monitoring)

I also created a [video demo](https://www.youtube.com/watch?v=KPqbA7ys24o) of it working on a mac m1 along with a few of my old colleagues cloning it with no issues reported. I have around 6-7 years helping maintain logs and metric backends and this is my second video on Grafana which is available on [Grafana's youtube channel](https://www.youtube.com/watch?v=AgV5DoWcY6I&t=1544s) from a meetup in 2017.

**Goals of this repo:**

* To trim down to the very basics of each service, to isolate them from each other so you can pick and choose what you want to use from the demo.
* I've configured it in such a way where you can scale it in a cloud environment and to give something to the developers.
* It's not dependent on keeping volumes on the machine, so you can use something like Amazon ECS without managing the volumes and use spot servers to help cut costs.
* It's not a lot of code or configuration, it uses a lot of existing tutorials already but made in such a way that I think anyone with some operational experience can use and get started with.
* It's also built in a way where the metrics are pushed to an S3 like backend using minio so you can keep and persist all the logs and metrics.
* Lastly, it uses Tenant IDs, so you can isolate offenders if you need to use this as a massive shared service for the company by rate limiting them until they stop sending you too many metrics/logs as we all are accustomed to see when we manage these type of backends.
* Since it is simple to spin up a Mimir or Loki cluster with a design like this, you could make multiple clusters and isolate components away even further

I hope someone out there finds this useful. I hope to add Tempo in the future along with a terraform deployment process for this stack.

https://redd.it/10gfu0t
@r_devops

Feedback Request: TCO Calculation for Apache Kafka

I'm working on calculating the total cost of ownership (TCO) for tools like Apache Kafka to determine when to build vs. buy.

I'd love your feedback -- what am I missing? What did I underestimate/overestimate? How can I improve this?

First, the criteria to consider when calculating TCO:

Up-front costs

software cost & licensing, if applicable
learning & education
implementation & testing (including data migration costs)
documentation & knowledge sharing
customization

Ongoing costs

direct infrastructure costs (e.g., hosting & storage)
backup infrastructure costs (e.g., failover & additional AZs)
supporting infrastructure costs (e.g., monitoring & alerting)
maintenance, patches/upgrades, & support
feature additions

Team & opportunity costs

hiring to replace the engineers now working with the new software
time spent on infrastructure that could otherwise be spent on core product

Now, an example using the above criteria:

Desired specs for our example deployment (I picked one of the smaller Heroku plans):

Capacity: 300GB
Retention: 2 weeks
vCPU: 4
Ram: 16GB
Brokers: 3

Assuming an engineer has an all-in comp package of $200k/yr (this would obviously be different in every situation, for every geo), year one would look like:

||Building (on AWS)|Buying (Heroku)|
|:-|:-|:-|
|software cost & licensing|$0|$21,600|
|learning & education|$7,692 (2 eng \ 1 week)|$3,846 (1 eng * 1 week)|
|implementation & testing|$15,384 (2 eng * 2 weeks)|$7,692 (1 eng * 1 week)|
|infrastructure costs (see above specs)|$12,117.60|$0 (included in software cost)|
|supporting infrastructure costs (monitoring, etc.)|$1,200/yr|$1,200/yr|
|maintenance, patches/upgrades|$15,384 (2 eng * 2 weeks spread throughout the year)|$7,692 (1 eng * 2 weeks spread throughout the year)|
|Year 1 TCO|$51,777.60|$42,030|

Directionally, this example seems correct.

What do you think? What am I missing? What did I underestimate/overestimate? How can I improve this?

Thanks!

https://redd.it/10g9bk2
@r_devops

Script or software that automatically populate specific profile in ~/.aws/credentials

cat \~/.aws/credentials


default
awsaccesskeyid = xxxx
awssecretaccesskey = yyyyy

foo
awsaccesskeyid = xxxxx
awssecretaccesskey = yyyyy
awssessiontoken = zzzzz

Every time I need to run `aws sts assume-role --role-arn arn:aws:iam::123456789012:role/xaccounts3access --role-session-name s3-access-example` then manually edit \~/.aws/credentials `foo` profile. I was wondering if there software or script that does it automatically for me?

https://redd.it/10ggej1
@r_devops

Hands-On: Kubernetes Gateway API With APISIX Ingress

A tutorial on using the new Kubernetes Gateway API with Apache APISIX Ingress. This is a hands-on walkthrough that you can follow on your own.

Read: https://navendu.me/posts/kubernetes-gateway-with-apisix/

https://redd.it/10gnldc
@r_devops

Why do some SaaS have multiple subdomains for each business domain?

What is the logic of this? I feel like it adds complexity. Only thing I can think of is BFF (Backend for Frontends) architecture. Essentially each frontend app getting their own api gateway.

Examples:

Shopify

* Auth screens and anything to do with accounts is on accounts.example.com
* Admin Dashboard has admin.example.com
* storefront entirely different domain and subdomain. hello.myshopify.com (the customization makes sense, since its public facing)

I want to know the benefits and logic of having an architecture like this. Security reasons? Increases complexity quite a bit I feel. Like JWT coming from account.example.com, but then also valid on admin.example.com.

I see Jira does this: start.atlassian.com id.atlassian.com, yourname.atlassian.net

https://redd.it/10gpiaw
@r_devops

Internal tooling ideas?

I am interested to hear the kinds of internal tooling people have created. Is there a tool you have made that had a significant impact on your team or organisation?

https://redd.it/10gsy86
@r_devops

backstage.io common Issues and Pitfalls

We are a \~500 developers organization, we are planning a project to onboard Backstage, in order to improve our developer experience, and better enable our developers.

What are the common pitfalls, gotchas, and issues we should be aware of when onboarding into Backstage? Any common plugins we should look into? Any popular tooling that is used with Backstage?

Any thoughts and feedback would be really great!

https://redd.it/10guddj
@r_devops

It seems like we are always in diapers. The anxiety of never knowing what we are doing and the future...

Reading posts here like this or this makes me feel better about how I always keep forgetting things or making a fool of myself in interviews after years of experience because I either forgot something basic from disuse or I haven't used some shiny new tool enough.

I always manage to do a good job, but sometimes I feel I am perpetually stumbling my way there.

Do you feel we are in a transition period?

I feel we live in a post Big Bang era of DevOps (and tech in general) where there are just thousands of tools that do similar things, many of them betaish, alphaish - half cooked. Perhaps the future will have fewer, but more stable and user friendly options, and we will probably be delivering internal developer platforms with these tools... and a lot of ChatGPT and AI tools :-)

Any predictions about the future along these lines?

https://medium.com/@nandovillalba/devops-engineer-perpetually-in-diapers-a2b125d5906c

https://redd.it/10gslt1
@r_devops

But really, why is all CI/CD pipelines?

So I've been deep in the bowels of our company's CI processes the last month or so, and I realize, everyone uses the idea of a pipeline, with steps, for CI/CD. CircleCI $$$, Buildkite <3, GHA >:( .

These pipelines get really complex - our main pipeline for one project is ~400 lines of YAML - I could clean it up some but still, it's gonna be big, and we're about to add Playwright to the mix. I've heard of several orgs that have programs to generate their pipelines, and honestly I'm getting there myself.

My question/thought is - are pipelines the best way to represent the CI/CD process, or are they just an easy abstraction that caught on? Ultimately my big yaml file is a script interpreted by a black box VM run by whatever CI provider...and I just have to kinda hope their docs have the behavior right.

Am I crazy, or would it actually be better to define CI processes as what they are (a program), and get to use the language of my choice?

https://redd.it/10gzdqg
@r_devops

"Accredited" DevOps Training/Cert Courses?

Does anyone know of any courses offered to prep for eventual certification in either SRE or CI/CD (DevOps Institute certs) that are considered to be "accredited"?

Seems that's the only way I can use my education funds at work and the usual programs I go to (AWS Coursework, Cloud Academy, etc...) don't see to mention anything about being accredited.

Thanks!

https://redd.it/10gxw59
@r_devops

fargate with react

Hi all!
I have some deploys in ECS (fargate) a frontend and a backend. The frontend is in a public subnet while the backend is private. The frontend uses react with axios, and initially I wanted to use service discovery, but.. I just forgot about the fact, the requests are made from client side not server side. So even though my service discovery work fine, it's useless.
Was wondering if this issue could be bridged using API Gateway or not ? Not very familiar with APIG.
Other thing was to deploy another middle server which would act as a gateway, but still, not the best solution because I have to hardcode the host in the react app, so I'd need a sub/domain for this as well to not depend on the IP.
Looking for a solution, or any ideas if you have... I have some apps and wondering how could I solve the issue the possible easiest and cost effective way.

https://redd.it/10h0asx
@r_devops