How do I deal with latency workloads in a multi-cluster Kubernetes-based platform?

I want to deploy an application and the idea is that it can be deployed globally where the effect of network latency is brought to a minimum. Can i get help on how to start researching whether this is possible?

https://redd.it/ywudpx
@r_devops

What will the benefit be? AWS Security.

Hello there,


So earlier today i finished up my required tasks to begin the transition to devops. Last time i spoke with my boss about it i was supposed to be moving on to docker, etc. However my boss told me to focus on getting the AWS Security cert. What benefit would there be to an aspiring devops engineer having a security cert? I was expecting him to tell me to get the solutions architect / devops cert for amazon.


What do you all think?

https://redd.it/yx24zb
@r_devops

Should I switch Jobs?

Hi All,

I've received and verbally accepted a job offer but haven't signed the contract yet or handed in my notice.

I'm getting cold feet and thinking it isn't the right decision after recieving the contract.

I'm currently a senior site reliability engineer at a scale up tech company and like my work and team. I only looked for a new role as it was announced there was going to be redundancies. It turns out I'm not on the to be axed list.

The job offer I accepted is to be a team leader at a subsidiary of a big tech company (I could put the big tech company on my CV though). They are offering an additional £5k a year salary and £20k in RSU that vest after 4 years. However, they require me to be in the office 50% of the time and their working hours are longer. On average with longer working hours and commute I would be doing an additional 8 hours a week. They also are offering 10 days less holiday a year.

The new job seems a good opportunity in terms of career progression as I would be jumping up to a team leader position but in terms of finance and benefits it seems much worse. Should I back out?

I would feel bad backing out as I did negotiate with them and had a celebratory dinner with the recruiter.

https://redd.it/yzwlw9
@r_devops

All-inclusive online game: Redis, kafka, both?

So I probably got the general idea: redis is more of an in-memory db, while kafka is more of an event bus/queue (among others). I know this is not the whole description, but it's all I know atm.

Now, I have a Unity game, online, multiplayer, turn-based, with different dev environments/stages each with its own postgres instance, all that ci/cd infra for both the game (which runs on pc, webassembly, mobile), editor (pc) and the back end (.net 5, cross-platform, but mostly ran on linux under docker).

There are also 2 systems for live content: one for game assets (asset bundles in Unity terminology), one for gameplay configs (general game rules and per-level configs), and coming soon a liveops tool to manage the game's economy, schedule events and such, which will need its own backend.

Question is simple: could I hit many of those birds with 1 stone, in terms of the communication protocol?

Here are a few examples:

- when a game client web build (web assembly) was uploaded to a certain bucket in google cloud, there's an event fired to a messaging system (e.g. by a github action) and a python script which waits for that event: stops the current live WebGL webserver, downloads and replaces it, then starts the new one, so 'play.game.com' always reflects the latest build, and ofc 'play-dev.game.com' uses a different build.

- general-purpose messaging between back-end components

- anyone can host a game server, so a game server has to register itself with the master server (this already happens, but it might need extending in the future and a messaging system seems better for faster development. I'm currently using websockets for Everything, which is overkill)

- player x invites player y to his game

- chat-like functionalities (I already have a third-party solution for this, but other chat-like messages would be needed over which I have full control - or even a completely separate, rudimentary chat as a backup system in case the main one stops for some reason)

- logging/analytics (I know third-parties have analytics packages and Unity Analytics will be used here, but it's not a silver bullet, and ofc I'll need backend analytics as well, to understand how all of the systems communicate and identify abnormalities)

I know how to 'just do it', but story of my career is: "I didn't knew that tool existed - what an easier life I could've had, had I know about it"

Other details: this is currently a 1-man project (excluding art) that will soon expand to probably 5-10 people, and I foresee at least 8-10y of runtime. Game is not yet available to masses, and I imagine it'll roll out slowly: 100 ppl in 1st month, 1000 in second month etc. Anything can happen, but we don't expect massive sudden loads on the infrastructure, so performance is currently secondary

https://redd.it/yzyalq
@r_devops

Introduction to Docker: A Beginners Guide

Blog post on Introduction to Docker: A Beginners Guide that covers all the basic concepts of Docker, its Architecture, and how it can be used for real-world applications!

Link: https://karanjagtiani.medium.com/introduction-to-docker-a-beginners-guide-for-2023-cbf9be911352

https://redd.it/yzwg2m
@r_devops

Awesome CoreDNS

https://github.com/mariuskimmina/awesome-coredns

I started an awesome list for CoreDNS ressources. There are probably still a lot of things missing but I thought it was already worth publishing so that other people can add to it as well (PRs welcome).

https://redd.it/z00siu
@r_devops

docker container to redirect host traffic to proxy server

I have an app(https://pawns.app/cli-download/) that does not support proxy server under ubuntu system, I want to know if it is possible to create a docker container that redirects all traffic to proxy server or how should I solve this scenario?

https://redd.it/yzxqsl
@r_devops

Looking for a pass with mtls

Hi everyone,

We’re starting a new project where it need to be really stable and fast on networking and autoscaling.

Another aspect asked is to be the best as we can cloud agnostic and we’re a small team so it need to be the simpler as we can too. And… stored in Canada

I know it sound a contradiction but at the days I worked on pivotal CloudFoundry release’s and it was not so bad.
Now when I’m looking for a paas, I see lot of solutions but where we need to keep a updated kubernetes to run the paas on it which is something we want to avoid if possible. Do you guys know something that can fit that ?

https://redd.it/z035d8
@r_devops

What according to you is a developer environment?

Hello Folks.

I have been hearing about this term for some time now and wanted to understand what does this mean?

In our organization, everyone installs vscode on their laptop, and the required language. Developer environment is kind of ready, is there anything else?..

Pls share your thoughts

https://redd.it/z02e3b
@r_devops

How to organize E2E testing?

I am in an org dealing with financial data (SQL). A lot of the functionality is just CRUD operations but there are some more complex parts such as FIX trading and dynamic PDF generation.

In this environment we have limited unit tests as most functions are to update or select from the database and a lot of calculations are using aggregate functions from the DB.

Truly complex calculations are scripted as functions and they have unit teststhey also dont need to be updated and aren the places wev run into issues where functionality doesn't meet expectations.

Integration/E2E testing is where we've found tests are valuable however if we setup the environment for each test, it'll take forever to test the entire system (800+ routes, 50+ jobs, 50+ reports).

Instead our approach is to create a "data story", throughout the tests, data is created by the testsused and updated as the tests execute. However this does mean at least some test suites are based on others, and that when ealier tests are change they break later tests which starts to become unwieldy.

Thoughts?

https://redd.it/z04aqw
@r_devops

TraceView - OpenTelemetry UI released

We have released an early version of our OpenTelemetry UI "TraceView"

https://github.com/asynkron/TraceViewDeploy

TraceView is intended as a tool for developers, to pinpoint issues in microservice architectures.
There are plenty of good and scalable tools for observability out there, DataDog, Grafana, etc.

But they do tend to focus on DevOps and SRE, e.g. focus on latency, focus on huge logs.

TraceView rather tries to combine and analyze this data to show meaningful views where you can reason about what is going on.

e.g. as a replacement for Jaeger or similar on local dev machines.

Any feedback is welcome

https://redd.it/z05bcc
@r_devops

Please criticize my SaaS architecture

https://drive.google.com/file/d/1JNnqSIbkSikTjtmjQYNtsDHbclL\_Pnot/view?usp=sharing

Monthly budget: 150 USD

I have also used the AWS calculator to sort of determine how much it will cost monthly, see here https://drive.google.com/file/d/1V-ZQrEyBYYl1PRaLJ5L\_SLjOeyRD5U81/view?usp=sharing

Idea: It's a serverless WordPress hosting infrastructure

1. The site owner sends a request to setup WordPress
2. There are two containers in the fargate instance (WordPress and DB container), Data are persisted in the DB
3. EFS for volume mount
4. & 5. Lambda function to deploy the static site to S3
5. The same with 4
6. Managing end-user requests to s3 bucket

​

This is just an high-level overview, I didn't go indepth on security and managing DDOs, however, I welcome all suggestions

GOAL: The goal is to reduce the budget as much as possible to 120-150 Monthly

​

Thank you

https://redd.it/z08vmc
@r_devops

A poor man's API

Creating a full-fledged API requires resources, both time and money. You need to think about the model, the design, the REST principles, etc., without writing a single line of code. Most of the time, you don’t know whether it’s worth it: you’d like to offer a Minimum Viable Product and iterate from there. I want to show how you can achieve it without writing a single line of code.

Read more

https://redd.it/z0a1zx
@r_devops

New to devex and worried my work isn't going to scale -- I could really use some advice; as broad or as narrow as you like

I started as a SWE at my current job, and kept stealing time from other projects to work on CI improvements and process automation. This work was well-received, management asked if I wanted to do DX full time; I said sure. For context, we're at about 30 devs right now.

I'm a department of one, for now, and have huge leeway to set my own "product direction," goals, KPIs, etc. This is really exciting, but I haven't done it before, and I'm not good at it yet. I started by holding meetings with eng management and running a developer experience survey, and came to the conclusion that my team's two biggest problems were broken development environments and tests that were difficult to run locally and flaky in CI.

Since assuming this role officially a few months ago, I've put the majority of my effort toward environment standardization. It hasn't gone particularly well. Major problems include:

* **Bugs in upstream tooling** \-- we're using VS Code's dev containers right now, and it's been a rough ride, with poor environment caching , intermittent port forwarding failures, and disappearing git credentials being particular pain points. We weren't doing great with purely local environments, though, and I'm not sure whether other solutions will be worth the pain of switching.
* **Sloppy custom tooling** \-- we have our own process runner that we wrote to replace Invoker, and a simple deployment UI in Slack, both of which break all the time and have poor DX. Neither was ever meant as a long-term solution, but both are now ingrained enough that switching will be an ordeal. While I'm aware of many other options here -- docker-compose or tilt for task-running; Terraform Enterprise for deploys since we use it for other infrastructure stuff already -- I'm again worried about making a bad bet.
* **Hidden information** \-- I get a lot of inbound about problems people could solve on their own... if they knew where the information was. Some of this is related to the aforementioned weird task runner and its bad DX; a missing dependency or occupied port will just manifest as "the app won't load" and then I have to go spelunking. CI is a big one, too: our integration tests, for instance, record videos and an incredible amount of debugging information but all of that is hidden in the CI runner's UI. Most devs don't even know we record it until I show them in a debugging session; I haven't settled on a good way to surface it.

So, I'm looking for advice from experienced practitioners -- I recognize that I've asked an unhelpfully broad swath of questions, but right now I'm not confident in my ability to decide which problem is most worthy of my attention, so I kind of need to ask about the whole job.

This could be as broad as "strategies for interfacing with devs and determining priority" or "here's how I evaluate tooling," or as specific as "this is a tool for environment management that has consistently worked for me." Basically, anything you would want to know if you were just getting started, moving over from being a SWE.

https://redd.it/yz3g21
@r_devops

Side work contracting

I have a company that is interested in hiring me for some contract work to support their junior DevOps person and also guide the DevOps side of the tech team. It's a small start up with very few people outside tech.

Currently their systems are a mess from what I can tell. So it will be a bit of work to resolve it all.

A bit about me, I have held DevOps or SRE roles for the last 4.5 years working at companies ranging from 30 person start ups to a 6000 person multi billion dollar org.

I currently make about $110 CAD an hour and am trying to come up with a fair rate to charge this prospective company. I would be doing the work after hours of my day job. I tried to look up rates online but came down to a range of $80-250 which I can't really tell where I should ask for in there. Based on what I said, what would be a reasonable range to aim for? I was thinking $150 since that's a bit above my day job but not too high. I was thinking a bit higher than my day job because of no benefits and PTO. Or does it make sense to ask for the same rate as my day job?

https://redd.it/z0d2rw
@r_devops

Tilt vs. bespoke Kubernetes tooling

I've been reading up on Tilt lately, and I think the concept of development in k8s clusters could be really beneficial for our team. As it turns out, somebody else on the team who was unaware of Tilt had recently had the same thought, and he's been building out his own tooling for cluster-based development within our internal dev CLI.

My initial instinct is that we should use an off-the-shelf thing for this since it's available. But inspecting my colleague's code, it seems like the job of actually starting/updating resources is only a small part of what he's put together -- much more of it is devoted to

wrapping that process in a CLI flow that's attentive to the particulars of our application
determining which services to update and when (the precise arrangement of k8s resources is done in Helm templates which are parameterized and triggered from the CLI)

Tilt's UI is phenomenal, and I'm tempted to use it to run our services for that reason alone. But I'm not sure whether there's benefit to going deeper than that. It seems like, in Tilt-world, the logic for what should update and when would go in a Tiltfile, and the interactivity would still go in a higher level tool (since AFAIK Tilt/Starlark has no mechanism for interactive user input). But given that my colleague has already done a lot of work on the orchestration in Go, is there any value in porting it? Or to ask the question a different way, is there anything cool that Tilt does for you that it might not be obvious you even want until you've done cluster-based development for a while?

https://redd.it/z0czs0
@r_devops

Scenario based question for DevOps Engineers out there

Trying to do a challenge I was given, I was wondering how different people would approach responding to questions like these in a real life environment. Just indulge me if you can :)

You need to set up a new product development effort for a new team. The product will be built in Python on AWS. It will be a database backed product with an APIs and an Angular webapp. The product will be containerized and will be deployed in AWS. The DevOps will heavily use Terraform.
What questions do you have about the setup of this product development effort? (Really looking forward to your answers on this)
What are your first steps?
Preferrable to do these steps in Terraform (you can skip this part)

​

Additional questions if you have time to answer (more of what your boss would look out for I guess):

What technologies do you suggest? Are they modern, well explained, practical?
What kind of questions do you ask about the effort? Do those seem like practical, curious DevOps questions?
How do they plan to set up the Terraform? Do they think of remote state? How much console work do they plan? Do they reference the API/CLI? Are they suggesting Terraform workspaces, modules, variables, conditionals, tfvars, etc.?
What are their bootstrap steps? Where do they plan to keep their files and state? Do they plan to use GitOps approaches? If so, how do they set that up?

​

So in essence what kinda questions would you ask to start off and set up when your team gets a new project, keeping in mind you're provisioning with Terraform.

https://redd.it/z0galk
@r_devops

Free Auto-Unseal cloud service for a private Vault instance

Hi all,

As the title says, I am looking for a free way to auto unseal my private vault using any cloud service, I already checked a couple and they already charge for everything which will be a pain to monitor.

Currently I am using this nice project but was wondering if a cloud solution would make more sense. Please advise.

Cheers,

https://redd.it/yx00fb
@r_devops

robocopy and hidden file

Does robocopy.exe copy hidden files by default ?

Do I require to give any parameters/options in the command ?

https://redd.it/ywubzx
@r_devops

What do you guys use for outage monitoring / incident reporting?

Seems like there are a lot of different services out there offering this kind of thing nowadays.

Which ones do you use, and if so, what should I look for when picking one? Any I should check out in particular?

For context, I manage the devops pipeline on a number of projects - smaller teams (under 10) covering a wide range of services.

https://redd.it/z0lcss
@r_devops

Need help with documentation for blackduck installation on kubernetes

Help!!

https://redd.it/ywsg67
@r_devops