I wrote a tool to speed up the testing stage by 95%

Consider starring on Github as it really does help a bunch w/ trying to break into big corp envs.

MIT License - https://github.com/nabaz-io/nabaz

Most tests shouldn't run, what I observed as an automation engineer is that when tests fail usually only a very small subset of them do (2-5) tests out of hundreds, or thousands.

In a Israeli cyber firm I worked for, running the full suite of tests in CI was absolutely impractical as they would collectivily take hours!

I wrote a tool that runs in production CI/CD of some software companies.

I decided to go open source, It collects code coverage separately for every test and then compares the per test coverage to changed code, if no changes were made the test is automatically skipped.

P.S

It also watches for changes in env vars, config files, and resource files. (coming very soon).

Would love feedback.

Yuval

https://redd.it/y653qa
@r_devops

Help to understand the pricing of CDNs

Google Cloud Storage (GCS) be used as an origin server for Fastly CDN. We currently use just GCS and have a monthly bandwidth of about 30 TB. For that, we have to pay about $6000.

According to the Fastly pricing calculator we would just have to pay about $2800 for that bandwidth. So with Fastly the content would be loaded faster (since it's a CDN and it has caching) and we pay less than half the price?! Why the hell would we not do it?

The question is just: We would continue to use GCS as the origin server because the media is already uploaded there. So I guess we would still have to pay GCS in some parts. But in which parts? Just for storing the data and no longer for deliviring?

Or would the two prices actually just add up and we would have to pay about $9000?!

https://redd.it/y66enj
@r_devops

Has anyone tried logging with Seq, what is the pros and cons of having Seq instead of ELK or Loki+Grafana stack?

I'm digging through the internet to find the comparison with Seq and others logging tools but seems like this is the best article that I can find: Choosing the Right Log Aggregation Tool | by Jacob Taylor | Medium

So, what is your opinion about seq, have you implemented it into your project and what is it pros and cons?

https://redd.it/y69v79
@r_devops

SDK's and DevOps

How are SDK's like Boto3 normally used in a DevOps environment?

I know you can use Boto3 to create, configure, or get data from AWS resources, but can't a lot of other DevOps tools do the same thing?

https://redd.it/y632px
@r_devops

Cloud Services

Sorry if this is the wrong place, I checked the rules but please remove the post if necessary.

I am a newer dev and my team has a job That runs about 50 times per month. It just consists of a few API calls and only takes a couple minutes to run in worst case scenario.

Our internal infrastructure for hosting it is really bad and consistently causes the job to fail. I want to look at a provider like AWS or Azure, but wasn't sure where to start. What kind of service should I be looking for? Just looking for a good place ro start my research as I have 0 cloud experience.

Thanks for the help.

https://redd.it/y6dkje
@r_devops

how many of yall run entire on-premises infrastructure vs either hybrid or cloud only?

wondering how common is running entirely on-prem.

We run basically everything out of ansible, gitlab and k8s

https://redd.it/y6e8oo
@r_devops

SW Support

Does anyone have any experience with SW support in a DevOps / Kubernetes environment.

The system I am working on, will be running containerised apps and I am wondering how this will be different from traditional software support.

Such as failure, restoration activities, set up and config tasks - I imagine from my own research this will be massively different under this kind of approach.

Does anyone have any good info? Or points in the right direction.

https://redd.it/y6gczk
@r_devops

GitOps features within Portainer + GitHub Actions

Portainer released easy GitOps features recently for end-users to boost their deployments to Portainer based on Git. This new blog post takes it another step to share how GitHub Actions can be utilized to achieve the same results in a programmatical way.

https://redd.it/y6k8zz
@r_devops

Conflict between engineers and sales

In my free time I've been contributing to open source projects for the last 4 or 5 years. Over time I've started to build up relationships with individuals inside of some large companies and organizations. Today my contributions came to the attention of the management at my employer (a software consultancy). The management asked if I wanted to be a little more intentional about growing these relationships and start making contributions during normal business hours.

At first I loved the idea. The idea of getting paid to work on these projects sounds absolutely thrilling to me. As the conversation continued though, they mentioned that we would pull in a few other folks to help out. Some of the folks they wanted to pull in are sales people.

Once I found out they wanted to pull in the sales people, I instantly became uninterested. You might even say I became defensive. I know tons of fantastic sales people who are genuinely great people, but for some reason this idea seemed so unnatural to the relationships that I've built on these projects. I've built up relationships that were genuinely based around shared goals of making software better. To introduce sales people into that just felt like I would be betraying my fellow engineers.

I have tons of questions. Here are a few:

1) Does anyone else feel this way toward sales? Any idea why?
2) Do you think my reaction was or was not justified? Can you elaborate on why?
3) Would you have invited the sales people to get involved? How would you have integrated them?
4) Do you think that sales people understand that engineers view them in this light? Any ideas on ways to align the two disciplines?
5) What are some other things that you would consider if you were in my position?

Thanks for any feedback you choose to give me! I genuinely appreciate it!

https://redd.it/y6p525
@r_devops

What are some of your core devops books?

What are some of your guys favourite devops books that you would highly recommend to someone just starting out?I’ve ordered The Phoenix Project as well as The Devops Handbook 2nd Edition.

https://redd.it/y6osu8
@r_devops

Move from Data Analyst to DevOps

Basically what the title mentions. What are the skills I would need to transition or are there any skills that a data analyst has that are transferable to DevOps

https://redd.it/y6nvz0
@r_devops

Why is eBPF trending as an approach to observability?

What does eBPF do/do better than instrumenting an app directly?

https://redd.it/y6s8bv
@r_devops

In need of extracurricular advice

SO, obviously there's the coding projects, hackathons, robotics, literally anything tech-y would help, but apparently everybody has those. I want your opinion in a few clubs (less common for people in tech) that maybe tech companies could see value in? (in addition to all the other tech-related side projects/ clubs):

1. Finance clubs in general, but more specifically investments and stocks (for the math, data analysis side)
2. Model United Nations experience (for the critical thinking, efficient communication, analysis, negotiation)

I'm interested in big tech companies for software roles.

https://redd.it/y6rstf
@r_devops

Help with CI/CD using multi-repo in Github Actions

Hello everybody,

We have a project composed of frontend (Service A), backend (Service B), and one microservice for processing data (Service C).

I have setup github workflows in each repository that automatically tests, builds and pushes docker images to repository.

The part I am missing is:


How do I test/deploy the product using all of the microservices? I want to setup a staging environment where I do e2e tests on all of the deployed services before I deploy them to production.

For example, I update Service C to version 1.5.0 and I want to test the compatibility with the current version of the other services.

My idea so far is to have a ''main repo'' for deployment and e2e testing scripts and a manifest file containing the current versions the pre-release service. On successful build the workflow of Service C will update the its version on the ''main repo'' and create a new pull request. This will trigger a workflow for deploying on staging environment, and if everything passes correctly, another workflow will be triggered for deploying into produciton.

Is there a better approach to this solution? I am having trouble understanding the staging environment as a part of the CI/CD process.

https://redd.it/y6cycp
@r_devops

Company has a crazy plan to bring sales and developers together, is it realistic?

Well to put simply, my company is primarily a software development company that works sometimes with external contractors to fulfill feature requests and fix bugs if our in house team is overloaded. We collaborate on a range of platforms but our primary project tracker in-house is Azure DevOps. Now is the time of year where we are in a mad rush to finish features before the holidays arrive and today management had a plan to help us (yes I was reluctant).

The plan was to integrate AzDo with some crm software i have never heard of (Dynamics365?) and allow people who are not technical (assume a sales person) gather feature/bug requests from customers. How the workflow would look: sales person asks what could be better > takes this data and dumps it into a crm ticket? > this will translate within AzDo as a feature/user story/bug >we complete this request and close it in AzDo > and then it auto closes out in this crm software.

My huge pain is the feature needs to be more specific and at a granular level which sales people can't seem to comprehend. I am probably just overthinking this and maybe features can be a bit broad to scope. I have a feeling though I am going to get alot more feature requests that I am going to have to write user stories for. My questions for you guys:

Is this even a real world solution for software dev companies?
Are any of you guys out there using a system like this?
Any issues you see or am I just overthinking logistics?

https://redd.it/y6z80v
@r_devops

Super light monitoring and alerting stack for personal projects

I run a small single-VPS web application. My CI/CD also runs on the same VPS on a cronjob. Maybe this setup will expand to two VPS in the future but at present high availability is not a requirement.

I dabbled a bit with self-hosting Elastic stack on another VPS but honestly Kibana seems super bloated and the resource requirements of the stack are greater than the system that it is supposed to monitor. Also played around with Grafana's cloud monitoring stack but like Elastic it felt bloated and overpowered for what I'm doing.

What I need:

- Application monitoring like response codes, latency and other application metrics. This could be instrumented in my code or written to a log file that is shipped and parsed by something like Filebeat.

- Basic system monitoring - CPU, memory, restarts etc.

- Set alert conditions that trigger a notification on PagerDuty or an SMS.

What is the lightest, bare minimum monitoring stack that can do this?

https://redd.it/y6zy58
@r_devops

Fastest, financially efficient way to deploy GPUs?

I've been working for weeks now to find the best way to deploy cloud gpus. There are lots of great serverless options these days, but unfortunately none meet the needs of my use case of being able to deploy individual ML models and control each instance. I've been tinkering with deploying virtual machines with gpus on Google cloud, but it takes far too long for my use case. Looking into using kubernetes, but it's not quite clear to me exactly the best way to set it up. Does anyone have any good ideas on the best way to deploy a GPU for ML inference in the fastest most economical way?

https://redd.it/y70kh1
@r_devops

What VSCode Customizations and Plugins do you use for optimal workflow?

I'm just interested in what you think you can't work without. What your essentials are.

May it be extensions, themes, other customizations or certain systems for optimized workflow.

https://redd.it/y6z5mf
@r_devops

A discussion about optimal DB design for saving aggregations

I have raw data.

There are 5 filters defined on this raw data: user_id,f2...f5.

There are 50k\~ permutations of f2...f5 filters.

There are 365 time windows. Each time window is 1 day.

For each time window, for each filter permutation, 50 different aggregations are saved.

I think of saving this data in a single aggs table, with a schema as follows:

user_id | f2 | ... |f5 | agg1 | ... | agg50 | start_time | end_time

That means every user, on average has 50k * 365 = 20m\~ rows.

When I query the data, in 100% of the time I will filter user_id, f2..f4.

Most of the queries are performing some aggregation on top of some agg column.

i.e, select count(agg1) where user_id=... f2..f5=... agg1=... start_time=...

Filter f5 and timestamps (start/end_date) are optional for filtering.

That means the query is running on a pretty "small" number of rows.

Some queries will update some rows and backfill missing ones.

There might be endless distinct user_id of course.

1. I assume I need to index the table by user_id,f2..f4 at least. Should I partition it also in some way? maybe by user_id?
2. Should I use time-series DB? Are there any benefits in this compared to "raw" Postgres?
3. As the number of user_id increases to infinity, this table will bloat. I assume this will deteriorate the performance of the DB. What solutions I may apply to keep the querying fast? Retention? Compression?

Maybe the whole DB design should be different. Please express your general opinion on it.

https://redd.it/y7apj3
@r_devops

Guys, I'm building a docker image to build my project APK. I have a small doubt.

I made a docker image to build my project APK. The docker image is 3.4 GB in size. I want to know if there's any way to download android SDK during compile time from docker. As this will considerably reduce the size of the image. As of now, i used the command line SDK manager to install a SDK on the container.

https://redd.it/y7d3ku
@r_devops

Currently stuck at work! Can anyone help me understand what the heck SDI is? (Server-Defined Infrastructure)

Hi everyone.

I was recently tasked by my current employer to learn about SDN & SDI's. Specifically, the product below. Now, if only I could understand what the heck I'm looking at. I have a related "background", but I'm super lost. Can anyone help me break these down? Here's an example...

Lenovo Server-Defined Infrastructure (**https://www.lenovo.com/us/en/servers-storage/servers/?orgRef=https%253A%252F%252Fwww.google.com%252F**):

ThinkAgile HX Series: Designed for easy deployment and manageability, Lenovo ThinkAgile HX combines Nutanix software with Lenovo’s #1 reliable, high performing platforms.
ThinkAgile MX Series: ThinkAgile MX Certified Nodes accelerate your adoption of Microsoft Azure Stack HCI solutions (Storage Spaces Direct) by streamlining the ordering process for validated configurations with an easy-to-use machine type.

Any direction or input is appreciated. Thank you all so much!

https://redd.it/y7dtze
@r_devops