Q4 Data on how Companies on AWS are Saving Money

Hi r/devops!

For the last 2 quarters I've been writing SQL against anonymized cost data for thousands of AWS accounts. There were some big shifts in Q4, among them:

* On-demand spend for EC2 was the lowest I've ever seen, clocking in at just 31%. Companies are using Savings Plans and RIs a lot more aggressively than they were.
* A lot of people moved data off the most expensive S3 storage tier, down to 64% of costs on Standard Storage for Q4 vs 80% for Q3.
* Graviton now represents over 50% of costs for Lambda, up from close to 0% when we started measuring it at the begging of 2022.

Hopefully these data points and others here ([https://www.vantage.sh/cloud-cost-report/2022-q4](https://www.vantage.sh/cloud-cost-report/2022-q4)) provide some inspiration for how your own teams are optimizing costs this year.

https://redd.it/10p9rgu
@r_devops

Transitioning into DevOps from a fullstack application dev/generalist?

Hey all,

Been making CRUD apps for about 7 years and I'm getting bored. Exposed to almost all popular application stacks, many 3-tier web architectures, and some more exotic approaches via distributed systems.

I'm interested in having a more holistic understanding of how the software I write is operated and taking responsibility for the complete software delivery lifecycle. I've done an entry level AWS cert (AWS SA - Associate), write my own pipelines, and have screwed around with terraform/k8s/helm/prom/grafana many times in my career. Particularly interested in specializing in AWS since I see it as good niche for now and for the future.

I am detailed and process oriented - I often end up unofficially maintaining our docs, driving client meetings, and distilling high level goals into actionable work units in addition to whatever deliverables I have. Ran a team at one point because of this.

My favourite parts of my job are the "Aha!" moments when debugging a procedure or system and figuring out the solution. I need to be busy to stay happy and am motivated by providing value to others (i.e., I'm fine with doing traditional/conservative approaches so long as the end result is valued by another human) while enjoying learning new things and adding tools to my toolbelt.


From the other side of the fence (as the grass is always greener), what do you like about performing this role full time and what do you wish could be better?

Appreciate your time and your thoughts.

https://redd.it/10p5974
@r_devops

Next to all the technical and dev/ops topics - what concepts or fields should you learn about to earn advantages and new chances in your career?

A question to everyone who doesn't want to stay the engineer guy forever or already reached past this point:

What are the concepts you learned about that impacted your growth the most and let you stand out?

I'm talking about ideas, books or whole fields of study that changed how you impact the success of the company you work for.

https://redd.it/10p2cqf
@r_devops

Best practice for server management (a better way for my method?)

I'm rebuilding an old task scheduling system.
My current plan is to have a main server ssh to a few hundred machines (mostly ec2s),
run a command on the server and come back later to pick up the result/logs/status.

Other than using bash at the low level, and maybe managing a queue of my own, any idea for better implementation?

First it will be used to gather system information, once stable we might be running our backups from it. (maybe managing a machines local cron and doing local checkups to collect the status/logs).

https://redd.it/10pelam
@r_devops

Completely checked out of new role (4 months in)

Would it be seen as a red flag if I started interviewing at other places already. For context, I have 5 years of SRE/DevOps experience. I started this new role mainly because of the compensation package offered and the tech stack. Fast forward 4 months I am starting to regret that decision.

The company itself is highly segmented due to the fact that they are acquiring other smaller companies and there are only 2 DevOps engineers tasked with this transition. With each new acquisition, I have to work with the CTO of that acquired organization to fix all of the siloed messes that have been created beforehand. One example is being tasked to secure a bunch of storage accounts that have public access enabled. You would think that you can just enable the “Only from selected VNEts” feature and call it a day but no, each storage account is tied to a numerous of production services and blobfuse mounts and there are over 100s of them so pinpointing this is a huge time waster. In addition to this, They have 3 SQL pools with over 500 SQL databases inside of each pool and I am tasked with creating smaller pools out of these. Again all this just for one of these “acquisitions” there are 3 other ones and more on the way. The CTO at the aforementioned place is also a know it all that is very hands on and often tends to dismiss your suggestions in lieu of his.

Any advise from any senior on here or someone with general advise would be highly appreciated.

https://redd.it/10pdz39
@r_devops

Creating a placeholder dummy cloud watch alarm?

I mean sure we can create some alarm with thresholds to make sure it never triggers but I was wondering if there is some easier way to signal the alarm is a placeholder alarm or some convention people follow? How would you do it if you had to create a fake placeholder alarm?

https://redd.it/10pdm1l
@r_devops

Centralized Alerting System

What is the recommended approach for efficient alert management?

Considering the current scenario of having multiple disparate alert sources (e.g CloudWatch, Robusta, Logzio) with varying structures? It becomes challenging to determine the required fields and monitor the open alerts, thus seeking a central solution to consolidate all alerts.

​

Can you share some of your experience please?

https://redd.it/10p1287
@r_devops

DevOps career

Hello all,

I have been in the IT industry for the past 8 years and I started from the bottom as a QA, learn to code and got a Developer role, got into Could technology and now i am contemplating DevOps. I have a done multiple projects with tools like Ansible, Terraform, and I am interested in the idea of CI/CD. My concern is, what is the career path of a DevOps? What’s after DevOps?

Let me know your thoughts.

https://redd.it/10pjpvy
@r_devops

Docker Question

Can anyone please answer the following question related to docker.
Two dockers are running on two sperate hosts. Hosts not able to ping or communicate to each other. Can docker ping or communicate each other without changing hosts network configurations and how?

https://redd.it/10ozgu1
@r_devops

Manually run cloud-init user data on bare-metal



We have a cloud-config made for virtual servers and we want to remove the virtualization layer and have the same setup on bare metal server. Is it possible to manually run that cloud-init user data script on the bare metal servers, after the server setup ? (OS install - Centos 7)

I have read about nocloud datasource for cloud-init and configuring os setup to fetch cloud-init data from an usb drive but for me it's not possible, I must find a solution to run run my user data via cloud-init manually after OS was installed

https://redd.it/10ou73c
@r_devops

Senior Devops to Staff Devops

I have been working as a Senior Devops Engineer and have a total experience of 7 years.
I am about to move to a new organization where I'm offered the role of Staff Engineer Devops.
How should I transition from my current role to new role smoothly?

https://redd.it/10ou6rm
@r_devops

Is freelancing better than full time jobs ?

Hello I am software engineer student with experience working as a serverless engineer, i just got my aws developer associate certificate and I started working as a DevOps Engineer , also started getting jobs on upwork
Now my question is, should I pursue my career as a freelancer or focus on my official job

https://redd.it/10omqsy
@r_devops

Learning Jenkins, is it still good?

I recently started learning Jenkins. I set up a Jenkins multi branch pipeline for a React project that installs npm dependencies, runs tests, builds, and then builds the static site it to a docker image and uploads to my registry. Overall, this process was a bit of a pain in the ass because the Groovy language is odd and testing takes like 4-5 minutes each time. For instance, I had troubles with the last step and I could only test it by going through the previous ones which took at least 4 minutes each time. I also had issues with the specifying the agent, where if I wanted the last step to not run in a docker container I had to move the agent declaration to the each of the previous 3 stages instead of keeping it as the default agent of the pipeline. I have not yet been able to figure out how leave it as the default and to just specify for certain stages to not use that agent.

So I am wondering, am I alone in finding Jenkins a little bit annoying? Should I continue learning it? Should I learn a different tool for CI/CD?

https://redd.it/10ofizp
@r_devops

Lambda to ECS

Hi! I'm building a web-application for doing file operations like converting formats or optimizing file size. I'm running this on Lambda functions and it works properly, but I know Lambda's not the most suitable architecture for converting a 20 min video from .MOV to .MP4 because it takes too long and if I get some traffic the costs can go high.

I'm thinking about migrating some of these "heavy" processes to ECS, and the architecture I have in mind would be something like this:

Current architecture: Our API backend receives the conversion request, calls, and waits for a lambda function until the conversion is finished.

New architecture planning: Our API backend receives the conversion request, creates an SQS message that spins up an ECS task for doing the actual conversion, and of course all of this would be asynchronous

​

Do you think it's the correct decision? Would you suggest another approach?

https://redd.it/10oaz40
@r_devops

How Does an Open-Source Workflow Engine Support an Enterprise-Level Serverless Architecture?

https://kubevela.io/blog

https://redd.it/10pskxl
@r_devops

Using ssh keys with ansible drone plugin

Hey there

I'm trying to switch to drone. I'm stuck in a situation where I need to use ansible drone plugin with my ssh key which is stored in secrets and passing the secret to plugin. It doesn't work.

Can anyone help me to figure out.

https://redd.it/10nygtl
@r_devops

Is it possible to have a reverse proxy for my setup with NGINX?

Hi, I'm currently working on a fullstack webapp, where I have two dockerized images:

A frontend in ReactJS that runs on port 3000 and is mapped to 80 on my digitalocean droplet

A backend in Node.js that uses Express to have REST apis (like /getinfo/:user or /postuser/:name) running on port 3001

Everything was working well when I just ran both of these images simultaneously on a droplet when it just had HTTP. Recently, I have added CloudFlare to my website which comes along with HTTPS, however, it seems Express doesn't really seem to work well with HTTPS (as I've tried with a seemingly endless number of solutions), and the preferred method is to have an NGINX server that acts as a reverse proxy to handle these requests as HTTP so that the Express server can handle them without me needing to modify the code.

The problem is, I have no idea how to do this, nor where to start. I've been searching the web for about 8 hours today trying to find a solution, trying several things, and none seem to work. Is there any way I can be pointed in a direction that could help me? It's been very frustrating basically being 'done' with everything except this last issue.

https://redd.it/10nt7wi
@r_devops

DevOps Learning

I’m a support manager with 14 years of work experience in Linux/MySQL support. I’m interested in learning about CICD/ Docker, other DevOps tools however I feel overwhelmed after learning one tool. I just finished a course on Git and have started learning Docker. Please help me with the order in which I should learn other DevOps tools and which ones. Thanks !

https://redd.it/10pwg7x
@r_devops

Revert merge changes on feature branch in Jenkins pipeline if build fails

Edit: my feature branch is connected to a website and I’m running selenium on it so I kinda have to push in the beginning when I merge the feature branch.


My Jenkins is set up so if someone makes a pull request to the main branch a Jenkins build is triggered with three steps
1: merge pr branch to feature branch
2: run tests on feature branch
3: merge main branch with feature branch and close PR

If steps 2 and 3 fail or have any error but step 1 passes I want the feature branch to be reverted.

I have an I’d statement to revert but idk if it’s right:

Failure{
Script{
If(currentBuild.result=="Failure" && (currentBuild.currentStage.name== "step2"|| currentBuild.currentStage.name== "step3")){


code to unmerge



Thanks so much.

https://redd.it/10pvshe
@r_devops

Different content but static URL

We want to have the internal report private. After every (main branch) pipeline run, we want the static URL to have the content updated with the new updated report. Is it possible to do this? I can use AWS or azure

https://redd.it/10pwri9
@r_devops

What are the problems that people are facing right now with different cloud providers?

I am building something in DevOps and I am reaching out to other users about the struggles they face while deploying their applications on several cloud providers like Digital Ocean, AWS, etc. What are the problems they want to be solved? Is it the dashboard UI/UX, pricing model, or the environment?

https://redd.it/10q0b5i
@r_devops