Any ideas for how to complete 100 vCPU-seconds worth of tasks in less than 3 seconds?

This app's workload is super spiky. Most of the time, no work is done. However, when a user presses a button (the time difference between button presses can be anywhere from seconds to days), we need to immediately run 100 different jobs in parallel, each which require a full vCPU, 1GB of RAM, and which will run for 1 second. The most important constraint is that we have to complete all 100 jobs in under three seconds before returning the results to the user (this is an explicit requirement, yes it's as bad as it sounds.) How do we achieve this while minimizing costs?

Yes, all the jobs are unique, yes, they cannot share their computation, yes, I hate this too.

Ideas we had so far include:

1) Invoke 100 Lambdas to do all the jobs. Since Lambda prices by GB-second, we would have 1 "starter" function with 128MB of memory reserved that runs for 3 seconds, and 100 "job" functions each with 1728 MB reserved that run for 1 second each. (Lambda scales vCPUs by memory; at 1728MB they give you 100% share of 1 vCPU, and any less than 1728MB would give you a fractional share of a vCPU - which doesn't work for our purposes.) This would entail 172,800 + 384 = 173,184 MB-seconds of usage, for a total price of $2.89e-3 per button press. However this doesn't really feel like the most optimal solution; 1728 MB of usage per job even though we only actually need 1GB feels wrong to me, plus the starter function feels a bit inefficient. Are any other serverless providers better / cheaper for this sort of thing?

2) Use EC2 on-demand autoscaling (or similar). This is actually cheaper per second than Lambda, but since on-demand has a minimum billing granularity of 60 seconds, this ends up being more expensive once you factor in the 60x blowup. Are there any cloud providers that bill by second for on-demand compute? How is the latency with on-demand compute?

3) Having a 36-vCPU server that just idles most of the time. This seems by far the worst, since any server of that capacity seems to cost hundreds per month, and we aren't expecting more than maybe 10k button presses per month (~$33/month using Lambda.)

Any more ideas? Thanks.

https://redd.it/12dkn0p
@r_devops

Studied 2.5 years for DevOps, what now?

I’m 39 year old guy based in UK London who didn’t attend college which essentially means formal education finished at age 16. For the last decade I’ve worked for myself in a totally unrelated field as a filmmaker.
2.5 years ago with no IT experience, I decided I want to be a DevOps Engineer. Since then I’ve been studying and have acquired a range of knowledge for the role, with certifications in AWS SA, Kubernetes, Terraform and Linux RHCSA.

Thanks to sources such as this subreddit I’m aware certifications alone can’t compete with experience so I have completed a number of projects displaying my proficiency with these tools, details for the projects can be found at the end of this post.

When I’m not going deeper in to learning Python I’m applying to DevOps roles every day on several job sites and recruiters as well as HR are reaching out but given my conversion rate, applications sent to contacts received, I think my strategy to get the job might not be fully optimised.

My question is if you were me and want to be employed within the next 3 months in a preferably DevOps type role or as close to it as possible, given your view point and unique perspective as someone already in the industry, what would you recommend as my next move?

I appreciate this is not an entry level role and most DevOps Engineers worked their way up whether they started from Tech Support or Software Development or elsewhere. With that in mind I imagine the only advice some are willing to give, is to start from the beginning and work my way up which I’m not entirely opposed to however it would be after exhausting every alternative. All responses are welcome. Thanks.




Project I

DEVOPS ENGINEER • PERSONAL PROJECT • AWS CI/CD CLOUD HOSTED C.V.

Successfully deployed a cloud-based website by utilising AWS services including S3, CloudFront, Route 53, ACM, and API Gateway, as well as infrastructure as code via CloudFormation, for an efficient and streamlined deployment process.
Implemented a visitor counter feature on the website using Lambda function and DynamoDB, which is triggered by API Gateway, showcasing proficiency in serverless architecture and AWS Lambda using Python code.
Ensured security and accessibility of the website by utilising an SSL certificate from ACM and hosting it in a custom domain in Route 53, demonstrating a strong understanding of AWS security best practices.
Utilised continuous integration and continuous delivery (CI/CD) through GitHub actions to automate the deployment process, showcasing expertise in DevOps and automation.

Project II

DEVOPS ENGINEER • PERSONAL PROJECT • IAC TERRAFORM SETUP AND PROVISIONING

Designed and implemented an automated infrastructure using Terraform, creating a VPC, EC2 instance, and security group to provide a secure environment.
Developed a remote execution script provisioner to deploy Jenkins, Terraform, and Kubectl on the EC2 instance, reducing setup time by 70% and minimizing human error.
Configured a role and policy for the EC2 instance using AWS IAM, enabling secure access and limiting permissions while allowing the instance to connect to other AWS resources as needed.
Integrated CI/CD pipelines into the development process using Jenkins and Terraform to automate code deployments to the infrastructure, ensuring fast and reliable deployment.

Project III

DEVOPS ENGINEER • PERSONAL PROJECT • IAC KUBERNETES AND JENKINS

Designed and implemented a robust CI/CD pipeline utilising Jenkins, Git, Terraform, EKS, and Helm, resulting in a streamlined deployment process.
Developed and integrated Terraform scripts to set up the EKS cluster, with automated provisioning of infrastructure resources such as IAM roles, security groups, and Kubernetes worker nodes.
Orchestrated the deployment of a microservices-based application using Helm charts, enabling automatic scaling and versioning of the application across the EKS cluster.
Leveraged git hooks to ensure version control, code quality, and security standards were maintained throughout the development

and deployment process.

Project IV

DEVOPS ENGINEER • PERSONAL PROJECT • ANSIBLE MONITORING SETUP

Successfully installed Prometheus, Grafana, Node Exporter, and Alert Manager using Ansible playbooks for monitoring system metrics and alerts.
Automated the deployment process by creating Ansible roles that start and enable the necessary services, reducing manual intervention and ensuring consistency across multiple environments.
Implemented dashboards and alerts in Grafana to provide real-time insights into system performance and status, improving overall system monitoring and troubleshooting capabilities.

https://redd.it/12dnvsk
@r_devops

Flask app container on fargate unable to connect on mongodb container

is there a way to specify a hostname on fargate for my mongodb container? Im just doing a poc and studying my way around fargate. this is why db is included on task.

https://redd.it/12dqy5l
@r_devops

Kubecon AMS

Anyone coming to Kubecon in Amsterdam?
I'll be standing at FluxNinja booth SU50. I've recently started working as an SWE ( new grad here ) so feel free to swing by our booth and have a chat with me or the team!!

https://redd.it/12doj0a
@r_devops

Setting up first Kubernetes Ingress

Looking for some advice on picking a Kubernetes ingress system with gateway support. Currently trying to decide between Project Contour, Emissary-Ingress and Gloo Edge (open source, not paid product)

I need to dynamically add new hostnames as customers are on-boarded, so ideally having something with easy instructions on how to make an RBAC role that can only make new routes & SSL certs using cert-manager.

I'm pretty dead set against using anything based on nginx since adding a hostname seems to pause all traffic during the reload.

https://redd.it/12dugnw
@r_devops

SSO for Third Party Users and Web App Auth Integration

I am looking for a solution that will handle SSO between some third party services we are hosting content for our customers. Such as Vanilla Forums, Mediawiki, etc. I have always been the administrator for SSO platforms such as Azure AD and OneLogin for in house tools that employees use with Active Directory and SSO integration. In this case, we just want our customers to have a seamless experience through our various apps that we provide to supplement our products. Without having to manually login to each separate application. We will obviously have many different domains. Thoughts?

https://redd.it/12dwqcq
@r_devops

Where do I find clients for consulting? Do I cold call?

Hello,

I’m a devops engineer looking to start consulting privately. Actually, I do have a client on the side right now but now much work. I have a registered business, website, but I don’t know where to begin with finding clients that need my services.

I’ve tried emailing some startups but no answer. Should I cold call? How do I know if my website or proposition emails are even effective?

https://redd.it/12dw41c
@r_devops

Spark simple query to Ceph cluster - Unable to execute HTTP request: Unsupported or unrecognized SSL message

I feel I'm missing something trivial here, please help.

Long story short, I'm trying to query Ceph using Spark and I'm getting an error "Unsupported or unrecognized SSL message".

Does anyone have any idea what to do?

More details on StackOverflow:

https://stackoverflow.com/questions/75952678/spark-simple-query-to-ceph-cluster-unable-to-execute-http-request-unsupported

https://redd.it/12dv9f5
@r_devops

AutoTune - a tool's whose goal is to optimize resources and costs in AWS Accounts - Feedback / Testers Requested

https://auto-tune.pateljay.io/#/

AutoTune's job is very simple. It is to clean and optimize cloud resources (aws). Stop wasting time updating and cleaning up unused resources across AWS Accounts to reduce costs. This tool works by modifying various cloud services configuration to a more optimal cost such as decreasing retention rates, optimizing requested hardware, enabling ondemand usage. Right now, all it supports is aws cloudwatch loggroups cost optimization.

https://github.com/jay-babu/auto-tune

Looking for feedback on the idea and any tips of where to go next with it!!

https://redd.it/12e0lqn
@r_devops

Spring Boot pod takes 60 seconds to become ready; trouble handling spiky workloads

This is running in kubernetes, and we've got somewhat unpredictable scaling demands so horizontal autoscaling is working but with the caveat that many requests are either failing or taking a very long time to fulfill during spikes. The result is poor end UX as the service is an intermediary step in a business process. The user must continue onward with the results of these API calls on the next screen in order to complete their work. One obvious solution is to keep a minimum replica set of underutilized pods available and scale at lower cpu/mem utilization metrics.

Any springboot devs have tips on startup times? Any stories of how you've transformed some archaic systems to tackle a similar problem?

https://redd.it/12e9n8n
@r_devops

Any tool for "Terraform Diff Coverage"?

I started making something of my own, but basically I want to assess the quality of a terraform/terragrunt repo by counting the percentage of resources that need change/the resources in general, as opposed to resources who are applied and healthy, in any module that exists under the main tree. For example:

Overall repo "coverage" - 65%

----> /dev
----> /eks/terragrunt.hcl (60% of resources applied, 40% diff)
----> /s3/main.tf (100% applied)
----> /prod
----> /eks/terragrunt.hcl (100%)
----> /s3/main.tf (0%)

My hope is that I can put a shield badge on the repository's README so that my team knows if the main branch has any diffs at any given moment, in case something happens in the cloud or a PR causes something to change

Ps. I am aware that in a perfect world I should lock the cloud and allow only IaC and my CI should cover it all, but I don't think this methodology is mature enough yet, at least not in my organization

https://redd.it/12edesn
@r_devops

Is it common to transition from Security automation engineer to devops/devsecop?

Hi everyone, currently I am working as a security automation engineer where my day to day job involves with automations (powershell, bash, aws ssm, soar tool), using cloud environments, constant use of CLI . Currently also learning stuff in references to the devops roadmap (cloud,iac) but I have no experience working in a development environment as my industry is more of a cybersecurity/IT than say normal development environments with their dev pipeline

My question is, how do I transition myself to learn from the dev side of things if I want to slowly transition to devops. Do I just blindly create projects in hopes that I can apply devops solutions to some of it or is it better to go to other more adjacent role?

Sorry if the question seem redundant

https://redd.it/12efbtw
@r_devops

Should I use base for a single command that runs every minute instead of crontab?

I am containirizing a laravel application. What I am making is a container that will run laravel cron scheduler:

FROM php:8.1-alpine

# Load nessesary php module here

COPY . /var/www/html

RUN echo "Installing composer \n" &&\
    php -r "copy('https://getcomposer.org/installer', '/tmp/composer-setup.php');" &&\
    php /tmp/composer-setup.php --install-dir=/bin --filename=composer &&\
    rm -rf /tmp/cpmposer-setup.php &&\
    chmod +x /bin/composer &&\
   cd /var/www/html &&\
   composer install &&\
   rm -rf composer

COPY ./entrypoint/entrypoint_cron.sh /usr/local/bin/entrypoint.sh

RUN echo "Fixing Permissions on Entrypoint Script \n" &&\
    chown root:root /usr/local/bin/entrypoint.sh &&\
    chmod +x /usr/local/bin/entrypoint.sh &&\
    usermod --shell /bin/bash www-data

ENTRYPOINT ["/usr/local/bin/entrypoint.sh"]
CMD ["php","artisan","schedule:run"]

The image will have the following entrypoint:

#!/bin/sh
while true
do
    "$@"
    sleep 60
done

But what complications my approach may have instead of using crontab inside docker container?

Laravel has its own configuration for crons and schedules: https://laravel.com/docs/5.8/scheduling

https://redd.it/12ef599
@r_devops

CICD help

is it possible to create a Java gradle build on circle CI and send that build to code deploy for deployment ? if so how ?

i created a job for build another for codeDiploy. but when codeDiploy sends files to server. build file is not found in bundle

https://redd.it/12edp48
@r_devops

Did I fuck up by choosing a wrong career at the end of the interview

I am a new grad applying to my first FT job and made it to a final round interview for a full stack web dev role recently. And everything was going great, senior engineers were impressed by my background of strong backend and AWS cloud skills as well as some frontend experience.

But in the last five minutes one of then suddenly asked if I wanted to be a devops as it fits my skill better. I was about to zone out due to the stress of the long interview so I was just mindlessly aiming to nail the role I was APPLYING for and didn’t really consider other roles that might work better for me.

So I said something stupid like “I am more interested in react than devops and will lean toward web devs.”

But after some thought now I feel like my background indeeds fits devops better. I did many cicd work and automation and created apis during my internship and it’s only my personal projects where I explored some of full stack and my front end skill is really weak compared to my backend knowledge and cloud.

I fear this is going to be bad. This is my first job so I kinda realized I should have chosen my strong suit as a career instead of betting on learning something new as an intern. Also apparently I really like AWS (passed cloud practioner and going to SAA soon) and appreciate higher salary as well.

Would it be bad if I follow up before the final decision to make sure that I am also considering the devops role? I feel like that would project the image that I don’t have a clear goal.

But all in all, would devops be the correct choice for me considering the possibility to fuck up as a junior?

https://redd.it/12elfof
@r_devops

Which cicd tool is the most worthy to invest my time in?

I'm currently a sophomore at a university, hoping to become a devops engineer someday. I have used jenkins and github actions before and have experienced the pain of using jenkins. I can also read a lot of hate for jenkins, despite this it seems to be the most prevalent cicd tool if I'm not mistaken.

As a devops engineer in the near future, what cicd tool is the most worthy to study? Should I go with github actions? maybe jenkins? or perhaps gitlab?

https://redd.it/12ekx2i
@r_devops

Thoughts on Auto GPT?

Auto GPT is this GPT-4 based automation tool that basically executes a series of commands given to it. I haven't tried it but it certainly looks like it could replace a lot of the trouble of setting up CI/CD and Test Automation or possibly skip some tools entirely. What do you think?

https://redd.it/12ensfm
@r_devops

Using Documentation as Code for a deployment guide

I am involved in a project where a significant portion of the deployment is done on the customer's premises. Due to the complexity of the deployment, our deployment guide is quite lengthy consisting of smaller guides from multiple teams. Although we aim to automate the entire process, we currently rely on a deployment guide consisting of about 80 steps, all of which are documented in a Confluence page. However, as we continue to update and modify the guide, the numbering of tasks often changes, leading to broken links and potential confusion.

Also i want to be to create a checklist from the documentation to ensure that every step is completed. Additionally, it would be helpful to create a Gantt chart that could be presented to management.

To achieve this, I suggest storing the documentation in markdown in a Git repository, allowing us to cut releases and reference version numbers and step numbers that remain static. By using headers in the markdown, we could easily create a task list using grep.

Has anyone else implemented a similar approach?

https://redd.it/12epvyc
@r_devops

Can we stop with the "DevOps is not a role!!!!" posts?

Title.

It may have started as a philosophy, but it IS a role now, no matter how much we may disagree with it.

These comments are getting so obnoxious and 99% of the time have nothing to do with the actual thread.

Just saw some guy asking about his career, saying he thinks his skill-set fits DevOps more than Web Development and there were a few "BuT DeVoPs iS nOt a RoLe!!!!" comments that had absolutely nothing to do with the guys question.

Please, just stop.

https://redd.it/12es2dk
@r_devops

What is the minimum price for studying, the AWS EKS?

Hi everybody, I need to fill the hand with EKS, what expenses are expected?

I plan to use Terraform to create and destroy the cluster,

Are there any hidden costs?

For example, 0.10 USD for each cluster (control plane), and which EC2 can I choose from the minimum?

https://redd.it/12esmw3
@r_devops