Sandbox environment for evaluating devops candidates?

We are hiring devops folks and I'm looking for a solution where we can use a lab sandbox where I can ask candidates to build out some simple infra and write some code against it. Does something like this exist?

Edit

I'm looking for a kubernetes sandbox where I can ask them to install a helm chart and write a quick python or shell script to query what the helm chart deployed. Nothing to complex

From previous experience we seem to get people who know how to drive cloud GUIs but not so much read docs, put something together and then write a script to use it. Meaning we want equal emphasis on dev/tooling and operations. Out of the people who say they know Jenkins most have not even installed it themselves, less of those used the containerized version and even less are familiar with deploying it to k8s.

When we evaluate devs we give them a simple take home quiz, which some never even run the code they submit and are surprised that we do. We give them a choice of two languages and some of them write it in both.

I'm looking to give devops candidates a way to do the same with but with simple infra.

I was hoping to give candidates a VM so they could

* Install k3s, a one line command
* Install as service in k3s, I really don't care what
* Write a simple script to interact with the service

Not really seeing any of that as rocket science. Then in the technical review we could dig into the how and why.

Edit

Very interesting the responses here, some helpful, some gatekeeping, some passive aggressive

​

https://redd.it/qqvnhv
@r_devops

Gitlab Runner and Maven - Guide With the efficient cache method.

Speed Up your build process.

https://www.bitslovers.com/gitlab-runner-maven/

https://redd.it/qrl4lj
@r_devops

Logs and tracing: not just for production, local development too

We're a super small start-up, but with prior experience of great observability toolchains. It meant we knew what good looked like, but didn't want to spend ages building or maintaining it.

With that in mind, we've started shipping logs and traces into Cloud Trace + StackDriver and been really surprised at how little effort it required to get a great setup going.

Equally, we've all been surprised by how much it changed our local development experience. Going from an error in dev to the trace view and all the logs inline is just one click, which is pretty transformative.

Happy to answer any questions, hope you enjoy the article!

https://incident.io/blog/tracing

https://redd.it/qrkm6n
@r_devops

new to devops job and lost

Coming from a fullstack dev background and trying to learn all the new modern tools for this company. I've never had to use containers and container services, AWS, CI/CD. Since I worked at a startup I was able to get away with all the tech decisions. I spun up VM's, used cheaper clouds without complexity (hetzner, digital ocean), never used containers and am still having trouble seeing the point to them when I can use VM's and configuration/provision tools like ansible and terraform to have the servers exactly how I want. But this new company does seem to have a lot of different environments (i had just dev and prod) but they have like 6. That might have something to do with it.

They have the dev team and devops team separate and both kind of keep to themselves unless one needs something from the other. The dev team uses rails+react and other tools.

We use:
- AWS (EC2, ECS, Cloudformation)

- Jenkins CI/CD, jenkinsfile pipelines

- Docker containers

- splunk for log


- New Relic (for app monitor I believe)


My issues are that I just don't know how to learn these tools, especially AWS. The guides I'm checking out are outdated. I don't know if AWS recently changed but even a 2 year old video on ECS clusters with docker was too outdated to follow, since too many things had changed in the AWS console.

Could someone give me some clarity... or maybe the resources they used to learn these services?

https://redd.it/qrnivn
@r_devops

The DevOps Master Class - Go Behind The Concept



his free course by John Savil, a true Master, is 12-hours of superb material on everything you have to know in order to incorporate devops into your workflow.

It really is a complete and structured resource covering every aspect of DevOps, code included. As it is structured it is meant to be watched in order, but that's just a recommendation.

Starting with the definition of DevOps, Savill shows that it is not just about the tools:

DevOps is the union of people,
process, and products
to enable continuous delivery
of value to our end users

https://www.i-programmer.info/news/90-tools/15006-the-devops-master-class-go-behind-the-concept.html

https://redd.it/qrs3cm
@r_devops

AWS pushing live data and saving it to process regularly?

Here's the situation I'm in.

I get sent a single JSON object with current stock prices, for 2000 assets, every 10 seconds. The moment it comes in, I want to split it up by asset and send the current price for a specific asset to any client listening via websocket for that asset.

At the start of a new minute, let's say 11:45:00, I want to take all the price data I received for an asset between 11:44:00 and 11:44:59, average it, and permanently save the average (actually saving as candlesticks not averages, but that doesn't matter for this example).

Basically, the client connects and I send them the archived candlestick data for the last 1000 minutes. Then I update the client every 10 seconds as new data comes in.

Candlestick data is just stored in RDS with each asset as a table (no particular reason why, just simple). Client sends XHR, web server queries RDS, values are returned, connection is closed. Simple enough.

To get the live data, the client opens a WebSocket connection to API Gateway. I have an EC2 instance receiving the JSON payload that splits it up by asset and stores it in a DynamoDB table called live_price with the asset id as the key. A Lambda function polls the DynamoDB stream, and when there is an update to an asset a client is subscribed to, it passes that data to API Gateway which passes it on to the client. The great thing here is that there can be thousands of open connections to API Gateway and none of that matters. I just need my one little EC2 instance that receives the payload and updates DynamoDB.

Here is the problem. What's the best way to store the live data and pass it to a process that can form it into candlesticks to save?

Instead of a single table in DynamoDB called live_price with {asset_id:price} I could create 2000 tables each called asset_id storing {timestamp:price}. Then every minute I have a worker thread that connects to DynamoDB, get's all the records for the last minute, turns it into a candlestick, stores it in RDS, and then removes it from DynamoDB.

Or I could have my EC2 instance that receives the JSON payload store the last minute of price data in memory. When a new price comes in, it updates DynamoDB. When the minute ends, it transforms the array of prices into a candle stick, saves it to RDS, and then empties the array.

But that doesn't really work. Because if a client connects at 11:44:30 then I need to send them the RDS candlestick data, as well as the raw price data from the last 30 seconds, and then push the new live price as it comes in.

Is the best way to just have 2000 DynamoDB tables and a worker routine that processes and cleans it up every minute?

Or is my entire design flawed from the start lol?

https://redd.it/qrm3vr
@r_devops

Do you guys wind up doing a fair amount of helpdesk for devs?

Hey all, I was just wondering about how much you have to help the dev teams you support. I have ~2 years of devops experience and one year at my current company. I don't really mind helping out the devs if they have issues but I was wondering where I should draw the line.

I've got a few big projects that I'm working on and am having a bit of trouble balancing helping out the dev team with issues and my actual projects I've been assigned.

https://redd.it/qrni69
@r_devops

CI - help needed

Hi everyone! I am developing a web API for a university project and I would like to implement automatic deployment with bitbucket pipelines. I run my vms and DBs on Proxmox in my lan and my team uses Vpn to connect to.

How can I do it? Use SFTP? Is it safe enough? My only firewall is the ISP router and some rules on ProxMox built in FW.

https://redd.it/qruhfr
@r_devops

How do you handle your containers and environment with your dev teams?

Hey guys,

My team and the associated developers are really interested in moving to containers for our custom apps/services. We've spun up a small Kubernetes POC and started migrating a few things over and its been great. What I am curious about is how to disseminate this change to our dev teams. As of right now, they build their stuff, mostly in .NET core/5 and we push via our CI/CD pipelines between our IIS environments and eventually to prod.

When we switch to containerization I am not sure the direction to give our developers and looking for advice. Do we have them install docker locally, have them build/modify their projects and then have them also dockerize it locally? Or maybe they don't bother with it, let them do their thing and work on our CI/CD pipeline to handle the dockerization component?

Anyways, new to this shift and just curious what you guys are doing, what advice you might have.

Thanks

https://redd.it/qs1n2r
@r_devops

#Kubernetes tutorial for beginners | K8s Monitoring and troubleshooting ...

https://youtu.be/jlsxIx2bzsg

https://redd.it/qs3lo4
@r_devops

Issue with VirtualHost settings?

<VirtualHost 127.0.0.1:80>
ServerName staging02.cherry.com



ServerAlias www.staging02.cherry.com



DocumentRoot /home/staging02/version/www.staging02.cherry.com
ServerAdmin [email protected]

SetEnv environment prod
SetEnv project staging02

UseCanonicalName Off
#CustomLog /var/log/httpd/staging02.cherry.com_log combined
#CustomLog /var/log/httpd/staging02.cherry.com-bytes_log "%{%s}t %I .\n%{%s}t %O ."

## User cherry # Needed for Cpanel::ApacheConf
UserDir disabled
UserDir enabled staging02

#<IfModule mod_suphp.c>
# suPHP_UserGroup staging02 staging02
#</IfModule>

SuexecUserGroup staging02 staging02

<directory "/home/staging02/version">
AddHandler php5-fcgi .php
Action php5-fcgi /php5-fcgi-staging02
AllowOverride All


</directory>
<IfModule concurrent_php.c>
php5_admin_value open_basedir "/home/staging02:/usr/lib/php:/usr/local/lib/php:/tmp"
</IfModule>
<IfModule !concurrent_php.c>
<IfModule mod_php5.c>
php_admin_value open_basedir "/home/staging02:/usr/lib/php:/usr/local/lib/php:/tmp"
</IfModule>
<IfModule sapi_apache2.c>
php_admin_value open_basedir "/home/staging02:/usr/lib/php:/usr/php4/lib/php:/usr/local/lib/php:/usr/local/php4/lib/php:/tmp"
</IfModule>
</IfModule>
<IfModule !mod_disable_suexec.c>
<IfModule !mod_ruid2.c>
SuexecUserGroup staging02 staging02
</IfModule>
</IfModule>
<IfModule mod_ruid2.c>
RMode config
RUidGid staging02 staging02
</IfModule>
<IfModule itk.c>
# For more information on MPM ITK, please read:
# https://mpm-itk.sesse.net/
AssignUserID staging02 staging02
</IfModule>
</VirtualHost>

I replaced this with:

<VirtualHost 127.0.0.1:80>
ServerAdmin root@localhost
DocumentRoot /var/www/html
ServerName staging02.cherry.com
</VirtualHost>

&#x200B;

And I was suddenly able to get the default Apache page instead of 503 varnish error, and I was able to go to [staging02.cherry.com/test.php](https://staging02.cherry.com/test.php). So I am thinking there's something wrong with the above config. What can I remove in that config and what values should I enter? I am thinking ServerAdmin is wrong, I should change the IP address among other things, but I am not sure if it would work. Should I remove all ifModule configs? If it worked with simpler VirtualHost settings can we rule out a problem with php-fpm and fcgi?

https://redd.it/qrwtpr
@r_devops

Career advice

Hello everyone, I'll keep this simple I'm a 26 year old having 2 year of experience in AWS and docker and other cloud technologies(not devops). However I am not working from last 1.5 years for few reasons. In this time I have learnt a few other cloud technologies such as k8s, anisible etc. I want to get back to work and I'm applying to several devops related roles for past 2 months but not getting any responses. I don't hold any certifications as of now. I'm only trying for devops roles as this is the domain I want to get started in. How can I improve my situation and get hired?

https://redd.it/qs8cyx
@r_devops

How to avoid being pigeonholed in the job market?

First, to be clear, I’m not actively looking for another job because I’m pretty happy where I am and would rather stick it out until I’m promoted into a salaried SRE role.

tl;dr/BLUF: I have a little over 3 years of experience doing work that was nominally full-stack but 90% back-end in practice. For several reasons, I have no interest in going back to business application/API development. I’ve made this clear to recruiters who have reached out to me but all I get are LinkedIn messages and emails about full-stack and back-end jobs doing just that. The response I often get is that I could start out doing that but make it clear in the interview that I’d eventually want to move into a more devops-related role. But I know that promises are cheap and easy to initially make and that it can be a battle to actually hold a manager to them and if I lost that battle, I’d have effectively locked myself further into a career path I have no interest in pursuing.

The wall of text, for context:

For the past few years, I’ve been in a program for career switchers that gives people actual experience more or less as SREs, but with varying degrees of autonomy. In my day-to-day, I’m fairly autonomous; if I hit a stumbling block on a task, I know how to find the information needed to unblock myself or ask an informed question when the answer can’t be found in internal documentation or with a Google search.

Prior to this job, I had a contract role that was similar but with more responsibility in that my day-to-day duties were basically writing and maintaining code that processed data from that company’s infrastructure. Just prior to my contract ending, I was offered a full-time salaried job there, but declined it to take this job and because the environment there was pretty toxic (think things like engineers and managers sabotaging other engineers and teams for petty reasons). I mention this to provide context about my experience.

I came into my current job with a little more experience than a lot of people who typically get accepted into this program, but my actual job title is extremely entry-level (basically intern++). The reason I even applied for it is that I was having the same problem: I didn’t have enough experience outside of hobby projects for recruiters or hiring managers to be willing to take a chance on me.

The reason it’s been so long and I still haven’t gotten promoted was largely out of my control (think something like having had to take a lot of time off to take care of a sick family member). That has been resolved for a few months and I’ve been able to fully concentrate on work instead of juggling that and shitty life circumstances. I’m deliberately being vague because specificity would out me to anyone at my current job who reads this sub regardless of whether I chose to use a throwaway account, which is why I didn’t bother with that.

While I’m working towards being promoted into a real SRE role, the earliest that could possibly happen is the second half of next year (fuzzing the timeline, but not by much) due to the way promotions are done. In the meantime, I don’t want to have that as my only option and even if I did, it would still be nice to have another offer in hand as additional leverage when it’s time to negotiate salary and equity.

Short of just taking the full-stack roles off of my resume and LinkedIn and effectively making it seem as though I have half my actual work experience and a weird multi-year gap in employment (not to mention being deceptive), how can I start being taken more seriously for the jobs that I want rather than the jobs I’ve had? I have applied for the sort of job I’d rather have, but I think because I don’t have a CS degree, my resume automatically gets screened out by the ATSes at those companies. I have a liberal arts BA but also a fair amount of work experience from having worked as a low-voltage tech for a couple of years and then as a technician at an electronics refurbishment startup before self-teaching for a year and then doing

a coding bootcamp to get into web dev.

I understand it from the POV of hiring managers and recruiters; it’s much easier to hire someone for the skills for which they have an established track record. I don’t blame anyone else for my predicament, but I just want to break out of the cycle even if it means staying longer in a job I took a considerable pay cut to take.

https://redd.it/qsb4wb
@r_devops

Ideas for a side project / blog tutorial showcasing modern SRE/DevOps skill?

Hey all,

I am an SRE at a big company, but I'm narrowly focused on some stuff that isn't necessarily directly relevant to the greater market for SRE/DevOps.

I'd like to write a blog/tutorial series showcasing how to use modern devops practices, CI/CD, infrastructure as code, and some "bleeding edge" cloud services. This is mostly to sharpen up my own skills, perhaps benefit others, and something to show potential employers in the future so I can stay relevant.

I'm less concerned with what the application actually does at this point (a simple "voting" app, or perhaps a mastodon social network or even a wordpress blog), this is more about the tooling and showing how to deploy a multi-region, highly available and fault-tolerant, cloud-native system.

&#x200B;

I'm fairly certain I want to include the following technologies:

1. Dockerized app(s) - running in AWS in multiple regions
2. Probably with AKS, perhaps ECS or "fargate"
3. Infrastructure completely described with Terraform
4. Amazon Aurora/RDS backend
5. Content (images etc) served by CloudFront
6. CI/CD - infrastructure and app to be deployed via Github or perhaps AWS CodeDeploy CI/CD
7. Monitoring/Alerts (Datadog or similar free trial/tier?)

&#x200B;

Looking for suggestions on what you all think are the most "relevant to the industry" services and technologies, preferably which have a reasonable "free tier" or pay-as-you-go so I can minimize the cost of this project.

"Buzzword bingo" is a bit of what I am going for, I want something to show in github (I can't share anything I do at my actual job).

&#x200B;

Thanks in advance for any ideas. What would you include if you were (or have) done something like this? What are the must-haves for experience to be hired as a DevOps/SRE on your team?

https://redd.it/qsg5eq
@r_devops

How prepare for DevOps Role?

I’m a recent graduate who got their first programming job as a Junior DevOps Engineer. Prior to this role the only experience I have with DevOps is with the one course I took in college, which I enjoyed a lot and briefly touched on the technologies that I will be using in this role as well, but that was 2 years ago and I haven’t done much with it since then. I want to be best prepared for this new role as a Junior DevOps Engineer and am wondering what resources I could use to be best prepare myself as I start on December 6th.

TDLR; Starting DevOps role with no experience, how should I prepare for this role?

https://redd.it/qshc6u
@r_devops

How would you build an DevOps person

I’m interested to know what skills everyone thinks the perfect devops engieer would have.

If you had a blank slate of a person with no IT experience. what skills would you prioritise them learning, what skills do you think a lot of engineers are currently missing or could be better at.

I understand each company/role is going to be different I’m just looking for what a good foundation would look like.

https://redd.it/qseq6j
@r_devops

Jenkins server crashes when trying to build a React app.

Hey guys, this is probably a very entry-level question, but I couldn't find an answer on Mr. Google. I'm trying to set up a simple CI pipeline for the front-end section of a client. It's a React app with 60k lines of code. Initially my pipeline was running npm run build on every merge request but this caused the server holding Jenkins (4GB of RAM) to ran out of RAM, ending the npm process. Now I'm running node --max_old_space_size=3096 node_modules/.bin/react-scripts build which seemd to work fine, but now the server is running out of RAM and sometimes even crashing the Jenkins instance.
I feel like I'm missing something, it doesn't seem like the app is that big. Should I double the RAM of the Jenkins instance?

https://redd.it/qsl7jy
@r_devops

In what stage to implement CIS

I have a requirement from infosec to implement CIS benchmarks. Would you put them in the packer stage so that the benchmarks are in the image or would you deploy the benchmarks later via config management?

I feel like the benchmarks are pretty static and all deployments will get it so they should be baked in the image.

https://redd.it/qspenk
@r_devops

#Kubernetes tutorial for beginners | Horizontal Pod Autoscaling in Kuber...

https://www.youtube.com/watch?v=3wITELXQGqE

Kubernetes tutorial for beginners | Horizontal Pod Scaling in Kubernetes HPA’s are stable resources in the autoscaling/v1 API group and their job is to scale the number of replicas in a Deployment based on observed CPU metrics. The autoscaling/v2 API allow scaling based on more than just CPU. You define a Deployment that makes use of Pod resource requests – where each container in the Pod requests an amount of CPU. You deploy this to the cluster. You also create an HPA object that targets that Deployment and has a rule that says something like: if any Pod in this Deployment uses more than 60% of its requested CPU, spin up an additional Pod. Once the Deployment and HPA are deployed to the cluster, scaling operations become automatic. One thing worth noting is that HPAs update the .spec.replicas field of the targeted Deployment. While this update is recorded against the Deployment object in the cluster store, it can lead to situations where the copy of the Deployment YAML file in your external version control system gets out of sync with what is currently observed on the cluster.

https://redd.it/qst81m
@r_devops

Podman vs Docker?

I keep hearing more about podman and I keep hearing worse things about docker (rg them charging for basic features)

What benefit does podman provide over docker? Does it have to do with it being purely open source? Is anyone using it in production?

Thanks!

https://redd.it/qstn3p
@r_devops