Career Advice

Am I being unrealistic looking for a 9-5 job non-scrum DevOps job without bleeding over into the weekends? I can only work at one speed and it’s slow and I’m wondering if I should leave DevOps altogether or try looking for a similar org as my last employer?
I’ve been working as a DevOps engineer for the past 5 years. Before working in DevOps I worked in various other IT positions for 10 years. I recently left an org ( 5 years ) as a DevOps engineer keeping a reasonable 8-5 without on-call and TC 110k. The reason for leaving was due to being acquired by a large electronics company and layoffs were approaching. My new job is terrible, 30% of my time is in scrum meetings and lots of bleed over into the weekends. It seems like my slowness with the current job is going to cause me significant issues. FWIW I did question my current employer about time expectations, scrum, and meetings during the interview process and received different answers to what is actually current with the company. TC to me isn't an issue if the position fits what I'm looking for. Thanks for listening.

https://redd.it/swbzwi
@r_devops

Using Prometheus metrics to trigger autoscaling in aws.

My company want us to stop using cloud watch as a part of cost cutting measures, but we are using cloud watch to trigger autoscaling in aws for certain scenarios, is there a way to achieve the same using Prometheus to trigger autoscaling when the cpu or memory usage goes high.

https://redd.it/sundmu
@r_devops

Has any one of you tried crypto mining on any public cloud provider ?

I haven't tried it myself, but nothing is stopping me from doing it.

Just wanted to hear about others experiences.

As ops people we can optimize the costs significantly imo. I'm not sure if it is allowed, but if you can run preemptive VMs then the cost reduces significantly.

And maybe Bitcoin or Ethereum mining can be difficult, but how about other coins. Take doge for example. Just for fun.

https://redd.it/swvtxd
@r_devops

adding Azure to my world of linux and open source systems?

I recently began my journey into Azure. My corporate job is switching to to Azure and I will need to look into it. I actually thinks Azure is great. Unfortunately it is too expensive as my personal playground, but I will be using the free stuff and keep my cheap VPS setup :-)

I installed powershell on my Linux laptop and played around with some webapps programmed in python and .net core? but is it worth it?

To me it looks like a lot of extra googling in order to make e.g. Python run i a webapp with a decent devops setup running github actions compared to using .net ( I believe it was done in a single mouse click, which fascinated me... Honestly I think I did spend around 2 days on making github actions work with my VPS the first time)

But I think i'm beginning to see a pattern here? it works with open source but things are so much easier if I purely stick to Microsoft technologies in Azure - share your thought with me?

https://redd.it/swxyx8
@r_devops

Container monitoring for microscale

I have a dozen containers on a VM. Some of them are always on, some are activated through a cronjob and live only a few minutes.

So far, monitoring is done like so:

              #!/bin/bash
              docker run \
              --rm \
              --log-driver=none \
              --volume={{ script_name }}:/app_data \
              --name={{ script_name }} \
              {{ script_name }} 2>&1 | /usr/bin/logger -t {{ script_name }}

so all things related to a container can be retrieved with journalctl -t {{ scriptname }}.

This works well, but I would like to:
1) get monitoring info from different containers to a single location
2) access that location from browser

So that means some sort of monitoring solution is required, yet one that doesn't add a lot of complexity, as my requirements are quite simple. I don't need auto-discovery, as will add container name statically, but need to see stuff by date, i.e. output equivalent to journalctl -t {{ scriptname }} --since X --until Y.

Could filebeat and elasticsearch offer what I'm looking for?

Has to be free and self hosted.

https://redd.it/sx11oo
@r_devops

Packer and aws AMI - how to get built ami id properly ?

When im building ami image at the very end of building log i get something like this:

...
==> Builds finished. The artifacts of successful builds are:
--> learn-packer.amazon-ebs.ubuntu: AMIs were created:
us-west-2: ami-0703a21445140541e

Offcourse i could get this id by some kind of log parsing - some regex, exploding or something similar. But Isnt there a 'cleaner' solution? Some kind of parameter or something? I was looking in the documentation and examples but i didnt found any.

https://redd.it/sx0i7r
@r_devops

Beginner in DevOps - how do I manage microservices (via Docker)?

I'm more of a programmer, but obviously to implement my code, I have to delve into DevOps.

Say I have 2 microservices, auth service & user service. And a MongoDB database in addition. I can put all these into a docker-compose file, which would be the easiest, wouldn't it? But if I want to replace the image with another, newer version, I have to do docker-compose down and everything goes down. It's my naive look on this, but I would like to get some help regarding this, the DevOps side. Thanks! ;-)

https://redd.it/sx3jak
@r_devops

Gitpod automates the provisioning of ready-to-code development environments.

I thought the r/devops subreddit might be interested in this project I just found!

https://github.com/gitpod-io/gitpod

https://redd.it/sv079l
@r_devops

Testing Github workflows

Hi there, uni student here learning/working on devops practices/engineering. I've recently started putting more time into writing github actions/workflows for CI/CD reasons. A bit of an issue I come across is the fact that I'll be doing a million commits to a github repository so I can test the action/workflow, which doesn't feel like a very handy process. Besides the solution of running github actions locally using nekos/act, are there any tips, solutions or advices you have for testing github workflows? Thanks!

https://redd.it/sx52lq
@r_devops

non deterministic docker container availability

Hi all,

I am running an app on a docker container that runs a specific application that i communicate with through port 10000. The image I am running on has been stable and unchanged for over a year, and it runs on multiple vms which are configured in the same way.

However, I get some unpredicted behaviour. Running from jenkins, on the same hosts, with the same configurations and the same image results in the container up and running in 30-50% of the times, with no constant pattern. There are no other applications or containers on the VMS.

Any thoughts on how to approach this ?

This is the run code but I think it's not so important since it doesn't behave the same

​

set -x

sudo systemctl restart docker

sudo docker network prune --force

sudo docker container stop $(sudo docker container ls -aq)

sudo docker container rm $(sudo docker container ls -aq)

​

port=10000

​

sudo docker run --hostname=quickstart.cloudera --name ${dockerName} --privileged=true -t -i -d -p 8888:8888 -p 8050:8050 -p 8051:8051 -p $port:10000 -p 25000:25000 -p 25010:25010 -p 25020:25020 -p 9870:9870 -p 8088:8088 -p 19888:19888 -p 14000:14000 -p 3306:3306 -p 8042:8042 -p 7180:7180 spark24_1 /tmp/cdh_startup.sh

​

sleep 240s

nc -zv $hostName $port -w 5

https://redd.it/sx9ieq
@r_devops

Managing pre-commits

I'm a primarily solo developer who has been working on improving my environment through devops principles. One of the setup tasks that I tire of is configuring pre-commit whenever I create a new repository. Since I hop languages and frameworks, it's not feasible to create a single copy and share it around my repo's.

Any ideas on how to better manage this aspect? I'm tempted to just make my own management CLI but wanted to hear of some other ideas before going down that road.

https://redd.it/sxeg9v
@r_devops

Job Hopping

Hi Guys,

A friend of mine joined a company just 4 months ago as a DevOps Engineer. Now he has got an offer from somewhere else for a 150% increase in salary. He is pretty confused since this is the second job he is leaving in less then 7 months and he keeps asking me for advice (the company where he is working now also offered him a 100% raise from his previous salary). I told him to stay where he is since he runs the risk of being labelled as a job hopper by potential employees but he is double minded since he thinks might not receive this much raise again as a new entry in the field. What would you guys recommend ?

P.S He is also worried about the company culture since where he is currently working right now is possibly the best culture according to him (not learning wise but the managers are very lenient and supportive and the company has very good reviews on glassdoor) and from where he has got the offer has average reviews on glassdoor (some good some bad).

https://redd.it/sx4hax
@r_devops

A little tool to help manage Elasticsearch/Opensearch clusters.

This tool helps to export/import data, move indices, perform backups and schedule these tasks. It is a docker image with a browser-based scheduler, code editor, and terminal.

https://github.com/bluxmit/alnoda-workspaces/blob/main/workspaces/elasticsearch-workspace/README.md

https://redd.it/sxovqn
@r_devops

Question on CI/CD with Multiple Environments & Corresponding Branches

The flow I want to follow:

PR from Feature to Dev -> Build & Test -> Merge to Dev -> Docker Build, Tag using GitHub Short SHA, Push Container to Registry, Deploy Container to Dev Env

PR from Dev to Main -> Merge to Main -> Docker Pull & Deploy Container to Prod Env

How do I pull the corresponding Docker Image From The Registry for the code I am currently merging to Main? Do I have to build again on the merge to main everytime?

The GitHub SHA is different in the two workflows because of the merge commit that happens. And cannot always pull the latest I believe because the Dev lifecycle could be faster than Prod.

I'm open to alternative flows that allows keeping code for multiple envs in their respective branches as well.

https://redd.it/sxpkni
@r_devops

DevOps Bulletin Newsletter - Issue 39

Hey folks,

DevOps Bulletin - Digest #39 is out, the following topics are covered:

* 🚀 "A ‘Hello World’ GitOps example" - This post walks through a ‘hello world’ GitOps example using Docker, K8s, GitHub Actions and Terraform.
* 💰 Why sometimes you should press the $100k button: With S3's unlimited object storage you can easily end up with a shitload of data and thousand dollar bill.
* ⭐️ hands-on video tutorial introduces ArgoCD, how it works, why you might need it and how to deploy a hello-world application to Kubernetes with it. All of that in less than 20 minutes
* 🚨 "Startup guide to incident management" - If you’re working at an early stage startup and looking to get some good incident management foundations in place without investing excessive time and effort, this guide is quite literally for you
* ☁️ Introducing a Google Cloud architecture diagramming tool; this interface provides a list of all GCP products and services in one spot. No need to worry about finding the right icon or uploading an image from somewhere else – just focus on building the architecture with the components you need right there in the interface.

Complete issue: [https://www.devopsbulletin.com/issues/s3-storage-bill-of-almost-100k-month](https://www.devopsbulletin.com/issues/s3-storage-bill-of-almost-100k-month)

Feedback is welcome :)

https://redd.it/sxs4b7
@r_devops

Youtube in vid ad skip;

A chrome extension for youtube that tracks ⠀other users skip behaviour and uses that data to skip at video ads such as skillshare, brilliant, world of war, raid shadow legends, ect Is this a good idea?

https://redd.it/sxthfe
@r_devops

Local Development with Kubernetes Service Accounts

I'm a devops engineer and i'm trying to convert an application from using an AWS service account w/ key and secret for authentication, over to using a k8 service account to assume an AWS Role.

The challenge im facing is that the app developers use IntelliJ on their local machines to test their code, which requires hitting AWS resources. This means that they have possession of the key and secret for the service account (in our dev environment at least) on their local machines. If their IAM user is terminated, they still have possession of those keys and security is not ok with that. We don't have a good mechanism for key rotation, nor is there a plan for one.

...Hence using kubernetes service accounts to assume an AWS role and grant the application the AWS permissions that it needs to function. Since we are using EKS and the containers already assume the cluster role by default, this has been super easy to implement. But it totally breaks the ability for devs to run the applications locally in intelliJ.

I'd love to set them up with the ability to run the application locally using docker desktop's kubernetes environment or something like that. But then they have to build the app and then deploy it locally, which is far slower and less streamlined than intelliJ. They are used to being able to run the application without even having to build it.

I'd love any and all suggestions as I am totally out of ideas.

https://redd.it/sxtjv4
@r_devops

Is there an easier way to SSH to ECS containers?

When I want to SSH to ECS I use

aws ecs execute-command --cluster <Cluster> \
--task <taskId> \
--container <ContainerName> \
--interactive \
--command "/bin/sh"

&#x200B;

But the ECS container gets rebuilt on every pull request so I have to keep going to AWS and get the new task Id.

Is there any other more convenient way to SSH?

https://redd.it/sxw5gj
@r_devops

What is Developer Experience? a roundup of links and goodness

https://redmonk.com/jgovernor/2022/02/21/what-is-developer-experience-a-roundup-of-links-and-goodness/

https://redd.it/sxwm41
@r_devops

question about internet speeds...

So, I'm looking at moving to a new house. The place is awesome but it's also pretty rural, and it seems difficult to find internet plans that are more than 50mbps. I'm used to having a much faster connection than that, like at least 300.

I need to know if I can do my job from home before I commit to purchasing a property... y'all think 50mbps would cut it?

https://redd.it/sxzd6h
@r_devops

Does anyone here use oath2 proxy in front of Atlantis?

I'm trying to figure out how to insert https://github.com/bitly/oauth2_proxy in front of https://github.com/runatlantis/atlantis via terraform but have several questions. First, is there any publicly existing terraform repo that can set this up automagically on Kubernetes? My google-fu is coming up short. Second, are there are any automagic scripts that can generate the config values for you via github api? The end goal really is to just figure out how to get this as automated as possible so I can add auth to atlantis. Any additional suggestions appreciated!

https://redd.it/sy0o6c
@r_devops