What do most people do for environment deployments with Git?

I had a similar question before, but now I just want to see what everyone else does either at their companies or personally. Currently using dev and prod branches introduces merge conflicts as the commits get extremely messy after a few months.

How do you separate dev, pre-prod (my company calls it go), and prod in their repositories for deployments?

I want to find a method that's just smooth and almost automatic when someone updates the helm chart.

https://redd.it/13wdtjr
@r_devops

How do I become a DevOps engineer?

Currently, I am a Quality Engineer with a total experience of around 1.5 yr out of which for 1 year. I have been on the bench not doing anything.

https://redd.it/13w1rna
@r_devops

'ekscli' vs. 'aws eks'

I see on https://docs.aws.amazon.com/eks/latest/userguide/getting-started.html you can wither use the GUI, ekscli, or aws cli to manage your cluster and interactions.

ekscli looks neat, but I imagine I will also need to use the normal aws eks style due to other aws command line options (e.g. aws sts).

What tool would you recommend getting framliar with and why?

https://redd.it/13vs5hd
@r_devops

How to prepare for DevOps Engineer Technical Interview and scenario based questions?

I am having 2 YOE (Currently on a Career Gap). I am currently looking to get into DevOps. Started learning AWS, Docker Kubernetes, Shell Scripting but the technical interview seems to be more overwhelming and focused on troubleshooting the scenarios? How to effectively prepare for those ? What are the tools that one must know before entering into DevOps scene.

https://redd.it/13vsi3v
@r_devops

What are some opinions and experiences when choosing between Elasticsearch and Loki?

Title says it all. Looks like Loki is a little better on resources, but curious to others' experiences with ES or Loki, choosing one or the other, for storing application and system logs.

https://redd.it/13wjs0p
@r_devops

Prevent access to .env on a shared VM (Guacamole)

I possess a VM that is shared among multiple users, and we all use the same Guacamole account with a shared username and password. My objective is to install a Node JS application on the server while ensuring that other users cannot access the .env variable. One potential solution could involve encrypting the .env variable to secure its contents.


Can this be done by Containerization? I believe the root user can access the docker secret variables

https://redd.it/13wka9v
@r_devops

What are my (AWS) options for running one single container reliably?

My team has just me doing "deployment stuff" and we're a tiny startup. I have one microservice/container with code that won't go into our monolith. I need to find somewhere to run it in production (on internal VPC's 10.10 network).

The ones I can think of:
\- ssh & docker run with restart/service options
\- nomad "cluster" -- seems heavy to me for one container?
\- lambda -- my "microservice" is a nodejs script that listens to port80, but probably convertable to lambda runtime interface

Things I don't know anything about (yet):
\- kubernetes -- also maybe too heavy for one container?
\- rancher? -- What is this? can it help me?

https://redd.it/13woti9
@r_devops

Cloud Dependencies Need to Stop F—ing Us When They Go Down

https://thenewstack.io/cloud-dependencies-need-to-stop-f-ing-us-when-they-go-down/

https://redd.it/13wqex0
@r_devops

Thoughts about my thoughts on performance / alerting metrics?

Curious for some feedback from folks, I was laid off a few weeks back and find myself dealing with the whole process of recruitment.

Something I consider a significant success that I like to talk about in interviews was writing a whole slew of "synthetic user transactions" as a method of gauging platform health.

That involved working with the app developers to plumb transaction time values for things like login, password reset, interaction X or Y as part of the API call responses.I came about with about 15 of them which fired every few minutes (or in some cases more regular) as a much more effective way to gauge how the entire system was working as opposed to "CPU High 90% - call Pagerduty".

I was (I guess I still am) quite proud of that as a monitoring solution. They were just AWS lambda functions performing tasks and checking the response times - but in that particular case it made it so much easier to identify what was bottle-necking and or "crapped out".

Thing is, I've had some fairly frosty reception to that when explaining it to recruiters / hiring managers and I'm wondering if I've missed something?

My take has long been, if you're paying for an instance of <insert service here> it's fine for it to hit it's peak CPU / memory / cache / whatever, you're paying for that - I'd care more about if the user experience is suddenly gone terrible?

I appreciate this is more of an SRE question.

https://redd.it/13ws7qv
@r_devops

how to package on-prem solution?

I have a SaaS (AWS stack) and want to package it for on-prem offering.
I don't have experience with "enterprise software" but from what I've seen most projects charge an annual license + annual support built in.

If I go and replicate the AWS stack on their environment (using client user/pass and then they change it), I'm thinking that:
a. I'm giving away all the source code.
b. How do I ensure that they don't resell the solution? This would probably need a contract but I don't know what kind and I worry it will complicate things.
c. If I give away all the infrastructure and code, how can I get paid for the second year?
d. I could also charge based on their users, but how can I monitor this?


I would appreciate if anyone is aware of a structure/packaging that makes sense for this, or any thoughts on this matter.

https://redd.it/13wuj9a
@r_devops

Architecture of OTT Platform

Totally noob in devops.

I was wondering if there's any way to understand how a typical flow or architecture works for a OTT platform.

https://redd.it/13ww6oc
@r_devops

Looking for advice on releasing my app while still incomplete

Hey everyone! I’m looking for advice on the most cost-effective way to release my app while there are still lots of things incomplete. Should I use a single server instance (AWS, GCP, Digital Ocean…) with all databases (mongo, postgres, redis), and codebases (node, golang, react, nextjs) of 3 different apps installed in ONE server; or use different cloud services for Database, Docker, Kubernetes etc.?

I need to release my MVP to show early users and get feedback. But it consists of 3 different loosely connected apps and I’m sure there are obvious insecurities. I would have to disable some authorization features in my MVP for the 3 apps to send API calls to each other, including from the client. I need to do so because I need users to test-use it and get their feedback to fix and improve ASAP (to get traction and raise investment). If I put them in a single server, I’m afraid a hacker can easily get access to root, users' data and confidential codebase (my startup is still in stealth mode).

Also, I need a server that is much more powerful than the free-tier server provided by AWS or GCP. I have a few thousand AWS credits but are not able to use them yet. What is the most cost-effective way to deploy my MVP? Any advice would be greatly appreciated!

What would you do if you were me?

PS: I have little experience with DevOps. Mainly backend > AI/ML > frontend. Therefore the questions.

https://redd.it/13wvg70
@r_devops

Any good write-ups on implementing IaC in your environments?

Any good place to read up for my write-up? :-)

https://redd.it/13wt09c
@r_devops

Strategies for converting an existing deployment to IaC?

Like many of you, I have my aws sandbox and dev fully automated with terraform. The environment configuration even roughly matches production. However, prod was built starting back in 2012 and is a delight of manual console changes.

Leadership is convinced the only approach is to build an entirely new production environment (six months or more of work) and eventually cut over and burn the old one to the ground.

I don't buy it. There must be a reasonable way to break down and do a service by service terraform import with some tweaking.

https://redd.it/13wr3fp
@r_devops

Ugh I fucked up and need some love

Usually I don't beat myself up but today I fucked up and just feel like an absolute idiot

Created new subnets for our AKS cluster
More IPs!
Rolled out new node pools very easy all looking good

Restarted each deployment and everything moved over nicely and I finished my day....

Thought I'd check slack on my phone to see if there was anything going on and yep .. incident call the region I was working in can't connect to 2 things....

I forgot that I needed to update these two things with the new subnets...

I finally get on the call as they are resolving and wrapping up

No one is pissed off, my manager said "fortune favours the brave, sometimes we get it right alot of the time we get it wrong, we just learn"

And he's right but I just can't stop beating myself up over such a stupid mistake...

As soon as I saw the incident room was busy I knew exactly what I forgot...

Time to find some long rizzla and forget today...

How was your Wednesday..

https://redd.it/13ww4cf
@r_devops

Engineering Methods Based on Fictional Characters

https://dathan.github.io/blog/posts/engineering-methods-based-on-fictional-characters/

A post on relating to other engineerings on how to estimate work to how to approach fixing problems. What methods do you use to communicate to others on approaching problems and working with engineers?

https://redd.it/13x8d3e
@r_devops

Opensource CD recommendations

We're in aws. I'm building a new pipeline in an in-house gitalb. I'm using Terraform to build and deploy an application on an ec2 instance via launch template and ASG. It also deploys a load balancer. It's working well now but I noticed I encountered difficulties when setting up rollback or standby. It doesn't work yet. I don't have any standby. I was given few solutions today such as ASG instance refresh. I have not played with it yet. Anyways, I was told that Terraform is not the right tool for what I am looking for. What would be your recommended deployment tools?

https://redd.it/13x9a13
@r_devops

JSON Schema as Source of Truth

I'm developing an app where the configuration data at runtime is best represented as a single JSON data structure.

I can ask users to manage their configuration using JSON or I could use something else and then convert.

Yaml is an obvious choice but I am wondering if I could also support an object/form-based user interface.

The thought is to generate a relational database from the JSON structure and then generate the forms etc.

I would treat the JSON schema as my data definition source of truth and generate as much as I could from that.

Has anyone done this? Does it seem doable? Are there exisitng tools to help or should I be doing it a different way?

The dream solution is define JSON schema and get everything else for free :)

Thanks

https://redd.it/13xbcr6
@r_devops

DevOps course for small companies and individuals

Hello everyone,

I've made a DevOps course covering a lot of different technologies and applications, aimed at startups, small companies and individuals who want to self-host their infrastructure.
To get this out of the way - this course doesn't cover Kubernetes or similar - I'm of the opinion that for startups, small companies, and especially individuals, you probably don't need Kubernetes. Unless you have a whole DevOps team, it usually brings more problems than benefits, and unnecessary infrastructure bills buried a lot of startups before they got anywhere.

As for prerequisites, you can't be a complete beginner in the world of computers. If you've never even heard of Docker, if you don't know at least something about DNS, or if you don't have any experience with Linux, this course is probably not for you. That being said, I do explain the basics too, but probably not in enough detail for a complete beginner.

Here's a 100% OFF coupon if you want to check it out:

https://www.udemy.com/course/real-world-devops-project-from-start-to-finish/?couponCode=FREEDEVOPS2306JEOZX

Be sure to BUY the course for $0, and not sign up for Udemy's subscription plan. The Subscription plan is selected by default, but you want the BUY checkbox. If you see a price other than $0, chances are that all coupons have been used already.
You can try manually entering the coupon code because Udemy sometimes messes with the link.

The accompanying files for the course are at https://github.com/predmijat/realworlddevopscourse

I encourage you to watch "free preview" videos to get the sense of what will be covered, but here's the gist:

The goal of the course is to create an easily deployable and reproducible server which will have "everything" a startup or a small company will need - VPN, mail, Git, CI/CD, messaging, hosting websites and services, sharing files, calendar, etc. It can also be useful to individuals who want to self-host all of those - I ditched Google 99.9% and other than that being a good feeling, I'm not worried that some AI bug will lock my account with no one to talk to about resolving the issue.

Considering that it covers a wide variety of topics, it doesn't go in depth in any of those. Think of it as going down a highway towards the end destination, but on the way there I show you all the junctions where I think it's useful to do more research on the subject.

We'll deploy services inside Docker and LXC (Linux Containers). Those will include a mail server (iRedMail), Zulip (Slack and Microsoft Teams alternative), GitLab (with GitLab Runner and CI/CD), Nextcloud (file sharing, calendar, contacts, etc.), checkmk (monitoring solution), Pi-hole (ad blocking on DNS level), Traefik with Docker and file providers (a single HTTP/S entry point with automatic routing and TLS certificates).

We'll set up WireGuard, a modern and fast VPN solution for secure access to VPS' internal network, and I'll also show you how to get a wildcard TLS certificate with certbot and DNS provider.

To wrap it all up, we'll write a simple Python application that will compare a list of the desired backups with the list of finished backups, and send a result to a Zulip stream. We'll write the application, do a 'git push' to GitLab which will trigger a CI/CD pipeline that will build a Docker image, push it to a private registry, and then, with the help of the GitLab runner, run it on the VPS and post a result to a Zulip stream with a webhook.

When done, you'll be equipped to add additional services suited for your needs.

If this doesn't appeal to you, please leave the coupon for the next guy :)

I hope that you'll find it useful!


Happy learning,
Predrag

https://redd.it/13xckxm
@r_devops

Thread starvation or clock leap detected in ECS with EC2 & Java Application

Hi,We use ECS with EC2, we have two ECS clusters one in which there are multiple micro- services in each cluster one all Java(Springboot) and one all python(Django). One of the Java Microservice started misbehaving, although I cannot see any disturbance wth metrices of ECS(CPU, Memory , IO), even healthy counts are there but it is almost that service goes into deep sleep, Logs mising for some time, No proccesing is done at that point of time. Following error is the only thing that i could find, anything anyone have found earlier?
[31mWARN [0;39m [[34mHikariPool-1 housekeeper[0;39m\] [33mcom.zaxxer.hikari.pool.HikariPool$HouseKeeper[0;39m: HikariPool-1 - Thread starvation or clock leap detected (housekeeper delta=1m35s328ms69µs44ns).

https://redd.it/13xe1t4
@r_devops

Is there a tool to visualize related stories, tasks, bugs, etc?

I am not too concerned about child tasks although that would be nice to have, but our tickets are full of related items and it's a muddle.

Right now, I am trying to figure out what can be closed, but there are a lot of interdependencies.

I will see that things are done better in future, but right now, what would really help would be a diagram, with boxes for tickets and lines between them (and status would be wonderful, preferably colo(u)r coded). Is there such a thing?

https://redd.it/13xeijf
@r_devops