ssh into vm's

Hi everyone my organization uses gcp and I need to ssh into vm's frequently. The problem is that there are so many of them and I need to copy ip every time.

I have made some bash functions for few of those to ease myself but I can't do that for all of them.

Is there any solution for this problem ?

https://redd.it/zeuqqk
@r_devops

How is feature deployment conflict handled in general?

Just curious, what if a company has 5 features to release at same night - and they could all conflict each other, who organize and foresees that, what methods are available?

https://redd.it/zeoxjm
@r_devops

Devops with A cots application

Anyone have experience in this realm? We've built up infrastructure-as-code and pipeline around a COTS app (JBoss/Java stack) but we struggle with silos of communication between us and the outside development team. It's a pretty niche application and we could have a better relationship with the vendor

https://redd.it/zeh43y
@r_devops

If you were the single AWS/DevOps engineer for a startup, how complex would you go?

So, joined a startup, great team, great potential, got two app dev dudes who are churning out some absolutely beautiful UI and functionality. The code needs some place to run, and we're going with AWS. I've been hired as the only AWS guy to help sort this out, and we need to start with AWS account / AWS Organization creation and build up from nothing.

How complex would you go in this situation? An argument could be made for, "just dump it in Heroku and go" for our team size, which is fair, but for various long-winded reasons, AWS is our ideal choice.

At first I was like, "alright let's make a Tooling Account for our pipelines," "let's get a Terraform Cloud account," and then the reality hit that: this is all on me to manage. When I started mapping out an OU structure for AWS Organizations like I would for a cloud environment supported by a "cloud team," but then I started wondering, "maybe some approach that is 'less perfect', but still 'good enough' is better..."

I need to find a balance of: "one dude can get this all up and running without going insane" and still "easy to manage/navigate."

https://redd.it/zeysbq
@r_devops

Wrote a blog on automating building Golden AMIs to make best use of developer's time

Modern cloud deployment success depends on immutable infrastructure and Packer an open source tool is here to help us in achieving the immutable infrastructure by creating golden images from a single source of configuration.


Packer helps in the faster deployment cycle, supports multi-cloud providers like AWS, GCP, Azure, etc., and provides scalability. I wrote a blog on how to automate building Golden AMIs with Packer, Ansible & CodeBuild. Do give it a read and share your feedback.


Disclaimer: It is posted on my company's website.

https://redd.it/zeyysd
@r_devops

I made a curated DevOps Engineer job board

Here it is: **https://www.devopsjobs.xyz/**

In my own search, I noticed there are no high quality job boards for DevOps engineers. There are plenty of targeted boards for frontend/full stack/backend engineers, but it's pretty noisy for DevOps.

I'd love to get some feedback. Ideally it could grow into a repository of curated DevOps roles with top tier companies and it would be possible to search by particular skills.

I'm also open to suggestions for companies/roles you think should be on this list :)

https://redd.it/zf0lfe
@r_devops

Security releases: Prometheus 2.40.4 and 2.37.4 are out

Security releases: Prometheus 2.40.4 and 2.37.4 are out.

Those releases fix a bug that would enable an attacker with access to the bcrypted passwords to bypass Prometheus' built-in basic auth mechanism.

Advisory: https://github.com/prometheus/prometheus/security/advisories/GHSA-4v48-4q5m-8vx4
Announce: https://groups.google.com/g/prometheus-announce/c/qVv\_BpMJdnM
OSS-Security: https://www.openwall.com/lists/oss-security/2022/11/29/1

https://redd.it/zf3lrl
@r_devops

Spacelift - opnions?

Have you used space-lift and what is your opinion of it in an enterprise production environment?

We are considering doing a POC not yet sure if it's worth it.

Any challenges, feature gaps, or pros that you have seen?

https://spacelift.io

Thanks!

https://redd.it/zf4lpw
@r_devops

Resigning from a role of being "the guy"

I joined a company that was a startup which got acquired within 2 months of me joining. When I came aboard, things were a mess. They had installation scripts for their product for 4 different distros of linux, code was poorly optimized, the documentation was non-existent so I pretty much the read the deploy scripts to learn the architecture. As I did that, I dockerized the application to make it easy to develop locally. The guy in charge had a habit of editing code locally and pushing it to test servers to test it there and would do stuff like print/exit on the server side instead of debugging... the whole thing has been a shit show. I learned that when they initially launched that the scalability of the app was non-existent.

With the new dockerized components(API, UI, various services), container orchestration/scaling via ECS and so on, I found myself to be the only one capable of actually managing the operations side of things. I begged them to hire someone with AWS experience and to let me let me spend time documenting it because I came in on the heels of them doing an MVP with very little code or infrastructure documented. I figured they just needed to be shown how to do this stuff and they'd take to it. To the contrary... I was now "the guy" and as long as I was willing to sacrifice to keep things going, they were happy to let me do it. I stayed way longer than I should've and came to the conclusion that I'd leave during my Thanksgiving "vacation" because I spent quite a bit of time helping them.

I put in my notice and now documentation is SUPER IMPORTANT. Oddly, the documentation they want is not actual documentation but like a cheatsheet of commands to run when X, Y or Z happens.

Hopefully they learn from this and document properly in the future. I'm documenting things that existed over a year before I joined the company. My lesson learned from this is that I shouldn't have done any of it. I should've left as soon as I saw the shit show.

Lastly, the president of the company set up a meeting with me yesterday to counter offer for me to stay. When I declined, he brought up the fact that there was a "non-compete" in place and that he would be contacting my new employer to inform them of the non-compete. The new employer is a very large Fortune 100 company and I'll be filling an entirely different role. Before this happened, I was open to helping my former coworkers occasionally on my personal time because I had a feeling of survivor's guilt, but after that threat, the company itself can fornicate itself.

TLDR: I joined a shitshow, made some modest improvements to operations/deployment/etc, turned into a slave for the company since they were happy with me just being THE GUY, then they kinda threatened me when I resigned.

https://redd.it/zf6t2j
@r_devops

How do you chose which log collector to choose - Fluentd, fluentbit, Logstash, etc

There are a few log collectors out there - Fluentd, fluentbit, Logstash are the more popular oned . When setting up your log collection pipeline how do you choose which log collector should you choose?

Anyone has made such choices before, would love to understand how you decided

https://redd.it/zf9k5e
@r_devops

Any new years resolutions? Or... how did you get out of your 'normal' and start your real DevOps journey?



Some seconds of introspective thinking ago I thought about how and when (or if :( ) I can get out of the loop of "write pipeline > test pipeline > fix pipeline >> yey, it finally works >> oh I could improve it > repeat".
Like some of you; I do this for around 30 devs... - alone.
I mean, it plays with my "building things gud" itch, but I don't see myself becoming any happier, more satisfied or with more shiny things at the end of the month accepting this kinda... eh... 'slavery'?.
It became my new normal some time ago and I literally have no real time at work to 'change things'.
Oh yeah, K8 gud and it might at least massively improve our resource allocation problems and solve our jenkins agent bondage, but yeah, much to do at the end of the year and it's one more thing on my "you actually MUST do this" list.

So... a nice, simple question to any of you who got out of this shit:

HOW THE FUCK?

https://redd.it/zf6hjz
@r_devops

Does anybody else find api calls to invalidate cloudfront cache flaky?

I seem to get sporadic errors submitting an invalidate cache requests via awscli. I really don’t think it’s me. It happens in several different networks, local and GitHub actions pipelines as well. Is this a known issue, does anybody else run into this problem?

https://redd.it/zfg7n2
@r_devops

Effort levels as a student

I'm currently a sophomore at University and I'm very interested in pursuing DevOps as a career. I landed an internship this summer but don't have much experience with Kubernetes, Docker, Terraform, AWS, YAML, ruby on rails, or Linux. Assuming burnout is no issue, would it be a good idea to start learning about these technologies throughout the next school semester prior to my internship?


I ask because I was once offered the advice that "it's best not to set the standard too high during my internships". In retrospect, the only rational explanation for such advice is that starting slow is preferable since rapid improvement is a very desirable metric in it's own right. Maybe someone else has heard or offered this advice before?

https://redd.it/zfhged
@r_devops

How do you integrate a static security analysis tool into the CI/CD pipeline

I looked at PrimeJS, which is the only static security analysis tool for javascript and it can't be integrated into a CI/CD pipeline, because you need to run the script through the browser. Is there a way to automate the check instead of being forced to run the scripts one by one? JSLint is a static analysis tool, but it doesn't check for security flaws.

https://redd.it/zfkc4s
@r_devops

DevOps or cloud engineer

Hello, I am a fresher graduate in BS information technology, and got a bit confused, I worked as Technical support for about 3 years and now I am working on getting my AWS cloud practitioner certificate, so I can apply for a cloud engineer role and then I can go for a DevOps role, but some of my colleagues advised me that I should learn DevOps immediately, but I got confuses, so please what should I do

https://redd.it/zf07vd
@r_devops

Confused about database migration tools and proprietary software

I'm currently working on a personal project that I may want to monetize in the future. I'm very confused about Flyway and Liquibase. It seems like, if you are using their library as part of your business, you are required to pay them a licensing fee.

Is this true? And if so, what is a free alternative that allows me to run SQL migrations in proprietary software?

https://redd.it/zfc46s
@r_devops

I think my manager lied. I want to get into devops. I dont trust him. Toxic work culture

This might be a long one.

Tldr : Manager put us on pip and claimed he didn't know what it was. He lied. Hr told us all managers did. I think he shot down my devops chance.

Me and my colleague joined over a year ago as apprentices in it support.

In comes our new manager. More harsher and less lenient. Me and my colleague do 80% of the work. Set ups, tickets. You name it. And everyone in our company saw this.

We work extremely hard. The months went on and it was annual review time
We both got put on PIP as an apprentice. Reason why? We don't have any certs yet ( we're apprentices halfway through our course) and don't do enough. The things he said made us feel unvalued, unworthy and sht. He literally said " if I had the power you wouldn't be part of my team, but you're apprentices".

.. I told my manager I want to get into devops after my course ends. He seemed happy and said he would speak to HR. I spoke to head of hr. And they had a meeting with my manager. My manager told me that HR said their isn't any roles available (my company easily creates new roles for many people) it's a startup.

He suggested I dedicate Fridays into learning devops and shadowing those from that field. He was meant to speak to our upper line manager about it. Its been 3 weeks and he hasn't at all. Now I'm doubting everything he's said about what was said in the meeting.

As that went by, we caught him out on a handful of lies. 1 of the biggest one was when he said to us he didn't mean to put us on improvement needed because he didn't know what PIP is. He claimed he didn't know that PIP is the next step closer to getting sacked. He's been doing IT support for years. We spoke to someone from HR and they told us all managers are told about what PIP

We got put off PIP and rewarded the rest of our bonus.


We assume he said this because he wants us to stay and regrets treating us like sht

Does anyone have any advice? I'm planning on leaving and getting into a devops bootcamp. My progression in this company is clearly going nowhere.

https://redd.it/zfbbuz
@r_devops

How to Unfederate a WAN Federation Over Mesh Gateways (Consul K8s)

Hi all, I currently have 2 separate KE private clusters.
One for the primary DC, and one for the secondary DC.
My setup is as per the following hashicorp article: https:|/ developer.hashicorp.com/consul/docs/k8s/deployment-configurations/multi-cluster/kubernetes

However, when I go to use consul-k8s to uninstall consul on my secondary DC. I start to see errors that the primary
DC can no longer connect to the secondary DC.

This makes sense, as Consul has been uninstalled. However, after reinstalling and rejoining the WAN federation on my secondary DC. The errors continue in my primary DC, and the secondary DC starts to show errors saying it can no longer find the primary DC.

Is there a correct way to unfederate DC's/k8s clusters that are in a WAN federation over mesh gateways?

I'm imagining that scaling this up must be tremendously hard, as if you install something incorrectly, and you potentially had 50 secondary DC's. These DC's would all start to show errors as soon as one DC was removed.

This seems like and incredibly common use case, so l am assuming I am incorrectly using consul. However, I haven't been able to find any answers online.

Thanks.

https://redd.it/zfqr7t
@r_devops

At what point would you want a DevOps teammate?

One man army, DevOps hero, or whatever companies or you guys call it here.
I just now read a comment about how being the lone bitch for everything in DevOps will retard your career.

So, please tell me if you know:
What would you do or tell your higher ups that could nudge everything in a better direction?

At the moment I'm mostly putting out fires and can't even proactively think about solutions that prevent them.
And the responsibility for all new builds, tests and releases is on me.
The same for automation... and then they develop something completely new, outside of our existing toolchain, and want a pipeline with test matrix in two weeks. Nice.

https://redd.it/zfs8js
@r_devops

K8s load balancing scenario on pod crash

Hi all! I have a question and it would be great to get some answers. I started the CKA training and let's say we have 3 pods hosting our online store and they're exposed via a load balancer. A random customer add an item on the cart and he's trying to pay and place the order. What happens if that specific pod where the customer currently is fails somewhere along the ordering process, let's say when the customer insert his personal data? Is he getting a blank page and losing everything and he has to start placing the order from scratch? What can we do to at least keep the product on the cart?What is the best practice for avoiding this scenario? Thanks everyone.

https://redd.it/zf5s8g
@r_devops