Reddit DevOps
270 subscribers
6 photos
31.1K links
Reddit DevOps. #devops
Thanks @reddit2telegram and @r_channels
Download Telegram
CloudFormation template validation in NeoVim

I write a lot of CloudFormation at my job (press `F` to pay respects) and I use NeoVim (btw).

While the YAML language server and my Schema Store integration does a great job of letting me know if I've totally botched something, I really like knowing that my template will validate, and I really hate how long the AWS CLI command to do so is. So I wrote a `:Validate` user command and figured I'd share in case anybody else was in the same boat.

vim.api.nvim_create_user_command("Validate", function()
local file = vim.fn.expand("%") -- Get the current file path
if file == "" then
vim.notify("No file name detected.", vim.log.levels.ERROR)
return
end
vim.cmd("!" .. "aws cloudformation validate-template --template-body file://" .. file)
end, { desc = "Use the AWS CLI to validate the current buffer as a CloudFormation Template" })

As I write this, it occurs to me that a `pre-commit` Git hook would also be a good idea.

I hope somebody else finds this helpful/useful.

https://redd.it/1jez6eg
@r_devops
Staging database - What is the best approach?

I have a staging environment and production environment. I want to populate the staging environment with data, but I am uncertain what data to use, also regarding security/privacy best practices.

Regarding staging, I came across answers, such as this, stating that a staging enviroment shall essentially mirror a production environment, including the database.

>[...\] You should also make sure the complete environments are as similar as possible, and stay that way. This obviously includes the DB. I normally setup a sync either daily or hourly (depending on how often I am building the site or app) to maintain the DB, and will often run this as part of the build process.

From my understanding, this person implies they copy their production database to staging. I've seen answers how to copy a production database to staging, but what confuses me is that none of the answers raise questions about security. When I looked elsewhere, I saw entire threads concerned about data masking and anonymization.

>(Person A) I am getting old. But there used to be these guys called DBAs. They will clone the prod DB and run SQL scripts that they maintain to mask/sanitise/transpose data, even cut down size by deleting data (e.g. 10m rows to 10k rows) and then instantiate a new non-prod DB.

>(Person B) Back in the days, DBA team dumped production data, into the qa or stage and then CorpSec ran some kind of tool (don't remember the name but was an Oracle one) that anonymized the data. [...\]

However, there're also replies that imply one shouldn't use production data to begin with.

>(Person C) Use/create synthetic datasets.

>(Person D) Totally agree, production data is production data, and truly anonymizing it or randomizing it is hard. It only takes one slip-up to get into problems.

>(Person E) Well it's quite simple, really. Production PII data should never leave the production account.

So, it seems like there are the following approaches.

1. 1:1 copy production to staging without anonymization.
2. 1:1 copy production to staging with anonymization.
3. Create synthetical data to populate your staging database.

Since I store sensitive data, such as account data (e-mail, hashed password) and personal information that isn't accessible to other users, I assume option 3 is best for me to avoid any issues I may encounter in the future (?).

What option would you consider best, assuming you were to host a service which stores sensitive information and allows users to spend real money on it? And what approach do established companies usually use?

https://redd.it/1jezs2f
@r_devops
For those of you who left the tech industry, what do you do for work now?

Why did you make the change?
Are you less or more stressed?
How did it change your financial situation?
Do you regret leaving?


https://redd.it/1jf01yg
@r_devops
Kubernetes command line extras

I have a few kubectl scripts set up. I have "kubectl-ns", which switches the namespace:

printf '%s\n' "kubectl config set-context --current --namespace=\"$1\""
kubectl config set-context --current --namespace="$1"
printf '%s: %s\n' 'Current namespace is' "$(kubectl config view -o json | jq '."current-context" as $currentcontext|.contexts[]|select(.name==$currentcontext)|.context.namespace')"

and "kubectl-events", which just lists events sorted by ".metadata.creationTimestamp", which... why was that not built in from the start??

It'd be nice also if there was a command to give you an overview of what's happening in the namespace that you're in. Kind of like "kubectl get all", but formatted a little nicer, with the pods listed under the deployment and indented a little. Maybe some kind of info output about something. Kind of like "oc status", if you're familiar with that.

And today I just hit upon a command line that was useful to me:

kubectl get pods | rg -v '1/1\s+Running'

Whenever I restart deployments I watch the pods come up. But of course if I just do "kubectl get pods" there's a whole bunch in there that are running fine and they all get mixed up together. In the past I've grepped the output for ' 0/1 '. Doing it this way, however, has the minor benefit of still showing the header line. It's a little nicer.

https://redd.it/1jf11ox
@r_devops
Suggestions around Hosting Jenkins on Kubernetes

I work in startup with lot of things we are managing on our own. Current Jenkins setup we have EC2 machines- Literally created manually with manual configurations. And as a nodes we have another set of Ec2 machines which are also used for some other things. Developers keep logging to that machines.

Has anyone Hosted on Kubernetes , So something like Jenkins Server on Kubernetes, and Nodes of Separate Kubernetes Clusters [Multiple Cluster in Multiple Accounts\].

Why jenkins only ? Lot of pipelines are built by devs so i don't want new tools. Its just hosting part as that is in my control. But there are problems are in scaling , Long Jenkins Queue. Whatever and what not.



https://redd.it/1jeyeis
@r_devops
Can I opt for Certified Kubernetes Security free retake immediately after failing ?

My CKS exam voucher is nearing expiry, so I wish to know that if i give my CKS exam today and i fail in it so can i retake it tommorow or maybe day after or there is some time frame after which only I can retake it ?

https://redd.it/1jf3es6
@r_devops
I just canceled a technical interview because of the "assignment". Can someone reassure me that I'm not wrong?



I was talking with a startup in Northern Europe. They needed someone to handle their migration to Kubernetes and be responsible for the entire platform, from code to production.

This is something I already have experience in, so it looked like a good opportunity. The first interview went well. I spoke with the CTO, who I would be reporting to. The company seems advanced and has a good product.

Then came the technical interview invitation. They want me to design their company's "golden path" from code to production and give a 40-minute presentation about it.

At first, I was suspicious because I felt the topic was like "get a consultant for free". And when I started to calculate how much time it would take me to research, design, and create this presentation, I stopped completely.

I asked them if a 10-minute presentation would work. Their response was, roughly, "No, but we can give you more time to prepare if you want."

I told them, "No, thank you," and canceled the interview. To me, this seems unethical and completely disrespectful of my time. Are companies really asking for all this time from applicants these days?

https://redd.it/1jf7tve
@r_devops
Here's a quick summary of my job search and the offer I received - Software Developer with 20+ years of experience

\-To paint a clear picture, I'm an older developer (56 years old), I don't have a college degree, and I haven't worked at FAANG. I started 24 years ago. The salary I was looking for was 160k to 170k, and fully remote work.

\-Started looking for a job: December 2nd

\-Applications/resumes sent: Around 40

\-Number of interviews: 2 (4 with the company that hired me, and 1 with another company. This second company is the one that contacted me).

\-Accepted the offer: January 10th. (Meaning only one month of searching, but the company that hired me started the process after the first week of searching)

\-I only used LinkedIn.

\-I only applied to jobs where my skills were a very strong match. Sometimes I made exceptions for opportunities in areas where I have extensive experience (usually in e-commerce or education). The company that hired me was a combination of a good technological fit and vertical experience (related to education).

\-I focused on companies in my NYC area so I could sell the advantage of being able to meet them in the company if they needed to. But none of them responded to me, even though it seemed like a good plan.

\-I ignored job postings that were older than a few days, and focused on the brand new ones that had less than 150 applicants.

\-I tailored my resume for each posting by removing any technology that was completely unrelated to the requirements.

\-I excluded all years of experience except for the last 15 years to avoid age discrimination and outdated technology.

\-I studied Leetcode problems.

\-using AI tools like chatGpt or interviewhammer

https://redd.it/1jf7zqm
@r_devops
Runs-on vs. terraform-aws-github-runner

Hey guys 👋

I’m planning on implementing both solution for POC and comparison for my client soon, anything I should be aware of / known issues?
How was your experience with either solution and why did you end up selecting one over the other?

Runs-on fairly new, and require licensing both offer greater flexibility (resource requests are made in the workflow manifest)

terraform-aws-github-runner is and enhanced version of Phillips’ original solution, well known and popular.

This is NOT an ARC (github k8s controller), I won’t spin up a cluster and maintain it just for that. Doesn’t fit my client needs.

https://redd.it/1jf593d
@r_devops
What are available career pathways for me to take as a junior DevOps?

So for record, I have 2 years of Software Engineering experience working on Fullstack web apps, and I am currently in a Junior DevOps position.

I am curious if anyone has any advice for me with my credentials on where I could potentially advance in my skillset. I am most likely going to do an Azure Certification, possibly both AZ-204 and AZ-104.

I am possibly interested in security as well. But I was wondering what are my options for advancing my skill set and what career pathways there are for me?

https://redd.it/1jfbi1u
@r_devops
Thinking of moving from New Relic to Datadog or Observe

My company is thinking of moving from NR to either DD or Observe. Wondering if anyone has done this change and how it went?

If so, how much of a lift was it to move from NR to DD or Observe?

I’m a bit concerned about how much time and effort it may take to move over & get everything configured - especially with alerts.

Any advice would be greatly appreciated !

https://redd.it/1jfbmly
@r_devops
Framing work experience

Hi DevOps community. I was hoping that the community could shed some light on how to frame a particular year of my work experience while looking for new roles? For context, I have 4 total years of professional experience. 1 of those years I worked as a Systems Engineer for a well-known IT management consulting firm that is primarily a DoD contractor (wont directly say the name of the company but it’s the one that “House of Lies” is based on), and while there I had an active Secret clearance. On top of that there was so much red tape that I was only ever assigned to two (very) slow-moving projects. I don’t know how to properly frame my experience there in interviews. Please be constructive but kind. Thanks everyone!

https://redd.it/1jfk5d0
@r_devops
Anyone use Cribl?

I have a team at work that is doing a PoC of the Cribl product for a very specific use case, but wondering if it is worth a closer look as an enterprise 0lly pipeline tool.

https://redd.it/1jfp117
@r_devops
Weird situation after reorg

Hey all. I am looking for some advice. As part of a reorg, I was transitioned to the ops team's manager, who manages a team of infra/devops engineers. Previously, I used to report to the engineering team director and I am the only devops guy managing an app.

It's been over 2 weeks but I haven't heard anything from this new manager. I even sent an email 4 days ago asking to set up a quick call, but no response. He also doesn't look to be on PTO, his status always shows available or in a meeting. I am feeling a bit stuck and left out. To add to the challenge, the other team members of this team manage totally different products/apps, so there hasn't been much overlap or opportunities to naturally connect.

Just wanted to get any ideas on how to approach this. I'm also worried about lack of communication going forward working with his team.

Thanks!


https://redd.it/1jfrd5r
@r_devops
AWS DevOps & SysAdmin: Your Biggest Deployment Challenge?

Hi everyone, I've spent years streamlining AWS deployments and managing scalable systems for clients. What’s the toughest challenge you've faced with automation or infrastructure management? I’d be happy to share some insights and learn about your experiences.

https://redd.it/1jfscf7
@r_devops
How to set realistic expectations for adhoc work

I'm a DevOps consultant and a previous employer. The feedback I got from my manager was that I wasn't scanning Slack enough for ad-hoc work. I was a team of 1 in charge of everything infrastructure and security related for the startup. Sometimes if I was working on something that required a lot of concentration and debugging I would not want to context switch to a slack thread partially if I wasn't tagged or sent a direct message.

Basically I was expected to constantly scan slack channels and respond to any issues developers were having asap and drop everything I was doing. For example one of the gitlab runners was slow and having poor performance. The gitlab runner was still operational but builds were taking 10 to 15 minutes longer than normal for a job that usually takes 10 minutes. My Manager told me because I didn't stop everything I was working on reply that I was working on a fix with 15 minutes and resolve the issue within 1 to 2 hours that I was at fault. I was told this days later after the issue had been fixed because I was worked on the fix for a slow gitlab runner later in the day.

I was not getting direct messages or being tagged so this would mean scanning the common slack channels every 5 to 10 minutes all day which seemed unrealistic if I am doing active development work through out the day on other features. I didn't want to seem lazy because I was willing to work 70 hour weeks if it was required but the client got mad because I would not respond to messages within 20 minutes at 8 PM at night when I was at the gym for a code review for something not urgent.

Is these just really odd expectations of devops at startups or has any else encounter unrealistic expectations from a manager similar to this and how you met them or convinced the manager of more realistic expectations?

https://redd.it/1jfr7pn
@r_devops
Need help for PipeLines

# TLDR;

Junior dev, the only one on the team who cares about pipelines, looking for advice on how to go about serverless.

# Thanks a lot

So I'm back. I'm the guy from this post. I'm very grateful for the help you guys gave me a couple of months ago. We're using Liquibase that a lot of you recommended and I managed to create a couple of pipelines in GitLab trying to automate a couple of things. I'm here because, while I enjoyed trying out Liquibase and building those little pipes, I'm pretty lost.

Let me explain:

## What we have

We started using Liquibase as I mentioned before and it's really helping. After that I decided to try Gitea and test some pipes (we were using GitHub Enterprise Server on-premises). Long story short, I really liked it, but I felt like it wasn't as enterprise-ready as GitLab.

We started using GitLab and with its sprint management and pipes the whole team was impressed. Well, more for sprint management. I decided that automating things was good, so I got to work and after a week I had a set of usable steps for pipes.

We are not using a repo for pipes because we are still trying it out, we only have a couple of repos and this repo is the only one that has pipes. I read that you can create a single repo for those and have another repo call the step on that or something.

Anyway we develop on .Net for BE and typescript with React for FE. I created 3 groups of pipes distributed in some stages:

- build

- test

- analyze (used for static analysis with SonarQube)

- lint

- deploy (used to publish a new version of lambda and push new files to S3 for FE)

- publish (used to apply that new THING on the various envs dev|test|demo|prod)

Maybe publish and deploy are used for switched things, but you get the idea.

Build, test, analyze and lint are executed on every commit on main (we are using Trunk but no one knows about it except me, I keep it a secret because some people don't like it)

Deploy is executed on tags like Release-v0.5.89 while publish on Release-dev|test|demo|prod-v0.5.89. We started logging the status code of the action executed by BE from both APIs and BusinessLogic to CloudWatch to track the error rate in a future pipe although I don't know how to use this data yet.

I feel like I need a little hint. Like what to look for or what the purpose of the next action should be. I was thinking about a way to auto rollback but our site is not in production so we are the only ones using it at the moment. Help?? 🥹

If it helps I can post the pipes via a pastebin or something tomorrow morning (Central European TZ zone).

Edit: fixed syntax and linting 😆. The first published was a rush through and i don't really read back what i wrote

https://redd.it/1jfuczh
@r_devops
The outdated and the new tools you use/prefer?

I'm a fresher (3rd year undergrad), I heard docker is getting outdated and container runtime is not docker anymore and it is containerd from senior, its a new thing for me , I have heard of containerd and never worked on it, what else are there like these to differentiate me from others?

https://redd.it/1jfxtqy
@r_devops