Reddit DevOps
267 subscribers
30.9K links
Reddit DevOps. #devops
Thanks @reddit2telegram and @r_channels
Download Telegram
Tips for job fair booth tech company

Hey guys

My employer is participating at a job fair for developers in poland.
Since we mostly own pretty average and boring stuff for advertising our company at fairs we decided to change it up.

So to make it quick:

What have you guys seen on fairs that caught your attention and got you drawn towards?
What’s important to you when attending such a fair as a developer? No gos and must haves?
Just give me everything you have in mind.

We really want to create something cool and worthwhile for both sides, so every opinion is appreciated.

Have a nice week 🤝🏼

https://redd.it/1174sk1
@r_devops
OMG I JUST PASSED THE GCP PROFESSIONAL DEVOPS ENGINEER!!!

I’m so excited ….I will write a full write as soon as it sinks in!!! I can’t believe this!!

It was TOUGH! in fact terraform associate is a cake walk compared to THIS!!!

https://redd.it/117qe5i
@r_devops
What were the ancient tools used in late 90's early 00's ?

I'm interested in stories and experiences from people that had to maintain a fleet of server two decades ago, without terraform, ansible or any modern tool that we take for granted today.

What was the smartest thing you invented or saw in action that would help the sysadmin do a task in 2 minutes instead of 2 days?

I can think of many different ways to automate things without configuration management tools, but what was actually used?

https://redd.it/117q820
@r_devops
AWS EC2 vCPU limit increase

I am working on a project that requires me to use a new account to create an instance with GPU. This instance requires 4 vCPU and my current allowance for vCPU is 0.
I have applied for three petitions to increase the VCPU but all of them got rejected. In the three petitions, I tried different combinations of number of vCPU request and also one different location and I also mentioned that it is an academic project.

I want to ask what I should do to increase my vCPU limit to at least 4. Are there any specific locations on which my change of getting allotted is higher? Does anyone have any experience with this?

This is the reply I get from them every time:
Service quotas are put in place to help you gradually ramp up activity and decrease the likelihood of large bills due to sudden, unexpected spikes.

https://redd.it/117psxg
@r_devops
First devops job

Hi all. How was your first job as a devops engineer after finishing a bootcamp or program? Did you feel that you knew how to perform all of the tasks that were asked? Were you thoroughly trained?

https://redd.it/117ucld
@r_devops
How do you keep track of the deployments and the branches they come from?

As the title states, there’s a little chaos on our testing environments because we are not keeping an evidence of some sort regarding the branches where the configmaps and deploymentconfigs come from. Sometimes people just deploy new pods and mess up the QA work. So how could we keep track and know (without digging into jenkins and fabric) what branch was used last time someone deployed a new pod?
For context we are working on a microserviced app, on-prem with openshift and use jenkins with helm charts for pipelines and bitbucket for version control.

https://redd.it/1175xkc
@r_devops
Noob Chef Question

When you are running commands using chef, does it automatically run those commands as root?

https://redd.it/117vjit
@r_devops
Documentation as a service

What do you use for docs? I work at a mid size tech company (~100 Devs) and our tech docs are all over the place 🙈 Confluence, GitHub Wiki, markdown in GitHub and even a couple of home cooked Hugo sites... So yeah, company wide doc search is essentially impossible...

Am I alone with this problem?

What do you guys use for internal docs? Would you recommend any (free or paid) tool that you can plug into GitHub for markdown or tech docs (sphinx, jsdoc, golang package docs, terraform module doc, helm charts etc...)?

For extra info, we are rolling out Backstage and are thinking of using their tech docs feature but it is tightly bound to service entities so wouldn't apply to team docs for example.

https://redd.it/117wpsf
@r_devops
How do you manage your PROD Access?

Hello Folks,

We want to provide our operation team some permissions (On Azure) during the deployment window and then revoke once they have completed the operation.

What is the best way to achieve it or how do you guys usually do it?

I'm thinking to have some pipeline which could accept some input like operation team name, grant/revoke as inputs and run some az cli commands.

Please share your thoughts.

https://redd.it/117w7q1
@r_devops
Company wide identity provider

In the future we want to use a central identity provider in our company to authenticate and authorize our internal and external users against our applications and APIs.

The project is mainly driven by middle management and product managers. I am in the role of the DevOps engineer and take care of the infrastructure of our applications. Deploying an identity provider kind of falls under my tasks.

I've been working with AWS for years, so a natural choice would be to go with Cognito. While I like AWS products in general, I have doubts about whether Cognito is the right choice. Especially since people who are not fit in AWS should also work with it (operate it). But also because Cognito cannot be customized to the same extent as other identity providers.

In our company, there is currently only (dangerous) half-knowledge on the subject. I'm familiar with the basics of OAuth and OIDC, and I'm just getting more familiar with how it works. However, I am still a bit confused about which criteria to use to select a suitable identity provider.

My questions are: How should I proceed? Which criteria are decisive in your opinion? Should the software developers have a say in the selection? Should we hire an expert to assist us? Is there anyone here who has switched from Cognito to another authentication service? If so, why? Is there another recommendation? Is self-hosting a viable option?

Although this might not be something DevOps engineers usually do, I really enjoy digging into the nuts and bolts of this topic. I feel confident, that given enough time, I can make an informed decision. But management wants us to start as soon as possible with the first prototype.

https://redd.it/117yqln
@r_devops
Company wide identity provider

In the future we want to use a central identity provider in our company to authenticate and authorize our internal and external users against our applications and APIs.

The project is mainly driven by middle management and product managers. I am in the role of the DevOps engineer and take care of the infrastructure of our applications. Deploying an identity provider kind of falls under my tasks.

I've been working with AWS for years, so a natural choice would be to go with Cognito. While I like AWS products in general, I have doubts about whether Cognito is the right choice. Especially since people who are not fit in AWS should also work with it (operate it). But also because Cognito cannot be customized to the same extent as other identity providers.

In our company, there is currently only (dangerous) half-knowledge on the subject. I'm familiar with the basics of OAuth and OIDC, and I'm just getting more familiar with how it works. However, I am still a bit confused about which criteria to use to select a suitable identity provider.

My questions are: How should I proceed? Which criteria are decisive in your opinion? Should the software developers have a say in the selection? Should we hire an expert to assist us? Is there anyone here who has switched from Cognito to another authentication service? If so, why? Is there another recommendation? Is self-hosting a viable option?

Although this might not be something DevOps engineers usually do, I really enjoy digging into the nuts and bolts of this topic. I feel confident, that given enough time, I can make an informed decision. But management wants us to start as soon as possible with the first prototype.

https://redd.it/117yqln
@r_devops
Calculate replicability score on a project/repository

In my current company, we are trying to renovate our work methodology and one of the goals is to create project assets and/or libraries that can be replicated across multiple clients in order to accelerate development and generate marginality.

We have been asked to generate an internally shared replicability score of a project based on language, technology, and complexity.

Based on your experience, if you were to calculate a replicability score for a project what metrics would you use? What analysis strategy would you use?

https://redd.it/1180ijl
@r_devops
Site Reliability Engineer - Automotive AI experience - Available Immediately - USA

Hi all,

Using this platform more as a punt more than anything else.

I've been referred a very talented Site Reliability Engineer who has been laid off recently by one of US's biggest AI organisations. Mid-way through a very difficult personal period, he has reached out to myself and one other recruiter for opportunities on the market. Unfortunately, the opportunities I have for him would require him to be on-site atleast once a week but prefers remote.

If there are any hiring managers in the US who are looking for great SRE talent, this candidate can be vouched for by his recent and previous organisations and has refrained from using Linkedin because of past bad experience with external recruiters.

Happy to share some more details about his profile, please feel free to DM me. He's available for interview early next week.

https://redd.it/1181sdq
@r_devops
In-terminal debugging and CLI assistance https://github.com/HeyCLI/heyCLI_client

Hello all,

HeyCLI is a GPT3 powered command line helper that allows you to use the terminal in natural language. See [https://heycli.com](https://heycli.com)

We have added two new features to HeyCLI:

1. In-terminal debugging of python, nodejs, kubectl, gcloud, aws and many other command errors. When you get an error, just type: hey debug the error above (or something like that, be creative!) 
2. we attached HeyCLI to some commands so you can type "yes" to execute them. 

To try these features, follow the instructions here [https://github.com/HeyCLI/heyCLI\_client](https://github.com/HeyCLI/heyCLI_client)

https://redd.it/118071p
@r_devops
How to get notified on Error Events in NewRelic?

I am currently doing this in the browser:

newrelic.noticeError(err);

I am seeing them arrive in New Relic/Events, but how do I make it to get notifications via Email when error is logged via client side?

https://redd.it/11842bs
@r_devops
Bootstrapping and updating CI/CD permissions

I'm trying to determine the best way to bootstrap and update permissions for CI/CD minimizing manual steps but also keeping it secure, least privilege. I'm sure this is a common need. I'd like to understand good practices around it that I might be missing.

For example, setting up a GitHub repository with GitHub Actions managing infrastructure and apps in AWS. In order for GitHub Actions to have authorization, I need to add an OIDC provider and a role restricted to the GitHub repo with a policy with reasonably least privilege.

I'm OK with this being done by checking out the repo and running the initial IaC with local AWS permissions (I use SSO so that no long-lived Access Keys in \~/.aws). Bootstrapping done, no problems.

But now as the applications grow, more permissions are needed by GitHub Actions to manage new infrastructure. I don't want GitHub Actions in particular to have Administrator Access on the account. So to add the new permissions, I would update the permissions in the bootstrap IaC and apply manually.

Everything is still in git. Recreating or replicating the environment wouldn't have any manual steps apart from running the bootstrap IaC. It seems like the way to go.

The only thing that rubs me the wrong way is that permissions to manage app specific resources would have to be managed outside of the app. They could be pulled in from the app repo/folder and the bootstrap admin would have to verify the diff when applying it.

So my question is: How do you handle the bootstrapping and updating of bootstrapped permissions for your CI/CD?

https://redd.it/1184jz3
@r_devops
CDKTF FAQ

I was recently tasked with a CDKTF project. As I started to dig into CDKTF, I had a lot of questions. I realized that there wasn't really a good FAQ online so I decided to share what I've learned. CDKTF can be kind of confusing as it's new and I suspect many people have questions around it (I saw a question the other day here about it). I plan on updating this FAQ it as I continue to learn more about CDKTF.

https://terrateam.io/docs/cdktf/faq

https://redd.it/11881mp
@r_devops
Packer provisioning on a Ubuntu 20.04 refuses to work

Trying to create a golden image with cloud-init support on Proxmox.
Without any provisioning I can get a iso/template but as soon as I want to configure something with Packers provisioning shell it just hangs forever. I could work around this with further provisioning using Ansible & Co but I refuse to accept defeat in this case.

Packer + Proxmox = Custom Cloud-Init template, then use Terraform to create actual VMs

I know I can use any cloud image from Ubuntu/Debian/etc and let Terraform use it as template however I wanted to include this as a step. For now I can continue to use my CI/CD starting at Terraform but I want to get this working.
Any reference I saw so far was just dishing out shell provisioning commands with sudo without any further tweaks. I let it run for hours just to confirm I wasnt too impatient.

PACKER_LOG=1 shows nothing, output just straight up stops when it should run the specified commands

There is a current problem with Proxmox 7.3 and Packer Plugin 1.1.1, forcing me to use 1.1.0
Also no DHCP, thats why I set the network settings manually

hcl file

packer {
required_plugins {
proxmox = {
version = "1.1.0"
source = "github.com/hashicorp/proxmox"
}
}
}

source "proxmox-iso" "proxmox-ubuntu-20" {
proxmox_url = "url"
vm_name = "packer-ubuntu-20"
iso_url = "https://www.releases.ubuntu.com/focal/ubuntu-20.04.5-live-server-amd64.iso"
iso_checksum = "5035be37a7e9abbdc09f0d257f3e33416c1a0fb322ba860d42d74aa75c3468d4"
username = "user"
password = "pw"
token = "token"
node = "proxmox"
iso_storage_pool = "local"

ssh_username = "packer"
ssh_password = "ubuntu"
ssh_timeout = "20m"
ssh_pty = true
ssh_handshake_attempts = 20

http_directory = "http"
boot_command = [
"<esc><wait><esc><wait>",
"<f6><wait><esc><wait>",
"<bs><bs><bs><bs><bs>",
" ip=${cidrhost("10.0.104.0/24", 80)}::${cidrhost("10.0.104.0/24", 1)}:${cidrnetmask("10.0.104.0/24")}::::${cidrhost("10.0.104.0/24", 1)}",
" autoinstall ds=nocloud-net;s=https://{{ .HTTPIP }}:{{ .HTTPPort }}/ ",
"--- <enter>"
]
boot = "c"
boot_wait = "5s"
insecure_skip_tls_verify = true

template_name = "packer-ubuntu-20"
template_description = "packer generated ubuntu-20.04.3-server-amd64"
unmount_iso = true

memory = 4096
cores = 1
sockets = 1
os = "l26"
qemu_agent = true
cloud_init = true
cloud_init_storage_pool = "local-lvm"
# scsi_controller = "virtio-scsi-pci"
disks {
type = "scsi"
disk_size = "30G"
storage_pool = "local-lvm"
storage_pool_type = "lvm-thin"
format = "raw"
}
network_adapters {
bridge = "vmbr1"
model = "virtio"
firewall = true
vlan_tag = 104
}
}

build {

name = "ubuntu-server-focal-docker"
sources = ["source.proxmox-iso.proxmox-ubuntu-20"]

# Provisioning the VM Template for Cloud-Init Integration in Proxmox #1
provisioner "shell" {
inline = [
"while [ ! -f /var/lib/cloud/instance/boot-finished ]; do echo 'Waiting for cloud-init...'; sleep 1; done",
"echo 'Sudo now'",
"sudo rm /etc/ssh/ssh_host_*",
"sudo truncate -s 0 /etc/machine-id",
"sudo apt -y autoremove --purge",
"sudo apt -y clean",
"sudo apt -y autoclean",
"sudo cloud-init clean",
"sudo apt update -y",
"sudo apt upgrade -y",
"sudo rm -f /etc/cloud/cloud.cfg.d/subiquity-disable-cloudinit-networking.cfg"
"sudo sync"
]
}

# Provisioning the VM Template for Cloud-Init Integration in Proxmox #2
provisioner "file" {
source = "files/99-pve.cfg"
destination = "/tmp/99-pve.cfg"
}

# Provisioning the VM Template for Cloud-Init Integration in Proxmox #3
provisioner "shell" {
inline = [ "sudo cp /tmp/99-pve.cfg /etc/cloud/cloud.cfg.d/99-pve.cfg" ]
}
}

user-data

#cloud-config
autoinstall:
version: 1
locale: en_US
keyboard:
layout: de
ssh:
install-server: true
allow-pw: true
disable_root: true
ssh_quiet_keygen: true
allow_public_ssh_keys: true
identity:
hostname: packer-ubuntu-20
password: "$6$qiEN6LwtNwuOZoim$8nvdVicI/.oDb5W4ynnyhToYKegBUGDEWgomK6kymT6xalkuQaoqHhAY4xcurVQ50wDEBhF.OzHUKkm4NvoNe/"
username: packer
realname: packer
packages:
- qemu-guest-agent
- sudo
storage:
layout:
name: direct
swap:
size: 0

https://redd.it/1189igt
@r_devops
Learning So Much Info & It's Hard To Juggle It All

Hey all,

Feel free to remove this post for redundancy reasons - it's not much different than what's been asked many times over on this sub.

I've been following the DevOps roadmap (I know some of you find this controversial - I understand), and just to clarify, I graduated last year with a Bachelor's in Computer Science, so I already have a decent development background (although I could definitely improve, which is something I am working on). I find all of this a bit overwhelming, but I have a strong desire to learn, and I spend time (I try to get to 2 hours of concentrated learning after work, I'm 22 years old and no other obligations mostly) after work learning every single day.

I'm currently in more of a Security-based role at a very large company, which does have me integrated with my company's cloud architecture, so I am learning AWS naturally from that, and would like to get the CCP cert in time. I like the work, and I think some of it will definitely translate to my next job, but DevOps has been an attraction ever since I graduated, and I like the idea of touching so many different tools and technologies, and there's a ton to learn (which again, I enjoy).

I guess the point I'm getting at is, how do you (or did you, depending on your current experience level) juggle all of this new information? For example, I was never exposed to Linux in detail in school. Currently it's something that I'm diving into deeper, but it takes time. Another example is Networking - Networking is huge. I want to learn a ton about Networking, it's interesting! But I find myself frustrated sometimes with the knowledge gaps I have on different topics.

I don't want to just be an average engineer, I want to be good. I'm willing to put in the time. I guess I'd just like some advice on how to juggle all of this information. I know I'm young and have a lot to learn, but I don't want that to be an excuse to take things slow (within reason).

Thank you all!

https://redd.it/118bwzu
@r_devops
Am I being ripped off?

Some context: I job switched internally at my company to be an Associate DevOps Engineer. (I work at a small startup >500 employees). They had an opening for a full DevOps engineer and I told HR I was interested in joining, but didn’t have nearly enough experience. HR ended up telling the team and the hiring manager for the role contacted me and basically said he’d love to bring me on but as an associate, and they’d have to hire a senior to help train me. I didn’t know much about DevOps, just knew a ton of networking, Python, and some bash/linux.

Long story short eventually I got brought on the team and I’m about 3 months in. The only downside is I’m making the same as what I made in my previous role (entry level digital forensics) - $55,000. I love the team and it’s so cool learning all the tools (K8s, Docker, Azure, Debian, Terraform, etc) but I’m definitely doing more work than I was before. There was no pay increase as it was seen as a lateral movement.

The upside I see is that I don’t have to spend 10 yrs as a sys admin to break into DevOps. And I get to learn pretty much everything or anything I want.

Am I being ripped off? Should I be making more? Or is this worth it for the experience?

https://redd.it/118d7ax
@r_devops