What are the production level issues that you've faced/fixed and are very proud of?

Please enlighten us with something that you've fixed or faced that everyone can learn from.

https://redd.it/y42l2l
@r_devops

Proxmox, packer Ubuntu autoinstall

Hi all,

I've been working through multiple issues the past few days trying to learn some devops tools in my own homelab. The end state is to have my self hosted gitlab run packer and terraform jobs with the gitlab runner I have on a kubernetes cluster.

I have gotten almost all of it in a working but not perfect state. The issue I'm having right now is packer spins up an http server to talk to proxmox during the automatic install when adding a cloud-init option.

From the documentation you can add the user-data file needed to a cd\_file declared in your hcl. However when I do this the gitlab runner gives the following error. I've looked at similar issues on hashicorps website and their packer documentation. Now I turn to you all for a little advice. I picked this up this week and feel unworthy of even posting in here, lol. Thanks for any guidance in advance.

```
Unsupported argument cd_files is not expected here.
```
```
# Ubuntu Server jammy
# ---
# Packer Template to create an Ubuntu Server (jammy) on Proxmox

# Variable Definitions


# Resource Definiation for the VM Template
source "proxmox" "ubuntu-server-jammy" {

# Proxmox Connection Settings
proxmox_url = var.proxmox_api_url
username = var.proxmox_api_token_id
token = var.proxmox_api_token_secret
# (Optional) Skip TLS Verification
insecure_skip_tls_verify = true

# VM General Settings
node = "hv1"
vm_id = "9006"
vm_name = "Hashi-ubuntu-server-jammy-1"
template_description = "Hashistack"

# VM OS Settings
# (Option 1) Local ISO File
# iso_file = "local:iso/ubuntu-22.04-live-server-amd64.iso"
# - or -
# (Option 2) Download ISO
iso_url = "https://releases.ubuntu.com/22.04/ubuntu-22.04.1-live-server-amd64.iso"
iso_checksum = "10f19c5b2b8d6db711582e0e27f5116296c34fe4b313ba45f9b201a5007056cb"
iso_storage_pool = "ISO_Storage" # Specify your storage pool
unmount_iso = true

# VM System Settings
qemu_agent = true

# VM Hard Disk Settings
scsi_controller = "virtio-scsi-pci"

disks {
disk_size = "20G"
format = "raw"
storage_pool = "pool1" #Specify your storage pool
storage_pool_type = "zfs" #Specify pool type
type = "virtio"
}

# VM CPU Settings
cores = "8"

# VM Memory Settings
memory = "16384"

# VM Network Settings
network_adapters {
model = "virtio"
bridge = "vmbr0"
firewall = "false"
}


# VM Cloud-Init Settings
cloud_init = true
cloud_init_storage_pool = "pool1" # Specify your storage pool

# PACKER Boot Commands
boot_command = [
"<esc><wait>",
"e<wait>",
"<down><down><down><end>",
"<bs><bs><bs><bs><wait>",
"autoinstall ds=nocloud-net;s=/cidata/ ---<wait>",
"<f10><wait>"
]

cd_files = ["./http/meta-data", "./http/user-data"]
cd_label = "cidata"
boot = "c"
boot_wait = "5s"

# PACKER Autoinstall Settings
#http_directory = "http"
# (Optional) Bind IP Address and Port
#http_bind_address = "0.0.0.0"
#http_port_min = 8802
#http_port_max = 8802

ssh_username = "ubuntu"

# (Option 1) Add your Password here
ssh_password = "ubuntu"
# - or -
# (Option 2) Add your Private SSH KEY file here
#ssh_private_key_file = "~/.ssh/ansible"

# Raise the timeout, when installation takes longer
ssh_timeout = "20m"
}

# Build Definition to create the VM Template
build {

name = "Hashi-ubuntu-server-jammy"
sources = ["source.proxmox.ubuntu-server-jammy"]

# Provisioning the VM Template for Cloud-Init Integration in Proxmox #1
provisioner "shell" {
inline = [
"while [ ! -f /var/lib/cloud/instance/boot-finished ]; do echo 'Waiting for cloud-init...'; sleep 1; done",
"sudo rm /etc/ssh/ssh_host_*",
"sudo truncate -s 0 /etc/machine-id",
"sudo apt -y autoremove --purge",
"sudo apt -y clean",
"sudo apt -y autoclean",
"sudo cloud-init clean",
"sudo rm -f

/etc/cloud/cloud.cfg.d/subiquity-disable-cloudinit-networking.cfg",
"sudo sync"
]
}

provisioner "shell" {
inline = [
"sudo wget -O- https://apt.releases.hashicorp.com/gpg | gpg --dearmor | sudo tee /usr/share/keyrings/hashicorp-archive-keyring.gpg",
"echo deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main | sudo tee /etc/apt/sources.list.d/hashicorp.list",
"sudo apt update",
"sudo apt install nomad -y",
"sudo apt install consul -y",
"sudo apt install vault -y",
]
}

# Provisioning the VM Template for Cloud-Init Integration in Proxmox #2
provisioner "file" {
source = "99-pve.cfg"
destination = "/tmp/99-pve.cfg"
}

# Provisioning the VM Template for Cloud-Init Integration in Proxmox #3
provisioner "shell" {
inline = ["sudo cp /tmp/99-pve.cfg /etc/cloud/cloud.cfg.d/99-pve.cfg"]
}

# Add additional provisioning scripts here
# ...
}


```

https://redd.it/y52mef
@r_devops

Cloud Agnostic Framework discussions

Where are there good discussion boards on creating cloud agnostic frameworks? Seems like having the ability to move cloud providers quickly and easily to get better performance/price and create more resilience would be top of mind for most companies.

https://redd.it/y40oiv
@r_devops

Help me, Obi-Wan Dev-obi....

So a little background: I am a moderately experienced IT support tech/system administrator who, about a year ago, completed my BS in Cloud Computing and Systems Administration from WGU.

Now, however you feel about that degree, school, what exactly 'moderately experienced' means is all well and good. But what I'd appreciate is some advice and maybe guidance if possible. I'm trying to expand my wheelhouse and get into cloud services, automation, etc.

The big problem i'm running into is that I lack the 'required' amount of experience or any proof that I can do the job at hand. So, the real question I am asking:

What are some programs/courses (think LinkedIn Learning, Udemy, etc) that I can follow or projects I might consider that I can use?

Thanks for any and all information. Ya boy appreciates ya.

https://redd.it/y421wp
@r_devops

Why is EKS (and AWS in general) so much more convoluted than GKE/GCP?

Due to free credits running out, I'm in the process of moving my company's small kubernetes cluster from GKE to EKS. I'm using terraform for replication across environments.

&#x200B;

Starting on GKE was really simple. I was able to use the raw GCP provider to set things up. The cluster really worked out of the box. Autoscaler included. Networking just worked. Typically, you may have to enable APIs along the way, but nothing crazy...

&#x200B;

My EKS experience has been hell. The number of small pieces you have to configure is pretty ridiculous imo. The auth system is annoying. I didn't even try to get cluster autoscaling set up and wen with the terrform blueprints: https://github.com/aws-ia/terraform-aws-eks-blueprints .It really feels like AWS is designed to help enterprises fill out their headcount.

&#x200B;

I'm just wondering why the experience seem SO much different between the two cloud providers. You would think they would want user experience parity.

https://redd.it/y5am95
@r_devops

How to add in one .htpasswd file in nginx access to two different domain?

Hello
How I can add to /etc/nginx/.htpasswd access for two
difference domain for two difference login and password?

https://redd.it/y5bx0g
@r_devops

Trying to learn pre-commit - how to handle multiple hooks in a small monorepo with different paths?

Let's say I have a monorepo that has the following structure:

\
\app
\file1.py
\file2.py
\terraform
\infra
\terragrunt.hcl
\module
\examples
\README.md
\web
\index.html
.pre-commit-config.yml
.terraform-docs.yml

My goal is to figure out how to target only the "terraform" path with the hooks related to terraform, the "app" path with hooks related to python, and so on. But I also want to keep it all in one .pre-commit-config.yml. Is this doable? If so, where am I going wrong?

.pre-commit-config.yml is as follows:

repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v2.3.0
hooks:
- id: check-yaml
- id: end-of-file-fixer
- id: trailing-whitespace
- repo: https://github.com/antonbabenko/pre-commit-terraform
rev: v1.76.0
hooks:
- id: terraformfmt
args: [terraform/]
- id: terraformdocs
args: terraform/module/


And here is my .terraform-docs.yml:

formatter: markdown table
header-from:
recursive:
enabled: true
path: .
output:
file: ../README.md
template: |-
<!-- BEGINTFDOCS -->
{{ .Content }}
<!-- ENDTFDOCS -->
sort:
enabled: true
by: name
settings:
anchor: true
color: true
default: true
description: false
escape: true
hide-empty: false
html: true
indent: 2
lockfile: true
read-comments: true
required: true
sensitive: true
type: true

It seems like it is not respecting the paths I've added as arguments. So some of it works recursively, but it doesn't work to target specific areas of the repo such as the README for the terraform module.

I've been looking at the documentation but it's not really making sense yet.

Any help is appreciated.

https://redd.it/y5gfkb
@r_devops

Novice developer needs advice on pushing code from local to prod

I shouldn't call myself a developer because imo it's a disservice to actual devs. But let me explain my situation.

I have a dev and prod environment in aws. Initially the dev environment was a replica of prod. My workflow was to push the code to the repo, then have Jenkins deploy to the different environments.

Over time as prod became stable, and needed less work, I've developed a bad habit of doing local dev work and deploying directly to the dev environment. Honestly it was because Jenkins was a pain in the ass to maintain.

Now that I'm ready to replicate the dev code to prod I'm at a loss on what to do and how to do it. For example, in the dev repo and environment is a fully "functioning" API with things hard coded that shouldn't be.

I want to deploy the code to prod but make it more secure before doing so. In addition, I don't like Jenkins. I understand it's awesome in small teams or enterprise environments but it's just me, and it's a beast to maintain.

My questions are, should I create a prod branch, remove everything that's hard coded and secure it? Then use bitbucket pipelines to deploy it to the prod environment?

I know this is a trivial thing... But I've forgotten what right looks like.

Any help you can provide would be greatly appreciated.

https://redd.it/y5ih9j
@r_devops

Books for DevOps

Basically the title any good books to get into DevOps i have searched and found that the most recent books are from 2018 or before while the concepts is the same, but I am afraid they maybe be using deprecated technologies in the book. what do you guys think?

https://redd.it/y5n005
@r_devops

Entry Level DevOps Question

I am preparing for a couple of entry-level/junior DevOps interviews by watching mock interview and researching typical questions online. However, many of the mock interviews don't specifically say if it is for entry level, mid, or senior.

Can anyone tell me what level position this mock interview is? It seems way to difficult for entry/junior level, or maybe I am just unprepared?

https://www.youtube.com/watch?v=Z\_bbozP6ZW4&t=406s

Thanks

https://redd.it/y5oxeb
@r_devops

IaC relation to DevOps

Disclaimer, I have very little hands-on experience with most DevOps tools and am just learning the concepts so much of this is still abstract to me.

I am having trouble understanding the link between IaC(Terraform in my case) and CI/CD.

I know you use IaC tools to provision cloud infrastructure.

I also know CI/CD is for continuous integration and deployment for software like an app program.

How does say a Terraform script for say a multi-tier architecture on AWS relate to CI/CD?

&#x200B;

Can someone give me a simple example of how Terraform is used in DevOps?(like how it fits in CI/CD?)

https://redd.it/y5q2cy
@r_devops

DevOps training for inexperienced

Hey guys, I come from a background of on-prem Win servers with manual config/deployments and lots of in house legacy code.
I was previously QA and am now moving into Product Ownership. I want to understand more technical areas especially around Linux, Cloud hosting, configuration handling etc... Could you recommend great resources suitable for someone without any previous knowledge?

https://redd.it/y5pv2h
@r_devops

Tips for a new Jr. DevOps Engineer

Hi everyone!

I’m starting a new role as a Jr. DevOps Engineer tomorrow. It’s been my dream to break into DevOps; I started as a SWE about 2 years ago working with Linux and Microsoft’s tech stack (.NET, Azure, PowerShell, etc.). I have a background in Lean Six Sigma and full-stack development as well.

If you were starting off in your DevOps journey again, what advice would you tell yourself? What would you do to make yourself reputable in the next 6 months that you wouldn’t have thought of as a newbie? Can you share a bit about what your first few months to a year were like after getting into DevOps?

https://redd.it/y5mn7t
@r_devops

Interesting Twitter thread - free training

https://twitter.com/ghumare64/status/1581497472043536385?t=t1qyqzxwGLi3Ok8aX_nHtA&s=19

Also, let me leave this Github repo with ressources about free training, free hosting and cloud free tier programs https://github.com/cloudcommunity

https://redd.it/y5xexi
@r_devops

Advice transitioning from Software Development to DevOps while unemployed with outdated skills?

I'm in a bit of a tight spot these days where I can't seem to get a job as a software developer, or anything related. I took a 2 year college course from 2005-2007 for computer programming, and was hand-picked by a company due to my proficiency in C++. I worked there for over 10 years on a suite of desktop applications related to GIS mapping. I worked with DevOps, QA, and Project Management extensively but the majority of my time was spent fixing bugs or implementing new features in the applications. I'm familiar with Jenkins and build farms, but I only ever used the CI/CD pipeline - I didn't manage it.

Due to struggles elsewhere in my life (I was depressed, morbidly obese, and frustrated with my sedentary self) I ended up quitting on good terms, but in a way that didn't facilitate any kind of a future in front of a computer, as I didn't want to do that anymore. Basically I've spent the past 5 years doing laborious jobs. The good news is that I've lost over 100 pounds and feel very happy with being alive, but the bad news is I can't seem to get a job as a software developer, or anything related, no matter how much I apply. I feel like I need someone to hear my story, then give me a chance to grow into a position much like an intern or entry-level person would, but I can't get past the resume submission process because I simply don't have the qualifications, or because the job is for interns and they actually require you to be in school to be accepted for it.

I seem to be starting from square one, and if I had a choice as to what to do I think I'd prefer DevOps over SD/QA. I feel confident that I could follow someone around for a few months seeing what they do, and do it just as well as them (which is probably true of most technically-inclined people with a willingness to learn), but as it stands I don't have the required qualifications, and I'm not really in a position to go back to college.

I have a willingness to learn, and a strong desire to build a career, and I'd even take an abusively low starting salary for the first year if it got my foot in the door, but I'm at a loss for how to move forward. Even if I watched a billion youtube videos and went through some online courses, I'm not sure how it would affect the job application barrier I've faced.

I guess my question is how does a middle-aged man with outdated software development skills get a job - any job - that lets me begin building a career, preferably in DevOps? Are there educational institutions that will teach and employ me, or perhaps some kind of staffing agency that actually has a good history of helping people re-enter a tech field? Thanks

https://redd.it/y5yde3
@r_devops

I wrote a tool to speed up the testing stage by 95%

Consider starring on Github as it really does help a bunch w/ trying to break into big corp envs.

MIT License - https://github.com/nabaz-io/nabaz

Most tests shouldn't run, what I observed as an automation engineer is that when tests fail usually only a very small subset of them do (2-5) tests out of hundreds, or thousands.

In a Israeli cyber firm I worked for, running the full suite of tests in CI was absolutely impractical as they would collectivily take hours!

I wrote a tool that runs in production CI/CD of some software companies.

I decided to go open source, It collects code coverage separately for every test and then compares the per test coverage to changed code, if no changes were made the test is automatically skipped.

P.S

It also watches for changes in env vars, config files, and resource files. (coming very soon).

Would love feedback.

Yuval

https://redd.it/y653qa
@r_devops

Help to understand the pricing of CDNs

Google Cloud Storage (GCS) be used as an origin server for Fastly CDN. We currently use just GCS and have a monthly bandwidth of about 30 TB. For that, we have to pay about $6000.

According to the Fastly pricing calculator we would just have to pay about $2800 for that bandwidth. So with Fastly the content would be loaded faster (since it's a CDN and it has caching) and we pay less than half the price?! Why the hell would we not do it?

The question is just: We would continue to use GCS as the origin server because the media is already uploaded there. So I guess we would still have to pay GCS in some parts. But in which parts? Just for storing the data and no longer for deliviring?

Or would the two prices actually just add up and we would have to pay about $9000?!

https://redd.it/y66enj
@r_devops

Has anyone tried logging with Seq, what is the pros and cons of having Seq instead of ELK or Loki+Grafana stack?

I'm digging through the internet to find the comparison with Seq and others logging tools but seems like this is the best article that I can find: Choosing the Right Log Aggregation Tool | by Jacob Taylor | Medium

So, what is your opinion about seq, have you implemented it into your project and what is it pros and cons?

https://redd.it/y69v79
@r_devops

SDK's and DevOps

How are SDK's like Boto3 normally used in a DevOps environment?

I know you can use Boto3 to create, configure, or get data from AWS resources, but can't a lot of other DevOps tools do the same thing?

https://redd.it/y632px
@r_devops

Cloud Services

Sorry if this is the wrong place, I checked the rules but please remove the post if necessary.

I am a newer dev and my team has a job That runs about 50 times per month. It just consists of a few API calls and only takes a couple minutes to run in worst case scenario.

Our internal infrastructure for hosting it is really bad and consistently causes the job to fail. I want to look at a provider like AWS or Azure, but wasn't sure where to start. What kind of service should I be looking for? Just looking for a good place ro start my research as I have 0 cloud experience.

Thanks for the help.

https://redd.it/y6dkje
@r_devops