Packer+QEMU+GitLab CI = Can't SSH

Hey all, I've got a local offline GitLab CI instance and I'm trying to deploy a new QEMU disk using Packer and I'm running into issues. Everything works locally on my machine, however on the gitlab-runner, it can't seem to connect through SSH and running with PACKER\_LOG=1 doesn't provide any insight - it just keeps attempting SSH and failing, which is "normal" and happens locally as well until the reboot and then it succeeds. I'll provide the files in play here with some of the "fluff" snipped out and if anyone spots something I may be missing or knows the issue, please let me know!

[deploy.sh](https://deploy.sh) which is called from the GitLab CI pipe:

...
CHECKPOINT_DISABLE=1 PACKER_LOG=1 packer build -var "http_path=${CI_PROJECT_PATH}" \
-var "vm_name=IFS_${CI_COMMIT_BRANCH}_${CI_COMMIT_TIMESTAMP}" \
-var "iso_url=file:/builds/${CI_PROJECT_PATH}/utilities/packer/CentOS-${CENTOS_VERSION}.iso" \
-var "kickstart=centos7-ks.cfg" \
-var "ssh_pass=${PACKER_SSH_PASS}" \
IFS_minimal.pkr.hcl
...

IFS\_minimal.pkr.hcl

# Variables snipped, but they're all just strings
locals { boot_command = concat(["<tab> text ks=https://{{ .HTTPIP }}:{{ .HTTPPort }}/", var.kickstart, "<enter><wait>"])}
source "qemu" "centos7-minimal" {
accelerator = "kvm"
boot_command = local.boot_command
boot_wait = "3s"
disk_interface = "virtio"
disk_size = "5000M"
format = "qcow2"
headless = "true"
http_directory = var.http_path
iso_checksum = "md5:a4711c4fa6a1fb32bd555fae8d885b12"
iso_url = var.iso_url
net_device = "virtio-net"
output_directory = "packer_images"
shutdown_command = "echo 'packer' | sudo -S shutdown -P now"
ssh_username = "root"
ssh_password = var.ssh_pass
ssh_timeout = "25m"
vm_name = var.vm_name
}
build {
name = "Build 1"
sources = ["source.qemu.centos7-minimal"]
}

centos7-ks.cfg

...
network --bootproto=dhcp --device=eth0 --activate --noipv6
firewall --enabled --http --ssh
services --enabled=network,ssh
...
rootpw --plaintext XXXXXX #matching what's passed above in ${PACKER_SSH_PASS}
sshpw --username=root XXXXXX #matching what's passed above in ${PACKER_SSH_PASS}

reboot

%packages
@core
net-tools
libssh2.x86_64
openssh-clients.x86_64
openssh-server.x86_64
openssh.x86_64
%end

Dockerfile for the runner

...
RUN apk update && apk add --no-cache \
qemu-img \
qemu-system-x86_64 \
libvirt-daemon \
virt-manager \
openssh \
openssh-keygen

RUN adduser -D -S -h /home/gitlab-runner gitlab-runner && \
addgroup gitlab-runner qemu && \
addgroup gitlab-runner libvirt && \
addgroup root libvirt && \
addgroup root qemu
...
RUN sed -i s/#PermitRootLogin.*/PermitRootLogin\ yes/ /etc/ssh/sshd_config \
&& echo "root:XXXXXX" | chpasswd #Same password as ${PACKER_SSH_PASS}
...
EXPOSE 22

https://redd.it/lco69c
@r_devops

retaining exit code

I have an rsync that that is piped to a sed. it looks something like this:

rsync some-files remote-location | sed 's/\r/\n/g'

Problem is, the thing that runs it and reads the output, is a python code. Because it is piped into a sed, even though it fails sometimes, it still returns succesful.

Is it possible to have it return the success code based on the rsync? Thanks ahead!

https://redd.it/lcl6zg
@r_devops

Change in devops workload after move to Kubernetes

If your company/shop made a move to kubernetes, as a devops team after you setup the kubernetes cluster, do you see reduced workload in devops and maybe more into observability/monitoring?

https://redd.it/lcndpa
@r_devops

What is the better approach to Helm charts with the same specification?

Is it worth to duplicate Helm charts if the application is different, but has the same template, except the docker image used for it. A simplified example is if you have like 5 sites, all of them are a static site (or some simple services). All of them has the same requirements like deployment, service, hpa and other. The only difference is the domain and the docker image.

It sounds logical to just use the same chart and create a different values yaml file for each, but is it a "best practice"? Would it be wiser to duplicate the chart for it? Or creating a chart with sub-charts?

Right now I have a chart for each of them, but when something changes, it's a bit of pain to update everywhere. It has better control over what happens in there, but painful.

https://redd.it/lchg6l
@r_devops

IaC and DevOps Standards for Azure Kubernetes Service

Is there an infrastructure as code and DevOps standardization and best practices that people would recommend like using github and ACR tasks to automatically push images then idk does that update ACI which then connectors update AKS. Whats the latest and greatest way

https://redd.it/lclgwx
@r_devops

We're trying to pick an apm ... has new relic changed for the better?

Saw that new relic made some changes (about time) to their pricing, integrations w/ OpenTelemetry and Grafana, and better alerting.

I used new relic before, but it has been a few years. Wasn't a fan at the time, but we're thinking about it now. I'm hoping to hear from people that use new relic if these changes are actually a big deal or if it's same old new relic, lol.

View Poll

https://redd.it/lchjbr
@r_devops

What monitoring tool should I use to monitor the sass services I use?

I want to monitor and receive alerts when the integration whit these services don't work as expected. For example y need to receive an email if we stopped receiving new leads in hubspot, or if our zappier zap stopped triggering.
What are the best tool for this?

https://redd.it/lcexp5
@r_devops

deploying project using Travis Ci and AWS EC2 through ssh

I have a laravel project deployed on AWS ec2 instance and I want to use Travis as CI/CD service but I want to write the .travis.yml script after_success to connect to ec2 through ssh and go to the project directory and pull the project using a git repository. so how can I do this

https://redd.it/lcej59
@r_devops

Once you identify a code issue, what do you do in order to help developers debug/replicate the issue? Do they look at the issue in production?

Title says it all

https://redd.it/ld1s06
@r_devops

automating terraform

Does anyone have reference to a fully working model of terraform CI/CD.

Looking to identify best practice in automation.
would like to see,

\- security testing

\- code quality

\- unit tests

\- module publishing

Would be great to hear peoples checklists for pipelines too

https://redd.it/lccm1z
@r_devops

Thoughts on Google IT Automation with Python certification program on Coursera

So I'm a former English teacher with a Master's degree in Composition and Rhetoric but for several reasons I want to make a career change. It's not really safe to teach right now because of the pandemic and I'm high risk for complications since I'm pregnant. My husband is killing himself working 2 jobs to make up for my lost income and frankly my income as a teacher was crap. So I wanted to learn a skill where I could make a good salary so my husband doesn't have to work so hard. And dangit I'd like to be capable of earning the kind of salary my husband does (even if it's just 81 cents for every dollar he makes lol). Anyway the fact of the matter is that I'm home full time so there isn't any reason I shouldn't be learning something new in the meantime.

Anyway, over the last year I've dabbled in C#, Go, Unity and Python but haven't quite stuck with anything just yet.

Today I signed up for two certification courses on Coursera. The Google IT Automation with Python specialization and the Google IT support specialization since I have zero background in IT.

Since my only real experience with tech is watching my husband go from IT to Senior Dev in 3 years, I'm not exactly sure what kinds of jobs to aim for when I feel ready to start applying for work. I had seen that a lot of the instructors in my courses were System Admins so I had asked my husband if that's where I should start and he said that I could do something like Cloud DevOps. So I've been doing some research and it seems that one does not simply apply for a devops position. It sounds as if I would need to start in an entry level IT role and work my up into one.

So is my understanding correct? If not could anyone fill in the gaps of what I'm missing? And does anyone have insight in certification programs like the ones I've started? Is it worth the 50-100 dollars a month?

I should add that I have a severe case of ADHD and can't medicate right now because I'm pregnant. So I thought that the structure and community that's built into these courses would be beneficial for my learning needs.

https://redd.it/lc6qt8
@r_devops

Build & deploy individual apps from a monorepo

# My Proposed Monorepo Architecture

Monorepo architecture pic

This layout is what I've come up with from reading through tons of articles and being convinced that Nx suits many of our needs where I work. The layout works fine for a company of this size because we need opinionated architecture to keep things running smoothly.

## Goal

I want to achieve isolated development, testing, and deployment while using a monorepo.

I have a few concerns with common workflows that I'm not sure how to approach just yet.

## Questions

Case 1: I want to work only on homepage, how can I avoid having to have the entire mono repo locally?

- I read about how Microsoft solved for this, but we don't have access to Azure DevOps at the moment and I wonder if this isn't solvable in some other way. Also, the open source tool they built requires a platform compatible with it, such as Azure DevOps.
- Are Git subtrees a solve for this? Would my monorepo look like a collection of child repos (subtrees) that, in reality, live in their own repos, and are just declared here for build/deploy reasons? This seems to ruin all the benefits of having a mono repo in the first place.

Case 2: I want to deploy changes I made to the homepage app without having to build & deploy every app in the monorepo. How?
- Is Jenkins able to use Webpack to solve for this and build & deploy only specified apps or changed apps? Even if all of it is in a single repo?

This may be a simple case of not knowing what to search for, but federated modules and monorepos are also relatively new.

https://redd.it/lc6ovb
@r_devops

My first month of being a DevOps freelancer

Me sharing the story about my first month of being a freelance DevOps Engineer. Feel free to reach out if you are thinking about doing the same.

https://brennerm.github.io/posts/first-month-of-self-employment.html

https://redd.it/ld9eg6
@r_devops

Having trouble deploying SSIS from cloud to on prem

So I am new to devops and I have been playing around with trying to duplicate the existing deployment system that I built myself. I got the SSRS to deploy using a powershell script, however I can't seem to get SSIS deploy to work. I have setup my releases pipeline and my deployment group fine, however it seems that the ispac file needs to reside on the actual server I am deploying on? I have noticed that when I build the project in azure using an agent task, it creates the ispac file in the temporary working folder but the Deployment group job cannot seem to access that file. So I have tried copying the ispac file using the file copy to windows task, but it can't see my server's unc path. Any help would greatly be appreciated. Thank you.

https://redd.it/ld95hi
@r_devops

SFTP -> MySQL ETL Service

I have a task that requires grabbing customer csv files from a SFTP server, transforming it, and loading it into a MySQL table.

Looking for suggestions of services out there. Something like stitcherdata but I need to output into a MySQL table.

Not looking to custom code this, we currently do this but we are to the point where we'd like to offload this to a service where it is their core competency.

https://redd.it/ldhhuu
@r_devops

Beginner Question, Are there any useful tutorials on deploying simple web apps (that use both client and server) hosted on the Google app engine

I am completely new to the creation of a web apps, and only have a limited amount of experience making small programs in python and java on my local drive.

The webpage needs to pull in stock data from APIs, based on the users selection (keyed into a text box) perform a computation on it using a python script, and display the output of the computation on the webpage.

I am aware that I need client side and server side scripts, from what I have learned so far I need Javascript on the front end and Python on the back end, but thats as far as my knowledge goes, it sounds basic but I'm suffering from an information overload and wondering if someone could point towards some resources on make a web app that can be deployed on GAE for the above criteria?

Its small, its only going to be used by 1 or 2 people at a time, so the simplest solution with least overhead would be the best!


Again really sorry for the beginner question.

https://redd.it/ldihea
@r_devops

Managing health checks at scale

We used health checks to monitor cron jobs, but it didn't scale, as we had to manually register each job. So I adapted a script from the healthchecks.io maintainer to automatically register the job, and it's been working great.

The article: https://dev.to/15five/managing-health-checks-at-scale-4731

The script: https://gist.github.com/caleb15/1a817ef5e58e8a8caf65190cff33806e#file-healthcheck-sh

Enjoy!

https://redd.it/lddc1u
@r_devops

anyone here have experience with slalom build or simply being a devops engineer/consultant at a consulting firm? Trying to understand the pros and cons

hoping for some insight into slalom build, but otherwise just doing devops in a consulting firm, whether as an engineer or consultant

https://redd.it/ldpqnc
@r_devops

Kubernetes cluster diagram

Hi, I'm a system engineer junior, and I'm on my first big project with cloud and k8s. Nowadays I practice those on my local computer, or in the free tier.

I Haven't seen that big clusters, as our production cluster with a lot of namespaces and \~50 pods and services in each other namespace. I want to draw a diagram explained that cluster. I have a tool: draw.io.

&#x200B;

My question is; Are there any tips and tricks or good practices to draw a huge cluster like this? Do I have to draw all different pods and services and deployments? Can anyone show me something like this?

https://redd.it/ld83z1
@r_devops

Flow to send mail when workitem is assigned

Hello i am trying to create a flow which will send a mail if workitem is assigned to any users. The mail will be sent to the one who created the work item or user which mail about the issue

https://redd.it/ld481u
@r_devops

How to create alerts in ELK

Hi there,

I am aware of x-pack edition of elastic stack comes with the alerting option ? Does the open source edition of the stack ELK provides the option for alerts ? If yes, how we configure alerts in the ELK stack ?

Thanks

https://redd.it/ld3lad
@r_devops