Help: GitOps+K8S via Terraform is driving me nuts

I'm honestly just frustrated.

I come from a simple CI/CD scenario where my containers are hosted on GitHub with their compose files, and the CI/CD would docker-compose up over SSH.

Now I have an infrastructure repo on GitHub that's public containing all my terraform code for my company infrastructure. Great, right? Everything managed in one repo: [https://github.com/ironPeakServices/infrastructure/](https://github.com/ironPeakServices/infrastructure/) So you have DNS managed on Cloudflare, terraform for my hosted k8s cluster, terraform for the k8s configuration and terraform for any helm packages on k8s.

It's not. If I don't have a scaleway cluster yet, the kubernetes terraform provider will error out (since it uses the kubeconfig from scaleway) and I can't deploy anything until I manually remove the kubernetes/helm stuff, run the pipeline and manually enable it again.

If I finally have a k8s cluster, if anything goes awry during the k8s configuration phase or helm installation, the pipeline will fail and my state will be borked. (Yes I have helm atomic operations enabled). I can't do anything after this since terraform will error out with things like \`namespace already exists\` or \`resource limit already exists\`, forcing me to manually remove k8s stuff out of my state file, destroy my cluster and trigger the pipeline with the k8s stuff disabled, en re-enabling it again...

I posted about these things on the terraform github page, but no help.

I really don't know how others are doing this successfully.

Refs:

[https://github.com/terraform-providers/terraform-provider-kubernetes/issues/775](https://github.com/terraform-providers/terraform-provider-kubernetes/issues/775)

[https://github.com/terraform-providers/terraform-provider-kubernetes/issues/708](https://github.com/terraform-providers/terraform-provider-kubernetes/issues/708)

[https://github.com/hashicorp/terraform/issues/24131](https://github.com/hashicorp/terraform/issues/24131)

[https://github.com/terraform-providers/terraform-provider-helm/issues/422](https://github.com/terraform-providers/terraform-provider-helm/issues/422)

https://redd.it/f9tbqu
@r_devops

From Plain Machines to Container Orchestration: A Complete Explanation

So, considering that "Kubernetes" is either a derogatory term or a holy blessing nowadays, it's a good time to figure out just what in the e-hell "Container Orchestration" is in general. Aside from the fact that most of the websites for these tools give you a nonsensical marketing definition, there's also a ton of required knowledge to even grasp the basic idea.

And so I put together a post that unifies all of the concepts required to understand container orchestration and how it works/helps. It starts with the simplest of ideas, the plain machine, and builds all the way up to clusters of virtual machines running containers. Instead of showing you the tip of the iceberg and claiming it as the whole, we dive beneath the waters and explore the full picture. Hopefully without freezing our brains out.

[From Plain Machines to Container Orchestration: A Complete Explanation](https://start.jcolemorrison.com/from-plain-machines-to-container-orchestration-a-complete-explanation/)

https://redd.it/f9vov8
@r_devops

GitHub vs GitLab

I've been using GitHub for a long time and it does almost everything I want. The big exception is CD. I've fully embraced GitHub Actions and I even use it as a CD tool for my Terraform deployments. However, the interface is decidedly CI.

GitLab, however, does have a proper pipeline CD interface. It seems like GitLab has all the features of GitHub and then some. Does that seem like an accurate statement?

​

I bring this up now because I have a meeting in a few hours with a team that wants to transition from Jira/Bitbucket to GitLab (one big reason is that our current Atlassian stack is in-house vs hosted). It sounds like they'd like to make this a standard and transition the rest of the org to GitLab as well (with my help).

​

I've always used GitHub + Other SaaS solutions to essentially accomplish what GitLab does in one package. What do ya'll think? I do know that there's a few teams with K8s deployments.....does GitLab help with that somehow?

https://redd.it/f9xfqv
@r_devops

What’s Your Favorite DevOps Interview Question?

If you are about to hire a DevOps engineer, what is that one question you will ask to test whether he/she is fit for the role?

https://redd.it/f9x2t9
@r_devops

Continuous Integration for Python Microservice

Hi everyone,

Currently a lead ML engineer that is very out of my element. I've inherited a code base that's a purely python microservice that serves ML predictions based on requests from a large internal monolithic app. The dev environment for the team is on multiple Windows 10 VMware virtual machines. I have access to a Windows 2012 VMware server. We don't have anything in place for CI/CD.

Instead of running a script to start the microservice, I was thinking of using Docker (with Docker Toolbox and enabling nested virtualization) so that a) I don't need to think about maintaining and running pip requirements.txt for upgrading packages and b) to streamline deployment in our users dev environment.

I can then use CircleCI to automate the "build" which essentially boils down to running unit tests for me upon code commit and emailing errors.
I'm only concerned with the deployment of the python microservice and nothing else.
Is this the best approach?

https://redd.it/f9tka5
@r_devops

setting path for the tools - maven, git

Should i set path manually in global config settings in jenkins or can i select install automatically option. Is there any difference, If so what difference does it make

https://redd.it/f9zy3x
@r_devops

Jobs for AWS Solutions Architect Associates

I'm studying for my AWS Solutions Architect cert and curious about what job titles to be looking for. For those who have completed this certification, what are your job titles? Thanks!

https://redd.it/f9yohp
@r_devops

Docker and GastbyJS

Hi, I am currently building a devops pipline for my school club. The projects we are working on use many different back end technologies (NodeJS, Flask, Django, etc). To deploy the back ends I am currently using Docker Images + Github Actions + Google Cloud Run. This was the best way to handel the variety of frameworks.

A new project is planning to use GatsbyJS and there aren't any good resources / dockerfiles available online. I was wondering if anyone had experience with this and could point me in the right direction. I was going to treat this similar to the NodeJS deployment with an additional build step. Thanks for the help.

https://redd.it/f9zsg0
@r_devops

Jenkins replacement

Our Jenkins jobs execute Ansible playbooks. Soon we plan to migrate some Ansible "tasks" to Terraform. Jenkins gets the job done but I can't look at groovy anymore, I don't like how Jenkins logs, I don't like it's plug-in system. What modern solution would you recommend? Thanks.

https://redd.it/f9pjxh
@r_devops

Critique my attempt at devops for python microservice deployment and best practices

Hi everyone,

Currently a lead ML engineer that is very out of my element. I've inherited a code base that's a purely python microservice that serves ML predictions based on requests from a large internal monolithic app. The dev environment for the team is on multiple Windows 10 VMware virtual machines. I have access to a Windows 2012 VMware server. We don't have anything in place for CI/CD.

Instead of running a script to start the microservice, I was thinking of using Docker (with Docker Toolbox and enabling nested virtualization) so that a) I don't need to think about maintaining and running pip requirements.txt for upgrading packages and b) to streamline deployment in our users dev environment.

I can then use CircleCI to automate the "build" which essentially boils down to running unit tests for me upon code commit and emailing errors.

I'm only concerned with the deployment of the python microservice and nothing else.

Is this the best approach?

https://redd.it/f9pr2m
@r_devops

My personal devops project. https://www.linkedin.com/pulse/how-i-am-using-bitbucket-github-heroku-my-cicd-vasu-bhardwaj/

Hi all , this was my first devops project
I used github, bitbucket and heroku.
Let me know what you think.

https://www.linkedin.com/pulse/how-i-am-using-bitbucket-github-heroku-my-cicd-vasu-bhardwaj/

https://redd.it/f9pz7i
@r_devops

I want to setup these in my desktop: Gitea (and Postgres), CI/CD, Harbor, logging/monitoring; need advice

I'm learning IaS but I don't want to spend money for cloud instances yet, I only have i5 Skylake & 16GB of RAM, it will have to suffice for now. Which is why I'm choosing Gitea instead of Gitlab, and since it's also my main dev aside from laptop, I'm aiming for container centric GitOps workflow. I've installed minikube but I'm not sure I should do it in Kubernetes yet, and my line of thinking is as long as the architecture works, I can migrate it in the future since I do want to learn Kubernetes.

My plan is

- ✅ use vagrant to build Ubuntu 18.04 VM (my desktop is hackintosh), provision docker & other dependencies,
- use docker-compose to pull
- ✅ Gitea,
- Postgres,
- Harbor,
- CI/CD (maybe Drone),
- and Prometheus/Grafana or ELK.
- sets networking so I can commit git, trigger scripts etc without ssh-ing into VM

Roughly this is what the setup looks like https://i.imgur.com/14e7yu9.png

I had a problem that I haven't find a solution for a whole day yesterday, attaching local VM folder to `$PGDATA` in PG container made the folder owned by root & caused initdb to fail. I tried the attaching read-only /etc/passwd trick mentioned in the official README but it didn't work. I'm 50-60% sure i can work around it even if I have to build a custom image.

But anyway, what I'm asking is:

1. Is this a decent/sane plan?
2. Is there a better way to do it?
3. How far am I from being junior level DevOps/SRE/whichever title?

https://redd.it/f9o4yb
@r_devops

Are AWS t3a bad?

Comparatively speaking, value for money. I've done some of my own tests compared to DO and Vultr, which score quite a bit better in performance. Even compared to other AWS EC2 offerings, it's not that good.

What's your opinion?

https://redd.it/fa7666
@r_devops

Cloud infrastructure automation for a SAAS company

I am wondering what is the strategy to use in order to create cloud resources upon subscription of a paying client for a SAAS product. Eg: custom CNAME or a blob storage.

How do you keep the state? How to validate the completion? How do you queue the creation tasks? What other points I am missing?

Thank you

https://redd.it/fa7zsq
@r_devops

Looking for feedback for our automated version increment workflow for CI

My problem that led to this workflow was that in many cases we had to manually increment versions in git. This was tedious and error prone since would frequently lead to either unexpected extra commits or collisions when multiple developers are working or both.

Therefore, I built tooling around the problem which provides version synchronization capability outside of manual developer workflow. Such new automated workflow could be used in the CI process as described [here](https://worklifenotes.com/2020/02/27/automatic-version-increments-with-reliza-hub-2-strategies/).

I would be very interested to get some feedback from community whether this is useful or not. And if not - how are you currently solving version increments?

https://redd.it/fac5cu
@r_devops

GitHub is Down ... Again ...

[https://www.githubstatus.com/](https://www.githubstatus.com/)


I'm sure other folks here are considering alternative solutions to GitHub, given the reliability issues we've been seeing since Microsoft has taken over.

I was considering a self-hosted GH Enterprise cluster, but haven't researched too much on what's needed to make that happen. What are you folks considering?

https://redd.it/fadojf
@r_devops

Software Disenchantment

A colleague shared this link in our team's off-topic chat channel and it created a buzz. I thought our community may appreciate it as well.

[https://tonsky.me/blog/disenchantment/](https://tonsky.me/blog/disenchantment/)

https://redd.it/fadz5c
@r_devops

Jenkins Deleting ssh.exe!

Any job I run on one of my windows agents deletes my C:\\Program Files\\Git\\usr\\bin folder and thus my ssh.exe so I cannot clone a repo via ssh. I am baffled...anyone have any ideas?

https://redd.it/fafgtn
@r_devops

What should I look out for as a candidate in a devops job interview ?

I've got close to 2.5 yoe in technical support, I recently interviewed at a startup and the interview was way too easy. So I started having doubts if I'm doing the right thing?
The things that were asked very basic tcp/udp difference, one simple coding question, how to design a backup systems and some questions about my current role.

https://redd.it/fafu87
@r_devops

How to grow as an SRE/DevOps Engineer

This post is more so for exploring what I can do to grow as an SRE/DevOps Engineer.
A little about me: I have a B.S. in Computer Science and have moved into a DevOps/SRE role where I am writing a decent amount of code/scripts (python, bash), rolling out a decent bit of infrastructure (chef, cloud services) and troubleshooting application/network issues. I would consider myself on the above-average side of programming but the average to below-average side of Linux knowledge. I am very comfortable making my way around the Linux OS but struggle with some of the more in-depth troubleshooting/concepts from a knowledge standpoint.
The inspiration for this all is from interviewing for SRE positions at a couple of different companies and either making the on-site, pretty much acing the coding portion and then not passing through the architectural side, or rarely not making it to the on-site. The frustration for me is that I am given little to no feedback and struggle to find where to learn/gain experience that is needed to grow (large scale systems) and ultimately pass through all the interviews.
I generally have used Linux academy to learn but am constantly looking for more places to learn from.


Any insight/perspective/guidance/encouragement would be greatly appreciated.

https://redd.it/fakrm2
@r_devops

Lack of tooling that helps understand large codebases

**How does the code base work?**

There is a lack of good in-depth tolling for understanding big codebases for architects/team leads/developers. On one side we have simple tools for test coverage and on the other side, complex and expensive products like code climate focused on velocity.

But there is a problem left unsolved in the middle. As an engineer working in a bigger team, it’s hard to get an answer about how does part of codebase work. Or why & how & in what places is this piece of the system used.

Keep in mind that you can use static analysis and “find all” to find specific usage of a function but what I am interested in is connecting concepts/modules together and showing their relationships in easy to understand way.

**It’s a new decade and I still don’t have a way to have:**

* A clickable diagram explaining all subsystems of my app and their dependencies to one another
* A simple way to see the high-level flow for a particular feature. User registration code diagram for an example
* A simple way to see which parts of code are undertested / overtested. Code coverage is a bad metric in general. It does not tell you anything about the popularity/usage of tested code
* A way to have a good understanding of what side effects will I have to think about **before** starting the work on a feature. To measure scope and time needed. (Oh shit user registrations need to be exported to intercom...)

**What general questions should you be able to answer easily?**

* I want to change the functionality of this feature for new requirements. How can I know all other subsystems interacting with the feature?
* I want to quickly get a high-level overview of major subsystems. What are the dependencies, are there any hidden connections that people tend to forget? What should I watch out for?
* How can I easily know where does the feature starts and ends? I want to know how the card payments work. Where to start?
* How much effort will be needed to refactor this feature? Are there any connections/interloop that I should be careful about?

**Real-world example**

What parts of the codebase do Stripe payment gateway touch and how is it implemented at a high level? What do I need to watch out for when migrating to Braintree?

**Can it be better?**

It took me 4 months just to grasp the major subsystems and their interactions with each other in my company. There has to be a better way.

I understand that deep and comprehensive docs could be an answer, but really how many companies will do documentation at all levels of the tech stack? Usually what I see is that you have a few people at the company who have an understanding of the architecture and you kind of have to piece it all together from them.

Why can’t we automate at least part of this process? Something simple to give to the developers, so they can get a quick understanding of the features and their relations in the whole codebase.

https://redd.it/fah0jw
@r_devops