Is a CI/CD pipeline the right solution for developing Terraform/Ansible code?

I'm in a bit of a hole here, and it's kind of getting deeper.

The backstory is that I work remotely for an MSP, our "DevOps team" consists of 2 people. Me and a fresh college grad with some AWS and Python experience, but not on the core infrastructure tools we actually use daily (Terraform, Packer, Ansible).

Our role is to create and maintain reference architectures for AWS using said tools. The usual pieces to be spun up would be VPCs, subnets, TGWs, Active Directory, Certificate Authority (Microsoft), SIEM (More logging since we have a separate anti-virus product), VPN, Jira, Vulnerability Scanner, etc. Whatever common pieces a client environment might need.

We have compliance requirements, so there's added overhead in hardening things like RHEL and Windows to CIS standards. STIG might have also been name dropped but not maintained.

As for me, my background is with Windows and VMWare with almost no prior experience with anything else like Linux, Cybersecurity, AWS, or Network Engineering/Admin.

​

In the past, people were utilizing 1000+ lines of User Data scripts to configure instances in ways it was never meant to do. A deployment typically involved running Terraform, sitting on your hands for 20-45 minutes, then logging into each individual instance and hoping everything was configured right. Recently, I was able to refactor a bulk of that into Ansible code so that deploying a SIEM would take 30 minutes instead of 120 man-hours.

The particular stress factor here is that my coworker is constantly tasked to write one-off Python scripts for other teams. So, if there's a major vulnerability in Software A because it hasn't been updated in over a year, I'm the only one who can fix it.

There's also a threat of an increasing and absolutely unsustainable workload:

Can we support GCP? Can we add Ubuntu, Debian, and AL2 to our AMI builds and get CIS hardening for those too? Can we add CIS hardening for PostGres/SQL/MySQL/NoSQL/MongoDB and every other flavor of database that GCP offers? Can we get CIS updated from v1.0.0 to v2.0.0 for RHEL8? Oh, don't forget to update from CIS v.1.0.0 for RHEL7 too because someone decided we still need to be using that. We might also want to add Azure too. We also need STIG.

It's a tad much for one person.

​

I've been eyeballing actually having a CI/CD deployment pipeline just because I feel like at least some of that can be addressed by it. In my organization, there are plenty of people familiar with the automation tools as well as other clouds. Typically, they'll even provide whatever fixes, changes, or updates they want. The 2 problems I see are that 1) I'm the only one held responsible for maintaining a central codebase and 2) the changes have to be tested and verified (e.g. whatever new software or CIS hardening still results in a working application install).

I used Jenkins before to push Terraform/Ansible code to my homelab. I feel like the workflow can be more manageable if I can tap into resources within the organization (other Engineers) and people can quickly test changes by making a Pull Request and seeing the results recorded in GitHub.

https://redd.it/zerehl
@r_devops

Anyone have experience deploying Atlantis via Helm on EKS?

So, I'm new to Kubernetes and Helm in general and this is my first real app deployment. Now, for the life of me, I cannot seem to get Atlantis running.

Context: completely private EKS cluster using VPC endpoints for all AWS API calls, VPC CNI, security groups for pods, deploying ALBs via Ingress, and secrets with ASM.

All of the AWS-level infrastructure seems to be deploying right. The ALB gets created on Helm install, the security groups gets applied to the pod, etc.

However, few things:

- pods get deployed and in Running state but never move to Ready

- the only warning/error event on the pod is the readiness and liveness probes fail with connection refused.

- if I remove the probes, no warning or errors but the pod keeps restarting, never enters ready, and eventually gets loopback error

- reviewing logs and events haven't been fruitful for me yet

- I have spun up a BusyBox pod in same namespace to test connectivity to Atlantis pod. Ping works but wget/nc on the designated ip:port (4141) does not work.

- security groups between kubelet and pods currently allow all traffic.

- I can exec into the Atlantis pod until it errors out

Any help or ideas?

https://redd.it/zeshn3
@r_devops

Job Hunting Toronto

Hey r/devops! Just wanted to get your opinion on the job market in Toronto, or Canada, or anywhere as of currently and into the next couple of months. Not sure if I should cross post this in r/askTO...

Some background:

I have postsecondary education. I have experience working for a VERY small company (basically me as the only developer) for \~1 year developing an MVP with flutter (lots of trial and error but eventually succeeded), revising their site, some development with Unity, and eventually the project was scrapped due to funding and I was out of work. I haven't worked for \~10 months due to serious family and personal reasons, but during that time when I had downtime I built a Flask app that had a postgres database pulling API data from Tripadvisor. I hosted the app on the AWS EC2 with a centos image. I learned to build a pipeline using tools such as: terraform, anisible, nginx, github actions, postgres, docker, docker-compose, grafana and prometheus, certbot. It was fully functional with a domain and I recently just took it down.

I can do the work, and I enjoy learning a lot but I'm not too sure how employable the skills I have are. I want to transition into Devops/SRE work but every job posting I see are looking for X amount of years of IT/Devops. I'm very early in my career and I'm not sure what else I need to learn to be more employable.

To get ready to interview in the next bit I'm not even sure how to approach it. Should I focus on building a pipeline again and understanding everything, should I do leetcode questions?

I know this is a lot but I would really appreciate any insight or direction. Thanks!

https://redd.it/zepj1t
@r_devops

ssh into vm's

Hi everyone my organization uses gcp and I need to ssh into vm's frequently. The problem is that there are so many of them and I need to copy ip every time.

I have made some bash functions for few of those to ease myself but I can't do that for all of them.

Is there any solution for this problem ?

https://redd.it/zeuqqk
@r_devops

How is feature deployment conflict handled in general?

Just curious, what if a company has 5 features to release at same night - and they could all conflict each other, who organize and foresees that, what methods are available?

https://redd.it/zeoxjm
@r_devops

Devops with A cots application

Anyone have experience in this realm? We've built up infrastructure-as-code and pipeline around a COTS app (JBoss/Java stack) but we struggle with silos of communication between us and the outside development team. It's a pretty niche application and we could have a better relationship with the vendor

https://redd.it/zeh43y
@r_devops

If you were the single AWS/DevOps engineer for a startup, how complex would you go?

So, joined a startup, great team, great potential, got two app dev dudes who are churning out some absolutely beautiful UI and functionality. The code needs some place to run, and we're going with AWS. I've been hired as the only AWS guy to help sort this out, and we need to start with AWS account / AWS Organization creation and build up from nothing.

How complex would you go in this situation? An argument could be made for, "just dump it in Heroku and go" for our team size, which is fair, but for various long-winded reasons, AWS is our ideal choice.

At first I was like, "alright let's make a Tooling Account for our pipelines," "let's get a Terraform Cloud account," and then the reality hit that: this is all on me to manage. When I started mapping out an OU structure for AWS Organizations like I would for a cloud environment supported by a "cloud team," but then I started wondering, "maybe some approach that is 'less perfect', but still 'good enough' is better..."

I need to find a balance of: "one dude can get this all up and running without going insane" and still "easy to manage/navigate."

https://redd.it/zeysbq
@r_devops

Wrote a blog on automating building Golden AMIs to make best use of developer's time

Modern cloud deployment success depends on immutable infrastructure and Packer an open source tool is here to help us in achieving the immutable infrastructure by creating golden images from a single source of configuration.


Packer helps in the faster deployment cycle, supports multi-cloud providers like AWS, GCP, Azure, etc., and provides scalability. I wrote a blog on how to automate building Golden AMIs with Packer, Ansible & CodeBuild. Do give it a read and share your feedback.


Disclaimer: It is posted on my company's website.

https://redd.it/zeyysd
@r_devops

I made a curated DevOps Engineer job board

Here it is: **https://www.devopsjobs.xyz/**

In my own search, I noticed there are no high quality job boards for DevOps engineers. There are plenty of targeted boards for frontend/full stack/backend engineers, but it's pretty noisy for DevOps.

I'd love to get some feedback. Ideally it could grow into a repository of curated DevOps roles with top tier companies and it would be possible to search by particular skills.

I'm also open to suggestions for companies/roles you think should be on this list :)

https://redd.it/zf0lfe
@r_devops

Security releases: Prometheus 2.40.4 and 2.37.4 are out

Security releases: Prometheus 2.40.4 and 2.37.4 are out.

Those releases fix a bug that would enable an attacker with access to the bcrypted passwords to bypass Prometheus' built-in basic auth mechanism.

Advisory: https://github.com/prometheus/prometheus/security/advisories/GHSA-4v48-4q5m-8vx4
Announce: https://groups.google.com/g/prometheus-announce/c/qVv\_BpMJdnM
OSS-Security: https://www.openwall.com/lists/oss-security/2022/11/29/1

https://redd.it/zf3lrl
@r_devops

Spacelift - opnions?

Have you used space-lift and what is your opinion of it in an enterprise production environment?

We are considering doing a POC not yet sure if it's worth it.

Any challenges, feature gaps, or pros that you have seen?

https://spacelift.io

Thanks!

https://redd.it/zf4lpw
@r_devops

Resigning from a role of being "the guy"

I joined a company that was a startup which got acquired within 2 months of me joining. When I came aboard, things were a mess. They had installation scripts for their product for 4 different distros of linux, code was poorly optimized, the documentation was non-existent so I pretty much the read the deploy scripts to learn the architecture. As I did that, I dockerized the application to make it easy to develop locally. The guy in charge had a habit of editing code locally and pushing it to test servers to test it there and would do stuff like print/exit on the server side instead of debugging... the whole thing has been a shit show. I learned that when they initially launched that the scalability of the app was non-existent.

With the new dockerized components(API, UI, various services), container orchestration/scaling via ECS and so on, I found myself to be the only one capable of actually managing the operations side of things. I begged them to hire someone with AWS experience and to let me let me spend time documenting it because I came in on the heels of them doing an MVP with very little code or infrastructure documented. I figured they just needed to be shown how to do this stuff and they'd take to it. To the contrary... I was now "the guy" and as long as I was willing to sacrifice to keep things going, they were happy to let me do it. I stayed way longer than I should've and came to the conclusion that I'd leave during my Thanksgiving "vacation" because I spent quite a bit of time helping them.

I put in my notice and now documentation is SUPER IMPORTANT. Oddly, the documentation they want is not actual documentation but like a cheatsheet of commands to run when X, Y or Z happens.

Hopefully they learn from this and document properly in the future. I'm documenting things that existed over a year before I joined the company. My lesson learned from this is that I shouldn't have done any of it. I should've left as soon as I saw the shit show.

Lastly, the president of the company set up a meeting with me yesterday to counter offer for me to stay. When I declined, he brought up the fact that there was a "non-compete" in place and that he would be contacting my new employer to inform them of the non-compete. The new employer is a very large Fortune 100 company and I'll be filling an entirely different role. Before this happened, I was open to helping my former coworkers occasionally on my personal time because I had a feeling of survivor's guilt, but after that threat, the company itself can fornicate itself.

TLDR: I joined a shitshow, made some modest improvements to operations/deployment/etc, turned into a slave for the company since they were happy with me just being THE GUY, then they kinda threatened me when I resigned.

https://redd.it/zf6t2j
@r_devops

How do you chose which log collector to choose - Fluentd, fluentbit, Logstash, etc

There are a few log collectors out there - Fluentd, fluentbit, Logstash are the more popular oned . When setting up your log collection pipeline how do you choose which log collector should you choose?

Anyone has made such choices before, would love to understand how you decided

https://redd.it/zf9k5e
@r_devops

Any new years resolutions? Or... how did you get out of your 'normal' and start your real DevOps journey?



Some seconds of introspective thinking ago I thought about how and when (or if :( ) I can get out of the loop of "write pipeline > test pipeline > fix pipeline >> yey, it finally works >> oh I could improve it > repeat".
Like some of you; I do this for around 30 devs... - alone.
I mean, it plays with my "building things gud" itch, but I don't see myself becoming any happier, more satisfied or with more shiny things at the end of the month accepting this kinda... eh... 'slavery'?.
It became my new normal some time ago and I literally have no real time at work to 'change things'.
Oh yeah, K8 gud and it might at least massively improve our resource allocation problems and solve our jenkins agent bondage, but yeah, much to do at the end of the year and it's one more thing on my "you actually MUST do this" list.

So... a nice, simple question to any of you who got out of this shit:

HOW THE FUCK?

https://redd.it/zf6hjz
@r_devops

Does anybody else find api calls to invalidate cloudfront cache flaky?

I seem to get sporadic errors submitting an invalidate cache requests via awscli. I really don’t think it’s me. It happens in several different networks, local and GitHub actions pipelines as well. Is this a known issue, does anybody else run into this problem?

https://redd.it/zfg7n2
@r_devops

Effort levels as a student

I'm currently a sophomore at University and I'm very interested in pursuing DevOps as a career. I landed an internship this summer but don't have much experience with Kubernetes, Docker, Terraform, AWS, YAML, ruby on rails, or Linux. Assuming burnout is no issue, would it be a good idea to start learning about these technologies throughout the next school semester prior to my internship?


I ask because I was once offered the advice that "it's best not to set the standard too high during my internships". In retrospect, the only rational explanation for such advice is that starting slow is preferable since rapid improvement is a very desirable metric in it's own right. Maybe someone else has heard or offered this advice before?

https://redd.it/zfhged
@r_devops

How do you integrate a static security analysis tool into the CI/CD pipeline

I looked at PrimeJS, which is the only static security analysis tool for javascript and it can't be integrated into a CI/CD pipeline, because you need to run the script through the browser. Is there a way to automate the check instead of being forced to run the scripts one by one? JSLint is a static analysis tool, but it doesn't check for security flaws.

https://redd.it/zfkc4s
@r_devops

DevOps or cloud engineer

Hello, I am a fresher graduate in BS information technology, and got a bit confused, I worked as Technical support for about 3 years and now I am working on getting my AWS cloud practitioner certificate, so I can apply for a cloud engineer role and then I can go for a DevOps role, but some of my colleagues advised me that I should learn DevOps immediately, but I got confuses, so please what should I do

https://redd.it/zf07vd
@r_devops

Confused about database migration tools and proprietary software

I'm currently working on a personal project that I may want to monetize in the future. I'm very confused about Flyway and Liquibase. It seems like, if you are using their library as part of your business, you are required to pay them a licensing fee.

Is this true? And if so, what is a free alternative that allows me to run SQL migrations in proprietary software?

https://redd.it/zfc46s
@r_devops

I think my manager lied. I want to get into devops. I dont trust him. Toxic work culture

This might be a long one.

Tldr : Manager put us on pip and claimed he didn't know what it was. He lied. Hr told us all managers did. I think he shot down my devops chance.

Me and my colleague joined over a year ago as apprentices in it support.

In comes our new manager. More harsher and less lenient. Me and my colleague do 80% of the work. Set ups, tickets. You name it. And everyone in our company saw this.

We work extremely hard. The months went on and it was annual review time
We both got put on PIP as an apprentice. Reason why? We don't have any certs yet ( we're apprentices halfway through our course) and don't do enough. The things he said made us feel unvalued, unworthy and sht. He literally said " if I had the power you wouldn't be part of my team, but you're apprentices".

.. I told my manager I want to get into devops after my course ends. He seemed happy and said he would speak to HR. I spoke to head of hr. And they had a meeting with my manager. My manager told me that HR said their isn't any roles available (my company easily creates new roles for many people) it's a startup.

He suggested I dedicate Fridays into learning devops and shadowing those from that field. He was meant to speak to our upper line manager about it. Its been 3 weeks and he hasn't at all. Now I'm doubting everything he's said about what was said in the meeting.

As that went by, we caught him out on a handful of lies. 1 of the biggest one was when he said to us he didn't mean to put us on improvement needed because he didn't know what PIP is. He claimed he didn't know that PIP is the next step closer to getting sacked. He's been doing IT support for years. We spoke to someone from HR and they told us all managers are told about what PIP

We got put off PIP and rewarded the rest of our bonus.


We assume he said this because he wants us to stay and regrets treating us like sht

Does anyone have any advice? I'm planning on leaving and getting into a devops bootcamp. My progression in this company is clearly going nowhere.

https://redd.it/zfbbuz
@r_devops

How to Unfederate a WAN Federation Over Mesh Gateways (Consul K8s)

Hi all, I currently have 2 separate KE private clusters.
One for the primary DC, and one for the secondary DC.
My setup is as per the following hashicorp article: https:|/ developer.hashicorp.com/consul/docs/k8s/deployment-configurations/multi-cluster/kubernetes

However, when I go to use consul-k8s to uninstall consul on my secondary DC. I start to see errors that the primary
DC can no longer connect to the secondary DC.

This makes sense, as Consul has been uninstalled. However, after reinstalling and rejoining the WAN federation on my secondary DC. The errors continue in my primary DC, and the secondary DC starts to show errors saying it can no longer find the primary DC.

Is there a correct way to unfederate DC's/k8s clusters that are in a WAN federation over mesh gateways?

I'm imagining that scaling this up must be tremendously hard, as if you install something incorrectly, and you potentially had 50 secondary DC's. These DC's would all start to show errors as soon as one DC was removed.

This seems like and incredibly common use case, so l am assuming I am incorrectly using consul. However, I haven't been able to find any answers online.

Thanks.

https://redd.it/zfqr7t
@r_devops