Patterns for deploying application infrastructure in ArgoCD based GitOps setup

So hello dear YAML engineers, I am currently working on a project where I am facing some dilemmas, in how to handle the continous delivery of applications in my setup. To be more precise I am unsure of the best way to handle the deployment of infrastructure which is ancilliary to application, such as databases, redis caches and similar. More details about setup and more concrete questions in section below.

​

Current tech-stack is basically following:

\- AWS fully in cloud

\- Kubernetes

\- Terraform for infrastructure

\- ArgoCD + Argo Rollouts for deployments

\- Jenkins for building and triggering stuff

\- Low level infrastructure such as networking, DNS setup, security groups, EKS cluster provisoning, logging and supporting functions are taken care of by separate platform team. We basically get a fully functional EKS cluster and AWS account, and are free to hack away at it, but only for purposes of application. We collaborate with platform team in case that we need some changes on networking ( which is basically never ), but we are otherwise free to provision anything and everything we need for our applications and we are free to install what we want in the clusters made available to us.

​

Current setup is that we have three repositories like following:

\- One repository is for application source code, build scripts, tests and etc. Plain and simple.

\- Second repository is for declarative description of resources, which we need to have deployed by Argo in an App-of-Apps pattern ( here app-of-apps is needed since we operate the same application in multiple different configurations )

\- Last is our infrastructure repository, where we manage our Terraform code, with one folder per cluster.

On the infrastructure side we are currently only provisoning application-related infrastructure, namely RDS instances, Elasticache for Redis, DynamoDB tables and Elasticsearch clusters from time to time.

My questions are following:

\- Would it make sense to keep the infrastructure code togehter with application code? If yes, then where would I provide the values to my Terraform modules, which differ on per-environment basis? Concretely if have nonproductive and productive environment, which are based on same TF module ( which should now live with application source code), but are supplied with different values for variables, where would I place these values and where would I specify the version of the module I want to have for which of the environments?

\- Does it make sense to treat deployment of application-related infrastructure as part of the continuous delivery pipeline? I would have something like a presync hook for ArgoCD, where Terraform resources are applied before application is deployed. Alternative to this would be having fixed "infrastructure environments" ( consisting only of ancilliary services used by application ) and then deploying application "into these environments", which would mean simply configuring the application with correct URLs.

I would also be interessted if there is tooling concetrated on deploying the application together with application-relevant infrastructure, while excluding the "lower-level" infrastructure. I have heard of KubeVela, but I have read very few experience reports online.

​

I am thrilled to hear your 2 cents on this.

https://redd.it/y0glfi
@r_devops

What are the biggest security challenges for CI/CD in 2022?

I am currently thinking about how to improve our build environment with regards to "I don't trust the admin", but "I want the admin to maintain my nodes". What are your challenges regarding security & CI/CD in 2022?

Maybe even challenges when adopting a SaaS CI/CD provider and rolling/maintaining your own platform instead?

View Poll

https://redd.it/y0cre3
@r_devops

Skipping GitHub Actions jobs while maintaining branch protection rules that require them

How to work around some quirks and limitations of GitHub Actions to skip CI jobs that aren't necessary in certain scenarios, without breaking branch protection rules that normally require those CI jobs to succeed:

https://blog.pantsbuild.org/skipping-github-actions-jobs-without-breaking-branch-protection/

https://redd.it/y0ku2a
@r_devops

Gradle and Software composition analysis

Why do SCA tools require a gradle build before scanning whereas for other projects just scanning package.json is sufficient without build?

https://redd.it/y0n45s
@r_devops

Best conventions to go to 2023

What conventions would you consider the best to attend in 2023? Looking for generalized conventions dealing with all facets of DevOps.

https://redd.it/y0nxgs
@r_devops

Diagram-as-code for cloud architecture

For visualizing cloud architecture and generating a diagram using a small DSL.

https://docs.tryeraser.com/docs/examples

https://redd.it/y0k2gz
@r_devops

loops in packer - dynamic sources/builds

Hi,

Looking for a bit of advice on how best to approach this.

Initially I had a packer file that runs ansible against a docker image, pushing the new image to a repo.

I then needed to do this for a number of images, so I added a source for each one and created a separate post processor to tag the specific images before pushing them.

The problem I now have is that the list of images is changing regularly, I have written a python file to loop over the different images/tags and pass the details into the initial packer file, running the builds one at a time but I was wondering if there is a way to do this natively in packer a d run the builds in parallel.

The list of images would be something like Ubuntu:focal, Ubuntu:bionic, Ubuntu:jammy though it would likely be longer and a mixture of images.

The python script works, just wondering if there is a better way.

Thanks

https://redd.it/y0swq2
@r_devops

Need help wrapping my head around massively bloated legacy application

I can't give too much detail as to not give away where I work since it's quite a large company.

But basically I'm on a very small team supporting somewhere between 15-20 applications. We're always so swamped - I've gotten better with dealing with the stress though. And learned a lot.

However there is one application that is just scary and it's partially broken and has been for months. My team lead made it but was gone for some time and it was changed a lot. It's an old web app that basically does a lot of HR processing.

We have multiple tickets for it but it's just been ignored until recently they are really on us to fix it. My team lead has begun training me on the app and I have many tickets and enhancements assigned to me for it.

While there is indeed a dev, test and prod version. I'm not sure that it's the same.

The documentation is a little sparse and there are dozens of stored procedures, batch jobs, and it's using SSIS as well which I've not used till now.

Basically all of the work is being handed down to me and I don't know if it's because I'm burnt out or if I just know I'm not capable of fixing this application but I just don't know where to start and am overwhelmed. Looking for some sort of guidance from someone on how to approach massive legacy projects like this that are broken and when documentation is sparse and there is just so much data and processing being done?

https://redd.it/y0s6t4
@r_devops

DevOps - Dev & Ops, Build & Maintain vs Ops/Customer Support - Prod Access

I know everyone has their own interpretation of DevOps, NoOps, etc.

Lets see how the following theory can work "Developers (a team) build and maintain a product." This team is cross functional but conceptually all are the same 'role' but have different strengths and have the same access otherwise we're just grouping Developers and Operation contributors together in a team.

Need to deploy a new service, the team spins up a new server.

Server runs out of memory and hangs, the team restarts it.

Team then builds new automation which they deploy to automatically restart it if it runs out of memory again.

Team gets tired of doing manual deploys and builds automated deploy, but still needs access to do the deploy since well they are running it and can modify it and what is in it.

All of that is focused on engineering and technology, not business and I think we all agree that falls under DevOps and access that the team needs to have which is already pretty high access since they built the deploy pipeline, the server, and everything that gets put onto it, meaning even if they don't have access to they prod keys, they can deploy scripts that do.

Lets say we're a movie theater and customer calls up our customer services/support line and complains that they reserved seat 2 but their virtual ticket says seat 20. Hopefully there is a tool that lets CS reassign the seat but we're a startup and haven't built it yet. Movie is later today so we don't have time to build a tool.

CS calls us up and needs help, do we have access to modify the ticket mysql database directly to change the seat. `UPDATE tickets SET seat = 2 WHERE id = ?;`

If we don't, who does?

In theory no one does, there is a tool for everything and never any bugs get through our process so there is never need for one off fixes but lets face it, that'll never happen.

Does your answer change if this is a bank or hospital instead of a movie theater?

Does your answer change if the organization is 3000 people vs 30 people?

https://redd.it/xzlhyk
@r_devops

Exited from consulting to a DevOps role in the industry, missing some aspects of consulting, how to get the best of both worlds?

Hello!

After spending 2 years in consulting right after my CS degree (ACN, cloud domain), I've accepted an industry role as a DevOps Engineer, that came with a title/rank bump, huge salary bump, and is fully remote - my new employer is in another country and doesn't have a branch in mine.

While I really enjoy the higher level of responsibility that came with the role and nature of the work - we're involved in day to day infrastructure operations and tech decisions need to be made carefully to not screw anything over, ability to gain much more breadth in my role, get very strong technical skills, getting to know the nuances of the industry of my current employer, and being around tech experts, I really miss some aspects of consulting and I'm also worried my career trajectory could be potentially slowed down.

I really loved the constant rotation of projects - some focused on really niche aspects of the cloud, variety of tasks (yeah even the ppts...), hybrid nature of the work, exposure to execs, clients PoC's/workshops, and sometimes building things from scratch. I think that also helped speed up my career trajectory a ton.

I'd like to stay in my current role for a bit, because the pay is great, and I really want to gain more breadth in my domain, but I cannot see myself growing into a ultra senior tech role in the industry/become a tech lead.

My ultimate goal is to combine tech with soft skills and get into management or solutions architecture.

In consulting, there was a ton of networking opportunities, management skill building just happened organically, because that is the requirement for higher levels.
I saw some of my supervisors just exit to former clients or using those networks at Director level. In a technical industry role, I feel that those opportunities are kind of limited or slowed down and the higher you go the more pigeonholed you get into pure tech roles.

I also don't want a pure remote job forever - I really miss the professional face to face interactions and also feel it's harder to build business relationships without meeting in person.

Thinking of ways to get the best of both worlds for now and next steps to take to stay on track.

I've had an idea to go to tech conferences/meetups and try to talk to other departments at my company and try to get into some BD/sales work to substitute for the aspect i liked about consulting.

What else would you recommend me to do to stay on track?
I guess going back to consulting doesn't make sense for now since I would still be at consultant level, not gaining tech depth, and lower pay.

Thanks for your insights!

https://redd.it/xzj9dt
@r_devops

DevOps Career

Hello everyone, I am trying to become a DevOps. I have an engineering degree, a bit of programming knowledge, Cloud Practitioner, Solutions Architect Associate, and Developer Associate certificates from AWS. What do you think I need to learn to be able to find a job as a DevOps?

https://redd.it/xzm3qt
@r_devops

What Certifications to do ?

Hi I recently passed AWS SAA exam and now thinking of doing another one in order to get into devops role maybe a junior DevOps engineer. My background is 5+ yrs in IT QA engineer. While keeping my job I am thinking of preparing and giving these certifications:


1. CKA

2. Terraform associate

3. Docker associate


please advise me if its a right thing to do

https://redd.it/y11l7q
@r_devops

How to learn DevOps being broke

Hello folks!

Not sure if anyone already asked this, but today I was talking with a friend and she's trying to find her path into SRE positions, but the openings always ask to have knowledge (and some experience) around some of the big cloud providers.

​

As we're from a third-world country (hello from Argentina) paying services like AWS/GCP and even DO can be pretty hard for someone that lives with the exact amount to survive.

​

So here is my question, is there any way to learn how to use these cloud providers in a cheap way?

https://redd.it/xz8lh4
@r_devops

Is devops really fancy name for sysadmin in many companies?

Is it true?

https://redd.it/y14d3z
@r_devops

Question: Docsets for SQL Server and other proprietary programs

Hi, i am looking for docsets for non open software. Docsets are offline documentations for tools like zeal or dash. Does anyone hast a tipp or source? Thank you very much!

https://redd.it/y15faz
@r_devops

DevOps Improvement Plan

I have fought hard for several years to implement DevOps practices in my organization but we are now stuck. Although configurations for our systems and applications are being managed by Ansible (AWX) and git, many of the playbooks and roles haven’t been developed in an idempotent way. Sysadmins basically took their old scripts and converted to Ansible using the command module. To make matters worse, there is little management and oversight of our git master branch. This has led to changes being made directly in that branch where over 100+ playbooks are now stored. Sprawl has taken place. Wading through the new complexity has slowed everything down. Proper testing has taken a backseat to throwing darts at a wall. We are troubleshooting more than ever in production.

Since our DevOps initiative is in danger of being scrapped for the “old way” of doing things I am attempting a draft overhaul plan for our current messy approach. It must be streamlined, scalable, and self-service. Everything must be measured so leadership can not only see the value, they can see problems as they develop. Sysadmins should be given zero room to continue bad habits. Has anyone ever attempted this before?

https://redd.it/y1610c
@r_devops

If a docker container is in deployment how do you exchange that to another newer docker container seamlessly (from user's perspective)?

If a docker container is in deployment how do you exchange that to another newer docker container seamlessly (from user's perspective)?

https://redd.it/y1c6g3
@r_devops

learning path: how to continue

Im kinda new in IT, mostly in DevOps. Since last summer, Im working as DevOps engineer, before that, I was QA engineer for 2 years.
I use a lot of things, like zabbix, VMware, Azure, Cisco IOS, (python, bash, powershell scripting) etc, but zero docker and container. You would say, its SysAdmin.

I wanna make plan to the future.
-learn Docker/k8s and become DevOps engineer without any specialisation
-learn security, like AppSec, and go for DevSecOps, or just security engineer
-learn Azure and become Azure DevOps engineer

Which one would you choose, and why?

I started to learn CKA, AZ900, CCNA, Comptia sec+. They are all basic stuffs

https://redd.it/y1i08h
@r_devops

New job with complex infrastructure - How do you approach?

I have been offered a (contract) job today, with a large multinational investment bank. (I suppose it would be hard to find an investment bank that isn't multinational! - but anyway) that is looking to modernise their Devops processes

They have a large complex infrastructure, and a considerable number of applications. Part of my job will be to decipher a lot of it and come up with strategies to achieve their goals.

At this point I don't know how organised / chaotic it will be, but I'm curious what approach you lot take when landing on such a site.

Do you have a process to bring yourself up to speed quickly? Any hints / tips / etc

(My previous experience has been mostly with startups so I've built everything that existed)

https://redd.it/y1ieze
@r_devops

AWS re:invent 2022 - what to see

What are your plans for this years reinvent in case you are going or attending in person ? Are there any must attend sessions or any other recommendations would be much appreciated

https://redd.it/y1m1z0
@r_devops

"2022" CI/CD design

Let's say you are in a situation where your CI pipeline unit tests the code, has docker containers built and you have helm charts built and published.

How would you design the rest of the pipeline to make sure your CD pipeline can deploy to multiple environments, including production as soon as a developer merges to master.

It will be interesting to see how the landscape has changed over the last few years.

https://redd.it/y1nftk
@r_devops