Helm: Is it possible to merge base values.yaml's with stage specific values.yaml's

Apologies for the title, but I can't come up with a better one right now.

I am sitting here for hours thinking about how to restructure our repository in the best way to avoid redundancy and manual tasks as much as possible. Please note that I am pretty new to the whole kustomize world. Also, as a side note, we are using argo-cd.

That said, I was hoping it would be possible for us to use a structure like this:

./
├── base/
│ ├── app/
│ │ ├── Chart.yaml
│ │ ├── templates/
│ │ │ ├── deployment.yaml
│ │ │ ├── service.yaml
│ │ │ └── ...
│ │ └── ...
│ ├── db/
│ │ ├── Chart.yaml
│ │ ├── templates/
│ │ │ ├── deployment.yaml
│ │ │ ├── service.yaml
│ │ │ └── ...
│ │ └── ...
│ ├── app-values.yaml
│ ├── db-values.yaml
│ └── kustomization.yaml
├── overlays/
│ ├── dev/
│ │ ├── app/
│ │ │ ├── values.yaml
│ │ │ └── ...
│ │ ├── db/
│ │ │ ├── values.yaml
│ │ │ └── ...
│ │ └── kustomization.yaml
│ ├── staging/
│ │ ├── app/
│ │ │ ├── values.yaml
│ │ │ └── ...
│ │ ├── db/
│ │ │ ├── values.yaml
│ │ │ └── ...
│ │ └── kustomization.yaml
│ └── prod/
│ ├── app/
│ │ ├── values.yaml
│ │ └── ...
│ ├── db/
│ │ ├── values.yaml
│ │ └── ...
│ └── kustomization.yaml
└── ...


My plan with this is to have base values.yaml's that contain all the values that are identical for every stage, and then to merge these base values.yaml's with the ones containing the values that are unique to each stage. I was hoping I could use of inheritance and just pass multiple values.yaml's to our kustomization.yaml's, but I couldn't figure out a way that works. After hours of trial and error, my brain is tired now, and I was hoping somebody could lead me back on the right track.

Also, please consider: We do not have Internet access from our customers GKE clusters. We are pushing our own GitLab repository to the GCP and everything else happens inside their clusters. Compliance reasons...

https://redd.it/11otrnc
@r_devops

what is your ideal branching strategy?

In context, I’ve probably experienced many kinds of branching strategy:

1. The so-called ‘each environment one branch’ strategy where u might have your environment, test, dev, staging and yada yada which follows a unique branch.. this seems to be the most popular one (from where I am). It’s all fine and dandy until the team decides to patch something in the prod branch and ‘forgot’ rebase to other branches. Not a good experience in general.
2. Similarly, there’s also this ‘each version one branch’ strategy. I mean… why would anyone do that?
3. The ‘git flow’ strategy or any other variations of it where you have a so called masterpiece branch with other sub main branches like it could be dev, rc-xyz, unstable-01 and yada yada.. then you have your devs working off any of these sub branches, whether is for a feature or something.. then someone came along and decided to patch something in your masterpiece branch and another person merges one of the sub branch into another sub branch and now a team lead decided that his rc-xyz is ready and tested and shall now be merged as part of the masterpiece branch.. suddenly you have yourself a spaghetti mess of conflicts. man, I absolutely hate this strategy. I have lost count the number of times disasters happens due to this. Having to merge/fix/cherry-pick/whatever one branch to another while resolving conflicts along the way probably took a good chunk out of my lifespan.

And then comes my ideal branching strategy overall - the one and only single branch, literally strategy. Why? Because:

- everyone work off the one and only branch, main.
- it makes PR a lot more simpler and easy, we only have to review once!
- encourages making explicit and good commit (no more having 1 commit that has a bunch of other nonsense)
- encourages responsible code change (you think twice before making breaking changes)
- regressions are handled almost immediately (things breaks and affects everyone, it is handled almost immediately before it goes to our QA)
- cleaner single line of history

When I implemented this workflow and strategy to my team, it does takes a while to get used to it but I now live happier managing 1 stupidly large monorepo (it’s 2 monorepos in a monorepo that I somehow inherited), 4 frontend, 2 ios and 2 android projects with teams across continents.

But I do want to hear your thoughts and see how I could improve it further, what’s your ideal branching strategy?

https://redd.it/11oroqf
@r_devops

Seeking Help With a Career Change

I am currently a GIS Coordinator for a small utility and do consultation on the side for a variety of organizations across the US. For a better career outlook, I am wanting to formalize other skill sets so that I can market myself to different positions.

I currently am most comfortable with Python, and have a large amount of exposure to JSON and HTML. I am comfortable with a lot of networking principles and system administration, and regularly tinker with a small home lab machine and virtualization (Ubuntu).

What I am trying to figure out, is how do I get started in formalizing these skills without going back to a formal institution? I have looked into resources like Coursea, but I am not sure how valid these solutions are and/or if they are worth the costs. Does anyone have suggestions?

​

Thanks

https://redd.it/11oqg1f
@r_devops

SVB Exposure

Does anyone know which startups have exposure to Silicon Valley Bank?

https://redd.it/11oy9ue
@r_devops

Github Actions in monorepos

Not sure if this can be useful to someone, but I just released `hawk`, a little tool to help you manage `github workflows` in monorepo.

Docs can be improved but it does the job 🙃

​

https://github.com/Rawnly/hawk

https://redd.it/11p1fp4
@r_devops

DevOps Practitioners from Top Orgs - What does it take to be one ?

TLDR;
If someone is looking to target some Top-notch organizations - ( Morgan Stanley, Goldman Sachs, Adobe, jpm , etc ) what does it take to be noticed and get shortlisted from the volumes of applications these organizations receive? Also, what "x" factor do they look for in candidates?

https://redd.it/11p95vt
@r_devops

Questions about Kubernetes and Terraform

I've never used Terraform and Kubernetes and the project hasn't started yet, so if you think I should switch technologies feel free to let me know. I usually use Docker and Puppet.

​

Suppose I have 3 dedicated servers and a VPS

​

1. A Dedicated server running Postgres on Kubernetes
2. A Dedicated server running a Go server on Kubernetes
3. A Dedicated server running Python sever on Kubernetes
4. The VPS is used to monitor those servers and for admin tasks

​

I have the following questions:

​

Let's say that the Postgres server is running low on resources, does Kubernetes notify me to buy a new server?

If I need to set alarms manually to buy a new server, then what are the best practices? How do you decide when it's time to buy a new server?

Can I buy a new server and not add it to any cluster and let Kubernetes decide if it wants to add it to any of the clusters mentioned above? Can I have servers on standby and then Kubernetes either use it for Postgres or Go etc?

How to protect against DDOS? Do I run all the clusters behind Cloudflare Or Do I buy additional firewall servers and run the clusters behind the firewall? Is it easy to set up Cloudflare?

​

So essentially, we're starting a new project, and we'll need at least 3 servers and a VPS as mentioned above, the project might scale up and if it does, I want to have answers to those questions so I know how to act.

https://redd.it/11p9mns
@r_devops

Uploading SQL data to remote server

I need to upload a small business's SQL data to a live updating server.The SQL server sits on a windows 7 computer and I was wondering what was the best way to do so.I wouldn't want to expose the SQL server to the internet for security reasons and FTPS uses SSH so I would also rather not having that port exposed.I was thinking of just having a shared google drive folder that would sync the changes. How legit is it to do? and are there any better methods to do so?

https://redd.it/11pd87d
@r_devops

How much should a Platform/Cloud/"DevOps" Team be responsible for?

In my organization, it sometimes feels like my team is responsible for too many things for us to be able to do a quality job on anything. In my experience of interviewing around for different positions, it has felt this was a shared problem in most companies, whether the employees working there share my perspective on this or not.

Namely, I'm talking about the Platform, or Cloud, or "DevOps" team, who is responsible for Terraform + Kubernetes stuff. Sometimes this includes CI/CD for the services, sometimes that's a separate team. Often, there is no separate team for DBA's and all DB technologies are thrown into this team. Monitoring/Logging is sometimes partially shared with a small SRE team, sometimes it is thrown into this team.

Basically, wanting to work on the layer below the product services often has put me face to face with the risk of becoming the toss-your-problems-in-the-bucket team. I find managing container orchestration, logging and monitoring, tracing, cloud infra management, databases, message queues, endpoint security (ddos prevention etc), and even more things to be too many to manage in a team of 4-8 people no matter the size of the project unless if you are in the first year of a startup, and unless if the other teams are heavily sharing responsibility rather than 90/10 split at best, if they don't outright ignore it unless you reach out and say their service is nuking the database from orbit.

How do you handle this in your team/project? How have you found ways to work better (or worse) with this? How do you share the responsibilities with the various domain (or otherwise) product teams, specifically when it comes to getting management buy in?

https://redd.it/11pev59
@r_devops

Price / performance sweet spot for Mac Mini iOS CI/CD

Hello.

Should be a quick one. Looking for recommendations on which Mac Mini hits the right sweet spot between price and performance for iOS/Android CI/CD.

I should note that I'm not looking to spend a huge amount as although I'm a software engineer in my day job, this is just for my personal projects so I don't need the fastest build times. Main priorities will be running lint, unit tests and deploying to firebase app distribution so nothing too complex (although I'll likely be using react-native so building iOS & Android in parallel would be nice).

Couple of options around the same price:

1. Second hand M1 with 16GB RAM & 512GB storage.
2. New M2 with 8GB RAM and 256GB storage.

Upgrading the M2 mini's to 16GB RAM or 512GB storage feels like awful value for money, so in reality it becomes: is the M2 with 8GB RAM adequate or better to go back a generation and get more RAM?

M1 feels like the better option out of the two but then again it's possible the M2's might have a couple more years support & OS updates in them.

https://redd.it/11pi1fv
@r_devops

How to run a web application hosted in ec2 server on demand? I want to minimise the application cost. Example turning off automatically the application (dns) when the user is away from his computer and only run when is working. I’m not concerned if ec2 runs 24/7h.

How to run a web application hosted in ec2 server on demand? I want to minimise the
application cost.
Example turning off automatically the application (dns) when the user is away from his computer and only run when is working. I’m not concerned if ec2 runs 24/7h.

https://redd.it/11pb7aj
@r_devops

How to run a website hosted in ec2 server on demand? I want to minimise application cost. I’m not concerned if ec2 runs 24/7h.

How to run a web application hosted in ec2 server on demand? I want to minimise the
application cost.
Example turn off automatically the application (dns) when the user is away from his computer and only run when is working.
I’m not concerned if ec2 runs 24/7h.

https://redd.it/11ovcau
@r_devops

creating a namespace using ansible module

Hi guys,
i'm trying to configure namespcae for nginx from ansible like this:

\- name: create ingress-nginx namespace
kubernetes.core.k8s_info:
api_version: v1
kind: Namespace
name: ingress-nginx
state: present


but receiving the following error:
FAILED! => {"msg": "Could not find imported module support code for ansiblemodule. Looked for either AnsibleTurboModule.py or module.py"}


any idea how to overcome this?

https://redd.it/11pnaad
@r_devops

Deploying app with Podman

Can you guys suggest a good article about deploying react app with Buildah, Podman, and Skopeo

https://redd.it/11pop7e
@r_devops

How to deal with devs pushing bad code

It’s became apparent to me, that code is being pushed straight into production without being load tested.

Take for example this week.

Devs pushed code into their repo. This gets built into an image and deployed, fine. But it gets deployed to production, with no change gate and no testing.

As soon as the client starts using this new feature, all hell breaks loose. The autoscaler was set up to track CPU, this new feature uses a damn tonne of memory, which was not relayed to anyone in ops. The pods get overwhelmed, and barely evade the OOM killer, but it means the site is sluggish and barely chugs along.

Someone, please enlighten me, my management don’t seem to understand that dev and ops need to be working together.

https://redd.it/11ppp66
@r_devops

IaC Best Practices with Bicep/Github Actions?

I'm currently leading a team in a small org who has chosen Bicep as their IaC language and Github Actions for pipelines.


While I'm not a technical resource on the team, I do spend a little effort researching and understanding so I can help guide them to technically sound solutions.


I read up on Bicep best practices, and set up some repos that use a main file to call modules for deploying resources, and then I set up a pipeline to push these files into Azure. All fairly standard stuff, but what was not clear to me is how teams tend to execute their bicep against Azure? The following ideas stuck out to me:

* Executing it directly from Visual Studio, add params in dynamically at runtime via VS UI
* Maintain a master branch with a parameters.json file, changes to the bicep code itself would be merged to master, and if a deployment is needed, a feature branch is created, the parameters file is updated and pushed, then the workflow is triggered against the feature branch. Feature branch is disposed upon completion.
* Set up the workflow to trigger when a change is merged to master (likely the param file).

Before you say "use terraform" - we are not now and will not ever be a multi-cloud shop. It will be Azure only until the end of time, a decision that was not in our hands. Because of this we chose Bicep over Terraform as we felt the barrier to entry was lower, and not having to maintain a state file help keep the process simple.

**For folks using Bicep/Github - how do you deploy your templates?**

https://redd.it/11pqwql
@r_devops

Pulled into a meeting… with the customer

Just want to ask you all how common it is, and how you’d react to being pulled into a meeting, to then have a client grill you about the performance of the service.

This happened a week ago. I was asked by delivery to join a meeting discussing why a customers SaaS app was so slow.

I joined the meeting, and almost straight away I was being asked about issues. OK fair enough, we have problems, let’s fix them. As the meeting went on, I started to question who this person was. I naively thought delivery had put me in front of a colleague that I hadn’t yet met. It was not a colleague.

20 minutes in, after repeatedly stating “we need more info to take a look at this , please send us this data”, I felt myself getting more annoyed. My colleague who was also on the call, didn’t say a word.

I left the call pretty unimpressed and quite frankly pissed off.

Who else has had this happen to them?

https://redd.it/11pph5p
@r_devops

Why is NewRelic so bad?

Really, I hate it. We had datadog before, but we were enforced to use NewRelic. We are currently migrating and I have to say that I really hate it. The docs have so many dead links, it’s a nightmare. Then the NewRelic explorer crashing. A SaaS app crashing in the browser was something I have never seen before. Then this whole pixie thing. I was excited at first, but none of the queries work. Seriously. I hate NewRelic.

https://redd.it/11prydv
@r_devops

Made a CLI tool for Markdown URL format check in Github repos

Hi.

Made a tool (called GMUV) which checks brokenURL's in Markdown format for public Github repos. Output can be stored to file or as a console output. Marketplace action is available. To those who wants to see execution example - click here.

https://redd.it/11ps81u
@r_devops

Git repository tags as a source in terraform module

As a source in terraform module we're using our own github repository with specific tag. That looks something like that:

provider "aws" {
profile = "default"
region = "us-east-1"
}

module "iam" {
source = "[email protected]:path/to-repo.git?ref=0.0.8"
name = "foo"
naming = "bar"
}

I want to make changes in terraform code that is tagged as 0.0.8 so can I push it as 0.0.8 tag or is it not possible and I should bump the tag to 0.0.9 for example?
If the answer is yes (make changes, update the tag to 0.0.9 and use that as a source) what to do with other tags that we're not using yet but will be using them in the near future - update those and bump the tags?
Another question - what is the purpose of master/main branch then?

https://redd.it/11pxixu
@r_devops

Are your product teams able to operate their services effectively without your assistance?

Hello all! 🙂 I'm building a new service for DevOps, SREs, and product teams. The product is designed to help product/dev teams run their services and handle incidents with less SRE/DevOps support.

In my case, it has been difficult to guide teams on how to operate their services due to various reasons, such as a lack of cloud knowledge and permission issues.

I wonder if there are others who also struggle to enable their teams to operate their services with confidence. If you face any problems related to this, would you please share your case?

https://redd.it/11pxxxl
@r_devops