What do you use to detect/be alerted changes to Kubernetes deployments?

Recently some people have unfortunately misused their access to the cluster, and ran some commands on one cluster instead of the other. The problem is that they have actual need for the permissions, so I'm trying to see how to get notified when someone changes something on the prod clusters for example.

I tried searching, but all I could find are solutions that require their own deployments/maintenance. Is there a lightweight method of detecting the changes?

https://redd.it/11l1uib
@r_devops

Service mesh users… a little help, maybe?

A few days back, I posted a Google form here.

I wanted to know a few things about DevOps/SREs who use Istio service mesh.

Specifically, I wanted to know their challenges and wishes to make it easier for them to use Istio.

A few replies popped up when I posted it...

One was quick to tell me that I'm running a free survey... well, that was a given.

The other 2 replies straight-up mocked me… but that's okay…

I don't expect everyone I see on the internet to be kind to me (although I try my best to be so).

See, the thing is, I'm a marketing guy.

I wanted to run a survey and validate the pain points, because I believe in starting from those who'd use the product.

My plan is to bring all the insights I get from the survey and give them to the Product team…

So they can create something meaningful and useful.

"I see… why not try survey platforms then?" you may ask.

There are 2 reasons:

1/ we don’t have enough budget (I bet you saw that one coming :D). And spoiler alert: nobody asked me to do this.

2/ I’d like it to be more targeted and get filled up by the actual users

Even if I had the money, I’d still run the survey here and give coupons or something to the participants.

For some reason, I trust Reddit communities…

Reddit is where I come for research, especially when I’m writing a blog…

And it’s always worth the time and effort.

(I've another account I use to actively participate in some communities. Not dev related, since I don't have enough ex's: experience & expertise.)

So… let me ask you a favor this time…


If you use Istio/any other service mesh, would you be kind enough to fill out a form?

It has only 4 questions, out of which 2 are multiple-choice ones…

Plus, your contact info is optional to submit the form.

I assure you that it wouldn’t take more than 2 minutes of your time… but it will be of great help for a stealth-mode startup like ours.

Last but not least…

3 great souls have already filled up the form… fill it up ASAP to join them (Just kidding :D)

Hope you'd fill it.

Good day, fellow human!

(If this post gets traction, I’ll wait to analyze the responses and then add the form below if they’re positive.)

https://redd.it/11l3vab
@r_devops

How do you keep track of your "Update debt" ?

A bit of context, I'm using ArgoCD to manage 3 clusters. We set auto-upgrade on the minor version of many tools and some of them are locked to a specific version.

My problem is that sometimes the chart version does not follow the same release lifecycle as the application version. (Like Prometheus-community chart, which release a major version of the chart for a minor of the app).

So How do you keep track of the last version of an app available versus what you have in your cluster?

I would like to avoid maintaining excel sheets manually updated every month :-)

https://redd.it/11l4246
@r_devops

Cannot get an Azure file share to mount in container instances

Trying to get a multi container instance up into ACI, wordpress, mysql, phpmyadmin. Everything works locally. I'm using Azure file shares for persistent data. Followed the documentation [here](https://docs.docker.com/cloud/aci-integration/), also checked my script against this [one](https://www.reddit.com/r/AZURE/comments/s6819x/azure_fileshare_volume_for_docker_compose/) and some others from googling and searching here and in the [r/docker](https://www.reddit.com/r/docker/) sub. Containers get created but cannot start because the mounts are missing. the mounts show up when I check the properties of the container but the wordpress and mysql both complain about missing data.

I have a storage account with 3 shares (because you cannot have duplicate names). Data folder for mysql and my database I am importing into mysql. The last share is my existing wordpress content.

version: '3.8'
services:
msmdb:
image: mysql
restart: always
environment:
MYSQL_ROOT_PASSWORD: somepassword
MYSQL_DATABASE: dbname
MYSQL_USER: dbuser
MYSQL_PASSWORD: anotherpassword
volumes:
- msmdata2:/initdb.d/mikdb.sql:/docker-entrypoint-initdb.d/init.sql
- msmdata3:/data:/var/lib/mysql
command: --default-authentication-plugin=mysql_native_password
container_name: mikdb

msmphpmyadmin:
image: phpmyadmin
restart: always
environment:
PMA_ARBITRARY: 1
PMA_HOST: msmdb
UPLOAD_LIMIT: 300M
container_name: mikphpadmin

msmwordpress:
depends_on:
- msmdb
image: wordpress
restart: always
links:
- msmdb:mysql
volumes:
- msmdata:/wwwroot:/var/www/html
environment:
WORDPRESS_DB_HOST: msmdb:3306
WORDPRESS_DB_USER: root
WORDPRESS_DB_PASSWORD: somepassword
WORDPRESS_DB_NAME: dbname
container_name: mikwordpress

volumes:
msmdata:
driver: azure_file
driver_opts:
share_name: mikwwwroot
storage_account_name: saname
msmdata2:
driver: azure_file
driver_opts:
share_name: mikdb
storage_account_name: saname
msmdata3:
driver: azure_file
driver_opts:
share_name: mikdbdata
storage_account_name: saname

https://redd.it/11l2yq4
@r_devops

How to quickly learn/understand the system architecture of any given application?

When you join a new project or switch jobs… how do you bring yourself up to speed with the new application and its inner workings. I feel the best (and most accurate) way is to observe the flow of traffic. How the requests come in and how they get routed to back-end.

https://redd.it/11l8wfa
@r_devops

Proof-of-Concept: Pass environment variables as docker secrets during runtime in your container with Docker Compose v2

Hi all,

I was going through some changelogs as well as a release blog post from Docker about their Compose v2 features and happened to stumble upon an interesting feature where you can let environment variables be passed into your container as a docker secrets i.e. instead of passing it as an env var you can pass the value at runtime as a secrets file.

Here the Proof-of-Concept repo

This mitigates the case where credentials as env vars are visible when performing docker inspect or docker compose exec <service> env

https://redd.it/11lc061
@r_devops

Strange Sonarcloud error message ?

Hi all, so to give a quick overview of the situation I am trying to run a code check using sonarcloud/sonarscanner in my Amazon EC2 server but keep getting the error saying "main component does not belong to specified organization" .What does this error message actually mean ?

https://redd.it/11l9wu0
@r_devops

Can one domain have two server locations?

I am using DigitalOcean with users in North America and Australia. My server is located in Canada. Is there a way I can add a second Australia server and redirect Australian users to the Australian server?

​

Or am I able to use a CDN or other third part service/software to help with this?

​

This is with the intention to increase the website load speed for Australian users.

https://redd.it/11l9uvg
@r_devops

Do you manage runbooks for operations and incident management?

Dear DevOps, I’m an indie developer developing a product to help DevOps engineers and software engineers generate runbooks and manage them up-to-date easily.

I would like to know if your company manages runbooks.

If you do,

What is the main purpose of runbooks?
Would you please share the runbook examples you have?

If you don’t,

Have you ever tried managing runbooks? Then what makes you stop using them?
How do you keep knowledge related to operations and incident management?

I wish to contribute to the DevOps community and industry, and your comments would be very helpful.

https://redd.it/11lg6r3
@r_devops

Best Enterprise engineering blogs

Some enterprises publish interesting blogs posts with their approaches of solving issues at large scale.

For example, this one from Atlassian about their CI/CD migration.

https://www.atlassian.com/engineering/how-we-migrated-complex-ci-cd-workflows-to-bitbucket-pipelines

And also enjoyed this post-incident post related with their 10-day partial outage

https://www.atlassian.com/engineering/post-incident-review-april-2022-outage

Another company that shares posts related to DevOps is Meta. https://engineering.fb.com/category/core-data/

​

What other large enterprises share their knowledge related with DevOps?

Share your favorite ones in the comments.

https://redd.it/11l9ag1
@r_devops

EKS , ALB , Route 53 help!


Hi , new on DevOps and AWS and im confused and maybe someone could give advice.
I followed the AWS Docs
So..

1.I created EKS Cluster
2.Deployed the ALB Controller /Cert-mng
3.Deployed my APP and Services ALB / ingress etc
4.Deployed mongodb as statefulset with an EBS Volume
5.Cluster autoscaler vertical

So it showed in the load balancers console as active without any target group.. BUT my questions are .

1.Is this the right method should i now create the Route53 ns record to point my subdomain to the ALB and should i worry about the DNS of ALB that changes , or how should i point Subdomain to ALB?

2.Or should i create from the console a ALB with a Target Group of my EC2 Node instances ?

3.When i deployed my app there are shown some Classic Load Balancers even it is deprecated from August 2021/22 IDK ?

4.My node group has an Auto-Scaling group , what happens with the EBS if my instances go up/down with it attach automatically with a instance or will it remain detached .

I see a lot of tutorials people implement different methods and i worry if it will not work correctly .

https://redd.it/11kxmtd
@r_devops

Storing build/deployment metadata in NoSQL

Many of our apps rely on key/value pairs during build / deployment phase.

These are stored in text format, as files, with source code.

Does it make sense to move all such metadata items, key/value pairs into a NoSQL database and implement webservices for persistence and retrieval ?

This is unstructured data sitting on files, moving it to a NoSQL DB, what is the benefit ?

Concern is:

1) Reliable system that is always up to serve key/value pairs for build/deployment

2) Over(ab)use pattern, since data is non-structured it is best to have it files. Otherwise a persistence layer has to be validated / developed/modified for every new artifact that dev team wants to create in a sprint.

Is this a Pro ?

1) Persistence API may help in enforcing some rules vs. free-form edits on a text file by development



Please suggest if this is something you would entertain in your project.

We could also have a mixed model, some info is in files (for mature, standard key/value pairs who need persistence validation) and some items (just started developing code/concept) may be in files.

https://redd.it/11i01nr
@r_devops

RabbitMq consumer not processing messages

We’re having a situation in one of our datacenters in which for some reason one of the rabbitmq consumers stops consuming. Consequently, the queue reaches well over 1000 and one of our app’s components stops functioning. Rabbitmq is installed as a package in the VM, while this component runs as a container in the same VM. Currently the workaround is to just restart rabbitmq, after which the dead consumer suddenly springs to life and starts consuming again. Anyway, what could be causing this issue? I've checked the app's logs and those of rabbitmq, all look normal, no errors reported.

https://redd.it/11lo4fg
@r_devops

Need help with a few topics

Hi everyone! I am a young and upcoming DevOps engineer, and there is a conference at my company, for which I want to participate in. The topic is to talk about new and upcoming accelerating technology in DevOps and cloud. I want to speak on an idea and have a demo on the same, can you guys help me come up with any topic that I can speak about? Would be great if it is DevOps related.
The last time I spoke on a green field implementation of CI CD on azure devops with the help of Infrastructure as code and talked about serverless infrastructure.
Would be great if someone can help me come up with some latest trends in devops or cloud that I can talk about

https://redd.it/11lqlgr
@r_devops

I have a big doubt, how to start studying.

I'm studying front but it's not something I really want. I really wanted to work as a devops, I need to be full stack to start studying and understanding tools like terraform, grafana, kubernetes…? Or is there another starting point?

https://redd.it/11lr3wp
@r_devops

Do we still need to subject our work to source code analysis if we will use no-coding in our DevOps work?

Do we still need to subject our work to source code analysis if we will use no-coding in our DevOps work?

https://redd.it/11ls1jx
@r_devops

For Small to Medium startups: Don’t Use Kubernetes Please!

https://thetechtrailblazer.blog/2023/02/14/dont-use-kubernetes-please/

https://redd.it/11ltkis
@r_devops

PSA datadog outage

I have a few cronjobs to post metrics and events to datadog, and they've been failing for the past four hours, now I know why :(
https://status.datadoghq.com/

https://redd.it/11luq9r
@r_devops

Specific routing not working, see my config

See here

So, as you can see I have a specific routing policy for three labels up, all those labels are added in their corresponding Alerts, and Continue matching subsequent sibling nodes is enabled on this specific policy so it can move and process the nested policy, which also configured to grab alerts by a label. Nothing is coming to my teams, but If I leave the matcher empty I get all the alerts.

https://redd.it/11lteqv
@r_devops

Solution Inventory for processes we develop

My team is responsible for developing RPA processes, Integrations, etc.
We are wanting some sort of Solution Inventory (or whatever you would call it) to document details for each process we create/maintain. We would want to document the process, type, what it does, primary support, business SME, support Tier Rating, Active/Decommissioned, etc.

Do you utilize anything like this and if so what do you use?
I don't think there is a proper piece of Azure DevOps for this. (We use Wiki for Process documentation/ReadMe) and our company has an App Inventory in ServiceNow so we could potentially make our own there for our Solutions we develop.

https://redd.it/11lx0v0
@r_devops

CICD dashboard?

We have corporate standards to have standardize CI/cd.
That runs out of gitlab or azure devops.
Where the source code is scanned by sonar.
And the binaries are scanned by veracode.

Is anyone aware of a dashboard that can aggregate data from sonar/veracode/gitlab/ado to show the status of apps across all these sources?

Or am I just going to have to build it?

https://redd.it/11lz9p0
@r_devops