LGTM Stack VS Google Cloud Operations Suite

Hello! I was wondering if anyone had any experience using either of these. Right now I have a project with a company to essentially improve the log management they use. Its a large enterprise level company but the team itself and the application they use is for internal staff, and it creates around 80-100GB of logs per week. Its hosted on a Kubernetes cluster.

They're currently using Google Cloud Operations Suite with FluentBit as the log shipper, where logs are sent to Cloud Logging. Metrics are monitored with Prometheus and there's no tracing. Alerts are also dealt with through Google Alerts.

I essentially wanted to implement the LGTM stack considering this has very good integration with Kubernetes running in microservices mode - I can configure tracing through Tempo and OpenTelemetry and also set up metrics through Prometheus for an observability stack showing logs, metrics and traces in Grafana.

However after a lot of research I still can't quite figure out whether this implementation would actually improve anything on thier end. There's no real information on Loki/lgtm stack vs GC Operations suite and I don't know if there would be any big differences in the cost/speed/resources/performance/etc. Is Loki better than Google Cloud Logging at what it does? Are Grafana Alerts better than Google Alerts? Are there alternatives I can use instead? Its a big company so the actual costs of the additional resources really don't matter as long as the solution works.

Thank you for any advice you can give me on this!

https://redd.it/1cig0ad
@r_devops

Generating IaC with drag-and-drop interface

Hello everyone, a couple of months ago, I wrote here to ask for your opinion on the tool I've developed, which allows you to generate IaC from a drag-and-drop interface. I've implemented several suggestions I received, including extending the number of components (the tool now covers all AWS RDS offerings except Oracle), adding VPC endpoint support, and improving architecture validation.

It would be great if you could check it out and maybe suggest some more features it's missing: https://app.archformation.com/

https://redd.it/1cihhdm
@r_devops

Help with alertmanager & webex

Hello colleagues,

does anyone have experience with migration of alertmanager alerts to webex teams? Currently we are in transition from slack to webex (don't ask me why) and we are migrating all of the slack alerts/notifications to webex. This is current configuration (relevant part of it) of alertmanager:


....
receivers:
- name: default
- name: alertswebex
webexconfigs:
- apiurl: 'https://webexapis.com/v1/messages'
roomid: '..............'
sendresolved: false
httpconfig:
proxyurl: ..............
authorization:
type: 'Bearer'
credentials: '..............'
message: |-
{{ if .Alerts }}
{{ range .Alerts }}
"**[{{ .Status | upper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] Event Notification**\n\n**Severity:** {{ .Labels.severity }}\n**Alert:** {{ .Annotations.summary }}\n**Message:** {{ .Annotations.message }}\n**Graph:** [Graph URL]({{ .GeneratorURL }})\n**Dashboard:** [Dashboard URL]({{ .Annotations.dashboardurl }})\n**Details:**\n{{ range .Labels.SortedPairs }} • **{{ .Name }}:** {{ .Value }}\n{{ end }}"
{{ end }}
{{ end }}
....

But the bad part is that we receive 400 error from alertmanager:

msg="Notify for alerts failed" numalerts=2 err="alertswebex/webex[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 400: {\"message\":\"One of the following must be non-empty: text, file, or meetingId\",\"errors\":[{\"description\":\"One of the following must be non-empty: text, file, or meetingId\"}],\"trackingId\":\"ROUTERGW......\"}"

The connection works, as the simple messages are sent, however these "real" messages are dropped. We also thought about using webhook_configs, but the payload can't be modified (without proxy in the middle).

Anyone with experience with this issue? Thanks

https://redd.it/1cij91j
@r_devops

Automate All The Things DevOps project now with GitHub Actions

As suggested by someone from this community, I've moved the pipelines on Automate All The Things from Azure Devops to GitHub Actions.

I didn’t know GitHub Actions was free when the repo is public. This makes it SOO much easier to get started with the project, so thank you stranger who suggested this!

https://github.com/tferrari92/automate-all-the-things

The Azure DevOps version is still there in its own branch.

Any other feedback or suggestions are always welcomed! There's always room for improvement.

Also.. Nirvana Edition with Backstage.io is coming soon!

https://redd.it/1ciiqie
@r_devops

Specializing within DevOps

There really is too much to know these days, what areas are there to specialize in?

My thoughts:

Kubernetes - I can see why some engineers love it. An awesome paradigm at the base layer and so much interesting built on top of it.

Observability - almost a science in itself and plenty to get into (or related to) be it monitoring, alerting, analytics, service management.

Platform management - building out a consumable platform, kinda like being a developer for developers.

Architect - the problem I have with this is developers are going to have their own software architects doing system design that the may overlap already with the infra side. Also many expect engineers to have software architects skills anyway. So where does that leave the cloud/DevOps architect? I feel there is not much mileage in this path.

Any others? As each year passes the more I think it is not a good idea to stay in the middle as a generalist and time to pick a path.

https://redd.it/1cikm6n
@r_devops

Can i use Prometheus and Grafana to build a localized cluster monitoring system?

I manage a computing clusters and want to monitor them locally. Never tried setting up a monitoring system on them before.

My idea is to setup Prometheus on all servers so i can export the data to Grafana, running everything locally.

I’ve tried using Netdata and it worked beautifully, i want the monitoring to be secure and netdata doesn’t cut it. Hence this solution.

Have you worked on anything like this in the past and what do you recommend?

https://redd.it/1ciod7f
@r_devops

Programmatically deploy apps to k8s cluster with ArgoCD+Argo Workflows

I'm using Argo workflows+ArgoCD, and I want to automatically deploy many apps app from git repos to my cluster. Am i supposed to just commit new deployment manifests to the git repo every time i want to deploy a new app?

Like,

1. Build image and push it to a registry with a workflow.

2. Generate a new manifests that deploys the built image

3. Push manifest to git repo

I've been googling for two days, but all I see are examples on how to update image tags for already deployed images. The closest I've found is this repository but they're directly applying the manifests.

https://redd.it/1ciptux
@r_devops

Deployment scenario for whole solution

We are developing whole solution backend, frontend, datalayer.

Everything is dependent from eachother. We can not deploy new backend version if frontend is not ready. We can not deploy new database schema if backend is not ready.

Each part of solution has own code repository. How to approach to the deployment?

Should I create separate repositorium with CD pipeline?
In that repo I will keep version of each part. When new versions are ready, I will update configuration with new versions and run CD pipeline?


How do you approach to that?

https://redd.it/1cim4bm
@r_devops

Shocking Azure bill

Hey there, which hosting platform do you use or would recommend?

https://redd.it/1cisern
@r_devops

How do you deploy your containerized Laravel apps to a dedicated server?

I am considering using containers for my Laravel app on my production dedicated server to be able to easily limit resources used by each service (the PHP app, the MariaDB database, etc.). However, I am not sure how to do this in a proper and secure way. I currently have limited knowledge in Docker and am curious how you handle this because I couldn't find any good tutorials online on the CD part specifically. I was thinking of using a solution like Deployer to get a zero-downtime atomic deployments but am not sure how this would work with Docker containers. I've also heard of Laravel Sail used for containerized development environment, but am not sure if it could be used in production.

https://redd.it/1citguz
@r_devops

Anyone Using Pulumi?

I’ve been exploring the CDK space recently and started looking more into Pulumi. I have read some absolute nightmares of the migration from Terraform, but I’m curious if anyone has had a pleasant experience? I was intrigued by the YAML abilities as we provide a lot of self service modules for developers. Would love to hear everyone’s thoughts

https://redd.it/1ciymgv
@r_devops

Any good CI:CD tools specifically for supporting Docker pipelines?

Wondering does something like this exist:

-> Self-hostable

-> Open-source

-> NOT Docker (the pipeline manager and server itself!) but:

-> Capable of deploying and managing containers running on Docker

Use-case:

Deploying Docker-based open-source software between a staging and production environments, both of which (of course) have Docker.

Things like Portainer are fantastic but ... aren't pipeline managers, of course. Looking for the extra component.

https://redd.it/1ciobx0
@r_devops

Devops skils set

I am managing a team of 4 platform engineers(devops) and we take care of CI platform for 1000 odd engineers. We use jenkins and have a custom pipeline library which generates a dynamic build pipeline based on user input. I have been working with this platform and team for 6 years now. Before that I have SDET experience with mobile SDKs. So my skill sets are python, java, (some experience building nodejs apps), jenkins, managing SAAS platform like JFrog and Github, Sonarqube, Docker, Kubernetes, Terraform, etc. What else I need to learn and prepare for job change? My job is less mainly managing CI platform and I don't have experience on CD platform or cloud administration. Nor I have any experience on logging and monitoring stack. Do I need to learn those(not just in theory)? If yes, do I learn with hands-on experience? I have also not worked on a platform at a larger scale. Would that be a problem? Please help a clueless fellow out. This is the first time I'll be looking for a job change after moving to this role and have no idea what's required for such roles in market. TIA.

https://redd.it/1cj5eql
@r_devops

Managing Terraform modules in GitHub monorepo, deployment and versioning questions

Lately I've been looking into ways to streamline my organizations various repos that contain Terraform and have a few questions:

For a bit of background we have 2 environments, 1 production AWS organization and 1 development AWS organization. All our development, sandboxes and testing happens within the dev AWS org.

My questions relate to a repo that is planned to be structured as follows, our modules are entire solutions with many resources in each rather than a module per resource type:

├── envs
│   ├── dev
│   │   ├── environment.tfvars
│   │   ├── main.tf
│   │   ├── outputs.tf
│   │   └── variables.tf
│   ├── prod
│   │   ├── environment.tfvars
│   │   ├── main.tf
│   │   ├── outputs.tf
│   │   └── variables.tf
└── modules
├── module_1
│   ├── v1.0.0
│   │ ├── main.tf
│   ├── v1.1.0
│   ├── main.tf
├── module_2
│   ├── main.tf
│   └── variables.tf
└── module_3
│   ├── main.tf
│   └── variables.tf

Onto the questions..

When using a monorepo like the above for managing Terraform modules, how do you go about best managing versioning? My current thoughts are to either have duplicated folders with the version in the name (like what is shown in the above folder structure under module\_1), or to instead use GitHub tagged releases (benefit much less code duplication).

To keep development fast, I'm thinking of allowing the dev env to be built from feature branches that don't require a pull request. With this way of working in mind, which of the following is best?:

* Merge to main at the end of each day and create a new tagged release (this could result in dozens of tags over a sprint).
* Merge to main at the end of each day and only tag a new release once development has completed.
* Develop out of the feature branch until work is complete before merging and tagging (this could mean a feature branch sticks around for a long time).


Apologies if these questions are basic, this is a very different way of working to what we typically do. We currently have seperate repos for environments (full code duplication between prod and dev) and work out of long running branches. I'm trying to move away from this to prevent our constant battle with drift between environments.

https://redd.it/1cj6t7w
@r_devops

Greenmask database anonymization tool release v0.1.12

A [new release](https://github.com/GreenmaskIO/greenmask/releases/tag/v0.1.12)

Introduces improvements and bug fixes.

[https://github.com/GreenmaskIO/greenmask](https://github.com/GreenmaskIO/greenmask)

This release introduces improvements and bug fixes

# Changes

* Fixed transformer parameters decoding/encoding issue
* Fixed TOC entries merge behavior when the data section is empty - important when you create a dump witout schema
* Fixed integration tests for S3 storage

If you are not familiar yet with Greenmask - it is a Database anonymization tool that brings wide anonymization functionalities and techniques. Check out the [Playground page](https://greenmask.io/playground/) to get started

https://redd.it/1cj6njg
@r_devops

Bitbucket Self-Hosting

Is there any chance that Bitbucket would be deprecating self-hosting in a near future?

https://redd.it/1cj9hdo
@r_devops

LFG, Llama3 cli helper

https://github.com/bjarneo/LFG

https://redd.it/1cja5hl
@r_devops

Personalised CV like website


Hello,

I've been contemplating creating a small, personalized serverless website to showcase my skills, professional experience, and interesting use cases. I'm curious if anyone has already developed something similar and would be willing to share their experience and any tips.

https://redd.it/1cj6hsq
@r_devops

What courses do you absolutely swear by?

Any really really highly recommended courses? Can be anything devops or IT

My company has used Udemy for a long time, but I've been in IT for awhile so never used it. On a curious glance, I looked at a Linux course on udemy last week, I've been using Linux for years so didn't imagine I'd learn a bunch.
It was amazing! So much interesting stuff I'd never come across before.

https://redd.it/1cjdaa9
@r_devops

Alternatives to rundeck?

As title says. What is your cron job interface? Something to allow service desk agents to run complex scripts.

https://redd.it/1cjgb72
@r_devops

Suggestions for ML model deployment

Hey folks,

Seeking urgent advice regarding my ML model deployed on Hugging Face. I'm looking to deploy it into AKS and implement CI/CD. Any suggestions on how to go about this? Open to any other approaches as well. Thanks in advance!

https://redd.it/1cji3sb
@r_devops