Jenkins jobs logging solution needed

Hi All,
I have around 200 Jenkins jobs running for a bunch of projects. Not all of them are deployment jobs only a very few & this query is not regarding them. I have other 170+ jobs which are created to run certain functionalities within few applications. They're like cron jobs (or batch jobs) you can say.

So these batch jobs are like taking file input from various SFTP servers & then executing them one by one.

Issue is that these jobs give success message even if one of the file from any SFTP server is not fetched. Let's say each job is fetching 10 files from different SFTP's and they miss out on 1 file but successfully execute 9 of those so it's still a success. Now it's not possible for me to go into console log of each job & see which of these executed all 10 file; if I'd do that then it will be very time consuming.

Is there any solution for cases like this where I can have a dashboard or anything which collects the logs from all specified jobs & I can check them all in as minimum time as possible? I was thinking something like ELK?

Thanks in advance.

https://redd.it/1geso87
@r_devops

Why do we need automated regression testing in CICD piplines?

Somke tests,integration,end to end. I am trying to grasp the whole picture. Why do we need regression testing? How it should be implemented? What are the pros and cons? Blog post or books on this,would be welcome.

https://redd.it/1gevfvz
@r_devops

Secure deployment on client's system

Hi,
I have an application which runs on multiple EC2 instances and around 10 Dockers running in that. It process some sensitive data.
Now what is the best method to deploy this on a client's AWS account? I need to protect my logic, sources code and some other data.
Since it's client's account, they can login to EC2 and see the contents. How can I prevent this? What are some best Industry practices?

https://redd.it/1gewgyz
@r_devops

GitLab CI/CD Pipeline with multiple ArgoCD clusters

We're migrating our azure pipelines over to GitLab, and so far the build process is fairly straightforward.

The problem is we're using multiple instances of ArgoCD, so the same application is deployed to multiple clusters with ArgoCD set up on each cluster.

I can't find a good way to reflect the ArgoCD sync status in the pipeline, and have this 'gate' the subsequent stages of the pipeline.

The pipeline looks something like this:

install dependencies
build/ push the image to Azure ACR
write the image tag to a git repo -> webhook from the repo to ArgoCD to trigger sync
ArgoCD picks up the change to the image tag and syncs

The problem is that in the final step we've effectively 'handed off' to ArgoCD, and the only way around this I can see is to have a stage that sleeps and periodically calls each ArgoCD endpoint to get the sync status, and once all apps have synced to mark the stage as successful/failed.

This seems quite a janky workaround, and the only 'good' alternative would be to move away from Argo and have GitLab update the kubernetes manifests - this is a big change that's not really feasible for us at this time.

Does anyone have any suggestions or a similar setup?

https://redd.it/1gexuhc
@r_devops

Terraform related query

Hi everyone, I just got certified as a CKA and I am super excited. As part of my next thing to learn I am giving a thought to learn terraform. I just want your thoughts on whether I should pursue, if not what should I pursue next.

Some background
I don't have any experience with it at all. Maybe I tried something here and there but not to the extent to crack the associate exam.

If you recommend me pursuing it, please drop the links to courses that will help in learning.

Thanks

https://redd.it/1gezwdh
@r_devops

Unveiling Jaeger V2, the next major release of the popular open source distributed tracing tool

After about 7 years and over 60 minor V1 releases, Jaeger is finally reaching V2 🥳
Great seeing this distributed tracing tool that started as an internal tool at Uber evolving into a mature open source under the Cloud Native Computing Foundation (CNCF), and now undergoing re-architecture to leverage the OpenTelemetry Collector framework (another CNCF project), which promises more flexibility, performance, extensibility and ease of use.

Here are some insights from Jaeger creator Yuri Shkuro:
https://medium.com/@horovits/be612dbee774

https://redd.it/1gf2g2c
@r_devops

WebSockets load balancing

Our setup: Client browser -> Cloudflare -> AWS ALB -> Ingress NGINX -> Node.js WebSocket server deployment with multiple replicas and autoscaling.

We're struggling with WebSocket load balancing. Some pods have more connections than others. When autoscaling adds new pods, new connections are often sent to existing pods rather than the new ones, further disbalancing the average connections per pod.

Has anyone tried to load balance WebSockets?

https://redd.it/1gf1ud4
@r_devops

AI agent, Multi-Cloud Support: AWS (using SageMaker) GCP (using Gemini)
Azure (using Copilot)

I'm building an AI assistant to guide the setup of cloud resources in a secure manner. 

Example prompt Setup a production-grade AWS foundation with the following requirements :
\- VPC in us-east-1 with 3 availability zones
\- Private and public subnets- Network segmentation for different workloads
\- Implement security best practices- Enable encryption for all services
\- Setup CloudTrail for audit logging
\- Configure AWS Backup for critical resources
\- Implement WAF and Shield for protection
https://medium.com/@rasvihostings/ai-agent-to-set-up-a-foundation-in-a-public-cloud-secure-manner-4a555d8fdd84

hashtag#googlecloud hashtag#aws hashtag#azure hashtag#gemini hashtag#SageMaker hashtag#Copilot hashtag#python hashtag#terraform

https://redd.it/1gf5m0b
@r_devops

Building a devops lab (and deliberately not posting this in /r/homelab)

Using cloud services is too expensive when a business class enterprise workstation plus 64 gigs of RAM ordered from crucial comes in at pretty close to a grand and then this machine lasts for 3 years. Now granted you don't learn cloud services when you do this but there are still an awful lot of tools and techniques you can focus on. It looks like the current optiplex machiens will even do 128 gigs of RAM and it's not THAT expensive to do that.

My dilema is if I'm better off using the macbook pro as a daily driver and spinning stuff up on this machine remotely and using something like proxmox, or if I'm better off running Ubuntu on it (and a lot of containers and VMs) or using Windows as Hyper-V is a decent platform along with WSL.

Curious what others here might do in this situation.

There are a lot of pros and cons to each choice. Focusing on using the mac and just using this box as a resource has some advantages, but using the desktop with either windows or linux might be an opportunity to get a nice new dual monitor setup.

https://redd.it/1gf7fcn
@r_devops

Made a list of free DevOps learning resources that offer digital badges

This is more for you to learn the tools, gain confidence to try more complex projects. So, if you don’t know where to start, here you go:

https://github.com/catinahat85/GitGudAtCloudNative/blob/main/learning-resources/README.md

https://redd.it/1gfdq7f
@r_devops

How do you set up your active directories?

What are the active directories set up you did that allows you to have a network drive to store files and shared across with everyone securely?

https://redd.it/1gff0t9
@r_devops

Learning DevOps

Hey guys I am a student and don't have money to spend on courses so please help me with some free resources to learn DevOps.

https://redd.it/1gffp7b
@r_devops

Flatcar open source project is accepted into the Cloud Native Computing Foundation (CNCF) - a community-owned cloud native Linux OS

Pretty amazing, this is the first time the Cloud Native Computing Foundation (CNCF) has adopted an operating system distribution.

As Chris Aniszczyk rightly put it: “A secure community-owned cloud native operating system was one of the missing layers of the CNCF technology stack".

Flatcar provides a lightweight Linux OS (derived from CoreOS) specifically tailored for hosting container workloads.

Here's the CNCF announcement for more details:

https://www.cncf.io/blog/2024/10/29/flatcar-brings-container-linux-to-the-cncf-incubator/



https://redd.it/1gfgqh0
@r_devops

Need Help: Using Red Hat Developer Subscription on Azure VM

Hi everyone,

I’m currently trying to use a Red Hat Developer subscription on an Azure VM, but I’m facing some challenges. Here’s the situation:

1. **Azure RHEL Subscription**: I created an Azure VM using the default RHEL image from the Azure Marketplace, which incurs some costs.
2. **Developer Subscription**: I want to switch to a Red Hat Developer subscription, but Azure doesn’t directly provide it.
3. **VM Creation Attempts**: I tried to create a VM using an ISO I downloaded, created a VM in VirtualBox, and then made a VHD from it. However, when I uploaded the VHD to Azure and attempted to create a VM from it, it didn’t work.

# My Questions:

* Is there a more straightforward way to use a Red Hat Developer subscription on an Azure VM?
* Can I create an Azure VM with the RHEL subscription from the Marketplace, then remove that subscription and attach the Developer subscription? If so, how would I go about doing that?
* Are there any specific steps I should follow to ensure a smooth transition from the Azure RHEL subscription to the Developer subscription?

Any guidance or suggestions would be greatly appreciated! Thank you!Hi everyone,I’m currently trying to use a Red Hat Developer subscription on an Azure VM, but I’m facing some challenges. Here’s the situation:Azure RHEL Subscription: I created an Azure VM using the default RHEL image from the Azure Marketplace, which incurs some costs.

Developer Subscription: I want to switch to a Red Hat Developer subscription, but Azure doesn’t directly provide it.

VM Creation Attempts: I tried to create a VM using an ISO I downloaded, created a VM in VirtualBox, and then made a VHD from it. However, when I uploaded the VHD to Azure and attempted to create a VM from it, it didn’t work. My Questions:Is there a more straightforward way to use a Red Hat Developer subscription on an Azure VM?

Can I create an Azure VM with the RHEL subscription from the Marketplace, then remove that subscription and attach the Developer subscription? If so, how would I go about doing that?

Are there any specific steps I should follow to ensure a smooth transition from the Azure RHEL subscription to the Developer subscription?Any guidance or suggestions would be greatly appreciated! Thank you!




https://redd.it/1gfeiob
@r_devops

What is the largest unexpected cloud bill you’ve received and what caused it?

Hey! As I wrote in the title - I’d like to hear your experiences when it comes to being charged unexpectedly high amounts of money for cloud services.

https://redd.it/1gfl6qa
@r_devops

Thoughts on crowd-sourcing on-call

Hey everyone, I'm working on a product to crowd source on-call responses.

Before I get any further with it, I'm wondering if there are people who would be interested in chatting with me about it. Would you be interested in helping respond to problems you know a lot about?

E.g. if you have some contributions to ClickHouse or have run it in production, would you be interested in responding to a page from a company having a ClickHouse outage?

Here's the service: https://pagersquad.com

You'd of course be paid by companies for being on-call and responding to outages. I figured it might be a decent way for people to get paid for knowing a ton.

https://redd.it/1gfn82s
@r_devops

Cloud or physical severs?

Do you recommend owning my own physical servers in order to host my own data and my web application or do you recommend going for a cloud solution, if so which cloud and how can I protect my self on being charged for high bills or fees I cannot pay?

https://redd.it/1gfotl3
@r_devops

Quick review of Pulumi ESC

I have been playing with Pulumi ESC since they went GA last month. Here are my quick thoughts on it:

1. It's very practical for centralizing configuration or secrets in environments that can inherit from each other
2. I like how ESC can get secrets from other stores like Azure Key Vault or 1Password. This feature makes it more than just a vault. Unfortunately, some sources like Bitwarden are not yet supported
3. I found that configuring OpenID Connect was quite challenging but once it's set up, being able to easily retrieve short cloud access tokens from an environment opens up a whole range of possibilities
4. ESC has interesting integrations with other tools (like Direnv of Terraform). I've only used the Pulumi IaC integration which is very handy
5. ESC is not completely open source (it seems only some parts like the CLI are) so you can't self-host it unless you pay a license 😕
6. The vscode extension is fairly basic but very nice to modify the environments
7. I didn't check the audit logs, but I'm sure that having environments that are auditable and versioned can be valuable for some companies

https://redd.it/1gfpnnb
@r_devops

k8s deployment workflow (GitOps, kustomize with ArgoCD). marketplace recommendations?

Hi there

I am setting up brand new deployment platform with a typical GitOps patterns (with kustomize & ArgoCD). We use GitHub Actions for the setups and noticed the marketplace for the GitHub Action: https://github.com/marketplace/actions/kustomize-github-action

I always create the bases/overlays with the proper kustomization.yaml in the folders. I never used the marketplace item before and not sure how to incorporate into the setups.

Looking for recommendations to see how community is setting up their kustomize modules (with or without marketplace items). If you used marketplace, how did you setup kubevals & other checks that's usually go under kustomize modules. 🫡

https://redd.it/1gfr1v2
@r_devops

Vybes, share what you are listening to in real time with friends

I recently published this social media that lets you share the music you're listening to (from your favorite streaming service) in real time with your friends.
You can download it here https://vybess.app/download
Let me know what you guys think

https://redd.it/1gfwdo6
@r_devops

Which job would you pick

I’m interested in hearing which job you would pick going off the compensation. Just for fun.

Job 1
Base: £50,000
Annual bonus: 9%
Pension: 4% employee 11% employer
Additional benefits: £120 per month benefit in kind for health and well-being
Company type: Corporate
Progression: level 2 of a possible 4.
Working pattern: fully remote

Job 2
Base £65,000
Annual bonus: 2-4%
Pension: 5% employee 5% employer
Additional benefits: None
Company type: Small to medium sized business
Progression: None
Working pattern: Hybrid 1 day every two weeks in office (20 mins drive)

Edit: Clarified Job 2 Hybrid working.


https://redd.it/1gfx2ab
@r_devops