Is your companies tech documentation also a mess?

Hey everyone, I work at a medium sized tech company and our documentation is all over the place. We use Confluence mainly, but also Gitlab readmes and Slack channels. The information is spread out and it's quite hard to find the information that you need. From reading this subreddit, it seems a lot of tech companies are in the same boat.

Has anyone used or built anything internally to help with this?

I'm thinking something like a semantic search across Confluence, Gitlab, and Slack channels. Anybody know of something that can do this?

https://redd.it/1djd7yz
@r_devops

How do you roll out components to your clusters?

I have a vague idea and would like some help detailing!

Say you need to upgrade Kong using TF and Flux
You have one AKS cluster in each region

Do you go in each cluster repo and upgrade one at a time

or

Do you go to one repo and upgrade to all simultaneously?


The problem is that there are many clusters, is there an architecture where you could just upgrade to all at once? What is the requirement there, to have applications running on infra that can handle it?

https://redd.it/1djflbb
@r_devops

Unmasking HTTP Logs: From Blind Spots to Full Visibility with Gleam and Quickwit

https://blog.kalvad.com/unmasking-http-logs-from-blind-spots-to-full-visibility-with-gleam-and-quickwit/?ref=kalvad-newsletter

The author came up with an original solution to monitor HTTP logs (NGiNX, Apache, Traefik,...) with all critical details like request and response bodies.
They use Gleam(friendly language for building type-safe systems) to build a proxy which captures all necessary details and send structured logs to a storage backend Quickwit.

What is your opininon on this approach? How do you produce detailed structured HTTP logs on your infra?






https://redd.it/1djgaa3
@r_devops

Be creative when it comes to naming environments

https://shippingbytes.com/2024/06/19/be-creative-when-it-comes-to-naming-environments/


Do you call your environment with canonicals name like prod, staging? I don't do that anymore because there is never a single prod, or dev does not stay dev for a long time. I have a different trategy

https://redd.it/1djiu4e
@r_devops

10 Books to Accelerate your Cloud Career

Not a comprehensive nor definitive list, but a list of books that have helped me personally a great deal. Interested to see if you agree, and to know which titles you would include in your top 10.

# https://medium.com/@jake.page91/10-essential-books-to-accelerate-your-cloud-career-f43f33d5e859

https://redd.it/1djjknq
@r_devops

What should be in your "interview" repo?

I haven't interviewed for a job in well over 10 years at this point so I am honestly just ramping up into what I'm going to need to prepare for. I fell into this position through being an onsite Data Center Sys Engineer, Virtualization Engineer, technically I'm in the SRE group, but do DevOps deploys and general infrastructure for our different cloud envs. I can script in bash, powershell, python, etc, but nothing that is ever going to impress anyone. I'm less a "move fast and break things" kind of person in favor of stability and making sure my team can sleep at night without outages kind of person.

I already know if they give me leetcode/hackerrank type questions I'm not going to pass. They gave us hackerrank questions at work to "review" for new hires and half of them were completely irrelevant to our day to day. I have never needed to build a hashmap from scratch since I primarily work in Terraform and Ansible which abstract that work and python has libraries for that. But the company also has a policy of "only hiring developers" for SRE and DevOps roles.

I've seen people talking about having a repo as a sort of "portfolio." I'm not trying to seed my startup or be a thought leader. I just want to do a good job at work and go home. I love learning new things so I'm not against remediating any skills gaps.

What are some standard projects I can work on doing mock ups with and but in a repo?

I prefer to work in Terraform and Ansible but I was thinking:
-bash scripts for Linux Admin
-powershell scripts for VMware Admin
-Python SDK for AWS deploys

https://redd.it/1djkm2f
@r_devops

code reviews become test plans

I have noticed more and more often that a devops code review becomes just a discussion as to how best to test the change. The most recent one was making an adjustment to a k8s network policy. Everyone looked at it and said, "looks okay to me but maybe you should test it."

This doesn't seem wrong to me. Just curious if others have seen this trend also.

https://redd.it/1djn8c1
@r_devops

Azure DevOps pipeline - anyone know how to make a change log from pipeline content?

Hello, I'm working with a project where I would like for every time a pipeline is run, that it creates a changelog update based on the contents of git.
I have seen others using pipelines go update the azure DevOps wiki, I'm wondering if anyone has had any experience with something that works?

https://redd.it/1djl8cy
@r_devops

Azure DevOps self-hosted agents and Artifacts Feed

Here is what I am trying to accomplish:

1. I have a Python package that I have published to an Azure Artifacts feed.


2. I want to install this Python package on a self-hosted Azure DevOps agent as part of a pipeline.


3. The pipeline will run some code that depends on the Python package.


To do this, I need to authenticate with the Azure Artifacts feed from the pipeline running on the self-hosted agent. Any ideas on how to go about this? I do not want to use a Personal Access Token PAT because it is linked to a user and not Group or Service account.




https://redd.it/1djpgjv
@r_devops

How to transition into a cloud based DevOps roles?

I am currently almost 2 years into working for an org with 0 exposure to cloud (the product uses cloud for data storage which is managed by the backend teams). We work on ensuring availability and making configuration management changes to servers and Jetsons on edge. I have worked on Bash and Python scripting, a lot of troubleshooting ubuntu running Cuda, Docker, Ansible, Grafana and Prometheus setups among others.

I'm trying to move out from this org for better pay but nobody's really doing similar work, and cloud native orgs are not really giving a chance for even an interview/call.

Should I rack in more experience at the same org and wait around being patient or am I going wrong somewhere?

https://redd.it/1djqs04
@r_devops

Recommend a monitoring alternative

We're spending 500k a year on datadog metrics + logging

We'd like to run our own regional kubernetes cluster where we can provide monitoring services for multiple products


Is there any stack someone could recommend that replaces the functionality of datadog but has a permissive enough license to self host (for internal use only, not providing access to tenants), and it should be able to use kubernetes to scale.

https://redd.it/1djsd94
@r_devops

Does anyone work with technical specifications document while freelancing in DevOps

Hello,

I was wondering if freelancers in DevOps do use technical specifications documents.

In French, we call it « Cahier des charges » and it’s a document that you’ll have as a template to specify each requirement and specification about what you are going to do in your task.

So this template will have some questions about the environment setup on the client’s architecture and what exactly are the results and preferences they want, then you evaluate it and decide what are the requirements and how much will it cost in total ( for you job + requirements fees )

Do you do that, if yes do you have any kind of templates you use or used before ?

Thank you in advance 🌹

https://redd.it/1djjogv
@r_devops

I Build Iudex, Low Cost Observability with OTel and AI

Hey folks, I'm the founder of Iudex, and we're building a super affordable observability platform to tackle the headaches of high log retention costs, confusing configurations, and solving log correlations using AI. We've designed Iudex to be low cost because we know how important it is not to have to think about how much data you're sending. We run completely on OTel, so if you're already using it, adding our exporter is trivial. And, we're focused on giving teams a low time to value. Our current features include: built-in log attribute filtering, keyword search, natural language search, traces, and service level metrics.

We're doing a soft launch and are looking for users who want good observability tooling but don't want to overpay for them. Join our waitlist, and as a thank you for being a beta user, you'll get 100 million logs per month for free!

https://redd.it/1dkq68c
@r_devops

Should I .gitignore everything by default?

I have started working with a team recently and the default behaviour is to git ignore everything and then add a not ignore to the .gitignore file for my content to be committed.

I have never worked like this and I am really struggling as no one can tell me why this is the pattern.

Is this common practice? Help? Please 🥲

https://redd.it/1dk6of3
@r_devops

What are your strategies when feeling overwhelmed?

I've been an SRE for a few years, thought I would shake it after a few years experience, but it still happens where I feel overwhelmed with things.

Not so much with the amount of work, which seems endless and I'm always playing catchup, but much more so with learning more and more of the technologies and it feels like I'm always behind.

https://redd.it/1dkuj2w
@r_devops

What Do Good IaC Practices Look Like?

I’m a startup founder looking to learn from experienced DevOps professionals about the best practices you follow when managing infrastructure through IaC (particularly Terraform).

I've been exploring various methods and tools myself, but I’m also particularly interested in understanding:

* How you structure your Terraform modules for scalability and maintainability
* Any tips for managing infrastructure drift
* Best practices for integrating IaC with CI/CD pipelines
* Strategies for ensuring security and compliance in IaC
* Tools and techniques you use for automated testing of IaC
* Any other tips and best practices you use on a daily basis

As a startup founder, I wish to learn and then improve our platform that eases or solves one or a couple of the problems that I’ll observe as a common pattern among all the workflows I might learn about.

My background is not in DevOps; it's in ML and Deep Learning. My startup, therefore, has been playing around with the idea to build an AI-driven Agent that enhances DevOps workflows. This Agent's capabilities would include:

* **Module Search and Documentation**: Searching for Terraform modules, pulling the right ones, populating them with the appropriate parameters for your cloud infrastructure, and presenting detailed documentation all in a clean well summarized format.
* **Configuration Parameter Retrieval**: Automatically checking & fetching configuration parameters from your cloud providers (such as AMI IDs) and populating your IaC automatically.
* **Searching through your existing IaC codebase** through natural language.
* **Drift Detection Tools and Real-time warnings.**
* **Natural Language Interaction**: Allowing users to interact with the Agent through natural language queries and plain English commands.

The speed improvement we expect to see is in having these various tasks that you as a DevOps engineer do daily but instead of diving into different pages, CLIs and windows yourself, we setup your most common pipelines which you can call easily through natural language

I recently learned how ubiquitous the use of Terraform modules for provisioning workflows is, and staying up to date with the vast number of modules and the services they render reusable that exist is unequivocally large. Code generation is the hot topic that everyone feels LLMs do well, but in reality, it's understanding and consuming large contexts (such as a product's complex cloud infrastructure).

I was wondering if such an Agent, with these capabilities, would be a tool that saves you time and energy. We’re not dead set on this being the idea. We're simply trying out things to see what clicks and what doesn't, and your feedback, criticism, & suggestions are crucial in helping us decide how to move forward!

Looking forward to your responses!

https://redd.it/1dk8uem
@r_devops

SSL cert for self hosted SAAS (Dockerised)


I set up a self-hosted version of a SAAS on a server. The software is completely dockerized including the Nginx server. I don't want to mess with the original Docker compose file because I want to be able to update it with the newer releases of the SAAS without much hassle.

The problem is since the port 80 is already being used, I cannot just install a Nginx server and set it up with a SSL. So what should I do in this case? How do I get a SSL certificate?

https://redd.it/1dkxqzx
@r_devops

Operations-as-a-service

I‘m currently in a fucky situation where we are supposed to govern, create and maintain standardized CI/CD jobs for ~700 devs, infra automation engineers, devops engineers etc. in an oldschool-ish org in the financial sector.
The scope is on-prem and in the cloud, so multiple tenants as well.

Most engineers can‘t be arsed to do shit themselves:
- they buy software which needs to run on the container platform but they have nobody to take care of the release management, they come ask us
- they ramp-up product teams with deadlines but don‘t have the staff with the necessary knowledge, they come ask us
- they don‘t have the justification for hiring a full time devops engineer because the product is relatively small, they come to us
- they don‘t care, they come to us
- we racked up massive tech debt, so shit breaks all the time
because we were trying to „enable“ engineers resistant to learning, they of course come ask us


The idea to prevent us from drowning and do our actual job would be to ramp-up a team which just takes care of these operative tasks for other teams and offer it as-a-service.


Thank you for listening to my rant


https://redd.it/1dkcjv7
@r_devops

Free vendor events

Last December my team was invited to the Datadog Holiday party. How do I get on the list for more events like this? More than willing to have vendors bug me if they pay for my dinner.

https://redd.it/1dklyj2
@r_devops

User friendly Kanban solution as a frontend for Azure DevOps?

Hello DevOps community,

I am seeking feedback regarding a concept which our team is developing - a far simpler/intuitive Kanban board to organize work around Azure DevOps system. Since we observe a problem where business or non-IT people are quite reluctant to go directly into Azure DevOps, Github or similar dev focused solutions to create work requests.

So we have decided to offer a scriptable two way sync between Teamhood and Azure DevOps. And now we are thinking for doing other integrations next. Does this solution make sense to you and are there any additional problems to solve so that IT people also get significant benefits?

Link to a solution presentation: https://teamhood.com/product/kanban-board/

https://redd.it/1dl08as
@r_devops

Best way for OnPrem postgres

Hello,

So we are shifting to OnPrem for some reason. For this we want to deploy PostgreSQL database on premise. What should be the best way to deploy this? Should we consider deploying in Kubernetes or as docker containers or direct installation using virtualization software's which you know will be present.

I am thinking of a stack for kubernetes:

Rancher

KubeDB

Cloudflare

Prometheus

Stash

Thing is replication is also important and Cloudflare for tunneling.
Any recommendations
P.S Might migrate other postgres databases to onprem also

https://redd.it/1dl1v6y
@r_devops