Reddit DevOps
270 subscribers
5 photos
31K links
Reddit DevOps. #devops
Thanks @reddit2telegram and @r_channels
Download Telegram
Career Advice Network Engineer -> Software / CloudDev / DevOps

Good day,

Looking for the advice for the above.

Essentially I am currently in a Helpdesk role with a company and looking at paths to further my career.

Preferably, the end goal would be for a remote position, however, that is not a requirement.

Current certification is primarily CCNA, of which I am pursuing my Cisco DevNet as well.

I've played around a bit with some software development, with a small number of languages, as well docker which i find rather fascinating. So not 100% on which path would work best for me, however, I am still researching what each position entails and would, of possible hear from people in similar roles already, who wouldn't , mind offering some guidance.

I have considered looking into a BSc in Computer Science from the University of London, however, with my current age, (31) I'm not sure how feasible that would be.

Any and all advice, suggestions, opinions are welcome.



https://redd.it/1e6gifa
@r_devops
Documentation

Shout out to y'all who spent hours writing those support documentation tasks which will never be read and stashed away in confluence until the end of time. Peace out homies.

https://redd.it/1e6ii75
@r_devops
Sysadmin here - do you manage your software yourself or let admins do it?

Hello,

Sysadmin here, currently updating software via SCCM, to get rid of some vulnerabilities. I've noticed that a lot of dev & devops users do not update their software (docker, python etc).

Since I'm a sysadmin, I'm more than happy to do it for you in bulk, but I'm aware that developer apps are very delicate and can break when updating.

So my question is - would you rather prefer to receive an email, giving you a month to update your apps (after that time, it's my time to shine) or you don't care and want admins to do it for you?

I realize the first option may not work, as probably a lot of people would just ignore an email.

All thoughts appreciated, thanks.

https://redd.it/1e6l38a
@r_devops
DevOps for industrial automation - SCADA, PLC controllers and the like (rant and a question)

/ ===== OPENING RANT =====
Hope you enjoy my writing
It provides context for the question
But it is not required to understand it
Skip to the next comment like this if you don't care :(
======================== /

I got hired as the IT team at a small company two weeks ago. I'm not even out of university and I'm already an entire engineering department, cool. We do mainly PV substations, construction and maintenance; but also home automation, power grid connections and the like. Since the company is small (less than 10 people) I also do the gritty industrial stuff, both in the office and on-site, in addition to being a code monkey and the like.

Hailing from the software engineering world, I have a very particular take on the process of creating
stuff. I have a nice modern code editor with bells and whistles, variable names are long, I write tests, commit changes to a VCS, run tests, maybe even automate running tests. Sometimes I even automate deployments! There's also the project management side - GitHub issues, projects, checklists, TODOs in code and out of code. Libraries are well documented (usually), or at the very least, I can look at the code.

Imagine the whiplash I got when I opened the SCADA software we use. It's older than me, the documentation is impenetrable (or maybe I just don't get it), and one of the main protocols is broken (though we don't know who's responsible, both implementors blame each other). Support for automating away boilerplate is almost non-existent. Did you know you can use AutoHotkey as an ad-hoc "code" generator? It's really neat! The SCADA uses JS as its scripting language. The engine has probably not been updated since I was born. It does not even have standard types, so you have to learn a custom `String` type. The system also uses a proprietary data format - it's a bunch of XML nodes glued with binary data. You can manually edit that, but only to a limited extent.
Okay, so maybe it'll get better when we get to safety-critical systems. After all, they better not fail. It would be quite unwise to, I dunno, not disconnect a 500 kW PV plant when the protection and management controller loses power, wouldn't it? This is not an edge case, right? You probably don't want the grid to exist in an unmanaged state. Wouldn't it be unfortunate if this specific scenario- who am I kidding, this happened today. No damage done, beyond a reputation hit, because it was done during project hand-off to the owner. Testing is done by finishing the contract and hoping nothing will explode or catch fire. I don't think the editor even supports tests, nor can I really check because it's proprietary software and a second copy has not yet been bought.
Version management does not exist. It's just not a thing. I copied the SCADA design file I was working on to my local computer, renamed it to include the feature I was working on and I upload it to the main server when I'm done for the day, with a README.txt in which I describe what's been done and what's left to do. I fuck something up? I better remember what I did, or revert to yesterday's copy entirely. Editor history is sketchy at best. Merging changes from two different people to two different things? You wish. One of us will need to manually copy the changes to their project. What changes? God himself could not diff those files and neither can I.
Project management? Done with notebooks. Sometimes. By some people. Usually we just wing it. I don't know what others are doing, and they don't know what I'm doing. I started writing READMEs, but I don't think this will catch on. 


/
===== QUESTION TIME =====
I apologize for any incoherence
I am currently kinda sick
And also falling asleep
Hope it was enjoyable anyway
========================= /

How would you go about DevOps for industrial automation? I can use Git with a self-hosted frontend for tracking changes, but that's not
really enough. Files are in a semi-binary format, so a standard diff isn't the right tool, and merging will basically have to not be done. I'm thinking of rolling a custom tool, specifically for working with those files, both for diffing and merging, but that will require reverse-engineering the file format and also plenty of time. Is there anything that can be done in the meantime? What about testing? I've read a paper about a similar situation, and, inspired by it, I'm considering hooking up the controller to a simulated IO device and either rolling my own or adapting an existing test harness to use real world hardware.
What about deployment? I think it's done rarely enough that doing it manually is fine, but still, not having to do that would be cool. Would something like Ansible work here?
If you have had experience dealing with similar systems, could you share any tips, mistakes you have made and the like?
Feel free to be imaginative - I have very few workflows and people to fight. Securing funding might be difficult, but I'm willing to give wacky ideas a shot.

This isn't really fitting here, so feel free to point me in the direction of more fitting subreddits, but I already wrote an essay so who cares if it's a bit longer. Any suggestions regarding automating some parts of the workflow? AutoHotkey really does help, but it's flimsy. One wrong press of a button and you entered a string of commands that did god knows what. Changing parameters requires opening a code editor and replacing strings. Is there anything short of rev-engineering the custom file format that would allow me to not do the same thing 30 times in a row?

https://redd.it/1e6lq7j
@r_devops
Anyone else sitting here waiting for Azure to come back up?

Been hours now, we are currently trying to move 25TB of data from one cloud hosting to another while hoping Azure Central US comes back up.

https://redd.it/1e6qe2o
@r_devops
How Do You Automate Your Status Pages?

Hi r/devops community,

I'm looking for advice and best practices on automating status pages for monitoring service health and notifying users of outages or performance issues. Specifically, I'm considering using Instatus to create and manage our status page.
Here's a bit of background:

I'm running multiple Kubernetes services, and I want each service to have its own component on the status page.
The goal is to automate the process of updating the status (Operational, Partial Outage, Major Outage, Degraded Performance, Under Maintenance) for each service.

Before I dive into implementing anything, I wanted to ask:

1. How do you automate your status pages?
2. What tools and processes do you use?
3. Any tips or best practices for integrating Kubernetes with a status page tool like Instatus?

I appreciate any insights or feedback!

https://redd.it/1e6xekh
@r_devops
Slow rendering some website pages and images

Hi r/devops community,

I run a blog called thenextscoop, and I face issues with some of the pages and images rendering very slowly, and some of them even fail to load. Is there any solution where I can check the website's health, latency and uptime? Earlier, I used a few browser Chrome extensions, but they did not give the right data in real-time.

I will be happy if any community can help me here... I appreciate it in advance.

https://redd.it/1e6y15h
@r_devops
clean overlay2 docker

Hello,
Is there any safe way to clean overlay2 ?

It's label studio docker image running using this command:


docker run -d -it -e LABEL_STUDIO_DISABLE_SIGNUP_WITHOUT_LINK=true -p 8080:8080 -v $(pwd)/mydata:/label-studio/data heartexlabs/label-studio:latest

please find more informations here:


root@vps-8b5453ed:/var/lib/docker/overlay2# du -sh * | sort -rh | head -n 5

129G 2cba2b496509f78e63274c6b9bcff18eb43c0fbe55c06ecfc684a1f883902aa3

836M ad32b03c1cae41ae7d2efc8973e5636c06602f78bc9af14b77a620965e914854

589M 10a810c769edf7e59c57d58cc693ac845fbf36ece72e46f55bcc8a9d07169b27

387M 74c3016cbdaad24ba3b5a58bb15ceba3e1130755dc8f17c343d0e6ba8a903637

156M m8yt2pa29iegnnlkp7zewi17g

root@vps-8b5453ed:/var/lib/docker/overlay2#

https://redd.it/1e6z840
@r_devops
Struggling with google cloud storage

I'm using Google Cloud Storage to update a CSV file from my website and when I manually check it it does update the CSV file, but when I read the file, it just does not include my last entry. I think google place a cache thing, but just want to know for certain why this is happening and how I can win it.

https://redd.it/1e7112m
@r_devops
Hey what to learn in Linux ???

Hi everyone i want to learn linux for devops but have no clue what to actually learn in it i went to roadmap.sh and the linux path was super overwhelming please anyone can tell me what to actually learn in it .


https://redd.it/1e6zjfg
@r_devops
Industry Trends | DevOps , Cloud

Hi all, what do you think are the most important trends shaping the future of DevOps and cloud computing? How are you preparing for them?

Because I think DevOps and Cloud filed is getting saturated and it’s not enough to survive as DevOps.(let me know if you think it’s wrong assumption)

https://redd.it/1e73ng9
@r_devops
Need expert advice

I am a devops intern who has been assigned tasks, like my first task is interesting to me but, I am on a week holiday and before going back and asking out my mentor or other mates, thought to feed my brain something.
Like the task is to "create a one stop access solution, like say we have developer's who once get their role will need default access based on the role that they get, say frontend or backend or anything like that

I need to provide such users access to certain software such as Jenkins, db access through a bastion, open search etc...

I was asked to make sort of a ui where once user is added I can do this checkbox to provide access and remove access, set time for access, etc....

Any insights will be usefull, I was said to look for AWS ssm so that the logs of the actions could be more clear and concise.

https://redd.it/1e746f0
@r_devops
How to Best Integrate E2E Tests into GitLab CI/CD Pipeline?

Hi all,

I'm looking for advice on the best way to integrate end-to-end tests into our existing GitLab pipeline. I am lost at this point. I'm doing this for the first time and I'm unsure how to handle it. Here’s a brief overview of our current setup, goals, and daily development workflow:

Current State:

Pipeline Tool: GitLab
Repositories: Backend (Spring Boot), Frontend (Angular Monorepo NX), Keycloak
Branching Strategy: Master-Branch, Feature-Branches

Daily Development Workflow:

When a feature is complete, we create a merge request from the feature branch to the master.
The pipeline is triggered on the merge request and includes the following steps: Prepare (npm etc.), Tests, Lint, Build, Trigger Deployment (manual trigger for the second part).
After successful pipeline completion, E2E tests can be manually triggered.
Merging can be done without triggering E2E tests or even if they fail, which currently has no impact on the merge process.
Clicking to trigger E2E tests temporarily deploys the changes to the dev stage.
Upon merging, changes are deployed to the master and rolled out to the dev stage.
During a release, we create a tag, and only the normal tests are executed, skipping E2E tests and deploying to the production stage.

Problems:

Manual execution of E2E tests.
Merge process can be completed even if E2E tests fail.

Goals:

Automate E2E tests for each commit/merge request.
Prevent merges if E2E tests fail.

Given our current constraints and setup, what would be the best way to achieve these goals with minimal disruption?

My initial thoughts were to containerize the frontend, backend, and Keycloak, and create a temporary E2E stage that would be terminated after a successful or unsuccessful job. However, it seems like a configuration mess and a waste of resources.

Are there simpler ways? I am open to any suggestions.

Thanks!

https://redd.it/1e759j6
@r_devops
As a DevOps architect, how would you ensure that an outage caused by CrowdStrike does not affect the development lifecycle and operations of your application?

🤔

https://redd.it/1e784tv
@r_devops
Master in Data Science or Software Engineering?

I am currently working as a Linux System Engineer in a company where I write bash scripts to automate tasks, monitoring infra and services using prometheus, sending alerts to slack channel, and deploying services in containers and currently building a kubernetes cluster on our infra. I wanted to get into the Data Engineering and DevOps/DataOps/MLOps field. I hold a bachelor's degree in Management Information System and currently applying for a Diploma followed by a Master program in a university in my country from the faculty of Computer Science and Artificial Intelligence. They have three programs available; Software Engineering, Data Science, and Cyber Security. But I don't know which is more pertinent to my field, Data Science or Software Engineering Master and had mixed opinions about which one to choose. I already have a fairly good background about Computer Science subjects like Algorithms, Data Structures, and coding in general.

https://redd.it/1e7ap1n
@r_devops
3 Essential Linux Command Line Tools for DevOps Engineers

A short video about using yq, sed/grep and curl. Getting better in using those commands is essencial for devops engineers (and not only). With proliferation of AI, there is even greater need to learn and understand the underlying technologies and tools. Hope it helps someone to learn a few new things or improve.

https://youtu.be/BYdrUJcU1yU

Related material:
- companion blog: https://medium.com/itnext/6-essential-linux-command-line-tools-for-devops-engineers-5cd23b578c50
- terminal slides: https://github.com/Piotr1215/shorts/tree/main/3-devops-tools

https://redd.it/1e78s5u
@r_devops
Advice on Running SAST and DAST with Veracode in Azure DevOps Without Access to Client's Source Code

Hi everyone,

I'm working on a project for a client where we need to run SAST (Static Application Security Testing) using Veracode. The client has provided the necessary endpoints for the DAST scan, and that part is straightforward. However, I’ve hit a snag with the SAST.

The client wants to integrate Veracode into their Azure DevOps pipeline but is not willing to share the source code with us. This brings up a few questions and concerns:

1. **Is direct access to the source code required to integrate Veracode with Azure DevOps and run SAST?**
2. **If the source code is not required, what are the alternative approaches to perform SAST under these conditions?**
3. **What specific type of access do I need in Azure DevOps to set up and configure Veracode for running SAST?**
* I assume I might need Project Administrator access to configure pipelines, deploy, and install/configure the Veracode extension, but any confirmation or additional insights would be helpful. if he's not okay to give us the Admin access, what are alternatives roles ?

Any advice or insights from those who have navigated similar situations would be greatly appreciated!

Thanks in advance!

https://redd.it/1e7cbjn
@r_devops
How to manage dozens of gitlab tokens in CI jobs?

Scenario: gitlab on-prem driven CI with many repos working together to provide a single infrastructure:

So we have a lot of tokens to manage. As gitlab now enforces a 1 year max token lifetime I've just had the realisation that hunting through CI variables in dozens of repos, recreating new tokens in other repos that that CI needs to access, with the appropriate permissions is not a sustainable approach.

So apart from better READMEs in each repo or a big spreadsheet, how do people manage dozens of tokens with varying permissions that need to renewed yearly and update the secret stored in the correct CI variable?

Unhelpfully gitlab deletes expired tokens and I don't see a convenient UI to list all project tokens across the entire account.

Curious... I assume this is a common problem with gitlab/github driven CI?

Many thanks in advance for any suggesstions, ideas, pointers... 👍😀

https://redd.it/1e7f5ot
@r_devops