Did anyone return to Infra or Pipeline from being an SRE?

I recently moved jobs to a place where I do SRE, 50% spent writing code and embedding into multiples the rest is writing integrations and capacity planning.

After a 10 months, I realized that I hate coding professionally. Trying to maintain other people's code and writing tests to cover edge cases is just draining. I used to just write tools and utilities with a specific purpose to automate a tedious task which I found very enjoyable, it clearly didn't translate to coding.

I'm just eyeing returning to do infrastructure work and automation by changing jobs yet again, does anyone else feel the same way?

https://redd.it/s83ppo
@r_devops

Any good online DevOps meetups?

Hey Guys\~ Does anyone know of any good online events to just hang out and discuss DevOps? I've been to meetup.com and eventbrite.com and a lot of the events seemed geared towards interview prep, finding a co-founder, start-up pitches, etc.

I feel like a lot of DevOps is learning through conversation and so I'm looking for something that is more coffee-talk type of atmosphere where people can talk about their projects, ask questions, learn best practices from one another, etc.

https://redd.it/s83htx
@r_devops

too sad for a title…

I’d like to get some thoughts from people that is or was hiring someone.
So I was in the last round of interviews for this nice company and I was rejected because according to the HM I could not speak appropriately and demonstrate skill on the subject. Ok, however the only question I got was for how long I’ve been working with the specific tech. And I’ve been working on a daily basis for a little over 4 years and hold multiple certifications on the subject. How did they come to that conclusion if no questions were actually made?

https://redd.it/s81iuj
@r_devops

Is local development that simulates the CI/CD remote pipeline and infrastructure, part of the CI/CD?

Being able to reproduce DEV/PROD environments locally, infrastructure, run automated testing, lint, etc locally part of a CI/CD pipeline? Or is it only when we think about remote automations?

https://redd.it/s85g05
@r_devops

Script to Track Crontab changes

Hi - is there a script in Linux that I can use to Track the changes made to the Crontab . Basically looking for who has changed the crontab , from which IP , at what time. Once a change to the crontab has been made ; an email alert should be triggered. I used inotify-tools in the past but not able to get the logic to track the crontab changes.

​

Any inputs are highly appreciated.

https://redd.it/s84rdf
@r_devops

Where should DevOps tools be placed?

In prod or non prod environments? This includes code repository, cicd, and scanning tools.

https://redd.it/s8axyy
@r_devops

Windows Program Slowdown in Docker Container



Hello fellow whale wranglers!

I’m trying to figure out why a windows app that’s running in a Docker container slows down over time. The program converts a proprietary raw file format into many individual images. This is data coming off of a cool microscope using what’s called 2-photon microscopy. I’m using Docker-Wine for running it on cluster.

This program starts out performing as well as it does on a windows machine, about 30 images/second.

After it converts a few thousand images, however, it begins to slow down, converting just 15 images/second. Eventually, after about 20k images, it slows down further to maybe 7 images/second. After 30k images, it slows to a crawl and takes about 10 seconds to make an image almost. There’s similar performance problems that occur when using a Windows VM. I’ve noticed that it seems to perform better for a longer period of time if you give it more RAM and processors, but the slowdowns are inevitable it seems.

On a windows machine writing to a local directory, this doesn’t happen at all.

So I’m wondering: What could be slowing things down so much when the program runs on a VM/ in a Docker container? How can I tell? What do I do about it?

If you’re curious about this problem, feel free to read on! There’s a wall of text inbound… sorry.

Likely/good questions:
So why not just do this on a windows machine or the microscope’s computer? We do about 7 recordings per day and this typically ends around 5/6 PM. The conversions are only capable of being run in series using this converter and a typical conversion takes about 25-30 minutes. Converting these into a different file format afterwards (from individual images to one large HDF5 file) takes another 25-30 minutes. Finally these things need to all be transferred to a server connected on a 1Gbps line (so original data, not the individual images, and the HDF5 file). Moving raw data alone takes about 4.5 hours to the server. Doing this transfer for the HDF5 file transfer as well would take probably another 4 hours. If another user of the microscope comes in the next morning, it’s likely the computer for the scope won’t be ready for them to use since it’ll be busy. The main idea is to make it so the microscope computer basically doesn’t do anything but run the scope itself.

Why not just put on an external hard drive and move things with a hot-swap drive that’s carried to the server? This isn’t a bad solution, but this will require coordinating schedules with members of our IT (who are great!) but are on different schedules than us, namely that people will image on the weekends sometimes and our IT team won’t be there to move things to the server/give us drives. Another issue with this is that it requires people to remember to do these sorts of things and coordinate even more on top of already intensive studies and crazy schedules. It would be best if things could at least be done from our servers. The complexity of doing things on our server I think is worth it if it means making the day to day schedules easier for users.

https://redd.it/s8attk
@r_devops

Leaving FTE for 12 month contract?

I do AWS DevOps/on-prem to AWS migrations, IaaC. Worked for the company as full time employee for a little over a year and I’m just not liking it, documentation from previous engineers sucks, constant chaos, bad processes. We are remote for now, but that could change.

I’m compensated really well, at 165K annually, plus PTO (20 days annually)/sick leave. There is absolutely job security, but I constantly feel like I’m struggling due to archaic management, being on call once a month, working weekends at times, some turnover in company etc. I’m definitely not happy, but the pay and benefits kept me from moving forward.

Few days ago, I got contacted by a recruiter to do a 12 month minimum contract, AWS DevOps engineer for a very reputable company, great reviews. $105 hourly on W-2, fully remote and flexible daily hours as long as work is done, I was also assured that workload would be just enough for 40 hours a week, as I asked 2 other contractors hired in August 2021 and they say they really like working there.

Should I make the move? I have no family to support and no debt. I’m also in at-will state.

Thank you!

https://redd.it/s8d8et
@r_devops

Harness launched source available continuous delivery

The Harness CD Community Edition is available for use free of charge.

https://redd.it/s8dzq2
@r_devops

Automate Deployments on Merge to Main Branch

Sorry for the beginner question, but I'm feeling really overwhelmed with this task and need some input from people much more experienced than myself.

I have 2YOE working as a dev. I'm now in a new role (6 months here) and it's a small company, even though my title is Software Developer, I'm doing a fair bit with servers because we need it and I think it's pretty fun.

Before I started, our deployment process was pretty basic. We needed to ssh into the server, switch user to www-data, cd into the relevant directory and run git pull, then run any relevant commands based on which files had changed. Some projects have a separate frontend and API backend so you'd repeat the process if there were changes across the full stack.

Also there are some projects that share one codebase, but have \~10 different customer-facing websites. So if you made changes to this codebase, you'd have to cd and run relevant commands in \~10 separate directories.

What I've done so far:

I've recently wrote a bash script that automates some of our deployment process. Now, we need to ssh into the server, switch user to www-data, run the deploy script, deploy script asks which project is being deployed, deploy script then cd's into relevant directories and runs any relevant commands based on which files have been changed.

I'm happy with what I've done so far, because if we need to compile frontend assets a deployment can sometimes take around an hour. But I'd like to completely automate this by having the script run (or a more automation-friendly version of the script) when code is successfully merged into the main branch. Removing the need to ssh into the server and run the script.

I've looked into Jenkins, read through the docs and watched some courses on YouTube and Udemy, but it all seems really overwhelming. They mention Docker, Ansible, AWS etc and none of this is relevant to me at this time. All I want to do is run the script when code is merged to main.

I've thought about setting up a cron job to check for changes every hour or so, but that seems inefficient to me and I'd rather do things the right way.

Some additional info:

* I only want to use free, open-source tools at this time.
* The client is in control of the server. No idea where it's hosted or even if it's on prem
* I'm using PHP, Ubuntu, Bitbucket.

https://redd.it/s8eifq
@r_devops

Who is tracking the performance of the CI pipeline and test success metrics in an organization?

I'm seeing organizations are more and more trying to optimize the CI issues.

I wonder who is accountable/responsible for the performance of the CI pipelines?

https://redd.it/s8fa5b
@r_devops

Some questions about influxdb / k8s

Hello,

Please excuse my broken English.

My company collects metrics from different cloud customers in InfluxDB, queried by Grafana. Most small customers are stored on a single, shared EC2. Some large customers have a dedicated EC2.

This model is showing its limits today:

\- no HA

\- performance problems occur and the only way to deal with them is to vertically inflate the EC2.

\- an incident on an EC2 can quickly impact x customers

A SaaS solution is not an option for my company at the moment.

From my point of view, I would tend to find a metrics storage solution that supports clustering and sharding, either licensed or open source, and to deploy all this in multi-AZ for HA. On the other hand, a new open source solution forces us to leave InfluxDB and rewrite a lot of dashboards.

Some colleagues suggested to move to containerization and to put influxDB in EKS, each client would have its own database with persistent volume, and we could separate clients by namespaces. I don't have much experience with K8S in production but this raises several alarms for me:

\-> increased complexity

\-> how will the service behave if a node change is scheduled (e.g. drain for update) or forced (node crash)

\-> if we use a cross-AZ storage like EFS, I'm afraid that the performance will not follow at all, InfluxDB is greedy in I/O. And EFS is much more expensive than EBS.

\-> I have the impression that using k8s for a stateful application, which does not scale horizontally, and which is potentially attached to an AZ does not bring much, but I do not have enough experience in orchestration to be sure

I'd like your advices ...

Thanks for reading.

https://redd.it/s8fwxm
@r_devops

Kubernetes Operations Survey

Hello everyone!

I’m doing a quick research about Kubernetes cluster operations. I would appreciate it if you can participate in this survey:
https://forms.gle/4iuyHuPX55wFVL137

Thank you!

https://redd.it/s8hub9
@r_devops

Can you refer multiple Azure cloud services as cloud infrastructure ?

I am using various cloud services on azure for a project such as VNet, Application gateway, AKS, VMs, PublicIP.

Is cloud Infrastructure a appropriate term to refer it ? I need to use it in resume.

For eg. I set up the cloud infrastructure.

I analyzed existing cloud infrastructure

https://redd.it/s8ch4j
@r_devops

How to do a complete security audit of your mongo cloud account

https://kloudle.com/academy/how-to-do-a-complete-security-audit-of-your-mongo-cloud-account

https://redd.it/s8a3j3
@r_devops

HTTP is working with custom domain, but not HTTPS.

https is working with appname.ap-south-1.elasticbeanstalk.com, but not working with custom domain. only http is working with custom domain

https://redd.it/s8ld33
@r_devops

Is there a cloud engineer here that would be down to chat?

Hey all! I might accept a cloud engineer position and I just wanted to see if there is anything in the job description or in my experience so far with them that stands out to you as red flags. I’m currently in an operations/app support role at my current company but since this job has the opportunity to involve more coding/automating, I wanted to move towards it. Any help is very much appreciated!

https://redd.it/s8mq3m
@r_devops

Is many Pipelines good or bad

In Corporate Organizations you often have thousands of unique Pipelines. I recently thought about, if this is a good thing or a bad thing.

I understand that it gives flexibility to the teams, on the other-side, it also means that there is a lower degree of standarization.

​

What are your takes?

https://redd.it/s7u0ja
@r_devops

How do you manage supply chain vulnerabilities of images and pacakges?

Should all images, packages and dependencies get updated - which will trigger a full system rebuild and requirement for full unit, integration and e2e testing or use a firewall or some other way of delivering updated images, packages and externally used libraries?

Supply chain security seems to be very expensive as you have to run daily scans for vulnerabilities due to the fact that you cannot know what vulnerabilities will be found when you even use the latest version today that can be vulnerable tomorrow.

https://redd.it/s8ovrb
@r_devops

What are your "most used" Ansible modules?

Disclaimer: I am mostly interested in VPS because I pay for my own servers, and it's way cheaper than AWS and whatnot. While I am interested in all aspects of tech, I don't have much use for AWS, Azure, or GCE specific modules at this time.



Hey everyone,

I've finally gotten around to start learning Ansible recently, have written some simple playbooks, etc.

I'm curious what you all think are the most important modules you refer to all the time, and which are the must-haves in my toolkit. Ones I have found important so far:

- package and distro-specific (apt mostly as I like Ubuntu servers on the cloud) package managers.
- user to setup non-root users for login.
- service / systemd.
- ini_file / lineinfile for configuration.

I guess I should look into ufw too.

What other modules would you say you use all the time and I need to learn ASAP?

https://redd.it/s8qd2i
@r_devops

Hosted SAAS Basic ElasticSearch or SOLR Options?

Hello, my shop is looking for something along the lines of a Hosted "Human Face" on SOLR or ES. We don't have the staff to build up our own environment, and are looking to a vendor to sell us a packaged solution. We have 20K fairly structured PDFs we need to search with mostly pre-set terms, and a wildcard. Any ideas welcomed!

https://redd.it/s7zzli
@r_devops