[ELK] Logstash Parsing issues.

Hey Guys!
I have ELK stack configured, where multiple sources (Filebeat) push logs to Logstash and it pushes logs to Elastic search.
However, there are multiple instances where logs were not pushed in ES because of some error such as:
\-Error parsing JSON
\-Unexpected character (, etc
I came to know about this issue when I checked var logs of Logstash.
Question1: How can I check which was the source when Logstash failed to parse and what log it was?
Question 2: How can I configure alerts for Logstash, triggered when any line failed to parse?

https://redd.it/erax55
@r_devops

Opentracing - Jaeger vs. Zipkin

I was wondering, have anyone used either jaeger or zipkin in production. What worked for you, what are the pros ans cons of the two?

https://redd.it/eraqve
@r_devops

GitOps (argo cd, flux or custom) and migrations and secrets

We have \~10 microservices mostly in Django, but we will add more. I'm trying to introduce GitOps (we have yamls and some helms) and some tools for releasing/promoting images to different environments and rollbacks... But I have a big problem with migrations:

1. I can handle "forward" migration in hooks/jobs/init containers in argo cd.

2. However, when I want to make rollback migrations let's say migrate database version 16 to 14, then it blows my mind how to do it. Argo cd doesn't support rollback hooks.

​

Another problem:

1. How to handle secrets in GitHub repository?

https://redd.it/eru0fg
@r_devops

Blog post: Building an Azure DevOps Pipeline [Soup to Nuts]

Hey guys, I just wrote a new blog post you may enjoy on the ATA blog.

Summary: Learn how to build a real Azure DevOps pipeline for an ARM template from beginning to end in this real-world, step by step tutorial.

https://adamtheautomator.com/azure-devops-pipeline-infrastructure/

If you'd like to write posts like this, get promoted and get paid, join us! https://adamtheautomator.com/friends

https://redd.it/ervvzy
@r_devops

«Consuming» existing on-prem infrastructure with IaC

Hi all,

I currently have a number of sites with on-prem infrastructure that due to various reasons (bandwidth, latency etc) cannot be migrated to cloud.

I also recently set up a new site, where I modified our existing cloud pipeline/IaC-framework (Terraform/Ansible etc) to deploy new site infra from scratch and it worked a charm.

I would like to extend the modified on-prem IaC-framework to existing sites as well. Unfortunately, for these I don’t have the luxury of a scratch deploy as all sites are nearly 24/7 operations.

Are there strategies or best practices for applying/transitioning in an CI/CD-pipeline for IaC into an existing infrastructure?

https://redd.it/eryfsp
@r_devops

Single-tenant application delivered as a multi-tenant service?

I'm trying to wrap my head around how this might work: Rather than create a multi-tenant application, my thought is to create a single-tenant application that is then deployed repeatedly for new customers.

I'm curious if anyone else has adopted this approach, and if so, how? What are some potential downfalls that would make this a pretty bad idea?

https://redd.it/erwzro
@r_devops

Building a Delivery Platform

https://blog.brujordet.no/post/devops/building-a-delivery-platform/

This is a blog post I wrote that initially started out as a confluence page where I wanted to lay the foundation for building a Deliver Platform at my new job. We don’t really have anything yet, and I needed to describe the basics so others can chip in.

This is all based on my experiences so I’m keen to hear your thoughts on this. Especially the approach to a golden path and how you’ve dealt with PaaSifying deployment to k8s for developers who don’t care much for DevOps. Is this pretty much obvious stuff that I’m talking about or do you have contradicting advice?

https://redd.it/erz1j5
@r_devops

Has anyone used or heard of KubeMQ?

We're in the process of migrating to Kubernetes and we were looking into alternatives for message queues. We found one that looked promising, KubeMQ, that claims to be a cloud native message queue that was specifically designed for running on Kubernetes. However, when we started looking into the adoption rate, we were a bit stumped... It seems like KubeMQ has zero presence online. Nobody has ever asked a question about it on stack overflow, nobody has written blog posts about using it (except 1 single person), nobody has compared it to kafka or rabbitmq (except KubeMQ themselves). It seems strange to me that something that seems so "legit" has never been tested or talked about by anybody, even here on reddit.

So, has anybody heard about this technology or used it?

https://redd.it/es0s78
@r_devops

Seeing the state of production in git and parameterising deployments that are very similar

Hi folks,

I was once mentored by a fellow who i am no longer in contact with and he recommended to me that all of the config for production remain in a single repository, as a sort of "source of truth" and I was wondering how that might be done and if it is still (if at all) a best practice?

My current thinking is that all config should live in the repository along with source code and the infrastructure required for that code to run in the cloud, so that all of the code required for a service exists as an atomic unit.

But, this causes problems when we e.g. want to deploy 10 services which are all essentially parametrised versions of each other, that is that the source code is the same but the configuration related to each of the 10 deployments is what functionally separates them. Do we separate this information into 10 repositories? For a concrete example we might have 10 data sources that we want to scrape from and all return the same format for data

I would be grateful for any discussion or resources on the matter.

https://redd.it/es1ipp
@r_devops

Minimal tracking of OS health indicators with alarms

I have a one off project that requires the tracking of:

* memory usage
* load average
* disk utilization

for a linux VM. The minimum functionality is to send notifications (email, Slack message) when those metrics exceed a threshold.

Knee-jerk responses might be to use Nagios, prometheus, SNMP etc. I want something simple with minimum effort (max: 1 day effort). What is your expert recommendation ?

https://redd.it/es07t8
@r_devops

Stopping Python-based backdoor Attacks with zerotrust networking

An overview of how to stop attacks like MechaFlounder with a zerotrust networking approach.

https://vimeo.com/383337193/65c5e52bcd

https://redd.it/es32i6
@r_devops

Automated testing for audio DSP software (VST)?

I have been using automated unit tests with TDD (test driven development) at work for physics software development and it made me wonder if audio DSP software could work with tests/TDD.

Some VST developers I know have posted videos about how debugging can be very time consuming for them, how they have noticed that certain parts of the audio processing are never actually executed (which I think is purely a code coverage issue), etc. which interests me in seeing if there is some way to do automated unit tests for TDD.

VSTs aren't standalone software, they run inside a VST host, such as a DAW (Digital Audio Workstation), almost all of which are graphical programs. So far the only way I have found to test them is to build to .dll file and use the VST in the DAW, which is purely manual testing. In addition some DAWs handle plugins differently.

I was thinking a sort of command line based VST host could suffice for setting parameters on the VST, hosting it, letting audio pass in and out, etc. [I did find Ecasound which apparently has command line support](https://lwn.net/Articles/509958/) but it is Linux only. But that is better than nothing.

I was also thinking I could use a recorder to write the output of the plugin to an audio file if that would somehow be useful in testing.

I was thinking some tests could be done fairly easily, such as:

1. Trigger a button press event, ensure that the state the button refers to is triggered
2. Call the various methods as normally, see if anything crashes/raises errors

However, checking if the audio sounds good seems more difficult because it is hard to quantify purely using tests, and it is subjective.

Does anyone have any ideas here? Thanks

https://redd.it/es3y5t
@r_devops

Looking for feedback on tool to fetch all your SaaS audit logs

Hi all,

I am building a tool that fetches the audit logs from all your SaaS tools (think G Suite, Okta, Github, Zendesk, Dropbox, Salesforce, etc) and pushes them into whatever existing logging pipeline you have (Splunk, ELK, Kinesis, etc). The product is in a beta state and I'm searching for testers who can try it out, provide feedback, and help guide my roadmap. If this sounds interesting to you I would love to chat - you can either direct message me here on reddit or shoot me an email at arkadiy{at}logsnitch.com.

Hopefully this fits the promotion rules in this sub but if not I can delete this post.

Thanks very much,

https://redd.it/es2we6
@r_devops

Do you often SSH into remote systems? What are your use cases? Do you have a plan to make it unnecessary in the near future?



https://redd.it/es5ztp
@r_devops

I'm using a makefile to automate terraform deployments. Does anyone know how to pass a Global Red Hat variable to a makefile so I can pass it via SSH agent forwarding. (code below)

echo "$efs_dns" | ssh -At ec2-user@$$(terraform output bastion-public_ip) 'ssh master.openshift.local "cat > efs_dns.txt"'

So this code block is supposed to get the global linux variable '$efs\_dns', pass it to 'master.openshift.local' jumping through a bastion to get the variable output to a file called 'efs\_dns.txt'. The problem is when it's all said and done efs\_dns.txt has some crap value instead of a dns name.

Edit: It seems like the bad output is a piece of the variable name..the variable is called $efs\_dns well the output text in the file is 'fs\_dns'

https://redd.it/es1chl
@r_devops

Moving from network engineering position to DevOps / software positions

​

Hi,

I'm a 23 years old network engineer with 4.5 years of experience at enterprises, I have the following knowledge:

\- Cisco networking - routers and switches also all the protocols.

\- Python scripting and network automation

\- Linux system administration

(not system admin experience but from network engineering perspective).

\- AWS fundamental knowledge

\- Check Point firewalls

\- Virtualization - VMware

​

These are my certs that I currently hold:

\- CCNP R&S

\- CCNA Security

\- AWS Cloud Practitioner

\- LPI Linux Essentials

I currently want to pursue a career in software development or devops and I'm not sure what is the best path for me.

I think about pursuing CS degree, but I got another feeling of doing AWS Solution Architect associate cert and Comptia Linux+ cert and go more to the "devops route"

(also learrning docker + ansible + jenkins afterwards).

I'm not sure if my best way is to pursue a degree or get certs for a better position (Devops/Software)

Thanks,

https://redd.it/ertlj4
@r_devops

Slow your network down

I wrote a blog post about using `netem` to simulate non-ideal network conditions.


Comments and feedback appreciated :)


Thanks


[slow your network down | bad gateway](https://badgateway.qc.to/slow-your-network-down/)

https://redd.it/errmsc
@r_devops

Process Control Solution

Hey, guys, I'm looking for a central ubuntu 18.04 process control solution with a web GUI that can control (stop, start, restart) services for few servers do you know something like this?

https://redd.it/erskjt
@r_devops

As a software engineering student how do I do DevOps

I am a SE student taking on a Capstone project class where I am assigned to a startup company. I have to figure our how and where to host their single web application and MySQL server. I know each choice is circumstantial so how do I make a decision based on my needs. I don't even know how much storage I need.

https://redd.it/errmjc
@r_devops

what are you devops engineers use Golang for ?

Hey

I have 4 month since I started learning golang because it is so cool to learn and has a lot of good features that other programming languages.

but my questions is what can you use it for as a devops / cloud engineer ? what are the use cases ?

https://redd.it/erri4t
@r_devops

Requesting a roadmap to a Big 4 SRE

Hey Reddit,

​

I've posted here on/off for a couple years now. I'm at a point in my career where I'm starting to specialize in a specific job (monitoring, alerting, and log aggregation) and I'm starting to get pigeonholed into it, but also despite several attempts to negotiate with my employer I have not received a significant base salary increase since I joined. Furthermore there are some unsustainable engineering practices we have that I simply no longer wish to participate in and I think it's best I begin to consider other employment options.

Although I understand the fundamentals of DevOps, I think my current skillset is more suited towards some flavor of SRE. I've identified several new skills and concepts I will need to master in order to be able to efficiently transition to a new role (specifically one at a big company) and I've already began studying the interview process.

I just wanted to ask Reddit in general for some advice: What skills would YOU say could get me a general SRE job at a big company and how competent should I be in them? Also, what skills do I need to master to ACE an interview? Would mastering the DevOps roadmap and data structures/algorithms be enough? And for recruiters out there, are there any skills that would really make a candidate stand out? And what skills do you find candidates are too weak in?

Thank you for taking the time to read and respond :)

https://redd.it/erolk2
@r_devops