(Blog Post) The Plucky Astromech Droid - A Star Wars DevRel Story.

I've been working in DevOps/Ops Developer Relations for a few years and I wanted to share some thoughts I had after a recent company summit.

Please Enjoy!

​

[https://dev.to/jaydestro/the-plucky-astromech-droid-a-star-wars-devrel-story-3nhd](https://dev.to/jaydestro/the-plucky-astromech-droid-a-star-wars-devrel-story-3nhd)

https://redd.it/f1h9vh
@r_devops

CICD: When to run unit tests?

Before or after build? Or both?

https://redd.it/f1jnvr
@r_devops

Continuous Testing CI tools for Desktop Automation

I have a few desktop applications that I have some automation tests for. I'm curious what the best tools for the job are for running these tests CI?

I use Azure Devops for source control and build pipelines, but historically these automation tests have been ran from Jenkins as all the agents are self hosted, and azure Devops has limits on how many jobs can run in parallel.

What other tools do people use to best run and display results of desktop automation runs? Keen to evaluate all tools and feature sets.

https://redd.it/f1give
@r_devops

What does devOps mean in practice ? What does a devOps engineer do. How is it different from a regular software engineer !



https://redd.it/f1gc8n
@r_devops

Question: Python isn’t something you’d find in production. Why?

I work with data science teams (python is their choice) as well as backend engineers (Scala/Java) and I hear this statement often. Why is it that you wouldn’t want python in your production stack?

https://redd.it/f1evno
@r_devops

How many hours per day/week on average can/do you actually spend focusing, at the computer, on your job?

​

In my 20s, I had jobs for years where I was expected to be in my desk seat at the office for 8 hours a day, because I was on shift, doing application support. We had some very busy days, but spent a lot of time browsing the web.

As I've grown a bit in my career into a more Sr. DevOps/SRE, and am in my mid-30s, I can't even sit in my chair for 8 hours a day without serious low-back pain (I'm trying many things to rehab this with standing desk and various exercises) , and theres no way I can spend 8+ hours being actually productive technically, day in and day out, without burnout. Sure, some days I will code for 10 hours if I'm really engrossed in a project, but this not my norm.

​

Most days I get up and have breakfast/coffee with my wife+toddler, sign into work and put in \~5 hours of *focus time* and then I am fried. I often try to get back into work, at least preparation/emails for the following day later in the evening, with varying degrees of success.

​

As a remote employee, I don't have face to face socialization with my actual coworkers, my meeting schedule is minimal, etc, so there isn't a lot of things to fill the gaps around my focus time to fill out a 40 hour week.

​

I'm always available via slack/phone. I spend a lot of time thinking about technical problems and solutions when I'm not at my desk, not even voluntarily, in the shower, while going for a walk, etc, and I think this is vital to my process. I don't know how to quantify that time or the value of it exactly though.

​

I'm curious your thoughts on this. I feel like there is probably a large percentage of people in our field in the same boat, but it's not something you can advertise or talk to your supervisors about without being accused of "time theft". Is the 40+ hour week reasonable? I feel like most us have a best part of hour day for focus, which is at best several hours long where the magic happens and the best things get done.

https://redd.it/f1cx0q
@r_devops

Example CI/CD using Docker,Gradle,Spring,Terraform,Helm,liquibase deploying to EKS kube cluster

Hey, So a few days a go, a few redditor gave me some help in fixing my terraform, so in the spirit of giving back, i thought I'll share this case study i've been doing. https://gitlab.com/roboops/case_study


1. Heres the Readme : https://gitlab.com/roboops/case_study/-/blob/master/README.MD
2. High level arch diagram: https://gitlab.com/roboops/case_study/-/blob/master/readme/CI-CD-High_Level_Arch.png


**The application:**

is a spring boot, writern in kotlin, that reads a file, adds it to a db and read the content. Nothing special, it does use liquibase to do the database update/management, that's about it.


**The ci/cd side:**

1. I use gradle to build the code, and bmuschko plugin to automate the creation of docker images (saves you having to write your own dockerfiles)
2. There's a jenkinsfile
3. helm chart , this helps do the deploy to your EKS cluster
4. Terraform , this will create a EKS cluster in your private subnet.
5. Uses k8 secrects to set the DB password


**Personal thoughts**

So i feel, this is a pretty good way of doing CI/CD for a typical java base microservice, deploying to a kubernetes cluster. I really like using gradle to do my build, as it means all my logic can go into the gradle file and there should be very minimal logic in my jenkinsfile. Makes debugging way easier IMO.


This could be made more production like, if I used some sort of ingress controller, service discovery, config management etc. But the point is more around the CI/CD side.

https://redd.it/f19iry
@r_devops

Do you find people you work with are really bad at using Slack?

Not sure if its just my company, but people seem really incapable of using Slack properly. Like they routinely miss messages entirely, "at" people (or the entire channel) at totally inappropriate times, hold conversations in the "wrong" channels, mix up channels with private messages, etc. It's basically a shitshow.

 

Maybe it's partly because I "grew up" on IRC, but there is a good number of senior developers and management types I work with who seem utterly confused by how to use Slack in a sane manner. I'm not sure if I should offer some kind of training on this as it seems so simple/easy to me.


 

</rant?>

https://redd.it/f2bzdv
@r_devops

Source of truth for Devops, because Github is just not enough!

Hi guys,

I’m trying to understand what’s the best way to improve the RCA (root cause analysis) process in a distributed system, both technical (microservices) and organizational (devops, devs, analysts).

No matter where I worked (as a Devops/SRE/Backend) I noticed a recurring problem - there are just too many moving pieces and not enough visibility to monitor them.

Detecting that **there is a problem** becomes easier by using tools like APM (datadog) + Exception management (Sentry) + Logs (Kibanha). But from my experience **even when you know something is broken, it’s hard to find out why.** It becomes even more prominent when working with multiple teams (dev, devops, infra, analysts) who use various systems & tools.


While debugging a problem I find myself forced to open several tools (Kibanha + Datadog + AWS + Github + Sentry + Slack) and use different techniques in order to pinpoint the real root cause (i.e. which recent event might explain the symptomes).

I know **Github** **should** **be the single source of truth**, but from my experience this is not the case for most problems. For example: infra changes that were done via the AWS console, manual schema changes, recent deploy/rollback, Cron runs that we forgot about, an undocumented DB change, etc.

The RCA tends to lead me away from Github and into the dark corners of the system. To mitigate this pain, I’ve found several solutions that helped us. For example:

* Important Cron runs status (start/end) are sent to Kibana
* Asked people to write about infra config changes in a dedicated Slack channel (but sometime people just forgot)

Some questions that I find interesting:

* Do you also feel this pain? Or is it just me?
* What are the best practices for tracking all of these changes?
* Did you implement some in-house solutions in order to solve this?
* How to reduce the time it takes to find the root cause?
* Is it just me or slack become a super important tool in the process of tracking changes?
* Do you measure your MTTR (Mean Time To Repair)?
* For those of you who are using Terraform / Pulumi: how easy is it to find problems that occured because of infra-related changes?

Would be happy to get any advice or feedback.

https://redd.it/f2c5h6
@r_devops

Wufei an alternative to kube tail

# [https://github.com/ericmcbride/wufei](https://github.com/ericmcbride/wufei)

(x-post from /r/rust)

Im currently an SDET at a company where we deploy our whole microservice architecture to separate namespace in kubernetes for testing. Since the beginning, debugging what happens in these test namespaces could be a nightmare. So last year we started using kubetail to pipe logs into a file. That helped tremendously but the log files became very hard to search. So in my spare time i wrote a project in rust I called Wufei to do the base functionality kubetail does but to pipe the logs to files based off of names of the pods-container.

First draft went alright and still is being used. Issues with it included CPU Usage (since I was using the td::process::Command;) would call kubectl logs <my\_pod> N amount of times, N being the current number of pods. When doing so, it would spin up a new pyenv (from AWS in my case) for every call which would cause a huge spike in CPU Usage. We worked around this as well, but I started thinking, theres got to be a better way.

That lead me to finding [https://github.com/clux/kube-rs](https://github.com/clux/kube-rs) a kube client that had async / await syntax. I realized that there was an open issue, so I tackled it, and now I've worked up the courage to post Wufei on here. This is my first post on reddit, and first time posting any code I've done on a public forum. Currently has been tested withe 140 pods with no problems.

Features include:- Update the log tailing for new pods that spin up- Log streaming to files- Log Streaming based off selector labels- Log Streaming based off json keys.

Im open to any feedback, and want to start giving back to the community. If you find any issues please let me know. There are more features coming soon, and I hope to have a more extendabile way of people using it (i.e. system package mangers) rather then downloading rust and using cargo to run it

https://redd.it/f1bbmz
@r_devops

Tracking third party tool and software releases

Currently we track new releases of third party tools and software manually. This would normally be via email lists or regular checks on the tools website. The sort of software we track are things like Kuberneres, helm, redis etc.

The problem with mail lists and manual checks is they are manual, can be missed and generates a lot of toil.

Is there a tool or a best practice out there that tracks tools and software releases? A central place that notifies you and keeps track.

https://redd.it/f18rfp
@r_devops

Kubernetes Question: Rancher or Kops or ___?

I'm trying to determine a good solution for kubernetes currently. Trying to get to production with it. Anyone have opinions on Kubernetes solutions? Leaning towards Rancher, but I'd like some counterpoints for Rancher. Any bad experiences/things I should know before committing to one or another? Thankssss

https://redd.it/f2hjzc
@r_devops

Blockchain Applications and Smart Contracts - Developing with Ethereum and Solidity: New earnings & professional growth are waiting

Blockchain Developers and Engineers, this valuable training program will equip you to define relevant Blockchain application use-cases, set up a development environment for Solidity, use the Solidity language and how to code a smart contract, create a test network to test applications without cost, and launch a smart contract in a live network. Training modules include: 1) Introduction to Blockchains and Smart Contracts - History of Blockchain technology, Consequences of double-spending avoidance, Objectives of different Blockchains, Adding smart contracts to Blockchains Determine relevant smart contract use-cases, 2) Ethereum: A Smart Contract Blockchain - Ethereum as a Blockchain for smart contracts, Using Truffle as a smart contract development tool, Ethereum addresses and transactions, Relationship between Ether and Gas, 3) Solidity: A Contract-Oriented Language - Solidity smart contract, Solidity declarations, Solidity function modifiers, Solidity error-checking, 4) Testing, Debugging, and Deploying Smart Contracts - Test smart contracts on a personal Blockchain, Debug smart contracts, and Deploy smart contracts on a test and live network, and 5) Smart Contracts - A Custom Token in Ethereum and Creating a token framework.

Register today at [https://tinyurl.com/r88l6uz](https://tinyurl.com/r88l6uz)

Career success awaits you, Lawrence Wilson – [Cryptocurrency, Blockchain and FinTech Academy](https://cryptocurrencyacademy.blogspot.com/)

https://redd.it/f2ib2u
@r_devops

How would you ensure availability of a couple of small wordpress sites?

The sites are running on a VPS with just apache and MySQL. What do you use, and what would you recommend? Cloudflare serving a cached version? Some other server architecture?

Any suggestions and advice is appreciated!

https://redd.it/f2heyq
@r_devops

Can a VPS run a constant stream of keras.model.predic_classes()?

So see, I have trained a DNN model to detect spams on my site. I can't run both the Node server, the Vue frontend, and the Python backend that contains the model object at the same time on my PC. So I can't test it on-the-go. However, I have tested the model alone, and it's pretty much perfect, with a kappa accuracy of near-100%. So my question is, can a cheap VPS run this class that a GTX 960 2GB can't run?

&#x200B;

Thanks.

https://redd.it/f2hn1j
@r_devops

Wazuh vs. AIDE?

Anyone have experience running Wazuh for host intrusion detection? Got any good or bad experiences to share?

Following the recommendations in one of the Center for Internet Security's benchmarks, a year or so ago I set up AIDE for host intrusion detection on all our systems. But... it's really not very good. Massive numbers of false alarms, the default rules in Ubuntu are just flat-out wrong in places, takes tons of ongoing effort to fine-tune the ruleset by hand, and, the kicker, no aggregation across hosts so you get 50 identical email reports when the same file changes on a set of hosts. And it doesn't even do real-time monitoring, so an attacker could get in, mess with our systems, clean up their files, and we'd never know they were there if their intrusion didn't happen to intersect with an AIDE scan.

It looks like Wazuh is probably a much stronger option and will fix many of our pain points, but this time before I take the plunge I want to find out if it's going to be just as bad in other ways.

https://redd.it/f2gan7
@r_devops

[Collecting ideas] How would you convince an admin that automation and DevOps mindset is a good thing?

I would love to hear your ideas, how an admin could be convinced that automation is a good thing. (ok, I'm feeling a bit lost with this guy)

Imagine the following sysadmin:

* VMs are created using the mounted ISO in VMware VSphere and the installation is: manually doing the install process for each machine (no templates)
* Docker Containers are started via `docker run` as root on all machines
* After adding a new container, a entry is added to the monitoring tool manually
* `ssh root@<IP>` is the default behavior
* `docker logs` is used for each individual container
* quote: 'git is evil'
* quote: 'Developers are viruses on my servers!'
* quote: 'A single PHP file is better than this shitty database API!'
* quote: 'Software architecture is not necessary and useless bullshit. It just slows down everything!'

Tell me what you think.

https://redd.it/f2fx69
@r_devops

Base OS for "bare-metal" Kubernetes

Disclaimer: I've mainly used already-running k8s clusters on cloud platforms and would consider myself as medium-skilled with it. Had a go at using Kubespray with Core OS Container Linux a couple of years ago but gave up due to requirements of the client to do with egress to external repos.

I'm looking to set up a cluster at home on a fairly resource-limited little server (16GB RAM, i5, 1TB disk). I'm planning to use VMs as worker nodes. I'm running ovirt on that node, but might abandon that as it has its own overhead (and seemingly memory leaks) that might be a bit too heavy for that little box. It would be nice to be able to add other nodes as required with minimal fuss. It's just to play around with k8s and run a few internal services, really.

I ask because I've heard CoreOS is now EOL and Fedora CoreOS is perhaps not quite ready for primetime. Are there any other mature-ish base OSes for kubernetes that can fill the gap?

https://redd.it/f2bbkj
@r_devops

Best CI/CD tools

Hi guys, planning to shift to another CI too, currently using GoCD in my project, can someone please help me with other CI alternatives currently hot in the market which you guys are using or have used in your project along with it's pros and cons. So that I can make a decision as to which one to go with..confused with so many options being thrown at me would be great to get some clarity..

https://redd.it/f2c7bz
@r_devops

The story of how I have spent 5 days getting dns to work in k8s, and it still doesn’t work

It all began when I tried K8s for the first time two weeks ago. I set up a one node one worker cluster on two RHEL8 VM‘s at home. Everything worked like a charm, no problems. Peace of cake, or so I thought.


We use swarm at work. Me, having read about what makes k8s superior, and having set up a little cluster at home: ‚Hey guys, how about I set up this new project with k8s?‘

And there I am. For the last 5 days, I have dedicated every minute of my life trying to get up a working k8s cluster. I cannot manage to get DNS to work inside pods on workers.

- I have worked through multiple tutorials multiple times
- I have tried a load of different linux distros and distro combinations
- I have tried using docker and cri-o as container runtimes
- I have tried cgroupfs and systemd as cgroup driver
- I‘ve tried flannel, weavenet, calico and cilium
- I switched to iptables-legacy
- I tried turning off firewalld
- I configurated sysctl to ensure iptables is not being bypassed
- I tried using internal and external nameservers on both nodes
- I tried nodes in different networks

Most of the attempts were made with two CentOS8 nodes, newest version of Docker and Kubectl/Kubeadm/Kubelet 1.17.2

In 70% of my attempts, I end up in a situation where the worker node can successfully join the cluster, pods can be scheduled on the worker and inside the pods, dns doesn’t work and the clusterip of the nameserver is not reachable. Pods scheduled on the master have functioning dns. (The remaining 30% of attempts had issues before I even got that far)

The fact that it is always the dns that doesn’t work makes me think that maybe, I’m just forgetting about a crucial aspect of dns, a simple and obvious configuration that I forgot. I’m really starting to feel like a complete idiot

https://redd.it/f2bh7z
@r_devops