Wazuh vs. AIDE?

Anyone have experience running Wazuh for host intrusion detection? Got any good or bad experiences to share?

Following the recommendations in one of the Center for Internet Security's benchmarks, a year or so ago I set up AIDE for host intrusion detection on all our systems. But... it's really not very good. Massive numbers of false alarms, the default rules in Ubuntu are just flat-out wrong in places, takes tons of ongoing effort to fine-tune the ruleset by hand, and, the kicker, no aggregation across hosts so you get 50 identical email reports when the same file changes on a set of hosts. And it doesn't even do real-time monitoring, so an attacker could get in, mess with our systems, clean up their files, and we'd never know they were there if their intrusion didn't happen to intersect with an AIDE scan.

It looks like Wazuh is probably a much stronger option and will fix many of our pain points, but this time before I take the plunge I want to find out if it's going to be just as bad in other ways.

https://redd.it/f2gan7
@r_devops

[Collecting ideas] How would you convince an admin that automation and DevOps mindset is a good thing?

I would love to hear your ideas, how an admin could be convinced that automation is a good thing. (ok, I'm feeling a bit lost with this guy)

Imagine the following sysadmin:

* VMs are created using the mounted ISO in VMware VSphere and the installation is: manually doing the install process for each machine (no templates)
* Docker Containers are started via `docker run` as root on all machines
* After adding a new container, a entry is added to the monitoring tool manually
* `ssh root@<IP>` is the default behavior
* `docker logs` is used for each individual container
* quote: 'git is evil'
* quote: 'Developers are viruses on my servers!'
* quote: 'A single PHP file is better than this shitty database API!'
* quote: 'Software architecture is not necessary and useless bullshit. It just slows down everything!'

Tell me what you think.

https://redd.it/f2fx69
@r_devops

Base OS for "bare-metal" Kubernetes

Disclaimer: I've mainly used already-running k8s clusters on cloud platforms and would consider myself as medium-skilled with it. Had a go at using Kubespray with Core OS Container Linux a couple of years ago but gave up due to requirements of the client to do with egress to external repos.

I'm looking to set up a cluster at home on a fairly resource-limited little server (16GB RAM, i5, 1TB disk). I'm planning to use VMs as worker nodes. I'm running ovirt on that node, but might abandon that as it has its own overhead (and seemingly memory leaks) that might be a bit too heavy for that little box. It would be nice to be able to add other nodes as required with minimal fuss. It's just to play around with k8s and run a few internal services, really.

I ask because I've heard CoreOS is now EOL and Fedora CoreOS is perhaps not quite ready for primetime. Are there any other mature-ish base OSes for kubernetes that can fill the gap?

https://redd.it/f2bbkj
@r_devops

Best CI/CD tools

Hi guys, planning to shift to another CI too, currently using GoCD in my project, can someone please help me with other CI alternatives currently hot in the market which you guys are using or have used in your project along with it's pros and cons. So that I can make a decision as to which one to go with..confused with so many options being thrown at me would be great to get some clarity..

https://redd.it/f2c7bz
@r_devops

The story of how I have spent 5 days getting dns to work in k8s, and it still doesn’t work

It all began when I tried K8s for the first time two weeks ago. I set up a one node one worker cluster on two RHEL8 VM‘s at home. Everything worked like a charm, no problems. Peace of cake, or so I thought.


We use swarm at work. Me, having read about what makes k8s superior, and having set up a little cluster at home: ‚Hey guys, how about I set up this new project with k8s?‘

And there I am. For the last 5 days, I have dedicated every minute of my life trying to get up a working k8s cluster. I cannot manage to get DNS to work inside pods on workers.

- I have worked through multiple tutorials multiple times
- I have tried a load of different linux distros and distro combinations
- I have tried using docker and cri-o as container runtimes
- I have tried cgroupfs and systemd as cgroup driver
- I‘ve tried flannel, weavenet, calico and cilium
- I switched to iptables-legacy
- I tried turning off firewalld
- I configurated sysctl to ensure iptables is not being bypassed
- I tried using internal and external nameservers on both nodes
- I tried nodes in different networks

Most of the attempts were made with two CentOS8 nodes, newest version of Docker and Kubectl/Kubeadm/Kubelet 1.17.2

In 70% of my attempts, I end up in a situation where the worker node can successfully join the cluster, pods can be scheduled on the worker and inside the pods, dns doesn’t work and the clusterip of the nameserver is not reachable. Pods scheduled on the master have functioning dns. (The remaining 30% of attempts had issues before I even got that far)

The fact that it is always the dns that doesn’t work makes me think that maybe, I’m just forgetting about a crucial aspect of dns, a simple and obvious configuration that I forgot. I’m really starting to feel like a complete idiot

https://redd.it/f2bh7z
@r_devops

This Week in DevOps: Feb 10th 2020

Hey Everyone,

I always have trouble keeping up with the latest announcements from AWS, Google Cloud etc... so I created a newsletter that outlines it all in one place each week. I try to give just enough info in the summaries to help you decide if you need to read the whole announcement.

I'm very interested in the communities feedback, specifically what else would you like to see covered in the newsletter? And do you find the summaries to be useful?

You can view the latest edition here: [https://thisweekindevops.com/2020/02/10/weekly-roundup-feb-10th-2020/](https://thisweekindevops.com/2020/02/10/weekly-roundup-feb-10th-2020/)

https://redd.it/f266j5
@r_devops

Automating ALB creation for Lambda

I'm looking into how to create a simple script when given the name of a Lambda function, the script would create and attach an ALB using the VPC, Security Groups, and Subnet info pulled from a get-function-configuration. I'd imagine someone has already figured out how to achieve this but my biggest obstacle is taking the json output from the get-function-config and separating out the parts I actually need. Any idea how to get this done in powershell or linux shell?

Thanks!

https://redd.it/f29n20
@r_devops

jenkins deployment

how do you all do a deployment in Jenkins using a Jenkins file? Jenkins runs as userA but my production app runs as userB.

Do you just set up ssh keys and scp it over or something more robust?

https://redd.it/f2pgv4
@r_devops

AKS seems to auto-update versions of pods without warning?

I wasn't sure if this too nuanced to post directly in r/AZURE or r/kubernetes but does anyone know the answer to this?


Recently (last friday) My team's AKS clusters all failed due to an update pushed to CoreDNS which retired the use of the syntax "proxy" in configmaps. This happened without warning and the pods restarted on their own.

&#x200B;

My assumption is that since CoreDNS is a part of the AKS managed solution, that this is on Microsoft, but my team is too poor to pay for proper microsoft support so it isn't as simple as sending in a support ticket to find out why this happened.


I can't see any indication that there is even any ability to prevent automatic fetching of updates anywhere in Azure and I can't track any configuration in the cluster itself so I am unsure what happened.

https://redd.it/f28y56
@r_devops

Do you think FinOps in your company?

I saw this number recently: "Unnecessary spending linked to cloud waste is forecast to reach $14 billion". I was surprised, but then I have realized that Cloud services can be expensive if you are not careful. I started looking into FinOps, and I wrote an article about things I have learned. Tell me what your opinion is about FinOps, and if you are using this method in your company!
\> [https://www.padok.fr/en/blog/finops-cloud](https://www.padok.fr/en/blog/finops-cloud)

https://redd.it/f2bhix
@r_devops

EBS consists alot of my costs. How to determine optimal EBS size for ec2 instances?

I currently am a greenhorn that just uses AWS for screwing around with stuff. I constantly build and terminate ec2 instances to get familiar with the process. I aim to be a SWE, but I want to know DevOps as well to deploy my personal projects.

On t2.nano AWS allocates 8GB of EBS by default. How much of it is necessary for a simple Ubuntu instance that is not production level, but just a playground to screw around with stuff? I know Ubuntu needs about 2GB, but I always read that when you install Ubuntu on a VM you should allocate at least 25 GB. So I need some clarifications here.

https://redd.it/f27sz2
@r_devops

cache problem with jfrog Artifactory HA on AWS

Hello devops,

does anyone implemented jfrog artifactory HA on aws environment ? if so how did you implement the chaching of the efs server on both instances ?

https://redd.it/f258sv
@r_devops

Logging service for QA environment (simple, fast)

We are looking to making our QA's job little bit easier by having our application(s) log into some kind of logging service with web UI that shows a log tail of currently running services.

I've looked at Azure's Applicaction Insights and while it does have some features that would be useful for our QA, I find that it isn't tailored for use case when QA wants to immediatelly see logs and metrics from last request she made against service she is currently testing.

Any suggestions for logging service that is :

* Simple to deploy
* Simple to use
* Shows immediately all requests made, with all log entries under that request
* Doesn't need to scale much, as there will be little of actual throughput

https://redd.it/f24etw
@r_devops

What to expect in a DevSecOps Role

Received an entry level SWE job offer from a big defense contractor about a month ago and accepted. Today I found out i was put on the DevSecOps team. Does anyone have any experience working in such a team and what the responsibilities might include?

https://redd.it/f2tyv1
@r_devops

How do you started using IaC and conf mgmt tools at your org?

We need to start from zero. The difference between the desired and actual state is super big right now and I don't want to reinvent the wheel. Currently our cloud footprint is quite small and I'm converting everything already created into TF config files, just that; next I would like to create modules from that config.

Some personal experience, workflow or reference would be great. Also...it becomes easier while you advance or just change the kind of problems you have?

https://redd.it/f2txdl
@r_devops

CI/CD for Terraform environments

Hi all,

I was wondering if anyone has encountered the same problem as we have in my team - we have fairly large environments on both Azure and AWS (scaling up to 100s of resources, lots of configuration). Deployment is done 100% via Terraform, including workspaces split for multiple environments (Dev, Integration, PreProd, Prod) and some custom modules we have. All Terraform code is stored in a Git repository with full version control, where about 5 engineers are working on the same environment simultaneously on average.

Issue is, when we started talking about CI (that was the first step) for our infrastructure, we couldn't find any workaround to deployment times - some resources take up to 40 minutes to deploy (for a single resource), which makes it practically impossible to run any kind of testing on the environment (the most basic smoke tests).

Has anyone implemented some CI method for large infrastructures? How did you manage to run a large number of tests (which take time as well, in addition to deployment time) on the environment? Any kind of help would be greatly appreciated.

Cheers.

https://redd.it/f2t3zn
@r_devops

Products to help manage python dependencies?

Is there a product that can act as an internal central package repo for pip? I'd like to control what libraries are available to which systems instead of just opening everything up to the public repo.

https://redd.it/f2rncv
@r_devops

Jenkins pipeline shared library vs parameterized builds

A little (oof... this turned into way more than a little) background on my situation is definitely warranted. I work on a team who is tasked with "modernizing" a lot of the deployment practices in an attempt to move to a Push On Green model for a decently large number of teams in my business unit (~70 teams and service deployables, I think).

We're a fairly large tech org, and have a fair amount of internal infrastructure that orchestrates the rollout to production:

* large "kitchen sink" jenkins shared library
* in-house artifact repository
* in-house deployment manager

Every project defines several jenkinsfiles for various tasks (updating libraries, handling branch deploys, build->QA->Prod workflows, etc)

Due to the combination of "kitchen sink"-ness of the shared library, as well as a considerable amount of organic growth, very few of the deployables are configured the same. You can tell what era a given project was written in by the rough skeleton of the jenkinsfiles it has. There's quite a lot of copy/paste code as teams have incorporated HOTFIX logic in some places, canary deploys in other, etc. And a slew of deprecated library calls that nobody has bothered to go in and update. It also requires a fairly intimate knowledge of the shared-library to do any non-trivial work.

One other drawback of the current system is that there is a fairly tight coupling between the build step and some post-deploy actions. In particular, stuff like JIRA ticket ids get set in the build step, and then comments/links are updated after the deploy.

This all makes iterating on modernization an extremely painful process, since quite a lot of time is spent waiting for a build to complete even though re-using a pre-built artifact would have sufficed.

Additionally, the team that manages the shared library seems to have realized that the current situation is pretty bad and is building a new version to Fix It For Real This Time. But really, it already does look quite a lot better. It's just not ready yet.

Ultimately, it seems like there's quite a lot of opportunity to attempt to standardize on some deployment models.

One thing I'm considering is setting up a handful of "Push On Green Deployment Jobs" that would accept as parameters: a job to build and publish the artifact (manged by the dev teams), a job that runs whatever tests the team wants, post-deploy monitors, maybe some canary logic, etc. To me, this has the advantage of hiding how the sausage gets made a bit and centralizes deploy logic into one place. Using parameters enforces a looser coupling of the build steps, so it should be easier to convert to the new library once it's ready. These jobs would be triggered by the deploy.jenkinsfile of the deployable project.

The other approach, and one that some other business units at my company have used with mixed results, would be to roll our own deployment library that exposes these same tasks, but lets them do whatever customization they want.

Ultimately, I feel like some teams will always want to manage their own deploys, and that's certainly up to them. But I'd like to do as much as possible to let them focus on the things they should care about (building, testing and monitoring) and not concern themselves as much with the deploying/rollback/hotfix details.

Anyway, I'm mostly just interested in whether anyone has had luck leveraging paramaterized builds as a way to manage and centralize this sort of logic.

https://redd.it/f2l4la
@r_devops

Who are you other than your job title ?

Do you have a side gig / What do you do for fun in your free time to break monotonicity of DevOps role and be purposeful ( earn some bucks maybe ) ?

https://redd.it/f2l052
@r_devops

Graduated with CIS degree. IT Help Desk -> Junior DevOps or Entry level cloud positions?

Help Desk to Junior DevOps?

I graduated with a Bachelors Degree in Computer Information Systems and spent the past 3 years in IT Support / Help Desk roles. I gained a great bit of knowledge and experience working in those roles as an intern (at a Fortune 500, at a Wealth/Finanace Company, and at a MSP).

I was lucky enough to accept a great job out of college doing IT & Software Support. About 5 months in, I realize that my heart really isn't in doing Tier 1 type Help Desk stuff (resetting passwords, program installs, recovering emails). I am starting to work with the software support team and realize all of these guys have a lot of coding knowledge in work in DevOps type roles. I figured I worked too hard in college and spent most of that time busting my butt interning to just be complacent with a job doing low level work. I am thankful for where I ended up... but I want to continue learning and climbing the ladder rather than be complacent.

My goal was to start studying for the AWS Certified Solutions Architect - Associate certification. After that, I wanted to grab a better understanding and study up on Linux (I have minor knowledge in Linux atm). After I got those 2 down - I wanted to dedicate my time learning Python and automation.

Is this a good learning track towards a Junior role? Has anyone moved out of a Help Desk role into a Junior DevOps role?

All suggestions help!! Thanks!

Resume: https://m.imgur.com/gallery/u51h9wa

https://redd.it/f2j64q
@r_devops

Scale testing

Hello everyone, I am working on a side project and I was curious on what people do to test large scale infrastructures/deployments. What do people do to do this?

https://redd.it/f317n4
@r_devops