Hikaru 0.7b just released with support for K8s watches. Now you can monitor Kubernetes activity with Hikaru, receiving events in the form of Hikaru objects.

Hikaru is a Python tool that provides you the ability to easily shift between YAML, Python objects/source, and JSON representations of your Kubernetes config files. It provides assistance in authoring these files in Python, opens up options in how you can assemble and customise the files, and provides some programmatic tools for inspecting large, complex files to enable automation of policy and security compliance.

https://github.com/haxsaw/hikaru

https://redd.it/p615pw
@r_devops

What would you use to get logs in a "pull way"?

Hi guys, I have been asked to set up logs collection/monitoring for +500 devices that don't support pushing logs to a log server, means I have to do an API request to get the logs for each device. I only have a small idea about the logging solutions out there and I would like to know your input.

Any suggestion is welcome, thanks in advance.

https://redd.it/p62tjy
@r_devops

DDOS protection by using secondary, smaller servers that receive traffic

A while ago I read about a method of protecting your servers from DDOS by basically having separate servers that accept connections from the outside and send the traffic to the main server. You have 2-3 of these smaller servers and in case of DDOS you switch between the smaller servers so when one fails you use another. Meanwhile, the main server is protected.

Does this ring any bells? I don't remember what it was called.

https://redd.it/p64det
@r_devops

DevOps Master Class Part 2 - Master Git

Available now at https://youtu.be/hQJktcBzJUs. Again zero adverts. Full playlist and repo linked in its description.

https://redd.it/p63v4v
@r_devops

Unpacking Observability: A Beginner's Guide

"Fundamentally, Observability is a data problem, and capturing the right data will ensure a properly observable system."


In the latest **#StoriesFromTheHerd**: Adriana Villela, Observability & Platform Solutions Manager at Tucows, shares a beginner’s guide to understanding Observability, why it matters, and how you can get started.

https://storiesfromtheherd.com/unpacking-observability-a-beginners-guide-833258a0591f

https://redd.it/p66dys
@r_devops

From Software Engineering to Platform Engineer Role

Hello everyone. Long story short I am coming from a background of software engineering for embedded systems with a masters in computer science. I was researching+planning to make a career change to DevOps and I applied for the role at a mobile network operator company. They gave me some technical skills tasks (basically an automation python script to GET information from a dummy API endpoint, populate an SQL database in a Docker container) and they told me that while I did very well, I am lacking the skillset in the QA testing and DBA areas for the DevOps role I applied for. Impressed by my submission, they offered me a role for a platform systems engineer to gain exposure to the industry for a potential shift to the DevOps department in the future.

I am trying to research and gain knowledge on the role responsibilities however I am having trouble finding information with respect to the telecom industry. Can someone explain and/or provide examples of the following with respect to the mobile network telecom industry:

1. Monitor and maintain infrastructure performance, capacity and availability, providing second and third level support for the resolution of incidents and operational issues.
2. Implement supporting database and middleware capabilities to support software module development and operations
1. What are middleware capabilities?
3. System KPI monitoring and analysis to pin point improvements
4. Working knowledge of database technologies, scripting and/or programming languages with the aim to increase system efficiency and lower human intervention time on any tasks
5. Working knowledge with Micro-Service architecture
1. They also require supporting and maintaining Java Applications so I am guessing Java Spring boot is one of the micro-services. What are other micro-services can be included in the architecture.

Apologies for the information overload. I am really excited for the career change and I want to do everything to shorten the learning curve before my official start date next September.

Thanking you guys in advance.

https://redd.it/p67n4k
@r_devops

Any useful anecdotes regarding the use of containers in vsphere

Looking at pairing down VMs with some vpshere upgrades and are weighing the pros and cons of containers in that environment.

It just seems like a weird way to go about it, but if it's less resource intense than a VM, that's likely to make my life a lot easier.

https://redd.it/p670ky
@r_devops

When varnish returns a cache, does it also return the cookie generated on the backend or only the css, images and html returned to the frontend?

When varnish returns a cache, does it also return the cookie generated on the backend or only the css, images and html returned to the frontend?

I had a problem where users would get the wrong cookie, so I am thinking it was due to the varnish caching, is that correct?

https://redd.it/p655y2
@r_devops

Any of you taking advantage of remote working and working two jobs?

I have been tempted to this since the start of the pandemic, but can't cause of childminding duties. It'd seem quite doable if you didn't have many meetings in a day

https://redd.it/p63tn0
@r_devops

Should I learn CI/CD or learn Containers?

My new job is basically AWS support and I work under Deployment. They will train me. Which domain or skill will be much more marketable now and down the line? CI/CD or Containers? What jobs are available out there that demand these skills?

https://redd.it/p6cr8a
@r_devops

Managing ENV vars and secrets

Hey everyone! Looking for some advice trying to centralise ENV vars and secrets.

Right now we have an application on k8s with way to many env vars. We need to keep a set for testing, staging, and production, a set for the CI/CD pipeline end-to-end tests, and a set for each developer.

Currently we have copies of everything at every place it's needed, which makes updating them a pain in the ass. What methods and tools are you guys using to centralise these things? Is there an API I can call to CRUD on env vars and secrets that's secure by default?

Thanks for the help!

https://redd.it/p635qu
@r_devops

Automating customer notifications?

hi, how do you automate your customer notifications for application maintenance? We're looking at Everbridge, but they're quite costly.

https://redd.it/p6276s
@r_devops

How are you protecting your staging environment(s) for your web apps?

By "staging environment" I'm referring to any non-production environment that you need to access over HTTPS, either ephemeral or long-living and potentially shared.

Are you using basic auth? If so, how are you managing usernames & passwords?

Are you using a VPN?

Are you using something else?

https://redd.it/p6fwax
@r_devops

Validating RESTful-ness of Flask API

Hi folks

My team maintains a small Flask API as part of our application, and we've run into a few scenarios where the endpoints we've written don't conform to the OpenAPI standards. Are there any tools out there that allow you to check compliance of a Flask API again the OpenAPI standard?

I did find https://github.com/p1c2u/openapi-spec-validator and https://github.com/p1c2u/openapi-schema-validator, but I'm pretty new to the devops world so I'm not sure if these are exactly what I'm looking for. Any recommendations or advice is appreciated.

https://redd.it/p6gjn2
@r_devops

Announcing SurplusCI - dedicated runners for your CI Pipelines

Hey all,

I just launched a new service called SurplusCI (https://surplusci.com), and would love if you gave it a try -- there's a demo which is only half a vCore but since it's dedicated it's not TOO terrible (I should probably raise that to 1 or 2 vCores so people get a better feel for the service).

As everyone here is ops-saavy -- it works just like you'd expect it to! I spin up VMs when you ask for more capacity and make sure the compute and storage scale as they need to. Would love to take questions on the infra if anyone is interested as well :). It runs on Kubernetes and uses some interesting addons/plugins (I actually use both KubeVirt and Kata-Containers).

r/devops members please DM me for a month free, I'll discount your subscription to zero, would love some feedback even if you end up not using it long term! I always wonder how much effort I should put into support Azure pipelines/BitBucket/Bazel and some other options. Hearing from people who are "plugged in" so to speak is valuable to me.

https://redd.it/p6gfdb
@r_devops

In Search Of: LOPSA Leadership Committee volunteers

Hi everyone, I am on the board of directors for LOPSA.org, the League of Professional System Administrators. We are a 501(c)(3) non-profit. The board is looking for volunteers for our Leadership Committee. This committee helps with our board election process including helping us find qualified candidates and running the election process. If anyone here would like to help volunteer for this committee, please reach out to me and let me know. I look forward to speaking with you and answer any questions.

https://redd.it/p6j0ui
@r_devops

How to deploy a Rails application using Chef these days?

Chef used to come with a cookbook called "application" which allowed to deploy web apps in different frameworks including Rails. That was back in Chef 11 days. Then breaking changes happened one after another, and I cannot even find the application cookbook on google. Is it replaced with another cookbook, or did Chef take a different path or what?

https://redd.it/p6ir2q
@r_devops

I want to log everything on kubernetes

Hey guys, I'm a junior DevOps and wanted to know how do you store your kubernetes logs.

I searched and investigate tools like graylog and Loki, but wanted to see if there's more.

Currently I'm using azure cloud so my clusters have azurefile as PVC and storage class.

https://redd.it/p61nld
@r_devops

terraform on mac m1 (arm64) ?

I need to install [email protected] but it looks unavailable: https://releases.hashicorp.com/terraform/0.14.10/

how can I use this version of terraform on the m1 mac? I've been trying/researching for a while.

https://redd.it/p6lm9r
@r_devops

DevOps Tools landscape chart/diagram generator

My manager asking me for some flashy diagram listings of all the devops tool chains we are using. I find it hard creating using powerpoint. Any easy online tool/generator, if anyone knows.

https://redd.it/p61e80
@r_devops

Federated prometheus and integration of kube-state-metrics

I have a number of k8s clusters which have installed a prometheus instance(using prom operator). I have a central separate prometheus which collects all metrics(cadvisor, kubelet) from all the k8s clusters using the /federated endpoint(using a load balancer service). I have recently installed kube-state-metrics on some of the clusters and i was able to expose those metrics using a separate LB service.


My end goal is to be able to have only one LB service exposed from each cluster and expose all metrics from that endpoint.


What i want is to be able to merge all metrics (cadvisor, kubelet, kube-state-metrics) into the local prometheus and expose all metrics externally using the federated endpoint of the local prometheus.


I was unable to find something similar online but i am sure other people have had experience on how to do this.

https://redd.it/p6mzv2
@r_devops