What are the hardest things you had to do as a devops engineer?

What are the hardest things you had to do as a devops engineer?

https://redd.it/eqmf0s
@r_devops

aws management and tagging

Hi guys, I have a big question for you. Imagine to have a very big aws infrastructure with thousands of resources but nothing is tagged. Now we want to tag everything because of IAM policies and better cost management.

How would you retroactively tag those thousands resources? No way I'm gonna manually tag every single line because it would be impossible.

I see 2 possibilities. (maybe 3)

1. tag father resources like EC2 and do a script to tag child resources (problem with shared resources).
2. activate an auto tagging system that when a resource call another resource a tag is added. No idea how to implement this
3. (tag everything in excel and import all tags. not very devops style)

Thank you for your ideas and suggestions and for sharing your tagging experiences

https://redd.it/eqdp04
@r_devops

Serverless Terraform remote backend

TL;DR:

I build a custom terraform HTTP backend. Link [https://github.com/derBroBro/TerraDepot](https://github.com/derBroBro/TerraDepot)

What do you think about this approach?

​

Long version:

For me, one of the largest problems in terraform is state handling. Especially if you have a lot of terraform projects, different accounts and team members it becomes quite messy.

Even if there are some builtin options like s3, gcp, artifactory or the offerings by hashicorp, they are altogether not what I was really looking for. Some are a mess from the permissions management (s3, gcs etc.) others (like terraform cloud) are more or less an included CI/CD System which we already have.

Therefore I spend some time on my holidays developing a backend as I would need it.

The result can be found here: [https://github.com/derBroBro/TerraDepot](https://github.com/derBroBro/TerraDepot)

What do you think? Do you have any further ideas or hints?

https://redd.it/er1v1n
@r_devops

Apprise API Docker Container - Over 50+ Supported Notifications

Hi guys,

It's taken a while, but I've managed to [build an API](https://github.com/caronc/apprise-api) around [Apprise](https://github.com/caronc/apprise) so that it can be used as a micro-service and/or [sidecar](https://docs.microsoft.com/en-us/azure/architecture/patterns/sidecar) for any project now. Apprise is basically the Swiss Army Knife of notifications; it can act as a universal relay to all of them out there (such as Telegram, Amazon SNS, Slack, etc).

The [Apprise API]((https://github.com/caronc/apprise-api)) is especially useful for system admins and devops who wish to be able to send notifications, but don't want to haul in *Python* libraries into their native *Go*application (*as an example*).

**The System**: [Apprise API](https://github.com/caronc/apprise-api)

**Docker Source**: [Here](https://hub.docker.com/r/caronc/apprise)

**Installation**:

# Retrieve container
docker pull caronc/apprise:latest

# Start it up:
# /config is used for a persistent store, you do not have to mount
# this if you don't intend to use it.
docker run --name apprise \
-p 8000:8000 \
-v /var/lib/apprise/config:/config \
-d caronc/apprise:latest

From here you can just point your browser and/or applications to `https://localhost:8000`

I'd be really curious on any feedback you have; enhancements, and or ideas you have that could make it better.

https://redd.it/er0rja
@r_devops

Azure DevOps Deployment groups

Looking for confirmation on my understanding of deployment groups:

Create a deployment group of like machines i.e. web servers

Install an agent (via the generated script or Azure add-on) on those machines and configure the appropriate demands. This will also register the machine as a target in the deployment group.

Set up a Release to these machines by configuring the deployment group in the Release.

Does that sound about right?

If so, I want to use terraform to create the machines, install the addon that registers them as targets, then automatically kick off a release once they are registered as a target. Does this sound possible?

Thank you for your time and thoughts!

https://redd.it/er1fv3
@r_devops

How do partitions of a database communicate? By microservices?

When a database is partitioned across multiple computers, how do the partitions communicate? Is it appropriate to consider each partition as a microservice? (I relate database partitions to microservices, because microservices are "partitions" of a larger program.)

https://redd.it/er4u30
@r_devops

Unothorized to create Kubernetes cluser on AWS using kops

I'm new to AWS and I'm trying to deploy Kubernetes for some tutorial purposes using kops.For that, I'm using a trial AWS account and I started following [this practical guide](https://ramhiser.com/post/2018-05-20-setting-up-a-kubernetes-cluster-on-aws-in-5-minutes/) to do it but I have an issue with this command :

kops create cluster --node-count=2 --node-size=t2.medium --zones=us-east-1a

which gives me the following error :

I0119 16:14:49.647701 13949 create_cluster.go:517] Inferred --cloud=aws from zone "eu-west-3a" error populating configuration: error querying for valid AZs in "eu-west-3" - verify your AWS credentials. Error: UnauthorizedOperation: You are not authorized to perform this operation. status code: 403, request id: a1f6e99f-76d6-4c3f-95bb-416cc2d67f2e

I wonder if it's not related to some limited access due to my trial AWS account !

https://redd.it/er1jbq
@r_devops

Build a 10 USD Raspberry Pi Tunnel Gateway

In this tutorial I'll show you how to build an Internet Gateway for your home network using a Raspberry Pi and a HTTPS tunnel for just 10 USD.

You can achieve a similar effect of an Internet gateway by enabling port-forwarding on your home router, however there are downsides to this.

An Internet gateway with the [inlets](https://inlets.dev/) OSS project will provide a persistent connection behind the trickiest of networks and let you expose your private and LAN services of choice to the Internet.

[https://blog.alexellis.io/raspberry-pi-zero-tunnel-gateway/](https://blog.alexellis.io/raspberry-pi-zero-tunnel-gateway/)

https://redd.it/equ1cs
@r_devops

Newbie : How do you document your work / knowledge?

Good morning everyone,

I recently started my account on Linux Academy, due to the good reviews and few comments on priors posts and I have to say that is gorgeous, I really like it and I thanks everyone who told me to try it.

---

As for the question, my main doubt right now is how to document and structure what I am learning from them and what I have already learned from myself. Because, if we are realistic I am not gonna retain all the information in my brain...

I would like to know which tools or methods do you use to organize your documentation when you are learning something new.

For the time being I think that what could work is to try to organize my existing repository and try to fit the valuable information from the course into it... but as said I am having a few doubts about this method.

---

As always thanks for your time, and have a great day ^^

https://redd.it/eqvbhp
@r_devops

ubuntu 18.04 auto-heal service on failure

Hey guys, I wanted to know if there's a way to auto-heal a service when he fails or a server restart on ubuntu 18.04

Thanks!

https://redd.it/eqtoi7
@r_devops

GitHub - oslabs-beta/Allok8: ⚡️A pretty swell Kubernetes visualization tool

With a group of enthusiastic engineers, they set forth on a month challenge to create an MVP for Allok8. It’s a conventional and uncomplicated web-based UI Kubernetes visualization tool. Though current tooling is readily available, their goal was to give insights down to the container level, especially when containers become interdependent on one another. They’d love the feedback and support of the open-source community as they’re always looking to grow! Tysm ~

https://github.com/oslabs-beta/Allok8

https://redd.it/er89la
@r_devops

Delayed Notifications Functionality: VictorOps vs OpsGenie

What do you guys think about the delayed notifications functionality on OpsGenie and VictorOps?

https://redd.it/erf5h7
@r_devops

"stress test" application to abuse CPU/RAM/HDD

Hi there,

I want to check and then harden my Kubernetes cluster's stress tolerance by deploying some kind of stress generating application that would:

1) Try to hog as much CPU as possible
2) Try to allocate as much memory as possible
3) Try to saturate disk IO with a lot of operations

The question:

Do you know, can you advice any application that does some harmful chaotic activities, tries to misuse resources, e.g. hogs CPU and leaks memory like there's no tomorrow? Does not need to be Kubernetes specific, just need something that works on Linux ;)

I need this to test if resource limits we apply on our Kubernetes cluster work the way we expect them to work, before deploying production applications there.

Thanks!

https://redd.it/ereddk
@r_devops

What is your code review process?

Do you have a limit for a number of code lines? Do you encourage people not to spend more than an hour on code review? Do you have any metrics that you watch (defect rate, avg time for code review,...), do you use any checklist so you don't forget on important things?

https://redd.it/erckwt
@r_devops

Splitting up long running CI tests into a nightly build

I've recently moved to a company that has some loooooong running tests (whole build takes around 23 hours). I'd like to split the longer running tests into a nightly build (we're using TeamCity). Is there a good way to do this? I can't seem to find documentation on best practices. They also have a bunch of failing tests. Is it DevOp's responsibility to fix failing tests? I've always seen that as a dev/qa responsibility. I'd love to hear some thoughts and ways you've implemented your build configs!

https://redd.it/erdeyt
@r_devops

Introduction to Istio

Folks. I posted an introductory article for [getting started with Istio](https://medium.com/swlh/getting-started-with-istio-524628c025). Any feedback appreciated.

https://redd.it/eri20u
@r_devops

Does anyone have thoughts on Sonatype Nexus? Looking for best practices for importing and exporting existing artifacts.

As stated, Im looking for best practices for importing and exporting repo contents and migrating to and from instances of Nexus. is the API any good ? And if not how have you gone about using it or not.

https://redd.it/erhxav
@r_devops

Nestjs run and watch files in docker-compose

I have a Nestjs project which i'm using as api in a docker container. After convert bundling and serving process to the webpack instead of nestjs itself, i'm confused how to serve and watch file changes in docker-compose with nodemon.

I could easily open two command prompt one for serve and one for watch mode. I wonder how you guys setting up this with containerzation way.

https://redd.it/erhp8e
@r_devops

What would be a good way to let end users manipulate a containerized gameserver in a k8s cluster through a web interface

Hey, I would like to host some game servers in a kubernetes cluster and allow users to manipulate their own „servers“ i.e. pods. Manipulation would include control over gameserver config files but also some control over resource limits.

I‘m not much of a web dev, but I would be able to code a Java backend for this project. Basically the only thing regarding this project I cannot do is creating a custom web interface. Ideally I would find either an existing web interface solution that serves my purposes or a way to make an existing web interface solution serve my purposes.

I‘m not really sure whether this is the right subreddit for this question. But I also don’t know what subreddit would be a better fit.

https://redd.it/erd7q4
@r_devops

Can you recommend network activity logger that helps to find network connectivity issue?

We are using VMs from Google Compute Engine offer. As our boxes are just old good Debian 10 it doesn't that matter which infrastructure is in use.

Our interconnected microservices (Java apps) sometimes reports losses of network connectivity. Between themselves and sometimes to managed products, like Google Postgress offer.

Application traces aren't helpful. Problems are floating.

I checked Google's offer https://cloud.google.com/monitoring/agent/

It looks like it is based on `collectd` and agent's capability is too generic:

https://github.com/Stackdriver/collectd/blob/stackdriver-agent-5.5.2/src/netlink.c

This stats you can get with `nstat` (iproute2). They are too generic (total numbers without any specific src/dst/port directions).

My strategy is to collect data to make an initial guess WTF is going inside a Google managed network. I can be wrong.

I think we need a tool that

* actively probes "ping" or TCP "open" port connectivity to well known hosts/services (DB, config server, LB, redis) including heartbeats of/to "health" endpoints where they are.

* captures stream of network interface for TCP anomalies. I am not network expert, so I don't know definition of TCP anomalies.

I can imaging that reordered packets or missing ACK (or whatever in Richard Stevens' books) or RST may provide important info for troubleshooting.

Of cause we need source / destination / port to narrow problems to particular subnets / services.

I have no clue how Google manages an infrastructure, what maintenance tasks they run and when.

How can a blindness be cured regarding network activity ?

https://redd.it/erldol
@r_devops

Someone asked me to explain k8 to him like he's 5

This is what i told him

​

>Okay.. Let's use this example
>
>Take a look at a cube of sugar.just one cube, it represents a container...
>
>While kubernetes is the sugar box that houses all containers.
>
>With over 400 cubes, I can move them all at once provided they are inside the box.
>
>However, you know it will be difficult to move individual cubes..

https://redd.it/erkstn
@r_devops