What to use for fast startup VMs

I'm working on a game server matchmaking system and I would like to be able to essentially create a VM on the fly to host a match when I determine that I have enough players to play a game. Nothing else on the VM is going to be running except for the game server itself.

I was looking into IncludeOS, which is a unikernel that advertises sub-second startup times, but getting unreal engine to compile with that would be insane. I'm looking at Apline and ClearLinux now, but I'm wondering if anyone has any advice on this before I dig too deep.

https://redd.it/dpb4rm
@r_devops

CI configs should be documentation

I've been toying with this idea for a few years - Whenever I try to follow documentation it's almost always out of date (especially the "install dependencies, start database, and get started")

I wrote an article (and made a product) around the idea that CI configurations themselves can act as documentation - Would love your feedback! https://layerci.com/blog/ci-configs-should-be-documentation/

https://redd.it/dpczar
@r_devops

Deploying with helm to a private GCP cluster

I have a CICD pipeline using circle and helm to deploy. I've been trying to set up a k8s cluster to send all egress and ingress through a NAT but I can't seem to find the configuration to allow deployments with helm when the cluster's traffic goes through the NAT and no external IP to the cluster. Are there established processes for deploying to a private cluster? I've seen this https://cloud.google.com/blog/products/devops-sre/deploying-a-production-grade-helm-release-on-gke-with-terraform and the associated repo as the closest thing to what I'm after but I'm wondering if there are non-hashicorp/completely native to GCP solutions.

Edit: I'm very new to devops and networking especially. If something is unclear, I will explain more. Thanks!

https://redd.it/dpdg1y
@r_devops

What is your team that does DevOps work inside your IT department called?

You might have other duties as well, as not everyone is necessarily exclusively doing DevOps stuff.

A lot of teams had the word "infrastructure" in them, but that isn't strictly accurate anymore, especially if you've gotten rid of your physical servers.

https://redd.it/dpdcmm
@r_devops

Any cool work related projects you guys are working on?

I'm working on getting terraform to a usable state for my team (running in to bugs and have git issues open) with their cloud backend service and AWS.

What about you?

https://redd.it/dp8dy6
@r_devops

Get up to speed on EKS and CloudFormation

Looks like at work we are going to be going with EKS and building it out using Cloudformation as we use CF for everything else. Still waiting on final word but hoping EKS.

I'm looking for resources to get me up to speed on best practices and potential issues that I am likely to encounter. I'm the only DevOps person and already know Kubernetes fairly well, on the skilled beginner/intermediate side of things.

My current plan is to create the EKS cluster, hook it up into what existing IAM and SG we have if possible, then containerize an example application (ours but only 1 part of the whole thing), idea would be to have an endpoint to be able to connect to and be able to send a HTTP request to the endpoint.

What resources will help me get up to speed on what I need to/should know for this?

https://redd.it/dpbdih
@r_devops

Appside secrets management.. How are y'all doing it these days?

Hey folks,

Got a design question.

My company is improving the way it looks at cybersecurity, and one thing that came up was the need to both version control configuration data but to prevent secrets from being leaked through that same vector.

Presently, our apps pull config data from environment variables, which are populated on deployment via an ansible push mechanism.

The threat model has the following vectors:

* A developer virtual machine image being leaked
* A developer virtual machine getting pwned via a network level attack
* Somebody getting access to our git remotes
* Disgruntled employee syndrome
* Somebody getting access (any access) in which they can expose Pam env vars on one of our service instances

Presently, we're transitioning to using Ansible. Ansible Vault has been looked at, but then there's still the issue of storing the AES key somewhere.

We've considered using AWS secrets manager to store our AES key for ansible vault but are additionally concerned about secrets being located in environment variables. I've considered writing an on-demand retrieval mechanism that statelessly pulls the AES keys from AWS Secrets Manager, but then there's the issue of round trip time to actually fetch the key each time. The application runs on a mix of node and PHP, so caching the key natively in application memory isn't really an option.

Reddit, wat do

https://redd.it/dp67io
@r_devops

How to wait for Postgres before running API services through docker-compose when psql can't be installed?

I have a docker-compose.yml file with a database and multiple APIs that I'd like to launch through a simple "docker-compose up" in the terminal. However, I'm running into the issue of my APIs starting up before the Postgres server becomes available.

I saw a solution for using a command statement in the docker-compose.yml file that runs a shell script with "psql" to test the Postgres connection before starting an API service, but the issue there is that my API images only have the API service on them (*no psql, python, ping, etc.*) and I'm pulling those images from AWS ECR so I don't have the option to install those.

To complicate things further, any command statement within docker-compose.yml gets in the way of the API image's command statement from the Dockerfile, which would ordinarily have it run its own docker shell script to start up the service. The Dockerfile that was used to build the image has a command to execute an external script to start the service.

Does anyone know of a solution for this issue? I'm a newbie and lost as to how to proceed.

https://redd.it/dpawwz
@r_devops

how do I add more disk space to my google cloud instance?
https://www.reddit.com/r/googlecloud/comments/dpkfc5/how_do_i_add_more_disk_space_to_my_google_cloud/

https://redd.it/dpknd6
@r_devops

Developers are increasingly interested in and adopting DevOps and Mini apps. Interest and adoption of cryptocurrency and computer vision is also increasing but to a lesser extent.

[https://www.developereconomics.com/resources/reports/state-of-the-developer-nation-17th-q2-2019](https://www.developereconomics.com/resources/reports/state-of-the-developer-nation-17th-q2-2019)

https://redd.it/dpm2dc
@r_devops

Building and deploying lots of microservices using werf and GitLab CI

[Here](https://medium.com/flant-com/building-and-deploying-lots-of-microservices-using-werf-and-gitlab-ci-3ce2b7d19450) is how we build & deploy to Kubernetes (via GitLab CI) many similar microservices (i.e. an application represented by multiple repositories). Our goal is to avoid an obvious pain of copying pipelines/infrastructure configs. We assume that deployments are described by Helm charts and werf is in use.

https://redd.it/dpmev2
@r_devops

What I learned at SREcon EMEA 2019

I recently went to SREcon EMEA 2019, and decided to write up my notes as a blog post. I tried to extract the common themes and trends I noticed rather than just the raw notes from the talks. It's very much my take!

[https://making.pusher.com/hot-sre-trends-in-2019/](https://making.pusher.com/hot-sre-trends-in-2019/)

Please let me know if you have any questions or things you disagree with.

https://redd.it/dpnv4z
@r_devops

Lesson from Deploying an ML Model with Kubernetes to GCP

Article mostly focuses on overcoming certain pain points in deploying a machine learning model and web application to GCP. Uses Kubernetes to deploy a containerized Flask API with a persistent volume for hosting the large model assets (.pkl files). Also, includes a short overview on machine type / pricing optimization for handling the high-memory utilization of the model.

[https://medium.com/@pat\_migliaccio/lessons-from-deploying-a-machine-learning-model-with-kubernetes-to-google-cloud-platform-ab84f71ab311](https://medium.com/@pat_migliaccio/lessons-from-deploying-a-machine-learning-model-with-kubernetes-to-google-cloud-platform-ab84f71ab311)

https://redd.it/dpp0iv
@r_devops

DevOps Projects / Practice?

Is there a service or resource that provides example projects for developing devops related skills? Something like LeetCode except geared towards providing example projects/problems focused around devops?

As an external learner (not currently in the field), I'm itching to test drive some of the knowledge I've gathered over the last few months but am struggling to find ways to practice it. Open to any suggestions and ideas!

https://redd.it/dpvhmv
@r_devops

I'm using ansible tower to provision VMs, blow away, and then recreate. What other tools to integrate or use to make my life easier?

New to automation, CI/CD, pipelining, etc. but am making good progress learning (i think). I create a bunch of VMs with specific IP addresses from a VMWare template using vmware\_guest module in ansible. I configure the new VMs with some additional playbooks, do some work, and then blow them away. Some times I want to tweak the VM names and IP addresses and start over. How do I get ansible to watch my inventory file for changes and automatically run my playbook? Do I need something like jenkins, or is that overkill?

https://redd.it/dptch6
@r_devops

Looking for best service such as sendgrid, ses, Postmark etc for outgoing mail from multiple apps with suppression list

Hi all,

I have decided to use a service to help the delivery of emails from all my apps and websites.

Can someone recommend a service that has a good API, has dedicated IPs ( that is not ridiculously priced ) that will work with office365, and others that mark new IPs as spam, to receive our emails. Has the ability to add email addresses to suppression lists that get bounced automatically. The suppression list would be great if configurable. such as any hard bounce will add it to the list for 30 days. If another bounce happens after 30 days it will add it again for 60 days, and so on.

we dont send many emails, maybe 300 a day at most. But we do require them to get delivered and past spam filters.

https://redd.it/dpxsct
@r_devops

Tutorial: Deploy a Docker Swarm Cluster on DigitalOcean (with Traefik, Let's Encrypt and Swarmpit)

I put together a lengthy tutorial on how to deploy a Docker Swarm cluster on DigitalOcean, together with Traefik (and Let's Encrypt certificates for services) and Swarmpit as a web interface. I realize Docker Swarm might be out of fashion in these Kubernetes times, but still!

I learned a lot by writing it, so see it as a learning exercise rather than a blueprint for production deployments. :)

Let me know if I can improve the tutorial! I really appreciate feedback! :)

[https://lunar.computer/posts/docker-swarm-digitalocean/](https://lunar.computer/posts/docker-swarm-digitalocean/)

https://redd.it/dq2hdv
@r_devops

Hey SREs: What are your favorite metrics to monitor?

I'm going out for an interview as an SRE, and I was thinking about which metrics were most important and why. Which ones do you find the most helpful in preventing and identifying issues?

* Response time, especially for database queries
* CPU and memory utilization per container, VM, and host (as appropriate) to spawn additional nodes
* Minimum available disk space
* Number of (GET) requests to monitor for spikes in traffic and DDOS attacks

Assuming a decent monitoring infrastructure, what else am I missing?

https://redd.it/dq54ur
@r_devops

Monthly 'Getting into DevOps' thread - 2019/11

**What is DevOps?**

* [AWS has a great article](https://aws.amazon.com/devops/what-is-devops/) that outlines DevOps as a work environment where development and operations teams are no longer "siloed", but instead work together across the entire application lifecycle -- from development and test to deployment to operations -- and automate processes that historically have been manual and slow.

**Books to Read**

* [The Phoenix Project](https://www.amazon.com/Phoenix-Project-DevOps-Helping-Business/dp/1942788290) - one of the original books to delve into DevOps culture, explained through the story of a fictional company on the brink of failure.
* [The DevOps Handbook](https://www.amazon.com/dp/1942788002) - a practical "sequel" to The Phoenix Project.
* [Google's Site Reliability Engineering](https://landing.google.com/sre/books/) - Google engineers explain how they build, deploy, monitor, and maintain their systems.
* [The Site Reliability Workbook](https://landing.google.com/sre/workbook/toc/) - The practical companion to the Google's Site Reliability Engineering Book

**What Should I Learn?**

* [Emily Wood's essay](https://crate.io/a/infrastructure-as-code-part-one/) - why infrastructure as code is so important into today's world.
* [2019 DevOps Roadmap](https://github.com/kamranahmedse/developer-roadmap#devops-roadmap) - one developer's ideas for which skills are needed in the DevOps world. This roadmap is controversial, as it may be too use-case specific, but serves as a good starting point for what tools are currently in use by companies.
* [This comment by /u/mdaffin](https://www.reddit.com/r/devops/comments/abcyl2/sorry_having_a_midlife_tech_crisis/eczhsu1/) - just remember, DevOps is a mindset to solving problems. It's less about the specific tools you know or the certificates you have, as it is the way you approach problem solving.
* [This comment by /u/jpswade](https://gist.github.com/jpswade/4135841363e72ece8086146bd7bb5d91) - what is DevOps and associated terminology.

Remember: DevOps as a term and as a practice is still in flux, and is more about culture change than it is specific tooling. As such, specific skills and tool-sets are not universal, and recommendations for them should be taken only as suggestions.

**Previous Threads**

https://www.reddit.com/r/devops/comments/dbusbr/monthly_getting_into_devops_thread_201910/

https://www.reddit.com/r/devops/comments/cydrpv/monthly_getting_into_devops_thread_201909/

https://www.reddit.com/r/devops/comments/ckqdpv/monthly_getting_into_devops_thread_201908/

https://www.reddit.com/r/devops/comments/c7ti5p/monthly_getting_into_devops_thread_201907/

https://www.reddit.com/r/devops/comments/bvqyrw/monthly_getting_into_devops_thread_201906/

https://www.reddit.com/r/devops/comments/blu4oh/monthly_getting_into_devops_thread_201905/

https://www.reddit.com/r/devops/comments/b7yj4m/monthly_getting_into_devops_thread_201904/

https://www.reddit.com/r/devops/comments/axcebk/monthly_getting_into_devops_thread/

**Please keep this on topic (as a reference for those new to devops).**

https://redd.it/dq6nrc
@r_devops

Joel Test for Devops or "How to find out if you want to work for a company"

Hopefully you've heard of the Joel Test for programmers. In case you haven't, they go like this:


1) Do you use source control?

2) Can you make a build in one step?

3) Do you make daily builds?

4) Do you have a bug database?

5) Do you fix bugs before writing new code?

6) Do you have an up-to-date schedule?

7) Do you have a spec?

8) Do programmers have quiet working conditions?

9) Do you use the best tools money can buy?

10) Do you have testers?

11) Do new candidates write code during their interview?

12) Do you do hallway usability testing?



I think that's a pretty good start, but I wanted to start a conversation about the equivalent for DevOps, maybe call it the Humble Test or something lol. Clearly a healthy dev environment will make for a healthier DevOps experience. What do you think? What's your 12 questions a company should answer about their DevOps culture before you should work there?

https://redd.it/dq67ob
@r_devops