Preparing for growth

Hi everyone!

I've recently released my first app for iOS on the App Store. Right now we've only got a handful of users, but it sounds like we've hit a stroke of luck and are going to be featured in a widely watched tech review TV show. Assuming things go well, we're going to have a large influx of people checking out the app at that point, so I'd like to do some preemptive scaling of my servers. Since this is the first time I've ever had servers running in production I was hoping to get some advice about the best way to proceed.

Details: My backend is a go API and MongoDB database hosted on (separate) DigitalOcean droplets. Both the API and the database are running on droplets with 8gb of RAM and 160gb disks. The server isn't doing a whole lot of continuous work, just login/authentication, periodic data storage/retrieval of JSON files with less than 10 fields (and normal users probably wouldn't create and interact with more than a couple of these files), and interacting with the Stripe API to handle payment processing.

Here are some of the questions I'm thinking about:

1) Should I create more api droplets and use a load balancer to distribute work between them? (This one feels like a high priority)

2) Should I redeploy the api droplet with a beefier processor?

3) Should I increase the disk size of my mongo droplet?

4) Should I make more mongo instances to make the database distributed? (This one seems like it would come with a substantial increase in complexity)

We've got almost 10K of credits with DigitalOcean through a startup program, so (at least currently) the price of running more powerful machines isn't a big issue.

Any and all advice or feedback would be greatly appreciated. Cheers!

https://redd.it/dzdqqk
@r_devops

how to intentionally configure a web server to return 500



I am seeking for guidance. My goal is to intentionally create a 500 to test if the website can properly show our customized page.

I have a web site hosted on website (docker on ec2 with nginx) behand a load balancer. I am aware that I can configure LB to return a fixed response code based on path.

Nginx I wonder what is the best practice to test such test case? I am thinking I can purposely miss configure the nginx to make a 500 page, or do something like this [https://www.digitalocean.com/community/tutorials/how-to-configure-nginx-to-use-custom-error-pages-on-ubuntu-14-04](https://www.digitalocean.com/community/tutorials/how-to-configure-nginx-to-use-custom-error-pages-on-ubuntu-14-04)

Express server: I am wondering how to test to see if our express server catches 500 and displays a custom error page?

Please advise.

Thanks

Jay

https://redd.it/dzeeo3
@r_devops

Looking to do an interesting demo

Hey guys,

Applying for a job and need to prepare a demo of something Devops related. I don’t have a lot of background in the field, looking for something SaaS based to do. Anyone an idea? Have this weekend to prepare.

Thanks

https://redd.it/dyy1hg
@r_devops

Best practices for managing Zookeeper configurations across environments?

We use Zookeeper to store our application configuration and passwords for apps that require database access. Today, these configurations are not in source control and as such, we don't have great visibility into current state across environments.

Assuming the plan is to continue using Zookeeper, what tools and/or methods have you used for managing additions/changes to application configurations?

Ideally, we're looking for a secure, version controlled (we use Git/Bitbucket) solution.

https://redd.it/e06lpc
@r_devops

Those looking for a roadmap to devops career

https://roadmap.sh/devops

Found this and it's probably one of the best explanations of what to learn.

https://redd.it/e031ex
@r_devops

Building ARM images with Packer

An easier way to build your ARM images (multiple modes, platforms supported).

Article: https://mkaczanowski.com/arm-images-with-packer
Github: https://github.com/mkaczanowski/packer-builder-arm

https://redd.it/e073e7
@r_devops

How did you set up Prometheus?

We have several Ubuntu servers that I am looking to set up Prometheus monitoring on. Would love to see what methods, guides, or githubs you guys followed in order to set it up.

Are you exporting metrics to a main server for dashboards? What metrics? Any tips?

https://redd.it/e02jl2
@r_devops

When _ shouldn't _ we use infrastructure automation (eg. leave a database out of cloudformation to prevent accidental deletion)

I'm most familiar with AWS' cloudformation and assume that other infrastructure automation tools are similar. It feels very uncomfortable to put a database or cognito userpool or s3 bucket in a cloudformation template that will also be used for code.


I know that some resources (eg. Aurora, S3) have flags to prevent accidental deletion -- however, I still feel uncomfortable putting them into cloudformation stacks because it then inhibits us from quickly deleting those stacks!


The alternative that I have come up with is a "state-stack" (bad name maybe) that contains resources that are stateful and hold data. Then, we export arns and values from that stack and grab it in other stacks. However, there can be a chicken or the egg problem with this approach... for instance, our networking stack should be able to be torn down, but a database like Aurora depends on those subnets.


Just curious for general thoughts on this topic although I'm sure each application is different.

https://redd.it/e0d3v2
@r_devops

Docker compose vs K8s for local dev/production deployment

Hey all,

Quick question.. I am trying to learn K8s/configs/etc but it is going to take a while to learn it all to deploy my eventual product to the cloud using K8s. In the mean time, I was trying to set up a local K3s similar to how it would run in production.. but alas, also taking a while to learn how to set everything up.

That said, I do understand Docker and Docker compose decently enough to run my basic API gateway, and services that it configures to, for local development.

Is there any major issue with using docker compose for local development and then the product in K8s in production.. that would result in possible issues that are not caught during dev? I am thinking spinning up the basic stuff, a couple of services that are being developed.. and thats it. But eventually when CICD builds/pushes this out to production.. I am hoping there are no hidden gotchas with using docker compose to run things for dev mode.

Eventually I want to figure out if the local K3S configs will work/run the same way in production (excluding any cloud deploy specific things I may need to configure).

TL;DR Can I use docker compose for local dev and K8s for production in nearly identical fashion so that I can get developing now and stop trying to learn all the K8s stuff right now without too much concern that it wont run in K8s eventually?

https://redd.it/e08nxx
@r_devops

Do service meshes like Consul lower the need for Service Accounts & IAM?



https://redd.it/e06zqp
@r_devops

[blog post shilling] A script to automate generating SSH config files and iTerm2 Dynamic Profiles for accessing AWS

An AWS project I am working on has over 300 EC2 instances dotted around various regions. Getting access to anyone of them in a pinch is a bit of a bit laborourious. You log in to the AWS console, search around for the right instance find all the details and then finally connect via SSH.

My initial thought was wouldn’t it be lovely if I didn’t have to type all that regularly, there must be a way to speed up this process. My plan was to use Python3 and BOTO to query the AWS API and store this information in an SSH Config file that will allow me to connect quicker.

**I put together a script which I have posted to GitHub here:** [**AWS-Iterm2-Profile-Generator**](https://github.com/n3rden/misc-python-scripts).

It assumes a couple of things;

* You are using a Mac with iTerm2 installed.
* You have all the EC2 keypairs named exactly the same as they are in the AWS console.
* Your instances have a “**Name**”, an “**Application**” and an “**Environment**” tag ([*Somewhat inline with Amazon’s tagging advice*](https://aws.amazon.com/answers/account-management/aws-tagging-strategies/)). If you don’t meet these requirements, it’s fairly trivial to modify the script to accommodate your requirements.
* You SSH to your instances using the internal IP address (*over a VPN or direct connect or whatever*)
* You have an Access ID and Secret Key ‘exported’ as variables in to your terminal session ([*for more information see here*](https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/setup-credentials.html))

If you can think of anything interesting I can add to make this more useful or improve things, please let me know.

​

*I typed this up in a bit more detail* [*here*](https://medium.com/@aidenvaines/automate-generating-ssh-config-files-and-iterm2-dynamic-profiles-for-accessing-aws-bb5dd684fe0f)

https://redd.it/e0jwie
@r_devops

An Open Source Stack for Managing and Deploying Machine Learning Models for Data Science Teams - DVC & Cortex - Tutorial

When a team of data scientists and engineers working on a model, things can get messy. Data is constantly being copied across machines, tweaks are being made on an ad hoc basis, and eventually, you end up with a model you cannot explain or reproduce.

Data Version Control (DVC) tool solves this problem using a version control workflow that will be immediately familiar to anyone who has used Git, DVC stores your model weights and training data in a centralized location, allowing collaborators to get started easily, while also tracking changes and ensuring an accurate version history.

In this tutorial, we’re going to use DVC to create a model capable of analyzing StackOverflow posts, and recognizing which ones are about Python. We are then going to deploy our model as a web API, ready to form the backend of a piece of production software.

While DVC makes your machine learning experiments reproducible, to deploy your model as a production backend, we’ll need another tool. As a final step in this tutorial, we’re going to integrate DVC with another open source tool - Cortex - that allows us to deploy DVC-generated models as web APIs, ready for production.

Full article: [An Open Source Stack for Managing and Deploying Models](https://towardsdatascience.com/an-open-source-stack-for-managing-and-deploying-models-c5d3b98160bc)

https://redd.it/e01uoo
@r_devops

Can you save battery by running Docker remotely?

My hypothesis is that if we could use a remote server for Docker builds, we could save battery. Let's look at a few ways we could do that, then try one of them out and finally review the experience.

[https://blog.alexellis.io/save-your-battery-with-docker-over-ssh/](https://blog.alexellis.io/save-your-battery-with-docker-over-ssh/)

https://redd.it/e02r1e
@r_devops

Setting up logging and monitoring infrastructure

Hi,
I wish to set up infrastructure mentioned in title, but at this point I am a little bit confused by variety of tools and how they can be connected in different ways.
My constraints:
1.low cost and low maintenance effort (I can spend some time to set it up right but I might not have enough time to take care of it when it suddenly crashes too often).
2. Scalable

For logging I was thinking about:
Client -> Apache Kafka -> Graylog +Grafana

For monitoring:
Client -> Telegraf -> Apache kafka (the same as before) -> Telegraf (again?) -> Prometheus (+ Grafana?)

I would like to be able to create some alerts) notification in future as well as add some custom metrics as needed.

Do I get this right or some connections are not possible/duplicate each other? Any advice, alternatives, as well as info about any of mentioned tools is very welcome.

https://redd.it/e0ilwe
@r_devops

AWS ElasticSearch Kibana

How do I access Kibana PUBLICLY ( with login ) in AWS Elasticsearch ?

https://redd.it/e0kgxa
@r_devops

[Packer] - Automatically configure network settings for VirtualBox images (*.ova) ?

I am setting up an \*.ova" VirtualBox image with Packer and later provide it with the necessary confiugrations over ansible. However, configuring the network adapters seem to be an absolute pain. By default when nothing is explicitly specified in my template I can not reach the guest from the host. So, I tried adding a bridged adapter with vboxmanage.

However, now Ansible is not able to connect over SSH anymore to do the configurations.

How can I ensure that the network adapters are configured in a way that I can always reach the guest from the host (all via the packer template, that is) ?

Hope to get some hints, as I am relatively new to the subject.

https://redd.it/e0k5ji
@r_devops

Vulnerability Management Tools

What are people using for their Cloud hosted VM's in term of vulnerability management and scanning?

I've recently been look at Qualys ,RAPID7 and Tenable

https://redd.it/dyli9u
@r_devops

Sending message to all service instances.

Hello, I have a situation where a message has to be sent to all the instances of the service running in the cluster(messages are for increasing or decreasing the amount of logs for a particular customer). For example, if there are 200 instances of the service are running, now i need to send info to all the instances. Have you ever faced a situation like this? What is the best approach you would provide for this problem?

One approach is to use RabbitMq and send a message to all the service instances. But the rabbitMq is not built for this use case, and I'm not sure if this approach will scale. Might need to maintain a queue for each instance.

https://redd.it/dyjz62
@r_devops

Tell me why I should not use kubernetes

Hi r/devops. Lately my company has been looking into possibilities to launch our service across the globe (using cloud providers) closer to end customers. We are now running several Rails apps with Redis and Postgres on our own servers in datacenter. Everything is managed by Ansible and some Terraform scripts.

Some parts of our app are containerized (and running on AWS ECS currently). We are pretty happy with it. But given toolkits available and availability of managed kubernetes (AWS EKS, GKE, Digitalocean etc..) we started looking into it. It looks promissing so now we are evaluating if we should switch to it.

Please give me as many reasons as possible not to switch to managed Kubernetes and stick to traditional VMs running rails, ngunx, redis, postgresql etc managed by terraform and Ansible

https://redd.it/dyiejk
@r_devops

I wrote a script to create a working ECS cluster on AWS using the AWS CLI.

It creates the ECS cluster and all the required elements; a task definition, service, EC2 instance, security groups, load balancer and target group. The end result is a simple Hello World app.

There is a clean up script too to take it all down afterwards. Hopefully it'll be useful to someone trying out ECS

[https://gist.github.com/gordonmurray/259eb3c52e66188ea4b0e3b420a6ccd8](https://gist.github.com/gordonmurray/259eb3c52e66188ea4b0e3b420a6ccd8)

https://redd.it/e0q7c9
@r_devops