Why aren’t ChatOps more popular?

A co-worker asked me this question and I’m curious to hear what other people think.

To be clear, I mean running/monitoring operations from chat interfaces/clients. Not using chatbots (like Hubot)

https://redd.it/exwb9u
@r_devops

Can Nginx handle thousands of virtual hosts?

Hi all. I'm working on an app hosted in Kubernetes that lets users add custom domains. Each time a domain is added, the app creates an ingress (virtual host) and a certificate in Kubernetes. The database used by Kubernetes (etcd) can only scale up to a point because of db size limits, plus there will be a limit to the number of virtual hosts that Nginx can handle. Because of these two reasons I am going to re-architect the app so it can spread users across many small Kubernetes clusters instead of a single big one. Questions:

1. How many virtual hosts can an Nginx instance handle comfortably from your experience? Does anyone have experience with, say, thousands of virtual hosts or more? I will need to do some testing but I would like an idea of what to expect more or less.
2. When there are many virtual hosts, can it happen that Nginx drops connections when reloading the config when a virtual host is created/updated/deleted?
3. Is Nginx quick enough when reloading a config that includes many virtual hosts?
4. Would something like Traefik or Haproxy be any more efficient with many virtual hosts and with dynamically reloading/updating its configuration?

Thanks a lot in advance!

https://redd.it/exti34
@r_devops

pipeline job not parsing IP's correctly

Hi,

​

I am trying to create a job that does an SCP to a couple of servers.

The list of servers comes from Extended Choice Parameter plugin: (values for instance: [1.2.3.4](https://1.2.3.4),[5.6.7.8](https://5.6.7.8))

The stage that SCP's is as follows:

stage('copy'){

sshagent (credentials: \['deploy'\]) {

def ips = "${environment.split(',')}"

for (ip in ips) {

echo ip

}

}

}

​

But instead of getting the IP's I get them separated to each line, for instance:

1

.

2

.

3

.

4.

​

And so on...

​

Can someone guide on how to correctly tim the IP's from a list?

https://redd.it/exq5l1
@r_devops

Designing for Reliability - Using an Internet Door Buzzer as an Example

Hi Reddit - I created an internet door buzzer a while back and being an SRE I decided to describe why I over-engineered it a little bit to increase reliability by a lot. Wrote a thing about it:
https://medium.com/@lutraman/designing-for-reliability-using-an-internet-door-buzzer-as-an-example-eb654b7121d0

https://redd.it/exm1sj
@r_devops

Launch Darkly vs Split.io

Guys,
I am new to feature flags - would love to hear about pros and cons of both Launch darkly and split.io, from people who have used both. And any other alternatives (suitable for big enterprises) that we should consider?

Thanks

https://redd.it/exthn9
@r_devops

Dumping initial data for dockerized MySQL database image.

Hello,

I'm trying to figure out what's the best way to dump related data from a cloud database to create a local development environment via docker image, but I'm having trouble thinking of a solution to getting all related to an record in a table. The database runs on MySQL 5.7

I'll try to explain what I mean:

* Table A is related to Table B and Table C and so on.
* Table B is related to Table M and Table N...
* Table M is related to Table V...
* Table V is related to Table Z

What I want to achieve is make a script/tool/whatever that would extract the data from all nested relation tables from A through Z by the ID(primary key) of a record in Table A.

https://redd.it/extwhz
@r_devops

Ansible playbook for 2 Node HA was clusters Active/Passive config with Pacemaker?

Looking for a good playbook that configures two Ec2 instances ( provisioned by terraform) in an active/passive state using Pacemaker, a cluster resource manager.
The way Pacemaker works is setting up a node as primary then copy an auth key to the secondary node.
What Ansible commands should I use to copy/move files from EC2-a to EC2-b? (I can use scp or copy/fetch).

https://redd.it/exmm3q
@r_devops

Dockerized Vim/Zsh/Tmux/Whatever IDE for more powerful code editing on server

Part of what I do in company where I work is devops and from time to time, I do development of tools right on servers.... (for example, I have been doing some networking stuff, with specialized hardware, I can test only on dev servers)

​

For minor file editing, it is more than ok to just use default vim config, or to clone minimal one, but, for bigger projects, I do prefer my whole vim setup...

​

Unfortunatly, it has grown a bit and has some dependencies, so, I have decided to create docker container out of it and to include some of my favourite tools, like jq, fzf, etc.

​

You can take a look at it at: [https://github.com/nemanjan00/dev-environment](https://github.com/nemanjan00/dev-environment)

​

I did put most of my dotfiles right into docker file...

https://redd.it/exzoyl
@r_devops

Bamboo Specs tutorials (for Bamboo 6.8+) ?

Hi all

So right now, the company I'm working for is focusing mostly on using Bamboo instead of Jenkins (much to my annoyance...). And it's beginning to hurt my job position.

I don't know Java, in fact my attempts at learning it have just lead to frustration and rage quitting. Does anyone know where a can find an up to date set of tutorials for Bamboo Specs?

I've got access to Pluralsite and LinkedIn Learning.

Thanks in advanced

https://redd.it/ey5n4l
@r_devops

Traffic shadowing - tools and advice

Hey, need to reproduce my production API inputs to a staging/dev server and ensure our next version can successfully process this real-world input. What tools did you use for that? any other advice and lessons you learned during this journey?

https://redd.it/eydm1z
@r_devops

Using output from a Terraform var in same module

Im using a Terraform [main.tf](https://main.tf) file and creating an AWS instance and attaching a VPC, EIP, gateway etc, and also configuring an RDS instance.

​

output "rds\_endpoint" {

value = "${aws\_db\_instance.default.address}"

​

terraform output gives the DNS name

and nslookup provides the IP address.

export db\_host = 192.x.x.x (ip address of RDS instance)

I'm trying to pass this variable as an environment variable using local-exec but its not possible as it says output is not defined. If I create another terraform module such as [database.tf](https://database.tf), can I then pass on the output to my [main.tf](https://main.tf) file? Not quite sure how this works..

Basically I want to SSH into a machine and pass this env variable to a Python script so its know the IP to access.

​

\# MySQL configurations

app.config\['MYSQL\_DATABASE\_USER'\] =$user

app.config\['MYSQL\_DATABASE\_PASSWORD'\] = $pass

app.config\['MYSQL\_DATABASE\_DB'\] = $db\_name

app.config\['MYSQL\_DATABASE\_HOST'\] = $db\_host

https://redd.it/eyh9l3
@r_devops

Teams of DevOps teams

Hi,

Apologies if this is slightly off topic (and rambling).. I'm asking for advice/insights on dealing with changes that DevOps enables in an organisation, rather than DevOps practices/tooling itself.

For context: the adoption of DevOps culture at the company I work for has allowed us to move towards "service teams" - basically autonomous DevOps teams that run a range of services e.g. customer-facing web apps, but also a lot of internally focused services - k8s platform, internal monitoring, security logging, development tooling etc. Each team runs many services, but each service is run by only a single team. Generally this model has has worked well for us so far.

Currently, we're at about 12 teams geographically distributed (and growing). Our primary tools for communication *between* teams are a Wiki, MS Teams (🤮but moving towards slack) and email.

Our problem is we're struggling with, what I think is essentially the N(N-1)/2 communication problem.

As we scale the number of teams, things are starting to get a bit more chatoic/brittle, and signal/noise ratio is getting worse. As an example, a customer recently suffered an outage because one team hadn't seen an important announcement, dropped into in a particular slack channel, by another team.

I'm curious if anybody else works in an organisation where this is a problem and whether they have insights, products or strategies they adopted to solve/mitigate it.

One avenue we are looking at is the "Team API" concept discussed in [team topologies book](https://teamtopologies.com/book), which looks really interesting. The Team API, from what I understand, is basically a agreed framework that standardises how teams interact within an organisation. It wouldn't necessarily solve the N/(N-1)/2 problem, but formalising and standardising might hopefully might make the problem slightly more manageable.

Jeff Bezos' "API mandate" is probably an extreme example of a Team API, but I've not seen any concrete examples of "Team APIs" being applied in much much smaller organisations.

Anyway, I'd be curious to hear from anybody else suffering the problem, or others with insights into how they've scaled DevOps teams.

Thanks!

https://redd.it/eyehxy
@r_devops

MS Teams down after they forgot to renew cert

[https://www.theverge.com/2020/2/3/21120248/microsoft-teams-down-outage-certificate-issue-status](https://www.theverge.com/2020/2/3/21120248/microsoft-teams-down-outage-certificate-issue-status)

https://redd.it/eycibi
@r_devops

Help deciding cloud architecture for IOT product

Hi!

I am working with an IOT startup who are currently working to create a proper backend for their product.

Basically we have a monitoring system which generates a packet, and we want to send this to the cloud and store it in a time series database. Next we run some algorithms that we have written based on the received data and generate insights which we send back to an android application.

As we don't have any proper backend yet, we are flexible with the suggestions for architecture but we have to look at the costing. I don't have much experience developing for any cloud architecture but I'm ready to learn. Could you please suggest what architecture would work best for the above mentioned scenario?

Thanks!

https://redd.it/eybp9n
@r_devops

Gitlab-CI cache options

Hi guys!

I’m really new gitlab user but last week i got a project to optimize our gitlab-ci with cache. I read the docs but i don’t understand tottaly how is it work.
In my project need to cache composer and npm packages, but i also want to understand how is it work.

Thanks any help!

https://redd.it/eye000
@r_devops

AWS ECS Fargate Metrics

Need Some help regarding AWS ECS Fargate. I have an ECS Cluster running with some tasks inside it.

I want to collect task-level metrics from it and use them for visualization. Data-dog ECS agent does the same kind of work, but its paid. I need a free solution for this.

It could be something like custom-beat that'll export Fargate metrics to Elasticsearch.

https://redd.it/eyb9gw
@r_devops

What makes a DevOps standup/scrum etc successful ?


I work on a small team at a \~100 person tech company.

2 "Sr. SRE" (including myself), a DBA, a buildmaster, and a newly promoted "Manager, DevOps" who is reasonably technical but was previously in a Sr. customer success type role sortof like a sales engineer but who was on our team. There is a totally separate small team that handles "Production Operations" and I have significant overlap with them in terms of responsibilities, often more than my own team, The only difference being I generally work on stuff before it is deployed vs after, in theory at least.

​

We've been having a monday morning standup call/meeting since I started over a year ago and frankly I feel like it's lacking and unfruitful. Nobody really comes prepared, we just rattle off what we are planning to work on that week, I sometimes references my jira queue etc, but there is little coherence to the whole thing and no followup on it, the next monday just rolls around and we do the same thing.

​

I've talked to some other companies that do standups daily or 4x/week. It seems like we should have a list of items prepared and then review our progress as a team at the end of the week or something.

​

So what makes a successful scrum, what should a lead of a small team like this be doing to ensure productivity and success?

https://redd.it/eyb76i
@r_devops

Junior IT here, trying to utilize Ansible for the first time, looking for advice of whether my first project is over my head

Hello eveyone,

**Background story - not mandatory**

So I've been working as a Junior sysadmin for almost 3 years now. I started off as help desk in my company but as time passed, I learned a bunch of things and really got into automating stuff with PS and Python.

I recently started watching some YouTube videos about Ansible and would like to learn it. I don't have much experience with those sort of things so I wanted to get some advice from this sub on whether my first project is too big for me, or if it's actually doable.

I'm part of an IT team. The guy who is in charge of our virtualization environment is pretty old fashioned and he likes to do things manually. We have an environment of over 1600 VMs and each week we deploy more. Even though he told me he wouldn't use it, I wrote a PoSH GUI script that clones a bunch of VMs at once. He'd get tickets asking for 20 new machines and would just clone them manually, so I helped out and he does use it nowadays.

--

**End of story - The thing I need help with**

I want to help on another matter and that's VM deployment customization. Our senior sysadmins usually deploys a bunch of CentOS/RH VMs and he configures each of them manually for stuff like how many disks the machine will have, size, partition type, file system type, etc.

Is there a way for me to create a script or a play book that will do those sort of things for me? For example, our CentOS template has 2 disks in it, both LVM and with a fixed size. If I deploy that template and edit it before I click "finish", the cloning will fail, so I just deploy it, then edit the disks, format it, etc.

Ideally, if it's doable using Ansible and if I understand playbooks correctly, then I would write a file in which I specify:

* Number of VMs
* OS
* Partition size
* FS type
* Since we're using LVM - PV, VG and LV names and paths.
* And other VMWare stuff like Datastore, hosts, folders, etc, which seems pretty straight forward.

I'd like to know if this project is beginner friendly, assuming I know my way around VMs and some basic scripting.

If it is, I'm not sure where to begin, what I need or what tutorial to follow first, so any tips would be great :)

Thanks!

https://redd.it/ey6wcc
@r_devops

Demo of multistage and canary CI/CD and cloud automation for microservices with GCP, Kubernetes, Terraform, GCB and ArgoCD/Rollouts

Just did a pretty complete and fully-featured end-to-end [automation demo](https://github.com/ilya-lesikov/gke-demo) on GKE. I think this might be helpful for both beginners and people with some experience.

What's in it?

* Multistage deployments (staging, prod)
* Canary deployments
* Horizontal pod/instance autoscaling
* Rollbacks, self-healing
* Distributed tracing, monitoring, logging, profiling, debugging

I used GCP, GKE/Kubernetes/Kustomize, Terraform/Terragrunt, GCB, ArgoCD, Argo Rollouts, Stackdriver, [hipstershop microservices](https://github.com/GoogleCloudPlatform/microservices-demo/tree/master/src). Setup/deployment is heavily automated so it will be easy for you to deploy it on GCP with Free Trial yourself.

The demo is here: [https://github.com/ilya-lesikov/gke-demo](https://github.com/ilya-lesikov/gke-demo)

https://redd.it/ey965p
@r_devops

DevOps knowledgebase/wiki

Thought this might be of interest to folks here:

[https://devops.yoga/](https://devops.yoga/)

(I'm not affiliated with this project, just saw it and thought it was cool.)

https://redd.it/ey8wm6
@r_devops