Simple way to keep configs in sync between 2x Apache web servers?

Wondering what a good way to do this is. Kind of reluctant to go down the full blown config management path (ansible, etc), as its literally only 2 servers that need to stay in sync.

 

Was thinking maybe store the configs in git and have a cronjob that grabs it every couple of hours and reloads the httpd? Although thats pretty clunky.

https://redd.it/dvfld3
@r_devops

Gerrit - Multiple Intermediates (Or steps)

I'm responsible for the CI pipeline at work. Current workflow is the following:

1. Push to gerrit
2. Code review and Jenkins Build
3. Submit

I would like to add another round of tests between the jenkins build and the submit. Those are selenium tests, a bit overlong to run and require deploying the application. I don't want to deploy the application and run those tests for every gerrit patchset, but I don't want developpers to be able to merge without those tests having been run.

​

The way I picture it would be the same flow as right now - back and forth until code review and build + ut are done, THEN start the back and forth with seleniun tests, THEN merge to master

​

It seems simple enough to do with a pull request merge. I would just need to have an intermediate branch in between that would include all commits waiting for selenium tests, and they could then be added to master when ready.

​

I can't quite figure out how this would work with gerrit though... Any ideas?

https://redd.it/dvfcrf
@r_devops

Visualize (And control) your helm?

Hi all,

I'm working as a devops for the last couple of years (mainly k8s) - and from my own personal pain i started working on a dashboard to visualize helm deploy, current status, and history (as a first step).

Show diff between versions.

And Later i plan on adding some control (mainly rollback).

I already started developing it but wonder if you some easy/cool tools that does what i need?

Anyway i would love to have some beta tasters for the dashboard, so feel free to contact me :)

https://redd.it/dvh0nm
@r_devops

Container Internals Series | Part 0 — System Architecture

Great article about container internals: [https://medium.com/@tiemma/breaking-down-containers-part-0-system-architecture-37afe0e51770](https://medium.com/@tiemma/breaking-down-containers-part-0-system-architecture-37afe0e51770).

Covers User & System space, syscalls, sysgroups etc. First part in a series.

https://redd.it/dvhkg7
@r_devops

Run Kubernetes+Istio+Knative locally, as a developer

What's the most lightweight/easiest way to run k8s + istio + knative locally as a development environment?

We'd love if our devs could run their services (multiple) locally and utilize Knative Eventing to test their event producer - customer setup.

Most of our devs use MacPROs and a few of them are on Linux.

https://redd.it/dvfvbm
@r_devops

Organising Terrform Workloads

I wrote about how I go about organising, or laying out, my Terraform code.

[https://opsfactory.com.au/organising-terraform-workloads/](https://opsfactory.com.au/organising-terraform-workloads/)

Hopefully you find it useful. Any and all feedback welcome, of course.

https://redd.it/dvn8ce
@r_devops

What is the scale at which companies start thinking about observability/monitoring/logging?

What is the typical scale at which companies/products start thinking about monitoring/observability? Is it based on the revenue, number of people on the dev team or on the number of users using the product on a monthly/daily basis?

I understand that the above variables are correlated - but want to understand of there are any standard accepted frameworks around this? Or is it just that people start worrying about monitoring/observability when things start breaking

https://redd.it/dvodks
@r_devops

Impacts of CI plus Pull Requests for large teams

Are you guys using CI plus Pull Requests for large teams (say 14) and different approvers?

In this situation the order of the approvals may result in some conflits, right? Or drop productivity a little?

Is it worth the cost? Isn't CI offering enough confidence on what is being published.

Thanks for sharing.

https://redd.it/dvj74s
@r_devops

A curated list of best practices for Kubernetes in production

We've curated a list of best practices for deploying and scaling apps on Kubernetes in production.
The list covers application development, administering and configuring clusters.
It's been a hard work collating and curating dozens of other articles and books.
I hope you find it useful.
[https://learnk8s.io/production-best-practices/](https://learnk8s.io/production-best-practices/)

https://redd.it/dvpzsi
@r_devops

Patterns and examples of Terraform projects that enable a continuous flow of changes and improvements

This is a 40 minute talk from GOTO Amsterdam 2019 by Kief Morris, author of O'Reilly's "Infrastructure as Code" & cloud practice lead at ThoughtWorks. Check out the full talk abstract and link below:

[https://youtu.be/1f\_P8ZH5BC8?list=PLEx5khR4g7PKT9RvuVyQxJLO8CZUJzNMy](https://youtu.be/1f_P8ZH5BC8?list=PLEx5khR4g7PKT9RvuVyQxJLO8CZUJzNMy)

People are adopting dynamic infrastructure technologies like the cloud, containers, and serverless so that they can easily make changes to their systems. Defining infrastructure as code should make systems consistent, reliable, and easy to manage, but an infrastructure codebase can easily become a complicated, fragile mess that is scary to change.

In order to routinely change, extend, and improve infrastructure, teams need to have confidence that changes will work correctly and that the impact of failures is low and easily corrected. This creates a virtuous cycle of continuously improving the quality of the systems. Teams can gain this confidence by applying appropriate design patterns and implementation practices.

Kief Morris shares patterns and examples of Terraform projects using pipelines, automated tests, and loosely integrated stacks to enable a continuous flow of changes and improvements. These patterns—which have been used at clients across the financial services, retail, public sector, and media industries—ensure that any part of the infrastructure can be easily rebuilt, minimize the “blast radius” for a given change, automatically validate changes for correctness, security, compliance, and other operational and architectural requirements, and structure projects to enable multiple people and teams to work on systems while minimizing coordination overhead.

**What will the audience learn from this talk?**

* The role of different types of tools and technologies in an evolutionary infrastructure
* Implementing automated infrastructure testing and validation
* Designing pipelines to deliver changes across environments and into production
* Handling data and configuration in a system where infrastructure is dynamically rebuilt, expanded, and contracted
* Advanced patterns for structuring and integrating infrastructure projects and pipelines in larger systems used by multiple teams

https://redd.it/dvq93o
@r_devops

Helm 3 is now released. No more Tiller!

Helm is the defacto package manager for Kubernetes clusters.

See the blog announcement here [https://helm.sh/blog/helm-3-released/](https://helm.sh/blog/helm-3-released/)

Apart from the removal of Tiller there are several other important changes such as the move of requirements.yaml into Chart.yaml

https://redd.it/dvubwy
@r_devops

Flight rules for Git: extremely detailed, scenario-specific standard operating procedures for using Git

I only just found out about this and wanted to share.

Git has become a central part of my devops experience as a Sr Solutions Engineer / DevOps Consultant. Not only do I find myself using it for more and more every day, I'm also constantly creating training materials for users of all ranges of experiences. This looks like a well I will certainly draw from, and use in my day-to-day work.

I hope somebody else finds this as helpful as I do.

[https://github.com/k88hudson/git-flight-rules](https://github.com/k88hudson/git-flight-rules)

https://redd.it/dvtgba
@r_devops

Make GitLab + Jira work together by verifying changes pushed to your repository based on Jira data

[How to install Git pre-receive hooks in GitLab](https://www.midori-global.com/blog/2019/10/29/how-to-install-pre-receive-hooks-in-gitlab) CE (Community Edition) and GitLab EE (Enterprise Edition) to make sure only those changes arrive in your product's Git repository that complies with your company’s or team’s agreed-upon rules.

https://redd.it/dvsoy0
@r_devops

Anyone here go from senior engineer to manager or director? How did you do it and how is it going?

I’m in the middle of trying that jump out now.

https://redd.it/dvt8vn
@r_devops

Need some guidance

Can I manage block servers (nginx) with PHP or JavaScript on the go?

Context: Trying to develop an app to control subdomains on my server (something like the free web hosting companies offer, "mySite.theirDomain.tld"). I'm pretty much done with the UI and in theory it works, if I create a new block server, it handles a subdomain; leaving aside the fact that I have to manually create the block server, I have to restart nginx too.

Is there a way to do this on the go and from the website?, perhaps every time that I create a website, under the hood, the block server is created alongside it to serve the subdomain. Could anybody give me some advice on how to approach this?

Thanks.

https://redd.it/dvtxlv
@r_devops

devopsdays NYC CFP Now Open.

NYC CFP is now open. Please send us your ideas for the March 3-4, 2020 event.

https://papercall.io/dodnyc2020

We are an all volunteer event bringing you talks, open spaces and networking on DevOps!

See you at the NY Academy of Medicine!

https://redd.it/dw1goy
@r_devops

Jenkins with ECR...

Why must getting Jenkins to work with ECR in a pipeline be such a royal pain? I've been at this a while now and made little progress. So far I have tried using just the regular docker way, I've tried the AWS cli and login shell command, I've tried adding in plugins - forget all I tried, and I've tried assigning a role to the EC2 instance that allows ECR access then just doing a docker push.

​

What options do I have to get Jenkins pipelines and ECR to work together? I'd like to save myself the time and just switch to docker hub or another provider but that's not an option for many business reasons.

https://redd.it/dw245o
@r_devops

Load balancer returning full certificate chain or all the clients have intermediate certs installed?

So recently we start adopting HTTPS over all inner resources such as jira, bitbucket, artifactory, teamcity, jenkins everything we have. All the traffic goes through Barracuda Loadbalancer which is kind of like a proxy, so every client connecting to Artifactory to upload packages built by Jenkins is connecting to Loadbalancer in the middle, where the certificates are returned to the client.

And the problem hit pretty hard - Loadbalancer returns only End certificate, and it is signed by the Intermediate certificate(which is signed by Root) and almost all the keystores (except browsers, which are mostly ok) have only the Root CA cert.

So I'm basically start receiving requests from different teams on daily basis that they are having connection issues related to inability of their client (curl, git, java, node.js ..anything) to verify certificate(because there is no Intermediate certificate around.

At first we were working this around just by manually uploading certificates into tools' keystores - like update java 'cacerts' once, and copy-paste it to all the instances on single machine, and then on all other machines(and there are few of javas of course on each machine, gotta search for all of them..).

Before that developers just were turning SSL check off.. which is even worse..

So I decided to dig into certificates to understand what's up, and then addressed a guy who could manage the Load Balancer(the proxy which serves the Cert. Chain to the clients).

And the guy told me that it would be better for us to just use Ansible for all the Linux machines to update all the keystores(differs on different distros as far as i'm getting it) and WIndows would be fine - it has some auto updates on the keystorage.


But my question is - why?

Why manually update hundereds of machines when we could update 1 single server to return 1 more certificate(Intermediate would be enough as far as i understand it at least).

Also need to mention, that for time being all the HTTPS traffic would go through this proxy, anyone goes anywhere using HTTPS would hit the proxy (even nodes of the jenkins server) - so there would be pretty huge load.

Could this thing about the amount of traffic be the reason the guy managing Loadbalancer could deny addition of Intermediate CA certificate into the returned chain?

Or are there any other technical issues preventing tune of the proxy to return not only End certificate but End Certificate+Intermediate CA certificate?

Sorry for wall of text, I'm pretty new to all the SSL stuff this is as exciting as it is confusing, which prevents me from shortening the post.

I'm just trying to understand why such a perfect (from my point of view) decision being declined and the alternative manual, repetitive and not reliable approach being presented?

I hope it makes sense to the readers.

And I need to apologize for my poor English.

https://redd.it/dw1oym
@r_devops

Opentracing to logfile and later feeding to Jaeger

Hi

We are offering small foortprint VMs within my company and don't want to include jaeger/zipkin in every VM.

​

Instead we would like opentracing from micro-services to log to local file in the VM. If an employee reports a problem, we then ask him to send us logs which we should then feed to centrally managed Jaeger/Zipkin server. This also helps us in saving resources as we feed the logs to Zipkin etc only when there is a problem.

is there any way to achieve this? So far reading all documentation seem that opentracing lib enabled micro-services need access to backend Jaeger/Zipkin.

Thanks

Angel

https://redd.it/dw3xqk
@r_devops

Executing script on github hooks?
https://www.reddit.com/r/learnprogramming/comments/dvs2q7/executing_script_on_github_hooks/

https://redd.it/dw6ee2
@r_devops