Helm 3 is now released. No more Tiller!

Helm is the defacto package manager for Kubernetes clusters.

See the blog announcement here [https://helm.sh/blog/helm-3-released/](https://helm.sh/blog/helm-3-released/)

Apart from the removal of Tiller there are several other important changes such as the move of requirements.yaml into Chart.yaml

https://redd.it/dvubwy
@r_devops

Flight rules for Git: extremely detailed, scenario-specific standard operating procedures for using Git

I only just found out about this and wanted to share.

Git has become a central part of my devops experience as a Sr Solutions Engineer / DevOps Consultant. Not only do I find myself using it for more and more every day, I'm also constantly creating training materials for users of all ranges of experiences. This looks like a well I will certainly draw from, and use in my day-to-day work.

I hope somebody else finds this as helpful as I do.

[https://github.com/k88hudson/git-flight-rules](https://github.com/k88hudson/git-flight-rules)

https://redd.it/dvtgba
@r_devops

Make GitLab + Jira work together by verifying changes pushed to your repository based on Jira data

[How to install Git pre-receive hooks in GitLab](https://www.midori-global.com/blog/2019/10/29/how-to-install-pre-receive-hooks-in-gitlab) CE (Community Edition) and GitLab EE (Enterprise Edition) to make sure only those changes arrive in your product's Git repository that complies with your company’s or team’s agreed-upon rules.

https://redd.it/dvsoy0
@r_devops

Anyone here go from senior engineer to manager or director? How did you do it and how is it going?

I’m in the middle of trying that jump out now.

https://redd.it/dvt8vn
@r_devops

Need some guidance

Can I manage block servers (nginx) with PHP or JavaScript on the go?

Context: Trying to develop an app to control subdomains on my server (something like the free web hosting companies offer, "mySite.theirDomain.tld"). I'm pretty much done with the UI and in theory it works, if I create a new block server, it handles a subdomain; leaving aside the fact that I have to manually create the block server, I have to restart nginx too.

Is there a way to do this on the go and from the website?, perhaps every time that I create a website, under the hood, the block server is created alongside it to serve the subdomain. Could anybody give me some advice on how to approach this?

Thanks.

https://redd.it/dvtxlv
@r_devops

devopsdays NYC CFP Now Open.

NYC CFP is now open. Please send us your ideas for the March 3-4, 2020 event.

https://papercall.io/dodnyc2020

We are an all volunteer event bringing you talks, open spaces and networking on DevOps!

See you at the NY Academy of Medicine!

https://redd.it/dw1goy
@r_devops

Jenkins with ECR...

Why must getting Jenkins to work with ECR in a pipeline be such a royal pain? I've been at this a while now and made little progress. So far I have tried using just the regular docker way, I've tried the AWS cli and login shell command, I've tried adding in plugins - forget all I tried, and I've tried assigning a role to the EC2 instance that allows ECR access then just doing a docker push.

​

What options do I have to get Jenkins pipelines and ECR to work together? I'd like to save myself the time and just switch to docker hub or another provider but that's not an option for many business reasons.

https://redd.it/dw245o
@r_devops

Load balancer returning full certificate chain or all the clients have intermediate certs installed?

So recently we start adopting HTTPS over all inner resources such as jira, bitbucket, artifactory, teamcity, jenkins everything we have. All the traffic goes through Barracuda Loadbalancer which is kind of like a proxy, so every client connecting to Artifactory to upload packages built by Jenkins is connecting to Loadbalancer in the middle, where the certificates are returned to the client.

And the problem hit pretty hard - Loadbalancer returns only End certificate, and it is signed by the Intermediate certificate(which is signed by Root) and almost all the keystores (except browsers, which are mostly ok) have only the Root CA cert.

So I'm basically start receiving requests from different teams on daily basis that they are having connection issues related to inability of their client (curl, git, java, node.js ..anything) to verify certificate(because there is no Intermediate certificate around.

At first we were working this around just by manually uploading certificates into tools' keystores - like update java 'cacerts' once, and copy-paste it to all the instances on single machine, and then on all other machines(and there are few of javas of course on each machine, gotta search for all of them..).

Before that developers just were turning SSL check off.. which is even worse..

So I decided to dig into certificates to understand what's up, and then addressed a guy who could manage the Load Balancer(the proxy which serves the Cert. Chain to the clients).

And the guy told me that it would be better for us to just use Ansible for all the Linux machines to update all the keystores(differs on different distros as far as i'm getting it) and WIndows would be fine - it has some auto updates on the keystorage.


But my question is - why?

Why manually update hundereds of machines when we could update 1 single server to return 1 more certificate(Intermediate would be enough as far as i understand it at least).

Also need to mention, that for time being all the HTTPS traffic would go through this proxy, anyone goes anywhere using HTTPS would hit the proxy (even nodes of the jenkins server) - so there would be pretty huge load.

Could this thing about the amount of traffic be the reason the guy managing Loadbalancer could deny addition of Intermediate CA certificate into the returned chain?

Or are there any other technical issues preventing tune of the proxy to return not only End certificate but End Certificate+Intermediate CA certificate?

Sorry for wall of text, I'm pretty new to all the SSL stuff this is as exciting as it is confusing, which prevents me from shortening the post.

I'm just trying to understand why such a perfect (from my point of view) decision being declined and the alternative manual, repetitive and not reliable approach being presented?

I hope it makes sense to the readers.

And I need to apologize for my poor English.

https://redd.it/dw1oym
@r_devops

Opentracing to logfile and later feeding to Jaeger

Hi

We are offering small foortprint VMs within my company and don't want to include jaeger/zipkin in every VM.

​

Instead we would like opentracing from micro-services to log to local file in the VM. If an employee reports a problem, we then ask him to send us logs which we should then feed to centrally managed Jaeger/Zipkin server. This also helps us in saving resources as we feed the logs to Zipkin etc only when there is a problem.

is there any way to achieve this? So far reading all documentation seem that opentracing lib enabled micro-services need access to backend Jaeger/Zipkin.

Thanks

Angel

https://redd.it/dw3xqk
@r_devops

Executing script on github hooks?
https://www.reddit.com/r/learnprogramming/comments/dvs2q7/executing_script_on_github_hooks/

https://redd.it/dw6ee2
@r_devops

What tools are needed for building/deploying servers using configuration files?

So here's what I'd like to setup. I can make an image or images using Packer tool. I can use Terraform to deploy an ec2 instance using the ami id or some tags. However, that's somewhat manual.

What I'd like to achieve is that I'd like my developers to be able to specify a json or yaml file in a git repo that they will be creating. For example, a developer would like to setup an Apache server. Let's assume I've already created an apache image called **apache-img** using packer. So the developer that needs to deploy an apache will create a configuration file that contains a property name say **base-image** pointed to **apache-img** string. I'm aware that I will have to parse his repo using server side git hooks(I can do this part). What I'm missing are the tools that will be able to build a new instance using the specified base-image and also the tool for deploying the instance to different environments.

If you have a link or links I can read, that would be great!

Or how did you implement yours? How does your developers deploy servers/instances of a specific application type like Apache, Java, etc?

https://redd.it/dw5q7e
@r_devops

Using environmental variables from host to a Docker image

Hi I have a Docker container running inside CircleCI and trying to pass in local environmental variables from the host to the Docker container.

and in the localhost I have declared these vars

export gituser= myuser

export gitpass = password

​

docker build --build-arg gituser=${gituser} --build-arg gitpass=${gitpass} -t nodejs .

FROM node:9

RUN apt-get install git

RUN git clone [https://$gituser:$[email protected]/$gituser/m](https://$gituser:$[email protected]/$gituser/deploy-happi-hello-world.git)y\_app.git

​

However, the syntax is not correct as it returns the error:

\---> 08a8c8089ab1Step 2/4 : RUN git clone [https://${gituser}:${gitpass}@github.com/lexarflash8g/deploy-happi-hello-world.git](https://$%7Bgituser%7D:$%7Bgitpass%[email protected]/lexarflash8g/deploy-happi-hello-world.git) \---> Running in 2ff3bbd16d88Cloning into 'my\_app'...remote: Repository not found.fatal: Authentication failed for '[https://:@github.com/$gituser/my\_app.git](https://:@github.com/$gituser/my_app.git)

​

I guess the right env vars arent getting passed how do i resolve this?

https://redd.it/dvytl0
@r_devops

Trying to move a server from being setup with ansible over to docker, need some advice.

Right now I have an SFTP server running as a google vm. Every once in a while I need to add a new user to this server, and I decided I should probably look into ansible to create new users and for recovery purposes. So I set up an ansible script, but this is the only thing that uses ansible in our whole system and there are a couple things that I couldn't get ansible to work with so it never really took off. Fast forward to now, I just want to make this a docker container I can build and deploy. My main problem is that I've never created a docker container to do more than run an application. For this to work I would have to create a user for each person who needs access, create the sftp directory for each client, and modify configs to contain entries for each client. Ansible was really great for that, you can have template configs that generate entries for each user and easily loop. How do I accomplish this with docker?

1.) How do I do a config template? Like \`sshd\_conf\` how do I enter an SFTP user entry for each person?2.) How do I create a user list, and loop through a bunch of commands for them (create each user in the list for example) instead of just executing an explicit RUN command for each user?

https://redd.it/dvzam7
@r_devops

Need guidance about what to study in 2019 - 2020

Hello guys,

First of all excuse me if this post shouldn't be here. I'm a junior sysadmin currently studying Higher Diploma on Administration and Networks but I really feel that I need to complement it with some extra studies at the same time. I already seen some interesting certifications (VMware, LPIC..) and I'd like to know what you think I'd need to focus on.

I really like the idea of turning to devops but I feel I still need a lot of knowledge so, I'd like to know what to study first. I've got knowledge about Linux but still need to study advanced Linux but with some LPIC preparation courses should be fine.

If you need some further information about me to advise me better just tell me.

Thank you all.

https://redd.it/dw0ebg
@r_devops

How can I use Terraform modules without commiting to my directory structure?

Hey all - so curious your thoughts on this situation. We are using Terraform to configure all of our AWS infrastructure which is fantastic, and we are also using modules to keep everything organized in subdirectories that own specific parts of the infrastructure.

So far it's been great, with this one hangup:
If I try to restructure any of the modules to a better format, Terraform will subsequently try to destroy all resources in the pre-defined module, and then try to recreate all of those same resources in the post-module.

Let me give you an example:
I created a module named "rds" for our rds database. Now we are spinning up a dynamo table along with some other data resources, and I'd like to instead have a module named "datastore" where all database elements are managed.

If I simply renamed the "rds" folder to "datastore" and switched the source where it's imported, Terraform will now try to completely delete and recreate the resources in that module because it sees them as being completely different, even though in reality all I did was a rename.

Is there anyway to refactor the modules/subdirectories without completely deleting all the infrastruture? Previously I've manually removed resources from the state and then re-imported them to the new module, but that is so tedious. Ideally, I could refactor the structure of our subdirectories/modules and not have to completely re-create all infrastructure in AWS.

Maybe I'm taking the wrong approach - let me know what you think! Would love a good solution to this.

https://redd.it/dvxv9o
@r_devops

The power of checklists

Thought you all might be interested in my post about the power of checklists: [https://www.transposit.com/blog/2019.11.08-the-power-of-checklists/](https://www.transposit.com/blog/2019.11.08-the-power-of-checklists/)

Would also be interested in hearing how checklists (runbooks, playbooks) work (or don't work) for you. In my mind there's a fundamental tension between automation (which is fast to execute and, within the bounds of bugs and flash memory failures, repeatable) and human centered playbooks (which are flexible and low investment).

https://redd.it/dwbelt
@r_devops

[Help] Planning to overcome ALB 100 rule hard limit with traefik in ecs cluster

Hi all, We are using aws ecs for hosting our saas application, since we use host-based header rule to map ecs containers with subdomains we are about to reach the hard limit of 100 rules in alb , other than adding an extra alb to the cluster we are planning to overcome this issue by using traefik as a reverse proxy.


Does anyone have experience with a similar setup (ecs + traefik) running in production?
Is there a better solution to this issue?

https://redd.it/dwdolz
@r_devops

gitlab ssh deploy keys at scale

Wondering what's the standard way to manage deploy keys for 1000 servers to clone private repos.

1) create a single key pair, register it manually and copy it on every server (pros: easy to setup, 1 key to manage. cons: sharing the same keys? I don't know if it matters)

2) create a key pair for each server and register them with gitlab API (pros: unique key for each server, but does it matter? cons: hard to setup, 1000 keys to manage)

https://redd.it/dwg4rr
@r_devops

Busting myths related to APM - adding monitoring will not make your app faster by itself

There are many myths to be busted when it comes to Application Performance Monitoring. One of such myths is the belief that adding APM or any other monitoring solution to your production tooling will make the application faster by itself.

It will not. It is equivalent to a belief that buying a sport watch will make you lose weight. Extending this analog – similar to a sport watch, APM is using sensors to capture data from the underlying system turning it into information.

The information exposed by the sport watch must be understood and used in order to gain anything. After all, if at the end of the day you are looking at the sport watch and see 3,000 steps done during the day, you are not becoming slimmer or more fit. But if you knew that 10,000 steps a day is a key to a healthy lifestyle and are motivated to follow this advice, you actually stand a chance. Just put on running shoes and go outside to take a \~45 minute walk, you have actually improved your life quality.

If we look at the example more closely then the key lies in understanding **when**, **how** and **why** to use the information provided. In the example above, it started by the person both **believing** in 10,000 steps being the key to a healthy life and **having the motivation** to chase the goal. After having those two key pillars in place, it all boils down to making sure the watch clocks in 10,000 steps per day.

The same applies to an APM. Buying any APM will not make the application faster or remove any of the availability issues. On the opposite, capturing this information tends to add some overhead (typically measured in low single digit increase in resource consumption).

The post continues in the [Plumbr APM blog](https://plumbr.io/uncategorized/apm-myths-busted-1-adding-monitoring-will-make-my-app-faster).

https://redd.it/dwaolh
@r_devops

What does Docker's demise say about open source as a business model?

All open source companies have looked to Red Hat for inspiration but the failure of Docker to make a business out of a revolutionary product has really gotten me asking the question, if this is all sustainable - Why cant someone like JFrog or Hashicorp run into the same issues? Would love to hear from you guys

https://redd.it/dw9yd6
@r_devops