Let’s Encrypt Invalidating Millions of TSL Certificates Due to Faulty Delivery Bug

>**Let’s Encrypt,** the most popular free certificate signing authority is going to invalidate **more than 3 million TLS certificates** **within the next few hours.** The reason why that happens is because they were wrongfully issued due to a Certificate Authority software bug.
>
>The bug was [confirmed](https://community.letsencrypt.org/t/2020-02-29-caa-rechecking-bug/114591) on February 29 and was fixed two hours after discovery. This changed how the domain name ownership was checked before issuing new TLS certificates.
>
>Affected website owners **have until 8PM UTC (3PM EST) March 4** to [manually renew and replace their certificates](https://certbot.eff.org/docs/using.html?highlight=renew#renewing-certificates), failing which visitors to the websites will be greeted with **TLS security warnings** — as the certificates are revoked — until the renewal process is complete.

If you'd like to read in-depth about this announcement and how to fix it, we've made this blog post about it: [https://www.bunnyshell.com/letsencrypt-bug/](https://www.bunnyshell.com/letsencrypt-bug/?utm_source=reddit.com&utm_medium=social&utm_campaign=lets-encrypt)

https://redd.it/fddtzs
@r_devops

Elasticsearch: how to create a HA, scalable, secure cluster?

Hi everyone, I have a task to set up an Elasticsearch DB with HA, scalability and security (e.g. encryption) requirements.

Questions: which "distro" to use? Open distro or AWS ES service or pure (and free) ELK stack or Elastic cloud? Any other options? How did you build your ES cluster? Any useful article about the topic? Thanks!

https://redd.it/fddint
@r_devops

Automation vs CICD tools

Hi guys,

Had this discussion the other day and would like to have other opinions for the sake of having a rich discussion. A colleague said he didn't agree on running a script for creating users on platform X via a CICD tool as we aren't really doing any CI or CD, we're just running a script to provision users.

I do agree with him but I understand it's just easier to do so nowadays and don't really know where I'd run them. Do any of you have a specific tool/platform you'd suggest for these kind of tasks, separated from your usual development CICD pipelines? Or do just believe it should be done in whatever CICD tool that is being used, for the sake of simplicity?

https://redd.it/fdd3dr
@r_devops

Sonarqube quality gate result checker for ci/cd pipelines

Hello

I made a simple sonarqube quality gate result checker for ci/cd pipelines. You can use it in almost every ci tool.

https://github.com/gungorugur/ciqube

https://redd.it/fdf10w
@r_devops

Mysql Multi-master replication setup on EC2

A few days ago, I have been looking for a solution where I can use a single EC2 instance to replicate multiple RDS mysql masters.
I did my research and realize that there is not a single blog that defines it completely step by step.

Generally, this topic has been on the boom for the purpose of cost optimization, for the techies who do not want to use aws-managed RDS replicas and for them who want to explore the insides.

>***What is the requirement:***

You can read this to differentiate RDS vs Mysql on EC2.
[Mysql on RDS vs mysql on EC2](https://serverguy.com/comparison/pros-cons-rds-vs-ec2-mysql-aws/)

We are using EC2 just for the replicas. Basically, we need to manage replicas of multiple MySQL masters on a single instance.

The first thing that will come in your mind is, how we are gonna do this in a single instance?

>***The answer is “Channels”, Mysql Channels are used to perform parallel replica operations.***
*one channel for each replica, that’s how you manage the multi-master replication.*

# Let’s do this

1. Launch an EC2 instance, Install MySQL on that.

**2.** Login to master DB and run the command “show master status” and note down the binlog file and position.
Note: Check the binlog retention and make it to 24 hours by using the below command: (Only for RDS Master)
\[call mysql.rds\_set\_configuration(‘binlog retention hours’, 24);\]

**3.** Take the dump of master using below command:
mysqldump -u \[user\] -p \[Password\] -h\[host\] — lock-tables=false — single-transaction — routines — triggers — all-databases | gzip -9 > \[backupname\].sql.gz

**4.** Restore dump using this command:
gunzip < \[backup name\] | mysql -u \[uname\] -p\[pass\]

**5.** After restoring the DBs you will need a backup of users from master:: In most cases, it will start replicating users after starting the replication, but it’s good to be safer side.
If you get some “ALTER” error while restoring the users after starting the replication it means your users are already replicated.

>***Take dump of users from master***

i) pt-show-grants — host \[host\] — user \[uname\] — password \[password\] — ignore root@localhost,’mysql.session’@’localhost’,rdsadmin@localhost,’rdsrepladmin’@’%’ > grants.sql

>***Restore Users to EC2 replica***

ii) mysql -h \[host\] -u \[unane\] -p \[password\] < grants.sql

**6.** Now, go to /etc/mysql/mysql.cnf and add below lines under \[mysqld\] section::


***This is to enable GroupBy in SQL queries.***
“sql\_mode = STRICT\_TRANS\_TABLES,NO\_ZERO\_IN\_DATE,NO\_ZERO\_DATE,ERROR\_FOR\_DIVISION\_BY\_ZERO,NO\_AUTO\_CREATE\_USER,NO\_ENGINE\_SUBSTITUTION”

***Change bind-address to 0.0.0.0 to bind to all interfaces.***
bind-address = 0.0.0.0

***Set your replica as read-only so nobody can make write operations***
read\_only = 1

***This is needed to start the replication***
master-info-repository=table
relay-log-info-repository=table

**7.** Finally, configure the replication

“ *change master to master\_host=\[host\],master\_user=\[replicauser\], master\_password=\[replicauserpassword\], master\_log\_file=\[filename you have noted down of master\], master\_log\_pos=\[position you have noted down of master\] FOR CHANNEL \[Any channel name\];*”

**8.** Start the Replication:
*START SLAVE FOR CHANNEL “Channel Name”*

**9.** Check the replication Status:

*SHOW SLAVE STATUS FOR CHANNEL “ChannelName”*

**10.** If you are getting any error while starting the replication, and an output of command “show slave status”,
then note down the error number, and add that in mysql.cnf with the below parameter and restart MySQL.


slave-skip-errors = ErrorNumber (You can add multiple error numbers separated with comma)

If you want to use Multiple Master RDSs and single ec2 Replica then follow the same steps with changing the channel names.

M**onitoring**
for the monitoring purpose, you can use mysqld\_exporter with Prometheus and Grafana.

https://redd.it/fd9rn8
@r_devops

How to automate spawning new domains for new branches?

Hi,

I'm using GitLab CI for our CICD. I currently have CI setup for our dev branch, but I would like to have a way of spawning a random-name domain for other branches.

Let's say this is a regular html/css page for the purpose of this example. I'm using nginx so I was thinking of including certain locations in the nginx.conf so that when developers create a new branch (e.g. feature/something) and push it would deploy to something like abc153421cba.mydomain.com
Any ideas how to achieve this, preferably best practice how these kind of deployments could be done?

https://redd.it/fd9m73
@r_devops

What questions should I ask on an interview?

I am a sysops with about 2.5 years of experience. My primary experience is in public cloud (mainly aws but also azure).

My main issue with is the amount of work versus the value of work. There is allot of bureaucracy and my role is more like a traditional sysadmin than a devops (even though the environment is all on the public cloud). The learning I do is on my free time.

Where I live, devops jobs are low demand low supply. But I have gotten an offer through an acquaintance for a devops job, and I have an interview.

My question is what questions should I ask to make sure that the job would not have the same conditions as the job I am currently in? My primary interest at this point is to gain experience in devops.

Any advice would be greatly appreciated.

https://redd.it/fd08vs
@r_devops

Operational stability from a data perspective.

At my company, we have a DevOps team that's applying the 3 ways of DevOps to our monolithic architecture. They've successfully achieved daily deploys, sensible change management, and implemented several processes to quickly recover from disastrous situations.

Unfortunately, the same care hasn't been given to other parts of our ecosystem. Specifically, our data architecture is kind of a mess right now. Our current architecture has already started fires that were tough and mind-numbing to put out. But in the midst of the chaos, emerged a data pipeline that was golden from a DevOps perspective.

I wrote an article which talks about how to build such pipelines: [https://medium.com/@talham7391/structuring-a-robust-data-pipeline-24ff67783782](https://medium.com/@talham7391/structuring-a-robust-data-pipeline-24ff67783782)

Here are some of the main points:

\- In order to build good data pipelines, data can't be an afterthought when designing systems that will inevitably generate data. Such practices will result in ad hoc, hard to test, and brittle data pipelines that will break.

\- If your company is shifting from monolithic to service-oriented architecture, you might be heading towards Data Armaggedon - a place where everyone is freaking out because data is disorganized and hard to access. Make sure you think about your data needs when making the shift.

\- Data pipelines, if they are critical to the business, should be treated the same as other critical services. Namely, you need to make sure you think about testing, automated deployments, observability, quick recovery mechanisms, etc.. etc...

If you're liking what you've read so far, I think you'll like the article! I go into more detail and I also talk about certain properties data pipelines should have that will make them a joy to work with.

https://redd.it/fczq6o
@r_devops

K8s terms for any newbies looking to learn

Came across this list. Solid overall. Good for anyone new to DevOps or K8s looking for a quick understanding of some industry terms. Missing a few things like Kube-proxy, DeamonSet, toleration & PSP tho.

&#x200B;

[https://www.portshift.io/blog/ultimate-devops-devsecops-kubernetes/](https://www.portshift.io/blog/ultimate-devops-devsecops-kubernetes/)

https://redd.it/fdu62z
@r_devops

An argument for building event driven microservices and avoiding distributed monoliths!!

Based on my relatively short experience of working with [Space Cloud](https://github.com/spaceuptech/space-cloud) and microservices, i quickly realized what Kelsey Hightower meant by _distributed monoliths_.

Here's my two cents on how you can overcome that problem by going event driven. https://medium.com/spaceuptech/the-right-way-of-making-microservices-communicate-b6cd517ae702

Would love to hear your thoughts and experience! All feedback is welcomed!

https://redd.it/fdvz6u
@r_devops

Kubernetes clicked when I learned about Deployments and Services

https://ljvmiranda921.github.io/notebook/2020/01/18/kubernetes-deployments/

https://redd.it/fdykn8
@r_devops

Former webhosting guy wanting to get into Devops after several years afk. Advice? Feeling knowledge is a mile wide but an inch deep.

HI folks. I have been doing data center technician work for several years and am looking to switch back to a Linux oriented job, more specifically devops. I originally was a network engineer for a major telecom after going through a ccna internship. I used that experience to get work at a small webhosting company several years ago and was trained to basically run it for the owner. It was a very simple and easy to manage and never got to expand knowledge past basic wp/ static site hosting (apache,bind9,postfix,etc..). After that I worked as a jr devops engineer for about two months before being fired for essentially not being able to keep up. I felt discouraged and went back to basically plugging stuff in for various datacenters for about 3 years.

I am at a point where I recognize my lacking skills but have immense ability to improve. I am unemployed currently and would like to do something in the next 2 months to get me on track to have a solid career in devops/ system engineering. I have considered going back to school for CS but am not prepared to do 4-5 years while working fulltime.

I am considering the LPIC path which I think I could at the very least knock out LPIC1 in the next 3 months. Then get a devops specific cert.

Does this sound like a good plan? Can I land a job with an LPIC and limited experience from several years ago, within the year? Would it be better to just brush up on linux learn python and go straight for a devops cert?

TLDR; Can I get a job with one or two of these certs? Should I just go for a devops cert right away?
https://www.lpi.org/our-certifications/lpic-1-overview
https://www.lpi.org/our-certifications/exam-701-objectives

https://redd.it/fe0i8u
@r_devops

CI/CD for Java Dropwizard + React app

My application is composed of three parts: a React frontend, Java Dropwizard REST service, and a Java scheduler (Quartz).

I was thinking of using Azure DevOps to deploy everything so I could have CI/CD, but they don't have Dropwizard as a framework and less customization.

Was wondering if anyone has any suggestions on what to use in this case? Should I use Docker?

https://redd.it/fe07rk
@r_devops

Jenkinsfile SCM best practice

Hi. We are planning to make a switch from TeamCity to the Jenkins. One of the most interesting features to us are Pipelines/ aka Jenkinsfiles.

I would like to hear how do you organize Jenkins in terms of scm? Do you have one repo containing multiple jenkinsfiles stored per project, or you have single Jenkinsfile within source code of project ( separate repo)?

Thank you for your opinions!

https://redd.it/fe1dzc
@r_devops

Best Way To Monitor Developer Environment on TV's?

Hi. So here at the office, we'd like to have some developer stats on the TV's in the office, so people can keep track of things and whatnot. We currently use Dashkiosk on pretty much 1 or 2 TV's right now to essentially test this out and have run into some HTML5 bugs with it. We have recent Vizio TV's, so they have an "OK" web browser built-in that we'd prefer to use over buying a dozen or so Intel Compute sticks. Is there a plugin for Dashkiosk that would resolve our HTML5 problems, or should we be using something else entirely, like DashPlay or another program?

https://redd.it/fe18fa
@r_devops

Pros/Cons: Containers vs immutable virtual machine images

Hi /r/DevOps,

I just started a new gig and find myself in a position where I need to pitch a pros/cons list of building a microservices application using containers or virtual machine images.

I'm familiar with both and have used them both to great success, but I have to convince management that containers are purpose built for our architecture. It's more or less a mix of python, Java, and node apps.

We're going to have about 70 total services, and if we follow the pattern of 1 service to one container/one image, it's a no brainer to me that containers (either on swarm or k8s) will win out in terms of operational overhead and cost.

How would you present pros/cons of each approach?

https://redd.it/fe4y75
@r_devops

Just got an offer as a Devops team lead, now what?

I have been escalating real quick In my career, I came out of college 4 years ago and now I got in a pretty good company by working 1 year in each of the following roles

Software developer > Scrum master > Profesional services consultant > Sysadmin > Devops team lead

This is my first time a as a team lead I might not be SR enough as many people here but you can count my experience x2. I'm really dedicated and work really hard from the beginning to the end of the day

I will be in charge of a team of 3 people working together with other devops teams in other countries but to be fair I'm a bit clueless where to start in this new role any advice is welcome especially in the "lead engineer" part

https://redd.it/fe99fw
@r_devops

Python For DevOps: Learn Ruthlessly Effective Automation

Has anyone had the pleasure of reading this book? I was wondering what people thought of it

&#x200B;

[https://www.oreilly.com/library/view/python-for-devops/9781492057680/](https://www.oreilly.com/library/view/python-for-devops/9781492057680/)

https://redd.it/fdv7vv
@r_devops

Python script for bulk find and replace in multiple files in multiple git Repositories

This script is useful when you want to change multiple part of multiple files in multiple git repositories in bulk. Script is creating a new branch from the given base branch and committing changes into it. Maybe push to remote also can be added with small change.

BTW, any feedback would be much appreciated.

I will drop this here maybe it can be helpful for someone else:

[Medium Post](https://itnext.io/bulk-find-and-replace-in-multiple-files-in-multiple-git-repositories-487c7c433b75)

[Github Repo](https://github.com/omerkarabacak/bulk-find-and-replace-in-git-repositories)

https://redd.it/fdxt9a
@r_devops

Authfailure: AWS was not able to validate the provided access credentials

I am trying to create my Gitlab CI/CD pipeline with AWS. The goal is to Terminate the Existing EC2 Instance, Run a new instance from a template, then Associate an Elastic IP to the new EC2. The runner I'm using is a Docker runner using the python:latest image. When I run my CI/CD pipeline I get

An error occurred (AuthFailure) when calling the DescribeInstances operation: AWS was not able to validate the provided access credentials My .gitlab-ci.yml is as follows:
```
stages:
- build
AWS_Install:
image: python:latest
stage: build
tags:
- Docker
script:
- pip install awscli
- export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID
- export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY
- export AWS_DEFAULT_REGION=$AWS_DEFAULT_REGION
- echo "running script :)"
- OLDEC2=$(aws ec2 describe-instances --filters "Name=instance-state-name,Values=running" --query "Reservations[*].Instances[*].[InstanceId]" --output text)
- aws ec2 terminate-instances --instance-ids "$OLDEC2"
- sleep 200.0
- aws ec2 run-instances --launch-template LaunchTemplateId=[launch-template-id],Version=12
- sleep 120.0
- NEWEC2=$(aws ec2 describe-instances --filters "Name=instance-state-name,Values=running" --query "Reservations[*].Instances[*].[InstanceId]" --output text)
- aws ec2 associate-address --allocation-id [allocation-id] --instance-id "$NEWEC2" --allow-reassociation
```
```
What I've checked/tried:
- AWS credentials: They are correct and valid
- aws configure: Everything sets correctly (checked using aws configure get)
- Ensured UNIX line endings were being used
- Adding a variable section to the YAML file to include environment variables
- Hardcoding credential values
- New user on AWS with all necessary credentials
- Using export to get the environment variables
- Running everything in one script rather than having a before script
- Having multiple stages/Jobs
```

https://redd.it/fdwxo3
@r_devops

Company would like to have a call to make a formal offer to start working, and answer any questions you may have at this point of the process.

I don't really want to have a video call and have to answer straight in the moment what I think, how much I want, etc.

I'm a much stronger negotiator when it's all written and digital. Any tips on how to handle this?

Should I e-mail back:

>Is it possible to get the offer in writing before having the discussion tomorrow?

I feel it's a strategic move to talk me down financially. I'm definitely looking to maximize the offer and I'm quite introverted, so this would make me feel uncomfortable.

https://redd.it/fdwel5
@r_devops