Where to place database?

For a green field shoe string budget project I need a Mongo database. Curent plan is to be using Kubernetes via KOPS on aws. I have a mix of containers that will be in the Kubernetes cluster as well as some Lambda functions that need access to that same database instance/cluster.

​

My requirements are easy to back up, low cost, okay performance - ideally same as the Mongo Atlas sandbox performance.

​

What path/where to put my DB, do you think I should go for this? I'm open to any options within reason

https://redd.it/dqdiq0
@r_devops

Tutorial: Deploy GitLab CE on Docker Swarm

I wrote a tutorial on how to deploy GitLab CE on Docker Swarm. It builds on [an earlier tutorial](https://lunar.computer/posts/docker-swarm-digitalocean/) I wrote about creating a Docker Swarm cluster on DigitalOcean. This tutorial isn't DigitalOcean specific, though, it just requires a Docker Swarm cluster with Traefik deployed on it.

Hopefully someone will find it useful. :)

Docker Swarm lives!

https://lunar.computer/posts/gitlab-docker-swarm/

https://redd.it/dr213z
@r_devops

Repository organization? Separate small dedicated apis vs one large one? Plz halp!

We have a small team and I'm in charge of moving our apps to containers using docker.

Currently we have several different apps, each with it's own repository. An app's repo consists of a web client and an api. Each repo has a folder for the client project and a folder for the api project. We are using hapi to write the apis.

Is this a good strategy? Using docker-compose and docker swarm we can easily deploy the apps, but different apps are on different versions of hapi; some outdated and will need refactoring. Our newer apps are on the latest versions.

Would it be a good idea to have one large api split into plugins for each app? An app's repo would contain only the web client. The large standalone api would have it's own repo and be deployed on it's own server. This way the hapi code could be kept on a consistent version between apps.

What are the upsides and downsides to each strategy, and are there other better ways I haven't considered?

Any help would be appreciated! Thanks!

https://redd.it/dr2xsk
@r_devops

logstash service not starting

I am using Amazon Linux 2 ( Amazon Linux AMI release 2018.03) I installed Logstash in it.

>wget https://artifacts.elastic.co/downloads/logstash/logstash-7.4.0.rpm
>yum install logstash-7.4.0.rpm

But I am unable to start logstash service.

Here are some attempts to start the service.....all failed.

>$sudo service start logstash
start: unrecognized service"

>$sudo service logstash start
logstash: unrecognized service

>$initctl start logstash
initctl: Rejected send message, 1 matched rules; type="method_call", sender=":1.2" (uid=500 pid=3521 comm="initctl start logstash ") interface="com.ubuntu.Upstart0_6.Job" member="Start" error name="(unset)" requested_reply="0" destination="com.ubuntu.Upstart" (uid=0 pid=1 comm="/sbin/init ")

As you see I have tried to start logstash service all possible ways but failed .

Is there a way I can start logstash service in Amazon Linux 2?

https://redd.it/dr2bka
@r_devops

How to specify specific subnet in Terraform when using for each

I am trying to create an ec2 instance that will hold my Jenkins server. I want this to be in a private subnet which I created using a for each loop. Below is my subnet for each loop and my ec2 instance resource. I have also included the error message. Idea is I want it in the first private subnet.

​

**ec2 instance**

resource "aws_instance" "jenkins" {
ami = "${var.ubuntuAMI}"
instance_type = "t3.micro"
availability_zone = "us-east-1a"
key_name = "me"
monitoring = true
vpc_security_group_ids = [aws_security_group.ssh_access.id]
disable_api_termination = true
subnet_id = "${aws_subnet.private[each.key]}"

tags = {
Name = "Jenkins"
}
}

​

​

**subnet resource**

​

resource "aws_subnet" "private" {
for_each = var.subnet_numbers_private

vpc_id = aws_vpc.Main_VPC.id
availability_zone = each.key
cidr_block = cidrsubnet(aws_vpc.Main_VPC.cidr_block, 8, each.value)
tags = {
Name = "Private-${each.key}"
}
}

​

**variable used by the subnet loop**

​

variable "subnet_numbers_private" {
description = "Map for private subnets"
default = {
"us-east-1a" = 1
"us-east-1b" = 2
"us-east-1c" = 3
}
}

​

**error message seen when doing a** `terraform plan`

The "each" object can be used only in "resource" blocks, and only when the
"for_each" argument is set.

https://redd.it/dr7b69
@r_devops

Are you familier with a tool which scan your cloud (AWS specifically) and notify on resources you can remove/smaller usage?

We are using ec2 instances (Standalone, part of ECS, DB) and we think how to optimize it. What tools map the usage, add automatically more logs, and analyze monthly results. Possible resources usage: increase, reduce, remove...
If you guys have experience with a tool that reduces cost with easy onboarding please write it.
Found these tools meanwhile:

1. Spotinst - Automated Cloud Optimization
2. NewRelic.
3. ParkMyCloud.
4. Cloudability
5. Mist

https://redd.it/dqznq8
@r_devops

Identity, Authorization & Authentication: Ory, Gluu, Vault?

Hello /r/devops!

I'm researching about the three topics in the title in order to come up with a self-hosted solution for my situation. I'm creating a service that's going to be used by multiples users within a hiring party, and multiple services used by that same party.

**The main idea and necessity is to have a Single-Sign-On system**. That itself isn't something concrete, because you can do it in different ways, so I'm still reading about everything that I can.

I came across multiples options, but I'm considering Gluu, Ory & Keycloak.

- Ory doesn't come with an identity management server but it appears to function without it? What am I missing? What can I use it in the void?

- Gluu looks very complete but I don't see many people talking about it? What's the deal?

- Vault is something that looks very powerful, they appear to have identity but I don't think they do what I need from them?

- Keycloak looks hard to setup and customize, but might do the job?

Have you ever gone through something similar? I appreciate any bit of knowledge that I can get!

*sorry if that's the wrong sub, I couldn't find a better one!*

*edit: Just to make it clearer, I need to have a central identity server for all future services and I will need to have "scopes" inside those, that's the authorization part, right?*

https://redd.it/dpt1og
@r_devops

Fun with Oracle

Not sure if this is right place to post this. I somehow think marking this NSFW could be a good idea :P

I maintain an old, large, monolithic application running on Oracle written entirely in PL/SQL. There are almost no tests, almost no documentation except code comments and most of design details is lost. There is no reliable testing environment, because configuration is very complex and undocumented, and because of some unfortunate design decisions it is hard to replicate the same results. In effect, unrestricted development in production by non-IT people was a norm. This was improved recently, because I (the only IT guy who seems to care) introduced (in fact enforced) merge request workflow, so at least there is somewhat reliable versioning and some review.

Looks like a horror story and some of you probably would like to run away screaming, but I decided to take this challenge, mostly because there was rising awareness that current situation is not right (this alone is a huge success).

So this is how it looked three months ago: I had Oracle with about 4TB of data in a single schema and about 100 fairly complex ETL processes loading about 2k tables implemented in 2.5M+ LOC of PL/SQL. Code itself is pretty simple, what is hard are dependencies (sometimes circular) between steps in ETLs.

I had some rough idea how to untangle this mess, but my first approach mostly failed. I have started with most obvious solution: testing environment as a copy from production and simple CI pipeline (in Gitlab) to test merge requests. This technically speaking works, but surprisingly slowed down whole process of _development_, because due to licensing and lack of resources I was able to run just one database instance and it was down too often. So currently testing env is just copy of production refreshed every night. No CI, just shared sandbox for everyone ¯\\_(ツ)_/¯

My second approach was to create separate instances of application, just by building it in different schemas. This worked well for me, because I could at least check if all my scripts work, but was pretty useless for other people, because building and loading data took even more time than before. This was a some success, because I learned much more about internal dependencies, so I now have some idea how to make this application more modular, so I could build smaller chunks.

Now, I think I gained some knowledge, but also noticed that others (remember, non-IT people) lost motivation. I realized that from their perspective things are _worse_, because _development_ slowed, is harder for them and all my ideas looks like failures or even _unnecessary_ burden.

I think I need to actually achieve something _fast_. My next step is to cut smallest, but still relevant, chunk of functionality and turn it into separate module/schema. First, to prove this can be done and there is hope. Second, to actually build actual CI pipeline and maybe even test and document something. My idea here is to wrap existing SQL and DDL scripts in Python, turn some of them into templates and then into `fabric` tasks, and then package them and put into local pypi (package repository), so I could build and configure new environments by pulling packages from repository. Basically, I want to turn my scripts into _application_ that builds environment with given configuration. And it does that fast. Other reasons behind this packaging are:
- a plan to move ETL processes to some python tool like airflow or mara
- I could use python testing tools (`tox`, `pytest` and maybe `utPLSQL`) to actually implement some tests
- I could use python docstrings to store some documentation and then use Sphinx to build some docs
- I could store configuration data in them (tables contents, as csv or json)

Does this makes any sense? Does anyone here at least _tried_ anything like this before?

https://redd.it/drafka
@r_devops

Is Pulumi really open source?

Why do I need to login to use?

​

I do not want to create the account before using any open-source tool.

​

Is it really open source? Or just the CLI open-source and backend proprietary?

https://redd.it/drb96u
@r_devops

Career suggestions?

- I love coding (Python), mainly for automating things
- I really like Linux as well
- Right now the money is where the cloud is...... my concern is that I don’t want to “follow the money” and then be doing something I don’t enjoy.

https://redd.it/drcy0b
@r_devops

Best library to interact with Gitlab?

Is there any official or stable library to interact with Giltab and also with the .git directory? I need a program which will make commits in an automated way. I don't care about the programming language but I am used to use golang.

https://redd.it/dref8l
@r_devops

Jenkins: Parse HTML report and fail build based on criteria

I currently have some security scans within my pipeline which generates a HTML report, the report is published using the jenkins plugin 'publishHTML' . This report indicated the severity level of the scans. What i would like to do is introduce the ability for Jenkins to parse the report and fail the build if it detects anything in the category of High or critical.

​

I believe i can do this with a bash script and grep but it seems messy. Anyone got any recommendations or similar experience.

cheers

https://redd.it/drf651
@r_devops

CI - Do I build docker container with app and scp it? Or do I build app and scp the app?

I am building a CI with Github and CircleCI.

When a developer tags a commit and pushes, CircleCI will see that it is tagged and run it's workflow. This will involve:

1. Running Ant to compile the app into the \`built\`
directory.
2. scp the contents of the newly \`built\`
directory to the production server which is /home/user/foo-repo/tomcat/ROOT.

The production server users docker-compose and has docker containers: 1) nginx 2) tomcat 3) postgres.

Am I doing this correctly by scp'ing the contents of the \`built\` directory to the production server at home/user/foo-repo/tomcat/ROOT?

Or should I be building the tomcat docker container on CircleCI, along with the app, and scp the docker container to production server?

https://redd.it/dr9yxd
@r_devops

Wikileaks architecture?

I'm not sure if this is the right sub, i want to learn how wikileaks system architecture works. Particularly the online submission platform, any resources/links to the same would help. Thanks in advance.

https://redd.it/dpzxo9
@r_devops

When will Kubecon start selling tickets for 2020 Amsterdam?



https://redd.it/drh20i
@r_devops

Docker vs VM | simply explained

Hi there!

I did a video comparing **Docker vs Virtual Machine**. 🐳

Please check it out, any feedback appreciated: [https://youtu.be/5GanJdbHlAA](https://youtu.be/5GanJdbHlAA)

Playlist from the whole Docker & Kubernetes beginner series: [https://www.youtube.com/playlist?list=PLy7NrYWoggjwPggqtFsI\_zMAwvG0SqYCb](https://www.youtube.com/playlist?list=PLy7NrYWoggjwPggqtFsI_zMAwvG0SqYCb)

https://redd.it/drivpt
@r_devops

How do people manage IP/Host black/whitelisting for logging? (AKA: how do I restrict those who can log to me)

Backstory: I'm currently managing the equivalent of an ELK stack for logging, where the Logstash is actually a Syslog-NG server, and I'd love to be able to restrict certain IPs from logging to my system.

This can be achieved (and I actually do it this way nowadays) in Syslog-ng using filters, but it doesn't scale very well, since for every new IP that I'd like to add to my white/black list I need to update the configuration and restart the services.

Besides the obvious answer of "well tell those IPs to stop logging to your stack", is there any intelligent way of addressing this issue?

Has anyone ever found themselves in a similar situation?

Thanks in advance.

https://redd.it/drlsbe
@r_devops

Cloud Engineer Interview Prep

I have an interview coming up for a role as a Cloud Systems Engineer. I'm coming from a background in Software Consulting, where I've supported both on-prem and AWS/GCP solutions, but this is the first time I've applied for a "Cloud Engineer" role.

Coming from a mostly development background, I'm used to seeing leetcode and whiteboarding. Can I expect the same or similar for a cloud engineer role?

It seems to be a young field, so there aren't much resources available online. What kind of interview questions should I expect to be asked?

https://redd.it/drhxvi
@r_devops

Bleeding edge DevOps tools?

Can you suggest some latest tools or software that would be relevant to DevOps, and seem like they will have big impact in the years to come?

Need to write a paper on the same.

https://redd.it/drkc7m
@r_devops

Terraform move from single file to modules

Recently came across a question and was wondering the correct answer to it. Let's say you have everything in terraform split between two files, a [main.tf](https://main.tf) and a [varibales.tf](https://varibales.tf) fle. I know this is not the correct way of doing it, so assuming you get given this, is there a way of moving towards modules while still keeping the exact resources that have been created already?

​

My first thought is no you can't but then it seems like a common case that this would happen and that it would make sense for this type of thing to be supported.

https://redd.it/droofp
@r_devops