How to specify specific subnet in Terraform when using for each

I am trying to create an ec2 instance that will hold my Jenkins server. I want this to be in a private subnet which I created using a for each loop. Below is my subnet for each loop and my ec2 instance resource. I have also included the error message. Idea is I want it in the first private subnet.

​

**ec2 instance**

resource "aws_instance" "jenkins" {
ami = "${var.ubuntuAMI}"
instance_type = "t3.micro"
availability_zone = "us-east-1a"
key_name = "me"
monitoring = true
vpc_security_group_ids = [aws_security_group.ssh_access.id]
disable_api_termination = true
subnet_id = "${aws_subnet.private[each.key]}"

tags = {
Name = "Jenkins"
}
}

​

​

**subnet resource**

​

resource "aws_subnet" "private" {
for_each = var.subnet_numbers_private

vpc_id = aws_vpc.Main_VPC.id
availability_zone = each.key
cidr_block = cidrsubnet(aws_vpc.Main_VPC.cidr_block, 8, each.value)
tags = {
Name = "Private-${each.key}"
}
}

​

**variable used by the subnet loop**

​

variable "subnet_numbers_private" {
description = "Map for private subnets"
default = {
"us-east-1a" = 1
"us-east-1b" = 2
"us-east-1c" = 3
}
}

​

**error message seen when doing a** `terraform plan`

The "each" object can be used only in "resource" blocks, and only when the
"for_each" argument is set.

https://redd.it/dr7b69
@r_devops

Are you familier with a tool which scan your cloud (AWS specifically) and notify on resources you can remove/smaller usage?

We are using ec2 instances (Standalone, part of ECS, DB) and we think how to optimize it. What tools map the usage, add automatically more logs, and analyze monthly results. Possible resources usage: increase, reduce, remove...
If you guys have experience with a tool that reduces cost with easy onboarding please write it.
Found these tools meanwhile:

1. Spotinst - Automated Cloud Optimization
2. NewRelic.
3. ParkMyCloud.
4. Cloudability
5. Mist

https://redd.it/dqznq8
@r_devops

Identity, Authorization & Authentication: Ory, Gluu, Vault?

Hello /r/devops!

I'm researching about the three topics in the title in order to come up with a self-hosted solution for my situation. I'm creating a service that's going to be used by multiples users within a hiring party, and multiple services used by that same party.

**The main idea and necessity is to have a Single-Sign-On system**. That itself isn't something concrete, because you can do it in different ways, so I'm still reading about everything that I can.

I came across multiples options, but I'm considering Gluu, Ory & Keycloak.

- Ory doesn't come with an identity management server but it appears to function without it? What am I missing? What can I use it in the void?

- Gluu looks very complete but I don't see many people talking about it? What's the deal?

- Vault is something that looks very powerful, they appear to have identity but I don't think they do what I need from them?

- Keycloak looks hard to setup and customize, but might do the job?

Have you ever gone through something similar? I appreciate any bit of knowledge that I can get!

*sorry if that's the wrong sub, I couldn't find a better one!*

*edit: Just to make it clearer, I need to have a central identity server for all future services and I will need to have "scopes" inside those, that's the authorization part, right?*

https://redd.it/dpt1og
@r_devops

Fun with Oracle

Not sure if this is right place to post this. I somehow think marking this NSFW could be a good idea :P

I maintain an old, large, monolithic application running on Oracle written entirely in PL/SQL. There are almost no tests, almost no documentation except code comments and most of design details is lost. There is no reliable testing environment, because configuration is very complex and undocumented, and because of some unfortunate design decisions it is hard to replicate the same results. In effect, unrestricted development in production by non-IT people was a norm. This was improved recently, because I (the only IT guy who seems to care) introduced (in fact enforced) merge request workflow, so at least there is somewhat reliable versioning and some review.

Looks like a horror story and some of you probably would like to run away screaming, but I decided to take this challenge, mostly because there was rising awareness that current situation is not right (this alone is a huge success).

So this is how it looked three months ago: I had Oracle with about 4TB of data in a single schema and about 100 fairly complex ETL processes loading about 2k tables implemented in 2.5M+ LOC of PL/SQL. Code itself is pretty simple, what is hard are dependencies (sometimes circular) between steps in ETLs.

I had some rough idea how to untangle this mess, but my first approach mostly failed. I have started with most obvious solution: testing environment as a copy from production and simple CI pipeline (in Gitlab) to test merge requests. This technically speaking works, but surprisingly slowed down whole process of _development_, because due to licensing and lack of resources I was able to run just one database instance and it was down too often. So currently testing env is just copy of production refreshed every night. No CI, just shared sandbox for everyone ¯\\_(ツ)_/¯

My second approach was to create separate instances of application, just by building it in different schemas. This worked well for me, because I could at least check if all my scripts work, but was pretty useless for other people, because building and loading data took even more time than before. This was a some success, because I learned much more about internal dependencies, so I now have some idea how to make this application more modular, so I could build smaller chunks.

Now, I think I gained some knowledge, but also noticed that others (remember, non-IT people) lost motivation. I realized that from their perspective things are _worse_, because _development_ slowed, is harder for them and all my ideas looks like failures or even _unnecessary_ burden.

I think I need to actually achieve something _fast_. My next step is to cut smallest, but still relevant, chunk of functionality and turn it into separate module/schema. First, to prove this can be done and there is hope. Second, to actually build actual CI pipeline and maybe even test and document something. My idea here is to wrap existing SQL and DDL scripts in Python, turn some of them into templates and then into `fabric` tasks, and then package them and put into local pypi (package repository), so I could build and configure new environments by pulling packages from repository. Basically, I want to turn my scripts into _application_ that builds environment with given configuration. And it does that fast. Other reasons behind this packaging are:
- a plan to move ETL processes to some python tool like airflow or mara
- I could use python testing tools (`tox`, `pytest` and maybe `utPLSQL`) to actually implement some tests
- I could use python docstrings to store some documentation and then use Sphinx to build some docs
- I could store configuration data in them (tables contents, as csv or json)

Does this makes any sense? Does anyone here at least _tried_ anything like this before?

https://redd.it/drafka
@r_devops

Is Pulumi really open source?

Why do I need to login to use?

​

I do not want to create the account before using any open-source tool.

​

Is it really open source? Or just the CLI open-source and backend proprietary?

https://redd.it/drb96u
@r_devops

Career suggestions?

- I love coding (Python), mainly for automating things
- I really like Linux as well
- Right now the money is where the cloud is...... my concern is that I don’t want to “follow the money” and then be doing something I don’t enjoy.

https://redd.it/drcy0b
@r_devops

Best library to interact with Gitlab?

Is there any official or stable library to interact with Giltab and also with the .git directory? I need a program which will make commits in an automated way. I don't care about the programming language but I am used to use golang.

https://redd.it/dref8l
@r_devops

Jenkins: Parse HTML report and fail build based on criteria

I currently have some security scans within my pipeline which generates a HTML report, the report is published using the jenkins plugin 'publishHTML' . This report indicated the severity level of the scans. What i would like to do is introduce the ability for Jenkins to parse the report and fail the build if it detects anything in the category of High or critical.

​

I believe i can do this with a bash script and grep but it seems messy. Anyone got any recommendations or similar experience.

cheers

https://redd.it/drf651
@r_devops

CI - Do I build docker container with app and scp it? Or do I build app and scp the app?

I am building a CI with Github and CircleCI.

When a developer tags a commit and pushes, CircleCI will see that it is tagged and run it's workflow. This will involve:

1. Running Ant to compile the app into the \`built\`
directory.
2. scp the contents of the newly \`built\`
directory to the production server which is /home/user/foo-repo/tomcat/ROOT.

The production server users docker-compose and has docker containers: 1) nginx 2) tomcat 3) postgres.

Am I doing this correctly by scp'ing the contents of the \`built\` directory to the production server at home/user/foo-repo/tomcat/ROOT?

Or should I be building the tomcat docker container on CircleCI, along with the app, and scp the docker container to production server?

https://redd.it/dr9yxd
@r_devops

Wikileaks architecture?

I'm not sure if this is the right sub, i want to learn how wikileaks system architecture works. Particularly the online submission platform, any resources/links to the same would help. Thanks in advance.

https://redd.it/dpzxo9
@r_devops

When will Kubecon start selling tickets for 2020 Amsterdam?



https://redd.it/drh20i
@r_devops

Docker vs VM | simply explained

Hi there!

I did a video comparing **Docker vs Virtual Machine**. 🐳

Please check it out, any feedback appreciated: [https://youtu.be/5GanJdbHlAA](https://youtu.be/5GanJdbHlAA)

Playlist from the whole Docker & Kubernetes beginner series: [https://www.youtube.com/playlist?list=PLy7NrYWoggjwPggqtFsI\_zMAwvG0SqYCb](https://www.youtube.com/playlist?list=PLy7NrYWoggjwPggqtFsI_zMAwvG0SqYCb)

https://redd.it/drivpt
@r_devops

How do people manage IP/Host black/whitelisting for logging? (AKA: how do I restrict those who can log to me)

Backstory: I'm currently managing the equivalent of an ELK stack for logging, where the Logstash is actually a Syslog-NG server, and I'd love to be able to restrict certain IPs from logging to my system.

This can be achieved (and I actually do it this way nowadays) in Syslog-ng using filters, but it doesn't scale very well, since for every new IP that I'd like to add to my white/black list I need to update the configuration and restart the services.

Besides the obvious answer of "well tell those IPs to stop logging to your stack", is there any intelligent way of addressing this issue?

Has anyone ever found themselves in a similar situation?

Thanks in advance.

https://redd.it/drlsbe
@r_devops

Cloud Engineer Interview Prep

I have an interview coming up for a role as a Cloud Systems Engineer. I'm coming from a background in Software Consulting, where I've supported both on-prem and AWS/GCP solutions, but this is the first time I've applied for a "Cloud Engineer" role.

Coming from a mostly development background, I'm used to seeing leetcode and whiteboarding. Can I expect the same or similar for a cloud engineer role?

It seems to be a young field, so there aren't much resources available online. What kind of interview questions should I expect to be asked?

https://redd.it/drhxvi
@r_devops

Bleeding edge DevOps tools?

Can you suggest some latest tools or software that would be relevant to DevOps, and seem like they will have big impact in the years to come?

Need to write a paper on the same.

https://redd.it/drkc7m
@r_devops

Terraform move from single file to modules

Recently came across a question and was wondering the correct answer to it. Let's say you have everything in terraform split between two files, a [main.tf](https://main.tf) and a [varibales.tf](https://varibales.tf) fle. I know this is not the correct way of doing it, so assuming you get given this, is there a way of moving towards modules while still keeping the exact resources that have been created already?

​

My first thought is no you can't but then it seems like a common case that this would happen and that it would make sense for this type of thing to be supported.

https://redd.it/droofp
@r_devops

How to deal with db migrations in rolling updates on managed instance groups

Hi,

i'm studying for the Google Cloud Professional Architect Certification.
For update deployments, everywhere is recommended (or mandatory) the rolling update feature (on a managed instance group, or on kubernetes pods), so some instances begins to update while others remain on the old version. This is obviously good because you have continuity and no downtime.

But, if I have, let's say, a django (or ruby) application with a managed relational database, and I need to apply a database migration (meaning a change of structure), and without it my new version breaks and with it my old version does not work, how is it possible to deal with it?

How do you keep that continuity?

https://redd.it/drl5yl
@r_devops

New DevOps platform for Game Engines!

Hello fellow humans, we’re launching our CI/CD platform that focus on the game industry get an early access right now to check it out.
[https://www.wololoci.com/](https://www.wololoci.com/)

We’re going to be interviewing at YC this winter so any feedback will be appreciated it.PD: and if you can show us some love at product hunt will be awesome

[https://www.producthunt.com/posts/wololo](https://www.producthunt.com/posts/wololo)

https://redd.it/drrvts
@r_devops

Does anyone use terraform for azure?

I know you can use Terraform to spin up different kinds of resources on Azure and I can find some info on the topic on Microsoft's website, I just don't find much on medium or on other types of blogs. Is there anywhere on here that uses Terraform to manage/ and spin up Azure infrastructure?

https://redd.it/drgjve
@r_devops

Embedded devops

I was asked to figure out a ci/cd pipeline for various embedded systems. I'm familiar with the tool chains involved however there is a huge problem that I haven't really found a good solution for.

Embedded systems need to be tested on actual hardware. This means having the board(s) plugged in.

Currently they have a few physical machine to handle this but it's not scalable and it wastes a lot of time from everybody involved.

I do have some ideas I want to try but they're kind of a long shot. If this is a solved problem I couldn't find it.

What are your experiences in this area?

https://redd.it/drehj9
@r_devops

Considering moving off Azure DevOps

For a number of years we’ve mainly been a Microsoft shop and using DevOps made sense at the time. It’s fairly easy to configure and for the most part no cost since it’s basically included with MSDN. That being said, in the last few years we’ve become a lot less Microsoft centric and considering moving to a different platform.

There are some things I don’t want to leave behind though. I really like how everything is tied together, almost like an ERP system. If you setup the boards and pipelines it’s really nice. For example, create a branch to fix a bug, associate the bug with that branch, then your commits get linked to that ticket. Later, when you merge to master and build/release, those subsequent items are linked as well. Long story short, I can create a release and very easily see what is different compared to the last one.

My question is, what is a typical setup outside DevOps that would provide that same type of functionality? We only have a handful of developers and while cost isn’t a huge factor, I really don’t want to go from paying $40/month (total) to hundreds.

https://redd.it/drw5tz
@r_devops