How to manage a large number of AWS Infrastructures?

Hello guys,

I hope someone with experience or a similar situation can give some help here. So we have a custom microservice App running on ECS with a lot of AWS infra parts and we manage this with Terraform ( Deployment, maintenance, improvements, patches etc ). We deploy App with Terraform to client's AWS accounts. So that's a lot of AWS accounts to manage given that we have separated AWS per env, prod, and nonproduction.
My question is how you guys manage or how you would manage let say 20 clients X 2 env = 40 AWS accounts/deployments/infrastructures? Let say I need to deploy some infra fix or feature with Terraform to all of them, how would you do that, as manually one by one is impossible?

I was thinking to use maybe AWS Systems manager, but I'm not really sure that it supports Terraform, maybe we should switch to CloudFormation?


Any help is appreciated :)

https://redd.it/kxehcs
@r_devops

Monitoring AWS ECS on EC2 - more than cloudwatch

I am looking for ideas or best practices on how to monitor the EC2 which provides the docker containers for ECS. Fargate is more expensive in our use case.

We are running some python Backends but also regular some proprietary software, which we can't adapt.

As cloudwatch has a) very limited metrics and b) a delay of up to 5minutes till you see the metrics, we want to use prometheus (node-exporter) and/or icinga agent.

We have both running for the rest of our infrastructure. Combined with ELK and grafana, this gives us great data. Just not for everything running on ECS.

ECS is configured with LoadBalancer and autoscaling, so discovery is necessary.

Obviously there is the option to add icinga-agent and node-exporter into the container itself or add it as a side-car. Updating the clients or configs or certs would potentially require a deployment of the app, which involves other Dev teams.

We kind of want to split monitoring from the actual app and be able to update configs or versions without touching the app container.

No idea if this is the right approach, we didn't find any solution, hence I am hoping for some friendly tips or better approaches to Monitor our apps.

https://redd.it/kxj0g4
@r_devops

Anyone here make a browser-based multiplayer game? I have a newbie question about sockets and want to ask someone with experience.

I'm trying to decide between using Unity and ThreeJS to make a browser-based multiplayer game. I intially thought they each had their own way of setting up and handling multiplayer functionality. (I was asking between the two, which one would be able to handle 600 players in one arena easily without necessarily customizing the underlying engine for performance enhancements.) I was told they both use websockets/ socket.io. Are all browser-based multiplayers using this same technology or is there another way to have multiplayer functionality?


Also, how many players at once can websockets handle? Could it handle a multiplayer arena of 600 players?

https://redd.it/kxhr2t
@r_devops

Automating an install of an application



Just wondering if some one know of a tool with which i can automate the installation of a windows software, i am not looking for cloning the machines etc, the application i am trying to install is kinda like the office application where each client has a different serial number, etc.

Also i need to be able to have custom location for the install, and i have few text to enter such as serial number, user name, company name etc.

I do not have much experience with Auto-IT and not sure if i can achieve this with it

Thanks

https://redd.it/kxeipp
@r_devops

API Documentation- Postman or Swagger?

I'm on a small team currently using Postman for our customer facing API but have a couple minor problems that we think could be fixed by swapping to Swagger, just wanted to find out if anyone else has had experience with the two. Doing some research, I found a couple reviews and comparisons but really I'm looking for more personal experiences and hopefully even discussion.

The main issues with Postman we'd like to solve:

* Theming- it'd be much nicer for the company if the API docs were consistent with theming when compared to the rest of the online presence
* Custom domain- it'd be preferable having a custom domain pointed at our docs, with a separate domain for our internal docs. This is possible with the premium tiers of Postman, but that still doesn't solve the other two issues
* Hosting control- My boss would prefer having control over the server itself, I think mostly for analytics purposes

https://redd.it/kx4zsc
@r_devops

Terraform and IAM policies documents

Hello everyone,

Can you give me some insights about what you're using for managing your IAM policies documents (JSON) with Terraform ?

Do you prefer using plain JSON or maybe the TF dedicated object (which is pushed by Hashicorp) https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam\_policy\_document

Thx

https://redd.it/kxa6a0
@r_devops

Question about the new AWS RDS CA

When updating the database to use the new CA, you get this popup:

"Before scheduling the CA certificate rotation, update client applications that connect to your database to use the new CA certificate. Not doing this will cause an interruption of connectivity between your applications and your database."

Does anyone know if this is required? We didn't have to import the CA originally, so why would we have to now? If it's a matter of the new root CA not being part of the trust chain of older distros, does anyone know if Ubuntu 18.04 has the new CA in its ca-certificates package yet?

Thanks!

https://redd.it/kx8lhl
@r_devops

Can I customize all my operating systems that I will install on kvm/qemu with Virt-customize?

**For my private cloud provider, I want to create a virtual machine from the operating system iso I want using kvm / qemu (since I was looking for an alternative to cloud-inite, I chose to customize it with virtual-customize) but I could not customize freebsd and openbsd images with virtual-customize*

https://redd.it/kx2nvo
@r_devops

lOOKING FOR A GIT SAAS IN EU (WITH FAILOVER)

I'm a bit confused that this doesn't seem to exist? Are my Google-fu skills just rusty?

We want to spend money to not host our own Gitlab so first we wanted to move to .com (US only)

https://www.stackhero.io/en/services/GitLab - Preview and no failover/HA

https://gitlabhost.com/ - Single tenant

Bitbucket / Github etc also all are US.

(Sorry for CapsLock title)

https://redd.it/kx2eor
@r_devops

Bit annoyed with everyone thinking their way is better than our current process (contractors, new starters etc.)

This is a trend I've noticed a lot over the years and frankly, does my nut in. Anytime anyone joins a project, they nit-pick. It might be subtle... 'oh... Why did you choose to do it this way?' or just blatant rude.. 'ah well I'd have used terraform cloud for this instead of managing statefiles in this way'. Would you? Glad you're a fucking psychic who can tell my bosses and BA's requirements from a single look at a repo!


First of all, code is 50% more difficult to read than write. That's a proven established fact. If my scripts look over complicated to you because you didn't write them. That's probably why.


And okay great you wrote this fantastic yaml pipeline which does a build in an amazing way, but is that way relevant to my use case?


And fine, my terraform modules are a little bit messy now. This is 18 months into a cloud transformation project and frankly sometimes you have to sacrifice code cleanliness for functionality because you need to get it working and the provider you use doesn't have the updated options, and changing provider will give you weeks of work, and you're a month into the crunch and things aren't going to schedule and that's why you've hired a contractor in the first god damned place.


In a perfect world we're all perfect engineers, but come into a project and understand that you're coming into something that's been on a journey you'll never quite understand fully. Come into it with an open mind and think about that time where you've been stuck and had to throw in a hack to make it just bloody work this time.


Stop being a judgemental prick please.

...Sorry.... Been a long few weeks.

https://redd.it/kxr39t
@r_devops

QUESTION Testing PRs across multiple repositories in a single build?

Current setup:

SCM Manager: GitLab

Build Server: Jenkins

Branching Model: Trunk-Based Development (in each repository)

I have the challenge of figuring out how to set up a way to trigger builds on Jenkins when PRs are created for multiple repositories. While this in itself is not a challenge, I want to be able to test them in a single run on Jenkins, since some of the repositories are tightly-coupled in such a way that our developers often work in multiple repositories to fix a bug / develop a new feature.

This leads to the issue. How can I create a PR on each repository and then be able to fetch those in the Jenkins build? I can set up a quiet period in Jenkins so the build waits for a duration before building - This should be enough time to create the PRs in all the different repositories.

By the looks of it, GitLab assigns each PR an incrementing number as name. These PR names might not always match, so I was wondering if there is a way to do some pattern matching to get the branches created from the different branches (labels, created by developer)?

Furthermore, is this even the right approach to take? We are currently on the steps of setting up versioning for each repository, but versioning the repositories will also slow down development time, since we'll have to wait for the packages being available before even being able to test if they work properly in other repositories.

https://redd.it/kx1nw4
@r_devops

How to run AWS CIS compliance and other checks with SQL and CloudQuery

I saw some interest on cloud compliance from a post yesterday on commercial product so I wrote a short blog how to run various compliance checks (CIS) with open-source tools and SQL - https://cloudquery.io/blog/how-to-aws-cis-compliance

https://redd.it/kxwsaj
@r_devops

Sometimes it's notjust technical stuff that needs devops

Having a technical idea is grand but being able to express that idea in business talk is still necessary.

This clicked for me today when realizing that all my top notch ideas were getting swept under because I was not properly laying out my arguments in a way that had a paper trail or was not to the right stakeholders.

One must be able to articulate these ideas and have workflow in place that make collaboration easy.

I'm looking tomove forward with project management software that has auto triggers in slack and deadlines so accountability and automation work hand in hand.

https://redd.it/kxz3ce
@r_devops

[Architcture] Multiples databases to query (Mongo)

Hello guys, we have several **mongo** instances (3, 4), with product data related on each of the instance. We would like to query 1 ID, or severals IDs, and get all its data across different instances and databases.

I was thinking to a kind of "GraphQL" Schema, where the app requires a product and the attributes it needs with ID by HTTP request. And then a "middleware service" does the multiple mongo query stuff. *Does it exist already? If not, how would you implement it ? :)*


Example:
An app web needs data on the number 156789, and wants to collect data from different databases where it matches the number 156789.
For a fruit could be: "give me data related to chemical attributes, reviews from people of this fruit, and basic info (name, color, average size, etc)" to the fruit number 156789. And via HTTP request a service would handle everything to give back the data.


GET fruit/345678

{
basic_info, chemicals, reviews
}

I am curious about any ideas or ways to handle this :)

https://redd.it/kxxs2e
@r_devops

Azure Pipelines Restore packages along side solution

When using the Nuget Restore task, what do i write in the Destination directory to make it restore the packages in a packages folder along side the solution.

I am asking this because, for some reason for one of our project, the Nuget Pack fails to find the packages when they are one folder above the solution. using the destination directory as "..\\packages"

what do i write to get it to go along side the solution.

is there a predefined variable for this.

​

i cannot hard code the path as i have it restore multiple solutions in multiple folders.

preferable a solution like this if it exists "$(Build.SolutionDirectory)/packages" but targeting the solution that is currently being restored.

https://redd.it/kxwwx8
@r_devops

Looking for tech blogs in the video game industry

Hey there, tittle pretty much says all.

Any recommandations about tech blogs? Programing, coding, AI, anything tech related to video games! Thanks!

https://redd.it/ky1yln
@r_devops

How do you manage your Github/Gitlab, etc?

I have a couple of people who are asking for private repo's in our Github, while the base permissions are "read" on every repo we have in our Github (which every repo is private to everyone outside our org).

​

My question is, what are you all currently doing? Do you have it set to where everyone can see every repo for max collaboration? Do you have repos siloed off to individual teams?

https://redd.it/kxwema
@r_devops

Hardening Docker and Kubernetes with seccomp

Hi /r/devops,

Container security is often overlooked topic, so I wrote an article that shows how you can improve security of workloads running in Docker or Kubernetes by leveraging seccomp profiles, which I think might be useful to some you who are interested in security topics.

So, here's link to the article: https://itnext.io/hardening-docker-and-kubernetes-with-seccomp-a88b1b4e2111

https://redd.it/kxqrke
@r_devops

Pulsar vs RabbitMQ

Hey all,

I'm apart of a small dev shop and we're looking at different message/queue services. Primarily we're looking at Apache Pulsar and RabbitMQ. I was hoping to get some feedback on what you guys think!


My main concern is operational overhead, we have a 1-2 man ops team, so ease of use is the most important. It's hard for me to tell which would be better suited, Pulsar is CNCF so it is built to be run on the cloud, but RabbitMQ has an officially supported k8s operator so it feels like that's a moot point.


Thoughts?

https://redd.it/ky52vk
@r_devops

Alert response automation vs systemic root-cause deep-dives

Background: Ex-Amazon turned first time founder building in the DevOps/SRE space.

I posted here a couple months ago and had some really good chats with experts and learned a lot about where the industry is regarding alert response automation. There seems to be a lot of unsolved pain here because we generated quite a bit of interest.

We've expanded our value proposition to not only reduce toil and MTTR, but also help uncover the systemic root-causes and contributing factors behind alerts. To address both the immediate pain and tackle the toil at the root.

Would anyone here be willing to jump on a quick call to share thoughts, war stories, and brainstorm with me?

Thanks in advance

https://redd.it/ky5eat
@r_devops

Monitoring iot devices and servers

For our company, we're looking into adding monitoring to our fleet of IoT devices and backend servers. We would like this monitoring to be exposed to customers through our own web portal.

In a previous job, we used Zabbix to monitor all the equipment we had and I'm very familiar with it. I played around a bit and although I can get everything I need out of it (monitoring wise), I wouldn't know where to begin to expose this to our website. Maybe I could add my own webhook under media types?

On the other hand, all our devices are also using MQTT for communication so maybe it would be better to use that instead?

I would prefer using a tried and tested package for monitoring because it would save a lot on development time.

Suggestions?

https://redd.it/ky1y7c
@r_devops