One pipeline to rule them all. Why?

I heard multiple companies, in recent months, say they are working on a single pipeline (or pipeline as a service) for the entire organization. My question is why is this a thing? Is this some “best practice” people are following?

Today I heard his company is asking him to fill out a 27 question spreadsheet and they will deliver him “the perfect pipeline”. This sounds like an anti pattern to me.

https://redd.it/f665pb
@r_devops

Best way to configure default settings for kubelet v1.17.1 (using Clearlinux) ?

I need to change default kubelet cgroup driver from "cgroupfs" to "systemd" (and maybe later some do further tweaking on kubelet config).

I am not sure how to do this properly with kubelet (I am using Clearlinux), some hints/links/example config file for kubelet would be nice.

https://redd.it/f65wb3
@r_devops

What Is DevOps? What is the work of DevOps Engineer?



https://redd.it/f5q16m
@r_devops

What's the use of codebuild being used with ECR?

I'm possibly going to have a junior Devops position soon and was trying to get a grasp of all these technologies. For the mostpart things have been making sense, yet I don't understand why codebuild would be used with ECS. Since you're using container images isn't it already built?

Is the whole point just to run tests on the applications?

https://redd.it/f6kjf2
@r_devops

CI/CD optimization

Hi

We are using circleci. For each check ins, it ends up with 2 test runs. One for CI and the other for CD.

This makes sense since there could have multiple PR going in at the same time.

However, for case where there is no new change, can the test run for CD be saved if it's exactly the same run as the one done for CI?

Thanks

https://redd.it/f6k7pk
@r_devops

Kubernetes Container Config

New to Kubernetes and am struggling with the best way to deploy a third party application that already has a docker image setup. The issue I am having is setting the configuration files (there are several in different subfolders) in the docker container. Locally I ran docker compose and mounted my local disk as the configuration.


I have been looking at ConfigMaps and that seems to be a bit clunky to map anything other than a series of files. Is this the typical way people are doing this and I just live with it or is there something easier?


(Running minikube locally while I learn but this will end up in Azure eventually)

https://redd.it/f6jtnp
@r_devops

Swapping homepage URL without disturbing URL paths and subdomains

Introduction:

* I have multiple Ruby on Rails applications served with Docker nginx in the same container.
* I have a domain name **website-example.com** from GoDaddy.
* On visiting **website-example.com**, we get to the landing page.
* API is exposed from the **website-example.com/getList** URL path.
* There are subdomains like **subdomain.website-example.com**

Problem Statement:

* I am building a new React application with NodeJS, but I want **website-example.com** to point to the new app.
* I want the exposed APIs to remain accessible via **website-example.com/getList**
* I want the subdomains to remain accessible like **subdomain.website-example.com**

How would I do this?

https://redd.it/f6j6c9
@r_devops

Monitoring your AWS Network Infrastructure

Monitoring your public cloud network is an unsolved problem, such as on AWS. It is far from complete, with many loose and fragmented solutions at best. For instance, filtering workloads with high inbound traffic, with specific sessions across VPC, Classifying short-lived bursts, long-lived sessions, etc. We have put together a short questionnaire looking for some feedback on possible offerings in this space.

https://forms.gle/9BH7s8sqhfDmsj117

Audience: 1) Anyone using public clouds such as AWS, GCP 2) SREs, IT solutions provider, security professionals welcome.

https://redd.it/f5k37w
@r_devops

What's your preferred way of doing HA scheduled tasks?

We have a couple crond containers that do things like cleanups and internal report/summarization tasks. These are, of course, not HA.

I've read Google's whitepaper on [distributed cron](https://landing.google.com/sre/sre-book/chapters/distributed-periodic-scheduling/) which is fascinating but we'd rather not roll our own if possible. I've also heard of people doing most of their scheduled task workload in Lambda which wouldn't really be what we're looking for. What do you guys use for HA "cron" or other scheduled tasks?

https://redd.it/f4z7af
@r_devops

K8s & Consul - Avoid cross-DC/AZ traffic

I have huge app in Kubernetes with about 90 containers on 5 k8s worker nodes. Nodes are spread between Availability Zones in AWS, like that:

node1 - AZ-A
node2 - AZ-A
node3 - AZ-B
node4 - AZ-B
node5 - AZ-C

Also, I got many small apps which use API from that huge app. All apps use external/public host of that app, so actually all traffic between apps goes through ELB.

I can use internal Kubernetes hostnames (now we got coreDNS), but I would like to avoid cross-zone traffic, so how can I implement consul to be AZ aware and prefer traffic between pods in the same AZ, e.g.

[Diagram](https://i.stack.imgur.com/6q4C8.png)

Something similar but with HAProxy and Redis:

[https://medium.com/helpshift-engineering/saving-inter-zone-transfer-costs-in-ec2-with-haproxy-c4217ac2ca04](https://medium.com/helpshift-engineering/saving-inter-zone-transfer-costs-in-ec2-with-haproxy-c4217ac2ca04)

https://redd.it/f4yo2z
@r_devops

Amazon.. holy crap...

I mean.. I just want to set up a simple couple containers. I spent 4 hours..and still have nothing running. I have had to go through cluster and policy and role and RBAC and holy shit. I am lost in why this is so forking difficult to use? I am not looking to scale 5000 servers and 4billion people. I just want to deploy my simple docker containers in a network so they communicate together.. and make it available. That's all. WTF is all this policy shit with 100's of things, and so on.

Gotta be some sort of template for common use cases?

https://redd.it/f52tzl
@r_devops

Free CI/CD tool for local dev use...

Hey all,

Been trying to figure out how to work in some CI/CD tool chain into my projects. One project in particular is my attempt at learning some microservices. I have a monorepo, which has a UI component (react), as well as some microservices that are essentially decoupled API endpoints.

I started reading up on GitLab, CircleCI, TravisCI, etc. All of those seem to be in the cloud, and while some have free options, none seem to run locally.

By locally, I was thinking I could find something fairly easy to run as a docker container that then grabs my project from github or bitbucket, can build individual microservices, the UI, etc, bundling each in to their own containers, and can then, using my docker-compose setup, stand it up to run tests. Eventually I would like to deploy to amazon or google, but right now, I am a lone developer. I don't know that I need (or want) to run all this sort of stuff in the cloud yet. So I was thinking it's 2020, there has to be some good quality open source CICD option out there (other than Jenkins) that is a little more "today" for lack of a better word.

TL;DR I would like to learn/run a "simple" CI/CD tool locally that works with my current Github project, figure out how to build separate services in my monorepo, stand up the app, run tests, and eventually be able to deploy it to a cloud service for production use.

​

Thanks.

https://redd.it/f4ocvb
@r_devops

Biggest drawbacks to Bitnami for launching WordPress sites etc?

If you regularly are deploying various websites for PHP, Laravel, WordPress, or CMS software, maybe even JS frameworks like Node etc then please offer me your opinion:

What are the negative aspects of Bitnami specifically?

It seems weird that VMWare aquired them last year when Bitnami seems big on the AWS Amazon cloud, what do insiders think is the future of Bitnami in relation to cloud servers overall?

With Docker and Kubernetes, etc, it seems even less relevant these days. Esp. if you compare with more advanced tools like Ansible.

Even Bash scripts can be organized easily now with like Mina etc.

Idk but seems Bitnami sucks from the few times I played with AWS servers... config seems pretty limited at least on the AWS versions... what's the point?

https://redd.it/f47vx7
@r_devops

Request and response validation against Swagger spec when using SAM cli local API gateway

Hi Every one,

I'm trying to write some tests against my AWS API gateway swagger specs using `sam local start-api --host`[`0.0.0.0`](https://0.0.0.0/).

It seems the fake API gateway does not validate the request nor the response against my swagger spec.

Is there any way to achieve this? like using Localstack or any other solution?

Thank you.

https://redd.it/f6vu33
@r_devops

How do I automate deployment of Logic Apps and workflows?

Hi,

I’m using Terraform to automate the deployment and management of Azure resources.

Terraform is great, and where there are gaps, I’ve used ARM templates called via Terraform.

However, I’ve run into a problem when it comes to deploying logic apps that require complex workflows and API connections to be created. Deploying a basic logic app is easy, but applying the workflow is where I’m stuck.

I understand that Terraform is for deploying and managing resources, and not deploying code or logic configuration, so I am looking for other methods in which this can be done.

Can logic apps have their logic workflow applied during CI/CD, or is there a way in which I could leverage ARM templates to deploy the Logic App and configure its workflow?

Open to any suggestions, and keen to hear how others are doing this.

Many thanks

https://redd.it/f46o2f
@r_devops

best open source way to visualize infrastructure?

we use mainly Terraform to create our infra, fully on AWS. we wanted something to create/auto-generate a VPC graph with up 2 date resource view.

I tried cloud mapper and CloudFormer, trying to auto-generate VPC graphs with resources created within, but they all seemed not so intuitive.. anyone has experience with some tools which can do this better?

PS: not expecting something magical like humans drawing boxes themselves (you can stick a fork in the whole concept if humans can do this better) . but something decent for an auto-generated infra graphs

And please don't even mention Terraform graphs

https://redd.it/f6t6wt
@r_devops

AWS SQS - An API to delay queue messages

Hi, I’ve always found scheduling queue messages to be a pain with SQS and SNS. SQS has support only up to 15 minutes. Anything beyond that requires you to build your own service to handle the delays.

I've always wished there was an API that just handled the scheduling for me. As such I’m developing an API to make scheduling messages to SQS and SNS dead simple. Call the endpoint with your payload and delay time, and it automatically publishes to your queue or topic when you need it. I have a prototype here:

[www.schedulerapi.com](https://www.schedulerapi.com/)

I’d love to hear some feedback on this idea and whether it solves a real painpoint. I’d also love to hear about the last time anyone had to do job scheduling - how well did your solution work? Would anyone hacking on their own projects find this useful right now?

https://redd.it/f6xfqu
@r_devops

Any tools to manage created pipelines?

Used to use Jenkins, not migrated to GitHub actions.

So some pipelines are still in Jenkins and some are in GitHub.

Wonder if there is a tool to manage(deploy/monitor/logging) all pipelines together.

https://redd.it/f715dh
@r_devops

We just released a free open source multi-cloud deployment platform built on Kubernetes. We would love feedback if you want to check it out

It's based on a simple CLI, uses Terraform under the hood, and it's free to use. We just really want to get some feedback from folks so we can improve it! You can check it out here:

[https://docs.convox.com/getting-started/introduction](https://docs.convox.com/getting-started/introduction)

You can read our announcement blog post here:

[https://convox.com/blog/convox-multi-cloud](https://convox.com/blog/convox-multi-cloud)

https://redd.it/f6vczc
@r_devops

crosspost from r/sysadmin: Book I found, Fundamentals of Windows Performance Analysis

[https://www.reddit.com/r/sysadmin/comments/f6wu7u/i\_found\_out\_about\_this\_book\_recently\_thought\_id/](https://www.reddit.com/r/sysadmin/comments/f6wu7u/i_found_out_about_this_book_recently_thought_id/)

Perf guide for Windows, centering on ETW trace analysis. Someone suggested I share here, so doing so., Hope you find it interesting.

https://redd.it/f6yb3f
@r_devops