Ansible is not very consistent

Hello, I am relatively new to Ansible and it's great when it works but a lot of the times for longer playbooks on a lot of hosts (~300) it doesn't work too well. 300 isn't even that many hosts honestly.

I am not doing anything special in the playbook and it is very inconsistent between invocations. Sometimes it works fine, and other times it randomly is not able to reach the client in the middle of the playbook and I get these messages

fatal: 10.2.216.198: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Connection closed by 10.2.216.198 port 22", "unreachable": true}
fatal: 10.2.216.195: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Connection closed by 10.2.216.195 port 22", "unreachable": true}
fatal: 10.2.216.152: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Connection closed by 10.2.216.152 port 22", "unreachable": true}
fatal: 10.2.216.104: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Connection closed by 10.2.216.104 port 22", "unreachable": true}
fatal: 10.2.216.204: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Connection closed by 10.2.216.204 port 22", "unreachable": true}
fatal: 10.2.216.164: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Connection closed by 10.2.216.164 port 22", "unreachable": true}
fatal: 10.2.216.139: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Connection closed by 10.2.216.139 port 22", "unreachable": true}
fatal: 10.2.216.169: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Connection closed by 10.2.216.169 port 22", "unreachable": true}
fatal: 10.2.216.187: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Connection closed by 10.2.216.187 port 22", "unreachable": true}
fatal: 10.2.216.160: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Connection closed by 10.2.216.160 port 22", "unreachable": true}
fatal: 10.2.216.90: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Connection closed by 10.2.216.90 port 22", "unreachable": true}
fatal: 10.2.216.202: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Connection closed by 10.2.216.202 port 22", "unreachable": true}
fatal: 10.2.216.150: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Connection closed by 10.2.216.150 port 22", "unreachable": true}
fatal: 10.2.216.96: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Connection closed by 10.2.216.96 port 22", "unreachable": true}
fatal: 10.2.216.122: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Connection closed by 10.2.216.122 port 22", "unreachable": true}
fatal: 10.2.216.176: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Connection closed by 10.2.216.176 port 22", "unreachable": true}
fatal: 10.2.216.91: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Connection closed by 10.2.216.91 port 22", "unreachable": true}
fatal: 10.2.216.93: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Connection closed by 10.2.216.93 port 22", "unreachable": true}
fatal: 10.2.216.206: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Connection closed by 10.2.216.206 port 22", "unreachable": true}
fatal: 10.2.216.143: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Connection closed by 10.2.216.143 port 22", "unreachable": true}
fatal: 10.2.216.107: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Connection closed by 10.2.216.107 port

22", "unreachable": true}
fatal: 10.2.216.130: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Connection closed by 10.2.216.130 port 22", "unreachable": true}
fatal: 10.2.216.186: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Connection closed by 10.2.216.186 port 22", "unreachable": true}
fatal: 10.2.216.106: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Connection closed by 10.2.216.106 port 22", "unreachable": true}
fatal: 10.2.216.159: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Connection closed by 10.2.216.159 port 22", "unreachable": true}
fatal: 10.2.216.128: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Connection closed by 10.2.216.128 port 22", "unreachable": true}
fatal: 10.2.216.192: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Connection closed by 10.2.216.192 port 22", "unreachable": true}
fatal: 10.2.216.168: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Connection closed by 10.2.216.168 port 22", "unreachable": true}
fatal: 10.2.216.121: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Connection closed by 10.2.216.121 port 22", "unreachable": true}
fatal: 10.2.216.132: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Connection closed by 10.2.216.132 port 22", "unreachable": true}
fatal: 10.2.216.177: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Connection closed by 10.2.216.177 port 22", "unreachable": true}
fatal: 10.2.216.111: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Connection closed by 10.2.216.111 port 22", "unreachable": true}
fatal: 10.2.216.97: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Connection closed by 10.2.216.97 port 22", "unreachable": true}
fatal: 10.2.216.197: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Connection closed by 10.2.216.197 port 22", "unreachable": true}
fatal: 10.2.216.92: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Connection closed by 10.2.216.92 port 22", "unreachable": true}
fatal: 10.2.216.148: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Connection closed by 10.2.216.148 port 22", "unreachable": true}
fatal: 10.2.216.142: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Connection closed by 10.2.216.142 port 22", "unreachable": true}
fatal: 10.2.216.95: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Connection closed by 10.2.216.95 port 22", "unreachable": true}
fatal: 10.2.216.193: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Connection closed by 10.2.216.193 port 22", "unreachable": true}
fatal: 10.2.216.208: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Connection closed by 10.2.216.208 port 22", "unreachable": true}
fatal: 10.2.216.163: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Connection closed by 10.2.216.163 port 22", "unreachable": true}
fatal: 10.2.216.114: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Connection closed by 10.2.216.114 port 22", "unreachable": true}
fatal: 10.2.216.131: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Connection closed by 10.2.216.131 port 22", "unreachable": true}
fatal: 10.2.216.191: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Connection closed by 10.2.216.191 port 22", "unreachable": true}
fatal: 10.2.216.109: UNREACHABLE! => {"changed": false, "msg":

"Failed to connect to the host via ssh: Connection closed by 10.2.216.109 port 22", "unreachable": true}
fatal: 10.2.216.120: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Connection closed by 10.2.216.120 port 22", "unreachable": true}

I have turned pipelining on in the playbook but doesn't seem to be the culprit because I see these issues whether or not I have that option. The controller node and all the clients are on the same network so I really don't it's the network.

Any ideas?

https://redd.it/sadxb8
@r_devops

Learning new technology/ tools for better job opportunities

I've been working in an AppDynamics project for about 3 years now. Part of my job involves just sitting on the tool on which no new deployments are made and is just being retained because the management has spent a hefty sum of money on it.

I was fortunate enough to get an entry level job as a devops analyst right after finishing my degree. In the beginning I showed good potential, made good contribution to the project and even got promoted to devops engineer along with a substantive increase in my paycheck but now things are starting to feel stagnant.

I'm not getting a lot of new learning or opportunities within my organisation and honestly I feel like here I would keep doing the same stuff for as long as the management wants, possibly years.

I look around for new skills to learn and get confused in all that is available, kubernetes, docker, aws, azure, splunk, data analysis using R or python, automation tools like ansible or chef and all that is available in the market.

What should I start off with and get into in order to get better opportunities and good career growth.

Any suggestions would be appreciated. Thanks

https://redd.it/sajs0l
@r_devops

Automatically push files in S3 storage after download

Hello guys, sorry for asking in this channel. but, please share your idea how to push torrent downloaded files into s3 storage(and after that delete them locally)? It is need to work with huge file, like ~10-30 GB

Now I use s3fs-fuse, and ask my torrent client move files at this folder, but it work so bad..(really slow and easy crashing)

https://redd.it/sar0za
@r_devops

Need architecture advice for building a 3rd party extensions marketplace for my own SAAS Flutter app

I'm currently building a SAAS collaboration platform for volunteering engineers that provides several integrated tools such as posts, chats, documents, events and goals. I've designed the UI such that several side panels and screens can host views provided by 3rd party developers. My goal is to design a marketplace system where external vendors can showcase their extensions that can be installed on the app to enhance the workflow of the users. Similar to how Jira Cloud or Wordpress extensions work.

Marketplace requirements:

Provide a public API that can be queried by 3rd party extensions
A system for managing developer identity (for management and marketing reasons)
Payments tracking (the core system will extract a transaction fee)
Hooks (slots) to insert custom widgets and a system to register them
A way to limit access to resources that the client consents to share
Rate limiting features
Ability to sandbox the 3rd party widgets (I suppose webviews are the way)
Performance considerations
Ability to revoke extensions
I expect the vendors to host their own servers for caching their own extended data sets.

I will be hosting this infrastructure on a self-managed kubernetes cluster in digital ocean. I have ability to use/configure tools like reverse proxy to satisfy such demands. I don't intend to go head first in going public, but I do want to have a solid understanding of what needs to be there such that the app can grow towards achieving these goals.

Burning questions:

How to register vendors ids and how to register them as API clients?
How to safely expose life cycle events, GUI hooks and REST APis?
How to track transactions?
How to safely display widgets and webviews from vendors in Flutter?

So far my experience with working with such integrations is limited (general overview from working with CMSes in the past and some hands on experience from working as a Jira Cloud extension developer). I've been researching on google/youtube to find guidelines on how to build the marketplace infrastructure but relevant materials are hard to spot. Any advice/guidelines/architecture/links/videos you can share I'll be happy to consume.

PS: There's a live read only demo here of the app which will host the marketplace and I have a discord where I'm already brainstorming extension ideas with the upcoming vendors.

https://redd.it/sas3hj
@r_devops

Hikaru 0.9.0b released

Hikaru 0.9b has just been released. It includes support for Kubernetes 1.19 and has now been tested on Python 3.10.

Hikaru is a tool that provides you the ability to easily shift between YAML, Python objects/source, and JSON representations of your Kubernetes config files. It provides assistance in authoring these files in Python, opens up options in how you can assemble and customise the files, and provides some programmatic tools for inspecting large, complex files to enable automation of policy and security compliance.

Additionally, Hikaru allows you to use its K8s model objects to interact with Kubernetes, directing it to create, modify, delete, and watch resources.

This release may introduce some breaking changes as the underlying K8s swagger spec broke some previous assumptions, so please read the release notes before upgrading.

https://redd.it/sb0mlq
@r_devops

Any good examples/advice on how to do a good DRYish Terraform setup for multiple providers/region/environments?

Currently using Terraform in a single AWS region and looking to expand to both a different cloud provider and other AWS regions while having prod, staging and dev and potentially different accounts.

Any advice or examples on how to setup Terraform to keep things clear but hopefully reduce or manage the cut and paste?

Open to Terragrunt.

Part of me wants to use CDKTF or Pulumi, but I'm not sure if it's worth it to open that can of worms, especially considering the maturity.

Any advice is appreciated!

https://redd.it/sb1va0
@r_devops

Zombie exception in AWS - a little humor for a long Sunday

Read point 42.10

AWS Service Terms (amazon.com)

https://redd.it/sb3ujr
@r_devops

Stop using static cloud credentials in GitHub Actions

If you're using GitHub actions to deploy to AWS, you might be using a static IAM user, or think you have to configure your own runners in order to securely get temporary credentials.

Well, that's not true! You can use OIDC to get temporary credentials securely. I wrote a blog post to explain how this is possible for all cloud providers:

https://www.leebriggs.co.uk/blog/2022/01/23/gha-cloud-credentials.html

https://redd.it/sb5ql7
@r_devops

devops desktop configuration

Hello. I'm trying to assemble the desktop configuration and want to learn and run locally devops tools like docker, kubernetes, terraform, ansible, VMs, etc and was just wondering is there any problems with running those with i5-12600K or Ryzen 7 5700G processor, because those are my two choices for the cpus? Thanks in advance.

https://redd.it/sb7xtc
@r_devops

How to choose the best versioning mechanism?

We are developing a cross platform CLI tool (in GO) and a docker image which should work together . You are welcome to check the project on Github it is an open source. Currently we are using semver, but abusing it and I will explain.
We have develop and main branches. On each push to develop CI increments the "patch" number of semver and publish a new "Pre-release" version for both CLI and docker, both will have the same version (e.g 0.1.1, 0.1.2, 0.1.3, etc..). On each push to main CI increments the "minor" number of the semver and publish a new "Stable" version for both CLI and docker (e.g 0.2.0, 0.3.0, 0.4.0, etc..). As you can probably guess there are several problems with this method, for instance if we need to deploy a "hotfix" we will push fix to main and the minor version will be increased (for instance making 0.2.0 less advanced than 0.1.22).

So the requirements are, having two component CLI and docker which need to work together, having both "sable" and "unstable" releases.

Would like to get your opinion, or good article which can help us choose which versioning mechanism and how we should implement.

https://redd.it/sbe7xg
@r_devops

How to create programmatic service level indicators (SLIs)

At New Relic, we believe that programmatically tracked service level indicators (SLIs) are foundational to our site reliability engineering practice. When we have programmatic SLIs in place, we lessen the need to manually track performance and incident data. We’re able to reduce that manual toil because our DevOps teams define the capabilities and metrics that define their SLI data, which they collect automatically—hence “programmatic.” Learn more here: https://newrelic.com/blog/how-to-relic/programmatic-service-level-indicator?utm\_source=reddit&utm\_medium=organic\_social&utm\_campaign=amer-fy22-q4-slm&utm\_content=blog

https://redd.it/sbgazj
@r_devops

Azure DevOps agent pools

https://youtu.be/yr3IgGd0iK?sub\_confirmation=1

https://redd.it/sbguro
@r_devops

A conversation about how to enable high-velocity DevOps culture at your organization

Found the title attractive, check it out https://stackoverflow.blog/2021/12/13/a-conversation-about-how-to-enable-high-velocity-devops-culture-at-your-organization/

https://redd.it/sbi0o4
@r_devops

API Rest help

Anyone good with REST APIs? Need some help, User contacted me saying they don't have access to the product they paid for. from January 8th, 2022. This is the original record I pulled (some stuff has been altered for privacy) but does anything look off? Your help would be greatly appreciated as I'm still very new to REST APIs

"Id": "aceea67a-abf1-11ea-97c5-379a72eb235", "state": true, "productId": "56e5e750-aa5e-11ea-b0f5-97883f2103b1", "organizationId": "ef7cfb28-abf0-11ea-ab98-d7c6011880ac", "ownerId": "ef7cfb28-abf0-11ea-ab98-d7c6011880ac", "ownerType": "ORGANIZATION", "createdAt": "2022-06-11T14:41:46.499Z", "updatedAt": "2022-12-21T16:41:42.588Z", "effectiveStartDate": "2022-06-11T00:00:00.000Z", "effectiveEndDate": "2022-12-31T00:00:00.000Z",

https://redd.it/sbgqlz
@r_devops

Filebeat autodiscover on k8s

Recently i got a new job in pure devops.

Lads here work mainly on k8s clusters. I have been tasked with creating network policies to secure all pods as much as possible and this led me to a few monitoring pods, like filebeat.

I come from a sysadmin background so i do have experience with beats and ELK, but bare metal style. So a filebeat service per server configured properly to handle server and app logs.

Here they are deploying a daemon set which spawns a filebeat pod per node, mounts ***/var/log*** and container logs, ***/var/lib/docker/containers*** and uses **autodiscover**.

Now this is the part that bugs me and i do not understand. How does the kubernetes provider autodiscover works? The way i see it right now, on these setups, is that each filebeat pod works the same way as a bare metal works. it monitors and parses all logs found under:

/var/log/containers/*-${container.id}.log

or

/var/lib/docker/containers/${container.id}/${container.id}-json.log

on it's perspective node(daemon set).

What i would expect from such a utility is to spawn a pod and autodiscover would somehow, with some extra configuration, the filebeat pod to find out with timed sampling that a new pod/node spawned and exec something like kubectl logs -f ${[pod.name](https://pod.name)} and not simply read the mounted logs. It seems like a wrong usage or approach to me but i might just be wrong here plus audodiscover mentions -nodes- but all i see here is information for pods and pods only.

https://redd.it/sbjapj
@r_devops

Provide SFTP access to Non-tech people

So an organization sends us files in our SFTP server and I fetch them and send it to our operations team but now the operation teams wants access to remove my dependency to make the process quicker. I get those files in our server in private subnet for security purpose.

How can I provide them access as they don't know linux commands?

https://redd.it/sbkl93
@r_devops

CircleCI Adds Free Unlimited Access Tier to CI/CD Service

<https://devops.com/circleci-adds-free-unlimited-access-tier-to-ci-cd-service/>

https://redd.it/sblwwt
@r_devops

CircleCI Adds Free Unlimited Access Tier to CI/CD Service

<https://devops.com/circleci-adds-free-unlimited-access-tier-to-ci-cd-service/>

https://redd.it/sblwwt
@r_devops

Pipeline Patterns

Is there a good reference page with examples on how to setup the following pipeline situations:

Full deployment?

Partial deployment with only updated files being pushed?

Pipelines that call other pipelines that include infrastructure then code related parts?

https://redd.it/sblmuf
@r_devops