Learning new technology/ tools for better job opportunities

I've been working in an AppDynamics project for about 3 years now. Part of my job involves just sitting on the tool on which no new deployments are made and is just being retained because the management has spent a hefty sum of money on it.

I was fortunate enough to get an entry level job as a devops analyst right after finishing my degree. In the beginning I showed good potential, made good contribution to the project and even got promoted to devops engineer along with a substantive increase in my paycheck but now things are starting to feel stagnant.

I'm not getting a lot of new learning or opportunities within my organisation and honestly I feel like here I would keep doing the same stuff for as long as the management wants, possibly years.

I look around for new skills to learn and get confused in all that is available, kubernetes, docker, aws, azure, splunk, data analysis using R or python, automation tools like ansible or chef and all that is available in the market.

What should I start off with and get into in order to get better opportunities and good career growth.

Any suggestions would be appreciated. Thanks

https://redd.it/sajs0l
@r_devops

Automatically push files in S3 storage after download

Hello guys, sorry for asking in this channel. but, please share your idea how to push torrent downloaded files into s3 storage(and after that delete them locally)? It is need to work with huge file, like ~10-30 GB

Now I use s3fs-fuse, and ask my torrent client move files at this folder, but it work so bad..(really slow and easy crashing)

https://redd.it/sar0za
@r_devops

Need architecture advice for building a 3rd party extensions marketplace for my own SAAS Flutter app

I'm currently building a SAAS collaboration platform for volunteering engineers that provides several integrated tools such as posts, chats, documents, events and goals. I've designed the UI such that several side panels and screens can host views provided by 3rd party developers. My goal is to design a marketplace system where external vendors can showcase their extensions that can be installed on the app to enhance the workflow of the users. Similar to how Jira Cloud or Wordpress extensions work.

Marketplace requirements:

Provide a public API that can be queried by 3rd party extensions
A system for managing developer identity (for management and marketing reasons)
Payments tracking (the core system will extract a transaction fee)
Hooks (slots) to insert custom widgets and a system to register them
A way to limit access to resources that the client consents to share
Rate limiting features
Ability to sandbox the 3rd party widgets (I suppose webviews are the way)
Performance considerations
Ability to revoke extensions
I expect the vendors to host their own servers for caching their own extended data sets.

I will be hosting this infrastructure on a self-managed kubernetes cluster in digital ocean. I have ability to use/configure tools like reverse proxy to satisfy such demands. I don't intend to go head first in going public, but I do want to have a solid understanding of what needs to be there such that the app can grow towards achieving these goals.

Burning questions:

How to register vendors ids and how to register them as API clients?
How to safely expose life cycle events, GUI hooks and REST APis?
How to track transactions?
How to safely display widgets and webviews from vendors in Flutter?

So far my experience with working with such integrations is limited (general overview from working with CMSes in the past and some hands on experience from working as a Jira Cloud extension developer). I've been researching on google/youtube to find guidelines on how to build the marketplace infrastructure but relevant materials are hard to spot. Any advice/guidelines/architecture/links/videos you can share I'll be happy to consume.

PS: There's a live read only demo here of the app which will host the marketplace and I have a discord where I'm already brainstorming extension ideas with the upcoming vendors.

https://redd.it/sas3hj
@r_devops

Hikaru 0.9.0b released

Hikaru 0.9b has just been released. It includes support for Kubernetes 1.19 and has now been tested on Python 3.10.

Hikaru is a tool that provides you the ability to easily shift between YAML, Python objects/source, and JSON representations of your Kubernetes config files. It provides assistance in authoring these files in Python, opens up options in how you can assemble and customise the files, and provides some programmatic tools for inspecting large, complex files to enable automation of policy and security compliance.

Additionally, Hikaru allows you to use its K8s model objects to interact with Kubernetes, directing it to create, modify, delete, and watch resources.

This release may introduce some breaking changes as the underlying K8s swagger spec broke some previous assumptions, so please read the release notes before upgrading.

https://redd.it/sb0mlq
@r_devops

Any good examples/advice on how to do a good DRYish Terraform setup for multiple providers/region/environments?

Currently using Terraform in a single AWS region and looking to expand to both a different cloud provider and other AWS regions while having prod, staging and dev and potentially different accounts.

Any advice or examples on how to setup Terraform to keep things clear but hopefully reduce or manage the cut and paste?

Open to Terragrunt.

Part of me wants to use CDKTF or Pulumi, but I'm not sure if it's worth it to open that can of worms, especially considering the maturity.

Any advice is appreciated!

https://redd.it/sb1va0
@r_devops

Zombie exception in AWS - a little humor for a long Sunday

Read point 42.10

AWS Service Terms (amazon.com)

https://redd.it/sb3ujr
@r_devops

Stop using static cloud credentials in GitHub Actions

If you're using GitHub actions to deploy to AWS, you might be using a static IAM user, or think you have to configure your own runners in order to securely get temporary credentials.

Well, that's not true! You can use OIDC to get temporary credentials securely. I wrote a blog post to explain how this is possible for all cloud providers:

https://www.leebriggs.co.uk/blog/2022/01/23/gha-cloud-credentials.html

https://redd.it/sb5ql7
@r_devops

devops desktop configuration

Hello. I'm trying to assemble the desktop configuration and want to learn and run locally devops tools like docker, kubernetes, terraform, ansible, VMs, etc and was just wondering is there any problems with running those with i5-12600K or Ryzen 7 5700G processor, because those are my two choices for the cpus? Thanks in advance.

https://redd.it/sb7xtc
@r_devops

How to choose the best versioning mechanism?

We are developing a cross platform CLI tool (in GO) and a docker image which should work together . You are welcome to check the project on Github it is an open source. Currently we are using semver, but abusing it and I will explain.
We have develop and main branches. On each push to develop CI increments the "patch" number of semver and publish a new "Pre-release" version for both CLI and docker, both will have the same version (e.g 0.1.1, 0.1.2, 0.1.3, etc..). On each push to main CI increments the "minor" number of the semver and publish a new "Stable" version for both CLI and docker (e.g 0.2.0, 0.3.0, 0.4.0, etc..). As you can probably guess there are several problems with this method, for instance if we need to deploy a "hotfix" we will push fix to main and the minor version will be increased (for instance making 0.2.0 less advanced than 0.1.22).

So the requirements are, having two component CLI and docker which need to work together, having both "sable" and "unstable" releases.

Would like to get your opinion, or good article which can help us choose which versioning mechanism and how we should implement.

https://redd.it/sbe7xg
@r_devops

How to create programmatic service level indicators (SLIs)

At New Relic, we believe that programmatically tracked service level indicators (SLIs) are foundational to our site reliability engineering practice. When we have programmatic SLIs in place, we lessen the need to manually track performance and incident data. We’re able to reduce that manual toil because our DevOps teams define the capabilities and metrics that define their SLI data, which they collect automatically—hence “programmatic.” Learn more here: https://newrelic.com/blog/how-to-relic/programmatic-service-level-indicator?utm\_source=reddit&utm\_medium=organic\_social&utm\_campaign=amer-fy22-q4-slm&utm\_content=blog

https://redd.it/sbgazj
@r_devops

Azure DevOps agent pools

https://youtu.be/yr3IgGd0iK?sub\_confirmation=1

https://redd.it/sbguro
@r_devops

A conversation about how to enable high-velocity DevOps culture at your organization

Found the title attractive, check it out https://stackoverflow.blog/2021/12/13/a-conversation-about-how-to-enable-high-velocity-devops-culture-at-your-organization/

https://redd.it/sbi0o4
@r_devops

API Rest help

Anyone good with REST APIs? Need some help, User contacted me saying they don't have access to the product they paid for. from January 8th, 2022. This is the original record I pulled (some stuff has been altered for privacy) but does anything look off? Your help would be greatly appreciated as I'm still very new to REST APIs

"Id": "aceea67a-abf1-11ea-97c5-379a72eb235", "state": true, "productId": "56e5e750-aa5e-11ea-b0f5-97883f2103b1", "organizationId": "ef7cfb28-abf0-11ea-ab98-d7c6011880ac", "ownerId": "ef7cfb28-abf0-11ea-ab98-d7c6011880ac", "ownerType": "ORGANIZATION", "createdAt": "2022-06-11T14:41:46.499Z", "updatedAt": "2022-12-21T16:41:42.588Z", "effectiveStartDate": "2022-06-11T00:00:00.000Z", "effectiveEndDate": "2022-12-31T00:00:00.000Z",

https://redd.it/sbgqlz
@r_devops

Filebeat autodiscover on k8s

Recently i got a new job in pure devops.

Lads here work mainly on k8s clusters. I have been tasked with creating network policies to secure all pods as much as possible and this led me to a few monitoring pods, like filebeat.

I come from a sysadmin background so i do have experience with beats and ELK, but bare metal style. So a filebeat service per server configured properly to handle server and app logs.

Here they are deploying a daemon set which spawns a filebeat pod per node, mounts ***/var/log*** and container logs, ***/var/lib/docker/containers*** and uses **autodiscover**.

Now this is the part that bugs me and i do not understand. How does the kubernetes provider autodiscover works? The way i see it right now, on these setups, is that each filebeat pod works the same way as a bare metal works. it monitors and parses all logs found under:

/var/log/containers/*-${container.id}.log

or

/var/lib/docker/containers/${container.id}/${container.id}-json.log

on it's perspective node(daemon set).

What i would expect from such a utility is to spawn a pod and autodiscover would somehow, with some extra configuration, the filebeat pod to find out with timed sampling that a new pod/node spawned and exec something like kubectl logs -f ${[pod.name](https://pod.name)} and not simply read the mounted logs. It seems like a wrong usage or approach to me but i might just be wrong here plus audodiscover mentions -nodes- but all i see here is information for pods and pods only.

https://redd.it/sbjapj
@r_devops

Provide SFTP access to Non-tech people

So an organization sends us files in our SFTP server and I fetch them and send it to our operations team but now the operation teams wants access to remove my dependency to make the process quicker. I get those files in our server in private subnet for security purpose.

How can I provide them access as they don't know linux commands?

https://redd.it/sbkl93
@r_devops

CircleCI Adds Free Unlimited Access Tier to CI/CD Service

<https://devops.com/circleci-adds-free-unlimited-access-tier-to-ci-cd-service/>

https://redd.it/sblwwt
@r_devops

CircleCI Adds Free Unlimited Access Tier to CI/CD Service

<https://devops.com/circleci-adds-free-unlimited-access-tier-to-ci-cd-service/>

https://redd.it/sblwwt
@r_devops

Pipeline Patterns

Is there a good reference page with examples on how to setup the following pipeline situations:

Full deployment?

Partial deployment with only updated files being pushed?

Pipelines that call other pipelines that include infrastructure then code related parts?

https://redd.it/sblmuf
@r_devops

Specifying IAM Role in Packer

So I am currently trying to default to using IAM roles/polices for packer to use rather than specifying environment variables.

So the question is in two parts which is:


1. When creating the IAM role, how is this specified within the HCL template?

2. When I use this specific IAM role, will packer require any other credentials to initiate the build or will it automatically recognise using the name of the profile?

Bonus question:

If I'm using Jenkins to create a pipeline, how do I specify the same IAM role to be used, will any work be needed on the slave (windows server) or on the Jenkins instance for it to initiate a connection to AWS to initiate the build?


I've done some research on how this is possible but all examples i haven't been able to visualise how it's possible within a working piece of code: https://www.packer.io/plugins/builders/amazon#iam-task-or-instance-role

Feel free to let me know if clarification is needed.

https://redd.it/sbke47
@r_devops

Help comparing CircleCI to Github Actions from cost perspective only

The pricing for both of these are rather convoluted and depends on number of users, number of minutes needed for the runners. So I want to set some constants to de-convolute it and just compare pricing between GitHub Actions and CircleCI, irregardless of the differing feature sets. Here are the constants:

Runner: Linux 2-core machine
Minutes needed: 1200/day
Number of users/seats: 10
Assume the use of dependency caching

Can anyone speak to the cost of just these two platforms with the given constants? Are there other variables that should be fixed to make the comparison more straight-forward?

https://redd.it/sbpx3h
@r_devops

What will better futureproof your career - learning serverless or containers/k8s?

Obviously if I had all the time in the world I'd learn both, but I'm curious what you all think might be the better option?

https://redd.it/sbs9pi
@r_devops