A library to improve scalability of yaml

Hi everyone, hope you all doing well!


Let me assume I'm not the only one who both love and hate YAML at the same time, love it because it's human-readable, and hate it due to its lack of composability and reusability.


#### The problem


There are `tags`1 in yaml spec, which supports custom rendering of field values. In theory, you can do anything with these custom tags, so you can make yaml scalable with these tags. But in reality, seldom case there is custom tag support in existing applications. One reason could be it's simply hard due to lack of parsing library support, or just not worth it to implement such support, and another reason I can think about is these tags make yaml files look overly verbose.


But to make yaml scalable, we have to overcome such obstacles, especially when it comes to existing DevOps solutions, most of which consist of many yaml files, and we want to manage these files for all kinds of use cases.


#### My solution


I implemented `rs`2, a golang library providing easy integration of custom rendering, using its unique rendering suffix syntax or ordinary yaml tag (with special tag prefix and certain limitations).


Developers can support very flexible yaml configuration with `rs`2, and still keep their config type-checked; End-users will be able to configure the app with almost unlimited flexibility once granted a proper set of renderers implemented by developers. Please have a look at the `rs`2 if you want to know how it satisfies both developers and end-users, and also check out our task runner `dukkha`3 built with `rs`2 making YAML files Makefiles.


I wish this library could inspire new ideas around yaml, or at least can improve yaml's user experience in existing or future tools (currently limited to tools written in go). More ambitiously, I'd like to make changes happen in the existing DevOps pipelines that rely heavily on plain text templating for yaml.


What do you think about it? Do you like the rendering suffix syntax? Or do you think I'm doing it in the right direction? Leave your comments here or in github discussions.


Looking forward to hearing from you!


Thanks for your interest!


P.S.


I'm a long-time follower of this subreddit, but using teddit for privacy reasons, I learned a lot from your posts. Recently I got the idea to improve existing tools using yaml, and here it is. Fix me if I'm doing posting the wrong way.


1: https://yaml.org/spec/1.2.2/#3212-tags
2: https://github.com/arhat-dev/rs
3: https://github.com/arhat-dev/dukkha

https://redd.it/rbosz5
@r_devops

Onboarding New Engineers

Typically for the first week(s) we have the engineer

Meet with the manager for a 1-on-1 session every day for the first few weeks then scale it down
15-minute meet and greets with all the primary teammates (BA, Dev's and OPs) on first week
Have a team zoom ice-breaker game day at end of the week (everyone is remote)
We create a support ticket welcoming them to the team listing out many of the applications used and ask them to familiarize themselves with the different tools.

What does your team do to onboard new engineers?

https://redd.it/rbzxud
@r_devops

Do you terraform your application registrations in Azure AD, or is there a more intelligent way that is less work?

Hey all

I've just written a module to handle this but the amount of work involved is quite staggering for our use case.

Is there a better (Microsoft native, perhaps) method of managing app registrations and enterprise applications vs doing it all in TF?

https://redd.it/rc0pjs
@r_devops

Customer Success Engineer? Curious about those with experience in a role such as this or similar

I am currently a DevOps Cloud Engineer and looking at switching companies. A role I've been learning more about is the 'Customer Success Engineer' (post-sales role) as it utilizes my engineering background with my soft skills / communication (which is something I excel in). Has anyone made a similar jump (to or from) in their career?

Thanks in advance to everyone for your time and help, it is greatly appreciated!

https://redd.it/rc0f6d
@r_devops

Why the heck is there not a repo for Prometheus?

So annoying that I can’t just install Prometheus from a repo. Makes installing it take extra steps especially when working with ansible and you have to run several tasks to find the latest version and download/extract it etc. it seems like there’s a repo for everything else but I’ve never come across one for Prometheus.

https://redd.it/rc33xz
@r_devops

Survey on Multi/Hybrid Cloud CI/CD systems

Hello there. I am a researcher doing a study on CI/CD system in multi/hybrid cloud environments. I am trying to evaluate if there is a use case for offering multi-cloud ready CI/CD systems. Here is the survey link - https://forms.gle/dLNGSJipWxKKBfVVA. I'd highly appreciate if you can help out with answering this survey. Thanks a lot in advance.

Context

This is a research study being conducted to investigate the multi-cloud/ hybrid-cloud use-cases for continuous software delivery (CI/CD) systems.

What is Multi-Cloud

The future of infrastructure platform is multi-cloud, meaning that users will invariably use more than one type of cloud for their infrastructure needs. This could mean they use a private cloud (on-prem deployments) along with one or more public clouds (AWS, Google Cloud, Azure, etc) or in some cases skip private clouds altogether and use one or more public clouds. Customers tend to avoid deploying all their workloads on the same cloud due to concerns of vendor lock-in and to protect against unfair pricing practices from being locked into one cloud platform.

CI/CD on Multi-Cloud

Given that there will be more than one cloud platform that users will use to deploy their software, it is expected that software they develop will need to be deployed/ delivered onto different cloud platforms, each with their own configuration, policies and semantics.

We are trying to understand if there is a use case to offer this CI/CD support for multiple cloud platforms and simplify user workflows that are manual today.

https://redd.it/rbm8jv
@r_devops

How often do you release to production?

I'm weighing up what I should aim for. I know very subjective, but interested to hear peoples experiences with what worked for them.

https://redd.it/rc4179
@r_devops

Automation of Pipeline Creation

Hello,

I'm a software engineer working on creating a CICD architecture for an organization with over one hundred repositories. All of our applications are hosted on AWS, primarily applications deployed through the serverless framework or fargate.

We are currently using Jenkins for CICD, but want to move away from it due to support for it going away company-wide. We want a solution in place that is self-managed.

Given the AWS centric nature of our org, I primarily explored AWS CodeBuild and CodePipeline as the means for an alternative. However, I've not been thoroughly impressed by the means to automate the process of creating pipelines. I've explored using cloudformation to script the process out, so a one time script that builds out the pipeline for our applications. Though, having boilerplate code in every repository to create the pipeline, and having someone run the script with the correct permissions seems awfully inconvenient especially during our transition over. Given that the vast majority of our repositories use the same deployment process, having this duplicated code seems unnatural and not DRY at all.

Am I missing something, or am I to ambitious to think there has to be a better way to automate this process? The boilerplate code required in each repository for jenkins is incredibly small compared to aws codebuild, where all we need to do is specify which type of deployment the application is. With CodeBuild we have to include a buildspec, the cloudformation template for the pipeline, which you could argue gives more customization for your build, however it also opens up a gateway of misuse and misconfiguration that could lead to issues that would be more difficult to debug and assist other teams with.

I'm not going to pretend I'm an expert in devops, but there has to be a more centralized and automated way of creating these pipelines. It is driving me crazy to think that this much boilerplate is needed and can't be automated in a better fashion. Is this really the approach people take using this platform? Manually creating each pipeline?

https://redd.it/rc3jkz
@r_devops

Am i a devops engineer ?

Hey ya'll. My name is Isaac, 24 years old and i'm from the Netherlands.

Currently i'm working for a secondment, and i've been employed at University of Leiden to do some workplace management by removing their old workplace, and placing a new one (new display (sometimes a dockingstation extra) keyboard, mouse)

But that's not what i want to do, i want to become a devops engineer.

Currently i can work with:

C#, Ansible, YAML, Json, Jquery, Ajax, PHP, CSS/HTML, docker (compose), and kubernetes. I can create a server in Linux, host it from on prem to the internet with a traefik reverse proxy/loadbalancing deployed. I could create a new WAN that has to be used, or configure the local WAN. I can code, script and now how to system management of Windows (xp, 7, 10 and now 10). i can create my own powershell scripts and deploy them on prem, and place my code in a local repository or github.

​

I do some work for customers in which i automate alot of workprocesses which would be done by hand. I could maintain contact with the Dev department and Ops department, but i do not know in what degree you should also have work experience as a manager to become a devops engineer (since you're almost managing two or more teams to adopt a different work mentality) I'm a nerd for sure. I wake up with devops and go to sleep with it. There is nothing else that i would want to become in my career (and i think i'm almost there) but my problem also seems to be i do not have any certificates.

TLDR: i have around 4 years coding experience with different languages all which are used to automate or develop things. I have around 4/5 years IT system management work experience and i would like to transition to becoming a devops engineer (or maybe i can already be considered one)

I've read alot, and the only thing i do not seem to have are the following things that in my opinion hinder me to become a full fledged devops engineer:

1) never managed a development or ops team, but i do know how they operate, and i can bridge them

2) do not have any masters degree, or certificates which prove i'm a engineer

3) never worked as a devops engineer (except for some freelance projects i do for customers i get)

​

someone i spoke with which works at google said that i'm already a devops engineer, and i just need to market myself as such. I could put up my resume, maybe that will give people more food to see what i've have done up till now (school and bussiness wise)

https://redd.it/rb0xfi
@r_devops

Just got an online internship at a local firm. Need some advice.

So its a 3 months training program, with assessments in between each month and after 3 months 2 people will be selected for the job. They gave us 5 people a login to acloudguru and directed us to follow the AWS SAA-02 course.

Everday we're supposed to email our daily progress to 3 people in the training team.

I am completely new to this sort of thing.
What i want to know is, is this the usual way of training new employees?

And am i just supposed to put something like "Chapter 7 EBS: Done" in the progress email? How is this credible progress they cant even track individual progress via acloudguru since we all have the same login.

im so confused because its all online and the only interaction i have with the company is through emails sent by HR.

https://redd.it/ravo0g
@r_devops

What do you use CodeCommit or git?

I mean I have just started using AWS, they have CodeCommit, so I was just thinking if it would be good idea to use CodeCommit instead of open source git?

https://redd.it/rcakp1
@r_devops

Anyone have a small devops project idea?

I just got an old Cisco switch! I’m excited to mess around with it so I was wondering if any of you seasoned pros had any ideas on how I can combine the following:

This handy dandy Cisco 2950 switch

Amazon web services

This old windows 10 pro machine (which I use for hyper-V capabilities)

https://redd.it/rcadd1
@r_devops

Digital Ocean challenge for Kubernetes 2021

Hi everyone I am new to k8s and learning. How do I go about starting this challenge? I am planning to do the Deploy an internal container registry.

https://redd.it/rb0a3n
@r_devops

Building dApps

Honestly speaking, I think devOps projects in cryto is gonna skyrocket by 2021

https://redd.it/raziy9
@r_devops

How to diagnose this repeating log output issue?

Hi, our team recently encountered a strange problem with our kubernetes cluster. "kubectl log -f" when the follow option is used, we get continously repeating log output. On the node, using crictl logs -f gets the same behaviour. I don' know where to look for issues. Does anyone have any suggestion? I can't get any valuable support from our cloud work nodes provider unfortunately. They claim it is the service application's problem.

https://redd.it/raraca
@r_devops

Authentication error using API Gateway

Hello,

I have an AWS API that calls a Lambda function, both built by SAM. When I test it via the console, it is works. When I hit it via a browser, I get:

{"message":"Missing Authentication Token"}

I have tried to use the following URL:

https://XXXXxf6cil.execute-api.us-east-1.amazonaws.com/Prod/POST/[email protected]
https://XXXXxf6cil.execute-api.us-east-1.amazonaws.com/Prod/[email protected]
https://XXXXxf6cil.execute-api.us-east-1.amazonaws.com/[email protected]

The URL for my production stage is:

https://XXXXxf6cil.execute-api.us-east-1.amazonaws.com/Prod

When I go to perform a (successful) test of API via the console, the request is what I have below. So basically, I thought that I could add the request on to the end of the production stage URL and use that?

/films?[email protected]

​

I haven't configured any authentication so I suspect the problem is elsewhere? There resource policy on the lambda function looks like it has been generated and applied correctly by SAM. This (films/POST) is currently the only method for this API. If I look at the Method Request, Authorization is set to "None".

https://redd.it/rce80r
@r_devops

AWS vs. Azure for dockerized auto-scaling applications

Hi there,

Our client has proposed that we should look over switching from Azure to AWS, but I want to make sure that this is a good choice. The applications are Python based (Django) Applications that run in Docker, some are just web-servers, some are for handling CRON-tabs. Served via NGINX and uWSGI and CI taken care of by Github actions.

I have been working with Azure for about one month, and I have run into so many quirks and issues that I've almost lost my sense of sanity sitting with their solutions. It's half baked, documentation isn't very good and error logging is catastrophic at times. Short and sweet - I hate it wholeheartedly and want to throw it into the fires of hell every other day, so I'm inclined to go towards AWS - but wanted to check with you who maybe have experience from both/the particular experience of our stack in AWS who can tell how it works for you.

Looking forward reading your replies. Thanks.

https://redd.it/ra7ihh
@r_devops

Don't Just Track Your Machine Learning Experiments, Version Them - Distributed Versioning vs. Centralized Tracking for ML Experiments with DVC and Git

Machine learning experiments often get split between Git for code and experiment tracking tools for meta-information - because Git can't manage or compare all that experiment meta-information, but it is still better for code.

The following guide explains how to apply DVC for ML experiment versioning that combines experiment tracking and version control: **[Don't Just Track Your ML Experiments, Version Them](https://dvc.org/blog/ml-experiment-versioning)** - Instead of managing these separately, keep everything in one place and get the benefits of both, like:

* **Experiments as code:** Track meta-information in the repository and version it like code.
* **Versioned reproducibility:** Save and restore experiment state, and track changes to only execute what's new.
* **Distributed experiments:** Organize locally and choose what to share, reusing your existing repo setup.

Experiment versioning treats experiments as code. It saves all metrics, hyperparameters, and artifact information in text files that can be versioned by Git, which becomes a store for experiment meta-information. The article above shows how with DVC tool, you can push experiments just like Git branches, giving you flexibility to share experiment you choose.

https://redd.it/rcgaio
@r_devops

Best DevOps courses (between beginner and advanced)?

I heard good things about Stephane Maarek's DevOps course, but I'm afraid it's a little too specific for me, since it includes AWS only, and the IaC part is with AWS CloudFormation only. But if I can apply the things I learn from that when I'm creating Terraform scripts for Azure, then that's totally fine with me.
And I also already completed Stephane's AWS Developer course. Oh and one more thing, sometimes I felt like the Developer course was too focused on teaching exactly enough to pass the Certification exam, and it skipped explaining some things more in-depth.


But if anyone can suggest a really good, possibly more generic course on DevOps topic, I'd be very grateful. Also it's okay if it's several seperate courses on topics that are neccessary for a DevOps engineer. I have motivation but am having trouble deciding where to start and how..

https://redd.it/rchk6q
@r_devops

Jenkins pipeline for npm project for code deployment to sandbox

Hi can anyone plz help me out with jenkins pipeline for deployment of 2repos in bitbucket to sandbox environment using npm commands

https://redd.it/rciq8l
@r_devops

Git and GitHub Tutorial – Version Control for Beginners

If you're a beginner developer and want to learn about Git and GitHub then this article is for you.

This tutorial will help you understand what Git and version control are, the basic Git commands you need to know, how you can use its features to boost your work efficiency, and how to extend these features using GitHub.

Check it out: https://www.freecodecamp.org/news/git-and-github-for-beginners/

https://redd.it/rchs9r
@r_devops