I want to log everything on kubernetes

Hey guys, I'm a junior DevOps and wanted to know how do you store your kubernetes logs.

I searched and investigate tools like graylog and Loki, but wanted to see if there's more.

Currently I'm using azure cloud so my clusters have azurefile as PVC and storage class.

https://redd.it/p61nld
@r_devops

terraform on mac m1 (arm64) ?

I need to install [email protected] but it looks unavailable: https://releases.hashicorp.com/terraform/0.14.10/

how can I use this version of terraform on the m1 mac? I've been trying/researching for a while.

https://redd.it/p6lm9r
@r_devops

DevOps Tools landscape chart/diagram generator

My manager asking me for some flashy diagram listings of all the devops tool chains we are using. I find it hard creating using powerpoint. Any easy online tool/generator, if anyone knows.

https://redd.it/p61e80
@r_devops

Federated prometheus and integration of kube-state-metrics

I have a number of k8s clusters which have installed a prometheus instance(using prom operator). I have a central separate prometheus which collects all metrics(cadvisor, kubelet) from all the k8s clusters using the /federated endpoint(using a load balancer service). I have recently installed kube-state-metrics on some of the clusters and i was able to expose those metrics using a separate LB service.


My end goal is to be able to have only one LB service exposed from each cluster and expose all metrics from that endpoint.


What i want is to be able to merge all metrics (cadvisor, kubelet, kube-state-metrics) into the local prometheus and expose all metrics externally using the federated endpoint of the local prometheus.


I was unable to find something similar online but i am sure other people have had experience on how to do this.

https://redd.it/p6mzv2
@r_devops

Creating a pipeline to update the AWS credentials monthly

Hi All,
Our AWS infrastructure updates the secret key and user access key id on a monthly basis. I want to create a pipeline in Jenkins which fetches the credentials from aws and updates the AWS credentials if the present ones are expired.
Can someone please help with the approach and how it can be done.
Any leads will be helpful.

https://redd.it/p5b7f6
@r_devops

Why Use Jenkins?

hey DevOps folks i'm a TFS to VSTS to Azure DevOps guy. I dabble with GitHub and Github actions but never Jenkins. So why do companies use it? is it the free factor? or is any good?

https://redd.it/p6qn7x
@r_devops

Linux Administrator to DevOps RoadMap : Programming, Administration, Containers, Infrastructure as Code, CICD, Logging, Monitoring, Cloud etc.(Useful to learn how your app is deployed using docker, kubernetes, with automation, and a CI pipeline)

Since about 2 years now I'm working on a series of short YouTube videos about basics of DevOps that helps you to move from Admin to DevOps engineer, You can find it here: https://www.youtube.com/channel/UCoOq-DtESvayx5yJE5H6-qQ/playlists?view=1&sort=lad&flow=grid

Contents:

1. Continuous integration and Continuous Deployment

2. Containers - Docker

3. Container Orchestration - Kubernetes

4. Logging & Visualization - Elk, Loki, Grafana

5. Monitoring - Prometheus, Splunk

6. Configuration Management - Ansible

These lessons that start with "Hello World," assuming one has absolutely no programming experience, and goes on to cover the basics on CICD, Containers, Kubernetes Pods, Deployments, Services, Setting up Jenkins, Docker, Podman, Minikube, Ansible, ELK, Git, most used commands, Pipeline jobs with Sonarqube quality checks, Artifactory for binary management, ansible playbooks, real use cases, deploying docker containers using ansible, how to convert your playbook in to roles, Monitoring docker containers, windows and linux servers, and more & more! . . All the lessons all hosted on YouTube and organized into a course. Hope you like it!

Refer the above link for complete tutorials.

https://redd.it/p6s0h2
@r_devops

Migrating from Node Redis to Ioredis: a slightly bumpy but faster road

Redis is a very important tool in the Ably toolbox, and since most of our backend code is written in Node.js, using the recommended Node Redis client was natural... but we started hitting some performance issues, and Ioredis came into the picture.

In this new blog post of the Ably Engineering series, we tell the story of how and why we migrated from Node Redis client library to Ioredis to suit our use case better, and a couple of bumps we faced along the way:

Migrating from Node Redis to Ioredis: a slightly bumpy but faster road

Hope you enjoy it, feedback is always welcomed 🤗

https://redd.it/p6ritj
@r_devops

meta Why do people keep posting YouTube videos instead of gitHub Repos? I want to read and replicate your infrastructure code, not watch a video.

Mini meta rant: how is it that YouTube videos keep getting posted here? Do other people really prefer YouTube content over a solid README with code examples? You know, like what you do on the actual job. Every other day it's another YouTube video and I'm getting a little tired of it. I'm not going to go through your 10 video playlist to learn concepts when there is no way to replicate it in my own homelab without manually typing in what you're saying.

https://redd.it/p6ti1u
@r_devops

AWS Cognito vs Auth0

I have an application that I've built and am in the process of rebuilding and am thinking of redoing the auth to use Auth0 instead of AWS Cognito since I have heard Auth0 is better and this seems like a good time to switch before I have to deal with migrations. I've done a bit of research on these two but am curious to see what this sub thinks of them in regards to management as well as just functionality.

https://redd.it/p6uezf
@r_devops

What are best practices integration testing as a platform grows?

When multiple changes are in flight across the various services/teams in a platform, how do you best handle this queue?

One option I've seen executed both poorly and well is to only allow one change at a time until it either hits prod or fails and is rolled back. This requires a strong automated testing suite to allow for a fast feedback loop but even then it doesn't scale well once you approach a large number services.

An alternative would be automating a full stack creation/termination for each change and test against current prod but this could create friction (sure, there's workarounds though) if co-dependent services make it to prod which haven't been tested against each other.

Anyone care to comment a workflow that's successful in their org?

https://redd.it/p6ubm1
@r_devops

Asking advise for cloud based startup

Me and a couple of my friends doing a startup from an idea we got together, I'm a .NET developer on Windows.

Our project's main logic is done inside Windows 10 Pro VM machines, so to do actual work we need Windows 10 VM machines, for that, I can get dedicated servers from a provider and run a bunch of Windows VMs to do the work, the problem is that I have no idea how to implement rest of the things, I even don't know what we need. So I'm asking you for help.

We need one more VM or database were we going to put already processed data, also we need a web page where users will request work from.

So my question is what do we need to accomplish this?

I just found out that API gateway can be used to connect services, etc.

I would appreciate some architecture guide as well, maybe someone already did similarly and shared it on Internet?

​

Imagine we doing "photo processing" and the main work is done in Windows 10 VM machines, we need users to upload photos and later get results, what do we need to set up such an environment?

All kinds of feedbacks are greatly appreciated.

https://redd.it/p6y6e4
@r_devops

AWS service catalog terraform

hey guys I am new to terraform. Is it possible to launch AWS service catalog product using terraform?

I came across this old thread https://stackoverflow.com/questions/56185045/how-to-launch-a-product-from-terraform but not sure about it.

https://redd.it/p6y5qy
@r_devops

Is there a way to export a ConfigMap file outside of deployment?

I have an application config that needs to be dynamically generated based upon Values. At the same time I would like to be able to still deploy the application based upon the same configuration that would be dynamically generated. Is there a way other than --dry-run --debug to print the contents of the configuration that would be built during install? Ideally I would like to just get the contents of the config file and not the entire contents of the charts. I am looking to save the contents to a local file for convenient local deployment for devs.

https://redd.it/p6sdom
@r_devops

Is there any point of learning .net core?

Hey, on my current project I am in a devops role but I have opportunity to help developing backend written in .net core. I am aware that .net core is a huge thing and people master this skill many years so here is my question. Is it worth to go for it? Will it help extend my devops knowledge? My main goal is to become devops expert . Will it help me develop better azure functions or will i find configuring msg brokers knowledge useful? (btw. I have some C# background acquired during my studies)

https://redd.it/p71o3e
@r_devops

Self-hosted directory of self-hosted apps

I started self-hosting web apps last year. As I have been discovering more apps, I made https://www.selfhost.is/ to collect and organize them, and gather useful deployment tutorials, for example Umami is one of the apps I am self-hosting. Hopefully it's helpful to people. Let me know what you think. You are welcome to share an app or a tutorial!

https://redd.it/p716ii
@r_devops

Terraform/Gruntwork or CloudFormation?

The client I am working for is deciding between sticking with CloudFormation or switching to Terraform. I have been looking in to Gruntwork and was wondering if anyone had any experience, talking points.

https://redd.it/p73won
@r_devops

Azure Service Bus

What is Azure Service Bus?

Message buses and queues are a powerful tool communicating between applications. They are used extensively in microservices to decouple applications and services from each other providing benefits like load-balancing, safe routing and control across service and application. Microsoft Azure Service Bus is a reliable cloud messaging as a service (MaaS) with message queues and publish-subscribe topics.

Concepts and Terminologies

Message: Transfer of data between different applications and services is called a message. A message contains business data such as sales and purchase order or any inventory status in different forms like JSON, XML or plain text.

Decouple applications: Service Bus improves reliability and scalability of applications and services. No consumer or producer are required to be online at the same time.

Load Balancing: Allow multiple consumers to read from a queue at an equivalent time, each safely obtaining exclusive ownership to specific messages.

Queues: Sender sends messages to queue and receiver receives the message from queue Messages stay in the queue until the receiver is available to receive and process them.

Topics: On the other hand, topics are same as queues the only difference is that same queue can be used or accessed by multiple receivers. Topics are useful in publish/subscribe scenarios.

When to use Azure Service Bus?

Azure Service bus is the “Swiss Army Knife” service of all other generic messaging services. It is a solid workflow and transaction handling and has robust facilities for handling many applications fault conditions.

Service Bus functions as a bridge between hybrid cloud solutions and systems that include branch office or work site systems. Systems that sit “behind the firewall”, are roaming across networks, or are occasionally offline can’t be reached directly via “push” messaging, but require messages to be sent to a verified pickup location from where the designated receiver can obtain them.

When the business application lies in the cloud and on-site datacenter, then Service Bus queues and topic subscriptions are the ideal choice. This model is especially fashionable SaaS providers in health care, tax and legal consulting, restaurant services, and retail.

Alternative solutions to Azure Service Bus

Azure Messaging provides a fleet of services that permits developers to select a fully-managed service that most closely fits their requirements The services follow common principles and provide flexibility to the developers. The other alternatives to Service Bus are Event Hubs and Event Grid which are complemented by Azure services like Logic Apps, IoT Hubs and Notification Hubs

Read More : https://www.taliun.com/azure-service-bus

https://redd.it/p6nm14
@r_devops

write technical blog in devops will help with future getting hired?

I started to write technical blog, in scribble style, about what I have done in my job and my free time, do you think it will help with getting hired?

Because I tend to forget a lot of stuff during interviews, my ideas, my thoughts, what I used to do and what to improve in details

link: https://yhc-bluemoon.blogspot.com/

but it really looks like scribble style, it's not meant to teach people how to do certain stuff nor tutorials

by the way, I want to move to Vienna, Austria. I am from Southeast Asia country, a self-taught person, with a degree in non-related field, it looks like it could be difficult to me to apply for visa in that sense, since RWR uses point system and EU blue card requires higher salary, but I gonna try. If anyone can help, please let me know.

https://redd.it/p756n4
@r_devops

Amber, Encrypted secrets management

Amber is an encrypted secret management solution tailored towards CI system. It allows the encrypted secrets to be stored as part of the git repository itself so that changes can be tracked over time. This is the announcement post with more details: https://www.fpcomplete.com/blog/announcing-amber-ci-secret-tool/

This is the github repository with the code: https://github.com/fpco/amber

https://redd.it/p77lvz
@r_devops

Grafana-Azure AD Integration

Hi there,

We have integrated Grafana with Azure AD to enable SSO login. We have created two groups Grafana-admin and Grafana-users on the Azure AD with different members. I need to assign the level of permission’s (admin, editor, viewer) so that the group “Grafana-users” only have the viewer access and “Grafana-admin” has the admin level access? I need help to understand where should I need to define this configuration?

Thanks, and I really appreciate your help on this.

https://redd.it/p798hc
@r_devops