Understand how Prometheus Monitoring works | Explaining Prometheus Architecture

Prometheus has become the mainstream monitoring tool of choice in container and microservice world.

[**In this video**](https://youtu.be/h4Sl21AKiDg) I explain following topics:

* **Why Prometheus is so important in such infrastructure** and what are some specific use cases
* **Where and why is Prometheus used with specific use cases**?
* How Prometheus works? What are targets and metrics?
* How does Prometheus collect those metrics from its targets?
* I explain **Prometheus Architecture with simple diagrams** and go through the main components: Prometheus Server, Pushgateway, Alertmanager
* Configuring Prometheus - Example YAML Configuration
* The **advantages** of Prometheus Pull System compared to alternative monitoring tools, which use Push System
* Using Prometheus Monitoring with **Docker 🐳 and Kubernetes**

A separate practical video for monitoring Kubernetes services with Prometheus will follow.

​

Thought I share it, could be helpful for some of you 🙂 Also appreciate any feedback.

https://redd.it/g7v7ny
@r_devops

Need recommendation for a CD platform

Hi,

I'm looking to host a $20 droplet on DigitalOcean, and I need a recommendation for the following setup.

1. Has an option to build container images (Docker) and save them to a local registry.
2. Deploys the said container images as a single-instance system on the same machine.
3. Routes HTTP / HTTPs traffic to the per-container domain (e.g. [myapp.mydomain.com](https://myapp.mydomain.com)) and requests LE certificates automatically.
4. Has a web GUI (optional).
5. Directly syncs with GitHub.
6. Is easy to install.

I tried Dokku and such and I didn't like the Duct-tapyness of it - I'm looking for something more enterprise-grade that has an on-premise option available.

https://redd.it/g80j53
@r_devops

Trying to understand Helm and multiple applications

Hi Everyone,

I'm working on architecting a structure for a complex application and need to understand if I'm approaching this incorrectly. I am new to K8s and Helm, so you've been warned. ;)

I have a collection of applications (Client apps) that consumes services from a single application (Server app). Since I would rather not spin up duplicate Server app for each Client app, I would like to get the charts to detect and use the existing services that are there. From my understanding, I might be able to accomplish this with requirements.yaml? But I can't find any documentation to confirm this.

To add to this, I plan on having multiple environments in the development cluster. I know I could use namespaces, but I'm hoping I can avoid it if it's not necessary.

PS, If Helm is the wrong approach, I am not married to it too.

Thanks!

https://redd.it/g7v40h
@r_devops

DevOps vs. SRE — Which is better for your career?

What do you guys think? Which is a better job title?
[https://medium.com/devops-dudes/devops-vs-sre-which-is-better-for-your-career-5694b5719d88?source=friends\_link&sk=7fde8bc1092eb01bf57cd79ba666f0d9](https://medium.com/devops-dudes/devops-vs-sre-which-is-better-for-your-career-5694b5719d88?source=friends_link&sk=7fde8bc1092eb01bf57cd79ba666f0d9)

https://redd.it/g7zgde
@r_devops

kubeletctl is an open-source client for kubelet with an option to scan for vulnerable containers

**kubeletctl is an open-source client for kubelet with an option to scan for vulnerable containers**

What can it do:

* Run any kubelet API call
* Scan for nodes with opened kubelet API
* Scan for containers with RCE
* Run a command on all the available containers by kubelet at the same time
* Get service account tokens from all available containers by kubelet
* Nice printing 📷

Check it out:[https://github.com/cyberark/kubeletctl](https://slack-redir.net/link?url=https%3A%2F%2Fgithub.com%2Fcyberark%2Fkubeletctl)

\#kubernetes #kubelet #kubeletctl

https://redd.it/g7ssbm
@r_devops

Openshift pipeline help

Need some good reading or video resources to understand how to design and implement entire pipeline on openshift or kubernetes. I understand kubernetes and openshift from infrastructure standpoint but I need to learn how to take traditional on prem application and convert to devops pipeline on openshift or kubernetes with entire build test ( all level of testing) deploy test scenarios. I have always been a infrastructure guy and never worked as software developer Thank you in advance

https://redd.it/g7sglh
@r_devops

Seeking for guidance.

Hello everyone,

I have some thoughts about which programming language i should learn (Pyhton || Ruby), and i want to share with you, and get some advice about it.

I have been working with ansible to provision the infrastructure for long time already, and using molecule with testinfra(pyhton) to test playbook, but i got a issue that put me in the situation of migrate to chef, and all it tools(InSpec, RSpec, serverspec, KitchenCI, etc), thing that i don't regret at all, and i kind of love it actually, because gave a test-driven provision, but all that is base on Ruby.
Now every time that i got an interview to DevOps position, some of the requirements always are python and bash for scripting, thing that is ok, but what if i can do the same scripting on Ruby??

Thanks in advance.

https://redd.it/g7s2ps
@r_devops

Migration from Docker Swarm to Kubernetes with same IP?

HI All,

I am working on migrating docker swarm based microservices to kubernetes using helm 3 charts. The migration job/script does the job of importing current config files ,volumes etc. Now all the services come up and we are able to validate the services. I am going to use metallb load balancer for services.

Any suggestions on how to switch the IP from docker swarm VM to kubernetes metallb online ?

Note that both the systems have single entry point for the microservices(api gateway).

https://redd.it/g7pnq4
@r_devops

Lacking some devops basics

So I’ve been working as a devops engineer for the past year. I had no prior knowledge about what devops was coming out of college at all. But I made due just getting hands on experience on aws, kubernetes, etc. but I feel like I’m missing some basic knowledge that I should know. I know how to fix certain issues, debug my way out of random requirements, utilize aws services all by googling stuff on the job. But i don’t know how to start learning and understanding why certain things work they way they do. Topics including SNI, why some tcp traffic needs Level 3 - 4 layer to work. Mostly network, certs, proxies related things. Not that I can’t google all this myself, but can anyone point me to the right direction of how to start or suggest any books that helped them really understand these abstract topics.

https://redd.it/g7pr1n
@r_devops

How come Amazon deploys 23,000 times a day? What are they changing so often?

ok so im new to devops. i came across this image [https://imgur.com/a/3uBZKBN](https://imgur.com/a/3uBZKBN) and i was wondering what exactly does amazon (and other companies) change in these so many deployments? Because i see pretty much the same website everyday

https://redd.it/g8ktuu
@r_devops

Praise dependabot! The github bot to manage your code's vulnerabilities

I just got on with a new project to perform an automation engineer role to help streamline the little resources this team has. First order of business was moving out of their private GitLab box that wasn't enforcing HTTPS to a GitHub Org, so we can be a little more confident in the confidentiality of our source code..

I enabled the dependency alerts under the new private Repo, and now there's this trusty bot named Dependabot scanning and submitting PRs to update the dependencies to clear all sorts of CVEs that have been posted on the tools in use. I've never seen this feature before, so I figured I'd inform the masses of this neat feature

https://redd.it/g8ncd9
@r_devops

Single node orchestration

Hey, everybody.

I'd like some advice for a containers orchestration for my new project. I have about ten services (web, app, database, static documentation, go server etc.). For these services I don't need a big infrastructure with several nodes, I don't need high availability either. A single node will be enough for me. It will be my server at home but I still want to take advantage of an orchestration to automate my devops and my CI.

Do you think single node kubernetes (with k3s) is overkill for that ? What else would you advise me to do?

Thank you all

https://redd.it/g8olrs
@r_devops

For what tasks do you configurations tools (Ansible, ...) if you have containers? Isn't this a anti-pattern?

Now that I have everything in containers I almost don't need a configuration managament tool like Ansible. I only use it to set Linux parameters with sysctl and install Docker (and the container orchestration).

I see some articles and tools for building container images with Ansible etc. Why? This seems much more complex than just Dockerfiles/scripts. I don't need to describe a 'desired state' with Ansible because I build the container from the ground up with scrips that easier to understand that Playbooks. One does not have to learn all the Ansible modules. I would even proclaim that building images with Ansible is an anti-pattern.

https://redd.it/g8jnas
@r_devops

How much Linux knowledge is required to be a Cloud Engineer?

How much Linux knowledge is required to be a Cloud Engineer?

I know the basics of the Linux CLI when it comes to commands. grep, ls, mv, etc, aren’t an issue. I’m just confused as to how much Linux knowledge one would need when pursuing a cloud role. I always see on job postings related to the cloud: “strong in Linux”. What exactly is a gauge of this?

Would I be wasting my time getting the RHCSA after my Network+? Should I just head for my first AWS cert after Network+?

I understand that certs aren’t the end all be all, and don’t guarantee a job etc.

https://redd.it/g8ki25
@r_devops

Critique/help with the MLOps plan for a small DS team

I work for a small (~4 person) data science team within a much larger organization. The team is responsible for making two machine learning models, creating a single set of very important predicted values, and creating reporting and data validation tools relevant to those predicted values. I came on board about 4 months ago with experience in data science, systems administration, and devops. I have a strong linux background and plenty of experience with Docker and Kubernetes.

I've been asked to improve the existing modeling pipeline. I've come up with a plan that I think is feasible given the organization's goals and (considerable) constraints, but I'm hoping to get feedback on potential pitfalls or things to add from people with more ML/dev ops experience than myself. I also thought it might be fun for this sub to think through what the ideal toolchain might be given a pretty serious set of constraints.

## Goals

- Make our pipeline more robust. No more undetected data issues or breaking commits. Automatic unit and integration tests on all commits/merge requests.
- Improve pipeline transparency and reporting. Make summary and performance statistics about each model more easily available.
- Make testing and comparing new models significantly easier. More clearly tie new model results/objects to the code that produced them.
- Make the whole pipeline run continuously and automatically (given new data or other triggers).

## Constraints

- No cloud infrastructure. Everything has to be on-prem.
- Absolutely no additional money. Zero.
- Need to keep the developer toolchain as light as possible. It has to be usable by a team with limited devops/linux experience.
- Infrastructure can be (and is) linux + Docker based, but it has to be simple enough that if I die it's easy to understand and maintain for someone with a moderate devops background. For the same reason, all infrastructure setup has to be infrastructure as code.
- Any rebuild has to be done within 6 months of one person's full time work. This includes all infrastructure setup, code refactoring, CI/CD setup, and new code.
- The pipeline/modeling itself has to be written in R.

## Tools Available

- Hardware is limited to 2 beefy SQL servers, 2 beefy Ubuntu VMs, and ~6 beefy Windows workstations.
- We recently upgraded to GitLab Silver for the whole organization and have all the features that go along with it.

## Current Setup

This is a relatively new team that had to get something up and running quickly, so they haven't yet had the time or resources to setup a mature ML pipeline or incorporate many devops best practices. However, they're committed to improving things and making the best system possible, hence why they asked for this plan. The current pipeline is:

1. **Data extraction/processing.** Data is stored entirely in SQL and feature engineering/data extraction is done via SQL views. The view definitions are stored in GitLab. There is one SQL server that is used for both reporting and modeling. Data extraction takes a *very* long time.

2. **Modeling.** The entire pipeline is written in R and is stored in a single, large GitLab repo. Scripts are manually triggered sequentially to run the actual pipeline and modeling. Data ingest/validation, modeling, model validation, and reporting are all roughly part of the same repo. This repo has no unit testing or integration testing.

3. **Reporting.** Reporting is done via R Markdown and a set of Shiny apps that exist separately from the main modeling repo. These reporting applications pull from the same SQL server as the main modeling scripts and report on the predicted values created in the modeling step. Model performance metrics are not available to the reporting apps.

Other notes:

- Intermediate data and model objects are not saved. The model specification and performance statistics of the best performing model are saved to an excel sheet. The predicted values produced by this model are saved back to SQL.

- Testing new models and/or functional forms is

done manually by editing the main repo's R code. Model outputs are not tied to specific commits or branches.

## Planned Improvements

Given my constraints, I'd like to make the following improvements:

- Disaggregate the steps of the pipeline into discrete repositories/tasks that can be individually run, tested, and worked on. Add unit testing to each of these repos that runs automatically (via GitLab CI/CD).
- Create an R package or packages that contains widely used functions and small datasets. Also add unit testing to these repos.
- Create a separate SQL server that mirrors the original server and is used exclusively for reporting.
- Use [DVC](https://dvc.org/) and [MinIO](https://min.io/) (running in Docker on a VM) to store the intermediate data produced by each step in the pipeline as well the final model objects. This is to prevent people from needing to constantly re-run the same data ingest scripts.
- Use DVC to define clear DAGs that automate the process of running the pipeline and collecting metrics on the results. Upload model metrics to a new table in SQL.
- Again use DVC to tie model output and data to specific commits and branches.
- Using the model summary metrics in both DVC and SQL, add some sort of reporting dashboard (Tableau, Shiny) that facilitates easy comparison of different models.

Those are my immediate thoughts for improvements, but I'm curious to get this sub's take as well. Additionally, I'd love to find an ML ops mentor if someone out there is willing to teach/talk.

**TL;DR:** You have 2 SQL servers, 2 VMs, a GitLab subscription, 0 money, and 1 person with linux experience. What's the most robust/transparent machine learning pipeline you can make?

https://redd.it/g8odm4
@r_devops

Organizing Developer Teams, Code and Resources as your Organization Grows (Part 2: Organizing your Codebase)

Hey guys,

Having worked with teams of all sizes and technical backgrounds, I have created a series on how to properly organize teams, code and cloud resources that fits well with the "DevOps" methodology of allowing teams to autonomously develop, test and deploy software.

All constructive feedback is welcome!


[https://diligentprogrammer.com/2020/04/25/organizing-developer-teams-code-and-resources-as-your-organization-grows-part-2-organizing-your-codebase/](https://diligentprogrammer.com/2020/04/25/organizing-developer-teams-code-and-resources-as-your-organization-grows-part-2-organizing-your-codebase/)

https://redd.it/g8g7ju
@r_devops

[Article] DevOps-as-a-Product

I'm always trying to come up with different approaches to help influence change within an organization. Check out this article and let me know what you think about trying to sell DevOps like you would sell any other product to business.
https://medium.com/devops-dudes/devops-as-a-product-64251c439340?source=friends\_link&sk=bc8d992e52e2e42cd5190fc789b10ed0

https://redd.it/g8jocs
@r_devops

Windows Update API - Automation of MSU Package Downloads

Hi all.

I've been researching this for a few days and I haven't had much luck so far. I'm looking for a way to programmatically download the MSU packages for Windows Server 2019/2016. Mainly cumulative updates. I have a specific use case where I need the actual MSU files and automation will be very important there.

I don't think there's some sort of API for [catalog.update.microsoft.com](https://catalog.update.microsoft.com) \- there apparently used to be an RSS feed but it looks like that's been removed.

Any thoughts or advice would be much appreciated!

https://redd.it/g8iik5
@r_devops

YAML File generator for kubernetes

When I am trying to create my own YAML configuration file in Kubernetes I faced a lot of challenges. One of the main challenge is " what are the option I have for a particular property or option for a YAML field? "

So I searched in Google and found some interesting Github project which is [kubergui](https://github.com/BrandonPotter/kubergui) but that project/tool is not helpful when we want to generate advanced YAML configuration for our Kubernetes cluster.

So I thought to solve this problem with my web development skills 🤪 and want to help others to easily generate YAML configuration file for their Kubernetes cluster.

So after working hard, I came up with the one tool that helps you to generate the YAML file and also helps you to know what are options available for a particular YAML property/field.

The tool is Kube-yaml-gen that helps you to generate the YAML file by selecting the available options.

You don't need to pay anything 😂 for this tool/app because it open source and hosted in Github pages 🤪.

Check out that project [here](https://github.com/MohanSai1997/kube-yaml-gen). If you found useful then give a star and share because this is my first open-source project.

Any suggestions or features to improve the project are accepted 😉😊.

https://redd.it/g8i8pk
@r_devops

I just created my first app (a Prometheus exporter for Jira Cloud) in Python! I'd love it if any of you want to give me your feedback.

As stated, this is the first app I've ever created. However, it's fairly simple, and I think it's fairly well-coded as well. I'd really appreciate any feedback you guys can give me.

[https://github.com/R0quef0rt/prometheus-jira-cloud-exporter](https://github.com/R0quef0rt/prometheus-jira-cloud-exporter)

It's amazing how quickly I'm picking up programming. I started as a career sysadmin, then moved into devops. If you can learn devops, you can learn programming. I've only been studying for about a month.

https://redd.it/g8faxx
@r_devops