BI / Analytics / Reporting in a modern microservice stack

Hi All

I'm a PM at an AgriTech BPM startup.

We're in the process of planning out the next version of our tech stack. We'll be breaking down our current +-5 year old PHP/jQuery/OrientDB monolith into (most likely) Nodejs/Vue.js/Couchbase etc microservices.

We currently use 2 separate components for reporting: Hand built frontend tables, graphs and the like for the basic "inline" stuff like KPI's and basic aggregated tables, charts etc, and then M$ Power BI for the more advanced dashboard-like stuff which we then embed. The latter is fed by ETL jobs that push data on a schedule into an Azure SQL DB. The licensing thereof is also damn near crippling TBH.

So with all that in mind, I'd very much like to hear your opinions on how/what/where to incorporate a reporting and BI solution. I'm basically trying to answer the following question:

Is it realistic to want to build one microservice that does everything from quick basic KPIs all the way up to complex dashboards and embedded reports?

* We're thinking along the lines of a Postgres instance with Cube.js and some form of Vue.js & D3.js frontend that can be embedded, or even something like Plotly's Dash (Python substack). Ideally we'd like to be able to eventually hire a data analyst or 3 that can run with this with minimal input from Devs.
* Is there merit in splitting the two ends of the spectrum into basically what we have now, namely one component that sits inline on the frontend, that pulls from the production DB, but only ever remains a mini framework for the basics like the aforementioned KPIs and basic stuff, and then treating the proper embedded reports and dashboards as a separate thing. Also bear in mind the possible AI/ML components that may come along in future.

P.S. I'm trying to stay away from the Docker & Kubernetes conversation on this specific point, but if it's relevant please feel free.

I've been accused of rambling on Reddit before, so please forgive if that's the case here.

Looking forward to your opinions.

https://redd.it/g77y88
@r_devops

I'm doing a survey in the devops community: What challenges do you face implementing continuous security into your workflows?

I'm hoping to do a talk on the several challenges that make devsecops difficult, what challenges do your security people struggle with compared to the old days of "pentest at the end". With continuous delivery, pipelines get more complicated and attack vectors are no longer just limited to the applications, so as a community we have to adapt to this, for example we struggle with the cloud infrastructure side a lot - most of the staff we have working on the terraform builds and cloud infrastructure pieces aren't actually infrastructure people, so it leads to exposed services and lazy configurations that are "function-first". Any comments or opinions like this would be much appreciated!

Thank you.

https://redd.it/g76tf5
@r_devops

What are good log analysis tools (not using Java)

Im looking to setup some remote log analytics but elasticsearch, elkstack, graylog etc etc are all Java based and not suitable to run on a small vm with 2GB ram (Java issues with not enough memory!), surely there is some solutions out there that are lightweight ? maybe running golang or something

https://redd.it/g76exg
@r_devops

Any tips/tutorials/cheatsheet to build skills in core networking concepts (vpn, subnetting,proxy, NAT, ssh tunnels, port forwarding...)

Working as a junior DevOps engineer it is frustrating I have a weak knowledge on networking... Need some guidance. How did you guys learnt networking ?

https://redd.it/g87f2c
@r_devops

Embarrassing Question

So I’ve been a DevOps Engineer now for about a year and a half, Windows sysadmin and application analyst for a few years prior to this and I’m embarrassed to admit I type horrifically.

It was never a huge deal before but I’ve found that as I’ve moved more into scripting, IaC and even high level app coding my current typing method is just super slow, inaccurate and cumbersome.

I’m not pecking each key bad, but I almost always need to be looking at the keyboard and I often am slow and still make errors.

I was wondering if anyone had any good tips or resources for learning to type quickly and without needing to look at the keyboard? I’ve of course just googled for typing tutorials and the like but I wanted to see if anyone had things they could say had helped them personally.

Thanks for the time all! Hope everyone is staying healthy and sane!

https://redd.it/g84mlk
@r_devops

[Article]Collection of workshop, books, trainings and articles about DevOps

I create a article with a good collection of workshops, books, ebooks, trainings, courses and articles about DevOps and DevSecOps, I hope it can help:
https://medium.com/me/stats/post/cfd86b4a7f43 https://medium.com/me/stats/post/cfd86b4a7f43

https://redd.it/g83wbd
@r_devops

Exposing internal services to developers

Hello!

I'm going to create three Kubernetes clusters for a small team of developers. I'm going to need a set of internal tools (e.g. Grafana dashboards). Would you propose to use SSO or VPN to expose internal services?

https://redd.it/g81for
@r_devops

Understand how Prometheus Monitoring works | Explaining Prometheus Architecture

Prometheus has become the mainstream monitoring tool of choice in container and microservice world.

[**In this video**](https://youtu.be/h4Sl21AKiDg) I explain following topics:

* **Why Prometheus is so important in such infrastructure** and what are some specific use cases
* **Where and why is Prometheus used with specific use cases**?
* How Prometheus works? What are targets and metrics?
* How does Prometheus collect those metrics from its targets?
* I explain **Prometheus Architecture with simple diagrams** and go through the main components: Prometheus Server, Pushgateway, Alertmanager
* Configuring Prometheus - Example YAML Configuration
* The **advantages** of Prometheus Pull System compared to alternative monitoring tools, which use Push System
* Using Prometheus Monitoring with **Docker 🐳 and Kubernetes**

A separate practical video for monitoring Kubernetes services with Prometheus will follow.

​

Thought I share it, could be helpful for some of you 🙂 Also appreciate any feedback.

https://redd.it/g7v7ny
@r_devops

Need recommendation for a CD platform

Hi,

I'm looking to host a $20 droplet on DigitalOcean, and I need a recommendation for the following setup.

1. Has an option to build container images (Docker) and save them to a local registry.
2. Deploys the said container images as a single-instance system on the same machine.
3. Routes HTTP / HTTPs traffic to the per-container domain (e.g. [myapp.mydomain.com](https://myapp.mydomain.com)) and requests LE certificates automatically.
4. Has a web GUI (optional).
5. Directly syncs with GitHub.
6. Is easy to install.

I tried Dokku and such and I didn't like the Duct-tapyness of it - I'm looking for something more enterprise-grade that has an on-premise option available.

https://redd.it/g80j53
@r_devops

Trying to understand Helm and multiple applications

Hi Everyone,

I'm working on architecting a structure for a complex application and need to understand if I'm approaching this incorrectly. I am new to K8s and Helm, so you've been warned. ;)

I have a collection of applications (Client apps) that consumes services from a single application (Server app). Since I would rather not spin up duplicate Server app for each Client app, I would like to get the charts to detect and use the existing services that are there. From my understanding, I might be able to accomplish this with requirements.yaml? But I can't find any documentation to confirm this.

To add to this, I plan on having multiple environments in the development cluster. I know I could use namespaces, but I'm hoping I can avoid it if it's not necessary.

PS, If Helm is the wrong approach, I am not married to it too.

Thanks!

https://redd.it/g7v40h
@r_devops

DevOps vs. SRE — Which is better for your career?

What do you guys think? Which is a better job title?
[https://medium.com/devops-dudes/devops-vs-sre-which-is-better-for-your-career-5694b5719d88?source=friends\_link&sk=7fde8bc1092eb01bf57cd79ba666f0d9](https://medium.com/devops-dudes/devops-vs-sre-which-is-better-for-your-career-5694b5719d88?source=friends_link&sk=7fde8bc1092eb01bf57cd79ba666f0d9)

https://redd.it/g7zgde
@r_devops

kubeletctl is an open-source client for kubelet with an option to scan for vulnerable containers

**kubeletctl is an open-source client for kubelet with an option to scan for vulnerable containers**

What can it do:

* Run any kubelet API call
* Scan for nodes with opened kubelet API
* Scan for containers with RCE
* Run a command on all the available containers by kubelet at the same time
* Get service account tokens from all available containers by kubelet
* Nice printing 📷

Check it out:[https://github.com/cyberark/kubeletctl](https://slack-redir.net/link?url=https%3A%2F%2Fgithub.com%2Fcyberark%2Fkubeletctl)

\#kubernetes #kubelet #kubeletctl

https://redd.it/g7ssbm
@r_devops

Openshift pipeline help

Need some good reading or video resources to understand how to design and implement entire pipeline on openshift or kubernetes. I understand kubernetes and openshift from infrastructure standpoint but I need to learn how to take traditional on prem application and convert to devops pipeline on openshift or kubernetes with entire build test ( all level of testing) deploy test scenarios. I have always been a infrastructure guy and never worked as software developer Thank you in advance

https://redd.it/g7sglh
@r_devops

Seeking for guidance.

Hello everyone,

I have some thoughts about which programming language i should learn (Pyhton || Ruby), and i want to share with you, and get some advice about it.

I have been working with ansible to provision the infrastructure for long time already, and using molecule with testinfra(pyhton) to test playbook, but i got a issue that put me in the situation of migrate to chef, and all it tools(InSpec, RSpec, serverspec, KitchenCI, etc), thing that i don't regret at all, and i kind of love it actually, because gave a test-driven provision, but all that is base on Ruby.
Now every time that i got an interview to DevOps position, some of the requirements always are python and bash for scripting, thing that is ok, but what if i can do the same scripting on Ruby??

Thanks in advance.

https://redd.it/g7s2ps
@r_devops

Migration from Docker Swarm to Kubernetes with same IP?

HI All,

I am working on migrating docker swarm based microservices to kubernetes using helm 3 charts. The migration job/script does the job of importing current config files ,volumes etc. Now all the services come up and we are able to validate the services. I am going to use metallb load balancer for services.

Any suggestions on how to switch the IP from docker swarm VM to kubernetes metallb online ?

Note that both the systems have single entry point for the microservices(api gateway).

https://redd.it/g7pnq4
@r_devops

Lacking some devops basics

So I’ve been working as a devops engineer for the past year. I had no prior knowledge about what devops was coming out of college at all. But I made due just getting hands on experience on aws, kubernetes, etc. but I feel like I’m missing some basic knowledge that I should know. I know how to fix certain issues, debug my way out of random requirements, utilize aws services all by googling stuff on the job. But i don’t know how to start learning and understanding why certain things work they way they do. Topics including SNI, why some tcp traffic needs Level 3 - 4 layer to work. Mostly network, certs, proxies related things. Not that I can’t google all this myself, but can anyone point me to the right direction of how to start or suggest any books that helped them really understand these abstract topics.

https://redd.it/g7pr1n
@r_devops

How come Amazon deploys 23,000 times a day? What are they changing so often?

ok so im new to devops. i came across this image [https://imgur.com/a/3uBZKBN](https://imgur.com/a/3uBZKBN) and i was wondering what exactly does amazon (and other companies) change in these so many deployments? Because i see pretty much the same website everyday

https://redd.it/g8ktuu
@r_devops

Praise dependabot! The github bot to manage your code's vulnerabilities

I just got on with a new project to perform an automation engineer role to help streamline the little resources this team has. First order of business was moving out of their private GitLab box that wasn't enforcing HTTPS to a GitHub Org, so we can be a little more confident in the confidentiality of our source code..

I enabled the dependency alerts under the new private Repo, and now there's this trusty bot named Dependabot scanning and submitting PRs to update the dependencies to clear all sorts of CVEs that have been posted on the tools in use. I've never seen this feature before, so I figured I'd inform the masses of this neat feature

https://redd.it/g8ncd9
@r_devops

Single node orchestration

Hey, everybody.

I'd like some advice for a containers orchestration for my new project. I have about ten services (web, app, database, static documentation, go server etc.). For these services I don't need a big infrastructure with several nodes, I don't need high availability either. A single node will be enough for me. It will be my server at home but I still want to take advantage of an orchestration to automate my devops and my CI.

Do you think single node kubernetes (with k3s) is overkill for that ? What else would you advise me to do?

Thank you all

https://redd.it/g8olrs
@r_devops

For what tasks do you configurations tools (Ansible, ...) if you have containers? Isn't this a anti-pattern?

Now that I have everything in containers I almost don't need a configuration managament tool like Ansible. I only use it to set Linux parameters with sysctl and install Docker (and the container orchestration).

I see some articles and tools for building container images with Ansible etc. Why? This seems much more complex than just Dockerfiles/scripts. I don't need to describe a 'desired state' with Ansible because I build the container from the ground up with scrips that easier to understand that Playbooks. One does not have to learn all the Ansible modules. I would even proclaim that building images with Ansible is an anti-pattern.

https://redd.it/g8jnas
@r_devops

How much Linux knowledge is required to be a Cloud Engineer?

How much Linux knowledge is required to be a Cloud Engineer?

I know the basics of the Linux CLI when it comes to commands. grep, ls, mv, etc, aren’t an issue. I’m just confused as to how much Linux knowledge one would need when pursuing a cloud role. I always see on job postings related to the cloud: “strong in Linux”. What exactly is a gauge of this?

Would I be wasting my time getting the RHCSA after my Network+? Should I just head for my first AWS cert after Network+?

I understand that certs aren’t the end all be all, and don’t guarantee a job etc.

https://redd.it/g8ki25
@r_devops