ELK deployment advice

Hello everyone,

At my work, we are thinking about deploying ELK stack on our VMs to analyze logs.
The entire stack will be at v.7.17 and all deploy on VMs (maybe EC2s, basically Linux machines).
The deployment restriction is ELK to be deployed as 1 Logstash/Elasticsearch/Kibana (1 per each/no clustering).
Downtime can be tolerated within reasons (blackout/disaster).

The current daily log size will be at most 500k lines (about 1-1.5GB total)
We'll have to keep them for at least 90 days so 1.5x90 = 135 GB total
Search speed can be less optimized but within reason for users (let's say searching for a specific event last month is within 5 secs)

The current idea is:
Logs will be read using Filebeat from each machine and sent to Logstash.
Logstash will then use some filters to process logs and send them to Elasticsearch and Kibana.

Filebeats/logs via HTTP requests from apps --> Logstash --> Elasticsearch --> Kibana

My questions are:
Is there any example/best practice on this?
Are there any pitfalls we should know to avoid?
Is an enterprise license required?
Are there places where we can learn more about ELK?

We are quite new to this so any recommendations are welcomed.
I've already recommended Datadog/Loki and other solutions but the end solution is ELK so we'll have to go with it.

https://redd.it/yboysz
@r_devops

sending logs to central storage from all ELK instances

hello everyone. what are you using/suggest for log aggregation from multiple ELK (and loki) instances.

We want this central storage to be able to connect to kibana/grafana as well.

I know there's victoria/thanos and other stuff for prometheus, but I am looking for something similar for logs thru ELK or promtail/loki.

https://redd.it/ybrqsb
@r_devops

Who would you target as a marketing campaign for devops services?

Starting my own IT devops services consultancy.
Having trouble identifying who i should target and the bca.
Was thinking
1) New companies that are just getting started
2) Specfic industry companies?
3) Big tech companies
4) Companies with existing jobs advertised for devops services?

Seeing if anyone has any ideas? In order for the company to require devops, they need to be at least well constructed, techy, and know what devops is, or would you target companies that know nothing about devops at all.

Thanks

https://redd.it/ybnp6s
@r_devops

scheduling

What is unitscheduling and stonebranch ?

Are they same thing ? Or different?

Can you please simplify the meaning of these words ?

I tried to find in Google but results are not clearly understandable.

https://redd.it/y9owq6
@r_devops

Need advice on new role

I recently started a new role as tech lead for a startup. Currently, we have overseas developers handling most of our product development.


Currently the workflow is

Create feature branch and work on dev.

Create pr, code review and merge in dev if all is good.

Dev is essentially our staging env. Once in staging qa eng (still outsourced company) verifies tickets and then typically from waht I have seen weekly or every 2 weeks code is merged into master and then master is essentially production.

Sucks and a complaint my boss (CTO) has is there is a huge bottleneck / holdup at the qa verify stuff step and then getting into production.


I have never done true trunk based dev but def would like to go in thst direction.

I was thinking of something like this.

Work on ticket in short lived feature branch using feature flags.

Create pr once done. Have code review AND demo at this step. Demo to teammate who is performing code review and a qa eng. At same time.

Merge into master. Delete feature branch.

Deploy to staging env.

Maybe have qa do some final checks and Create short lived release branch.

Deploy to prod. (Daily?)


Would love help to figure out what may be realistic in this situation

https://redd.it/ybwee1
@r_devops

DevOps is Bullshit

Cory O’Daniel, CEO of Massdriver, gives his thoughts on the broken state of DevOps and the future of platform engineering.

https://blog.massdriver.cloud/devops-is-bullshit

I'm curious to hear everyone's thoughts on this. Everywhere I've been, DevOps seems to be more of a burden than a boon on the engineering teams.

https://redd.it/y94xhx
@r_devops

Terraform modules that create the same resource

I have two Terraform modules that create cloudflare_zone resources on the same zone. The first module works fine, but the second one fails because the zone already exists.

Is there a way to let Terraform "merge" the two modules together?

https://redd.it/yc1qtr
@r_devops

Running cypress tests for svelte app with Github Actions

I'm really new to Github Actions. I have a repo with a 'frontend' and a 'backend' folder. Backend is fastapi, frontend is svelte and inside the frontend/app, there are 'src', 'cypress', 'package.json', 'cypress.config.js', 'rollup.config.js'.

In the root folder of the repo, there is the .github folder, and there the main.yml. So I want to build and run my svelte application (and the fastapi backend) and then do cypress tests on them. Running fastapi backend works so far. For the frontend I tried runnning svelte on a separate step than the cypress tests, but then cypress.visit() couldn't find the localhost, where the app is located.

That's why I put everything on the same step:

- name: Install packages
run: |
cd frontend/app
npm install
npm install cypress

- name: Cypress
uses: cypress-io/[email protected]
with:
build: npm --prefix frontend/app run build
start: npm --prefix frontend/app run start
config-file: frontend/app/cypress.config.js
wait-on: 'https://localhost:3400'

The "Install packages" steps works fine, but in the cypress step I get an error

Error: ENOENT: no such file or directory, open '/Users/runner/work/<repo>/<repo>/package-lock.json'

So it never checks the right path (frontend/app) so I probably ignores the --prefix parameter. I also tried: (build: cd frontend/app && npm run build) but it still checks the root dir.

Does anyone has a clue why this happens and how I can let them find the right package file? Has been very frustrating.

Thanks

https://redd.it/ybxzs4
@r_devops

Suggestion on plan to deep dive into devops part of application in production.

Hello folks I'm a ruby on rails web developer(>3 YOE), and so far has very minimal "ops" knowledge.For smaller client works and personal projects I've been using - CI: github-actions/circle- CD: Heroku to Handle all the deployment of PR builds, staging & production apps.Currently working for a larger client, here it is devops heavy(with a large dedicated devops team),& I'm kinda clueless & awestruck by the work the devops have done here, hence this plan for deep dive.Also, since Heroku going through some major overhaul, rather than looking for a similar alternatives I would like to deep dive into the operations part of web apps.After some research I came up with a personal ROADMAP (thanks to roadmap.sh & other sources from this sub).Along with 2 personal projects CI/CD setup as to put my learning into practice.

{
networking: "basics", "Protocols", "nginx", "Web Server", "Network Tools"
linuxscripting: ["ruby/go", "bash", "terminal commands", "Compiling apps"],
cloud: ["aws","IAM", "ec2", "vpc", "aws-cli", "s3" "rds", "load balancing", "Auto scaling groups", ],
containerization: ["docker", "docker-compose", "Docker Swarm", "Swarm cluster", "ECS", "ECR", "Fargate", "Fargate cluster", "ECS load balancing with ALB"],
cicd: "jenkins", "github actions", "gitlab", "travis/circle/semaphore" "heroku", "Pipeline Setup With Jenkins and AWS ECS",
sysprovisioningconfiguration: "Terraform","Ansible",
monitoringandlogging: "Prometheus", "Grafana", "cloud watch", "ELK" ,
orchestration: "kubernetes", "eks",
intermediateproject: {
// No orchestration required
// Should I use Jenkins/other CI tools like circle/gitlab/github actions
appcomponents:
"rails_app monolith(front end using jsbundling-rails or webpacker",
"postgress_db"
,
ci:
"PRs should run test suite, static code analysis(rubocop + eslint)", "check for 90% test coverage", "only merge if green run",
cd: "PR merge to master must auto deploy the containerized app to production",
"deploy a rails_app monolith(including front end) + postgress_db"
advancedproject: {
appcomponents:
"rails backend",
"postgres db",
"sidekiq (requires redis)",
"react js frontend",
"setup metabase for the DB (optional future feature)",
"setup rabbitMQ for messages (optional future feature)"
,
ci:
"PRs should run test suite",
"create PR builds for testing",
"static code analysis(rubocop + eslint)",
"check for 90% test coverage",
"auto create PR apps to testing", // is setting up pr builds too much???
"only merge if green run",
cd:
"PR merge to master must auto deploy the containerized app to production",
"deploy",
,
setupmonitoring: ["Prometheus", "Grafana", "cloud watch", "ELK"],
automate: ["provisioningconfiguration", "terraform", "ansible"]
}
}

Am I overdoing anything here? or am I missing out on anything crucial?.
I know Jenkins kinda has a bad rap around here, I personally would like to learn it.
I'm planning to achieve this in 4-5 months, is it too much?.
Any suggestions regards to above points would be much appreciated.Thanks.

https://redd.it/yc5pnj
@r_devops

GitOps with Advanced Cluster Management for Kubernetes

Manage multiple clusters and deploy apps across several target clusters with Argo CD and Advanced Cluster Management for Kubernetes: https://piotrminkowski.com/2022/10/24/gitops-with-advanced-cluster-management-for-kubernetes/

https://redd.it/yc8bp8
@r_devops

New OSS Project: Helm-Dashboard, the missing UI for Helm

Some time ago, we at Komodor, started working on a new OSS project called Helm-Dashboard. Now, it’s finally ready for users, and I’m excited to share with the world. You can see its GitHub repo here: https://github.com/komodorio/helm-dashboard


So Helm-Dashboard is basically a GUI for Helm, and it's designed to solve some of the more acute painpoints of Helm users by visualizing changes in Helm charts. The goal is to help beginner Helm users to get started with Helm, and for more experienced users to speed up operations.


I invite everyone who is using Helm to give it a try and share your feedback. Feel free to open GitHub issues, send PRs, ask questions in a dedicated #helm-dashboard channel on our Slack Kommunity: https://join.slack.com/t/komodorkommunity/shared\_invite/zt-1dm3cnkue-ov1Yh\~\_95teA35QNx5yuMg

Give it a star if you liked it :)

https://redd.it/yccfb5
@r_devops

Helm Manager - App health / Helm Release TTL / Helm Rest API / Helm Web UI

We are working on a tool that helps better manage Helm applications.

[https://github.com/JovianX/Service-Hub](https://github.com/JovianX/Service-Hub)

Helm Manager includes:

* Show application health (All deployed workload resources are healthy).
* Helm releases Time-To-Live(TTL) schedule for temporary apps.
* Manage Helm releases across multiple clusters.
* Helm Restful API with basic RBAC
* Helm UI with basic RBAC

I would much appreciate your feedback and thoughts!

If you like the project, or find it helpful please STAR ⭐ the repo.

https://redd.it/yceela
@r_devops

Devs deploying to Production and separation of duties?

Ok. So I work in security and I have to care about PCI, SOC2, SOX, Alphabet soup, etc…

How are you creating a compliant separation of duties scenario that auditors are going for?!

I know people are doing it. I want to drive developer enablement and push us to streamline the whole path to production.

https://redd.it/ycdafh
@r_devops

DevOps Interview questions

What are some common interview questions for devops in aws and python? I have an interview coming up and need to prepare for it….anything helpful is much appreciated!

https://redd.it/ycjmhv
@r_devops

DevOps Bulletin - Issue #74: AWS prices in your Google Sheets

Hey folks 👋🏻
Check out a sneak peek of the topics covered on this weekly issue:

* 🔐 "**EKS Security Checklist: 10 best practices for a secure cluste**r" - The best way to tighten your cluster security is to implement practices that have become industry standard and are recommended by the Kubernetes community. Here are 10 most EKS security tactics every team needs to protect its clusters.
* 🗂 "**Searchable Pod logs on Kubernetes in minutes**" - Containers generate huge volumes of log data. In a production environment, SREs, DevOps folks need a central location to query and analyse these logs, to do their job well.
* 📦 "**Quirks, caveats, and gotchas In SQLite**" - This document strives to highlight the principal differences between SQLite and other SQL implementations, as an aid to developers that are porting to or from SQLite or who are trying to build a system that works across multiple database engines.
* 🤖 "**An easier way to install tools for GitHub Actions**" - This blog post covers how to use Arkade in your GitHub Actions to get the typical kinds of tools you'll need to build code and run end-to-end tests.
* ⚡️ "**CloudFront vs Cloudflare, and how to reduce response times for both by \~35%+**" - Turn on Origin Shield if you use CloudFront and care about response times, even for dynamic / uncacheable content.
* 💡 "**Solving common problems with Kubernetes**" - This article is aimed at engineers who need to deploy their code using Kubernetes, but have no idea what Kubernetes is or how it works.
* 📹 Podcast of the week goes to "Leaving the Cloud" by Rework Podcast. **Cloud services have been used by tech companies for many years**, but it’s not the only way. Hear why 37signals is making the decision to go in another direction. David Heinemeier Hansson, co-founder of 37signals, and Eron Nicholson, Director of Operations, discuss why 37signals is making the move away from the cloud.
* 🛠 Project of the week goes to "**The AWS Pricing Google Sheets add-on**" - The add-on allows you to incorporate the latest AWS pricing data in your Google Sheets spreadsheets. This makes it easy to perform cloud cost analysis directly in Sheets without error-prone copy-and-paste from pricing website.
* 📕 Review of "**Infrastructure as Code, Patterns and Practices**" - The book teaches you to automate infrastructure by applying changes in a codified manner. You’ll learn how to create, test, and deploy infrastructure components in a way that’s easy to scale and share across an entire organization.

Complete issue: [https://www.devopsbulletin.com/issues/aws-prices-in-your-google-sheets](https://www.devopsbulletin.com/issues/aws-prices-in-your-google-sheets)

Feedback is welcome :)

https://redd.it/ycdy1i
@r_devops

Am I missing something about SAML?

We're currently undergoing a SAML implementation for third party services to simplify MFA enforcement and user management but SAML seems to be missing some things.

1. When a user leaves, if we delete the user from the IDP then they still exist in the third party service. Some services will still charge for that user.

2. Some services don't support role mapping so we have to go into the service UI anyway.

3. Some service providers don't let you either disable non-saml based login or enforce MFA.

Am I missing something here? Seems like it's not solving many of the problems that we'd hoped it would? Sure it makes creating users simple but often assigning the right roles and removing users doesn't seem much improved?

https://redd.it/ycocer
@r_devops

Next gen deployment strategy for a Hackathon?

Our group is running a Hackathon in a few weeks: brainstorm a project idea, and split into a groups and each build that project.

In preparation for that, I'm thinking about setting up a new environment for the project, maybe something like: Fly.io or Rancher or K3s, where we could try some new methods of building and deploying apps.

Fly.io seems to be doing a lot of interesting things. ISTR I've seen a demo or Rancher where it integrated into VSCode and you could get your code change up and running with a custom URL to test it out, something like that might be slick.

Our current environment is: Windows workstations, test locally, commit and push, then build in Jenkins, which creates Deb packages and pushes those to the Linux staging environment. It'd be nice to try something completely different. Our apps will likely be Java backend, Vue frontend. Probably will need something that can connect back to our existing database via a VPN, or be set up on a VM in on-prem.

Any ideas on something worth looking at getting set up to try?

https://redd.it/ycq8zt
@r_devops

Where do you generate your trace header?

I'm curious about where the distributed trace header (W3C Traceparent, etc) is typically injected into the request in people's application architectures.

I can imagine client devices, LBs or gateways being the origin, but would like to hear from folks who have actually implemented tracing.

https://redd.it/ycmyqc
@r_devops

Terraform -> Ansible -> Kubernetes -> Docker Containers

Am I missing anything? All those technologies seem to synergize with each other. Is there overlap, could this be simplified or improved?

1. Terraform to provision infrastructure
2. Ansible for configuration management of that infrastructure
3. Kubernetes for docker container orchestration

https://redd.it/ycup6d
@r_devops

Platform Engineering: DevOps evolution or a fancy rename?

What is Platform Engineering? How is it different from DevOps, SRE or PaaS? Who should use it, and when should we introduce it?

As a veteran of the PaaS, and a DevOps enthusiast at present, I decided to delve into this topic, peel off the hype, and see what it’s about in practice. This also inspired my recent episode of OpenObservability Talks with George Hantzaras, Director, Cloud Platform Engineering at Citrix Systems.

https://horovits.medium.com/platform-engineering-devops-evolution-or-a-fancy-rename-a0cad2cdc819

https://redd.it/yd2fnx
@r_devops

DevOps Projects

Hi, I'm a beginner in DevOps and Cloud areas and I like to do projects to get some hands-on experience.

I will share a small project I did recently. It's a backend application and I wanted to deploy it to the Kubernetes cluster. So here are the things I did.

Infrastructure:-

1. Provisioned an Azure Kubernetes cluster (AKS) and Azure container registry using the console.

2. Then I wrote the Terraform Infrastructure as Code and tried to deploy those resources using IaC.

3. Configured the Terraform backend.

4. Added the Terraform code into a GitHub repository.

5. Configured GitHub Actions workflow and tried to provision resources using the CI/CD pipeline.


Application Deployment:-

1. I wrote the Dockerfile for the application.

2. Built the application Docker image and pushed it to the Azure container registry.

3. Wrote the Kubernetes manifest files for the application deployment.

4. Deployed application to the Kubernetes cluster manually.

5. Then added the Dockerfile and Kubernetes manifests to my backend application code repository.

6. Configured GitHub Actions workflow and tried to deploy the application using the CI/CD pipeline.

By doing this simple project I got some hands-on experience in Terraform, Docker, Kubernetes and GitHub Actions.

Can you guys share some good project ideas like this? It will helpful for self-learning guys like me.

Thanks for your idea and your time. 🫡

https://redd.it/yd2k77
@r_devops