Kubernetes monitoring: Which tools your reviewed and why you've chosen them?

Hi,



We are at the moment reviewing multiple tools for monitoring the microservice app in KubernetesAt the moment, the Datadog looks a bit pricy but more attractive. More integration options, tracing, super popular with the community, seen peers using it.

There are other options, like LogZ, Dynatrace, and others

Logz \- cheap, but looks like this is just managed ELK. So, we would have to cook it, just give more resiliency

Dynatrace \- more expensive, though popular. But, haven't heard any peers actually using it. Even though Gartner in the report puts the as industry-led (really??)

Own ELK - we would need to manage it, spend time and costs for maintenance and cook it. And,still would be quite far from any polished by millions of users products.

Sentry \- they have a simple helm chart for tracing errors. But this is not a login solution

Cloudwatch \- hemm..hopefully, no one suggests it. It is horrible from UI/UX and tracing standpoint. More like a dev tool. It is a hell of a ride with distributed microservices...Someone can say to use X-RAY with it, but then we have to make changes to our apps and instrument a lot.

What have you chosen and why?

View Poll

https://redd.it/nwtmzz
@r_devops

Fluentd delete old processed log files

Hello Everyone,

I have use case where i need to stream apache access and error logs to kafka brokers, to which i plan to use fluentd. I am completely new to fluentd so i am still figuring out how it works. 1 thing am not able to figure out or not been able to find the answers online is how to get rid of the log files which are already processed by fluentd? I mean after pushing the events to kafka i don't want these log files sitting in the machines for no reason.

Any help or suggestion will be really helpful.

\-Thanks

https://redd.it/nwsvyc
@r_devops

Thoughtworks have moved GitOps to “hold” on their tech radar due to the complexity in implementing good branching practices. I must say, I quite like GitOps and have not had trouble with branches. What are your thoughts on GitOps?

https://www.thoughtworks.com/radar/techniques?blipid=202104006

https://redd.it/nwwlfl
@r_devops

Does a entry level job as database administrator (excel, back end program) abled me to transition well to a career in devops?

I always thought that if you want a career in IT you have to start working as helpdesk, IT support, cloud technician and all but does starting office work like a database administrator handling Microsoft excel, data entry and few back end database processing able for me to transition to other IT section in devOps, cloud architect etc?

Or is it not really good job to start yourself to transition into other more technical IT role?

Anyone veteran or knowledgeable can explain about this?

https://redd.it/nwy7an
@r_devops

Is there a deployment management tool for k8s that doesn't try to be a full-service CI platform?

I'm very new to k8s, and trying to write a push-button deployment tool for my team. We've been using Helm to deploy for ages now, and my thought was originally to write a cloud application that just ran Helm commands -- but as I've worked on it I've become a bit nervous about Helm's mostly client-side state management model (as of Helm 3).

A few times in development, I've had to kill deployments halfway through and discovered that the Helm metadata stored in k8s secrets is just out of sync with the cluster. I'm also not seeing great tools for defining failure states -- I want deploys to fail fast with no automatic retries, but I want failing nodes to be replaced automatically after a deploy registers as successful, and I'm not sure how to express that idea in Helm. To put it another way: a new deployment should have a backoff limit of 0 until it passes its initial health checks and becomes the primary deployment, because if you fail on initial rollout it's probably a bad deploy -- but after a successful rollout it should try to replace failed pods.

I don't really want to commit to a whole CI/CD platform just to solve this problem -- we've had lots of other issues with CI/CD and I don't particularly want to entrust this whole mess to some new provider just because they're good at doing k8s. But it seems like a relatively easy-to-isolate task: just rolling out new deploys and tearing down old ones. Is there something that fits into the workflow here? Argo? Spinnaker? I don't have a very good sense of what tools are useful for what.

https://redd.it/nwzpf6
@r_devops

HSM in the Cloud for non-HA usage?

Our product has 4 apps (Windows, macOS, iOS & Android) with 90% run in the cloud (Android, iOS & macOS are built entirely on cloud infrastructure) but for Windows, in spite of it being the most straightforward of the 4 build processes, at one stage we have to grab the .exe and pull it onto a local PC sat in our office (which is my home office as we're fully remote), sign the thing with signtool using a Digicert physical key, and then upload it back to the cloud pipeline to complete processing.

This obviously introduces a point of failure/bottleneck in our process, which is that if that machine goes down and i'm on vacation, or my internet goes down for an afternoon, or the machine needs maintenance and i'm not about, then we can't build our Windows app.

One solution here would clearly be to buy a second Digicert key & NUC and have it in another employee's office as a failover (i'm the only DevOps so they wouldn't be solving the maintenance part so much), but i'd rather pull the entire step into the cloud and use cloudHSM (we're an AWS shop) to do this process.

Doing some research, I initially assumed this would be cost prohibitive ($1.45p/h for 1x CloudHSM = $1450p/m, vs $699p/y for the Digicert key), however I found this article which suggests that if we don't need HA (we don't - builds only happen 2/3x per week on avg) it would obviously lower the cost dramatically.

The only security issue I can see is around the movement of the .exe in order to sign, however I believe I could mitigate this dramatically by keeping everything within the private VPC of the CI.

Does anyone have any experience with doing this, and what gotchas am I missing?

https://redd.it/nwie93
@r_devops

Understanding Kubernetes Architecture

A simple blog on understanding the Kubernetes Architecture in terms of Master-Worker Nodes. Let me know what you think of it.

https://dev.to/pghildiyal/understanding-kubernetes-architecture-2k0l

About Me: I am a DevOps Engineer, developer and Co-Founder of Devtron Labs(An Opensource Kubernetes delivery workflow).

https://redd.it/nvrqnx
@r_devops

Do you have a remote job ?

This is not about a personal preference, this is about your current situation at work.

View Poll

https://redd.it/nx31qm
@r_devops

iis web.configs best practices in production?

Hey all! In the past year, I’ve slowly been working more in a DevOps capacity for a medium sized company. We’re still a windows shop using iis to host .net web apps. We don’t really have any pipelines setup yet. Code is manually pushed to the servers(yes I’m crying too). Long story short, can y’all point me in the right direction as in,how to appropriately manage web configs that are constantly needing to be changed? Ansible, GitHub? How do I stop the madness?

Sincerely,
Frustrated

https://redd.it/nx4wot
@r_devops

New open source project - Opta

Hey folks, I've been working on a new open source project called Opta and it's finally at a stage where I'm looking for feedback! Please check it out :)

https://www.youtube.com/watch?v=nja\_EfpGexE

Opta is a platform for running containerized workloads on the cloud. It abstracts away the complexity of networking, IAM, kubernetes, and various other components - giving you a clean cloud agnostic interface to deploy and run your containers. It's all configuration driven so you always get a repeatable copy of your infrastructure.

https://github.com/run-x/opta/

https://redd.it/nwzli6
@r_devops

What are you automating?

I am new to this discipline and constantly read that knowledge of a preferably a scripting language (Python/Ruby) is required for automation. I am a Ruby/Python developer myself, but trying to understand what are SREs and "DevOps" engineers automating.

https://redd.it/nwwryv
@r_devops

creating a powershell container with M1

hey guys,

maybe it was a mistake, but I got myself a M1 MacBook and I am struggling a bit more than expected.

I have to use powershell for some old legacy azure stuff (which I want to migrate somewhen) but have to test it first.

Is there any ready-to-use docker-container to ramp up fast to get my scripts up & running? Any hints how to do that in case it does not exist?

Best,

dejeckehoot

https://redd.it/nx843k
@r_devops

Getting error Avoid using {....} Multiple lines block

Getting error when i run cookbook and my code is


# Add AD USer to local admin Group when AD User Not present

group 'administrators' do

members 'TNGLAL\SVC_Prod_RDK'

onlyif {

result = powershellout (checkformembership)

result.stdout.chop == 'False'

}

action :modify

append true

end

Please help me with it

https://redd.it/nu9r8m
@r_devops

Best practice for PCI app deployment on VMWare

Hi All,

Could someone please help me to understand the CI/CD best practice for deploying PCI (Payment Card Industry) application fleet in VMWare(On-Prem)?

Tech stack for provisioning :Terraform / VCD ProviderAnsible

I was exploring k8s and nomad workload orchestrator but those would be overhead to manage just for 1-2 applications

TIA!

https://redd.it/nu8sm2
@r_devops

Is DevOps a good career path?

I good as in pay, job availability and future prospects. I am thinking about taking it as an elective but unsure if it would be useful as someone who has a lot of backend courses. I could also pick Scrum.

https://redd.it/nua2u1
@r_devops

MLOps.toys - A curated list of DevOps tools related to Machine Learning

Just launched this small website that contains resources related to MLOps (the combination of Machine Learning & DevOps).

https://mlops.toys/

The website is completely open source and we'd love your help! Thanks :)

https://redd.it/nxdc0o
@r_devops

Load Testing on WebSockets

I'm looking for software recommendations for load testing websockets. Apache JMeter doesn't have native integration with websockets and the plugins don't seem to work.
Ideally it should be something where I could specify all websocket messages at once along with the authorization headers.

https://redd.it/nxekxg
@r_devops

Consolidating tools in our pipeline?

Currently we're in the process of evaluating/adding additional stages into our Azure DevOps pipeline and there's growing concern that we're heading towards maintenance hell.

We have Azure DevOps that holds our code and runs our pipelines. Tickets are hosted via YouTrack (used to have TeamCity, too, but moved to ADO).

Right now we want to add SonarQube and TestRail for code analysis and testing.

We like ADO, so it would be good to have tools integrate with it without having to sweat, but it looks like we'll be running a lot of services to manage our development. Just the maintenance of those tools and services looks like extra work that maybe shouldn't exist.

Is this the reality of the business that you either pay through your nose to get a complete package, or suffer through the maintenance of each separate component? Or did we just go down the wrong path and there's an industry standard "better way" to deal with the very basics? Is YouTrack a "liability" that could be consolidated into ADO for example? Or is there something preferable than ADO? Or everyone has their preference and everyone will suggest something different? :)

We're a .NET team, if that matters.

https://redd.it/nxexc4
@r_devops

Xms and Xmx for Spring applications

fellow devops/admins/developers,

while running your spring application in Kubernetes or in VM, do you set Xms and Xmx as same or different like Xms512m -Xmx4096m. Why?

What is the advantage of one over the other?

https://redd.it/nxetfk
@r_devops

puppet throws me an error

Notice: /Stagemain/Main/Hudon::Dbhwebsite/Mariadb::Userhwebsite/Execdb-hwebsite-user: Dependency PackageMariaDB-server has failures: true
Warning: /Stagemain/Main/Hudon::Dbhwebsite/Mariadb::Userhwebsite/Execdb-hwebsite-user: Skipping because of failed dependencies
Notice: /Stagemain/Main/Hudon::Dbhwebsite/Mariadb::Granthwebsite-hwebsite/Execdb-hwebsite-hwebsite-grant: Dependency PackageMariaDB-server has failures: true
Warning: /Stagemain/Main/Hudon::Dbhwebsite/Mariadb::Granthwebsite-hwebsite/Execdb-hwebsite-hwebsite-grant: Skipping because of failed dependencies
Notice: /Stagemain/Main/Hudon::Dbhwebsite/Mariadb::Granthwebsite-hwebsite/Notifyhwebsite-hwebsite Add user : hwebsite@localhost: Dependency PackageMariaDB-server has failures: true
Warning: /Stagemain/Main/Hudon::Dbhwebsite/Mariadb::Granthwebsite-hwebsite/Notifyhwebsite-hwebsite Add user : hwebsite@localhost: Skipping because of failed dependencies
Notice: /Stagemain/Main/Hudon::Dbhwebsite/Mariadb::Granthwebsite-blacksmith_backup/Execdb-hwebsite-blacksmith_backup-grant: Dependency PackageMariaDB-server has failures: true
Warning: /Stagemain/Main/Hudon::Dbhwebsite/Mariadb::Granthwebsite-blacksmith_backup/Execdb-hwebsite-blacksmith_backup-grant: Skipping because of failed dependencies
Notice: /Stagemain/Main/Hudon::Dbhwebsite/Mariadb::Granthwebsite-blacksmith_backup/Notifyhwebsite-blacksmith_backup Add user : blacksmith_backup@localhost: Dependency PackageMariaDB-server has failures: true
Warning: /Stagemain/Main/Hudon::Dbhwebsite/Mariadb::Granthwebsite-blacksmith_backup/Notifyhwebsite-blacksmith_backup Add user : blacksmith_backup@localhost: Skipping because of failed dependencies
Notice: /Stagemain/Main/Hudon::Setuphwebsite/Userhwebsite01: Dependency PackageMariaDB-server has failures: true
Warning: /Stagemain/Main/Hudon::Setuphwebsite/Userhwebsite01: Skipping because of failed dependencies
Notice: /Stagemain/Main/Hudon::Setuphwebsite/Hgproject::Projecthwebsite/Sshauthorizedkeyhwebsite01-gitlab-runner: Dependency PackageMariaDB-server has failures: true
Warning: /Stagemain/Main/Hudon::Setuphwebsite/Hgproject::Projecthwebsite/Sshauthorizedkeyhwebsite01-gitlab-runner: Skipping because of failed dependencies
Notice: /Stagemain/Main/Hudon::Setuphwebsite/Apache::Fcgiphp5-fcgi-hwebsite01/File/etc/php-fpm.d/php5-fcgi-hwebsite01.conf: Dependency PackageMariaDB-server has failures: true
Warning: /Stagemain/Main/Hudon::Setuphwebsite/Apache::Fcgiphp5-fcgi-hwebsite01/File/etc/php-fpm.d/php5-fcgi-hwebsite01.conf: Skipping because of failed dependencies
Notice: /Stagemain/Main/Hudon::Setuphwebsite/Nginx::Vhosthwebsite-front-web/File/etc/nginx/conf.d/hwebsite-front-web.conf: Dependency PackageMariaDB-server has failures: true
Warning: /Stagemain/Main/Hudon::Setuphwebsite/Nginx::Vhosthwebsite-front-web/File/etc/nginx/conf.d/hwebsite-front-web.conf: Skipping because of failed dependencies
Notice: /Stagemain/Main/Hudon::Setuphwebsite/Hgproject::Projecthwebsite/Sshauthorizedkeyhwebsite01-rsync-ssh: Dependency PackageMariaDB-server has failures: true
Warning: /Stagemain/Main/Hudon::Setuphwebsite/Hgproject::Projecthwebsite/Sshauthorizedkeyhwebsite01-rsync-ssh: Skipping because of failed dependencies
Notice: /Stagemain/Main/Hudon::Setuphwebsite/Apache::Fcgiphp5-fcgi-hwebsite01/File/etc/httpd/conf.d/module.php5-fcgi-hwebsite01.conf: Dependency PackageMariaDB-server has failures: true
Warning: /Stagemain/Main/Hudon::Setuphwebsite/Apache::Fcgiphp5-fcgi-hwebsite01/File/etc/httpd/conf.d/module.php5-fcgi-hwebsite01.conf: Skipping because of failed dependencies
Notice: /Stagemain/Main/Hudon::Setuphwebsite/Apache::Htpasswdhg.hwebsite01/Exectest -f /etc/httpd/conf.d/htpasswd.hwebsite01 || OPT='-c'; htpasswd -b ${OPT} /etc/httpd/conf.d/htpasswd.hwebsite01 hg 'john123':

Dependency PackageMariaDB-server has failures: true
Warning: /Stagemain/Main/Hudon::Setuphwebsite/Apache::Htpasswdhg.hwebsite01/Exectest -f /etc/httpd/conf.d/htpasswd.hwebsite01 || OPT='-c'; htpasswd -b ${OPT} /etc/httpd/conf.d/htpasswd.hwebsite01 hg 'john123': Skipping because of failed dependencies
Notice: /Stagemain/Apache/Servicephp-fpm: Dependency PackageMariaDB-server has failures: true
Warning: /Stagemain/Apache/Servicephp-fpm: Skipping because of failed dependencies
Notice: /Stagemain/Main/Hudon::Setuphwebsite/Hudon::Remotefile/etc/nginx/ssl/hivimax.com.bundle.crt/ExecHudon_retrieve_/etc/nginx/ssl/hivimax.com.bundle.crt: Dependency PackageMariaDB-server has failures: true
Warning: /Stagemain/Main/Hudon::Setuphwebsite/Hudon::Remotefile/etc/nginx/ssl/hivimax.com.bundle.crt/ExecHudon_retrieve_/etc/nginx/ssl/hivimax.com.bundle.crt: Skipping because of failed dependencies
Notice: /Stagemain/Main/Hudon::Setuphwebsite/Hudon::Remotefile/etc/nginx/ssl/hivimax.com.bundle.crt/File/etc/nginx/ssl/hivimax.com.bundle.crt: Dependency PackageMariaDB-server has failures: true
Warning: /Stagemain/Main/Hudon::Setuphwebsite/Hudon::Remotefile/etc/nginx/ssl/hivimax.com.bundle.crt/File/etc/nginx/ssl/hivimax.com.bundle.crt: Skipping because of failed dependencies
Notice: /Stagemain/Main/Hudon::Setuphwebsite/Hostwww.hivimax.com: Dependency PackageMariaDB-server has failures: true
Warning: /Stagemain/Main/Hudon::Setuphwebsite/Hostwww.hivimax.com: Skipping because of failed dependencies
Notice: /Stagemain/Main/Hudon::Setuphwebsite/File/home/hwebsite01: Dependency PackageMariaDB-server has failures: true
Warning: /Stagemain/Main/Hudon::Setuphwebsite/File/home/hwebsite01: Skipping because of failed dependencies
Notice: /Stagemain/Main/Hudon::Setuphwebsite/Hgproject::Projecthwebsite/File/home/hwebsite01/.ssh: Dependency PackageMariaDB-server has failures: true
Warning: /Stagemain/Main/Hudon::Setuphwebsite/Hgproject::Projecthwebsite/File/home/hwebsite01/.ssh: Skipping because of failed dependencies
Notice: /Stagemain/Main/Hudon::Setuphwebsite/Hgproject::Projecthwebsite/File/home/hwebsite01/.ssh/id_rsa: Dependency PackageMariaDB-server has failures: true
Warning: /Stagemain/Main/Hudon::Setuphwebsite/Hgproject::Projecthwebsite/File/home/hwebsite01/.ssh/id_rsa: Skipping because of failed dependencies
Notice: /Stagemain/Main/Hudon::Setuphwebsite/Apache::Fcgiphp5-fcgi-hwebsite01/File/usr/lib/cgi-bin/php5-fcgi-hwebsite01: Dependency PackageMariaDB-server has failures: true
Warning: /Stagemain/Main/Hudon::Setuphwebsite/Apache::Fcgiphp5-fcgi-hwebsite01/File/usr/lib/cgi-bin/php5-fcgi-hwebsite01: Skipping because of failed dependencies
Notice: /Stagemain/Main/Hudon::Setuphwebsite/Apache::Htpasswdremoteapi.hwebsite01/Exectest -f /etc/httpd/conf.d/htpasswd.hwebsite01 || OPT='-c'; htpasswd -b ${OPT} /etc/httpd/conf.d/htpasswd.hwebsite01 remoteapi 'azsxdcfv': Dependency PackageMariaDB-server has failures: true
Warning: /Stagemain/Main/Hudon::Setuphwebsite/Apache::Htpasswdremoteapi.hwebsite01/Exectest -f /etc/httpd/conf.d/htpasswd.hwebsite01 || OPT='-c'; htpasswd -b ${OPT} /etc/httpd/conf.d/htpasswd.hwebsite01 remoteapi 'azsxdcfv': Skipping because of failed dependencies
Notice: /Stagemain/Main/Hudon::Setuphwebsite/Hosthivimax.com: Dependency PackageMariaDB-server has failures: true
Warning: /Stagemain/Main/Hudon::Setuphwebsite/Hosthivimax.com: Skipping because of failed dependencies
Notice: /Stagemain/Main/Hudon::Setuphwebsite/Hudon::Remotefile/etc/nginx/ssl/hivimax.com.key/ExecHudon_retrieve_/etc/nginx/ssl/hivimax.com.key: Dependency PackageMariaDB-server has failures: true
Warning: /Stagemain/Main/Hudon::Setuphwebsite/Hudon::Remotefile/etc/nginx/ssl/hivimax.com.key/ExecHudon_retrieve_/etc/nginx/ssl/hivimax.com.key: Skipping because of failed dependencies
Notice: