prometheus + grafana stack in k8s. is it a silver bullet?

I'm <3 years junior DevOps and I have experience with 2 companies that both have 20\~30 devs. Both companies use prometheus + grafana stack for their monitoring especially k8s nodes & pods. However having used these stack I got some curious that are other companies using same thing as me. Because these were not that bad in most cases though, but some trivial bugs or several-years-old issues gave me a hard time. (or maybe they were all my fault)

And it seems there are 4 major way to deploy prometheus based monitoring stack in k8s..
prometheus-operator/prometheus-operator,prometheus-operator/kube-prometheus,bitnami/charts/kube-prometheus,prometheus-community/helm-charts/kube-prometheus-stack,... they really confuse me

\---

So I wonder

\- Are you using prometheus + grafana for k8s monitoring?

\- If yes, are you using them without any problem?

\- If not, what are you using now?

&#x200B;

Thanks

https://redd.it/10qy7j9
@r_devops

What’s a project, personal or professional, that you can never find the time/resources for? Why?

For me it would be to update my personal website with a blog portion that contains well documented posts of my completed projects. I have the projects done. I have the knowledge and experience. I know what to do to get this all completed. I don’t really have any excuse, especially considering that if the big project was for work I’d already have the documentation completed.

https://redd.it/10r0i5w
@r_devops

Monthly 'Shameless Self Promotion' thread - 2023/02

Feel free to post your personal projects here. Just keep it to one project per comment thread.

https://redd.it/10r0ixm
@r_devops

Best Free Full-Stack monitoring Suite?

We currently have a hybrid Datacenter-AWS environment. We are currently using Dynatrace, but it is too expensive. What suite do you use to monitor Applications, Servers, Containers, networking, DB, logs, etc?

https://redd.it/10r1ca2
@r_devops

Test environment for branches?

Is there a standard procedure for manual testing different branches in a repo?

For example I have Azure DevOps server and I have an environment set up for my dev branch (site, db) so people can click stuff in the current general state of the app.

If I start implementing a feature on a separate branch, how would I handle the environment? Like say I want to show off the progress to see if this is functionally the right direction or whatever.

I would need to provision a new environment, frontend, backend, db, domains, etc. How would I achieve this in an automated manner?

https://redd.it/10r043l
@r_devops

looking for ways to learn AWS or other cloud service providers in a production environment

I have good experience managing infrastructure in a private cloud for more than 5 years, with strong k8s and docket fundamentals. However my job didn't involve me working on any public cloud providers. Every job I am looking for asks for strong experience in atleast one of the CSPs. So what is a good way to learn and use one of the CSPs at production level so that I can justify my experience. Don't want just introduction to AWS or something else.

https://redd.it/10r5tzt
@r_devops

Code Automation Help

Hi all,

I'm trying to figure out what the most efficient way is to automate my app deployment process. Currently using GitLab to store code for an app and using an AWS EC2 instance that it gets deployed on, I'm currently limited to EC2 only. My current process is, I have a Packer script that builds out the AMI image and includes a line to clone my git repo for the app code.

This is a manual process each time I update so I'd like to change it where whenever new code is committed in GitLab, it can automatically kick off the Packer script to create the newly updated AMI image. Is this possible to integrate with GitLab CI/CD or any other ways of doing it?

Thanks

https://redd.it/10r83zs
@r_devops

Frequent Targetgroupbinding reconciliation on an EKS Cluster

Hi Everyone,

Viewing my event logs, my EKS cluster tends to perform a lot of reconciling (less than 15mins continuously) by Targetgroupbinding with exposed service containers.

I understand it has to perform leader board election but checking if this is normal behavior

https://redd.it/10r9jst
@r_devops

What to expect for first DevOps interview? (with DoD-based contractor)

Hi everyone, while I think I have a federal "DevOps" role in the bag (seems to be taking forever for a solid offer letter though). I was invited by two DoD based companies to do an interview with them for mid-DevOps roles. I'm DevOps now, I think?

I use docker, Ansible, I create bash or python scripts to lockdown systems or to install sw prereqs, I do site reliability (but without monitoring software), I troubleshoot finicky service scripts, I ACAS scan systems for security vulnerabilities, I create and backup Linux systems, and of course more. So I do DevOps stuff but then I look at these job listings I applied for and I don't know cloud (@lat least no work experience), I don't know load balancers, I haven't produced code for a project, I don't know kubernetes, I don't practice leetCodes. Am I walking into a mind field for this interview? I have "PowerShell" scripting on my resume but now that I think of it, I have scripted anything in PowerShell since I made my few "Windows baseline configuration" scripts 2-3 years ago. 😶 I just use them quite often.

DevOps broadness is awesome when you're on the jobs but nerve-racking when you are trying to land a position. My current position was for Systems Administration but I kind of got thrown onto a very fast moving agile development project where I learned my above skills.

So what is expected of me if I don't know some of these technologies, what are some ways you guys navigate unknown technology questions in interviews?

https://redd.it/10ra89y
@r_devops

Monitoring mobile apps

How do you monitor your mobile apps to see if it has issues loading?
Story: We have mobile apps available from both Playstore and App store. We get issues sometimes when the app doesn't load at all or shows a blank page. Web application(browser based) loads fine at all times. Hence all our monitoring checks pass as it's looking at web applications only.

My question is how do you ensure uptime on mobile apps? Please advise.

https://redd.it/10qxsh4
@r_devops

Folks In Healthcare Industries, what do you use for application monitoring?

Tl;dr: Large hospital, primarily Windows Desktop/Server, Epic. We have infrastructure monitoring covered, what do you recommend for application monitoring when there is TONS of health care applications already in the environment?

Without disclosing much, I work a hospital. Rather large one. We're an Epic shop. We run our own datacenter. Been here for 4 years, came onto the DevOps team recently. Started in operations, went over to the systems administration team so it's a huge plus as a lot of the monitoring tools I used to be the operator on, I am now the maintainer.

I had a realization today that we don't have a clear tool to do application monitoring. Infrastructure is certainly covered as we have vROps, Loginsight, NagiosXI and other solutions at our disposal to monitor the underlying infrastructure. I've tried working with Telegraf, which is integrated with vROps now; had my gripes and such with it but it does OK; just not the best or what I would want for my org for an application monitoring perspective. Best we can do right now is Windows Services monitoring, but I know we can do better than that.

There must be at least 200+ applications in the environment total. That's just a guess. I checked out roadmap.sh and it mentions Jaeger, New Relic, App Dynamics, Instana, OpenTelemtry. Checked out Dynatrace but the agent is 5gigabytes.... just seems a bit big.

What's your take?


Edit: I have a test environment at work that is usable. Additionally, my own homelab at home. So I'm ready to tear at something. Just wanted to gear it towards Healthcare applications specifically.

https://redd.it/10rdzc2
@r_devops

When did you shift to devops and where do you see yourself going amidst recession and do you thinks devops will stand strong amidst Artificial intelligence in coming years. Give solutions as well to overcome to all these questions please. Thank you!

..

https://redd.it/10rd6tk
@r_devops

Continuous Delivery vs Deployment

https://adamdingman.net/posts/continuous-delivery-deployment/

https://redd.it/10rgrsa
@r_devops

I was exploring Nginx web server and got error as permission denied. I don't find where. I have given every permission.

This is my nginx.conf:

events{}

http {

server {

listen 80;

server_name localhost;

root /home/mrburnwal/Downloads/DhirajCV-master;

}

}

Error from error.log:

2023/02/02 09:06:11 [notice] 810306#810306: signal process started
2023/02/02 09:06:18 [error] 810307#810307: *8 "/home/mrburnwal/Downloads/DhirajCV-master/index.html" is forbidden (13: Permission denied), client: 127.0.0.1, server: localhost, request: "GET / HTTP/1.1", host: "localhost"

and my file permission that:

drwxrwxrwx 2 mrburnwal mrburnwal 4096 Nov 20 2021 css
-rwxrwxrwx 1 mrburnwal mrburnwal 2915 Nov 20 2021 index.html
drwxrwxrwx 2 mrburnwal mrburnwal 4096 Nov 20 2021 media

it should have run successully with this config. But getting errors. Ans yes, I have reloaded nginx after making changes as well.

systemctl reload nginx

https://redd.it/10rfzo9
@r_devops

My 2023 plan

I'm a noob to devops, but not a noob to software development. Anyway, my current job is titled as devops but it's actually code fixes and maintenance and shit, hardly an opportunity to learn cool new shit, particularly cloud native stuff, which I'm interested in. I've made a plan to learn cloud native stuff in my free time. Here's the plan below and let me know what you think:

Goal: deploy and manage full stack app in k8s

\- pick a cloud from either Azure, AWS, GCP

\- learn ArgoCD

\- learn Rancher

\- try out GitLab CI/CD

https://redd.it/10rig0b
@r_devops

Thoughts on using GPT tools with databases

Team members have discussed the implementation of an AI product for querying databases. While I have some initial reservations, I am also intrigued by the product's innovative features. I understand that it may appeal to small/medium companies with limited resources, but I am interested in exploring its potential and determining if it offers any advantages over developing a solution using open-source tools.

I would welcome any insights or perspectives from those who have knowledge or experience with similar tools.


For reference:

https://twitter.com/python\_spaces/status/1620607399299280897

https://redd.it/10rjuei
@r_devops

CVE vs CWE

Hi all, so my company is moving from Veracode to Mend(White source) for code scanning - I’m trying to do a small test to check if mend is able to catch all the vulnerabilities caught by veracode for the same library. I noticed that mend wasn’t able to catch some vulnerabilities that veracode could figure out. We also use codeQL for CWE scans and I don’t have that data yet with me- but I wanted to know how much of a difference will it make if I do get the CWE data for the same library- will it be able to make up for the discrepancy as CWEs are different from CVEs. I really need some help here! Thanks a lot in advance!

https://redd.it/10rk576
@r_devops

Azure DNS query logs for analysis

Hello Everyone,


I am working on a project where I have to do analysis of the DNS Queries (A record,AAAA record,CNAME record) on Azure Platform to make cloud infrastructure related decisions.


I have checked the metrics option in the Azure Portal that seems to be with limited with the scope i.e. query volume of A records.

I am looking for solution with more insights on the DNS Queries, can this be achieved using any Azure Services or any kind of scripting.


Thanks in Advance

https://redd.it/10rja9d
@r_devops

Pearson VUE cancelled and refunded my 'AWS Professional devops engineer' cert test with no reason.

Anyone else have this issue? I signed up and payed earlier in January. Course showed as active upcoming. Today I received email stating my cert test time/date has been cancelled and refunded with no other reasoning. Is this normal? How can I guarantee this won't happen again or closer to the exam date?

Update: I contacted PEarsonVue customer support and they had no answers. They told me to apply again

https://redd.it/10qzh59
@r_devops

Boast the Potential of DevOps with CI/CD

Article link: https://www.impactqa.com/blog/boast-the-potential-of-devops-with-ci-cd/

&#x200B;

The above article discusses the potential result of the combination of continuous integration & continuous deployment (CI&CD) with DevOps. Do you think they did right? I am not an expert in this field but they did omit a few important aspects.

https://redd.it/10rn9w8
@r_devops

Dockerized Jfrog Platform

Hi friends,

My team and I are looking to install the Jfrog platform (Artifactory, Xray) but we’re hesitant if we should run these services as docker containers. Especially the Postgres DB for each service. We are a supporting around 100 developers.

Update: it will be self-hosted.

https://redd.it/10rnskd
@r_devops