Monitoring and Metrics replacement project - looking for suggestions opinions

​

Hi all,

I'm a System Engineer with limited knowledge around monitoring and metrics working on a small team. I was handed a project to work on cleaning up what we have (SolarWinds) and gathering basic metrics for business critical items to be observed and shared within IT. I wanted to present my overall landscape and see what more knowledge folks had suggestion wise.

**The current landscape: \~**

* 200 servers
* 80 different physical locations (offices)
* fair amount of networking equipment
* 17,000 elements (data points/thingymajigs) in SolarWinds speak

We are using SolarWinds to monitor and alert via email and teams, nothing groundbreaking here. We have a mix of on-prem hardware, AWS, and Azure. For the most part it’s basic out Monitoring of servers and network equipment like up/down, CPU, Mem, Disk, bandwidth, with a few oddballs sprinkled in but nothing critical on the oddball stuff. We do have a bit of SQL monitoring and IIS monitoring but none of it is tuned in the slightest.

**The need:**

* Agentless (I know this is a limiting factor for greater detail)
* General cost on par with SolarWinds (not terrible because we are relatively small)
* Ability to the monitor basics above
* Need the ability to pull in data from other sources into a presentation tool such as Grafana. Some of the metrics I'm being tasked to pull in will not be easy or practical to pull from SolarWinds or Zabbix, or whatever we choose from the monitoring side. So the ability to pull in data from other systems, CSVs, APIs, or just text would almost be a requirement.
* The big "WANT" is, 1 central place to display & present the data.

I’ve started looking into things like Zabbix + InfluxDB + grafana, PRTG, (Datadog soon), and generally lurking in this forum for the last few weeks researching stuff. But as with a lot of us out there in the business this isn't my primary responsibility so there is only so much brain space to devote to it :).

As an aside, SolarWinds is pretty quirky and clunky but does things “OK” as my org needs them. I’m pretty sure with a stupid amount of work i could make SW do most of what needs to be done. But the level of effort involved there might be 2-3x more than dropping in multiple other systems.

https://redd.it/u3y6d8
@r_devops

Cake v2.2.0 released 🎉

Highlights of this release

⭐Opt-in script caching for significant performance gains

⭐New aliases for using dotnet workload search

⭐Fixes in UNC and relative path collapsing

⭐Improved handling of .NET runtime reference assemblies

More details in the release blog post

https://redd.it/u47o82
@r_devops

Service Meshes - Reasons to avoid them?

I've read a lot about Service Meshes and like a lot of the possibilities but have seen a lot of things saying they should be avoided until absolutely needed.

I haven't seen good descriptions of why, however, outside of "it takes a lot of effort".

Can someone explain why it wouldn't make sense to start out with a service mesh with defaults and then invest in individual features when it makes sense?

Are there legit horror stories or is it just scars from early adopters that have been mostly worked out?

https://redd.it/u4ae46
@r_devops

Manage cloud infrastructure as data in PostgreSQL

https://blog.iasql.com/os-iasql/

https://redd.it/u3ukrx
@r_devops

I'm Implementing DevOps in my Organization. Which CI/CD tool to prefer

I'm currently asked to choose from one of the three Jenkins, GitHub actions and Travis CI to use as CI/CD tool but I don't know much about any other tool except for jenkins.
My organization is a growing startup, all the codebase is in GitHub and currently I'm the only DevOps person who itself is an intern. I really don't wanna do something that will haunt me back later.
Any suggestions??

Edit: I'm highly familiar with Jenkins though

https://redd.it/u4nw0x
@r_devops

most cost-effective backend

my backend runs fine on a high-end consumer-grade computer. It doesn't need a GPU. I am wondering what is a cost effective backend service. AWS ec2 seems too expensive.

https://redd.it/u4n3eq
@r_devops

Unix command cheatsheet

https://tvkoushik.medium.com/unix-command-cheat-sheet-for-busy-developers-ec1bea462861?sk=bd5e8eac34546e9ca294612f3582687a

https://redd.it/u4r2a4
@r_devops

Azure Devops > Simple File Copy with Deployment Groups

I am defining a Release Pipeline - and I want to simply copy files (the deployment assets) to a directory on the remote agents (on-prem). I do not want to specify credentials. The deployment agent should pull the assets from Azure, similar to how the "IIS Web App Deployment" task group works.

This is a simple console app that I am deploying. I cannot find an appropriate task to do this. Anyone have any ideas?

https://redd.it/u4j5q0
@r_devops

Roles on continuous delivery

Hi every one, due to a university protect I was investigating about the roles in a software company or software development team, that implements continuous delivery, all I can find was the roles in scrum, and what I need is a more general set of roles.

I really appreciate any kind of help, and sorry for my broken English.

https://redd.it/u4h67v
@r_devops

New Route53 Cli release in - Get info about your records from the terminal - quickly!

New Release - r53

Example:

r53 -q
my.company.domain.com

It will return a list:

Hosted Zone ID + Web URL
The target behind (Load balancer, Lambda, etc) + Web URL to target
Recursively expand records
Verify NS match with dig

Install:

$ brew tap isan-rivkin/toolbox

$ brew install r53

New features:

\- Exposed SDK on top of CLI

\- Now support JSON output --output-json

https://github.com/Isan-Rivkin/route53-cli

​

\#aws #route53 #golang #go #dns #networking

https://redd.it/u4u0it
@r_devops

Jenkins vs gitlab

Hi people. About CI/CD, I would like to know your opinion between Jenkins and gitlab, the pros and cons. Can we talk about?

https://redd.it/u4kxj0
@r_devops

Are “devops” salaries stagnating?

A decade ago if I made rhe mistake of picking up the phone when a San Francisco headhunter called I would ask for their salary range out of curiosity before I hung up, it was always $140k-$160k.

Today it seems to be $150k-$170k, unless you’ve written books or promoted the hell out of yourself on twitter.

I make almost twice that consulting on managing offshore eng orgs and would never consider a real job again, but why are salaries so low and stagnant in a city where the cost of housing has doubled in 10 years?

they would give me a $140k-$160k salary range and I would hang up.

10 years later I’m given the same salary range

https://redd.it/u4yf7k
@r_devops

How to backup and restore db on postgres?

Hi everyone,
I plan migrate db on postgres 9.3 to 13, when perform restore i have a many error, due to underscore symbol on table name, for example all tables have like _table1 etc.
Please help, how do restore db ?

https://redd.it/u4xebq
@r_devops

How does Circleci compare to github actions and gitlab

longtime lurker here... not a full time dev ops but noticed that most people prefer GitLab or Github actions... I have been using circleci for a while now... wondering if i'm missing some features staying with circleci, so wanted to get some input... We use cci to run a pipeline to k8s cluster, i can create a new cluster in few lines if needed.. also creating docker images and pushing them to registry is pretty seamless...

Whats the hive mind input on Circleci?

https://redd.it/u509au
@r_devops

is kubernetes for entry level devops engineers ?

I am self-taught and I happen to like kubernetes.
I dont have a job yet.
On last interview I was emphasised how bug kubernetes is and that there are people who only do that.
Get I get a entry level job focusing on kubernetes? Can I push it as my selling point ?
Thanks

https://redd.it/u51hf7
@r_devops

My company is starting git, how screwed am i?

Let me explain. Against my stern warnings, they are instituting an environment based system. So the Item is done on local branch, it gets merged to test. Where it gets crazy is from test to stage. They are going to cherry pick each individual item from test into stage. Never merging.

I personally am in charge of implementing this. Are we as screwed as I think we are? Will we be solving merge conflicts on every cherry pick? Should I just get it over with and shoot my foot right now?

Pretty much I want to know from people who might be more experienced than me if this is as bad as I think it's going to get? Am I just nervous and overexagerating?

https://redd.it/u54r5v
@r_devops

My company is starting git, how screwed am i?

Let me explain. Against my stern warnings, they are instituting an environment based system. So the Item is done on local branch, it gets merged to test. Where it gets crazy is from test to stage. They are going to cherry pick each individual item from test into stage. Never merging.

I personally am in charge of implementing this. Are we as screwed as I think we are? Will we be solving merge conflicts on every cherry pick? Should I just get it over with and shoot my foot right now?

Pretty much I want to know from people who might be more experienced than me if this is as bad as I think it's going to get? Am I just nervous and overexagerating?

https://redd.it/u54r5v
@r_devops

medium / small company version list of faang?

Hey Everyone,

As the subject states, I am curious if there is a "FAANG" type list for medium/smaller tech companies? I tend to enjoy the small to medium size companies (less politics more impact usually), so just curious what are companies that fall into this category?

If people are looking for some examples, I would say some medium size companies are:
Splunk, PagerDuty, Hashicorp, Slack, Atlassian, etc.

Any feedback would be greatly appreciated. Thanks!

https://redd.it/u57n76
@r_devops

Interview questions for employers

Hi /r/DevOps it's easy to find search results on google on interview questions for the person applying but I'd like to make a list of intelligent questions one should ask the interviewer to get a good feel of what DevOps looks at their company.

One I was going to ask is how does on call look at your company. Is it repetitive? Do you have a process to ensure you won't get another on call event generated for the same event?

Sorry if this was already answered, I couldn't find it anywhere.

https://redd.it/u5g156
@r_devops

Is my raise enough?

I’m heavily involved in AWS, CI/CD, and architecting and engineering solutions at an east coast higher ed (not in NYC)that’s hemorrhaging IT staff and has no concrete WFH policy. Is a 6.5% raise and a 4% bonus split over two payments good or am I being too critical? Salary now at 130k. Recruiters contact me for positions starting at 150k.

https://redd.it/u5monh
@r_devops

Need help setting up self hosted loki logging mechanism for docker containers.

I am trying to create a monitoring mechansim for our microservices. After an extensive research, I found out using prometheus + loki + grafana is the most scalable, least challenging and most maintainable solution for us since we are going in the self hosted direction.

My current problem is that while prometheus + grafana seemed pretty straightforward, I cannot add loki to the mix. The documentation is a bit confusing.

​

There are apparently two options for Loki + docker for logging.

1. Using promtail + loki for logging, and mounting docker container log files in promtail environment.
2. Using loki logging driver for containers that I want to log. (source)

The first one is not only confusing, but also way less maintainable, since I probably have to lunch a new promtail instance + config files for each time a new type of microservice is getting deployed. Plus there's this hassle of service discovery and container filtering that I have to write manually in the promtail config yaml that is daunting to say the least.

​

The second one while seems pretty straightforward if you use the centralized grafana solution, I cannot find any examples of it being done in a self hosted situation. Has anyone done this? If so, I need a bit of an explaination to how to exactly set up a very small, reproducible example.

https://redd.it/u5qd19
@r_devops