Kibana - segregating logs based on a specific field

I am wondering if it is possible to filter/segregate data inside Kibana based on specific field.

For example, we have a multi-tenancy production environment and sometimes we have to work with external partners. Sometimes they request access to logs which we cannot give, since they would see a logs/sensitive data which does not belong to their project.

Obviously, one solution would have one kibana instance per tenant/client but at the moment this is not so easy for us to achieve since our entire backend running on EKS is tenant agnostic; meaning all services process events/data for all tenants.

I wonder if there is a way of segregating data inside kibana based on a specific field; so show only the logs of tenant A and give it to external developer. Our kibana is running on managed elasticsearch provided by AWS.

https://redd.it/ogct81
@r_devops

devops from interview to hired. What are your guys story when it came to getting your first devops gig?

Currently practicing devops by studying for certs, creating projects, and studying LeetCode. Got my first devops gig last year which was supposed to be a regular position but got thrown into more of a senior role even though the title itself didn't reflect it. Learned a lot though there wasn't much coding (at least for me). I did get to do a whole lot of sysadmin which was nice and I helped in shaving down time for OS install automation, IaC, etc. Now, here I am, laid off because of COVID (for a while now actually). How did you guys get your first and/or second gigs? Curious to know what all of your experiences are like. Open to any questions that you may have as well!

https://redd.it/ogdchl
@r_devops

External retrieved data integrity?

Hello /r/devops

Not sure if this is the right place to ask, but since DevOps might involve with some cryptography I decided to ask my question here.

For a new system we're building, we are systematically retrieving data from an external system (Basically via a cron job pulling new information from a REST API). One of the technical requirements is that we should be able to proof that the data we locally store, is indeed coming from the external data source after X amount of time. My first thought was to store some kind of reference to the SSL certificate used in the request and/or the certificate chain and do something cryptographically with the data to verify it's integrity (Not sure how yet, it's my first gut feeling on how to approach this since the SSL certificate is identifying the server the data came from).

Anyone has experience with this kind of subject? Or is able to point me in the right direction?

Cheers!

https://redd.it/oge39q
@r_devops

Live now Tim Kadlec testing WebPerf Optimizations with Cloudflare Workers & WebPageTest

Join the stream >> https://twitch.tv/webpagetest

https://redd.it/ogc46g
@r_devops

Service Connections on Azure Dev Ops

Hey Guys, someone know if its possible to create a Sonarqube service connection on Azure DevOps, calling a internal webserver or its mandatory that the webserver must have a public ip?


https://imgur.com/XTUjDSm

https://redd.it/ogf2ab
@r_devops

Canonical Survey highlights skills shortage in Kubernetes, with 54.5% pointing to a lack of in-house skills/limited manpower as their biggest challenge when migrating to/using Kubernetes and containers.

Canonical recently ran an interesting survey where they gathered data from 1200 respondents on hybrid and multi-cloud operations, Kubernetes, VMs, bare metal, goals, benefits, challenges, operators, advanced usage, edge, and more. This blog post digs into 5 of the data points we found most interesting.

https://redd.it/ogi31q
@r_devops

GITLAB CI/CD pipeline for embedded systems

Hello,

I am currently trying to develop a CI/CD pipeline in Gitlab that performs build, Static code analysis, unit testing, flashing an Stm32 using Raspberry pi, and smoke tests of the code on the stm where it communicates the results to the Raspberry pi.

I am currently trying to make the flashing stage of the pipeline where I have a Gitlab Runner on the Raspberry pi. As I understand it, I have to specify the port before running the docker image (when performing "docker run" ). However, I am uncertain as to how I will be able to the access the USB port on the Raspberry pi to flash the device when I am not performing the "docker run" command. Do I have to specify the port in the Entry point of the dockerfile, or is there a better way of doing it? Also, is there a way to make the runner notice which USB port is connected to the stm32 so that I do not have to hard code in the port?

Thank you

https://redd.it/ofk8hb
@r_devops

Measuring 40x and 50x errors on a K8s service endpoint with Prometheus?

Hi community,

Any idea how to go about this, can't find much clear info on google, to measure the Errors (40x, and 50x) on my service endpoints. My services are up and when I delete pods just for a test, I can see in the blackbox metrics that the prometheus gets and error, but not specified like 40x type or 50x.

https://redd.it/ogitx4
@r_devops

DevOps w/emphasis on Dev

I've been around on this sub for a while; mostly lurking. I created a new account today so as not to eventually dox myself.

My impression is that a lot of folks here are in what seems to be a more Ops focused role. My team is "Engineering Operations" and our job is to develop tools that help dev teams implement devops or just generally deliver their code more quickly. Aside from using DevOps within our own team, we don't manage any infrastructure, that's done by an entirely separate team that we work closely with.

Is this fairly common? How many of you basically focus entirely on development of tools? Is that just your role, or the role of your team?

https://redd.it/oggqzy
@r_devops

Do my French salary is enough ?

Hi,
I’m actually working as a devops engineer in a French fun-tech.
I have 3 years of experiences.
I’m working on technology such as Jenkins Kubernetes Salt Ansible.
Developing with Python and Groovy.

Actually my project is the migration of an Java applications into a modern application with spring (I’m only working on the infrastructure part).
I have actually migrate 99% of the application on Kubernetes with cluster on aws, azure and on-premise cluster (which I have setup alone).

My salary is 44000€/ Years and I’m living in Paris.

Actually thinking about finding a new job to have a better salary.

Did you think I’m wrong ?

Thanks for your response and sorry for my English !
I’m available for any of your questions.

https://redd.it/ogeqwx
@r_devops

GitOps - the Evolution of DevOps (list of resources)

Sharing a curated list of GitOps resources: articles, books, podcasts and videos to help anyone interested in the topic get a profound understanding of it.

https://github.com/microtica/gitops-resources

The idea is to create a centralized place with the most valuable GitOps content.

This will be a growing list available on GitHub, so please feel free to contribute.

https://redd.it/ogx6ta
@r_devops

How do you manage Jenkins pipelines at scale?

Working in a company with over 1500 employees. 7-8 different languages. Adding complexity of versions like Python 2 and 3, Java 8 to 11 etc.

How do you manage pipelines at scale?
Keeping 1 pipeline per language? (Then how to keep things flexible)?
Keeping pipelines in repo?

Or any other way to manage?

https://redd.it/ogwps4
@r_devops

HELP Azure Pipeline DevOps



Hi guys,

I'm here because I need some advice. After all, It's my first Azure Pipeline. I'm currently developing an Azure Pipeline and I already have Build Pipeline with these stages:

Info
Copying
Test
Security
Build
Updating

So, I want to use a Release Pipeline and migrate two stages from Build Pipeline to Release Pipeline. Specifically, the stages are "Build" and "Updating" that you can find the code here.

Finally, if you see other advice from my code, please let me know.

Thank you in advance for any help you can give!

https://redd.it/ogypps
@r_devops

Any of you ever take one of those “nice to haves” out of your backlog without telling anyone to work on during your spare time?

We have a rather large back log with things we’ll probably never get to any time soon. A good chunk of these issues I’m not really familiar with. I just picked one issue, and plan to try to “build” it out myself off hours for learning/experience.

https://redd.it/oh1dqi
@r_devops

Security metrics and activities

Hi folks. I’m really interested to learn more about the kind of security checks you guys do as part of DevOps. From pipeline stuff to how you manage or ensure security or your containers? Also interested in resources to learn about the kind of security metrics you can and should gather in your environment? Any pointers to good resources will be appreciated.

https://redd.it/oh0xnt
@r_devops

No Escape from Data+ML Lock-in

Reflecting on a recent a16z article on cloud costs, it occurred to me that there's no workload portability for Data & ML. I dug into it some in this post based on my experience but would love to hear from others if this makes sense. I've had some peers review it too and it seems to hit the mark, despite some gaps in my knowledge and experience (e.g. core network infra world). Have others observed this too? What am I not thinking of from a Data+ML workload portability perspective?

The domain (data+ML) is still nascent so use cases, tech and pricing are far from mature but from my perspective, it seems like there's a clear product strategy on Cloud providers' part to build these irreproducible strategic assets that lock you into their platform.

https://redd.it/ogqcnn
@r_devops

Difficulties of viewing backlogs when you use a single project with multiple areas in Azure Boards

All the MS documentation these days seems to suggest the use of a single project for almost all use cases. It also looks like they suggest using areas to divide this project so that you can limit what you view in some views as the Product Backlog .

However for me I have a problem in that as soon as I use the filter to view one area then the hierarchy goes away and it changes to a flat view.

I seem to share this problem with many others who have been complaining about this since 2018. Do you also have this problem? If so then how does it effect you and do you have some resolution for dealing with it? Also if it effects you then might be a good idea to upvote here.

https://developercommunity.visualstudio.com/t/boards-backlogs-please-keep-displaying-the-hierarc/366475

On checking the above link in more detail it appears that the person who is working at Microsoft who it is assigned to has had no issues or activity since Feb 21 :-(

https://redd.it/ohbfyh
@r_devops

Current market rates for 100% remote (US)?

I am the business manager for a very small (\~10 people) cloud consulting company & it's been a real struggle to find good DevOps/Platform Engineer folks to join our team. The existing guys are all really smart & amazing at what they do and have worked together in the past at other (much bigger) consulting firms. They are just stretched so thin right now due to how in demand we are & I really want to find some great new people to add to the team.

I am realistic in that I know we are a very small shop and do not have any kind of brand recognition other companies do, so I want to ensure the package we are offering is good enough for the realities of the market.

Currently:

* 100% remote (US only right now), will always be remote.
* 20+ days paid vacation, plus sick leave, plus personal leave.
* Medical/Dental/Vision/Life insurance (2 options, Platinum/Silver), 75% premiums paid by us.
* Additional training and certificates encouraged and paid for by us.
* Hands-off "management", not completely flat but very close.
* Opportunity for career progression and growth with the company and input in shaping the culture and structure.
* We plan to add 401(k) options next year.

Right now our (very broad) salary bands are:

Junior -- $60k-$100k
Mid-level -- $100k-$140k
Senior -- $140k-$180k
Senior/Management - $180k-$200k

Does this suck? Would appreciate feedback as the average rates I see on Google don't seem to align with what I see in this subreddit.


Ty!

https://redd.it/ogxryu
@r_devops

NewRelic

When I connect to a website using newrelic, certain pages show in uBlock Origin that I connect to log-api.newrelic.com. Other pages (such as the home page or other navigation pages) only show me connecting to js-agent.newrelic.com. I just wanted to know why this is the case? Thank you!

https://redd.it/ogwtmx
@r_devops

Looking for high quality advanced training in Devops topics ?

I checked out a lot of resources in udemy, kodekloud, and LinkedIn but they are very beginner focused and it’s like deploy hello world or create a Docker container

Where I can learn advanced topics or practice labs that are complex and can prepare you for interviews?

Something like build a full CiCd pipeline with testing and code coverage linking several repositories

Are there some good GitHub forkable projects to play around with? I prefer the structure of a course or video to understand the concepts and how and why

Expectation from interview is you know it way in depth and not basic “what is a vm vs container”

Preferably want to learn more about nginx ingress controller, service meshes, and Gitops. I just see companies say they are looking for terraform and kubernetes experiences, with Cicd

It can be a huge monster in how complex it is.. and I want to learn concepts someone who is a subject matter expert

I found books to be more detailed and technical .. a lot of video content is basic YouTube explain a concept but it’s on a high school level or very general

Or they talk theory when you should know in’s and outs, gotchas, and industry best practices .. it’s super rare to find these training materials

Do you guys craft your own training study agenda and work to make progress? It’s easy to hit a ceiling where your skills are not leveling up because the practice is not challenging enough

https://redd.it/ohhajc
@r_devops

Deploying a .netcore app on elastic beanstalk

I have been trying to deploy a .netcore app on elastic beanstalk using amazon linux 2 and .netcore 2.1 with nginx and getting a unhandled exception error.
I have been also manually trying to deploy on a instance as well and having issue with the same error. Here is the error,

web: Unhandled Exception: System.Security.Cryptography.CryptographicException: Unix LocalMachine X509Store is limited to the Root and CertificateAuthority stores. ---> System.PlatformNotSupportedException: Unix LocalMachine X509Store is limited to the Root and CertificateAuthority stores.

From NGINX error log:
connect() failed (111: Connection refused) while connecting to upstream, client: xxx.xxx.xxx.xxx, server: , request: "GET / HTTP/1.1", upstream: "https://127.0.0.1:5000/", host: "xxx.xxx.xxx.xxx"

I have created a public certificate using aws cert manager and attached it to a load balancer on beanstalk configuration with enabling listen port for https.

Previously the development team was deploying on azure webapp where i have been told that azure itself used to attach a SSL certificate to its app or service and they had nothing to worry about.

Now they are planning to deploy on AWS as well. Which i am assigned to.

I have been trying to do this for a week and not able to deploy it with this Certificate error.

Any devops engineers who have worked with deploying .netcore app on linux?

Please advise/suggest if there is anything wrong or something that i am missing to do.

https://redd.it/ohi960
@r_devops