random question: What is the average percentage of errors you'll be facing in your daily devops work routine?

Just wanted to know how much a devops person spends a day on solving the errors

https://redd.it/ydnh06
@r_devops

Regex based web tail for Kubernetes

https://youtu.be/86wHchb2LxI

Like a kubectl logs... but through a web console and a regex based filter

https://redd.it/ydrk48
@r_devops

Prometheus metrics on Cloudwatch

I am trying to integrate Prometheus with our servers to get application level metrics, so I was following this "Monitor and scale your Amazon ECS on AWS Fargate application using Prometheus metrics" blog.

The cloudformation template works and I am running cloudwatch agent in my cluster but for some reason I cannot see prometheus metrics in Cloudwatch. The only difference I could find is that in this blog they are using public subnet but my server is in private subnet and in my case I am using a common security group with all traffic allowed in both inbound and outbound rules for the cloudwatch agent and my server.

​

Any suggestion on what I am doing wrong?

https://redd.it/ydtlw0
@r_devops

Advice please. Reddit gold for help. :)

I am an experienced developer but a DevOps n00b. I recently taught myself the basic of Kubernetes / EKS / Jenkins and Cloud Formation.

I have a large project where I will be rolling out the same code base for 60 different clients, it's a cross of Sass/PaaS I guess? Where each customer has their own Kubernetes cluster running their app(s), a Redis cluster and an Elastic Search Cluster.

I have a Jenkins server setup that can does the following:

- Takes the base repos an builds the Docker images for the apps
- Deploys a Kubernetes control cluster via EKS inside VPC.
- Deploys customer 1's applicaiton to that cluster.
- Deploy an Aurora DB cluster inside the same VPC for the application to uses.
- Deploys Elastic Search and Redis clusters for the app to use inside the VPC.

I can manage applications updates etc via the Jenkins job by passing in a new image version for the app etc..

Ok so that's all working, great! But now I have this situation:

I need to roll this out for 60 different customers. The larger customers will sit in their own VPC, having the entire cluster to themselves.

Smaller customers may share a VPC, i.e. share a DB/ES/Redis cluster just have different databases and indexes etc.

Each customer will need to have their infrastructure and/or application updated at some point, but not neccessarily all at the same time.

My question is how should I manage all of this? I really want a dashboard where I can see all of the clusters, all of the customers on those clusters, and the versions they are currently running for each of their applications and infrastructure.

Should I look to create this dashboard manually, should it be a job in Jenkins? Should I look at Gitlab or something similar?

How would you manage all of this? What tools should I learn to help me?

Thank you in advance! Good answers will be rewarded with Reddit gold!

https://redd.it/ydvoaf
@r_devops

I have got 2 offers, one with better pay, worse stack, second better stack + more grow opportunities but less pay

Hey guys,

I have got 2 job offers where the first one is an insurance company, they offer better stack (Terraform, AWS, Python) with guaranteed AWS Certification that they sign you for ( "AWS Solutions Architect – Associate" ), there is even one day a month only just to study (paid),

The second one is a bank, they offer less preferable stack for me (internal cloud, Linux, NGINX, Tomcat, RedHat and Azure DevOps) but with much better pay, they also need on-call rotation 24/7 unfortunately, which would be once in 6 weeks,

Which one do you think I should pursue?

​

The one thing is that I accepted the bank offer already, which I can pull out from of course, but that might make me blacklisted.

https://redd.it/ydvxy5
@r_devops

Devops Niches?

Can I focus on a niche in devops that doesn't have on call duties?

I am an older programmer and currently coming back into the job market (out to help my spouse beat cancer) I have a background in both dev and sys admin.

Recently have discovered devops and find I relate a lot to the role. I am enjoying working with the cloud and recently achieved my SAA. But I haven't worked formally as a devops, but find myself really enjoying terraform and kub.

I want to be realistic and I may pivot to data eng just to keep my sanity working for a company. I tend to like a solitary role and that's why I am not keen on going straight back as a full stack dev. Standups and demdanding clients were never my joy of software dev. LOL.

Any advice?

Thanks!

https://redd.it/ye6pmc
@r_devops

What is the common practice for wrapping public helm charts?

Hi,

I am looking for some advice on the industry's common practice toward wrapping public helm charts.

Let's say for instance, that there is helm-chart-x that creates a service, ingress and gets me 90% of the way toward creating "an internal (to the biz) service", additionally hypothetically let's say I need to add some extra k8 resources such as an additional ingress for the remaining 10%.

Would the common practise be to make the public chart a subchart of a custom helm-chart for the internal-service or would folks recommend decoupling things and either having a separate helm-chart or simply kube manifests which configure the remaining 10%?

Intrigued to know which way folks would go, as I'd like to establish a bit of a pattern towards this.

https://redd.it/ye1aa4
@r_devops

Things that suck to implement in Python, but not in Golang?

I've been using Python for a long time now, ever since I discovered that C++ and Java were not terribly useful for devops type tooling many moons ago.

Python has done pretty much everything I ever needed it to, but recently I had to implement threading in a tool to cut down on run time. Wrapping my head around async.io was time consuming, and I ended up using concurrent.futures, which was only marginally less painful.

At the end of it, I was left with the impression that threading in Python was a very hackish affair, and the people that had been complaining about the GIL (global interpreter lock) for a long time had a really good point.

Are there any other areas that Golang really excels at vs Python for devops / SRE type use cases?

https://redd.it/ye8zr5
@r_devops

In the interview process. Take home test is iac based and asked to build multi tired application.

I'm applying for Jr roles.

One company asked me to build multi tired application but they said focus on terraform and gitops process. do they mean build the app and also the iac infrastructure to deploy it?? Or just a multi tired architecture capable of hosting a multi tired application?

https://redd.it/ydy5sm
@r_devops

Pants 2.14: Support for templating and deploying Helm charts

Pants 2.14 was just released, and includes supports for a new deploy goal, with an initial ruleset for Helm.

https://blog.pantsbuild.org/pants-2-14/#do-more-of-your-workflows-in-pants-with-the-experimental-deploy-goal-with-initial-support-for-helm

Pants will automatically:

1. Infer the Docker image dependencies of your Helm charts
2. Build and publish image dependencies
3. Post-process the Kubernetes manifests to use the published image names
4. Run the Kubernetes deployment resulting from the post-processing

Please check it out and let us know what you think!

https://redd.it/ye53su
@r_devops

azure - vm size for the cluster and for the node pools?

I'm creating a cluster for a client (yay employed!) but client wanted to use Azure. I had just light contact with Azure before but I'm willing to learn.

Client reserved a VM for 3 years for us. He wants a managed AKS installed on one of those VMs.

My question is: I'll create the AKS selecting the reserved VM; will I be stuck with a single node pool? As I understood when I create a node pool I have to state the size of the VM as well.

If I use the reserved VM (F8 instance) and create nodepools with smaller sets (a system node pool with the bare minimum, another node pool for an application), will I be using (and paying for) several VMs? If so I'll have to stick with 1 node pool and manage replication on the containers/replica-sets?

Or it will create VMs on that reserved bigger VM and I'll not be paying extra?

For example, F8 has 8vCPU and 16GB RAM, which means I can "create" 4 node pools DS2 with 2vCPU+4GB RAM?

I'm reading the https://learn.microsoft.com/en-us/azure/aks/use-multiple-node-pools docs but nowhere it solves my doubt.

Thank you!

https://redd.it/ye2y2i
@r_devops

Unable to install Virtual Box on my macOS Monterey (AppeM1 Pro chip)

Is there a different hypervisor that's compatible with my machine? I'm new to Mac's and Virtual Box is the only hypervisor that I've used so far to configure vms.

https://redd.it/ye1id9
@r_devops

With all these DevOps tools, what’s the use case for using Python?

Is using Python still a necessity? I know a lot of these tools use Python under the hood, but do you have to know it?

https://redd.it/ydt1su
@r_devops

How did the Solarwinds attack and log4j affect your work as a Junior Admin?

title asks it all

https://redd.it/yehina
@r_devops

What happens when a node with local persistent volume goes offline in K8s?

From what I've understood about local PVs, they are allocated on a single node itself. What happens when this node goes down? Is there any way to set up replicas (or something similar) of a local PV on different nodes?

Note: I'm referring to multi-master multi-node cluster.

https://redd.it/yeizu5
@r_devops

Nginx ingress controller: how to insert whitelist-source-range annotations globally, but conditionally at the same time?

Hi Guys,

I wonder if that's even possible ;)

We have one ingress controller resource which monitors multiple Ingress resources in our k8s cluster.

Let's say \~40 Ingress resources. Some of them have whitelist-source-range defined to allow access only from selected IP ranges. Some of these Ingress resources are publicly opened.

Now, this mechanism works perfectly fine. Nothing wrong about it, except we need to maintain, monitor and track correctness of white-source IP ranges in number of Ingresses.

Therefore, we were thinking if there is a way to conditionally inject the whitelist-source annotation into selected Ingress basing on URL e.g. if ingress URL contains magic text, say ".notpublic.", then insert white-source annotation.

Another way would be to have two ingress controllers: open for public access, and second one for private access (with whitelist-source-range annotation).

Appreciate you help and input.

https://redd.it/yekoi3
@r_devops

Devops or cloud engineering

Hi everyone,


I am about to finish my master's degree in IT in the next few months and was thinking of getting into devops or cloud engineering. I know most of you say there is no such thing as junior or entry devops roles, but I am moving to Melbourne or Sydney after graduation and there are some job posts there for junior or entry level devops engineers.


I have 4 years of work experience in mechanical engineering, and no IT work experience although I have done some tinkering with HTML, CSS and JS in the past.


I have got 4 months left till I graduate and would like to gain some solid skills in that period by self-studying.


Which career path might be more suitable for me?

https://redd.it/yemrud
@r_devops

What's the best way to update/manage a self-hosted k8s/k3s cluster?

My aim is to make my home server and deployments more accessible externally and also add in some more comprehensive management tools for a better "more professionally maintained" environment.

I have a simple k3s cluster running and was wanting to add in TF so I can store and use secrets from my vault instance and possibly make some TF modules for deploying new apps which would create and update Cloudflare DNS entries/cert-manager definition/nginix ingress/(authelia config maby too)

I do have azure arc connected so I could potentially have a GHA connect via that proxy, or I could try expose the cluster externally and connect it to GHA somehow, or use kubectl proxy connected to a subdomain e.g cluster.domain.com then whitelist certain IPs for access or something.
I've also been researching Crossplane which looks like it could be fun and powerful to help manage the cluster instead of TF for applying changes, but that may be over complicated at this point.

Yes, this is more complicated than just managing some yamls and using a VPN like I do now, but was wanting something more comprehensive for learning purposes as well.

https://redd.it/yekmon
@r_devops

open-appsec Machine Learning-based WAF open-source code is now published on GitHub

open-appsec is a new open-source initiative that builds on machine learning to provide enterprise web application and API security with the visibility, protection and manageability that is required by modern workloads.

The engine is powered by two machine learning models:

A supervised model that was trained offline and fed with millions of requests, both malicious and benign.
An unsupervised online model that is being built in real time in the protected environment. The online model is updated constantly based on inbound network traffic.

Since it's not based on signatures, it proved to be effective against zero-days attack such as Log4Shell, Spring4Shell and the recent Text4Shell - blocking them using the default settings and with no update.

We are glad to update that code of open-appsec is now fully available in **GitHub**. Thank you for those of who participated in the early review and provided comments.

See more details in this blog https://www.openappsec.io/post/open-source-code-is-now-published-for-open-appsec-machine-learning-based-waf

The project is still in beta and team is eager to get your feedback about the product and the code. Please use the community page at https://openappsec.io/community

https://redd.it/yenjff
@r_devops

Is FANG DevOps all about coding

So I have been contacted by a few FANG companies recruiters offering to participate in interviews for being recruited into DevOps/SRE positions. All of them include multiple coding exercises to proove you're worthy of their money showers. From what I am told they do a lot of in-house coding to replace well known monitoring, configuration management software because it is not capable to handle their insane size of infrastructure. So basically it is much more of writing code than using usual infra management activities with open-source software solutions used globally I am more used to. I myself come from infra background with little actual development experience so I am not even sure why I draw their attention. Also having experience working in Enterprise and large infra using well known tools I find it strange that FANG find those tools incapable. Are the DevOps working in FANG all geniuses or is it because coming from Developer background they don't know much on how to properly configure those tools for large scale? Maybe there's another reason I am not aware about?

https://redd.it/yevhaw
@r_devops

Has anyone had luck getting a four-day workweek?

Currently, despite working in a FAANG (or perhaps because of that), I get several jobs offers every week. The range salary is most of the time lower and sometimes much lower than the one I have currently, which is kind of expected I suppose.

The thing is, I'm kind of burned out from the long work days and stress, so I'm looking for options, even if they pay less. Still, I have the feeling that just changing to another company will not fix it, best case scenario I will work fewer hours (maybe), for less money.

A four-day workweek, however, I think it will make a big difference. So when contacted, usually after is clear that the salary range they offer is below my expectations, I offer the option of a four-day workweek with noticeably reduced salary expectations. So far, despite the urge, some of them have to find someone, no luck.

For the record, I'm in Europe, and in my country, they are companies using this system, and even the government funded some trials about it, so it is not something crazy to ask, especially in a field so in demand.

Could anyone get this kind of arrangement? Is really that difficult to find a position that allows it?

https://redd.it/yezm14
@r_devops