Using Litmus Chaos Engine on Kubernetes with Autonomous Monitoring for Verification

This [Github repository](https://github.com/zebrium/zebrium-kubernetes-demo) installs a demo app environment with multiple services on a Kubernetes cluster and runs a series of chaos experiments on the cluster. It then uses Autonomous Monitoring to automatically detect the issues caused by the chaos experiments. It's pretty cool!

https://redd.it/fhihxl
@r_devops

Kubernetes Cluster API updated to v1alpha3 and now supports launching clusters on 7 platforms

Cluster API allows you to bootstrap and manage clusters from Kubernetes itself. It can be seen as an operator for Kubernetes clusters.

The quickstart has been updated for v1alpha3
https://cluster-api.sigs.k8s.io/user/quick-start.html

Cloud platforms
* AWS
* Azure
* GCP

On-prem
* vSphere
* OpenStack
* Bare-metal (metal3 via openstack ironic)

dev
* docker (think KIND - kubernetes inside docker style)

https://redd.it/fhkl0o
@r_devops

Twelve Dot One Factor Apps (Blog Post)

I have been following the Twelve-Factor App methodology for a considerable amount of years. I mean, just like the Pirate Code, I see them more as guidelines than rules. There Is a factor I’d like to challenge. Can tell you how and why in this blog post I have written just for you ;)

[Twelve Dot One Factor Apps](https://zemanel.eu/posts/2020/03/twelve-dot-one-factor-apps/)

https://redd.it/fhk4xz
@r_devops

How to add external cluster to spinnaker

I have two Kubernetes clusters, I have Spinnaker deployed to dev and want to be able to run pipelines on prod. I can't seem to figure out how to add another cluster, I see something in the helm chart values file about adding a secret but not exactly sure what it fully means and google has been less than helpful today.

​

How can I add an external kubernetes cluster to my Spinnaker setup which was installed via helm - no text editor in the halyard container to create files with

https://redd.it/fhklcs
@r_devops

Back-end engineer wanting to change to full time DevOPS engineer, need some advice.

Title says that all, I've been doing back-end engineering for two years. After some serious thinking. I've decided to dig to DevOPS but I have no Idea from where should i start. I have been using docker for a while, as well as cloud, so I am not a total beginner. I want to get sufficient knowledge to get a job. Even as a Junior DevOPS engineer would be great

so if you could give me some video courses/books/resources I would be very grateful.

Much appreciated

https://redd.it/fho8t9
@r_devops

Are there any Legitimate DevOps bootcamps?

Edit: I should have added, I am seeking in person bootcamps.

https://redd.it/fhtqxl
@r_devops

Advice needed on monitoring solution

Hi,

I am asking you for advice regarding monitoring solution that will fit our set of constraints. Our team doesn’t have a lot of Ops experience, and we have to choose monitoring software based on our limited experience and knowledge. That’s why I am asking you for help.

The problem:

We will have to monitor a couple of VMs (about 20, owned by us), and a couple of servers not owned by us (no cloud). What we would need to monitor (at least for now):

1. **VMs** (owned by us): disk, cpu, ram usage, server availability
2. **VMs** (owned by us): hdd mounts (our infra is quite bad, and our hdds keep unmounting) - from monitoring system we would like to know when hdds were unmounted to know where the processes have been interrupted
3. **VMs** (not owned by us): disk, cpu, ram usage + obtaining info from custom software by running shell commands and reading info from stdout.
Main problem I have here is that those servers are not owned by our team. We can ssh to them, run shell commands, but for 99% we won’t be able to install any agent on them. Unfortunately we have to monitor those servers because we are running some intensive computations on them.

4. **sFTP** (not owned by us): availability and disk usage
5. Last but no least we will have to monitor few web-services that are running in docker containers

I think that for our needs it would make sense to use metric based monitoring (but I might be wrong here).
For now our infrastructure is very static (VM IPs don’t change), however from time to time we have to migrate to new VMs. In the future we will move to OpenStack to have more flexibility (no cloud allowed for now). We're also aiming to have most of our infrastructure as a code, so it would be nice if the configuration of proposed solution supported that.

What are your suggestions for monitoring solution here (especially for point 3)?

https://redd.it/fhqsr0
@r_devops

For Fun: How many of you think your infra is bad if you're not getting alerts (devops recursion)

Instead of accepting the fact that you've done a good job and are looking for more options to ensure your confidence, as its important to be on top of it. devops recursion.

https://redd.it/fhtncl
@r_devops

North American company - weird D-Day plan?

I work for a company that is mainly a North American company. As a developer I work remote a lot.. like I never see my desk. Well I had a call and I was put in a rotation to be on site in my local office once a week during this crisis. I’m not in a hard hitting zone yet but we are gaining numbers fast. But I am just wondering if anyone else is getting this. I think it is more “if IT can do it so can you” type of thing.

https://redd.it/fhtg4g
@r_devops

Can I use a Public IP address to access a web server on a Private Subnet IF I'm accessing it from the public subnet that has access to it?



https://redd.it/fhnho0
@r_devops

Do you use anomaly detection/outlier detection feature in your monitoring tool?

Many monitoring tools like AWS Cloudwatch/ DataDog/ New Relic and others provide some sort of anomaly detection option? For example - Cloudwatch introduced [this](https://aws.amazon.com/blogs/aws/new-amazon-cloudwatch-anomaly-detection/) in October last year. DataDog has [outlier detection](https://www.datadoghq.com/blog/outlier-detection-algorithms-at-datadog/) algos which one can use.

Do you folks use these features? Why or why not?

If yes, what is the key improvement - does it improve the quality of alerts or reduces false positives?

https://redd.it/fhm2u7
@r_devops

What DevOps or SecOps content could I create that would actually be helpful?

I'm the developer advocate for a devops/cybersecurity company

It's my job to create content like blogs, videos etc that the community wants, it can be anything that is helpful to the community. I'm curious to know what kind of content you guys feel is missing in the community at the moment?
Any topics I could create information on that you would find particularly interesting or helpful?

https://redd.it/fhjnl3
@r_devops

embedded devops engineers

For all the embedded devops engineers out there.

how often do you meet with devops engineers embedded in different teams. Our idea is to eliminate where possible duplicity in solutions, having a standup every day feels like a lot considering I already have my project teams's standup. Any tips anyone can give on how to keep in sync with the engineers on different teams with the view to creating shared libraries of infrastructure code?

Thanks!

https://redd.it/fhg25b
@r_devops

I'm being offered an SRE position @ Microsoft tomorrow. Comp expectations?

​

Greetings,

​

I am currently employed as a fully remote "Sr. SRE" with a company of \~120 employees. My salary is 145k, some bonus, and decent benefits (100% of my medical, 50% of family, 401k but no match, etc.). This has been a great setup for me and my family as we bought a house in a LCOL area close to my family.

​

I've done quite a few interviews lately to see if I could increase my earnings or get into something exciting. After two rounds of interviews with Microsoft, I've been told I am receiving offer details tomorrow.

​

The original job posting was for "SRE II" . The job involves supporting internal systems involved with running and expanding Azure datacenters.

​

If I accept the job, it will require relocation to sit at the Redmond campus. My wife and I lived in Seattle for many years until recently and know and love the area. It's just really expensive, and traffic sucks. Microsoft's benefits are going to be way better. But strangely I have no idea what salary, signon, relocation assistance, equity etc they are going to offer.

​

I am going to be 36 next week and have been working most of the time since I graduated high school. I've slowly worked my way from tech support to IT to sysadmin to devops/SRE over the course of that time. No college degree.

​

Any insights would be appreciated. The relocation is only going to make sense if the total comp is significantly more than where I'm at.

https://redd.it/fhjdr0
@r_devops

What do you use to store events?

I'm part of an organization that does a lot of IT operations, much automated, some manually.

We get a lot of automated events; the two biggest sources are our monitoring system, and failed cron jobs.

What do you guys use to store and manage such events?

Currently we have a ticket system (OTRS) that can be used by email, but it is approaching its capacity limits, and it's simply not designed for this task.

I'm looking for:

* A way to make events visible to anybody interested
* Some sort of deduplication: if a cron job sends an email every 5 minutes, there should be a way to deal with them all at once
* Some sort of triaging: a way to mark an event or stream of event as not important / dealt with / whatever, and only give me a few of events that haven't been triaged
* Escalation: some events must trigger alerting to the appropriate teams

I'm sure we're not unique in needing a way to deal with this, but my searches haven't turned up a ready solution for this.

https://redd.it/fheoy3
@r_devops

Deploy Gitlab, Docker and Kubernetes

Hello everyone,
I am currently Building a CI/CD system which should consist of Gitlab, Docker and Kubernetes.

I startet with a Gitlab instanc in a Docker Container and wanted to intigrate Kubernetes next my Question now is „should i run Gitlab in a Container? and where is the best place to install Kubernetes (in the same Container?)?

If you need more Information of have trouble understanding my gibberish don‘t hesitate to ask! Thanks :)

https://redd.it/fhhg17
@r_devops

Using Prometheus and Grafana with synthetic monitoring

Hi,

We published a blog post on how you can use Prometheus and Grafana to graph and alert on the data coming from our synthetic monitoring solution. It's quite in-depth and practical. The writer John Arundel dives into how to connect and cross relate metrics and create practical SLA-type dashboards.

​

I'm very curious about how other Prometheus users are mixing "passive" metrics collection with "active" pinging and synthetic solutions.

Full disclaimer: I'm the founder of Checkly

[https://blog.checklyhq.com/monitoring-website-performance-with-checkly-prometheus-grafana/](https://blog.checklyhq.com/monitoring-website-performance-with-checkly-prometheus-grafana/)

https://redd.it/fhf9c9
@r_devops

What's the best approach?

I'm a recruiter (boo) working in the cloud/DevOps space providing candidates for clients in Germany.

I know just how much you guys get contacted about new positions, and probably a whole lot of them aren't even relevant to you.

I'm wondering when you've had really positive experiences with recruiters, how did they contact you in the first place? What did they say/write to you that caught your attention?

I'm a firm believer there is no "one size fits all" option as everyone has different motivators and different interests I'm just interested to hear how I can improve the experience for my candidates.

https://redd.it/fhcqmi
@r_devops

Great resources for anyone new to the world of DevOps/DevSecOps

Last post like this got me A LOT of Karma so I'm trying again 😄

This is a good list of blogs, Twitter accounts & podcats which are great for any newbies looking to learn more about the industry. Way more of the ones mentioned here have podcasts, although for some reason this list doesn't include them. I also suggest IT Revolution's blog & Networkworld

[https://www.portshift.io/blog/kubernetes-cloud-resources-devops/](https://www.portshift.io/blog/kubernetes-cloud-resources-devops/)

https://redd.it/fhesor
@r_devops

which json log handler are you using?

I am following [this](https://www.datadoghq.com/blog/python-logging-best-practices/#log-in-json-format) guide about python logging best practices and they recommended using a json handler for my loggers.

I am using a centralized logging solution (AWS CloudWatch) and this a solid piece of advice, however the json logger [library](https://github.com/madzak/python-json-logger) they recommended haven't been committed to since sep-19 and hasn't been released on PyPi since mar-19.

which one are you using and would recommend? or should I just copy some parts of the library and have it part of my source code ?

https://redd.it/fhd70m
@r_devops