How to effectively monitor and respond to events in Openshift?

Hi

I got challenge to project dashboards or another solution where support the support team will see when there is a problem with application and where will be some place where in where in case of failure immediately they can see point of failure and incident response path. In small company with traditional java apps we can use nagios and check status of servers and load, ram, cpu and is java process. But now it will be something different that will be around 60 namespaces with around 30 microservices in one namespace. So I think that integration with nagios isn't good because this is changing enviroment and there will be many false postives.

We thought about Grafana. One main dashboard with all namespaces and there will be some value that will determine whether the namespace is working properly and then when we click on this tile with the name of namespace then we will see all tiles with microservices in this namespace and then when we again click on tile with name of microserivice some information what happens with each microservice (ram, cpu, requirements etc.). But that make problem that we will have a lot of dashboards in Grafana, a lot of clicking, a lot of queries to influxdb and prometheus and I'm not sure that isn't it too aggravating. For example 60 namespaces with 30 microservices it is 1800+dashboards in Grafana only for applications in private cloud. So maybe its always go into openshift and see whats going on. I don't know. I don't have too much expirience with this system. Maybe someone know how big companies monitor microservices and how they respond to failures? They have kibana with elasticsearch too.

https://redd.it/en8m29
@r_devops

I've tried everything, but I still can't prevent servers from kicking me out of SSH session

I have tried to login via OpenSSH (windows 10 peasant). Got inside my ec2. Then kicked out after 2-4 minutes. I've edited various KeepAlive settings into every ssh\_config file I could find on my machine. Didn't work.

I tried using mobaXterm. Kicked out. I edited the SSH to "keep alive" on the settings. Still kicked out. I scrounged the web for mobaXterm timeouts even after enabling the keepalive and what i've found is to connect via SCP. Didn't work.

I am really at the end of my wits here guys. Neither hard configuration on the native OpenSSH client, or GUI configs through clients like mobaXterm work. I know I can config the keepalive in the server, but A) too much of a linux newb to pull it off in the 2 minute time window the SSH gives me B) it's bad practice anyways.

What should I do?

P.S: I am being kicked out from ec2 AWS and DO droplets if that matters. I don't use passwords - only private keys.

https://redd.it/en995k
@r_devops

What is AWS CodeStar in AWS DevOps?

As per my understanding, it is a cloud-based totally service for growing, handling, and operating with software improvement initiatives on **AWS**. You may speedy increase, construct, and deploy applications on **AWS** with an **AWS** CodeStar undertaking. An **AWS** CodeStar assignment creates and integrates **AWS** services for your undertaking development toolchain.

**Please share your understanding in the comments below?**

https://redd.it/en45h3
@r_devops

What better DevOps tooling would you like to have?

Hi, I am one of the founders of Commandeer. We are currently building out our IAC tooling. It let's you run command line services like Ansible, Serverless, Docker and Terraform from a desktop app. On the left hand-side is your playbook.yml, serverless.yml etc, and on the right is the conole output. I'd love to get some input as to what tooling you would love to see built in as we are tackling this right now. We are planning on integrating [Circle.ci](https://Circle.ci), Azure Pipelines, and Jenkins as well. The tools let's you deploy to AWS or to LocalStack. [Ansible Runner](https://imgur.com/gallery/wFjw9ow) [Serverless Runner - GIF](https://imgur.com/gallery/XkknoJL) [LocalStack Dashboard](https://imgur.com/gallery/Z8wxxe2) Thanks in advance!

https://redd.it/ene603
@r_devops

Need Advice: Buildtime vs Runtime Debate

I need advice on buildtime vs runtime config management. We are configuring our Bamboo Agents as code with Packer and Ansible to build an AMI image. I have a developer who has written several bash scripts which essentially look at aws tag metadata and update bamboo config (such as what bamboo server it points to, pulling for changes to a bash script repo and pulling latest versions). So he puts some of these scripts on the image at build time which allows for runtime updates/changes. However there is some conflicting ansible code that installs the agent that needs a build time variable defined to point to a certain bamboo server. If we continue down that path it would lead to 1 AMI per enhancement. So my question is, what's best practice here? 1 AMI for any environment. This doesnt allow us to do local test/development of our ansible code. Or AMI per environment? That would require building several more bash scripts and smart logic there. I personally lean to 1 AMI per environment because I want an immutable image and not have tons of bash scripts but I also see the argument for 1 AMI.

https://redd.it/enfw2o
@r_devops

Looking for a new CI/CD platform with UI similar to Concourse.

I have been using the tool [concourse](https://concourse-ci.org/) for a few years and I am now exploring new job opportunities to start fresh with a brand new CI/CD platform. One thing I really love about concourse is its UI ([here's an example from their website)](https://concourse-ci.org/examples.html).

* It's very visual and easy to dive into individual resources, jobs, tasks, tabs, etc.
* It shows what resrouces and jobs connect to other resources and jobs.
* The url's are all human-readable and can very easily be inserted into scripts without having to lookup complex UUIDs.

I've been trying out some other SAAS CI/CD tools like Gitlab and CircleCI, but their dashboards are not quite what I'm used to (at least from what I have found so far). What are some CI/CD tools in widespread use today that have a visual map showing pipelines?

Edit: I'm also looking for one that makes it easy to create multiple pipelines from the same repository.

https://redd.it/eneu3u
@r_devops

How long it usually takes to build a CI/CD

Hi,
I am wondering on average how long it usually take you to build an automated infrastructure and CI/CD For your application regardless of the technologies / tools used (AWS, Terraform, Ansible, Packer, Kubernetes, Jenkins, Cloud Based CI)... Etc

https://redd.it/endipd
@r_devops

Auto-registering K3OS to Rancher help...

Hey all! Looking for some help with a cloud-init script I'm working with. The end goal is to have a K3OS cluster that self-registers to a Rancher instance on boot. I've got the basic framework spun up, but I'm having trouble figuring out the `cURL` script. The idea is that the `k3s-init.yaml` script will:


* curl `call-rancher.yaml`
* put them into k3s `/manifest` folder
* automatically apply all manifests in the `/manifest` directory on start

Secondly, the `call-rancher.yaml` script is supposed to call a remote rancher and do the token dance to auth with Rancher.

Here are the gists:

[https://gist.github.com/jdbohrman/3db315b0a9273f67699d771bc59ccec6](https://gist.github.com/jdbohrman/3db315b0a9273f67699d771bc59ccec6)


[https://gist.github.com/jdbohrman/3b56e69612fb59b722c6534d0820a879](https://gist.github.com/jdbohrman/3b56e69612fb59b722c6534d0820a879)

https://redd.it/en41qa
@r_devops

Salary Sharing Thread

* Company/Industry:
* Title:
* Location:
* Years of Exp:
* Salary + Overtime:
* Relocation/Signing Bonus:
* Stock and/or recurring bonuses:
* Total comp:

https://redd.it/enj3wo
@r_devops

Windows build server recommendations

Hi. I'm a web dev primarily but I'm working on a large C++ project at home and would like some recommendations for setting up a build server.

My personal PC doesn't have great specs, compiling the project takes a bit time (1 hour+) and my PC isn't really usable during the process. So I was thinking I might set up a basic EC2 instance to use to build the application and Jenkins to execute a pipeline for this (downloading/compiling/storing the artifacts).


I would rather not spend much money on this as it is for a hobby project, would a basic EC2 instance be able to handle this? Anything else I should consider? Is Jenkins a good choice or any better suggestions maybe?


Thanks.

https://redd.it/enj1l7
@r_devops

What do you do as a DevOps Engineer? Are you paid overtime for being on call or when you are deploying IaC?

What do you do as a DevOps Engineer? Are you paid overtime for being on call or when you are deploying IaC?

https://redd.it/enj05c
@r_devops

I made go library to pull AWS product/feature announcements

Hope some of you can make use of it!

https://github.com/circa10a/go-aws-news

https://redd.it/eng961
@r_devops

This connection is Not Private

I recently signed with pre-made Wordpress droplet on DigitalOcean. I set everything up with www when it prompted me.

But now when I try to load up the website I am getting an error. I tried typing the name with and without www

I am using letsencrypt SSL for which I used the option with www

​

Any ideas how I can solve this? Could it be because Nameserver has not propagated completely yet? Do I need to do some URL redirect? Any feedback on this is appreciated.

Thank you

https://redd.it/enirnu
@r_devops

Maybe You Don't Need Kubernetes

As 2019 has gone by, [this article](https://endler.dev/2019/maybe-you-dont-need-kubernetes/) is one that stands out when i think back. I see lots of posts here from people wanting to learn the new buzz, but IMHO, a far more valuable trait is the ability to find the right tool for the job.

This becomes increasing important if you work (or want to work) at a small/medium sized company where you truly have the potential to influence the company's outcome. Chances are, that in a larger organization, you might be more valuable for being nose deep in k8s - and if you enjoy this - good for you!

The takeaway from the article is more or less: don't use Kubernetes just because everybody else does. Carefully evaluate your requirements and check which tool fits the bill.

https://redd.it/enphaw
@r_devops

Which US State or City in your experience offers the best for an overall DevOps experience, in terms of salary, quality of living, healthy balance of work/life, job satisfaction, and the number of available jobs? Why?

I'm assuming someone who makes 100K in Silicon Valley doesn't have the same quality of life as someone who makes 100K in Texas.

https://redd.it/enrc9g
@r_devops

Created an Octopus Deploy PowerShell module

Still in progress (just started it an hour ago or so). I've been working a lot with Octopus and saw a ton of great REST API examples, but no modules to interact with it. This is going to be a fun project :).

[https://github.com/AdminTurnedDevOps/OctopusDeploy-PowerShell](https://github.com/AdminTurnedDevOps/OctopusDeploy-PowerShell)

https://redd.it/enqcv6
@r_devops

What to use for CD in 2020?

At my previous job, we had a **Jenkins** cluster that used to build (CI) and continuously deploy to all the different environments. However, there was a lot of effort that went to continuously scale and manage Jenkins. Managing pipelines via Jenkinfiles was also mostly copy-paste and a lot of the server configuration could not be managed as infrastructure-as-code.

At my current job, we are using Github actions (for CI) to build and create docker images that are then *manually deployed* to our **Kubernetes** cluster. I want to enable continuous deployment to remove the friction between code push and rolling it out to different environments. Our setup includes **Vault** for secrets and the CD solution should be able to fetch the secrets for different environments to then deploy the app to Kubernetes. For this obviously, it will need the right privileges and we are not comfortable an external system having such access to such critical parts of our infrastructure. (Are the concerns invalid?)

It's 2020 and I feel that there has to be a better way of doing things than going the self-hosted Jenkins way especially if it could be infrastructure-as-code. I am looking at [https://landscape.cncf.io/category=continuous-integration-delivery&format=card-mode&grouping=category](https://landscape.cncf.io/category=continuous-integration-delivery&format=card-mode&grouping=category) but not sure which is the right tool.

Any advice, suggestion is welcome.

https://redd.it/enpacv
@r_devops

Our field deserves more than pissant low-payed contracts and I'm tired of it. Maybe we should unionize.

How the hell are we, as DevOps Engineers, with our experience and our knowledge, have no choice than to be pulled more and more into 6ish month (maybe +) contracts (aka contract to fire) that can end in a heartbeat and both don't offer benefits nor any job security?

I have been told by recruiters that this "contract" model is the new trend and seen it on the ground that it's taking over our field.

Why do we have this experience and don't get so much as a sick day or even the basic benefits of health insurance? Wages are way down when I have to seek some way paying for rent myself.

I am disgusted by this new model that takes us for granted and I feel like I'm nearing retirement at 37 with the insanity of it all. Requirements are up, competition is up, but wages and everything else are down. Benefits are absolutely zero when contractors are the second-class of DevOps Engineers.

Yet somehow we are still in-demand.

This is supposed to be an in-demand field but from where I'm at in the Bay Area it's a goddamn race to the bottom on the job market. The Bay should be an icon of success for its workers, not a dumpster fire of workers' rights.

We get nothing here nearly all the time.

I've always been loathe to the unions in other industries because they've never helped me. That being said, I think we need to rise up against this race to the bottom and I want some real organization to combat this contract dystopia.

I am tired of being exploited and wrangled around and shit on and I want a better life for everyone here and in this field.

Thank you for reading my rant.

https://redd.it/enm9ae
@r_devops

Starting a DevOps Consulting Firm

Has anyone tried starting a small consulting firm specializing in DevOps? Can you share you success/failure stories and maybe provide any tips for me?

https://redd.it/enlqgl
@r_devops

Ansible Alternatives?

We're looking for fairly basic remote command execution (update config files on many machines etc.) for a few dozen servers (too many servers to deal with manually but not huge). Ansible looks straightforward, but it doesn't support 2FA thru a jump box with keyboard-interactive (Google Authenticator). They have no plans to support this either. Any other alternatives?

https://redd.it/enydjx
@r_devops