[Question] Running SaaS agent on K8s cluster

I'm working on a SaaS that helps K8s users with observability and management. It also provides correlated view across clusters. For that purpose there is a need for some agents to be running within the cluster and reporting metrics to the SaaS backend. The rest of the things is accessible through web site once the data reported to the SaaS service. The product uses read-only access to non sensitive cluster APIs.

I want to remove barriers for entry and simplify the use as much as possible. If you're a Kubernetes I'd appreciate your feedback on how would you like to run such SaaS service (assuming you know what it is and need it).

Have few options in mind. Please feel free to suggest anything beyond those:

1. SaaS web site uses a wizard and asks if you're a public cloud user. If in AWS/GCP/Azure will show instructions to create service account. The SaaS will then connect to client's account, get list of managed K8s cluster. User selects a cluster and the SaaS deploys necessary agents on the cluster using the namespace of the choice. For self managed k8s clusters, asks for ip address/credentials and does the same thing. No direct management, configuration and deployment is needed.
2. User deploys a helm chart which contains an operator that handles agent deployment and version updates. It has write access to its own namespace only. Deploy the operator once, the rest is automated for life.
3. User deploys, upgrades and configures agent manually. Can be involved.

https://redd.it/g1yfvs
@r_devops

Jenkins and running Projects/Builds as another user?

Hi everyone,

Quick question here. So, I built out our team's Jenkins instance about a year ago. We have an Ent Git that I use to push PowerShell scripts to and have Jenkins jobs that pull and run the scripts in Ent Git's remote repo. Simple enough and everything works great! But...

When I setup Jenkins, I have the service currently running with the "Log On" as my AD \_adm account that has Account Operator. I recently created a service account (srv\_jenkins) to have as the account in the Jenkins service "Log On". Worked just fine. I could log in to Jenkins web UI just fine. But wait... I go to run a job, specifically this job, like many of the ones, query AD/DNS/DHCP/other directory services or modify AD Objects. Well the job failed. I'm assuming that because the jobs run as the user account in the "Log On" tab for the Jenkins service? Is this correct?

If so, what is the work around, if I don't want my AD \_adm account to be running the Jenkins service, but need the jobs that would require Account Operator/Domain Admin to run the PowerShell scripts to run? Thanks!

https://redd.it/g1xs3b
@r_devops

One Size Fits None

As somebody who has been involved with many projects, included some "One Size Fits All" pipelines. I thought I'd document some problems I've ran into, as well as some things that help this transition. Hopefully this helps at least one of you in the same boat.

https://medium.com/devops-dudes/one-size-fits-none-22cf9a1725ba

https://redd.it/g1w1bo
@r_devops

Workflow for Kubernetes DevOps

[Workflow for Kubernetes DevOps that can evolve.](https://medium.com/@imarunrk/workflow-for-kubernetes-devops-15f0dbb560ff)

The developers, application/cluster operators, architects, and security team wants to contribute to the Kubernetes YAML continuously to keep the Infrastructure matching to the evolving organization strategy and policy, which demands a workflow. [Read More](https://medium.com/@imarunrk/workflow-for-kubernetes-devops-15f0dbb560ff)

https://redd.it/g229e4
@r_devops

What's the difference between Configuration Management tool and a Provisioning tool?

I hear this frequently, but can't find a decent definition. For instance, "Puppet is more of a configuration management tool, whereas Ansible is more of a provisioning, configuration and deployment tool. "

Redhat says: "Provisioning is not the same thing as configuration", and then in the next paragraph: " Server provisioning includes setting up the physical hardware in a datacenter, installing and configuring software, ..."

https://redd.it/g20rv3
@r_devops

Cloud agnostic vs using specific

Do you strive to build and deploy your apps as cloud agnostic? Or do you see value in using a specific cloud provider and leverage benefits of their managed services/unique offerings or neat integrations (*e.g. using AWS cloudformation, SAM, lambda, fargate dynamoDB etc.*)?

by cloud agnostic I mean doing everything across your software stack to be able to move the apps to other cloud provider e.g. - using abstractions in your software or configurations not to directly use specific cloud service, using terraform for IaC, serverless framework, kubernetes for orchestration, anthos? etc.

**what informs your decision to go cloud agnostic or cloud specific?**

**which technologies/tools/ frameworks promote cloud agnostic/portable cloud native engineering?**

​

Personally, I see no clear answer, but rather take it as a project specific decision

Cost of cloud agnostic is e.g. increased operational cost, infrastructure cost, loss of speed by not using cloud provider specific innovation, reduced engineering speed for a need of maintaining layers in between your software and cloud providers

A motivation for cloud agnostic can be though: need to move workcloud across clouds, avoid vendor lock-in / compliance to enterprise strategy , legal / customer requirements

Happy to hear your thoughts!

https://redd.it/g21kga
@r_devops

Why does sorting by New not work properly in this sub?

When I sort by New in this sub only, I get a reverse sort starting about 14 hours ago. Scrolling down I see newer posts up until about 2 hours ago. After that, it starts again at about 11 hours, down to 2 or 3. This pattern repeats a few times before 1d old posts appear, and then a few 2d old posts, then some more 1d old posts.

Most other subs I'm on (haven't checked them all) behave as one would expect.

https://redd.it/g21wr8
@r_devops

Rsyslog and Journald - Log processing in Linux

Hey there,

I do not really understand the forwarding from Journald to Rsyslog.

Basically I understood it in the way that the 'pipeline' is built up as follows:

>Kernel logs through printk() → /proc/kmesg ← rsyslog → writes to log file according to rules in rsyslog.conf
Userspace logs → /dev/log ← rsyslog → writes to log file according to rules in rsyslog.conf


This results in the various log files, such as /var/log/syslog etc.

As in the rsyslog.conf snippet, messages from the facility 'user' are also written to syslog and are therefore stored twice, ***right?***

# # First some standard log files.
Log by facility.
#
auth,authpriv.* /var/log/auth.log
*.*;auth,authpriv.none -/var/log/syslog
#cron.* /var/log/cron.log
daemon.* -/var/log/daemon.log
kern.* -/var/log/kern.log
lpr.* -/var/log/lpr.log
mail.* -/var/log/mail.log
user.* -/var/log/user.log

So far so clear, if I've got it right.

Journald forwards the logs to syslog (standard in Debian according to [https://manpages.debian.org/testing/manpages-de/journald.conf.5.de.html](https://manpages.debian.org/testing/manpages-de/journald.conf.5.de.html)).

If both rsyslog and journald read from /dev/log (or the link and socket provided for syslog) and journald on Debian has ForwardToSyslog enabled by default, ***why don't all messages duplicate?***

I thank you in advance and hope that you can help me.

https://redd.it/g20mte
@r_devops

A CloudNative conversation

I’ve been working on an always evolving area of systems and infrastructure engineering over the last decade, which started for me as Automating Everything possible to doing DevOps to Everything as Code and now doing CloudNative stuff. I’ve had many conversations with myself and others on this path and here are those voices in form of many frequently asked practical questions and their answers here [https://medium.com/@ankur.floss/nativity-in-the-cloud-f14d9b197306](https://medium.com/@ankur.floss/nativity-in-the-cloud-f14d9b197306).

https://redd.it/g235as
@r_devops

Gotopus: a minimalistic tool that runs arbitrary commands concurrently

# [Gotopus](https://github.com/lherman-cs/gotopus)

Github: [https://github.com/lherman-cs/gotopus](https://github.com/lherman-cs/gotopus)

You define your commands with their dependencies and Gotopus will take care of the rest, running them concurrently when possible.

# Features

* \[X\] Concurrently run steps, speeding up running time
* \[X\] Local or remote configs
* \[X\] Easy to install
* \[X\] Circular dependency detection
* \[X\] Clean step definition with [YAML](https://en.wikipedia.org/wiki/YAML)
* \[X\] [Builtin and user environment variables](https://github.com/lherman-cs/gotopus/blob/master/README.md#environment-variables)

# Installation

curl -sf https://gobinaries.com/lherman-cs/gotopus | sh

# Basic Usage

Usage: gotopus <url or filepath> ...

-max_workers uint
limits the number of workers that can run concurrently (default 0 or limitless)

examples/basic.yaml:

jobs:
job1:
steps:
- run: sleep 1 && echo "job1"
job2:
needs:
- job1
steps:
- run: echo "job2"
job3:
steps:
- run: echo "job3"

To use `basic.yaml` above, you can run the following command:

gotopus basic.yaml

Or you can simply give a URL to this file:

gotopus https://raw.githubusercontent.com/lherman-cs/gotopus/master/examples/basic.yaml

https://redd.it/g25w8i
@r_devops

How do you automate configuration of PaloAlto Networks infrastructure?

I'm fairly new to devops, working in it for about 2 years now. I've become accustomed to build/automation tools such as docker, packer, terraform, gitlab, and a lot of scripting, mostly with GCP. This has been sufficient so far, in automating much of the images and infrastructure we use to deploy our software.

I was just introduced to configuration of a variety of elements within our paloalto networks (PAN) system (which appears to be hosted by one of our teams - not SaaS). To my horror, everything was performed manually via a web UI, by someone highly trained to do so. To make things worse, this person then went into GCP and hand-manicured a number backends and external load balancers, duplicating the effort we have terraformed in our internal load balancers, but pointing the LBs at the PAN.

Aside from the GCP manual editing, is anyone familiar with IAC for PAN, and how have you done it?

https://redd.it/g1zooy
@r_devops

What is a normal monthly cost for Azure CICD?

We only have a team of 3 devs for frontend (I'm one of them), and make around 2-3 PRs a day. Frontend + backend with QA, Staging, Production, etc, there are a total of 15 containers.

I was shocked to find out the cost for such a small company is in the mid 4 digits. Is this normal pricing for Azure's CICD service? I would have thought it'd be in the hundreds at most.

https://redd.it/g1wg15
@r_devops

ServiceNow DevOps, CMDB and Self Service Portals

1. Has anyone seen ServiceNow’s DevOps product? Looking for some reviews. Change Automation sounds like a compelling story. Not sure if it can also help with dependency identification.

2. Are CMDBs still a thing? How do you do dependency identification in change management without a CMDB?

3. How do you all utilize self service portals? Check-ins kick off builds is something I understand, but for a new app, where do you specify which build to kick-off (Gold/Silver/Bronze servers) etc.?

4. How do you like ServiceNow’s story around the need for a single dashboard showing the flow of a ticket from conception (idea) to release?

https://redd.it/g1pyxv
@r_devops

Ask r/Devops - Basic help with Jenkins&Kubectl

I am trying to get a very basic Jenkins-GKE CI/CD pipeline going with GitHub integration. Push change to GitHub, webhook to Jenkins (ubuntu VM) server running in GCloud, build Docker image and push to docker hub, update deployment in GKE with kubectl with "set image".

My problem is that Jenkins is running with the user "jenkins" and it always fails at the kubectl part. It gets the notification from GitHub, builds the new docker image, pushes it to Docker Hub but can't access my kubernetes cluster with kubectl.

The error:

`+ kubectl set image deployment/hellowhale hellowhale=timorwell/hellowhale:11 error: Missing or incomplete configuration info. Please point to an existing, complete config file:`
`1. Via the command-line flag --kubeconfig`
`2. Via the KUBECONFIG environment variable`
`3. In your home directory as ~/.kube/config`

I spun up the cluster on GKE with the gcloud-cli with my regular user on the VM, so that user can access the cluster no problem. I copied the .kube/config gile to /home/jenkins/.kube/config but it still doesn't see it. I chown-ed it to the jenkins user too.

Why can't it use my config file? Am I approaching this problem from the wrong place? Is there a better way to integrate kubectl into my Jenkins pipeline?

I just tried adding the `--kubeconfig` flag pointing to my config file but that results in "must be logged in to the server" error...

I'm trying to learn all this rn, thanks for the help. And yeah I know this isn't stackoverflow but I just like reddit more due to it being more "human" lol.

https://redd.it/g1vioz
@r_devops

DevOps test



Programming Pre-Screen

Conference room scheduling. Find the nearest open conference room for a team in which a team can hold its meeting. Given n team members with the floor on which they work and the time they want to meet, and a list of conference rooms identified by their floor and room number as a decimal number, maximum number of people it fits and pairs of times they are open - find the best place for the team to have their meeting. If there is more than one room available that fits the team at the chosen time then the best place is on the floor the closest to where the team works.

E.g.

rooms.txt

7.11,8,9:00,9:15,14:30,15:00

8.23,6,10:00,11:00,14:00,15:00

8.43,7,11:30,12:30,17:00,17:30

9.511,9,9:30,10:30,12:00,12:15,15:15,16:15

9.527,4,9:00,11:00,14:00,16:00

9.547,8,10;30,11:30,13:30,15:30,16:30,17:30

Input:

5,8,10:30,11:30 # 5 team members, located on the 8th floor, meeting time 10:30 - 11:30

Output:

9.547

Please explain: how you solved the problem and how it would behave based on the different parameters (number of team members, longer meeting times, many rooms with random booking times). How would you test the program to ensure it always produced the correct results?

For extra credit, can you improve the solution to split the meeting across more than one room if say only one room is available for a fraction of the meeting and another room is free later to hold the remainder of the meeting during the set time. If you want to make this more powerful - assume that the number of room splits can happen in proportion to the length of the meeting so that say if a meeting is 8 hrs long then the algorithm could schedule it across say up to 4 rooms if a single room was not available for the whole time



You may code the response in any DevOps programming language you like

• Bash

• Perl

• Python

https://redd.it/g24epo
@r_devops

Jenkins freestyle job ansible pluging sudo issues

So I'm running a playbook in linux based jenkins using the ansible plugin. For linux to linux, it works just fine. However, with windows I'm running into an issue where it keeps trying to do ssh instead of winrm I've narrowed it down to the fact that when jenkins runs "ansible-playbook" it's not issuing a "sudo". I tested it in the cli directly. If I run ansible-playbook without sudo, it attempts to connect as ssh. When I run it with sudo, it will attempt as winrm. Same behavior as jenkins.

&#x200B;

So my question is, how do I get the ansible plugin to issue a sudo to ansible-playbook? There is the sudo check box in the plugin, but that is deprecated for version 2.6 and up. There is become but based on the documentation, that's switching user context on the target, not elevating ansible-playbook to sudo status.

I can always make a shell command but doing it via plugin would be nice.

https://redd.it/g1sc0l
@r_devops

How to analyse logs from docker in my dev environment?

Hey guys. Been feeling quite a bit of pain with log analysis from distributed web applications.


I'm trying to find a way to pipe my docker logs to a tool that would allow me to at least highlight words, maybe filter lines based on regex. Ideally would aggregate multiple log streams based on session ids and stuff to reconstruct a timeline.

Any ideas?

Thank you!

https://redd.it/g1s85p
@r_devops

How much are you paying for datadog compared to your total AWS/cloud costs?



https://redd.it/g1n9a3
@r_devops

What hasn't been Dockerized?

During the quarantine I have been using my time to learn Docker, working towards my certification. But I also want to show some examples on my GitHub of me taking existing applications and building images. I started searching through various projects but almost every repo I looked at already supported Docker deployment. Anyone have tips of finding apps that haven't been Dockerized yet?

&#x200B;

Cheers

https://redd.it/g2fnru
@r_devops

Using github actions - should I have separate workflows for CI and CD?

I'm a bit confused as I've seen so many conflicting examples of github actions workflows where in one workflow the example shows **testing, build & deploying** all in one and others show **testing & build** as one and **deploy** as separate one.

 

My understanding is that:

* CI: You continuously merge your code into master, running a build and test job on it to make sure everything is ok before merging.

* CD: You continuously deploy your app.

 

Therefore, should I have one workflow called `continuous-integration-workflow.yml` and another called `continuous-deployment-workflow.yml` or have it all in one single file?

https://redd.it/g1pkch
@r_devops