Cost center versus profit center

You may heard of the cost center vs profit center issue where people in profit centers get paid and treated better than people in cost centers. People at profit centers make money for the company where as people in cost centers cost money for the company. I can't help to think that a typical devops team that works in silo is an cost center where as member in a feature team with a devops focus is in a profit center. In times of company down sizing, individuals in cost centers are usually the first to go

What are certain things that developers in a cost center do, that I should avoid if I want better career trajectory. Things that I can think of are : maintaining systems, sec ops, building infrastructure and automated tests. Don't get me wrong, these are good skills to have but it is very easy for the business to dismiss these as they cost the company money instead of earning the company money

https://redd.it/kdebzt
@r_devops

Detect and Block Exploit Attempts for Kubernetes Vulnerability: CVE-2020-8554 Man in the Middle (MiTM) Attack Using Kubernetes Service Resources

Read the blog here.


TL;DR

Kubernetes CVE-2020-8554 enables an attacker to intercept traffic from other pods (or nodes) in the cluster if the attacker can create or edit services and pods. This vulnerability was originally discovered almost a year ago, revealing a design flaw that affects all Kubernetes versions
Exploiting this weakness requires at the minimum RBAC permissions to create, update or patch Service resources, specifically:
An attacker that is able to create a ClusterIP service and set the spec.externalIPs field can intercept traffic to that IP.
An attacker that is able to patch the status.loadBalancer.ingress.ip field of LoadBalancer service status.loadBalancer.ingress.ip can intercept traffic to that IP.

What you can do about it:

At this point, CVE-2020-8554  does not have a software update that mitigates this issue. Users are advised to implement fine-grained access restrictions and can use RBAC policies and admission controllers such as this one, OPA Gatekeeper Constraint or others. 
Scan Kubernetes audit logs for evidence of attempts to exploit this CVE. The creation of a new Service or modifying an existing Service leave traces in the audit log.
Monitor Kubernetes resources and entities for attempt at service creation or modification that allow attackers to intercept traffic.
Use admission controllers policy logic to deny and alert on external-facing or unauthorized Ingress Controllers and Services.

https://redd.it/kd0978
@r_devops

Hi team greetings can I know some best practices to store hashicorp vault address and token.

Currently I am doing by storing and restore those using env variables.

But is there any other best practice available for production point of view.

Thanks in advance.

https://redd.it/kcv2r2
@r_devops

DevOps without ops

At our company we lack ops in a way that the ~6 people team also operates and fixes stuff whenever needed. Now people decided it would be best to implement DevOps, which in their mind means developers also doing operations with rotating schedules and on-call duties. The desired state is doing Devs for about 80 % time and 20 % for ops.

Everywhere I look DevOps means better communication between the two teams, but I can't find anywhere anything about not having dedicated team, the closest thing I can find is SRE. However I'm told this is not as uncommon in practice so I would ask everyone here if you have experience with this or can point me to a blogpost or something.

https://redd.it/kcu98y
@r_devops

Perfect gitflow CI in scrum - opinions?

This could be a cross-post between r/Git, r/Scrum and r/DevOps, but the clou of this topic I believe is closest to r/DevOps. I'm not a young dev anymore, but I'm quite new to DevOps so I'm curious about your opinions, suggestions, your own experiences, etc. What I worked out has been working pretty well so far, but when it comes to CI/CD I'm always looking for improvement.

​

Some background

My team (less than 10 devs) works in Scrum: 2-week sprint interval ending with live release (with code freeze 2 days prior to that). The usual story flow: READY > DEV > CODE REVIEW > QA > RELEASE > LIVE QA. We incorporate 3 test instances that always correspond to 2 branches (newest sprint (development) branch is always deployed to staging/TA, newest release candidate is on PRE instance). Docker/helm-chart/kubernetes handles instances and its services, Gitlab serves our repositories.

​

Our variation of Gitflow

We're using a custom variation of Gitflow. It's pretty much the same as Gitflow, except that we use chronologically numbered sprint branches instead develop. Release branches are numbered in same fashion, so it makes it really easy to tell in which calendar sprint a given scope of features was added. This also fixes a problem of prematurely "smuggling" features that normally would have been already pushed to "develop", but were not yet approved by QA. One can say this could be solved by QAing feature branches before they are merged to common "development" branch, BUT we do need to test multiple different features when they are combined together (integration test is even more important for us than testing a feature on its own). That's why we use sprint branches, their main purpose is to automatically deploy to STG/TA instances on any commit so that QA can test all features right away. In case a given feature is not approved, and there is not enough time left to fix it before end of sprint (rarely, but happens), instead of messing with cherry-picking we can just merge only the approved feature branches into release branch (and skip sprint branch). In case the whole scope of the sprint branch is fine (90% of the time) we just merge sprint branch to release. The release package is tested again on PRE and if it's fine it's released, followed by merging to master. Critical hotfixes are pushed with hotfix/x.y branch names (where x is the version of the release to be patched, y - incremental number starting from 1).

​

Automation

This obviously implies plenty of micromanagement around branches, but I've managed to automate it with gitlab API and scripts.

\- sprint/x branches are automatically deployed to STG/TA on any commit.

\- similarly, release/x and hotfix/x branches are deployed to PRE (once built can be can be manually deployed to LIVE from same pipeline)

\- merging sprint/xx into release/xx creates sprint/(xx+1) and protects sprint/xx (so prevents pushing to old branch by accident)

\- merging release/xx into master creates release/(xx+1) and protects release/xx, it also creates 2 new merge requests: sprint/(xx+1) into release/(xx+1) and release/(xx+1) into master

\- there are usually 2-3 days between merging sprint to release and merging release to master (last 2-3 days of the sprint), because of that in that time normally automating adding a MR from new sprint to new release would not be possible (because a new release branch needed for target branch is not yet created), that's why we must always have additional branches 1 sprint ahead of current sprint

​

Pros

\- most pros resolve from a fact that inspecting branches is much more convenient than tags, i.e:

\- comparing diff between 2 release branches is possible even from gitlab UI (easier to inspect scope of release)

\- chronology, easier to rollback

\- docker images are named after branches, so it takes very little time to check in k9s or app actuators what's currently deployed (instead I'd have to check by commit hash or pipeline number)

​

Cons

\- necessity

to have an eye on using correct branches that change every two weeks due to Scrum, i.e. target branch in MR if sprint ends (such mistakes can be prevented by protecting old release branches)

https://redd.it/kdmihv
@r_devops

Learn Git Visually?

This is an illustrated book that explains how git works at its core. which surprisingly is rather simple. Learn git visually i.e. learn core git concepts not just memorizing commands. It will stick longer and it will be easier to understand if you can associate how git works under the hood with everyday analogies.

Learn git concepts not commands

https://redd.it/kdof5f
@r_devops

How to monitor servers using SMPP + Nagios/Centreon?

The scenario I have is a system that transmits information about the servers over SMPP. Since SMPP being known for being an SMS protocol, the server information is inside the SMS content in JSON format. This data is currently being parsed added to Prometheus.

In my little project at the moment the data from Prometheus is displayed in Grafana and if I want to keep tabs on alerts I need to keep on watching it all day long.

My question is, how can I have Nagios implemented (if applicable)? I believe I could be consuming the data from Prometheus and working the alert administration in Nagios/Centreon for example (or maybe Prometheus alert manager), and this way no one would ever have to be staring at the statuses on Grafana.

Is it possible? Or what is the best way to have those alerts handled?

I am pretty new in the topic and never really implemented or used such software.

https://redd.it/kdq8pt
@r_devops

Assistance with likely job title change - not sure of appropriate title

Hey guys,

So I need some help here (omitting some details for privacy reasons) - I am currently an “Application Support Engineer” at a SMB. Due to unfortunate circumstances involving my former boss, I am now reporting to the president of the company.

Currently, I am responsible for my company’s entire AWS multi-region infrastructure including security/compliance and managing costs, as well as creating/maintaining automation through an enterprise job scheduler and operational monitoring via Datadog (infrastructure/applications) and Grafana (Salesforce data). I am fluent in Powershell and Python, and working towards learning Docker and Terraform as well as getting better acquainted with source control via Bitbucket/Git. I do not have any certifications and I only have an associate’s degree.

There is currently an open question regarding whether I will be receiving a promotion and title change but I am not sure what title would be appropriate. I am preferring to keep my job responsibilities technical while maybe becoming a supervisor to other people on my team until we find a replacement for my former boss. However, I want to transition my title into something that better describes my current responsibilities.

I was thinking “Lead Cloud Engineer” as we are not quite suited yet in house for traditional DevOps/SRE functions, slowly but surely getting there. I am hoping maybe some of you might be able to assist here. Any help or advice is greatly appreciated. Thanks in advance!

https://redd.it/kdrw4t
@r_devops

Sparky CI on k8s cluster

Sparky is a lightweight CI server written on Raku. It has a Raku API and plugin system where plugins could be written on many languages.
Initially the server was written to be run on a single machine and did not scale well. So it only could handle a small/medium load, mostly working in localhost mode.

Now, with the help of k8s cluster Sparky could be easily turned into industrial scale CI server.


More details in this post - https://sparrowdo.wordpress.com/2020/12/15/sparky-on-k8s-cluster/


PS comments and feedback are welcome

https://redd.it/kdsu5i
@r_devops

Best tool to create a landing page with several products

hello, I want to ask you for a recommendation, I have many product selling in a marketplace, and I want to create a website to sum up all those product in one place (like a Brand), in the website every product you click on it, it redirect you to the product's page in the marketplace, so what are the choices that I have (difficulty, price, time...) which tool do you recommend? and thank you for your time

https://redd.it/kdws75
@r_devops

How to GitOps with a procedural approach?

I'm currently messing around with various pipeline methods and things to see what would work best for our company. We work in a regulated industry, so making sure everything is meeting an infrastructure qualification is really important.

I'm convinced we can do this qualification with GitOps - having all our infrastructure as code and having all changes recorded in source control (and tested as part of the pipeline). This works great for a declarative approach like CDK / Cloudformation. But as AWS users will know - it often takes a while to get things into Cloudformation and some things still need to be done on the CLI.

Currently I'm thinking 95% of the stuff will be in Cloudformation, but how can I handle that 5% of things that need to be done through the CLI in a more procedural style?

How can I handle these two different styles?
How can I run tests on a procedural script? (CFN templates have things like cfn-nag)

https://redd.it/kdy4gn
@r_devops

Running Gitlab-ci + docker + services + volumes driving me crazy

I'm trying to run browser tests in Gitlab-ci. To do this, I'm using services to run httpd, php-fpm and mysql. The problems I'm encountering all have to do with how the code is injected in the containers.

I was assuming that the project would be volume-mounted into the containers as one would probably do if you were to do something locally with docker or docker-compose, but that doesn't seem the case. The code is somehow injected after the container has been started without the use of docker volumes. This create a number of problems that I haven't found a solution for.

To start of, the httpd container is the standard httpd container, with the exception that at the end of httpd.conf, I do Include ${CI_PROJECT_DIR}/ci/vhost.conf to include the vhost config for the project, which is stored in the actual repository. When the container starts, CI_PROJECT_DIR will contain something like /builds/mygroup/myproject, but /builds is still empty so the container fails to start.

Another problem is that the services aren't using the same path/filesystem. If I create a file in a PHP script, that file isn't available in the httpd container making it hard to test something like a file upload. Also, any commands I execute in the job that alter the filesystem (like prepping a config file) aren't replicated to the service containers either, and I don't really see an easy way to get that in there since services don't have a before_script section.

So either I'm totally missing something here, or Gitlab-ci just isn't fit for this kind of CI jobs unless I create a custom-tailored and all-containing container for each and every project. Any tips or advice?

https://redd.it/kdxt34
@r_devops

Analyzing Maven dependencies

Hey, I have a project with 20 repos, each with up to 200 modules. The problem is that the repos have dependencies to each other which are circular. This means at the moment all repos have to be built on any code change. Is there any good tooling to help me find common dependencies to extract them into a library repo? Any good tips are appreciated 😌

https://redd.it/kdtyrj
@r_devops

Have a techinal Interview for a Cloud engineer(1-3Y exp) role in 2 days. I've worked as a freelancer up till now and it's going to be my first interview for such a role and I really want this to work out. what should I prepare myself for?

I was approached by a recruiter for this job and boy does the job description fit me or what. I made it smoothly through the first round, now there's a technical interview scheduled in 2 days and I'm kinda skeptical as this is my first time sitting in an official technical interview for a company of this size (It's a big multi-national cloud consulting firm). I've like a year of experience working as a freelance cloud/web dev guy for small and medium-sized companies and I have the AWS SAA cert under my belt, but this is a first for me.

I really want this job due to finances being very tight. So, people of /r/devops what topics should I brush up on and what type of questions can I expect? Will there be whiteboarding? Will they ask me to write some automation scripts there and then? Please help.

https://redd.it/kdrwgb
@r_devops

How services like gmail makes backup of it?

Considering its one of the largest companies on the world that serves most of services to the public then I am wondering how do they manage such amount of data backed up?

Let's say I have some important emails from a 10-15 years back, considering how much emails is running on a gmail domain and how active spambots are I am able to believe that these emails are not backed up with help of another disk drive especially that the storage drives has its lifetime.

https://redd.it/kdpkce
@r_devops

I have a small training budget i need to use before EOY - any suggestions for quality commercial/paid courses or resources like books?

just as the title says.

https://redd.it/kdlc6s
@r_devops

Beginner here, would love if someone could help me a bit

Hey! How are you doing? Hopefully you are feeling good!

I'm kinda lost not going to lie :)

I'm not sure if I'm posting on the right subreddit, but hopefully someone will help me.

I'm trying to deploy a small simple Django app using kubernetes and ingress-nginx. I'm not sure how to put everything into production and how to create a simple development flow for everything using travis and git and gke.

I would be really grateful if someone would help me with some advice and such!

https://redd.it/ke5cdb
@r_devops

Need help with collection of logs from Nifi , storing in Elasticsearch and displaying in Grafana.

My Nifi is running in a container separate from the one containing fluentd, elasticsearch and grafana. I wanted to configure fluentd to collect logs from nifi processors and forward it to elasticsearch and later query the same from grafana and display. How can I achieve this? I am able to see the count of logs as of now in grafana, but in my docker logs I get this warning everytime a new log is generated from nifi:

fluentd_1 | warning: 299 Elasticsearch-7.2.0-508c38a "[types removal] Specifying types in bulk requests is deprecated."

https://redd.it/ke71v2
@r_devops

Automation tools for VMware + potential hybrid in the future?

We're looking at tooling options to finally move to automation. We're heavily VMware on-prem, mostly Windows, but likely to introduce more Linux over time, and likely to use hybrid cloud (Azure) in the future.

Are there any obvious tooling options that I'm missing in this list, or any that would be recommended over others?

\- VRealize
\- Packer/Ansible/Terraform
\- SaltStack

​

Thanks

https://redd.it/ke9om8
@r_devops