Perfect gitflow CI in scrum - opinions?

This could be a cross-post between r/Git, r/Scrum and r/DevOps, but the clou of this topic I believe is closest to r/DevOps. I'm not a young dev anymore, but I'm quite new to DevOps so I'm curious about your opinions, suggestions, your own experiences, etc. What I worked out has been working pretty well so far, but when it comes to CI/CD I'm always looking for improvement.

​

Some background

My team (less than 10 devs) works in Scrum: 2-week sprint interval ending with live release (with code freeze 2 days prior to that). The usual story flow: READY > DEV > CODE REVIEW > QA > RELEASE > LIVE QA. We incorporate 3 test instances that always correspond to 2 branches (newest sprint (development) branch is always deployed to staging/TA, newest release candidate is on PRE instance). Docker/helm-chart/kubernetes handles instances and its services, Gitlab serves our repositories.

​

Our variation of Gitflow

We're using a custom variation of Gitflow. It's pretty much the same as Gitflow, except that we use chronologically numbered sprint branches instead develop. Release branches are numbered in same fashion, so it makes it really easy to tell in which calendar sprint a given scope of features was added. This also fixes a problem of prematurely "smuggling" features that normally would have been already pushed to "develop", but were not yet approved by QA. One can say this could be solved by QAing feature branches before they are merged to common "development" branch, BUT we do need to test multiple different features when they are combined together (integration test is even more important for us than testing a feature on its own). That's why we use sprint branches, their main purpose is to automatically deploy to STG/TA instances on any commit so that QA can test all features right away. In case a given feature is not approved, and there is not enough time left to fix it before end of sprint (rarely, but happens), instead of messing with cherry-picking we can just merge only the approved feature branches into release branch (and skip sprint branch). In case the whole scope of the sprint branch is fine (90% of the time) we just merge sprint branch to release. The release package is tested again on PRE and if it's fine it's released, followed by merging to master. Critical hotfixes are pushed with hotfix/x.y branch names (where x is the version of the release to be patched, y - incremental number starting from 1).

​

Automation

This obviously implies plenty of micromanagement around branches, but I've managed to automate it with gitlab API and scripts.

\- sprint/x branches are automatically deployed to STG/TA on any commit.

\- similarly, release/x and hotfix/x branches are deployed to PRE (once built can be can be manually deployed to LIVE from same pipeline)

\- merging sprint/xx into release/xx creates sprint/(xx+1) and protects sprint/xx (so prevents pushing to old branch by accident)

\- merging release/xx into master creates release/(xx+1) and protects release/xx, it also creates 2 new merge requests: sprint/(xx+1) into release/(xx+1) and release/(xx+1) into master

\- there are usually 2-3 days between merging sprint to release and merging release to master (last 2-3 days of the sprint), because of that in that time normally automating adding a MR from new sprint to new release would not be possible (because a new release branch needed for target branch is not yet created), that's why we must always have additional branches 1 sprint ahead of current sprint

​

Pros

\- most pros resolve from a fact that inspecting branches is much more convenient than tags, i.e:

\- comparing diff between 2 release branches is possible even from gitlab UI (easier to inspect scope of release)

\- chronology, easier to rollback

\- docker images are named after branches, so it takes very little time to check in k9s or app actuators what's currently deployed (instead I'd have to check by commit hash or pipeline number)

​

Cons

\- necessity

to have an eye on using correct branches that change every two weeks due to Scrum, i.e. target branch in MR if sprint ends (such mistakes can be prevented by protecting old release branches)

https://redd.it/kdmihv
@r_devops

Learn Git Visually?

This is an illustrated book that explains how git works at its core. which surprisingly is rather simple. Learn git visually i.e. learn core git concepts not just memorizing commands. It will stick longer and it will be easier to understand if you can associate how git works under the hood with everyday analogies.

Learn git concepts not commands

https://redd.it/kdof5f
@r_devops

How to monitor servers using SMPP + Nagios/Centreon?

The scenario I have is a system that transmits information about the servers over SMPP. Since SMPP being known for being an SMS protocol, the server information is inside the SMS content in JSON format. This data is currently being parsed added to Prometheus.

In my little project at the moment the data from Prometheus is displayed in Grafana and if I want to keep tabs on alerts I need to keep on watching it all day long.

My question is, how can I have Nagios implemented (if applicable)? I believe I could be consuming the data from Prometheus and working the alert administration in Nagios/Centreon for example (or maybe Prometheus alert manager), and this way no one would ever have to be staring at the statuses on Grafana.

Is it possible? Or what is the best way to have those alerts handled?

I am pretty new in the topic and never really implemented or used such software.

https://redd.it/kdq8pt
@r_devops

Assistance with likely job title change - not sure of appropriate title

Hey guys,

So I need some help here (omitting some details for privacy reasons) - I am currently an “Application Support Engineer” at a SMB. Due to unfortunate circumstances involving my former boss, I am now reporting to the president of the company.

Currently, I am responsible for my company’s entire AWS multi-region infrastructure including security/compliance and managing costs, as well as creating/maintaining automation through an enterprise job scheduler and operational monitoring via Datadog (infrastructure/applications) and Grafana (Salesforce data). I am fluent in Powershell and Python, and working towards learning Docker and Terraform as well as getting better acquainted with source control via Bitbucket/Git. I do not have any certifications and I only have an associate’s degree.

There is currently an open question regarding whether I will be receiving a promotion and title change but I am not sure what title would be appropriate. I am preferring to keep my job responsibilities technical while maybe becoming a supervisor to other people on my team until we find a replacement for my former boss. However, I want to transition my title into something that better describes my current responsibilities.

I was thinking “Lead Cloud Engineer” as we are not quite suited yet in house for traditional DevOps/SRE functions, slowly but surely getting there. I am hoping maybe some of you might be able to assist here. Any help or advice is greatly appreciated. Thanks in advance!

https://redd.it/kdrw4t
@r_devops

Sparky CI on k8s cluster

Sparky is a lightweight CI server written on Raku. It has a Raku API and plugin system where plugins could be written on many languages.
Initially the server was written to be run on a single machine and did not scale well. So it only could handle a small/medium load, mostly working in localhost mode.

Now, with the help of k8s cluster Sparky could be easily turned into industrial scale CI server.


More details in this post - https://sparrowdo.wordpress.com/2020/12/15/sparky-on-k8s-cluster/


PS comments and feedback are welcome

https://redd.it/kdsu5i
@r_devops

Best tool to create a landing page with several products

hello, I want to ask you for a recommendation, I have many product selling in a marketplace, and I want to create a website to sum up all those product in one place (like a Brand), in the website every product you click on it, it redirect you to the product's page in the marketplace, so what are the choices that I have (difficulty, price, time...) which tool do you recommend? and thank you for your time

https://redd.it/kdws75
@r_devops

How to GitOps with a procedural approach?

I'm currently messing around with various pipeline methods and things to see what would work best for our company. We work in a regulated industry, so making sure everything is meeting an infrastructure qualification is really important.

I'm convinced we can do this qualification with GitOps - having all our infrastructure as code and having all changes recorded in source control (and tested as part of the pipeline). This works great for a declarative approach like CDK / Cloudformation. But as AWS users will know - it often takes a while to get things into Cloudformation and some things still need to be done on the CLI.

Currently I'm thinking 95% of the stuff will be in Cloudformation, but how can I handle that 5% of things that need to be done through the CLI in a more procedural style?

How can I handle these two different styles?
How can I run tests on a procedural script? (CFN templates have things like cfn-nag)

https://redd.it/kdy4gn
@r_devops

Running Gitlab-ci + docker + services + volumes driving me crazy

I'm trying to run browser tests in Gitlab-ci. To do this, I'm using services to run httpd, php-fpm and mysql. The problems I'm encountering all have to do with how the code is injected in the containers.

I was assuming that the project would be volume-mounted into the containers as one would probably do if you were to do something locally with docker or docker-compose, but that doesn't seem the case. The code is somehow injected after the container has been started without the use of docker volumes. This create a number of problems that I haven't found a solution for.

To start of, the httpd container is the standard httpd container, with the exception that at the end of httpd.conf, I do Include ${CI_PROJECT_DIR}/ci/vhost.conf to include the vhost config for the project, which is stored in the actual repository. When the container starts, CI_PROJECT_DIR will contain something like /builds/mygroup/myproject, but /builds is still empty so the container fails to start.

Another problem is that the services aren't using the same path/filesystem. If I create a file in a PHP script, that file isn't available in the httpd container making it hard to test something like a file upload. Also, any commands I execute in the job that alter the filesystem (like prepping a config file) aren't replicated to the service containers either, and I don't really see an easy way to get that in there since services don't have a before_script section.

So either I'm totally missing something here, or Gitlab-ci just isn't fit for this kind of CI jobs unless I create a custom-tailored and all-containing container for each and every project. Any tips or advice?

https://redd.it/kdxt34
@r_devops

Analyzing Maven dependencies

Hey, I have a project with 20 repos, each with up to 200 modules. The problem is that the repos have dependencies to each other which are circular. This means at the moment all repos have to be built on any code change. Is there any good tooling to help me find common dependencies to extract them into a library repo? Any good tips are appreciated 😌

https://redd.it/kdtyrj
@r_devops

Have a techinal Interview for a Cloud engineer(1-3Y exp) role in 2 days. I've worked as a freelancer up till now and it's going to be my first interview for such a role and I really want this to work out. what should I prepare myself for?

I was approached by a recruiter for this job and boy does the job description fit me or what. I made it smoothly through the first round, now there's a technical interview scheduled in 2 days and I'm kinda skeptical as this is my first time sitting in an official technical interview for a company of this size (It's a big multi-national cloud consulting firm). I've like a year of experience working as a freelance cloud/web dev guy for small and medium-sized companies and I have the AWS SAA cert under my belt, but this is a first for me.

I really want this job due to finances being very tight. So, people of /r/devops what topics should I brush up on and what type of questions can I expect? Will there be whiteboarding? Will they ask me to write some automation scripts there and then? Please help.

https://redd.it/kdrwgb
@r_devops

How services like gmail makes backup of it?

Considering its one of the largest companies on the world that serves most of services to the public then I am wondering how do they manage such amount of data backed up?

Let's say I have some important emails from a 10-15 years back, considering how much emails is running on a gmail domain and how active spambots are I am able to believe that these emails are not backed up with help of another disk drive especially that the storage drives has its lifetime.

https://redd.it/kdpkce
@r_devops

I have a small training budget i need to use before EOY - any suggestions for quality commercial/paid courses or resources like books?

just as the title says.

https://redd.it/kdlc6s
@r_devops

Beginner here, would love if someone could help me a bit

Hey! How are you doing? Hopefully you are feeling good!

I'm kinda lost not going to lie :)

I'm not sure if I'm posting on the right subreddit, but hopefully someone will help me.

I'm trying to deploy a small simple Django app using kubernetes and ingress-nginx. I'm not sure how to put everything into production and how to create a simple development flow for everything using travis and git and gke.

I would be really grateful if someone would help me with some advice and such!

https://redd.it/ke5cdb
@r_devops

Need help with collection of logs from Nifi , storing in Elasticsearch and displaying in Grafana.

My Nifi is running in a container separate from the one containing fluentd, elasticsearch and grafana. I wanted to configure fluentd to collect logs from nifi processors and forward it to elasticsearch and later query the same from grafana and display. How can I achieve this? I am able to see the count of logs as of now in grafana, but in my docker logs I get this warning everytime a new log is generated from nifi:

fluentd_1 | warning: 299 Elasticsearch-7.2.0-508c38a "[types removal] Specifying types in bulk requests is deprecated."

https://redd.it/ke71v2
@r_devops

Automation tools for VMware + potential hybrid in the future?

We're looking at tooling options to finally move to automation. We're heavily VMware on-prem, mostly Windows, but likely to introduce more Linux over time, and likely to use hybrid cloud (Azure) in the future.

Are there any obvious tooling options that I'm missing in this list, or any that would be recommended over others?

\- VRealize
\- Packer/Ansible/Terraform
\- SaltStack

​

Thanks

https://redd.it/ke9om8
@r_devops

Is there a practice to incorporate blue/green into names of resources?

I think about following practice of blue/green deploy and wonder how people name duplicating assets?

For blue/green I have to temporarily duplicate resources, like prod-app-1, prod-redis-1, prod-job-1. I think that I have to alter naming schema from ENV-APP-COPY to: ENV-COLOR-APP-COPY, resulting in resources: prod-blue-app-1, prod-blue-redis-1, prod-blue-job-1 this release and prod-green-app-1, prod-green-redis-1, prod-green-job-1 next release.

This also means instead of marking Git/Hg with master I will move colors blue / green and use master / old-master just for informational purpose (to hind which color is a new release). I mean that I will embed blue / green into apps config because those names represent variations (current and previous release) while master denotes the tip so same name cannot be used in releases (how could you express different "master" in blue & green env? by naming blue-master & green-master?).

Prefixing resources with a color I can avoid conflicts between old/new deploys (only LB + DB are shared between releases) and can dispose no longer unused resources later.

Any ideas?

https://redd.it/ke945x
@r_devops

K8S troubleshooting made easy

Hi everyone,

In the last couple of months, we have been working on Kubernetes native product to help dev+ops teams troubleshoot efficiently. We provide system-wide visibility and control. For each service, we automatically construct a coherent view, including relevant deploys, config changes, and alerts. You can then drill down and explore suspicious changes and their ripple effect. Our platform sits on top of your existing source control, CI/CD tools, the underlying infrastructure, and monitoring tools and has quick and seamless integration.

So, in case you are running on top of Kubernetes and want to join our closed beta - DM :)

Complete service view

Deployment tracking

Full changes diff

https://redd.it/kebjm2
@r_devops

Three-Part Guide to How to Set Up as CI/CD Workflow - would love your feedback!

We just published the last post in our three-part series on setting up a CI/CD workflow. Hopefully it's a useful guide! Please let us know what you think? Easy to follow? Anything you'd add? Cheers!


Part One: how to set up a CI workflow using GitHub and AWS. https://monadical.com/posts/set-up-ci-workflow.html

Part Two: how to set up a CD workflow with Docker on AWS, using ECR and ECS. You’ll learn how to create a pipeline to get your image from the image repository to your production cluster. https://monadical.com/posts/set-up-ci-workflow-part-two.html

Part Three: introduces testing and linting to the workflow, and shows you how to adapt what you’ve learned so that it works for different containers by using a containerized Python application. https://monadical.com/.../set-up-ci-workflow-part-three.html

https://redd.it/kebjyf
@r_devops

Do we need powerful laptops with all the tools installed?

With the emergence of browser-based tools like Gitpod, GitHub Spaces, and others, we might want to rethink the needs we have for development environments. Did you try any of those? WDYT?

​

\>> https://youtu.be/QV1fYt-7SLU

https://redd.it/kef5rd
@r_devops

Flipaclip

Hello, I’m a newbie and wondered about FlipaClip app... Can I make similar one with unity game engine? How is this possible to build sth like this app?
Its a cartoon/animation maker app...
Danke

https://redd.it/kei79k
@r_devops