Discussion: Agility

I've been reading The Phoenix Project. I quote, "...get your build process automated." "Get humans out of the deployment business. Figure out how to get to ten deploys a day." (Chapter 30)

Getting 10 deploys a day was just a number that Erik threw out as a baseline for Bill to work towards. But, that number can be at any threshold that works for your business.

I wanted to open this up to a discussion about agility in your day to day.

1. What does your current build process look like? (Is it fully automated from Dev to Prod? Is human interaction required?)
2. How did you get your build process automated? (tools/solutions used)
3. How many deploys are you pushing a day/week/month?
4. What is your deploy frequency? (on demand/daily/weekly/monthly?)
5. What constraints or bottlenecks, if any, are you still hitting?

https://redd.it/lb99en
@r_devops

Client side javascript usage - best practice how to analyze it?

So we have moved our applications to the cloud in Azure Kubernetes Services. We have the prometheus stack installed for some interesting analysis.

Our main website is a react clientside website. We want to monitor how our users use the website. I am talking what menu items they press, what configuration preferences they have stored in their local storage etc.

We already have a method to store "server side" metrics via Prometheus.net (it's a .net application).

I have looked into Azure Application Insights. They mention things like monitoring Retention, Funnels and Users, sessions, and events analysis. I am not sure if this makes sense for our scenario.

I am curious, what is the best practice way to gather this type of clientside data?

https://redd.it/lb5nxo
@r_devops

CI-CD Platform (A new initiative) --- Trying to make CI-CD processes smoother

Hey Everyone,

I am a Product Manager & is driven by the aim of bringing innovation to B2B SaaS & delivering ground breaking product initiatives.

One of the opportunity areas identified is “Making DevOps teams less busy & more productive”. We envision to achieve this by conceptualising & building a “CI-CDaaS Platform” to offer seamless experience and efficient software delivery pipeline management without hiccups and much manual intervention.

If your daily life involves a tussle with the CI-CD processes, understanding the challenges faced by you will be of UTMOST IMPORTANCE to us and will help us to identify the right problem areas to solve.

Below is the link to a short survey to get your thoughts:

https://forms.office.com/Pages/ResponsePage.aspx?id=2bCoUwDZzEidflk13I1bF0GBbFQhc-5BqssHSqnq37NUMFk2QUFVWTZVSFJKNUxPUUdSMFlaSlBDWC4u

I would sincerely appreciate your response.

Cheers...!!

https://redd.it/larjvh
@r_devops

What belongs in an RPM vs a Config Management Tool (Ansible)?

Currently my team is trying to put things like:

Creation of new users on a machine
firewalld rules

into RPMs. This seems wrong to me though I can't really place why. I am newish to the team and only 3 years into my career though, so I don't have as much perspective as others. Is there a good reason why those things should be in an RPM versus an Ansible role? More generally, how do you delineate what belongs in an RPM versus Ansible?

I don't think the other more senior developers on my team like Ansible and would prefer to put as much as possible in the RPM and document the rest of what needs to be setup. The RPMs and documentation would be given to other teams. However, I manage our own VMWare environment where I would much prefer to use Ansible to update development clusters.

To be fair, I think my predecessor put too much in our Ansible roles and, rather than using group_vars, put all variables in an included variable file. Every dev on our team then runs the Ansible playbooks to update their own cluster. Furthermore, every development cluster has its own yum repository (running in a docker container. No one on our team uses or really wants to use docker) which the Ansible roles copy RPMs to. In short, I think some of the design was overly complex and left a bad first impression to some of the other developers.

I am sorry for being scatter brained, but the more I type, I wonder if this is more of a process / education / culture problem, rather than a what tool to use problem. Does any of this seem whacky?

Thanks, I appreciate any help. I don't really have anyone at work to talk with for advice.

https://redd.it/lbdhzi
@r_devops

Session 4 Manage Container Images | #Docker Images | Openshift tutorial ...



· Install git and Download a Demo Application

· Create/Modify Docker file for the Application

· Build the Image from Downloaded Application using Docker File

· Run the Container from Local Image

· Tag and Push the Image to Docker Hub

· Remove the Image.

https://redd.it/lbhonp
@r_devops

wrap.sh: an experiment in devops UX

My team has been looking into ways of improving the devops experience for a while now, and testing our theories through the spontaneous creation of a bunch of small tools.

One theory we examined goes as follows:

1. The projects with bad CI experiences have poor, slow and low-signal feedback from their CI pipelines.

2. The most direct signal you can get is from direct access to the broken version of the project

3. So if a project lets you connect directly to the pipeline when a test fails, that should come with a good CI experience.

This led to one of the more promising experiments: Wrap.sh - an attempt to increase feedback and visibility in CI pipelines. It's also open-source.

We don't fully know what to expect from this thing; it might be useful to someone, maybe for OSS projects using end-to-end tests. Or maybe not.

In any case, we'd love to hear your questions and feedback regarding the tool (or the theory it's based on).

The links:

https://wrap.sh

https://github.com/layer-devops/wrap.sh

https://redd.it/lbdmgo
@r_devops

Containers monitoring solution

Hello Folks,

I am working on a task for comparison on different monitoring tools currently out there for containers/k8s. I understand Prometheus has some native support like service discovery for Kubernetes, but i would like to understand what are the close competitors to look for. I am looking for about 6-7 tools for a deeper evaluation. A quick Google search returns many tools but would like to hear from the community experts about your preference on monitoring k8s clusters. Any useful inputs on this is highly appreciated.

Thanks

https://redd.it/larf53
@r_devops

Solving ArgoCD Secret Management with ArgoCD-Vault-Plugin

Hi everyone, i wanted to share an ArgoCD plugin that i have been working on that allows for connecting to Vault in a simple way that does not require an Operator or CRD. The plugin is in its early stages and only supports a couple backends but we look forward to any contributions/suggestions or ideas you may have!

https://werne2j.medium.com/argocd-secret-management-with-argocd-vault-plugin-539f104aff05

https://redd.it/lbpcpp
@r_devops

How do you automate AWS AMI updates?

I currently manage most of our infra with terraform. I have a module that returns the latest AWS AMI for a particular service (EKS, ECS, etc). This means that whenever we run a terraform plan for a project that uses the service, the plan will include an AMI update if AWS has released a newer AMI. This has worked fine but I'd like to make this a little bit more stable. I'd like to have the latest AMI run for a while in our non-prod environments and then have some sort of approval process so that production gets updated later. Any ideas on how to make this work? Or any ideas for an alternative approach?

https://redd.it/lbrfhs
@r_devops

A guide to the best SRE tools

A beginner's guide to common SRE/DevOps tools and incident management automation (monitoring, oncall, IaC):

https://www.getcortexapp.com/post/a-guide-to-the-best-sre-tools

https://redd.it/lbtomt
@r_devops

transfer thousands of files of any size with optimization

We have been doing a mix of manual process and some scripts to transfer files of various sizes from one system to another. Basically there are shares where people may dump hundreds or thousands of files of varying sizes. We then move these files to another location.

We want to use a tool that would automatically optimize speed/perf based on file size and amount and transfer the files. (nifi maybe?)

https://redd.it/lboo0s
@r_devops

Keeping track of the infrastructure

Hi there,

the cloud application my company is developing did not start with orchestration in mind (and it's to late to do so :D).

We have some hosted components in Azure (fixed set) together with some managed machines in OVH (scalable).

For two purposes:

\- dynamic topology

\- monitoring

to be able to have a service where every component can "check in" and that other applications can use to reliably get info on the current online components.

Do you guys have any suggestion?

Thanks a real lot,

https://redd.it/lbnw5t
@r_devops

Prometheus exporter to retrieve the DockerHub rate limit counts as scrape target

This exporter allows you to retrieve the DockerHub rate limit counts as scrape target for Prometheus as Gauge metric.
Multi arch docker images are available (arm/arm64/amd64) with a complete docker-compose example.

I hope you find it useful.

Docker Hub Rate Limit Exporter Github Link

https://redd.it/lbigg8
@r_devops

What do you use to manage on-call alerting on AWS?

Hi, we have a current system where we use cloudwatch and ms teams notifications for alerting if something happens in production.

However, management requires a 24/7 support. Therefore I'd like to use a system with scheduling for 24/7 support and ability to call/alert developer in case severe incident with production happens.

​

What are the best tools for the job?

https://redd.it/lbldob
@r_devops

Help me with setting this up

Hey guys, I'm working on a project which has a production database hosted in AWS RDS. We work on a separate local Postgres database from inside a Docker container. The APIs are to be uploaded to Lambda. This is where things get complicated. Whatever changes or migrations we make to the development database are not reflected in the RDS. I want the final changes to be applied to the RDS automatically through some pipeline. Is there any guide that helps with this kind of problem?

Sorry if I sound noob, this is my first time working with large techs. Thank you.

https://redd.it/lbl04s
@r_devops

Need help from someone with AWS CI/CD and VPC experience

I hope this does not violate community guidelines, but I really need some help with an AWS project. I am working on a MENN App in AWS and we are having a lot of issues with CI/CD from CodeCommit --> CodeBuild --> CodeDeploy for Lamdas. I am also having no luck connecting Mongo Atlas to a Lambda. We are all full stack devs and I have a background in systems administration so I was able to setup VPC peering, but can't get IAM authentication to work for connecting to Mongo Atlas from Node.JS Lambda. I don't have a lot of money, but would be willing to pay if anyone could help.

https://redd.it/lc4b6q
@r_devops

Help with specific metrics around platform for increasing headcount

Unsurprising story: ask for head count fails because DevOps / platform / infrastructure isn’t easily quantifiable like revenue of an external product / feature.

Really need someone to share either/both:
1. specific measures they use that are effective when talking to executives
1. how these are scrapped / generated

Disclaimer: I’m aware of Accelerate metrics but this is one of those challenges beyond that. A product team that makes money can show they’re making more money by adjusting those metrics; it is difficult if you’re 1-2 deviations off.

https://redd.it/lc2q8y
@r_devops

How does manual testing fit into CICD and trunk based development?

Struggling how to understand how people do CICD and trunk based development with or without manual testing. Surely you can go straight to prod if you pass all automated regression tests - but is anyone ever still performing manual tests? And at what point in the development cycle/process is manual testing performed? Is it a gate before release is actually "deployed"?

https://redd.it/lc10mf
@r_devops

Career Advice I want to move from Civil Engineering to DevOps engineering?

I'm currently doing my bachelor's degree in Civil Engineering technology in South Africa and when I graduate I may become a civil technologist/engineer. However, I want to branch into DevOps. What is the best route for me to become a DevOps engineer ? Is there a bridging honours or masters I can do to become a DevOps Engineer?

https://redd.it/lc3u1z
@r_devops

Declarative API's

I am wondering whether there's actual use case or its an advanced users feature that is a nice-to-have.

​

Will declarative API's, infra as code capabilities affect your decision when choosing a tool/platform?

View Poll

https://redd.it/lc1qnb
@r_devops

Looking for simple local build system

I'm looking for some kind of simple generic build system that will run entirely locally on my Windows machine (not Docker) that will basically do 4 things:

Execute a sequence of commands
Capture the commands and output
Collect generated files from a build and put them somewhere
Maintain the history of builds, logs, and files

Even better if it could automatically do a lot of things a CI/CD system would do, e.g.

Checkout a Git revision (from a locally hosted Git repo, or a Github repo)
Setup environment variables
Run tests
Generate some reports
Generate a manifest
Identify and collect artifacts

https://redd.it/lby4ta
@r_devops