Reddit DevOps
269 subscribers
5 photos
31K links
Reddit DevOps. #devops
Thanks @reddit2telegram and @r_channels
Download Telegram
Building a devops "compiler"

Hey all, I'd like to discuss with this community a concept I've been working on and refining for a little over a year now, and would like to see if other people here would think this is a step in the right direction or misguided in some way.

I've worked in several different places as a software engineer and/or infrastructure engineer for years now, although in the last few I've found a real passion in building and optimizing automated infrastructure. I've found an incredible amount of satisfaction in spending months building out what is essentially a "factory" for delivering code and automating its various facets. Once it all comes together, you end up with this really nice, well-oiled machine that takes code as input and outputs running, certified, and observable applications.

What I've noticed doing this is that there isn't actually that much difference between what various software development teams want. I will narrow the domain down to the following types of applications:

- Listening for and responding to HTTP requests at some URL (e.g. websites, APIs)
- Performing some operation on a timed schedule (e.g. cronjobs)
- Listening for and reacting to some non-HTTP event (e.g. SFTP listener, Kafka/queue-based listener)

Notably, this does not include any sort of AI application, which I have never worked with and will not pretend to understand their devops requirements, or desktop applications. I'm sure there are others as well. The general gist is that I'm applying these ideas to software that is most comfortable sitting in a cloud-run VM and reacting to a push- or pull-based event. These types of applications can typically be run "statelessly" (as in they don't need hard drive storage attached to them, not that they don't use a DB or perhaps bucket-based storage mechanism), and can be scaled up or down/replicated easily.

For this domain of applications, a software development team will want the following:

- A CI loop that listens to their PRs, runs automated functionality, performance, and security tests, provides in-PR feedback as to the results of those tests, and certifies the PR based on the results of those tests and, optionally, the existence of a human review; additionally, on a merge to main, this loop will semantically version the changes based on a well-formatted commit message and tag the commit-level with that version, and generate artifacts for this new version (code coverage results, docker image(s) with appropriate tags, helm chart(s)/kubernetes resource package with appropriate tags)
- A CD loop that listens to newly generated artifact versions and automatically updates appropriate environments with the new version; this is typically automatic for a dev environment, but dependent on engineer approval for staging or prod environments
- A networking stack that automatically provides their application with a URL to communicate with once deployed, DNS for that URL, and TLS certs that are regularly rotated
- An observability stack that provides: a UI for quickly parsing through indexed application logs, a dashboard for visualizing metrics, a UI for searching through and using traces, and an alerting system that can utilize all three to generate notifications to the relevant on-call team
- A pre-built package they can quickly include in any codebase that will automatically instrument their code for the aforementioned observability stack, providing easy code hooks for shipping custom logs, metrics, and traces out of their code
- An identity provider connected to the networking stack that solves the authentication and UI login problem; preferably, authentication should be offloaded to the networking stack, such that a request reaching actual code can be assumed to be authenticated; the code can do further custom authorization checks once the request arrives, or this can also be offloaded to the networking stack
- A central secret and configuration management UI that automatically hooks into deployed code, provides the necessary config, and automatically rotates secrets
regularly; this system should allow for the concept of shared and app-specific configuration, particularly for microservice-based architectures
- A pre-built configuration package they can easily include into any codebase that knows how to read the automatically provided config
- Infrastructure that will automatically scale their code up or down based on need, and scale itself up or down based on how much the code needs to be scaled

I'm generalizing and I'm sure there are some additional details I could add, but on the whole, the work I've seen devops teams pour time into has typically been to implement some subset of the above. I have seen teams have varying levels of success doing so with a whole bunch of different tools, some home-grown, some commercial, some open-source, but all of them quite different. And I'm sick of it.

We all want the same thing, and there is, GENERALLY SPEAKING, an "ideal" devops pipeline and infrastructure out there that checks all the boxes, but it requires strong interplay between assumptions made by the infrastructure and codebases fulfilling those assumptions. That brings me to my core thesis:

We need to start viewing devops and infrastructure like compilers, not editable pipelines

When you start writing code, you don't rewrite your compiler or your interpreter to fix your janky code. The compiler/interpreter knows what's right and what's wrong and publishes a clear interface for how you need to be doing things, and if you do them wrong, you fix your code, not the compiler/interpreter. Too many times have I seen teams force changes to a pipeline due to poor coding and software development practices rather than problems in a pipeline. Once those poor practices have been normalized it is incredibly difficult to change them. If you normalize good practices from day 1, everyone gets used to it and no one grumbles.

Our devops pipelines should provide a clear interface for what is considered acceptable code, and code should be modified to fit that interface, not the other way around

If the code fulfills the interface, infrastructure can make SO MANY assumptions that provides that same code a huge number of pre-configured systems and advantages that then don't need to be configured by a dedicated devops team.

A crucial part of accomplishing this successfully is providing a very easy-to-use entrypoint to that interface. Usually, I implement this in the form of "template" repos that act as quickstarts; this is a repo in the target coding language that has a main file with a clear entry point to start writing code and all the observability and config packages pre-configured. This repo will also typically include a compliant Dockerfile and Makefile. Make sure that as few build files have to be configured as possible. The idea here is to create a "pit of success": the easiest path is the correct path. A developer can clone the repo, start writing code, push the new repo, and the pipelines take care of the rest.

The trick is always to balance out maintaining enough customizability that most applications can fit the interface, without making it so generic that making things works requires a lot of app-specific work, but I believe there is a balance that fits MOST applications; not necessarily ALL applications, but MOST.

I've been building out such a system using exclusively open-source tools for the past year and I think it works quite well and is applicable to most of the companies I've worked at. What I like about it is there's no vendor lock-in and can be deployed single- or multi-tenant, so it could be deployed entirely separately or used as a PaaS. I'm having some trouble motivating myself to finish it recently and figured I'd get feedback from the devops community to hear if:

1. They apply these sorts of principles at their own jobs

2. If so, have they found it to be useful, and have they found success in doing so; if not, what technical or non-technical barriers have they encountered

3. If they know of any tools or companies that accomplish this or some subset of this (and
I don't mean things like DataDog or CircleCI, which in my mind fulfill a very small, specific subset of these)

If you made it this far thank you very much for reading!

https://redd.it/11jflm2
@r_devops
Damn I just passed Cisco Certified DevNet Specialist - DevOps specialisation!!!


OMG I am over the moon!!!!

I can’t even write still shaking lol!!!

Cisco DevNet Professional I managed to bypass the associate level!!!

A lot of overlapping concepts with this and GCP Professional DevOps

I will write on this later…

https://redd.it/11jln8y
@r_devops
Licensing temporary Windows Servers on KVM Hosts

So I’m trying to figure out how to license Windows Server for temporary dev & test servers. How do you guys do it? Do you just use the trial period or do you use some form of license?
The problem I see with the trial period is customising the image while still not starting the counter and using the current date.

Any tips and thoughts are appreciated!

https://redd.it/11jle3r
@r_devops
From Startup to Success: Why Adopting a DevOps Approach Is Essential

Learn why a DevOps approach is crucial for startup businesses to achieve success. From improving collaboration to accelerating time to market, discover how DevOps practices can help your business thrive in the competitive landscape. Don't fall behind - make DevOps a priority for your startup today. Scoop of Everything about Devops for Your Business Growth.

https://redd.it/11jqtv6
@r_devops
Getting CISCO Networking Certifications

How useful are Cisco certifications in terms of the knowledge they could provide for DevOps (If I already know the basics of Networking)? Is that worth going deeper in knowledge about networking with certifications from CISCO

https://redd.it/11k1wgg
@r_devops
Weird git branch rule

I am working on a place that has a weird branch rule.They said that 2 branches could not be merged into the main, under no circumstances.These branches are used to test stuff.If you want to test something, you should merge your branch in the test branch or another specific branch for it. Since you cannot merge these branches back to the main, what is the point?They use gitlab flow (at least that is what they claim)

It seems like these break everything I know about CI. Does anyone worked in something similar?

Edit: I forgot to mention, we are no allowed to create branch from theses 2 branchs. Theses branch's only run unit tests.

https://redd.it/11k1d65
@r_devops
How do you create your Kubernetes configuration?

I was wondering how do you create your Kubernetes resources so that everything is stable and secure.
There are many ways of improving the quality of the artefacts created, but I wonder if you are actually dedicating some effort to generate the best configuration manifests possible, or getting it out of the way.

View Poll

https://redd.it/11k5m75
@r_devops
Legality of employer not paying for oncall?

My employer capped the oncall hours they pay out. In theory it's "to give us more wlb." Post layoffs we're all now going over the limit because there are fewer people oncall. So we're working for free. Is this legal? This is in California.

https://redd.it/11k4a5r
@r_devops
Should CI/CD tooling build & deploy its own configuration and infrastructure?

An ongoing conversation I'm having with a colleague regarding our Jenkins infrastructure. Our Jenkins' deployment is specified by several layers of infrastructure/configuration-as-code: terraform, Ansible, CasC, shared libraries, packer images for build agents. Each of these is specified in git, and changes require testing, validation, packaging, and release automation.

Some such changes are currently built/tested/deployed manually. Some are managed by an external system. Some are managed by Jenkins itself. We are in a long conversation about what's the most "correct" system on which to manage these processes.

Doing things manually: pros, simple to reason about; cons, risks around human error, repeatability, velocity

External system: pros, segregation of duties, no "recursive" loops; cons, getting into a turtles-all-the-way-down situation

Self-management: pros, fewer cicd platforms to maintain, pipeline code written using same syntax and elements as other pipelines; cons, if a bad release breaks CICD you might lock yourself out, any vulnerabilities in the CICD platform might be magnified if it has permission to alter itself.

This question might be more relevant for self-hosted solutions than SaaS tools. However, to those who would suggest we move away from Jenkins: (a) easier said than done, there's a decade of technical inertia behind our installation that would need to be migrated to a another tool (b) it's stable, performant, and well understood, there's no urgent business need to migrate (c) the theoretical problem would certainly exist with any other self hosted tool, and would have arguable parallels in a SaaS solution.

This isn't an urgent problem, things work mostly pretty well. I'm more interested in what an idealized architecture would look like, and how other people are approaching this topic.

https://redd.it/11k6nip
@r_devops
How many of you manage Kubernetes on remote servers vs cloud managed servers?

I have yet to meet somebody who is managing K8S on remote servers, as everybody I know is doing K8S the cloud managed way (mostly AWS). Is this industry standard at this point?

https://redd.it/11ka9hi
@r_devops
Looking for feedback on first DevOps Strategy

So my company is redesigning our websites, and as such we are building everything from scratch including our git and github repos. We are a small team (5) devs. But our website is quite sizeable (\~5000 pages) its almost entirely static content for products and documents. We have no DevOps person persay but since i have the most experience with github and our deployment tool i have become the defacto devops person. My concern is I have no actual training in this area but neither does anyone else. I would like to get some feedback on the strategy i am planning for this project.

​

So what I have planned out thus far:

We have 1 repo with all the website projects in it. I want to define 4 Special protected branches.

1. main branch: will be the primary source of code all other branches will be based on it.
2. Production branch: will be deployed to our server for live hosting
3. Staging branch: for testing features in a live environment
4. Development branch: for developing new features that require a deployment but do not need access to production databases

My idea is that we will use these branches to do testing and deployment and then create feature branches when we are added new features based on main.

​

We are using Github actions to automate deployments via Octopus Deploy to a windows 2019 IIS Server.

The workflow is fairly simple and i think will work for our intended purposes, however I am weary about keeping the git branches inline. What are the best ways to deal with this?

​

Also any feedback in general is welcome.

Thank you.

https://redd.it/11kdge1
@r_devops
Upload Ansible Scripts as Artifact in JFrog?

Is it possible to upload Ansible Script as an Artifact in JFrog? And that will be called by my AMI Setup from BitBucket repo.

https://redd.it/11kd535
@r_devops
thoughts on aws/live coding interview

Hey everyone, I I a technical interview coming up where I was told that I will have access to an aws account with basic coding, possible kubernetes, and fixing some stuff in an account. I have worked with aws for a few years hut have mever been in a "hands on live" interview like this. What is best to prepare or things you can think of that will be best ways to practice?

https://redd.it/11kc7gz
@r_devops
Would you say developing an application is DevOps?

I was hired for a DevOps role, and for some time I think I could say I was indeed working on a DevOps role (working with monitoring scripts, iac) but what I've been doing lately is developing an application. I can't say I'm not learning anything, such as working with the cloud, because this application does provision stuff on the cloud. But this application is actually part of the product that is sold by this company.

So I have to ask, would you say this is DevOps?

https://redd.it/11kja7m
@r_devops
Describe your thoughts on Agile in five words or less.

Title says it all

I'm giving a talk on this

All viewpoints welcome!

https://redd.it/11kj08e
@r_devops
1 WEEK TO GO: Register for Python Web Conf Today!

Join Pythonistas from around the world for the 5th annual Python Web Conference (March 13-17). Tickets include 5 days, 65+ live talks, expert-led tutorials, social events, an exclusive pass to all conference recordings for 90 days, cool swag and more. Don’t wait, buy your ticket today!

That's not all, we're offering an exclusive 15% discount code for past Python Web Conference attendees! To register, use the discount code "PastPWCAttendee” at checkout or check out this link 👇

https://ti.to/six-feet-up/python-web-conf-2023/discount/PastPWCAttendee

Full Schedule: [https://2023.pythonwebconf.com/schedule](https://2023.pythonwebconf.com/schedule)
Register today: https://ti.to/six-feet-up/python-web-conf-2023

See you on March 13!

https://redd.it/11kk0u5
@r_devops