Business strategy, Devops, and dollars

I've been struggling for the past few years around the business justification behind devops that I'd be interested in hearing people's opinion about. Its fun and exciting to geek out about the engineering involved with the field, but I feel that it very rarely one of those mission critical aspects that actually matters to my employers and the people writing my paycheck.

Can I shift-left people, process, and technology on a CI/CD pipeline? Yeah, no problem. How about geo-distributed/zero-downtime deployments? You bet. Event-driven distributed systems with HA/SLAs/DR/Business Analytics/FinOps while still adhering to regulations, corporate policies, and ITIL change management while taking an iterative approach towards delivery to navigate uncertainty that can still provide some amount of value if the project/program gets cancelled half way through? Its a hassle and I hope all interested parties are ready for a challenging conversation, but sure. But to what end? How does this move the needle in a way that matters?

What I've struggled with boils down to a single question: in what ways is devops part of your business strategy? By that I mean, the arena your company competes in and how you intend to win over others in the space. If you envision the conversations between your management, executives, board members, or shareholders, how does devops come up? My experience is that quality, availability, support, and analytics is implied or assumed at best. More often than not, its considered part of the cost of doing business that they'd just as soon (and often do, sometimes even rightfully so) deprioritize when the going gets tough. I can't imagine devops showing up as a line item on an A3 or a future state operating model.

I feel like the total cost of ownership behind devops adoption is in the tens of millions of dollars range for most medium-sized businesses (say a 5-person "devops" team, development changes, cloud/tooling spend along with custom integrations, education/training, countless hours in meetings, the opportunity cost of time being taken away from feature delivery, etc). Its been a trend over the past decade that even though I've worked in this space for a long time as an engineer, advisor, and starting to border on organizational management that if asked to justify my paycheck for a particular employer or the overall value of the devops in general, I simply couldn't do it. DevOps techniques and capabilities aren't cheap -- a single, very talented, senior level engineer probably costs the company ~200-250k in total compensation. This is a space I work in and can really matter to some companies, sectors, and products but not the type of "industry best practice" that every business gets a measurable ROI from contrary to popular belief/salesmanship/conferences/etc.

Is this just me being out in left field or chasing my own tail around impossible expectations? I'd be interested to hear how others grapple this question.

https://redd.it/1btodmf
@r_devops

How do you run one-off scripts in Github Actions?

I have a Flask app that uses Flask-Migrate and SQLAlchemy however, one thing that is constantly bugging me is that whilst I have a Github Actions to build, test and deploy, handling data migrations is something that I can never do within the pipeline.

Two specific examples of my what I want to use a one-off scripts :

1. I can generate Alembic migrations but I am never sure where exactly to run a SQL script to backfill data? And ensure it only runs the one-time I do so.
2. Updating the db say I wanted to update the db to facilitate a new feature (no schema change) or cleaning up bad data. Again running once.

I have seen examples where people used their migration tool to also manage existing data, correct me if I am wrong, these tools are supposed to only handle schema changes and not the existing data?

How do you usually do it? Currently, I just manually inspect by connecting prod db through DataGrip but I know this isn't best practice.

​

https://redd.it/1btuphk
@r_devops

What do you guys like getting at trade shows?

Hi all,

I'm helping the GTM team at a trade show. Being an engineer myself, I hate being sold to, but you know, business... Anyhow, they've asked what would be some good things to bring to the booth to start conversations. I've been to some of these things and know that most booths will bring t-shirts, stickers, pens, cables, etc.

Is there anything useful you've gotten at a trade show? Anything a bit more unique or something that stands out?

Any help is greatly appreciated! Thanks in advance.

https://redd.it/1btyh9r
@r_devops

Introducing Kusion v0.11.0: might be helpful for boosting collaboration between application developers and platform engineers

Declaration: I'm one of the members of the KusionStack team

Hi guys, I'm happy and humble to introduce the latest version of Kusion, an opensource platform orchestration tool that might be a booster for collaboration between application developers and platform engineers. Platform engineers can design, build and publish a reusable building block for application resources and config data, which can be easily adopted by application developers. More info can be found here.

https://redd.it/1btz48i
@r_devops

Do you enjoy Gitlab CI?

I am really annoyed with it. I didn't use much else, so hard to compare. But what really puts me off is the way it handles variables - one has to work around the fact that it doesn't expand them when passing them to downstream - typical thing: define some workflow, set up some stuff like image names by using stuff like commit or tag. Pass such name to downstream multi-project pipeline and guess what. Nothing works because it gets passed as string to downstream and it is expanded in downstream project, with its variables. Or am i missing a clever trick?

Other annoyance is that i have to either build super image containing everything and do everything in one job (build and package mostly) or pass the stuff around as artifacts, which is painfully slow.

Sorry, if i am annoying you,, i really wanted to vent my frustration. By the way i am grateful that GitLab is open source and free and overall pretty OK. I'd love to hear your experience!

https://redd.it/1bu1hlw
@r_devops

Resume Feedback Please

I'm currently looking for either a Cloud Engineering or DevOps Engineering position, I've not been getting much of a response. A few lukewarm responses but no actual interviews in a while.

I have a strong background in System Administration working at a Managed Service Provider, with one of them requiring some Cloud administration. I switched over to Cloud Engineering and ended up being the 'devops' guy in the projects that I was working on because of my sysadmin background. I'm looking to switch to another devops role.

Would one of you kind folks scan my resume please? I'm going to write it from scratch. But I'd love some feedback, maybe there is something that I'm missing.

​

Full Resume

Imgur01

Imgur02

https://redd.it/1bu31dt
@r_devops

How does your team approach problems you don't have much experience with ?

So, recently landed in a team that seems to be responsible for everything from start to an end while consisting of few ppl.

Managing DBs, profiling queries ? Check

Defining standards for security ? Check

Managing two clouds in parallel ? Check

Managing On-Call, Monitoring, Logs, ES ? Check

Implementing Zero-Trust ? Check

Implementing Developer Platform ? Check

Maintaining Terraspace, Kafka, Rabbit, Kubernetes, Clickhouse, Vault, Okta, Backstage, Jenkins, GitLab and few more.

A lot of moving parts. All of us have pretty solid understanding of the bigger picture, but when it comes to details - when stuff is actually implemented, a lot of those details seem to fall through and are not taken into consideration during plannings/brainstorming.

This is mostly due to inexperience with quite a few of those managed or not services (team rotated over the years).

I'm curious how other teams are managing complex setups to make railroad tracks meet in place instead of having to adjust everything at the end coz stuff was not discussed and aligned correctly beforehand?!

I know it's sometimes impossible to think about everything, but would like to decrease as much as possible those turnouts at the later stages of development.

What's your go to strategy ?

https://redd.it/1bu2p62
@r_devops

Devops US citizen sponsored abroad?

I'm a senior engineer and a US citizen. I'd like to move long term to another country in EU or Asia. I'm aware of the pay difference and the "digital nomad visas". I'm asking specifically if any Devops / SRE US citizens have been sponsored by a country abroad?

https://redd.it/1bu49ns
@r_devops

Az Solution Architect exam, Memorization is sick

So I'm studying to take the Az Solution Architect expert exam, and I'm tired having to Memorize SKU features of each service and it seems that it's a requirement for the exam.
Really guys if IT is about memorization then what's the difference between us and medicine field, this should be included in resources alongside the exam "same as k8s resources in CKA exam"; because in actual world we won't be remembering these as lawyers remember articles to perform in court. On the contrary we will just open Microsoft docs and check about SKUs features and that's it!

https://redd.it/1bu7cd9
@r_devops

Cloud Engineer in Singapore

Hello Everyone!

I am curious as to how someone from the States can get a job in Singapore as a Cloud Engineer/DevOps Engineer. I am also curious as to what the salaries are in Singapore, and overall how the market is over there.

I am a Cloud/DevOps Engineer with five years of experience looking to move to Singapore, thus wanted to learn about some of the experiences y'all have had.

Thanks!

https://redd.it/1bu8os3
@r_devops

How to focus GitLeaks on a smaller subset

We have a few mega-repos with several if not dozens of solutions. I wanted to see if I could make a GitLeaks YAML task that would focus in on a particular path. The documentation seems to say that we could do it like
scanlocation: '$(Build.SourcesDirectory)/path/to/your/subfolder'


But in practice it doesn't seem to be finding it. Has anyone done here had some experience with this?

https://redd.it/1bua53g
@r_devops

Terraform workspace: how to do it?

Hello,

I am experimenting with workspace for handling three environments (dev/test, staging and prod)...

I got it work under the assumption that the three environments are similar in terms of type and quantities of services... but this is not the case in real life...

let me explain:

my terraform (for now/testing) is pretty simpler:

* **in dev/test**: it creates an S3 Bucket
* **in staging:** it creates one S3 Bucket + 2 VMs
* **in prod:** it creates - one S3 Bucket + 4 VMs

the idea is to have a terraform that allows to create 1 S3 + 4 VMs, but in each environment it should just create a subset...

how do I get this done? is there a way if .tfvars does not have a variable, it skip the instruction?

any suggestions?



https://redd.it/1bu9uvm
@r_devops

Dealing with Burnout

In the last year, I’ve basically burnt out. In just the last 4 months, I’ve worked almost every weekend, worked late almost every day, and there has been exactly 1 week that I’ve worked 40 hours or less.

I used to genuinely enjoy my job, liked coming to work, and liked the problems I was working on and people I work with. I wake up every day dreading going in, every single thing everybody says gets on my nerves, every task im assigned pisses me off, and every week I tell myself I’m going to protect my time and not work this weekend, just to do the same.

A couple weeks ago my manager said something along the lines of “u/marsmanify likes working weekends and nights” and when I called out that I didn’t he replied “could’ve fooled me”.

I am given tasks that HAVE to be done after hours or on weekends. Sure I could just not do it, and let the tickets roll in Monday morning, but it would be my ass on the line if I did.

I have no degree, no friends/family in the city I live in, and the job market is tough so just quitting is more or less out of the picture.

What can I do to manage burnout and not ruin my career by snapping on someone? Taking vacation isn’t an option because I had to submit my time off through June in January or February

Edit: I average 50-60 hour weeks, and I’ve worked 24-30 hours straight several times. 40 hours is what my work week should be, not what I’m currently working.

https://redd.it/1bue1k4
@r_devops

can packer be told to not create an image?

Is it possible to have `packer` not create the image? If you wanted to troubleshoot some aspects of provisioning and not wait for the image to get built to tweak code and what not. Basically just a time saver. Not sure if any of this made sense...

https://redd.it/1buf6zu
@r_devops

Design requirements of a new configuration-management system?

I am building a new [configuration management system][1]; of the old-regime ([CFEngine][2], [Puppet][3], [Chef][4], [Ansible][5]).

Here are the [proposed] constituent parts, all of which can be assumed to have CLI, GUI, REST API, RPC, web-frontend, mobile-frontend, and shared-library interfaces:

- **System and network de/provisioner**: abstracting APIs of various cloud and bare-metal environments
- **Registry**: What the above↑ populates, can be completely proxied—and just call the vendor(s) APIs—or built on a separate key/value store (possibly fancy, e.g., with DNS builtin); or some combination (with the key/value store being the cache)
- **Packager**: images; archives; OS|distro specific packages
- **Package managers *(large number of these)***: rather than 'recipes'/scripting create a different—yet composable/dependable—package-manager for each 'thing'; where 'thing' could be a specific web-server, database (incl. clustered variants), WordPress, Open edX… all versioned
- **Configuration updater**: Utility functionality to update (idempotently) configurations, e.g., add/remove a route in [nginx][6]
- **Execute on remote server(s)**: which might be as simple as a [polyshell][7] `curl`'d or `scp`'d that proceeds to run the desired package manager(s) using aforementioned *Registry* to locate & authenticate, and update what was run successfully where. This should let it be cross-platform (Windows, Linux, macOS, OpenBSD, SunOS, &etc.)

By design, it would not implement features like:

- **Load-balancing**
- **Health checking**
- **Secret handling / vault** - NOTE: will however need to implement integration between this and *Registry*
- **Web-server**
- **DNS server**
- **Other protocol specific servers** (e.g., email)
- **CI/CD**
- **Security scanning**
- **Distributed system guarantees**

(as these are the domain of system(s) deployed by aforementioned package manager(s) and/or vendor-provided services [e.g., cloud DNS])

---

What is missing from this new system design?

[1]: https://en.wikipedia.org/wiki/Software_configuration_management
[2]: https://en.wikipedia.org/wiki/CFEngine
[3]: https://en.wikipedia.org/wiki/Puppet_(software)
[4]: https://en.wikipedia.org/wiki/Progress_Chef
[5]: https://en.wikipedia.org/wiki/Ansible_(software)
[6]: https://en.wikipedia.org/wiki/Nginx
[7]: https://github.com/llamasoft/polyshell

https://redd.it/1buibmf
@r_devops

Where should I focus first in order to go from a tradicional tech stack to a devops approach?

So here is my situation, I'm a full-stack developer (Real full stack, I do from server configuration to frontend development, and even less related tasks like QA or data analysis) that in the past years have assumed a mixed DevOps engineer and development role (Have moved development environments to docker and enhanced the internal it processes overal). Lately I'm working into developing my skills and started tweaking with some tools like Jira, GitHub actions, some of the aws tools an stuff like that, but the more I go down the rabbit hole the more I need to learn (or even relean from a new perspective).
We are currently working with aws to almost everything, repos hosted on GitHub and not many automations on either. We use a custom ticket/issue system that we can't drop easily to use a more standard approach but I want to look for a two-way synchronization with something like Jira using webhooks and/or apis. Automatic testing and code style enforcement also aren't implemented yet. We have tons of independent tools in a pretty diverse environment but we lack a tool to centralize all those so we can have less tedious manual tasks.
I think the first thing is setting the correct DevOps mindset in order to start learning things, testing them on my personal projects and then take them to my job. I would like to get some good material to read/watch (I don't really mind) in order to help me getting my ideas sorted and having a global vision of the options out there to help me decide where shall I focus first given the current status of my job tech stack. I don't mind looking at paid courses as in my job we have credits to buy courses that gets lost if we don't use them.
I have read the pinned post, but most of the material I found is more oriented to learn from scratch rather than updating knowledge.
PS: Sorry, I have ended rambling instead of going to the point. I have tried to condensate the post, but I ended with my ADHD brain dumping info on a post.

https://redd.it/1buitid
@r_devops

Exciting News! Microsoft is now offering FREE certification courses.

Yes, you read that right: No Fee ✔️ No Subscription ✔️ No Registration Required ✔️ Just grab your laptop and start learning!

1. Microsoft Azure Fundamentals

\-> Course Link: https://learn.microsoft.com/en-gb/training/courses/az-900t00?wt.mc_id=studentamb_371890

2. Python for beginners

\-> Course Link: https://learn.microsoft.com/en-us/training/paths/beginner-python/?wt.mc_id=studentamb_371890

3. Introduction to GitHub

\-> Course Link: https://learn.microsoft.com/en-us/training/modules/introduction-to-github/?wt.mc_id=studentamb_371890

4. Microsoft 365 Fundamentals

\-> Course Link: https://learn.microsoft.com/en-us/training/courses/ms-900t01?wt.mc_id=studentamb_371890

5. Develop Generative AI Solutions with Azure OpenAI Service

\-> Course Link: https://learn.microsoft.com/en-gb/training/courses/ai-050t00?wt.mc_id=studentamb_371890

6. Developing Solutions for Microsoft Azure

\-> Course Link: https://learn.microsoft.com/en-gb/training/courses/az-204t00?wt.mc_id=studentamb_371890

7. Microsoft Azure Administrator

\-> Course Link: https://learn.microsoft.com/en-gb/training/courses/az-104t00?wt.mc_id=studentamb_371890

8. Configuring and Operating Microsoft Azure Virtual Desktop

\-> Course Link: https://learn.microsoft.com/en-gb/training/courses/az-140t00?wt.mc_id=studentamb_371890

9. Designing Microsoft Azure Infrastructure Solutions

\-> Course Link: https://learn.microsoft.com/en-gb/training/courses/az-305t00?wt.mc_id=studentamb_371890

10. Microsoft Azure Data Fundamentals

\-> Course Link: https://learn.microsoft.com/en-gb/training/courses/dp-900t00?wt.mc_id=studentamb_371890

11. Microsoft Azure AI Fundamentals

\-> Course Link: https://learn.microsoft.com/en-gb/training/courses/ai-900t00?wt.mc_id=studentamb_371890

12. Designing and Implementing a Microsoft Azure AI Solution

\-> Course Link: https://learn.microsoft.com/en-gb/training/courses/ai-102t00?wt.mc_id=studentamb_371890

13. Microsoft Security, Compliance, and Identity Fundamentals

\-> Course Link: https://learn.microsoft.com/en-gb/training/courses/sc-900t00?wt.mc_id=studentamb_371890

14 .Data Engineering on Microsoft Azure

\-> Course Link: https://learn.microsoft.com/en-gb/training/courses/dp-203t00?wt.mc_id=studentamb_371890

15. Microsoft Security Operations Analyst

\-> Course Link: https://learn.microsoft.com/en-gb/training/courses/sc-200t00?wt.mc_id=studentamb_371890

Some Extras!

1. Elevate coding prowess with Visual Studio Code Docs:

Link: https://code.visualstudio.com/?wt.mc_id=studentamb_371890

2. Gain insights from Microsoft's developers:

Link: https://devblogs.microsoft.com/?wt.mc_id=studentamb_371890

3. Explore Azure Cloud Services:

Link: https://azure.microsoft.com/en-in/?wt.mc_id=studentamb_371890

4. Join Microsoft Tech Community for collaboration:

Link: https://techcommunity.microsoft.com/?wt.mc_id=studentamb_371890

That's a wrap!

Hope this was helpful for you.

https://redd.it/1buk9sh
@r_devops

Any free tool to analyse tomcat access logs?

Team,

My Tomcat running on RHEL8, I want to analyse localhost access log generated by tomcat in pattern

>pattern="%h %l %u %t \"%r\" %s %b %D %F"

Tried internet, didn't find any suitable tool for this.

grep from linux and analyse is tedious task right now and also don't have access to client machine to install monitoring tool, I do copy logs to local machine and then analyse.



I don't need fancy, but atleast basic all details and STATS would help

https://redd.it/1buj4ke
@r_devops

Multi-Tenant Customer deployment on Kubernetes

Hi Folks,


My team is planning to transition of current single-tenant application to a Software-as-a-Service (SaaS) model, which will serve multiple tenants (customers) on Kubernetes.


We're currently using GitHub Actions for continuous integration (CI), GitOps for continuous deployment (CD), and Kubernetes for deployment of the multi-tenant application.


I'm wondering about the best approach to achieve this transition. I'm not very much familiar with Helm, so I'm curious if it's possible to accomplish this without Helm.
Your precious thoughts will be help full to giving me a kick-start

https://redd.it/1bumx14
@r_devops

Seniors - What is your best piece of advice to newly transitioned DevOps engineers?

I'm under a manager who has a large ego, and highly incompetent at his job. I'm new to DevOps so I would have loved a manager/senior that can offer me great advice and tips. However, this manager is always telling me things are pointless and hiding opportunities from me. I know the best advice would be to swap jobs, but I can't do that currently - life reasons.

My manager isn't a DevOps engineer but instead a Senior Developer - So it seems to me like it might be an insecurity thing about him managing someone on topics he doesn't know much about. For example, he told me that I shouldn't bother getting Azure certifications because there a waste of time (I have absolutely none) and I apprecite that experience is more value, but I think it comes more from insecurity with him not having any certs.

What advice would you give to a new DevOps engineer trying to be the best engineer they can

https://redd.it/1bunioa
@r_devops

Gitlab issue .gitlab-ci.yml does not exist in plank project

Hey on gitlab I have 2 accounts 1 mine personally and second is for an organization, so locally branch project does not show any issue, but inside organization gitlab I have issue .gitlab-ci.yml does not exist, why it tries to run a pipeline without .gitlab-ci.yml file in blank project?

The other question around .gitlab-ci.yml in different branches in my case if I have some jobs on main branch and does not have it in fechure branch the pipeline combine them and even whe I run pipeline from fechure branch it also add job from main one. (again locally everything works correectly and pipeline on different branches only run jobs which in specific branch)

If you have any Ideas why organisation Gitlab is messed would be nice, thanks.

https://redd.it/1bungv6
@r_devops