Version control table changes with a data warehouse in a multienvironment setup?

When working with traditional databases like MySQL, we can use migration tools such as Python Django Migrations to write down changes to our tables in code. We can then run these changes in our CI/CD pipeline while deploying updates to new environments, thereby applying the changes to the environment's database. Typically, these changelogs are stored in the database itself.

For data warehouses, we could also store the changelog in the database. However, some data warehouses (like BigQuery) do not allow any data mutation within the first 30 minutes of a single row insert. This means that only read operations are allowed, making rollbacks (deleting a record) a difficult and time-consuming process.

Do you have any ideas or solutions for dealing with this issue?

https://redd.it/12fszhd
@r_devops

List of DevOps / Engineering Capabilities

Hello! I want to start a list with all DevOps, SRE and Engineering Capabilities. For example, CI, CD, Test Driven Development, Observability, Feature Flags ... all of that. Has anyone seen a list like this?

What capabilities are trending? What capabilities do you use?

Thanks!

https://redd.it/12fuxy5
@r_devops

How do you deploy ephemeral Windows Server with Terraform on-premise?

I'm trying to deploy Windows Server instances with Terraform on my Proxmox homelab.
Including the whole user setup, partitioning and software installs.
It's the preparation for doing this on VMWare Cloud Director at work... and because there is no VCD provider I sadly can't do it with Hashicorp Packer.

I found CloudBase-Init for doing this in Windows, but I'm not sure if this is the best solution.

The last goal is to do this in KubeVirt - i just now got all the ball rolling for Terraform & Cloud-Init based K8 cluster deployments.

​

Did anyone here do this already?
Any tips?


PS: We have legacy software that needs windows as the target system. I want to automate the app & environment deployment for internal testing.

https://redd.it/12fse6m
@r_devops

Finally reading "Heart of Darkness" by Conrad and slowly realizing "The Phoenix Project" is the same story but for Ops/programmers

The similarities are striking and undoubtedly intentional: allusions abound.

To my mind it's the same story:

- Individual is restless (in his office/duties)
- Individual is hired to solve a problem for a Company
- Individual discovers his predecessor was violent (by turns violence in HoD become Arrogance in TPP)
- Individual discovers his new office is cult-like
- Individual begins to dig deeper and discovers the breadth of that cult-mentality which renders his Company incapable of functioning

etc. etc.

Both are basically a re-telling of Plato's Cave too, which is curious to me.

I am only half-way through HoD and lent my copy of TPP to my boss some years ago so the likenesses are only from memory. But the more I read of HoD the more convinced I am that it's a retelling of that story.

What's your opinion on the similarities or discontinuities between the two?

Is all a stagnant Org needs is some semblance of the (independent) perspective? That seems naive in and of itself. idk.

just thinking out loud; please don't admonish me for my first draft of thoughts on this.

https://redd.it/12g6usa
@r_devops

Introducing the Comprehensive DevOps Learning Series


Greetings, r/devops community! I’m excited to announce that I’m starting a "Comprehensive DevOps Learning Series." This series of posts will be designed to provide a thorough and detailed understanding of DevOps concepts, practices, and tools for professionals with various technical backgrounds.

Whether you're just getting started or have experience in the field, this series aims to serve as a reliable resource for advancing your DevOps expertise.

Throughout this learning journey, we will explore the foundations of DevOps, practical implementation techniques, real-world applications, and much more. The series will be structured as follows:

1. Introduction to DevOps: Acquaint yourself with the historical context, underlying principles, advantages, and culture of DevOps.

2. DevOps Practices and Tools: Familiarize yourself with key DevOps tools and techniques, including CI/CD, IAC, and containerization.

3. DevOps Lifecycle: Delve into the DevOps lifecycle and learn how to implement it effectively in your projects.

4. DevOps Methodologies and Frameworks: Understand how to integrate DevOps with Agile, Scrum, Kanban, SRE, and ITIL to achieve better outcomes.

5. DevOps in the Cloud: Examine the integration of cloud platforms such as AWS, Azure, and GCP to optimize your DevOps processes.

6. DevOps Case Studies and Success Stories: Analyze real-world success stories of companies like Netflix, Etsy, Amazon, LinkedIn, and your company too!

7. DevOps Challenges and Solutions: Overcome common obstacles and pitfalls by employing industry best practices and expert insights.

8. DevOps for Small Teams and Startups: Discover how to adapt and scale DevOps principles to suit small teams and startups.

9. DevOps Career and Certification: Enhance your career with insights into job roles, skills, certifications, resume development, and interview preparation.

10. Community and Resources: Identify essential books, blogs, websites, influencers, and conferences to stay informed in the rapidly evolving DevOps landscape.

11. Conclusion/collaboration: Summarize your learning experience and explore future avenues for growth within the DevOps domain.

My goal with this series is to create an engaging, technically focused learning experience where community members can exchange ideas, resources, and insights to foster collective growth. I encourage you to actively participate in the discussions and share your knowledge, resources, challenges, and successes. With this foundation, let's embark on our journey towards a deeper understanding of DevOps principles and practices.

Notes:
Links will be added here to the different posts as I can write them. My current goal is to post weekly, and there’s going to be multiple posts per topic that I’ve outlined above.

https://redd.it/12gcft0
@r_devops

Azure resources like the ones that Adrian Cantrills have for AWS?

Looking to learn more about Azure and I've used Adrians courses and material before for AWS, which are excellent.

Any equivalents for Azure?

https://redd.it/12ge90i
@r_devops

The History and Evolution of DevOps

Series index

Hello, r/devops community! I’ve already prepared the first real post in the series:

As we dive deeper into the Comprehensive DevOps Learning Series, this post will take you through the history and evolution of DevOps. Understanding the origins of DevOps will provide valuable context for grasping its principles, methodologies, and the rationale behind its growing adoption.

The Pre-DevOps Era

Before DevOps emerged as a popular methodology, software development and IT operations were generally treated as separate entities. Developers focused on writing code, while operations teams were responsible for deploying and maintaining the software in production. This division often resulted in bottlenecks and delays, with both teams working in silos and having different goals and priorities.

The Birth of DevOps

The term "DevOps" was first coined in 2009 by Patrick Debois, a Belgian consultant and Agile practitioner, who organized the first "DevOpsDays" conference in Ghent, Belgium. The movement was inspired by the need to break down silos between development and operations teams to improve collaboration, reduce friction, and increase efficiency.

However, the seeds of DevOps were sown earlier, around 2007, with a few pioneering companies exploring ways to address the challenges of an increasingly fast-paced software development and deployment landscape. This shift can be traced back to:

Agile Development: The Agile Manifesto (2001) emphasized collaboration, iterative development, and continuous improvement. Agile principles laid the groundwork for better communication and teamwork between development and operations.

Continuous Integration (CI): CI practices, pioneered by Martin Fowler and Kent Beck, automated the integration of code changes, fostering a culture of collaboration and paving the way for DevOps.

Lean Manufacturing: DevOps also drew inspiration from the Toyota Production System, a precursor to Lean Manufacturing. The focus on minimizing waste, streamlining processes, and incorporating customer feedback resonated with the goals of DevOps.
DevOps Evolution and Adoption


Since its inception, DevOps has evolved significantly, driven by technological advancements and changing organizational needs. Key milestones in its evolution include:

Infrastructure as Code (IaC): The emergence of IaC enabled IT infrastructure to be managed through code, making it versionable, reusable, and reproducible. Tools like Puppet, Chef, and Ansible contributed to IaC's growing popularity.
Containerization: The introduction of containerization technologies like Docker and Kubernetes transformed the way applications are developed, deployed, and managed, making it easier to maintain consistency across environments.

Cloud Computing: The rise of cloud platforms such as AWS, Azure, and GCP provided scalable, flexible, and cost-effective resources that significantly enhanced the potential of DevOps practices.

Microservices Architecture: The transition from monolithic to microservices architecture allowed developers to build, test, and deploy individual components of an application independently, further enhancing the benefits of DevOps.

DevSecOps: The growing importance of security in the software development lifecycle led to the integration of security practices into the DevOps pipeline, giving rise to the concept of DevSecOps.

Site Reliability Engineering (SRE): Google's SRE framework brought a more structured and measurable approach to managing large-scale systems, aligning well with the goals of DevOps.
As more organizations recognized the value of DevOps, its adoption grew, and it became an essential approach for companies aiming to deliver high-quality software quickly and efficiently.

In the upcoming posts, we will delve deeper into the principles, benefits, and culture of DevOps, as well as practical tools

and techniques that have emerged from this evolution.

Share your thoughts on the history of DevOps in the comments below.

https://redd.it/12ge71h
@r_devops

Best study tools to learn azure and azure devops?

Hi guys I need to learn Azure as a whole and Azure Devops as a focus. What courses, youtube videos, or instructors do you recommend? I have my AWS Solutions Architect professional exam, so I have a fair bit of cloud experience, just nothing on azure and need to learn it asap for a new role.

https://redd.it/12gelug
@r_devops

DevOps to Product owner

Hello everyone. I've been struggling over the past few days to decide which path I need to pursue and would appreciate your thoughts. I have 4.2 years of IT experience and 2 years as a DevOps support engineer, and I have been offered two positions: associate software engineer and associate DevOps engineer. I've spent the last 6 months upskilling myself in Terraform, AWS, Kubernetes, Gitlab CICD, and other DevOps tools to become a DevOps engineer but I m not sure if I will love this in the future.

I am now confused about what role to take. If I opt for the Associate DevOps Engineer position, I will only be in the DevOps space, such as Senior DevOps or Cloud Architect. If I opt for the Associate Software Engineer position, I can move later to any role (Senior Software or Product Owner). However, I want to move to a product owner role in the future as I would have better influence on products, but I am not sure which role would be most suited if I decide later to become a Product Owner.

I discussed with a few product owners in my company, and they advised me. It would be tough for DevOps Engineers to transition to product owner compared to Software Engineers as they have more knowledge of product compared to others

https://redd.it/12gj4es
@r_devops

How should I approaching wanting to move to a different team under the same senior director?

I work as a SysAdmin and support engineer for complex system legacy app that is being decoupled into new applications using React UI, Azure, Google Anthos, MySQL, etc. My team was rebranded as "DevOps Support Engineers," which mainly involves running Jenkins jobs, monitoring Kubernetes pods, and Splunk alerts. We contact DevOps engineers or dev leads to fix issues. My daily role is still supporting the legacy app doing SysAdmin tasks, but also incident, problem and change management. My co-workers are a little more involved in the legacy app working with PL/SQL code, etc.

I want to transition to the DevOps team and I'm working on improving my skills in Kubernetes, Terraform, Google Anthos, Azure, and Jenkins (Groovy).

I have a good relationship with our senior director who manages both teams. How should I approach him and/or direct manager about joining the DevOps team?

https://redd.it/12gkeso
@r_devops

Do I really need DevOps?

Hello,

I am managing an automation (RPA) practice in my organization which is in the business of Asset Management.

The automation practice was established in my org two years ago. From then till today, the automation infrastructure (we're on UiPath) is on a single node on-premise setup. In these two years, a lot of business verticals have become highly dependent on automation for every day critical business tasks. Therefore we have come to a conclusion that we need to plan for a high-availability and disaster recovery setup.

We've also taken a call to build it on Azure + on-premise (Hybrid model). Now my question is, should we plan for introducing DevOps into the automation practice? Could this be a use case where DevOps can make an introduction and spread from there?

My primary reason for DevOps is that:

1. We're going to have a complicated infra setup, it will be easier to maintain the environment across DEV/UAT/PROD
2. It will also help us reduce deployment time

Note: My organization does not have DevOps as of now.

https://redd.it/12glare
@r_devops

Hired as an external DevOps engineer with unclear objective - what should I do?

I've been hired as an external DevOps engineer into a project and I don't know how I feel about it. I share my situation here because some opinions of more experienced engineers could be valuable for me and perhaps others are or have been in similar situations and would like to discuss it or learn from it.

# Description of the situation

The project has been built up by three developers over the last three years and is fairly complex. The problem is that all the developers will leave the company in the next 1-3 months. I have been told by my manager that it is is my job now to "conserve the knowledge". It is planned to hire additional developers at a later point, but it is not communicated with me when this is planned and what skillsets these new team members will or should have.

They already have documented a lot of their processes, but without having worked in the team, I feel like it is impossible to know whether what has already been documented is complete or should be extended.

I could go through the documentation and look for things that should be documented or add notes to the existing docs where I believe something is missing. But there is no way for me to ensure that everything will be documented. One person cannot "conserve knowledge" in a couple of months. I could do this after having worked in this project for several months, but not in the short timeframe of 1-3 months.

Another difficulty is that most of the code is written in languages I don't have any experience with, which was communicated before I started. I have experience related to the "DevOps" technologies they use, but apart from that I really feel a bit lost with the existing codebase. Again, they knew from the beginning which technologies I had experience with and which not.

Another point is that I don't feel like I learn so much about the project by reading their documentation and code. The only way to learn what is going on in this project is to actively work on it. So I proposed to work on open tasks while checking documentation to get more familiar with everything. The response was that there are currently no open tasks for my skillset.

This is why I believe my job doesn't really make any sense. One person cannot "conserve knowledge" in a couple of months. This is the job of the developers during development in the form writing documentation and assuring a certain level of quality regarding the documentation. This development process should always see the possibility that people may leave at some point. It cannot be fixed by one person in a few months.

What do you think? Am I right? What should I do now to not end up in a situation that will be very difficult to handle for me?

https://redd.it/12gp2jo
@r_devops

Which Cloud provider to build infra for LLM-based app

Hi everyone,

I am working on an app leveraging LLMs at its core (we use ChatGPT as our main AI API).

I am having a look at the different Cloud providers to see which one would be the best to host my infrastructure.

AWS seems to be the regular choice for startups to build their Cloud infra, but because I will rely heavily on ChatGPT API, Azure seems to be a good choice as well.

My criteria are:

* Pricing (are there big differences in VMs cost)
* Off-the-shelf products available on the marketplace (from databases to compute, through APIs)
* Credits for startups (how much and for how long)

I did not mention GCP but could be a good choice as well.

What is your opinion on this choice of CSP for an LLM-based startup? I would love to have your opinion!

[View Poll](https://www.reddit.com/poll/12gwfey)

https://redd.it/12gwfey
@r_devops

What did your career up until now look like?

I completely abused ChatGPT to create a roadmap of the 30 most important fields in DevOps (without the tooling).
It spilled out a skill matrix too, but I still don't know what to make of it.

So reading about other people's career development might be the best way to get a feeling for what might come.

So, how did you get where you are now?
I love stories of people that earned their worth.

For me, I literally have no 'formal' education but got a job as 1st level support with some sysadmin tasks, later getting much further into networking and virtualization, worked with decades old hardware essential for production and had to code a simple replacement for the software running on it. After we got a new teammate our workload reduced so far that I had the time to use containers and implement real monitoring and went on a hunt for flaws in our security.
From starting in that shithole up to my present role in "DevOps" for another employer it took only two years and the last pieces of my sanity.
Now I'm only a glorified linux admin and somehow have to get our workflows and tooling from ~2014 up to today's standards. Mostly alone with some help from our normal sysadmins that I have to teach ~everything~ in the long run.
It's a grind and I don't know if it's time well used.
Is working yourself up through the history of CICD, system/cloud administration and software development the way to do this? I mean, a high peak is nothing without a wide base, right?
Right?

https://redd.it/12guyym
@r_devops

Containerization Article

Hey Everyone!


I recently wrote an introductory article on containerization. I'd love to get this subreddits thoughts (when I asked ChatGPT what it thought, it said it liked it). I have plans to write more technical articles in the future, and I figured you'd be the best people to get feedback from.

https://medium.com/@ace2930/the-power-of-containerization-why-you-should-care-ded839e1b4b9


Some more about me: I currently work as a Senior ML Engineer, and have been doing deployment/DevOps-y work for 4 years now. I do a mix of both application coding and deployment/IaC/dev tooling, and looking to expand into technical writing.


Feel free to DM me about it too!

https://redd.it/12h3q8s
@r_devops

Chef berks package

For the chef tools, there is a command "berks package" that is used to " to vendor, and then archive the dependencies of a Berksfile."

What does vendor mean here, and why would it be necessary to archive Berksfile dependencies?

https://redd.it/12h5hfh
@r_devops

Do you use external uptime monitoring?

Hello,

I am toying with an idea to run a boutique uptime monitoring service for indie devs and early startups and I would like to ask you all if you actually use any external uptime monitoring service?

By uptime monitoring service I mean one that tracks your downtime, gives you a status page, or perhaps even manages your on-call rotations.

If yes, which one do you use and what you like or dislike about it?

https://redd.it/12h7ou2
@r_devops

Automation of your team's local environment setup

I would be interested to hear your approach about how you implementing automated processes for setting up local dev environments for new hires or for new devices and to know what tools and processes you have used to make the process more efficient, especially when onboarding new team members or using multiple laptops.

For instance, one approach could be to use scripts that download all the necessary binaries in their pinned versions for day-to-day tasks (eg. `kubectl`) and also set up the necessary configurations and access to different environments.

https://redd.it/12hapa1
@r_devops

terracove - open-source to instantly test the health of your terraform/terragrunt repository

https://github.com/jatalocks/terracove

Hey r/devops.

After much trial and error, I made a prototype of a tool that my team pretty much needed but that didn't yet exist to my knowledge.

What my problem was originally is that we had a lot of technical debt and drift on our terraform/terragrunt repositories, and tons of leftover modules that haven't been planned in years of missing CI. Instead of going one by one, planning it and checking if it's valid, I built terracove.

What it does in parallel is basically init/plan/show using terratest on every subdirectory on your repository tree or provided paths. The output is either a JSON summary or a custom made Junit XML test file you can ingest into your tests reader. It took it around 8 minutes to map the entirety of our bloated repository.

All you have to do to set it up is either:

brew tap jatalocks/jatalocks
brew install terracove
terracove --junit --json .

Or (with the correct ENV variables and secrets if necessary):

docker run --rm -v $(pwd):/data ghcr.io/jatalocks/terracove /data --json --junit

I hope anyone finds this useful, Any feedback is welcome!

https://redd.it/12hcafq
@r_devops

Quick question about release management

I'm interested in understanding how many hours a week are spent (in total) company-wide to ensure that software version status information is accurate.

For example, if every week a release manager needs to meet with 3 team leaders for 10 minutes each, then that constitutes 1 hour a week (2 participants * 3 meetings * 10 minutes).

Consider how many hours it takes to:

1. ensure that all tickets are up-to-date and that their status is accurate

1. Compile a release's overall status - to understand what has been deployed to integration/staging, what has been tested, etc.

2. attend meetings (even one-on-one) to sync on status

​

Thanks!

https://redd.it/12hemca
@r_devops