Learn 9 CI/CD tools for FREE

Hi everyone, CloudClan, a community that I've co-founded is hosting a CI/CD Workshop Marathon where in-demand CI/CD tools like GitLab CI, CircleCI, Jenkins, Tekton, ArgoCD, Azure DevOps, GitHub Actions, Concourse, Cloud Build will be taught every Sunday from April 30th
You can find more details about the workshop at www.cloudclan.co/workshop

https://redd.it/12ncdss
@r_devops

Is AutoGPT gonna take over DevOps?

I'm a student. I haven't really worked for a firm as of yet. I got this ad in the emails today for a prompt course on AutoGPT :

"I just tried AutoGPT on my computer, showed it to my wife, and we stood in awe.

My Windows machine started to work autonomously on a task to research and execute a business idea. It opened my Chrome browser, visited URLs for research, downloaded information and stored it on my file system, created Python code files on my computer, debugged, improved, and executed them.

In a fully autonomous way!

It was like watching an intelligent being working on a task - AutoGPT is self-aware and uses advanced project management techniques such as PLAN, ACT, ASSESS, and ADAPT. Occasionally, it asked me if I wanted to proceed, but otherwise, it was acting autonomously.

This was nothing like prompting. This beast prompted itself, iteratively and very smartly, to solve the high-level task I had given it. Before it worked on a macro or micro problem, it researched the best way to solve it online.

As my wife was scared (rightly so) that the AI may do something shady or dangerous on my computer, I switched it off. But I'm sure I'll span a small Amazon machine soon and put it to work on one of my side projects

If you want to fully learn and master the art of effective prompting and prompt engineering, check out our full course on the academy: <AcademyName> "


I'm really scared of AI.

https://redd.it/12n6r82
@r_devops

CI/CD good practices: Should the CI/CD pipeline refactor the code?

I am working on some personal projects on GitHub and I thought setting up a super simple CI to ensure proper code formatting would be fun and useful to try/learn and get some basic of DevOps


I am setting up the pipeline for a Python project on Github using Github Actions, and I want to ensure proper code formatting with the package black. Should the CI pipeline modify the code itself if the test fails or only check if the code complies with UTF-8 standards?

What is the standard?
Should a CI/CD pipeline be able to refactor/change the code or just test it?
What are the best practices for CI/CD pipelines in this situation?
Is there any source where I can find good practices for CI/CD pipelines?


Thanks in advance :D

https://redd.it/12m0e88
@r_devops

Portainer (swarm) nginx reverse proxy

Hello,

I am desperately trying to use nginx as a reverse proxy for my portainer installed on another machine.

Here is the configuration of nginx inside server {...} in the default configuration file:

location /portainer/api/websocket {

  rewrite /portainer/api/wesocket/(.*) /api/websocket/$1 break;

  proxy_pass https://localhost:5050;

  proxy_set_header upgrade $http_upgrade;

  proxy_set_header connection 'upgrade';

  proxy_http_version 1.1;

}

location /portainer/api {

  rewrite /portainer/api/(.*) /api/$1 break;

  proxy_pass https://localhost:5050;
}

location /portainer {

  rewrite /portainer/(.*) /$1 break;

  proxy_pass https://localhost:5050;
}

Here is my yaml file for my portainer:

version: '3.2'

services:
  nginx-proxy:
    image: nginxproxy/nginx-proxy:latest
    networks:
      - proxy
    ports:
      - 9600:80
    volumes:
      - /var/run/docker.sock:/tmp/docker.sock:ro
      - ./vhost.d:/etc/nginx/vhost.d:ro
  agent:
    image: portainer/agent:latest
    environment:
      AGENT_CLUSTER_ADDR: tasks.agent
      AGENT_PORT: 9001
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /var/lib/docker/volumes:/var/lib/docker/volumes
    networks:
      - agent_network
    deploy:
      mode: global
      placement:
        constraints: [node.platform.os == linux]
  portainer:
    image: portainer/portainer-ce:latest
    command: -H tcp://tasks.agent:9001 --tlsskipverify
    volumes:
      - portainer_data:/data
    environment:
      - VIRTUAL_HOST=https://portainer.yourdomain.com
      - VIRTUAL\_PORT=9000
    ports:
      - 5050:9443
      - 8000:8000
    networks:
      - proxy
      - agent_network
    deploy:
      mode: global
      placement:
        constraints: [node.role == manager]

networks:
  proxy:
   driver: overlay
   attachable: true
  agent_network:
   driver: overlay
   attachable: true

volumes:
  portainer_data:

If in my browser I navigate to https://<my server>:5050 it works well. But if I navigate to https://<my server>/portainer I get a page not found.

I have tried many configurations and alternatives but I am unable to getting it work despite my time spent on this problem.

Thank you very much in advance for any help

https://redd.it/12nkcf8
@r_devops

I made a database with 500+ places to promote your products


Just finished compiling list of places where you can promote your product.

What is included in the list:

1500+ Startup directories
110+ Facebook groups tagged by sector
170+ guest post opportunities
50+ Slack/Discord list
100+ Tech Journalists Email
100+ Online communities


Wanted to share the list with all! To get a free download just enter 0$ or you can buy me a coffee entering more! Cheers!


Link: https://gumroad.com/l/jbpoy/


Would be great if you left a star rating for the product in exchange on Gumroad ⭐️ 🙏

Upvote the post for all to see! ⬆️

https://redd.it/12o2grd
@r_devops

Tired of people trying to make tools IaaS agnostic

I cannot count the number of new devops tools which have attempted to provide "IaaS agnosticism" beyond what kubernetes and terraform can provide.

IaaS providers provide different stuff. There's no common IaaS spec. Networking, IAM, Compute, GPU support, chipset, etc, all work differently.

And yet I think I've seen more product demos for devops tools claiming to have abstracted away underlying IaaS platforms than tools that do anything else. The deal is you just have to use their platform or tool exclusively instead. What platforms do they actually support at the moment? Well, just AWS of course, with plans to support Azure and GCP by Q3 of 2024.

Then you try their product out. First, you select an environment, and have to select the cloud provider. You have to select the exact instance type you want from that cloud provider. There is no ability to migrate workloads between cloud providers with zero downtime.

If the tool was actually cloud agnostic, we would have to give up all the extra stuff that drives us to pick one provider over the other. I can't use KVS since GCP and Azure don't have a compatible service, as a single example.

To me, it feels like the equivalent of trying to make a programming language which is language independent. Where only features supported in all languages are present in this "language independent" programming language. It totally misses the point of why we pick specific programming languages for certain tasks in the first place.

Whenever I see this on a list of 5 ot 6 features a new devops tool offers I just shake my head and move on.

Edit: I said "tool cloud agnostic" but I meant "tools attempting to abstract away IaaS providers" if it wasn't clear. Obviously devops tools that do something specific are useful (estimating costs as an example) and can also be cloud agnostic

https://redd.it/12o96gi
@r_devops

The Role of Automation in DevOps: CI/CD Pipelines

Series index

Hey r/devops community! In this installment of the Comprehensive DevOps Learning Series, we will explore part of vital role automation plays in DevOps, with a particular emphasis on Continuous Integration (CI) and Continuous Deployment (CD) pipelines.

By embracing automation, teams can streamline their development and deployment processes, enhancing efficiency and ensuring a consistent, high-quality software product. Please be aware that there are many different tools and software ecosystems that solve similar problems. There are only a few mentioned here, and further discussion in the comments is encouraged.

## Why Automation Matters in DevOps

Automation is a cornerstone of the DevOps philosophy, as it helps to eliminate manual, error-prone tasks and accelerates software delivery. It also fosters collaboration by allowing development and operations teams to work together more seamlessly throughout the entire software development lifecycle. By automating key processes, organizations can achieve the following benefits:

Improved productivity: Automation frees up valuable time and resources, enabling teams to focus on higher-value tasks, such as feature development or process improvement.
Faster time to market: Automated processes expedite the delivery of new features and bug fixes, ensuring faster software releases and a more rapid response to customer needs.
Enhanced consistency and reliability: Automated tasks are less prone to human error, resulting in more consistent and reliable outputs.

## Continuous Integration and Continuous Deployment

CI/CD pipelines are central to the automation process within DevOps. Let's take a closer look at each of these concepts:

### Continuous Integration (CI)

Continuous Integration is the practice of integrating code changes from multiple developers into a shared repository frequently, often multiple times a day. CI enables teams to identify and resolve integration issues early in the development cycle, reducing the risk of conflicts and costly fixes later on.

Typical CI pipeline components include:

Version control system (e.g., Git, SVN)
Build system (e.g., Jenkins, Bamboo, TeamCity)
Automated testing tools (e.g., JUnit, Selenium)

### Continuous Deployment (CD)

Continuous Deployment is the process of automatically deploying code changes to production environments after they have passed through the CI pipeline and met established quality criteria3. CD ensures rapid and reliable software releases, enabling teams to deliver new features and improvements with minimal delay.

CD pipelines often involve:

Configuration management tools (e.g., Ansible, Puppet, Chef)
Infrastructure as Code (IaC) platforms (e.g., Terraform, AWS CloudFormation)
Container orchestration tools (e.g., Kubernetes, Docker Swarm)


Conclusion

Automation, particularly in the form of CI/CD pipelines, is a critical element of DevOps practices, enabling teams to work more efficiently, minimize errors, and accelerate software delivery. By embracing automation and integrating CI/CD pipelines into their workflows, organizations can reap the benefits of streamlined processes, improved collaboration, and consistent, high-quality software products. There are many valuable guides on specific tools, deeper dives into why these principles work, and published books describing the DevOps world. Share your favorite, and your thoughts, experiences, and questions in the comments below!

Sources/additional reading:

Duvall, P. M., Matyas, S. M., & Glover, A. (2007). Continuous Integration: Improving Software Quality and Reducing Risk.

Fowler, M. (2006). Continuous Integration.

Humble, J., & Farley, D. (2010). Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment

Automation.

https://redd.it/12opa8w
@r_devops

Fearless Distroless

With the rise of Docker came a new focus for engineers: optimizing the build to reach the smallest image size possible.

A couple of options are available.

Multi-stage builds: A Dockerfile can consist of multiple steps, each having a different Docker base image. Each step can copy files from any of the previous build steps. Only the last one will receive a tag; the others will be left untagged.

This approach separates one or more build steps and a run step. On the JVM, it means that the first step includes compiling and packaging, based on a JDK, and the second step comprises running, based on a JRE.
Choosing the smallest base image size: The smaller the base image, the smaller the resulting image.

In this post, I’m going to focus on the second point.

Read further

https://redd.it/12ooagr
@r_devops

Is Jenkins still the king?

A lot of people on reddit seem to recommend gitlab, or drone.io, but if you get on indeed and search for jobs there are tens of thousands of posts looking for people who know Jenkins and only a tiny fraction of job listings interested in any other ci framework. Is it worth investing time into anything else? It's my decision and while the other options seem more friendly I don't see any point in learning them if I'm not going to be able to use them in the future.

https://redd.it/12ovcoe
@r_devops

Anything on par with HashiCorp Vault

Any alternative to Vault for a multi-cloud / on-prem environment?

https://redd.it/12oxakd
@r_devops

How you deal with pain-in-the-ass team member?

We are few DevOps guys in our mid 30s working on the product.. we also happens to be good friends since college.


The problem is one of the guys have a huge ego - he is good person outside of work, knowledgeable, perhaps more experienced than us in certain areas, but every time his approach is questioned or better alternative is proposed he he starts getting protective. Instead of constructive discussion he leaves sarcastic remarks, putting others down.


His recent target is me as he seems even more agitated since I got my AWS SA Pro and Azure SA expert certs while I never rubbed anything to his face.


We try to give props to the guy since it seems he has very brittle ego but honestly I don't know if I want to continue working with them/him.


So I'm puzzled how to deal with that kind of people and what causes this kind of attitude?


Any reasonable non-conflicting suggestions are highly welcomed.

https://redd.it/12ox31l
@r_devops

How would you manage Perforce?

Hello everyone,

I work for a game development studio based in UK. We have a few hundreds of developers and, as usual in this industry, we are using Perforce.

Since the DevOps team was short-staffed, the company tried to offload Perforce Helix Core and Helix Swarm administration by hiring a third party to do it. However it has come to my attention that this backfired and the third party needs to be hand held into solving any problem that appears and even came to the ridiculous of having the entire system down due to forgetting to renew the SSL certificate of their Authorization-handler website.

Now I have been thinking and am open to see what you guys think: in an ideal world with no limitations and full support of your companies how would you setup your Perforce/other centralised version control tools?

https://redd.it/12ozyz3
@r_devops

First position as a dev ops engineer

For those of you who are hiring managers, what do you expect out of a entry level devops engineer?

https://redd.it/12p3hpn
@r_devops

Saw a techfluencer in LinkedIn with all AWS, Azure and GCP cert.

Seems like it is good for marketing. But is it worth studying so many certs?

https://redd.it/12p4bnw
@r_devops

KCL v0.4.6 is Coming — Rust-Based IDE Extension, Helm/Kustomize/KPT Integrations

The KCL team is pleased to announce that KCL v0.4.6 is now available! This release has brought three key updates to everyone: Language, Tools, and Integrations.

+ Use KCL IDE extensions to improve KCL code writing experience and efficiency
+ Helm/Kustomize/KPT cloud-native community tool integrations
+ Improve the KCL multilingual SDK for easy application integration.

See here for more: https://kcl-lang.io/blog/2022-kcl-0.4.6-release-blog/

https://redd.it/12p75ny
@r_devops

View Kubernetes Secrets Quickly with a Single Command

Ever struggled to view kubernetes secret value as we have to identify and decode it.

Not anymore, view this youtube short and learn how to view the secret value with a single kubectl command.

https://youtube.com/shorts/XIRBdqAJkag?feature=share

https://redd.it/12ozske
@r_devops

Help Project repo structure with common code, dependencies, IP protection issues

Hello everyone,

I'm currently struggling on taming a brown field project for an embedded device, particularly on enabling basic CI processes, mainly due to the poor codebase structuring the project has. For context, on a high-level, the product application is distributed, that is, we have multiple subsystems (e.g., vision, navigation, safety-control) that communicate either via a kind of a message broker or a point-to-point communication channel (e..g, using I2C). Some of these subsystems may be deployed to different hosts, with different OS and platform architectures (i.e., different build toolchains might be required).

Currently, the way the project is structured is a bit of a mess (simplified example):

\- Repository A, for subsystems X and W, to be deployed on platform Alpha;

\- Repository B, for subsystem Y, to be deployed on platform Beta;

\- Repositories [C, D, E\], for subsystem Z, to be deployed on platform Alpha. Some of these repos depend on each other (yuck...);

\- Repositories F and G, one with all necessary build toolchains and third-party dependencies, the other with "common code" such as definition of data structure for the messages being exchanged between subsystems/components. These are either pulled to create the build environment or used as dependency for some subsystems (in the form of git submodules).

Hopefully, you can imagine the headaches resulting from managing the build environments, dependencies, and submodules. My initial thought either to join everything into a mono-repo, with a "common" folder for the code shared between subsystems, and each subsystem residing in a dedicated folder comprising the corresponding source code and everything it needs to be built (toolchain, third-party packages, etc). Another approach, was to have one repo per domain/subsystem in a way that it is decoupled and fully buildable in isolation (ala microservices). Alas, I'm not sure how to deal with common code using this approach and the impact it has on the CI process...

To complicate things, we have externals working on parts of a subsystem (say repos C and D for subsystem Z) that cannot have access to an IP-protected part of the same subsystem (say repo E). My thoughts to "solve" this was to merge the "public" codebase for this subsystem into a single repo (H=C+D) and have the IP-protected code being imported as a git submodule (the externals wouldn't have permissions to the corresponding repo, and thus, couldn't pull it). I could also follow a similar approach for a mono-repo with the IP-protected part as a submodule but...the externals must be exposed to as few parts of the project as possible.

Finally, I need to find a way to integrate everything into a cohesive product comprising the different subsystems (e.g., the product has subsystem X v1.0.0, subsystem W v.1.0.1, subsystem Y v.1.0.5, etc) and make sure everything works (via system tests, etc). The mono-repo would easily solve this; for the multi-repo approach I would have an "umbrella" repository importing each subsystem as git submodules at the necessary tag/version (plus any helper scripts that I would need to build, glue, and deploy everything).

Does anyone experienced a similar scenario? How would you solve this and how would your CI process look like? Is there any book discussing these kind of issues, strategies on how to structure projects using GIT repositories, etc., that you would suggest? Most of the content I seem to find relies on scenarios with perfectly clean or basic situations...

Thank you everyone, and apologies for the long post!

https://redd.it/12paw3k
@r_devops

Jenkins guides

Any places considered best references to study Jenkins ?

https://redd.it/12pkdwd
@r_devops

OnCall Fiasco

Friends, I’ve recently started a new job that has brought a challenging work-life balance. This includes unpredictable weekly deployments and releases (9 PM), including Fridays. And be on an oncall every three weeks.

I'm curious if this is normal, especially for senior positions – Is it common for higher-paying roles to demand this much off-hour work?

https://redd.it/12po3hi
@r_devops

Any real benefit on going with Datadog for AWS monitoring?

I have been wondering why some steps were taken by my current company and this is one of them.

We only use AWS, hence we only need to monitor AWS services. Therefore I got puzzled on why would someone go the extra mile and pay for Datadog when we have an AWS built-in product like CloudWatch. I could understand using Datadog would be good if we used multiple clouds, but that is not the case and we do not foresee any change to this within a couple of years.

Is there a big difference that would make going for Datadog reasonable if a company only uses AWS services?

https://redd.it/12pofo8
@r_devops