Designing a data recovery strategy

Hi everyone. If this is considered vendor spam, lmk and I'll take it down.


I work with Rewind and we have a thing coming up on March 31, where our CloudOps team is walking through the process of drafting and testing our data recovery strategies. Things like who is involved, what's in a runbook, ways to test the strategy (we do tabletop), and so on.


We are hoping to get some others to join us (it's basically an AMA format) to share stuff they have picked up over the years. Here's a link if anyone is interested: https://rewind.com/mistakes-to-avoid-when-designing-your-data-recovery-strategy

https://redd.it/11z4mtr
@r_devops

Do you use custom Prometheus metrics?

If you use Prometheus for metrics, do you mainly use it for infrastructure metrics or are your applications instrumented with custom metrics? If you use custom metrics, what kinds of things do you track?

https://redd.it/11zj6rt
@r_devops

A Guide to Delegated Administrator in AWS Organizations and Multi-Account Management and how to secure the Root Management Account

www.cloudquery.io/blog/guide-aws-org-delegation

Delegated Administrator in AWS is a secure way of using non-management accounts to manage multiple accounts within your AWS Organization.  Read more about our research and how to setup delegation securely. Check out our research and guide on setting up delegated administrator, the IAM permissions necessary, and security benefits of multiple accounts and delegated administrator accounts, and why using the root management account can be insecure.

Disclaimer: I'm the author.

https://redd.it/11zkf8a
@r_devops

What is the least frustrating/prone to error way to use Terraform in a multi-account AWS Org deployment?

New to AWS and have been considering different tradeoffs for laying out the start of our footprint. I've read conflicting approaches on what method is used with terraform to handle a multiple account/single org setup.

It seems there are a few blogs from AWS and Hashi that favor workspaces but they seem to be dated by about a year. Quite a few stack overflow posts where people consider local workspaces the devil.

The only hard requirement I have at this point is TF 1.15 and it has to be done in house (no TF Cloud)

Appreciate any experiences / tips, thanks.

https://redd.it/11zmuik
@r_devops

F5 Nginx Plus ingress controller

Does anyone use the f5 version of nginx ingress controller?

I’m trying to understand how it’s worth the amount of money they are asking for when compared to the free kubernetes ingress-nginx.

What are your experiences?

https://redd.it/11zoeeq
@r_devops

The best way to monitor hpILO,IPMI and DRAC

I am new to devops and trying to find opensource way to monitor my servers through remote access control.
There are so many options if I check online , grafana , Prometheus etc
I am new to these as well and having a basic information about these tools , could please point me where I can go step by step and have some sort of system which alert me about the problems.

Thanks in advance

https://redd.it/11zqtna
@r_devops

Cicada - Build CI pipelines with Typescript

Hey r/devops! I've been working on a project to let you write CI pipelines using Typescript and run them anywhere (locally or in any CI provider). Wanted to hear any feedback you have: cicada.build

This was born out of our frustration with building and maintaining complicated CI/CD pipelines using YAML - it was hard to reuse logic, IDE support was pretty bad, and we were constantly dropping into bash to write obscure one-liners.

Would really appreciate any feedback. Is this something you'd use?

Also, there is a waitlist for now since the SDK is in active development, but we're onboarding new projects every week, it will be public and open-source soon!

https://redd.it/11zs341
@r_devops

What salary should I be expecting for a DevOps/SRE/Platform/Cloud engineering job in Canada?

What salary should I be expecting for a DevOps/SRE/Platform/Cloud engineering job in Canada?

A bit about me that could help in answering this question:

* I have 1 year of SDE experience in India (at an MNC) and I will have a master's from the University of Toronto in CS by end of 2023.
* I will have 16 months (7 months part-time, 9 months full-time internship) of experience working in DevOps with a startup by the end of 2023 (my graduation date).
* I have worked with AWS, K8s, Docker, and other CI/CD tools.
* I am 2x AWS certified (SAA, DVA).
* Planning to get CKA/CKAD if it can help in getting interviews or salary negotiations.

Sorry if the question is a bit direct. I really want to know what I should expect when I start hunting for jobs. My classmates say that we should be getting somewhere close to 120k - 150k since we have a masters from the University of Toronto. But I think it's a bit too much, most of my classmates are freshers and their idea of salary is just based on the numbers they see from level.fyi. I just want to get a general idea from the community so that I don't lowball myself.


Thank you.

https://redd.it/11zrs5l
@r_devops

Gitlab CI/CD pipelines: rule for pipeline NOT to run if specific variables ARE defined

Hello people of the internet,

​

I for the life of me cant wrap my head around why this pipeline is not working as I expect it to.

so its quite self explanatory, what i want to accomplish is for the first job ".pact" to only run if the variables IS\_LIBRARY, IS\_BASE\_IMAGE etc IS NOT defined. If for example IS\_LIBRARY is defined it should not run any of the other jobs like pact:acc.

These variables are either defined or not in repos themselves in their .gitlab-ci.yml files, example down below

​

---

variables:
IS_LIBRARY: "yes"
PYTHON_VERSION: "3.9"

include:
- project: reddit/pipelines
file: python-pipelines/.gitlab-ci.yml
ref: master

​

This is how the .gitlab-ci.yml looks which just does a include of the pact pipelines .yml file:

​

include:
\- local: '/hi\_reddit/deploy.pact.yml'

​

---

.pact:
image: pactfoundation/pact-cli
artifacts:
paths:
- pact-*
script:
- |
pact-broker can-i-deploy --pacticipant $CI_PROJECT_NAME --version $CI_COMMIT_SHORT_SHA \
--to $ENV --broker-base-url $PACT_BROKER_URL \
--output json > $ENV > pact-error || true
echo ok > pact-$ENV
if grep '"unknown": 1' $ENV; then echo maybe > pact-$ENV; fi
if grep '"failed": 1' $ENV; then echo no > pact-$ENV; fi
rules:
- if: $IS_LIBRARY || $IS_BASE_IMAGE || $BATCH_NAME || $IS_KAFKA_SCHEMA || $ONLY_SECURITY_SCAN
when: never

"Pact:sys":
stage: pre deploy sys
extends: .pact
variables:
ENV: "sys"
rules:
- if: '$CI_COMMIT_BRANCH == "dev" || ($CI_COMMIT_BRANCH == "master" && $TRUNK_BASED == "true")'
when: on_success


"Pact:acc":
stage: pre deploy acc
extends: .pact
variables:
ENV: "acc"
rules:
- if: $CI_COMMIT_BRANCH == "dev" || $CI_COMMIT_BRANCH == "master"
when: on_success



"Pact:stage":
stage: pre deploy stage
extends: .pact
variables:
ENV: "stage"
rules:
- if: $CI_COMMIT_BRANCH == "master"
when: on_success


"Pact:prod":
stage: pre deploy prod
extends: .pact
variables:
ENV: "prod"
rules:
- if: $CI_COMMIT_BRANCH == "master"
when: on_success

https://redd.it/11zls8w
@r_devops

Feedback on my free AI based Slackbot to Simplify DevOps

Hey there!


I built a Slackbot that lets you easily monitor and automate your DevOps tasks.


I hated wading through team chats, wiki docs and outdated shell scripts. I wanted a way to manage and troubleshoot my infrastructure, right from Slack. My Slackbot combines all these siloed knowledge bases and automates tasks for you.


Currently, it runs against Notion and AWS. Here's a quick video: https://www.youtube.com/watch?v=HX9CD-kxKj4


Try it out in the public Slack playground (no downloads required) : https://join.slack.com/t/promptops/shared\_invite/zt-1rl40rt90-ntijxFM5wFWnNNkyy\_nzJA


I'm hoping to connect with more folks to improve the experience :)

https://redd.it/1202bd8
@r_devops

April Fools Day Idea?

As we draw nearer to April 1st and you can’t think of a good prank to pull on the coworkers?

Feel free to use mine!

Submit a PR containing all of your code on a single line.

https://redd.it/1203vin
@r_devops

Have you used eBPF in real production? worth the switch?

I hear about it everyday on Twitter on how great eBPF is etc.

https://redd.it/120a074
@r_devops

Python command line and client API for Burp's Enterprise Edition API

Wrote this https://github.com/olliewuk/bseept which allows you to drive all of Burps' DAST scanner from the command line. Additionally it can be used as a client library in your own Python code.

https://redd.it/120bw0p
@r_devops

Android on the cloud

Hello everyone! I recently stumbled upon Android-x86 and I must say, it's absolutely amazing! However, I'm facing an issue with running this operating system natively on an EC2 instance. I was hoping to remotely connect to the instance from my Android phone to greatly enhance its overall speed. My phone only has 4 gigabytes of RAM and lacks the computing power to emulate or play games like PUBG Mobile. By creating an EC2 instance with Android-x86, I can control it remotely from my phone and enjoy increased internet speed, computing power, and gaming capabilities. I would greatly appreciate any suggestions or guidance on how to set this up. Thank you!

https://redd.it/120g6dl
@r_devops

DevOps Engineer here looking for something interesting to work on.

Hey, I have been a DevOps Engineer since the past 5 years with multiple companies. But since the past 6 months I have lost all interest to work or do something new. I have just been doing the bare minimum and getting by. I feel I need something interesting to ignite my passion again for technology and DevOps. Anyone here has any ideas?

https://redd.it/120iiew
@r_devops

Golang, Docker “FROM scratch,” Kubernetes Cronjob, Istio or How to Make Them Friends

**In this post, I'd like to share our DevOps engineer article about how to run a k8s Cronjob with the following conditions:**

* our application is written in the GO language
* the Docker image is built on the "FROM scratch" base
* auto-inject is enabled for the istio-proxy.

Docker has become an indispensable tool for developers and DevOps engineers, making it easier to package, distribute, and deploy applications consistently and efficiently. One of the most popular programming languages for building microservices and scalable applications is Go (Golang). In addition to being an essential tool for developers and DevOps engineers, Docker is often used in conjunction with Kubernetes to manage and orchestrate containerized applications at scale. This is where Istio comes in - it's a service mesh that can be used with Kubernetes to help manage and secure communication between microservices in a containerized application.

## Building Docker Images for GO Applications

Multi-stage builds are a powerful feature in Docker that allows you to build an application in one stage and then copy the compiled binary to a second, minimal stage. This provides several advantages, including:

* **Reducing the size of the final image:** By using a minimal base image, you can keep the size of the final image as small as possible. This makes it faster to download and deploy and reduces the attack surface of the image.
* **Improved security:** By using a minimal base image, you can reduce the number of dependencies and potential attack vectors. Additionally, by not copying any build artifacts or temporary files, you can further reduce the attack surface of the image.
* **Improved build time:** Multi-stage builds allow you to compile the application in one stage and then copy the binary to the final image in a separate stage. This can improve build time, as you can use a faster build environment for the first stage and then use a smaller, more minimal image for the second stage.

The "FROM scratch" base image is a special, minimal base image that contains no files or dependencies. It provides a clean slate for building your application and is often used as the base image for multi-stage builds.

Here's an example of how you can use a multi-stage build to build a Docker image for a Go application:

# Build stage
FROM golang:latest AS build
WORKDIR /app

COPY . .
RUN go build -o myapp
# Final stage
FROM scratch

COPY --from=build /app/myapp /myapp
CMD ["/myapp"]

In this example, we first use the golang: latest image as the build stage. This image provides a Go environment with all the necessary dependencies to build our application. We then copy the source code to the /app directory and run the go build command to compile the binary.

In the final stage, we use the scratch base image, which provides a minimal, empty environment for our application. We then copy the compiled binary from the build stage using the COPY --from syntax. Finally, we set the CMD to run the binary, which will be the default command when the image is run as a container.

(This example doesn’t show all best practices of building Docker images because it’s not a goal of this article. That’s why the 'latest' tag is used, etc.)

Using a multi-stage build and the "FROM scratch" base image, we can create small, secure, and efficient Docker images for our Go applications. This approach also provides a consistent and reproducible build environment, making it easier to manage and deploy our applications.

## Istio

Istio is an open-source service mesh technology that provides a set of APIs and tools for managing microservices in a cloud-native environment. It is designed to simplify the management of the communication between microservices and to provide a platform for adding new functionality, such as traffic management, security, and telemetry.

**Advantages of using Istio:**

* Istio helps to manage the complexity of microservices communication by

providing a unified control plane.
* It provides built-in security features that help to secure communication between microservices.
* Istio makes it easier to manage and monitor microservices by providing centralized visibility and control over the network.
* Istio provides an API-driven approach to traffic management that makes it easy to manage and control the flow of requests between microservices.

## Problems with using Istio in Kubernetes Jobs and docker images built using “FROM scratch”

Kubernetes Jobs (cronjobs) are used to run batch jobs, and when the main container is completed, the Istio proxy sidecar continues to run, which can cause issues with resource usage and security. 

To terminate the Istio proxy sidecar, you need to run the following command in your main container:

wget -q --post-data='' -S -O /dev/null https://127.0.0.1:15020/quitquitquit

However, this can be a challenge when the main container is built using the "FROM scratch" Docker image, as it does not include the wget/curl tool, which is required to terminate the Istio proxy sidecar.

In such cases, there are a few alternative options to consider:

* Modify the Docker image to include the wget tool: You can modify the Docker image to include the wget tool by using a base image that includes it or by installing it directly in the Dockerfile. This solution would require modifying the existing Docker image and could potentially impact the size of the image.
* Configure Istio to automatically terminate the proxy: Istio can be configured to automatically terminate the proxy when the main container is completed. This can be done by setting the terminationGracePeriodSeconds property in the Kubernetes Job definition. This solution would require modifying the existing Kubernetes Job definition and may require additional configuration changes in Istio also we need to be sure that our main container is completed. 
* Using init containers to copy tools into the main container.

Let’s see what the third option looks like. 

An init container is a special type of container that runs before the main container in a Pod.

In this case, the init container can be based on the **busybox** image and have the following steps:

* Copy /bin/sh and /bin/busybox to a volume (e.g. emptyDir) shared between the init container and the main app.

The command for the main container will be set as follows:

set -x; /myapp || EXIT_CODE=$?; /share/busybox wget -q --post-data='' -S -O /dev/null https://127.0.0.1:15020/quitquitquit; exit ${EXIT_CODE}

The above command runs the main application /myapp and sets an exit code stored in the EXIT\_CODE variable. When the main application finishes, the Istio proxy sidecar is terminated by sending a wget request to https://127.0.0.1:15020/quitquitquit. The exit code of the main application is then returned to ensure that the correct exit status is passed back to the system.

Here is an example of a Kubernetes CronJob YAML definition that implements the solution:

apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: my-cronjob
spec:
  schedule: "0 0 * * *" # run every day at midnight
  jobTemplate:
    spec:
      template:
        spec:
          volumes:
            - name: share
              emptyDir: {}
          initContainers:
            - name: init
              image: busybox:musl
              command: ["cp", "/bin/sh", "/bin/busybox", "/share/"]
              volumeMounts:
                - name: share
                  mountPath: /share
          containers:
            - name: main
              image: my-main-container-image
              command:
                - /share/sh
                - -c
                - set -x; /myapp || CODE=$?; /share/busybox wget -q --post-data='' -S -O /dev/null https://127.0.0.1:15020/quitquitquit; exit ${CODE}
              volumeMounts:
                - name: share
                  mountPath: /share
              livenessProbe:
               

tcpSocket:
                  port: 15020
              readinessProbe:
                tcpSocket:
                  port: 15020
          restartPolicy: Never

In addition, the above definition includes liveness/readiness probes for the main container to check that the Istio proxy container has started and is ready to serve traffic. Keep in mind that this cronjob spec doesn’t have special annotations/labels to enable injecting istio-proxy sidecar. 

## Conclusion

When using Istio with Kubernetes Jobs, it is important to consider the limitations of the main container's Docker image and the tools that are included. By exploring alternative solutions, such as modifying the Docker image, using a different tool, or configuring Istio, you can ensure that your Kubernetes Jobs run smoothly and securely with Istio.

https://redd.it/120jdfe
@r_devops

heads up Github host key issue

https://github.blog/2023-03-23-we-updated-our-rsa-ssh-host-key/

This will probably cause a lot of problems for people over the next few days!

https://redd.it/120jnds
@r_devops

Syntax error Unexpected token after deployment

Hi guys. I have this app which is deployed on Kubernetes. Was previously using a local host now shifted to domain and deployed. But i cant solve this issue that when i try to access the app through domain it gives an error that “uncaught syntax error: Unexpected token ‘<‘ at main.f2d33…… “. Please help me

https://redd.it/120n0ii
@r_devops

Did you become a Devops Engineer instead of SWE because you suck at leetcode?

Not to belittle anybody - we all know the devops space is extremely complex so we don't have to hide behind SWEs when it comes to solving hard problems.

However, after I initially became an SRE "by accident" (the job was simply offered after a SWE internship), I only stayed on the path because the interviews don't require (as much) Leetcode as normal SWE gigs and pay equally well. Even though I find YAML boring and actually dislike overengineering using 100 tools.

View Poll

https://redd.it/120pbgv
@r_devops