Need help with understanding deletion of previous docker image in pipeline

Hello guys! I'm pretty new to devops and looking for advise.

I'm trying to implement this strategy called streamline: [https://medium.com/@jbradyaudio/streamline-a-super-efficient-branching-and-ci-strategy-ffa864aa99d4](https://medium.com/@jbradyaudio/streamline-a-super-efficient-branching-and-ci-strategy-ffa864aa99d4)

I'm doing it to learn things.

One step is confusing to me: Deleting old image from PR.

It goes like: devs creating branch named "feature-xxx" of master. Then they do some commits and open PR (or MR in gitlab terms). Pipeline runs tests and builds docker image, then image pushed to registry, then some other tests... Then devs push another commit on top to this PR and pipeline should delete previous image.

I end up with tagging images with `"${CI_MERGE_REQUEST_ID}_${CI_COMMIT_SHORT_SHA}"`, then i have a bash script to call gitlab build-in registry api, find images that have tag started with `${CI_MERGE_REQUEST_ID}` and delete it.

PREVIOUS_TAGS=$( \
curl -s --header "PRIVATE-TOKEN:${GITLAB_API_KEY}" "https://gitlab.com/api/v4/projects/${CI_PROJECT_ID}/registry/repositories/${REPO_ID}/tags" | \
jq -r ".[] | select(.name | startswith(\"${CI_MERGE_REQUEST_ID}\")) | .name" | \
paste -sd, \
)
# then make it array and iterate over with for loop and curl --request DELETE

Gitlab ci pipeline example:

stages:
  - lint   
- clean
  - build
Lint:
rules:
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
stage: lint
image: node:lts-alpine
script:
    - ./node_modules/.bin/eslint "**/*.{vue,js}"

Clean:
rules:     
- if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
stage: clean
image: alpine:3.14
before_script:
    - apk add --no-cache curl jq bash
script:
    - bash ./.cicd/clean-prev-build.sh

Build:
rules:
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
stage: build
image: docker
services:
- docker:dind
before_script:
- docker login -u $CI_DEPLOY_USER -p $CI_DEPLOY_PASSWORD $CI_REGISTRY
- export IMAGE_REPO=$CI_REGISTRY/demo
- export IMAGE_TAG="${CI_MERGE_REQUEST_ID}_${CI_COMMIT_SHORT_SHA}"
script:
- docker build -t ${IMAGE_REPO}:${IMAGE_TAG} -f ./.cicd/Dockerfile .
- docker push ${IMAGE_REPO}:${IMAGE_TAG}

Sadly $CI\_COMMIT\_BEFORE\_SHA always 000000000 for merge request pipelines so i can't use it.

How bad is my solution? Any suggestions to make it better?

https://redd.it/oxsh56
@r_devops

Infra as a code is cloud lock-in

https://flicksfix.com/posts/infra-as-a-code-is-cloud-lock-in/

I've wanted to spark a discussion regarding the movement of doing "everything as a code". I've tried really hard to make it work for my own servers / boxes, but it turns out its a lot easier to maintain them using good-old ssh with some bash scripts here and there instead of full-blown weapons like Ansible / Terrraform.

On the other hand I feel like at work terraform is a blessing when working with cloud providers and I get this big disconection between my own hobby projects and work that I do.

Do you feel the same about it?

I've shared some thoughts and example in (disclaimer: my own) blog post.

https://redd.it/oy0zhb
@r_devops

Creating and Automating a Runbook for an application hosted in AWS

Hi,

I'm a newbie to aws. Looking for a way to create a runbook and automating for deploying applications hosted in AWS.Appreciate your thought on this

https://redd.it/oxq7jd
@r_devops

We built an observability tool for CI Pipelines to help developers identify why their tests fail. It’s free for open-source projects

Hi everyone,

We’ve been working on our new product, Thundra Foresight. Our aim is to help developers to understand test failures and to bring observability into their CI Pipelines for higher productivity and lower CI costs.

It’s free for open-source projects.

# What’s the point to write here?

We’re looking forward to getting feedback on our tool to serve developers in the best manner! And, you can explore foresight (how it works) with the most popular open-source projects.

Thank you for your support!

https://redd.it/oxu2f2
@r_devops

Future of Application Performance Monitoring Pricing

Hello all,

I'm exploring what would revolutionize the pricing for Application Monitoring tools, and would really appreciate your inputs and experience with APM tools such as Dynatrace, Stackify, AppDynamics, etc.

The idea is to make sure that the users/buyers of an APM tool are able to forecast the pricing as well as keep it flexible based on their dynamic infrastructure needs.

The current industry is all over the place with the pricing models and lack simplification and flexibility.

What would be the best pricing method for the future of an APM tool/

View Poll

https://redd.it/oxpri0
@r_devops

Moving out from the on-Prem ghetto

Hello all,

I am a management consultant who ended up falling into a devops role at a client where I ended up building their cluster and their devops tools all on-Prem. Where I live (Asia) the demand appears to be primarily aws/azure related and everyone seems to lose interest when I say I worked solely on Prem. Am I targeting the wrong markets or should I move out - and how do I get substantive aws/azure experience?

https://redd.it/oy9wkq
@r_devops

Caching and real-time notifications in a fully serverless AWS based web application with…

https://towardsaws.com/caching-and-real-time-notifications-in-a-fully-serverless-aws-based-web-application-with-5ae9b67add78

https://redd.it/oyagg2
@r_devops

Deploy .NET project to Hetzner

Is there a way to deploy docker with Visual studio on Hetzner, something like you can do on Azure? Or what is the best/quickest way to deploy .NET project to Hetzner?

https://redd.it/oybn8k
@r_devops

Deploy .NET project to Hetzner

Is there a way to deploy docker with Visual studio on Hetzner, something like you can do on Azure? Or what is the best/quickest way to deploy .NET project to Hetzner?

https://redd.it/oybn8k
@r_devops

Learning one cloud provider or multiple?

Is it better to invest X amount of time into learning one cloud provider or the same amount of time into 2?

Obviously with the first option I would know it more in depth, but the second could make it easier while job searching right?

Also, is there that much info when it comes to each provider? GKS & EKS should work similar enough (it's the k8s themselves that you work with most of the time), then the IAM management (probably the biggest differences here) & then storage management. It's not like I need to be an expert in the provider to do devops, I only need to know the relevant parts...

https://redd.it/oybg3f
@r_devops

PowerShell vs REST API

I'm writing a script that will onboard and offboard users that will be used by helpdesk and HR. Some of the items i need to communicate with are in Azure and Exchange online (regarding mailboxes). How should i decide on using PowerShell cmdlets vs REST API calls when interacting with Exchange and Azure?

https://redd.it/oyacf5
@r_devops

Trying to set up chef16.14.1 on debian 9 remote servers

Hello. I'm attempting to get chef set up on a group of 5 remote servers all running debian 9. I've got the download, dpkg'd it on each server, and tried to run sudo chef-client, but I get the below output

[2021-08-05T07:12:04+00:00] ERROR: Running exception handlers

Running handlers complete

[2021-08-05T07:12:04+00:00] ERROR: Exception handlers complete

Chef Infra Client failed. 0 resources updated in 01 seconds

[2021-08-05T07:12:04+00:00] WARN: Failed to read the private key /etc/chef/client.pem: #<Errno::ENOENT: No such file or directory @ rb_sysopen - /etc/chef/client.pem>

[2021-08-05T07:12:04+00:00] WARN: Error while reporting run start to Data Collector. URL: https://localhost:443/data-collector Exception: No HTTP Code -- I cannot read /etc/chef/client.pem, which you told me to use to sign requests!

[2021-08-05T07:12:04+00:00] WARN: Failed to read the private key /etc/chef/client.pem: #<Errno::ENOENT: No such file or directory @ rb_sysopen - /etc/chef/client.pem>

[2021-08-05T07:12:04+00:00] WARN: Error while reporting run start to Data Collector. URL: https://localhost:443/data-collector Exception: No HTTP Code -- I cannot read /etc/chef/client.pem, which you told me to use to sign requests!

/opt/chef/embedded/lib/ruby/gems/2.7.0/gems/chef-16.14.1/lib/chef/compliance/runner.rb:23: warning: Chef::Compliance::Runner#logger at /opt/chef/embedded/lib/ruby/2.7.0/forwardable.rb:154 forwarding to private method NilClass#logger

[2021-08-05T07:12:04+00:00] FATAL: NoMethodError: undefined method \logger' for nil:NilClass`

I assume I've missed a step or 3, but I've no idea what those steps might be. The docs don't give much in the way of help I can make sense of. Can anyone help me out, please?

Thanks in advance,

Phil

https://redd.it/oyep0v
@r_devops

macOS Monterey : Check new Powerful features here

Apple's macOS Monterey public beta is available now. WWDC announced this year, the 18th major macOS update isn't adventurous as compared to its previous versions. But it is a satisfactory OS update with a bunch of surprising features.

https://www.theencrypt.com/macos-monterey-features/

https://redd.it/oyf6bf
@r_devops

Multi Cloud soultions & suggestions

I am looking for a solution(s) that can handle CI/CD for multi-cloud and possibly something that can suggest if moving to other plublic might save money.

So for example: If an application is deployed in AWS but maybe deploying in Azure might be cheaper. I would like to get suggestions and approximately how much savings.

For handling multicloud, I'd like the solution to have
1. Be able to deploy to multiple public clouds at once
2. Be able to deploy to other public clouds if application is down

Are there any sort of solution for the above?

https://redd.it/oyfrky
@r_devops

I tried AWS Serverless monitoring with Dashbird - here's my honest review

As a startup, we always want to focus on the most important thing — to deliver value to our customers. For that reason, we are a huge fan of the serverless options provided by AWS (Lambda) and GCP (Cloud Function).

Monitoring the execution of the Lambda function starts to become a real issue when you have tens or hundreds of functions running at the same time. When we first started, we relied a lot on Cloudwatch Logs. Instead of creating a serverless monitoring tool ourselves, we were exploring off-the-shelf monitoring options in the market. Dashbird.io was one of services we explored. In this article, I will share the experience of using Dashbird.io, particularly on the onboarding process and the main offering by the platform: https://aws.plainenglish.io/aws-serverless-monitoring-using-dashbird-io-4714fe222e2a

https://redd.it/oygu6c
@r_devops

Heroku and DevOps

A new client is intent on hosting production on Heroku. Is this as impractical as it sounds to me and do any of you have anything good or confidence building to say about it?

I'm worried about scaling and being tied into a PaaS that seems to be extremely heavy handed at config.

Seems like setting up CI/CD is pretty easy.

https://redd.it/oyir58
@r_devops

Fortify Thoughts

Just curious, what is everyones thoughts on Fortify?

https://redd.it/oyhxf8
@r_devops

GitLab Commit 2021 - DevSecOps - Upgrade your development workflow with interactive code maps

Earlier this week, I gave a talk at GitLab Commit 2021 about how we can build our understanding of code, share that knowledge with other people, and use new tools for guidance and navigation as we create, fix and improve code. If you weren’t able to attend the event, I’m going to recap my talk here. If you prefer to watch vs. read, scroll down for a 14 minute video of my talk.

Here’s the problem I outlined:

> Acquiring knowledge about code is hard. And communicating it across teams is even harder.

So when we make structural changes to code, I think it’s important to pause for a minute and confirm our assumptions before we start changing things. Getting extra context and getting oriented is really important when making performance changes, refactoring or fixing tough bugs -- because navigation of code becomes less apparent when we make these types of complex changes.

This is particularly relevant in DevSecOps, because security has specific skill sets and values. Bringing security into DevSecOps requires everyone to learn more about people in adjacent roles, and learn a bit of their language and value system in order to collaborate and share context efficiently.

Here’s the example I shared:

Take the lifecycle of a bug fix, a basic communication process that happens every day between 3 different people in different roles. You’ve got:
1. The person reporting the bug (e.g. another developer, a QA tester or security tester)
2. The developer fixing the bug
3. The code reviewer

There are 2 handoff stages here: from the bug finder to the developer, and from the developer to the reviewer. At each stage, a knowledge transfer needs to happen between people with very different experience levels with the code in question. In fact, it’s possible that none of these 3 people really know this particular area of the codebase that well. But they all need to work together to get a code change written, reviewed and approved.

The bug finder knows that the dev team wants as much descriptive information about the bug as possible -- what the tester was doing, what they typed in, what buttons they pushed, what they saw, etc. This description can be accompanied by some screenshots and for UI bugs, this works great. But what about bugs in application behavior, where it’s the backend that’s misbehaving? Maybe the user is presented with inaccurate information or data belonging to someone else! Or they expect to go to page X but end up on page Y. What then?

Here’s the solution I proposed:

As a developer, wouldn’t it be great if the bug finder could send you the equivalent of a screenshot, but of the backend internals? How about a map of code paths, including all the dynamic and complex stuff like HTTP requests, caching, user session interactions, security and SQL?

This is all possible with code maps. And by code maps, I mean a visual presentation of code (other than the code as text) that makes a design aspect of the code easier to understand.

Making code maps is possible with AppMap, a free, open source tool that makes runtime recordings of code. AppMap works equally well with large and small projects, web applications, and microservices. It captures all the information I described above, and bundles it into a portable JSON format. It’s also easy to write programs that process, analyze and display AppMap data.

> Quick sidebar: Last week we released a GitLab integration for AppMap! Now you can upload AppMaps to the AppMap Cloud and log in with your GitLab account. You can also attach links to AppMaps stored in AppMap Cloud to your GitLab merge requests. Here’s a short demo video if you’d like to see this in action.

Here’s the demo I gave:

In my talk, I showed how to use AppMaps to transfer information about bugs from testers to developers, and from developers to code reviewers. Start

watching at the 5:00 mark in the video to watch the demo.

Next steps you can take:

If you’d like to try using code maps to accelerate code understanding and improve communication with your team, you’ll need to:

[Install AppMap](https://appland.com/docs/quickstart/) for VSCode or JetBrains
Install and configure the AppMap client agent for Ruby, Python, or Java
Show your dev team or pen testers how to run the app with AppMap enabled, and use the AppMap browser extension to [record what they see and do](https://appland.com/docs/reference/remote-recording.html)
Show everyone how to upload and download AppMaps - from your own internal repository, or from AppMap Cloud

Presentation video

https://www.youtube.com/watch?v=iC0NrcqhStI

Any comments or questions? Come find us on our Discord server or on Twitter. I’d love to hear from you!

https://redd.it/oyh2qg
@r_devops

A Makefile for docker monorepos

We have open sourced today an opinionated Makefile that eases the handling of monorepositories.

Any question, feedback, contribution are more than welcome!

https://github.com/enspirit/makefile-for-monorepos

https://redd.it/oymsdz
@r_devops

AWS: Connect to EC2 instances using Systems Manager (SSM)

Hey everyone,

I'm sharing this video to show you how to securely access your AWS EC2 instances other than SSH. This option is great for those instances that need to have a public IP attached or need to be placed in a public-facing subnet for one reason or another.

Also, the Systems Manager may not be all that well known, but it might be depending on how you are using AWS. Lastly, there are pre-requisites to getting this working, but bounce around this video, it's for you anyway! :)

https://www.youtube.com/watch?v=jrmctZ5CmPk

Let me know what you think, and thanks again!

Cheers!

https://redd.it/oynq7v
@r_devops