Docker-compose based CI/CD?

Has anyone implemented or used a CI/CD pipeline that runs docker-compose services?

I’ve used CircleCI and GitHub Actions at previous jobs, and my new job uses a self-hosted TeamCity instance that is able to run your docker-compose services as build steps in the CI/CD pipeline.

This is incredibly convenient compared to something like CircleCI, as it’s way easier and more consistent to specify testing/build steps in docker-compose and run them both locally and in CI/CD. HOWEVER, our DevOps team did a lot of work to make it possible for TeamCity to do this.

I was wondering if anyone has any experience with a platform that natively allows the use of docker-compose in the CI/CD pipeline? Or alternatively, if anyone has experience implementing this kind of solution.

https://redd.it/lhddud
@r_devops

Do you know how to program or write scripts and what is the difference?

And which one matters when hiring?

https://redd.it/lhcc31
@r_devops

AWS CDK: Unable to access secretArn if I run Lambda on SAM Local

I'm reposting this from another sub. Maybe you guys can help? :)

I tried creating a secret in Secrets Manager via CDK:

const secret = new secretsmanager.Secret(this, 'Secret', { /* options */ });

I'm trying to get `secret`'s `secretArn` and pass it into a Lambda function's environment:

const opts = { environment: { SECRET_ARN: secret.secretArn } };
const lambdaFunc = new lambda.Function(this, 'simoncpu', opts);

My problem: In a real environment, I can read the `secretArn` via `process.env.SECRET_ARN` but in SAM Local, all I can see is something like `simoncpuSecret12345`.

How can I access the secretArn in SAM Local?

https://redd.it/lhbg0c
@r_devops

Has anyone used AWS's DevOps Guru?

AIOps is becoming a thing, as clients are starting to ask about it. I'm about to start a poc for an e-commerce platform, and will update this post with my findings.

In the meantime I'm curious if you guys used it, have thought about it or used alternatives.

Cheers

https://redd.it/lgxg13
@r_devops

How to write terraform in a provider independent way?

I thought that the advantage of Terraform was the ability to write infrastructure independently from the cloud provider.

However after a quick look, I don't see it...

I mean I don't see a way of creating a virtual machine in an abstract way independent from AWS or Azure.

https://redd.it/lhkj1y
@r_devops

Monolith vs. Layers?

Start a new Infrastructure as Code (IaC) terraform project may be very challenging. This article explores 2 opposite implementations in detail to organizing your code: a single monolith or multiple layers. Please feel free to share if you have any other advice!

https://redd.it/lhkus6
@r_devops

Deploy machine learning NLP models with Docker Swarm

Hi,

As we're using Docker Swarm extensively behind NLP Cloud (https://nlpcloud.io), I made a post about how to set up Docker Swarm and deploy a cluster with Docker Compose:

https://juliensalinas.com/en/container-orchestration-docker-swarm-nlpcloud/

I also included important details about deploying machine learning NLP models to production, and discussed a couple of things regarding Swarm vs Kubernetes.

Hope you'll like it!

https://redd.it/lgueem
@r_devops

Help on design for learning project - single-tenant monolithic application for multiple users

I am working on doing a small-ish project that is to learn about deploying monolithic single-tenant applications on Kubernetes. The project is a simple Node.js backend with a Vue.js front end (think of hello world with an admin dashboard to change the text) and uses MySQL for the database. The database will be in RDS on a tiny instance.

Since the application is single-tenant, that means I need a new set of resources per user. I'd like to I'd like this to be automated in some way where I have say a signup page or API endpoint - doubt I will build a proper UI for this project - where when the 'user' is created that new resources would be deployed to the Kubernetes cluster and the RDS updated.

I will be using AWS for this project since I know it well so that means I am open to using AWS services if they are cheap or free to use. My plan is to use EKS, NLB, nginx-ingress, SQS, Node.js, Vue.js, Aurora MySQL on RDS, built using Terraform so I can destroy when I am not working on it - RDS excluded in destroy.

My question/reason for this post is to get some ideas on where I would start with a project like this on the DevOps side of things. I have a rough Node.js API done and nearly done the Vue.js UI as well so time to do the DevOps stuff past local dev.

If you were building a system as I've described, what would you do and more important to learning, why would you do it that way? What are some security concerns I should look out for?

If anything is unclear just ask

https://redd.it/lgyopr
@r_devops

Deploying message queues in a testing-compatible way

Hey, How do you deploy message queues (e.g. topics, subscriptions, all of this stuff)?

My wish is to find a mechanism that is testing-compatible like DB migration. What do I mean by that? The definition of the DB structure can be defined using some code-level migration framework, this allows to find (some) deployment failures early during testing. Put in other words, when my integration tests pass on a developer machine with docker-compose - I know that the migration is valid. One more risk is tamed. Could you propose a MQ definition technique that will work both in the cloud and over docker-compose locally?

I use multiple different MQ products so not referring to a specific stack or cloud.

https://redd.it/lhhulh
@r_devops

Ubisoft's new tech blog: How transverse teams allow innovation in a competitive industry

Ubisoft Montreal has a new blog about anything tech. In this first article, they talk about how transverse teams can help devs to still be able to innovate, in an industry where it is sometimes hard to allow such room.
Enjoy the reading!

https://redd.it/lhncj8
@r_devops

What do the best DevOps Engineers do every day? (I want your input)

I'm curious what your thoughts are on the activities that DevOps-dedicated engineers do every day. Doesn't necessarily have to be someone in a "DevOps" function. Could be a software engineer with DevOps.

What would drive their daily routine? What priorities do they set?

How does this set them apart from standard ops people?

https://redd.it/lhjjil
@r_devops

A short article that explains how to deploy a NodeJS app packaged with Docker on AWS ECS

I just shared my experiment of deploying a Docker App on AWS thanks to ECS

https://dev.to/raphaelmansuy/deploy-a-docker-app-to-aws-using-ecs-3i1g

I have not tried Kubernetes on AWS: pro or cons compared to ECS ?

https://redd.it/lhl6nm
@r_devops

DevOps survey

Hello everyone!

I am an undergraduate student and conducting a survey for my 4th year project on the maintenance of DevOps artifacts. The survey is targeted at software practitioners currently working or have had experience with DevOps in the industry.

The survey has ethical approval from the University of Glasgow. no. 300200051

I would highly appreciate it if you could take part! It should approximately take 10 minutes or less to complete the survey.

The results will also be posted to get some more insight on DevOps in the industry!

https://glasgow-research.onlinesurveys.ac.uk/devops-in-the-industry-2

https://redd.it/lhiynt
@r_devops

wait: can I use terraform instead of docker-compose?

I'm just reading about terraform, and I see that you can specify Docker as provider.

Does that means that in principle I could use Terraform to do everything I was doing with docker-compose.yml?

Am I missing something?

What are the pros/cons, a part -I guess- the obvious one that terraform could be provider independent ?

https://redd.it/lhhnjn
@r_devops

Get DevOps Assessment

Want to improve your DevOps capability? With DevOps assessment, we will help you to identify your strengths, weaknesses and suggest next steps for your DevOps journey. https://www.softwebsolutions.com/devops-survey.html

https://redd.it/lhgtox
@r_devops

Jenkins Notification Update

Is it possible to send a custom Jenkins build notification to one slack channel while at the same time sending the default error message/notification to another channel?

So for example:
job success message will be sent to #release, and
build failure message will be sent to #jenkins channel

https://redd.it/lgxxqp
@r_devops

Custome metrics pod kubernetes

Hello,

I'am new in Kubernetes world.

I want to deploy a pod with custom metrics. I want to generate a random number and expose it to prometheus to monitor.

How can i do that ?

https://redd.it/lgqwqv
@r_devops

Developer needed for SOAR/SIEM platforms

We're looking to hire a developer with skills building plugins/add-ons for popular SOAR & SIEM platforms. This would be ongoing work for at least 3-6 months.

Please reach out if you are a good fit.

https://redd.it/lhubwj
@r_devops

Artifactory "best practice" for automatically deleting old artifacts from Jenkins.

I'm currently looking for the best way to automatically delete old artifacts uploaded from Jenkins.

Our folder structure in Artifactory goes "project name/release version/build number".

Using rtBuildInfo I'm able to limit it to automatically delete old builds. However, this doesn't take into account version numbers. I can get around this by using a different build name depending on versions, but this quickly gets messy as we create new release versions.

The project does not use Maven or other common build tools supported by the Jenkins-Artifactory plugin, so I'm limited to using most of the generic pipeline functions.

Any ideas for best practice here would be greatly appreciated.

https://redd.it/lhu8qv
@r_devops

How to manage multiple single-tenant infrastructures

Can anyone guide me on how to efficiently manage multiple single-tenant infrastructures and avoid common traps and pitfalls?

I have vast experience in managing multi-tenant SaaS product - one infrastructure with multiple users. We use Terraform for IaC and Azure DevOps for deployment. It's pretty easy and straightforward on how to do upgrades of infra, etc.

But can't really imagine how to maintain IaC with upgrades and deployments for multiple infrastructures at once. Furthermore, different customer can have different infrastructure configuration (scaling or some additional resources provided). Moreover, the creation of a new environment must be automated with correctly configurated infrastructure and deployment).

If I use Terraform I assume I should go with state per tenant with a configurable module of our infrastructure, am I right? How is deployment handled in this scenario? I don.t wanna manage pipeline per customer and I would rather use one general configurable pipeline for all customers, is it a good idea?

https://redd.it/lht8jp
@r_devops

AWS EC2 launch configurations vs launch templates

[Original Source](https://brennerm.github.io/posts/aws-launch-configuration-vs-template.html)

At first sight AWS launch configurations and templates may seem very similar. Both allow you to define a blueprint for EC2 instances. Let's have a look at their differences and see which one we should prefer.

## They grow up so fast

Launch configuration are old. In terms of cloud technologies they are essentially ancient. During my research I found articles that date back to 2010. It's hard to find exact details but it seems like they have been introduced together Auto Scaling Groups (ASGs) or shortly after. This also explains why there are only compatible with ASGs. Want to create a single EC2 instance based on an launch configuration? That is not going to happen.

Settings that are supported include:

- the EC2 image (AMI)*
- the instance type (e.g. m5.large)*
- an SSH key pair to connect to the VM*
- the purchase options (on-demand or spot)
- an IAM profile
- one or more security groups
- a block device mapping to specify additional storage volumes
- a few more minor things

_* marks required values_

Changing any of these parameters is not supported due to launch configurations' nature of being immutable. This means instead of updating it in place you need to delete and recreate it.

All in all launch configurations have a very specific use case and a set of configuration options limited to the basic parameters. Let's see how launch templates compare.

## The hot stuff

The first big difference is the wider range of AWS services that are compatible with launch templates. Additionally to ASGs it can be used in managed EKS node groups and to create single EC2 instances.

Regarding configuration options, they support a bit more than launch configurations like network settings and a few more advanced details (interruption behavior, termination protection, CloudWatch monitoring, ...).


The main difference here is that every setting is optional. As you can see in the image above you can set the value "Don't include in launch template" for every parameter. You can essentially create a launch template that specifies nothing. That's kinda pointless but you get the idea.

Combining this with the ability to source values from existing templates and you can start to imagine all the options that arise. Similar to something like Docker images you can start to create your base template(s) and inherit more specific templates from them.

As nice as this may sound I just want to advice you to be cautious with doing this. Depending on organization and your upfront template "architecture" planning this may work really well. But it hasn't been just once that I've seen this ending up in dependency hell. (including rhyme in blog post ✅) So think about if you don't wanna stick with independent templates especially when factoring in the next feature.

Launch templates support versioning. Meaning while a single version is immutable you are still able to make modifications which will result in a new one that you can refer to. In my opinion this workflow provides a much better user experience compared to the delete and recreate approach that you need to go through with launch configurations. But again it adds complexity of managing the references in your ASGs and child templates.

## Which is the better choice?

So, how did both do? Is there a clear winner or can I give you at least a recommendation which one you should prefer?

I'm not sure how things really evolved so take the following with a grain of salt. To me it seems like launch configurations have been created out of necessity when introducing ASGs. Afterwards the folks from AWS noticed that having an EC2 blueprint could be useful for other services as well. Cause it was probably easier to create something new instead of making the existing solution more generic they