Looking to change from Bash files that transform text files to a CI/CD system

Hi,

Currently I have a bunch of shell files running bunch of Python scripts that receive a zip file, which I manually unpack, and transform data in different stages.

The bunch of Python scripts part is necessary. I would like to set up this in a CI/CD system where myself and other devs can alter the python scripts on our git repo. Then at the end of the day scripts are pulled, the data gets processed in stages, and if one of the stages fails, we get an e-mail or similar.

I guess I described it too much, but I was looking at Jenkins. Would this be the ideal solution?

Thanks in advance

https://redd.it/lh5c82
@r_devops

What are your best tips for avoiding alert fatigue?

Is it a matter that can be fixed by choosing the right system, or just by company culture and a dedication to removing noise?

https://redd.it/lh3wkw
@r_devops

Knowing what you know now if you could best advise junior devs on how to be competitive for dev ops what would you say is number 1 thing?

Im a typical junior dev but a big picture with solid communication skills who enjoys planning code with goals and i enjoy leading others.

Id like to take the route that is the most logical and also the strongest competition wise to enter into the dev ops engineering field.

Which of these is number top of your list if you are a junior dev aiming for dev ops?

View Poll

https://redd.it/lh6yeg
@r_devops

Resources to get the networking fundamentals I need?

I've been working in the DevOps world for about half a year now. Got some certs and experience at a client on azure. Currently diving into openshift for the next client but i keep realising my networking knowledge is sub-par.

Did this fundamentals course on udemy but it was basically what is a router, switch etc but not how to configure all that stuff. Some colleagues don't think this is relevant, but i just think different and wanna have a solid foundation.

So i learn best with video/reading + labbing. Do you guys have some advice on how to get up to date in networking?

Saw David bombals CCNA for instance? Also heard that it the CCNA might go too deep?

I really wanna go quite deep and really understand networking (eventually want to cloud architect in 3-5 years) and don't wanna half-ass my job.

TLDR: Work in DevOps role , wanna get a solid foundation in networking and looking for resources. Any suggestions?

https://redd.it/lhet5u
@r_devops

Terraform Troubleshooting Methodologies

This post covers the troubleshooting and general usage techniques one might want for all the main subcommands of terraform. Hope this is some useful stuff folks!

init

Scroll through the list of errors until you see one you think you understand. Fix it. Hope that also fixed some of the other ones. Repeat. Drink.

plan

In theory, this command will tell you what terraform is going to do. In theory, giving Quentin Tarantino enough money will tell you what he's going to do, so that's kind of like this command.

apply

Directed-acyclic graphs are great until something goes catastrophically wrong and you literally can't know what it is, on account of your run being locked harder than Tomas Haake's calves. He should just a play normal beat - "Boom, da! Boom boom, da!"

state list / state show

These tell you about everything terraform knows about - so maybe they should be named after your sister!

import

P̴ͅr̷͉á̡̡͓̫͎̱̼̦y̤̥͓̯,̧̛̝ ̻͜͟f̵͔͉̮͞o̷̤͔o̵̱͙̜̭̖̹̕͜l̵̵̹̪͈̣!̺̱̹͙͕͘ ͍̘͙̘ ̡̝̜̙̱̯͎̭̩͝P̪͚̝͈̟͙͞ͅͅr̷̲̤̣̤̦a̙̖͖̦y̧̘̻̮͎̕,̛͖̀͜ ҉̠̯̦̪͔̪a̡̩͡n̢̢̮̗͇͚̟̰̰͡d̸̻̝̮̟͚̯̭̲́ ̧̤̦̲̖́͝r̨̻̼̯̰u̜͔͎̯͝͞n҉̛͖̺̖̩͉̟̩.̟͓̠̀͞ͅ ̶̢͍̥̥̖̩̞̙ ̶̺͢O̡̤͉͈̠̥̱̲̝̰͟ǹ̷͖͓͈͈̀l̕͏̬̪y̴̛̤̙͝ ̹̯̩f͏̵͓ͅo̖͚̹͍͚͈̩̺ͅo҉̴̞͉̜̩̩̝͈ͅḽ̷̛̤̦̘̫͖s̪̫̜̀ ̵͉̙̲͝ȩ̖͙͎͔̠̬̪͝ͅn̷͈͕̙͕͚͕̟ͅt͉̘͙̼̱͚̯̱͚ę̯̺̠̰͝͡r̨̛̠ ̷̡̢̺͓̗̟͖̤̦̖͔h̕͏͔̦ẹ̮͈͈̤͍͚̀͜r͕̮̜̱͔͔̀͘e̘̺̟̠͈!̨̣̝̥̦̟̯͞ͅͅ

https://redd.it/lhe3sd
@r_devops

Docker-compose based CI/CD?

Has anyone implemented or used a CI/CD pipeline that runs docker-compose services?

I’ve used CircleCI and GitHub Actions at previous jobs, and my new job uses a self-hosted TeamCity instance that is able to run your docker-compose services as build steps in the CI/CD pipeline.

This is incredibly convenient compared to something like CircleCI, as it’s way easier and more consistent to specify testing/build steps in docker-compose and run them both locally and in CI/CD. HOWEVER, our DevOps team did a lot of work to make it possible for TeamCity to do this.

I was wondering if anyone has any experience with a platform that natively allows the use of docker-compose in the CI/CD pipeline? Or alternatively, if anyone has experience implementing this kind of solution.

https://redd.it/lhddud
@r_devops

Do you know how to program or write scripts and what is the difference?

And which one matters when hiring?

https://redd.it/lhcc31
@r_devops

AWS CDK: Unable to access secretArn if I run Lambda on SAM Local

I'm reposting this from another sub. Maybe you guys can help? :)

I tried creating a secret in Secrets Manager via CDK:

const secret = new secretsmanager.Secret(this, 'Secret', { /* options */ });

I'm trying to get `secret`'s `secretArn` and pass it into a Lambda function's environment:

const opts = { environment: { SECRET_ARN: secret.secretArn } };
const lambdaFunc = new lambda.Function(this, 'simoncpu', opts);

My problem: In a real environment, I can read the `secretArn` via `process.env.SECRET_ARN` but in SAM Local, all I can see is something like `simoncpuSecret12345`.

How can I access the secretArn in SAM Local?

https://redd.it/lhbg0c
@r_devops

Has anyone used AWS's DevOps Guru?

AIOps is becoming a thing, as clients are starting to ask about it. I'm about to start a poc for an e-commerce platform, and will update this post with my findings.

In the meantime I'm curious if you guys used it, have thought about it or used alternatives.

Cheers

https://redd.it/lgxg13
@r_devops

How to write terraform in a provider independent way?

I thought that the advantage of Terraform was the ability to write infrastructure independently from the cloud provider.

However after a quick look, I don't see it...

I mean I don't see a way of creating a virtual machine in an abstract way independent from AWS or Azure.

https://redd.it/lhkj1y
@r_devops

Monolith vs. Layers?

Start a new Infrastructure as Code (IaC) terraform project may be very challenging. This article explores 2 opposite implementations in detail to organizing your code: a single monolith or multiple layers. Please feel free to share if you have any other advice!

https://redd.it/lhkus6
@r_devops

Deploy machine learning NLP models with Docker Swarm

Hi,

As we're using Docker Swarm extensively behind NLP Cloud (https://nlpcloud.io), I made a post about how to set up Docker Swarm and deploy a cluster with Docker Compose:

https://juliensalinas.com/en/container-orchestration-docker-swarm-nlpcloud/

I also included important details about deploying machine learning NLP models to production, and discussed a couple of things regarding Swarm vs Kubernetes.

Hope you'll like it!

https://redd.it/lgueem
@r_devops

Help on design for learning project - single-tenant monolithic application for multiple users

I am working on doing a small-ish project that is to learn about deploying monolithic single-tenant applications on Kubernetes. The project is a simple Node.js backend with a Vue.js front end (think of hello world with an admin dashboard to change the text) and uses MySQL for the database. The database will be in RDS on a tiny instance.

Since the application is single-tenant, that means I need a new set of resources per user. I'd like to I'd like this to be automated in some way where I have say a signup page or API endpoint - doubt I will build a proper UI for this project - where when the 'user' is created that new resources would be deployed to the Kubernetes cluster and the RDS updated.

I will be using AWS for this project since I know it well so that means I am open to using AWS services if they are cheap or free to use. My plan is to use EKS, NLB, nginx-ingress, SQS, Node.js, Vue.js, Aurora MySQL on RDS, built using Terraform so I can destroy when I am not working on it - RDS excluded in destroy.

My question/reason for this post is to get some ideas on where I would start with a project like this on the DevOps side of things. I have a rough Node.js API done and nearly done the Vue.js UI as well so time to do the DevOps stuff past local dev.

If you were building a system as I've described, what would you do and more important to learning, why would you do it that way? What are some security concerns I should look out for?

If anything is unclear just ask

https://redd.it/lgyopr
@r_devops

Deploying message queues in a testing-compatible way

Hey, How do you deploy message queues (e.g. topics, subscriptions, all of this stuff)?

My wish is to find a mechanism that is testing-compatible like DB migration. What do I mean by that? The definition of the DB structure can be defined using some code-level migration framework, this allows to find (some) deployment failures early during testing. Put in other words, when my integration tests pass on a developer machine with docker-compose - I know that the migration is valid. One more risk is tamed. Could you propose a MQ definition technique that will work both in the cloud and over docker-compose locally?

I use multiple different MQ products so not referring to a specific stack or cloud.

https://redd.it/lhhulh
@r_devops

Ubisoft's new tech blog: How transverse teams allow innovation in a competitive industry

Ubisoft Montreal has a new blog about anything tech. In this first article, they talk about how transverse teams can help devs to still be able to innovate, in an industry where it is sometimes hard to allow such room.
Enjoy the reading!

https://redd.it/lhncj8
@r_devops

What do the best DevOps Engineers do every day? (I want your input)

I'm curious what your thoughts are on the activities that DevOps-dedicated engineers do every day. Doesn't necessarily have to be someone in a "DevOps" function. Could be a software engineer with DevOps.

What would drive their daily routine? What priorities do they set?

How does this set them apart from standard ops people?

https://redd.it/lhjjil
@r_devops

A short article that explains how to deploy a NodeJS app packaged with Docker on AWS ECS

I just shared my experiment of deploying a Docker App on AWS thanks to ECS

https://dev.to/raphaelmansuy/deploy-a-docker-app-to-aws-using-ecs-3i1g

I have not tried Kubernetes on AWS: pro or cons compared to ECS ?

https://redd.it/lhl6nm
@r_devops

DevOps survey

Hello everyone!

I am an undergraduate student and conducting a survey for my 4th year project on the maintenance of DevOps artifacts. The survey is targeted at software practitioners currently working or have had experience with DevOps in the industry.

The survey has ethical approval from the University of Glasgow. no. 300200051

I would highly appreciate it if you could take part! It should approximately take 10 minutes or less to complete the survey.

The results will also be posted to get some more insight on DevOps in the industry!

https://glasgow-research.onlinesurveys.ac.uk/devops-in-the-industry-2

https://redd.it/lhiynt
@r_devops

wait: can I use terraform instead of docker-compose?

I'm just reading about terraform, and I see that you can specify Docker as provider.

Does that means that in principle I could use Terraform to do everything I was doing with docker-compose.yml?

Am I missing something?

What are the pros/cons, a part -I guess- the obvious one that terraform could be provider independent ?

https://redd.it/lhhnjn
@r_devops

Get DevOps Assessment

Want to improve your DevOps capability? With DevOps assessment, we will help you to identify your strengths, weaknesses and suggest next steps for your DevOps journey. https://www.softwebsolutions.com/devops-survey.html

https://redd.it/lhgtox
@r_devops

Jenkins Notification Update

Is it possible to send a custom Jenkins build notification to one slack channel while at the same time sending the default error message/notification to another channel?

So for example:
job success message will be sent to #release, and
build failure message will be sent to #jenkins channel

https://redd.it/lgxxqp
@r_devops