I need help setting up CD/CI with Docker Compose

Hi,

I'm a full-stack web dev who needs to break into DevOps (my app got more complex than I anticipated) and I'm really new to Docker and pipelines forgive me if my question makes no sense or has already been answered.

I set up my dev environment via a compose file with some containers, but I'd like to eventually work towards something like this:

\- I push code to GitHub

\- some CD software notices and starts a testing environment, with my new code and some throwaway dbs (postgres, redis), then runs my unit and integration tests.

\- if there was no error, then my testing environment gets shut down and the new code is run in prod, on my actual dbs (the dbs wouldn't be in containers).

I've never really done CD / CI before, so I'm completely lost.

​

Thank you!

https://redd.it/pxb6rm
@r_devops

Have you done Cloud DevOps

Our management wants to expand our computing grid to the cloud, but our setup is not straight forward. We run \~100GB RAM simulations from \~100s repositories at the same time with \~10GB output log, while using semi-custom SCM (something on top of git). Locally everything is easy, just have one main NFS, repos get cached on this share and the output is thrown back to this share.

We don't have a dedicated DevOps, even though the department is in hundreds of people. We (who are definitely not DevOps) have two options:

we need to duplicate our cache, but this means the whole new NetApp in the cloud. (seems like a dumb solution to just close what we have locally, but into the cloud) Easy(?), but expensive(!)
(ATTENTION crazy idea ahead) Compile a docker image for every simulation. Package all repos and all tools, then upload the image on the cloud, execute. (Image is \~30GB, seems crazy to me)

The ideal would be if we could attach Cloud nodes over VPN to our NFS, but it is gonna be slow. Have you ever had any similar problems? Who to ask for help?

TLDR: how to allow Cloud to access 100s of GB of local files to run simulations?

https://redd.it/px7jnu
@r_devops

Free intro to Linux commandline/server course starting 4 October 2021


This free month-long course is re-starting again on the first Monday of next month.

This course has been running successfully now every month since February 2020 - more detail at: https://LinuxUpskillChallenge.org - daily lessons appear in the sub-reddit r/linuxupskillchallenge - which is also used for support/discussion.

Suitable whatever your background, and aims to provide that "base layer" of traditional Linux skills in a fun interactive way.

Any feedback very welcome.

https://redd.it/pxe7cd
@r_devops

Migrating from cloud to VPS

A sad story short - one of our clients wants to migrate around 350 different application and DB servers from Azure VMs + Cosmo DB to a basic kvm hosting on some random small hosting provider.

The problem is that currently all of the environments use Terraform for IaC and we use managed Kubernetes service for part of the applications (the rest are on VMs).

The new provider doea not have any API or managed services, just pure basic VPS with SSH access.

My approach is to spin up multiple K8s and Rancher clusters to use for all apps from the current Kubernetes clusters and few Ansible managed VMs for the databases (MySQL and Postgre). Create Ansible playbooks for the rest of the applications, then do rsync of all stateful (yes) apps and database syncs between the old and new databases before the DNS switchover.

What would you do differently?

https://redd.it/pxaziw
@r_devops

Production Kubernetes: Building Successful Application Platforms ebook

The brilliance of Kubernetes is how configurable and extensible the system is, from pluggable runtimes to storage integrations. For platform engineers, software developers, infosec, network engineers, storage engineers, and others, this book examines how the path to success with Kubernetes involves a variety of technology, pattern, and abstraction considerations. Learn and read about this ebook from here: Production Kubernetes: Building Successful Application Platforms ebook

https://redd.it/px9e0t
@r_devops

How to avoid bad assumptions during incidents

https://incident.io/blog/how-to-avoid-bad-assumptions-during-incidents

Having been a responder in countless incidents both large and small, most of my most painful mistakes have come from assumptions, whether they were accidental or plain wrong.

This article discusses an incident where the assumptions were wrong or misleading almost every time we made them, and provides a few tips to try avoiding the allure of assumptions dressed up as facts.

https://redd.it/px576t
@r_devops

End to End DevOps Services | Azure DevOps Services



Ksolves DevOps Consulting Services and Solutions is an end-to-end solution designed to overcome the hurdles presented by the constant market upgrades. The service will allow you to remain on your toes and continuously integrate, deploy, and deliver business processes. We are a DevOps as a service company consolidating application modernization and revitalizing enterprise agility. Get hold of a varied list of services and solutions to automate deployment, analytics, and boost continuous delivery!

https://redd.it/pwwmcj
@r_devops

Senior Network Engineer

I am senior network engineer with 8 years of experience wanting to jump into devops, I already have an associate python cert and working on my cloud certification, any other advise and what skills should I acquire? and also maybe some side projects I can work on so I can really practice some devops skill. thanks!

https://redd.it/pwshfv
@r_devops

Fetching the LOGS

Hello everyone,

I want to fetch the EC2 instance logs and the service running inside it in each region of my AWS account. I am trying to get this logs in ElasticSearch (Opensearch) which is now a service in AWS. So i have used filesbeat but when i insert my ELK(ElasticSearch) in filesbeat.yml it stops responding as it is not getting access to my ELK. I have open all the port in my EC2 instance. I guess this error will be remaining same for logstash also.

Any solution will be appreciated.

Thank you

https://redd.it/pwi4go
@r_devops

Is this a sign of a bad company?

Ive been in my role over a year now. Work from home job. Got a good mac book laptop but only one LED scree to plug the laptop into. I want a second screen and Ive asked my manager many times if i can have one. The company has offices and at thr offices there are atleast 30plus screens exactly like the one I have just standing there acculating dust. I've asked my manager three times for another screen and he always asks me to ask someone else or deflects the question.
I dont get this type of thinking since this is something that will help me do my job better and and help them make more money. Is my thinking toxic or am I at the wrong company?

https://redd.it/pwccs3
@r_devops

How can I set up a Putty tunnel so I can use Chrome to access a webapp in a private subnet?

Is there a write up for this? I've not found it yet. Even a step in the right direction would help :)

https://redd.it/pxnrhj
@r_devops

MFA not allowing WebAuth(touchID) in AWS

I am the admistrator and I have access to the root account. So when a new person joins I created an account for them by giving them appropriate permissions. By this time when that person triee to choose WebAuth option it showed

Missing required input

Please try again. If the error persists, please contact your administrator.

But other methods are working fine.

Any idea why this is happening?
Thanks

https://redd.it/pxp1ps
@r_devops

What is CNCF Strimzi?

There’s just an endless plethora of tools out there for the novice and the expert. Sometimes, it even seems like the open source community can’t catch a break until something new hits the web. This time, we’ll walk readers through another exciting addition to the Cloud Native Computing Foundation (CNCF) – Strimzi.

Read more here:

https://www.p3r.one/strimzi-kafka-on-kubernetes/

https://redd.it/pxpvef
@r_devops

Recommendation for faas/serverless framework?


What is the most mature and recommended serverless/faas franework? Such as openfaas, knative, kubeless, etc. Support for docker containers is a plus

https://redd.it/pxrcwr
@r_devops

Revving up Continuous Integration with Parallel Testing

Is your CI/CD pipeline slow? Do wait times make you feel unproductive? Parallel testing is an indispensable technique for reducing wait times. And mastering it is key to getting the most out of CI/CD.

Continue reading article here:

https://semaphoreci.com/blog/revving-up-continuous-integration-with-parallel-testing

https://redd.it/pxwajn
@r_devops

terragrunt graph External DependencIES ?

% terragrunt graph-dependencies | dot -Tsvg > graph.svg

Module eu-west-1/sqs-catalog-ingestion-main depends on module eu-west-1/eks-ocean, which is an external dependency outside of the current working directory.

Should Terragrunt run this external dependency?

Warning, if you say 'yes', Terragrunt will make changes in eu-west-1/eks-ocean as well! (y/n)

what changes is this talking about? what will this run? I don't want to change/run anything except to make graphs. I cannot tell what/where the changes maybe...

https://redd.it/pxxc7p
@r_devops

What is holding you back from using machine learning features for your projects today?

When I ask development teams why the are not using a machine learning model for a certain use case I keep hearing lots of different answers (listed below). I'm curious to see what people in this subreddit think.

View Poll

https://redd.it/pxx11j
@r_devops

Project Information Management Tool

Hey guys,

Me and my team are wondering if there's a tool (open-source or licensed) from where we can store the information of our projects (such as Title, Completion Status etc.) which will be in .pdf or .doc and index it using references (ex. show me projects that are not completed or show me projects with contract > X). This tool must be web based (host-client) so we can host it in one of our machines, not an online provider. If you can achieve this using different methods, please let me know as well.

Thanks in advance!

https://redd.it/pxqrnj
@r_devops

Azure Pipelines strategy question

TL;DR: Is it better to have a single, grand unified pipeline for a project, or multiple specialized pipelines?

\---

We're migrating from Jenkins to Azure DevOps. I have four different Jenkins projects that I want to duplicate in Azure pipelines. Let's call them CI, CD, Release, and Test. These four projects all work from the same Git repository, based on different triggers.

I did the CI one first. It's a single-stage, single-job pipeline that does everything. Call that Pipeline version 1. It's where I did all of my learning.

For version 2, I thought it best to break up the flow into multiple jobs in multiple stages. With this architecture, I was able to combine all four Jenkins projects into a one-size-fits-all pipeline. It works great.

Now I'm setting up triggers and hooks for this pipeline. But I'm having second thoughts about the one-size-fits-all strategy. Would it be better to break it out into four separate pipelines, each with its own triggers and Git hooks?

Theoretically, with the Infrastructure-as-Code paradigm, either way will work. The Azure pipelines YAML is flexible and versatile enough to do whatever I want it to do. But what's the best way to do it?

And in case that question is unanswerable ("define 'best', Mr. Zyzmog"), what are the pros and cons to the one-size-fits-all vs. four separate pipelines?

https://redd.it/py0p5d
@r_devops

Difference between Reverse Proxy, Load Balancer and API Gateway

I am seeing different companies taking different approach. I am not sure anymore where each should be actually used. On top of that tech like Kong make me question whether API Gateway should be one thing for all. Some perspective into this would be really appreciated.

https://redd.it/py1q54
@r_devops

Is triggering container builds on GIT merge bad practice?

Backstory, I've been a dev for over 10 years, worked with docker/containers for +5 years, deployed multiple production apps for corporates and start-ups.

Recently, I've been hired to build a project that is hosted on AWS/K8s. The client has their own external infrastructure team. I asked them if they could set-up a simple CI pipeline that would compile the docker Images and push them to ECR, each time we merge into master. But they are telling me, in their expert opinion, that we shouldn't kick off builds on merge? However this is what I have done at many Fortune 500 companies and start-ups?

Typically the dev process would be:
Work on Feature Branch -> Open PR to Dev branch -> Approved by PM -> Merge into dev branch -> Open PR from Dev to Master branch -> Approved by PM -> Merge into master branch -> *starts build*

Is this bad practice? If so please can you explain why?

https://redd.it/py217j
@r_devops