Seeking help with moving a locally running, short-lived Docker container to CI / deployment process?

I own a git repo of end to end tests. They are portable in such a way where anyone can clone the repo, create a docker build, then run docker compose up. These tests are designed to run against a publicly available web app. There are no other dependencies to worry about which is quite nice. There is no need to worry about test result reporting either. Absolutely everything is taken care of by the Docker container. Essentially it runs a shell script via CMD and exits on its own when the command completes after 10-11 minutes.

However this type of thing is kind of in a "silo" because it's run from my local laptop, manually triggered after a deployment, and triggering these tests is not automated in any way.

Does anyone have any tips and resources to integrate this type of thing into Jenkins? It would be great to maintain the ease and portability of running these tests via Docker Compose.

I thought about maybe running this container in ECS (we are an AWS shop) but couldn't tell if it was overkill. I also think it wouldn't help because I would just be trading my local laptop for ECS and still have the same exact problem. I would still maybe need Jenkins itself to automate some kind of schedule/trigger for this (and I'm mostly clueless around Jenkins).

Thanks in advance for any help. Appreciate it!

https://redd.it/z0zkqp
@r_devops

Starting out with my first project

Just deployed my first application (a simple bookstore) on the test cluster. The application consists of a frontend and backend in Go interacting with Postgres DB, currently made accessible through ingress (no proper dns yet :( ). Had to deal with db connection issues and making the frontend accessible. Learnt/learning a lot during the process. I am currently creating a helmchart for the application.
As the next iteration to this project, I am planning to host this on AWS using Terraform. Later on add some Gitops functionality.

Any suggestions on workflows / functionality I need to add into it to gain more experience is much appreciated.

https://redd.it/z3v8v3
@r_devops

30% OFF Azure Data Factory Basics for Azure Data Engineer DP203+Lab

Learn Data engineering on Azure Data Factory real world projects in 1.5 hour. Start your career as Azure Data Engineer - https://www.udemy.com/course/azure-data-factory-for-azure-data-engineers-with-hands-on-labs/?couponCode=E2F0178F625C337E3861

In this course, you are going to learn how to build data pipelines with hands-on labs using azure data factory

This course covers demos for all concepts used in real-world projects from an industry expert with hands-on Labs.

Anyone can spend 1.5 hours time and become an Azure Data engineer!

This course will provide what is data factory, what are the various core components in azure data factory.

This course is useful for those aspiring to become azure data engineers.

Those who want to switch their career from soft engineering to data engineer

To expand their skills from AWS, GCP data engineer to AZURE data engineer

to gain an in-depth understanding of the latest trends in azure data engineering

This course covers the following topics.

Basic concepts of various components in ADF

What are ETL and ELT

What is Linked service,

Demo for creating Linked service

What is a data set,

Demo for creating Data set

What is Integration run time

What is an Azure Data lake

Demo for creating Azure data lake storage account

What is Cosmos DB

Demo for creating Cosmos DB

What is COPY activity

Demo for creating a data pipeline to move data from ADLS 2 to Cosmos DB using COPY Activity

What are control flow activities-

Get Metadata activity

Demo for getting Metadata activity

Filter activity

Demo for Filter activity

ForEach activity

Demo for ForEach activity

Demo for other Control flow activities

What is DATA FLOW Activity

Demo for removing Duplicate data using DATA FLOW Activity

The course is useful not only for freshers but also for experienced data engineers to gain more skills in azure data engineering

https://redd.it/z3wfg2
@r_devops

Does lead time - cycle time = queue length?

I've just started reading Accelerate and I'm trying to wrap my head around the tempo and stability metrics and where Kanban metrics fits in.

They mention development lead time, which they define as "from commit to production time". Do they mean "commit" to doing the work, or "commit" the code?

Both are very different things.

I would imagine the high level software delivery team performance measurement they would be trying to get would be the length of time it takes you to complete work after agreeing to do it? Or is it purely focused on once you have written the code, how long it takes that to get into production?

I'm starting to realize that their are two types of backlog items:

1) ideas or minor issue fixes that you might do one day (not committed)

2) work that you have agreed and committed to which is added to your queue

How do you handle these differences?

https://redd.it/z186s3
@r_devops

Going from junior to mid-level

Been in industry for 3 years and I've come to learn yoe doesn't mean anything. I still feel like a junior and need help being told what to do next sometimes. I might start looking a new job and one of the things that's worrying me is being able to bridge the gap between junior to mid. I don't want to be in a position where they think I'm more senior than I really am and then when start the job it all goes to shit.

What are some things I can do/you did to make yourself go from junior to mid or senior? I want to be able to hold my own.

https://redd.it/z17g9h
@r_devops

Couple of doubts regarding YAML's syntaxes.

Everything in YAML, at least visually seems to be the exact same key: pair combination, I've familiarized myself with all the YAML syntaxes but I'm having a really hard time figuring out whether what I'm entering is an an Array, Sequence, Set, Map, Collection, Dictionary or an Object.

1. I simply use [ \] for simple values and { } for key: value pairs. The only real difference I could notice is that when using [ \] , you can have duplicates inside but if you enclose the same content within { } , then the duplicate items disappear. Am I missing something more important here?

2. Also can someone please explain the purpose of ? Sets in YAML, is it only there to assign null value to the key and delete any duplicate entries? And why are they emphazing in bold that it's "Unordered" unlike "Ordered" Lists.?

3. You can use * to call all the values of & , so why use << on top of it, isn't it redundant?

ref: &ref

girl: F

boy: M

Gender: *ref

4. The most confusing of all for me is Tags, I read the YAML documentation about tag handles, verbatim, shorthand. The more I read, the more confused I get.

I know that !! is used to specify data type and also you can enter a tag using the key: value pair method, but have no idea on how to properly use the single quotation ! tag and the %Tag directive.

%TAG !e! tag:example.com,2000:app/

\---

!e!foo "bar"

!<tag:example.com,2000:app/foo> "bar"

To me this code seems to function exactly like an Anchor / Alias combo rather than an actual Tag. So essentially, we're replacing the !e! with the URL. Isn't the primary goal of #Tags is to help find your content and resolve your search queries faster, I don't see it happening anywhere here.

&#x200B;

Sorry if the questions are too basic or outright dumb.

https://redd.it/z0xkec
@r_devops

Bicep templates

Hello.

Is there anywhere a big repository with bicep templates which you can look up for examples? Microsoft have a few but usually they're not filled with "values" but rather empty and from there i'll have to try my best until something doesn't work and then fix it.

Currently it just feels like i'm at times reinventing the wheel since some resources won't accept values which others would usually be accepting.

https://redd.it/z10yl2
@r_devops

do you action alerts in non-live environments?

At my company, we have alerting set up for all envs (dev/qa/live). I as the devops person responsible for multiple services, each of them having 10-20 alerts set up, am in charge of setting up the alerting and maintaining it/steering the dev team to do it.

Now our live services usually work fine and we rarely get alerts fired there, but a lot of times, it's the dev/qa alerts that fire, most of the time because of some failing dependency or bad data used in dev, or similar reason.

I'm usually inclined to ignore these and focus on actual work, but I'm pushed from above to action these alerts - refine thresholds, tamper with queries, etc. (or make sure developers take care of them).

How do you approach alerting in your teams?

https://redd.it/z0tyrt
@r_devops

Please recommend Student Information System solutions for higher ed (colleges and universities)

Looking for potential solutions to replace our university SIS which is Oracle People Soft. A few requirements our university needs for the SIS to have:

1. an active directory used for account creation, reading, updating, deleting, and authentification.
2. a built-in LMS like moodle.com to handle student enrolments, and attendance, create exam schedules, and be able to process final grades
3. a dashboard for reporting and analytics
4. a complete and comprehensive program for accounting control to track daily financial transactions
5. an online payment gateway for student payments and account balance updates
6. the SIS should be able to process online applications and student placement information
7. workflow processes for making grade changes, creating the course and major sheet exceptions

Currently, the existing solution has MOST of the things listed above as custom apps and integrations. I understand there is no perfect solution that will offer all of these, but any recommendations to handle most of the requirements with a ready-made solution are highly appreciated. Obviously, some things will have to be made as custom integrations, which is no problem, I'm more than welcome to opinions and feedback on potential options. Thanks in advance!

https://redd.it/z1ai3p
@r_devops

How much collaboration you need to fix your config when your deployment is broken?

I wonder if you work on isolation or you set up some kind of war room when you are facing a minor or major outage in your application.

We are thinking on adding capabilities for online simultaneous editing of config files for quick fixes or long term configuration definition, but I really don't know if it makes sense, specially if you are using GitOps or any git based configuration workflow. I can see some value, but I don't know if it is better to open a Zoom call and share your screen to work on it instead of having some kind of Etherpad like experience, even if we (Monokle) can do real time validation of your config files.

Do you see value in such a feature? What would be the requirements that would make this unacceptable if there are not there?

https://redd.it/z48k3b
@r_devops

Should i migrate from Kustomize to Helm?

Hi!
I'm currently facing some "limits" of a kustomize approach.. Basically i need a sort-of "Preview Environments" based on PR. I managed to get something that works using ArgoCD and it's PullRequestGenerator which use a specific kustomize overlay to deploy those environments when a PR is opened.
But the problem is: I need to pass some values to kustomize from ArgoCD, and i guess that there isn't a way of doing that easily.

Let's say that a PR is opened from a branch: feature-12

This will trigger a GH Action that push the container to a private registry tagged with branch name and other stuff.

ArgoCD then will be notified from a webhook that a PR is opened and will create a namespace (named like the branch) with all the stuff deployed.

From ArgoCD i can pass to kustomize some values, like prefixName and images, and that's fine.

But ideally the stuff just deployed from the PR should be reachable from specific URLs like feature-12.example.com or feature-12-api.example.com..

I cannot pass that value to kustomize.
So i think that the only way is to migrate to Helm and then pass those value with Values.yaml

Any suggestions about that?

P.S: Also kustomize insert namePrefix on a lot of stuff and those stuff are usually also in some container environments of the deployment.
Like an env that refer to: redis://redis-service:6379 when deployed to dev using dev as a namePrefix, will become: redis://dev-redis-service:6379, so the env must be patched as well. And while this is fine in a scenario with 3/4 different environments, that's not feasible in a scenario with multiple environments dinamically created by ArgoCD

https://redd.it/z49fdx
@r_devops

How to access a Cloud SQL instance in GCP from a different project?

I have 2 GCP projects. Each has its own VPC and one of them has a Cloud SQL instance. Thing is, after doing VPC peering between those two separate VPCs I still can't access the database from the VPC in the other project. Anything I'm missing here?

https://redd.it/z49wud
@r_devops

Is it possible to update .net core WebAPI in IIS without taking it down?

I'm currently using jenkins build server, basically building+deleting+deploying(copying), but the endpoint goes down while doing this.

Is it possible to use f.ex. Azure DevOps to update WebAPI in IIS without taking the application down?

https://redd.it/z49pvs
@r_devops

Need some help starting out

I've beed a software dev for like 5 years and now trying to get into devops, so I was playing around with networks and terraform.

I am trying to set up a haproxy and multiple services behind it. Haproxy has public ip, servers with services don't.

&#x200B;

What is the proper way to configure hosts on the network that don't have public IPs? They can't access internet so I can't download required packages. Should they use a proxy to access internet? If yes, how? Or should they just have public IPs and firewall? Or should they download packages from local storage?

&#x200B;

Would really appreciate any tips.

https://redd.it/z4czak
@r_devops

Where do you host your software versions for users to download?

I was uploading all my installers to our web hosting provider KnownHost, but they said it was not for storing those types of files. We moved to AWS with our own custom web server but that's giving us trouble. I'd like to move to something standard, we need to transfer our installers from our Azure build server, and then have a link to download the file.

Any recommendations?

https://redd.it/z4d26q
@r_devops

A typical Friday

Early morning on a black Friday, surfing around to see if there is anything to buy but end up with nothing, so I went to look at a personal project that hasn't been touched for a while, then ... make test failed due to Azure moved or dropped some of its open datasets, try to find another one but found azure cli is not installed, running brew but got 503, then found github is having problem with packages and pages.

TGI"B"F

https://redd.it/z4ftds
@r_devops

Confusion question

I am often confused when asking online community for help with a technical problem. The feedback is often something isn't clear or needs more details. These question are often gears to very specified tool. I am mostly trying to state my problem with a few lines of code and possible document I am using. My question is how does one formulate a very clear and detail tech question about something they are not fully understanding ? it seems like some people don't understand the reason something is unclear because the person asking for help doesn't know how to explain it. which often happens to newbies.

https://redd.it/z0w0cm
@r_devops

DevOps Gig

Can someone share any resources where we can find devops side gigs ?

https://redd.it/z0nuex
@r_devops

Are you using Cloud Composer in production?

Hi, we are looking into Cloud Composer to stop managing Airflow clusters ourselves (which is a pain). Are you using Cloud Composer in production? If yes, how are you dealing with environment upgrades (snapshots, etc.)? Are the upgrades smooth? I have seen that Cloud Composer 2 Environment upgrades is still Beta functionality.

https://redd.it/z47k8m
@r_devops

open-appsec (open-source machine-learning based WAF) - updates

We have added a new Killercoda playground that allows deploying open-appsec for NGINX - https://killercoda.com/open-appsec/scenario/simple-appsec-for-nginx

For more information about this option see the docs at https://docs.openappsec.io/getting-started/start-with-nginx

If you haven't Star the GitHub project already, please consider doing it. It helps us as a young project: https://github.com/openappsec/openappsec.

Thanks and have a great weekend!

https://redd.it/z4np2f
@r_devops

What’s the coolest thing you did this year?

Just curious what cool shit other people have been up to over the past year or so?

Any plans for some cool shit for next year?

Interested in both personal / professional projects.

https://redd.it/z4pgrn
@r_devops