How to run integrationtest with docker-compose and jenkins

Hello.

I have a spring boot project in which I'm using a tag to run my integrationtests

My compose file is straight forward with a network, three services in which one depends on the other two. Works brilliantly.

I'm at my integrationtest stage but I can't get it to work.

This is my stage:

steps {
sh 'docker-compose -f docker-compose.dev.yml up'
sh './gradlew integrationtest'
}

and while it builds all my services and I see my health checks on both non-3rd-party services it never goes down and runs my ./gradlew integrationtest

How do I move on to the next cli command when all three services has succeeded?

This is my cleaned up version of docker compose:

version: "2.4"

services:
rabbitMQ:
image: rabbitmq:management-alpine
ports:
- 5672:5672
- 15672:15672
networks:
- backend
subscribe:
image: image from personal registry
ports:
- '8080:8080'
networks:
- backend
push:
build:
dockerfile: Dockerfile
context: .
networks:
- backend
ports:
- '8181:8181'
volumes:
- ./auth:/auth/:ro
depends_on:
- rabbitMQ
- subscribe

networks:
backend:

https://redd.it/pos7d7
@r_devops

A client wants in two weeks as a deliverable a document of how they can implement DevOps internally in their company. They are a traditional business in the sense Devs and Ops are definitely in separated silos. What are some ideas / thoughts that I could insert in this document?

I know that DevOps implementation is unique per environment, organization, client, etc.

But I was thinking... are there any "general rules" I should follow when thinking about how I can set up some guidelines so my client can implement DevOps in their company?

I did some research and found the following recommendations from Google Cloud:



1. Form a Site Reliability Engineering team: Site Reliability Engineers (SREs) are a mix of application SMEs and operations SMEs who are focused on operational reliability through procedural automation. Provide these SREs a mandate to automate every aspect of operations. The SRE team will continue to be responsible for the deployment of all infrastructure, but will do so via repeatable declarative configurations.

2. Only permit automated access to production resources:Taking developer access away from production resources can be a psychological challenge for Agile software developers and should be avoided. Instead of removing access, insist that all production access be scripted and tested in a staging environment. Software engineers will begin to work with SREs to develop reusable libraries and tools to probe and reconfigure production infrastructure. These tools can be used, maintained, and further enhanced by the SRE team.

3. Separate your environments: Google recommends creating at least three environments:
1. Dev: Environments where software engineers have full access to play, test, and do their work. While the SRE team will probably not need regular access to these development environments, the SRE team's automation capabilities can be used in these projects to ease the creation of desired infrastructure.
2. Stage: A single staging environment where the SRE team and software engineers collaborate on implementing and testing automation capabilities.
3. Prod: A single production environment where the SRE team uses scripts and automation developed and successfully tested in the staging environment to affect change.

4. Fully implement Continuous Integration: To free the SRE team from having to deal with production issues stemming from minor development errors:
1. All code paths need associated unit tests.
2. All checked-in code must pass a peer review, which includes checking for unit tests.
3. All checked-in code must pass an automated build process that implements the full battery of unit tests.

It is recommended to set up a code repository structure that is capable of containing an entire ACME enterprise architecture and integrate with the CI/CD process.

5. Fully implement Continuous Delivery: Once a successful build has been created, automate its rollout to the staging environment. Despite the ability to automate the process, most organizations elect to have a manual review of the new build in a staging environment and have someone "press the button" to initiate the phases of the production deployment.

​

​

I don't know if there are other ideas / guidelines I could use for this document. Do you have any ideas from your current / past experience with DevOps?

Thanks in advance!

https://redd.it/pot6gu
@r_devops

Is there a way to monitor per-pod disk i/o for pods using Rook/Ceph including per-osd pod usage?

I am trying to find a way to monitor per-pod/node and per osd/pod disk usage and i/o operations in order to debug some problems with i/o on some nodes.

Right now i am using the default prometheus monitoring instance and kube-prometheus-stack but none of the metrics seem to show any connection between the Ceph/OSD usage and pods/pvcs including the default exporter for Rook/Ceph:

https://github.com/rook/rook/blob/master/Documentation/ceph-monitoring.md

https://redd.it/poqdcn
@r_devops

What are your hours? 9-5?

I was wondering how many hours a day people in devops usually work. With my new job starting soon, I wanted to set realistic expectations for myself for what is standard in the industry. What is your daily schedule?

https://redd.it/povfwj
@r_devops

Aws data api or just expose the database network to the public?

I'm trying to set up a database to connect to an aws lambda, and there doesn't really seem to be any straightforward way to do it. I always have to set up a bunch of other services for access that cost another $20 a month each (expensive for just a hobby project).

Is aws aurora with the data api enabled accessible by a non-VPC lambda (without a ton of setup and costly services)? Or should I just use a regular mariadb RDS, set a really long password, and expose the network to the public?

I gotta say, the lambdas were WAY easier to set up than this. Am I missing something obvious, or is database hosting just like that for now?

https://redd.it/pouul4
@r_devops

Infrastructure as SQL

After having worked with numerous engineering teams of different sizes, as small as few people and up to several hundred, we have seen first hand how difficult it is to safely make and revert changes to infrastructure in a micro services architecture. This is a consequence of the current Infrastructure of Code solutions not having truly good ways to define dependencies between the different pieces of infrastructure.

What software you have deployed on what services and the interactions between them is not a program, it is information about your infrastructure. Changing your infrastructure is a set of operations to perform, a program. A SQL database is a set of information and SQL queries read or change that data. We are humbly putting forward our ideal way to describe cloud infrastructure that is familiar and makes the relations between pieces of your infrastructure first-class citizens. There is more information about Infrastructure as SQL in this post.

https://redd.it/pomj83
@r_devops

Istio Install/Upgrade Strategies

Anyone have experiences (good or bad) with Istio upgrades in production or suggestions on deployment strategies (ie. canary, blue/green etc)

Also interested to hear of experiences with the various install options (ie helm vs operator vs istioctl etc)

https://redd.it/poygit
@r_devops

Key areas in first DevOps position.

I recently started a new DevOps position in my job. I took over from a guy who left the company. I don’t have much actual DevOps experience so I’ve been trying to pick things up as I go.

I spend a lot of my time working on Linux Virtual Machines so I’ve been trying to improve my Linux knowledge and my scripting.

I also plan on getting hands on with Jenkins and other automation tools. I’m also working on a project using AWS and Azure.

Any season DevOps pros please advise me how to succeed at this role. I don’t mind doing the hard work but want to make sure I’m on the right track

https://redd.it/poyeuw
@r_devops

https://firehydrant.io/blog/firehydrant-plugin-for-backstage/

FireHydrant launched the first incident management plugin for Backstage. With the Firehydrant plugin, we introduce FireHydrant’s incident management and analytics in Backstage, so you can quickly and efficiently manage your incidents within Backstage. Teams can keep organized and easily identify information about services like recent active incidents, incident analytics service healthiness, time impacted, and MTT (Mean Time To ) data. The FireHydrant plugin will soon be available on SaaS Backstage provider, Roadie.

https://redd.it/poy15k
@r_devops

What is the most complicated CI/CD system you had to deal with or build one?

Being there many tools and ways of doing this, which is your most complicated setup you had to deal with?

https://redd.it/poo8p0
@r_devops

k3d - k3s in Docker

Is it recommended to use k3d fro production?
I have just started exploring around k3d and its really amazing tool to work upon especially for local Kubernetes development but I am not sure we should use it in production or not.

As of now in local development, I haven't faced any such issues that would restrict me to use it in prod, but being a beginner I am not very much comfortable with the production use-cases,

so can anyone help me with this?

https://redd.it/pp8ol4
@r_devops

How do you make work visible according to the first way

According to The First Way, which is about ensuring work flows from left to right i.e. request - > running in production, the work should be made visible. That way you can mange things like amount in "work in progress" etc.

How do you make all the smaller tasks visible? Like changing a config, help someone test something, do a 30 minute explanation to the new guy, etc.
Do you create cards for every little activity?

I can't quite decide on what would be a good solution for tracking all the "messy" stuff that is not nicely described as a story as part of a feature.

https://redd.it/pp9b09
@r_devops

IaC for a small set of bare metal servers

Hello! I have a relatively small set of bare metal and VPS servers that I use mainly for my own personal needs. My main way of managing them at the moment is just SSHing into them and doing administrative tasks that way. Pretty much everything that I will be running is running inside a Docker container and I configure these using a local docker-compose file. I'd really like to just have a single GitHub repository where I can manage a list of hosts:

Install Docker if its missing
Start Docker containers with the proper configuration parameters (ports, volumes, etc.)
You can think like docker-compose but for multiple machines
Administrative tasks

My ideal workflow would be to push to a separate git branch, create a PR, and have the changes be "dry run" against my current infrastructure. Merging to main will actually apply the changes to my servers.

I've been testing an Ansible set up to handle this, but in my experience it can be somewhat slow since for every machine it needs to rerun every task (even if the task determines nothing needs to be changed) every time a change is made to the playbook. I also don't really want to delete tasks because if I decide to add another server it'll need to run those tasks for it anyway.

Are there any alternatives that you would recommend trying? Is there any way to speed up how Ansible runs playbooks?

I appreciate any ideas for this, thanks!

https://redd.it/pp9tkg
@r_devops

Amazon EventBridge to Salesforce using API Destinations

https://towardsaws.com/aws-eventbridge-to-salesforce-using-api-destinations-8b956ec45b79

https://redd.it/pp9gap
@r_devops

Do I need nginx?

Recently I started my first internship during my computer science bachelor. During this internship I need to improve the proces of delivering student projects to teachers within school. For this task I came up with a system which requires students to Dockerize their projects, put them on github where they will be automatically published to a VPS using GitHub actions. These urls won’t be visited by more than 10 people at the same time. After presenting this design to my “boss” he stated that I should use nginx, I tried understanding what it does and why I need it but I can’t really comprehend why I should use it. Is it necessary to implement this and why?

https://redd.it/ppbksz
@r_devops

Do you need a new job?

Mayden are looking for new hands please - https://mayden.co.uk/working-here/

https://redd.it/pp9f0i
@r_devops

Release dashboard for multiple rounds of testing with a pipeline

Any suggestions for what's a good way to visualise all the test results from a build in one place? There would be unit tests, smoke tests, regression tests, e2e tests in a pipeline (different frameworks) and I am trying to find a way to see all the results combined

https://redd.it/ppewzk
@r_devops

What would you use to configure VMs?

I'm starting to get more into infrastructure management, and I'm very familiar with Docker and Kubernetes and using tools like Helm & Kustomize to install things on my cluster and Terraform to provision my cluster (on GKE).


I want to start looking at doing more things on VM instances - how would you go about installing and configuring the VM runtime? For example, making sure certain binaries, packages, or languages are installed on every VM? Is this where things like Ansible, Chef, and Puppet come into play?

https://redd.it/ppeglu
@r_devops

Cloud Platform(s) Management

I have been a cloud platform user & admin for some time now. I am familiar with some major cloud platforms that offer Infrastructure as a service such as AWS & Azure. I have noticed that many of these platforms are really lacking when it comes to their administration, identity & user management. Every time when it comes to onboarding new users or updating user permissions or creating new roles in the systems it gets really complicated, cumbersome & repetitive.

​

Moreover, this becomes more & more complicated when a System Admin such as myself has to deal with multiple different cloud platforms with a small team. There are both similar things to do in each platform yet some platform specific things in regards to IAM, resource monitoring, reporting etc.

​

To the fellow Cloud Admins or System Admins out there do you use any such tools where you can control/manage all the major Cloud Platform like AWS, Azure & GCP centrally & automate some of the steps?

https://redd.it/ppguxo
@r_devops

Learning Path + Lab Building for a Beginner

Hey everyone,

this is my first reddit post, after reading for a couple of years - so please have mercy ;-)

My professional background is 10-15 years of experience in enterprise datacenter infrastructures, in the fields of Engineering/Implementing/Administering/Migration/Monitoring of Windows Server, System Center Suite, Exchange, ADFS, Virtualization, File-Systems, as well as little experience IT-Project Management and team-leadership. Three years ago I changed my position to a Cloud Engineer, working with Microsoft 365, Intune and Azure. Once I was part of a project with a DevOps approach and during that time I became a better understanding of the technology and the way of work, which excited me a lot till this day. Unfortunately due to customer demands, my projects shifted from Azure to 90% Microsoft 365, which does not really excite me anymore. It appears to be that my company doesn´t have a lot of Azure (DevOps) Projects in the pipeline.

So it seems like I won´t get any to little DevOps exposure anytime soon. Already being in a senior position, working only for max 35hrs/week and the need of feeding the family makes it hard to apply for entry positions. Therefore, I made the decision to create a learning path by my own, to refresh some knowledge, close some gaps, educate myself about products and methodology and to make myself more attractive to the market.

I would like to share with you my estimated learning path and would be very interested of your feedback. During a Udemy Sale, I acquired several videos regarding DevOps tools. I would like to build several projects according to the learning path, to build up on the stuff I learned.

I am aware of the most common websites, articles and books, nevertheless If someone has a really nice hint, I would be very thankful.

Learning Path 2021 / 2022

Learning Path 2022

I am aware that I might struggle with some parts and I need more time to learn for a topic - then so be it. I would like to know if this is somehow realistic and a "good path"? The ice tip on the cake would be some ideas for projects, building up according the acquired knowledge.

I am looking forward for your input and much regards.

https://redd.it/ppatqh
@r_devops

Statusgator dashboard

Hi.

Anybody here using Statusgator to monitor service status of websites? I was wondering if there is a way to remove the "Powered by StatusGator" footer.

Thanks.

https://redd.it/ppc3oa
@r_devops