Multi tenant platform for running docker containers

Guys, i'm looking for open source platform for my users to sign up, request pool of resources and then run containers in each own pool. I have a cluster of virtual machines i want to split between my users (even multiple users on same VM).

I tried kubernetes+dashboard with RBAC/namespace resource allocation. While it perfectly limits cpu/memory usage, but couldn't limit disk space usage per namespace. Even with small persistent volume limits i could open container and create 10 Gb files inside without any restriction. Besides some malefactor can possible mess up cluster for every1 else.

My other idea was to run cloudstack to spin up small VMs for each user, but my CPUs don't have virtualization enabled.

I like how Loft is using virtual k3s clusters, but was hoping to find free alternative.

Lxd and kata containers looks interesting. I hope to find open source service to manage cluster preferably with web-ui. Or any other solution that comes to mind.

https://redd.it/ll7cz0
@r_devops

Need help understanding the concept of stateful serverless.

Hi all,


I'm trying to understand how tools, like cloudstate, from an architecture meaning.
I get it why FAAS are no good for working with data because you need to call the database every time that the function runs and it's gonna take some time.
But I'm struggling to understand how this tool is working that you can have both serverless architecture and stateful functions?


Thanks!

https://redd.it/llb6gg
@r_devops

Red Hat and IBM now have a Linux and Private Cloud Administration Specialization on Coursera.

If you're a Coursera subscriber and you're interested: https://www.coursera.org/specializations/linux-private-cloud-administration-power-systems

https://redd.it/llaop6
@r_devops

Scaling up with serverless computing

The rise of serverless computing is an interesting trend, but many teams still don't quite know why, how and when to use it.

My friend took part in quite a few projects based on the framework and wrote a case study that answers just these questions. In this you will learn why serverless is perfect for the service-based architecture, prototypes or MVPs, but not quite the best one for monoliths. Give it a try and let me know what you think, I can pass questions to the author.

https://redd.it/lla5ji
@r_devops

problems with spinnaker artifacts in gcp

Good morning everyone, someone with spinnaker experience, I am having trouble defining the artifacts to be able to deploy in a gke cluster, I did the installation provided by Google in gcp, and I am trying to get the artifacts (in this case manifest from k8s) from two different places, first from a gcp bucket and second from GitHub In both cases I get the same error, that the object cannot be resolved.

error for github artifact:

Failed on startup: Unmatched expected artifact ExpectedArtifact(matchArtifact=Artifact(type=github/file, customKind=false, name=k8s/deploy.yml, version=null, location=null, reference=null, metadata={id=8bed9f1f-0806-445d-9c14-f661c568922c}, artifactAccount=c-bordon, provenance=null, uuid=null), usePriorArtifact=false, useDefaultArtifact=false, defaultArtifact=Artifact(type=custom/object, customKind=true, name=null, version=null, location=null, reference=null, metadata={id=7dc222d9-1103-4564-95fc-0eebf0f4fcbd}, artifactAccount=custom-artifact, provenance=null, uuid=null), id=73cd74b1-4089-4a65-833c-1cb917414f51, boundArtifact=null) could not be resolved.

error for gcp bucket:

Failed on startup: Unmatched expected artifact ExpectedArtifact(matchArtifact=Artifact(type=gcs/object, customKind=false, name=gs://demo-pipeline-1/deploy.yml, version=null, location=null, reference=null, metadata={id=10025080-76c4-4926-96bb-75ca24f32262}, artifactAccount=gcs-account, provenance=null, uuid=null), usePriorArtifact=false, useDefaultArtifact=false, defaultArtifact=Artifact(type=custom/object, customKind=true, name=null, version=null, location=null, reference=null, metadata={id=7dc222d9-1103-4564-95fc-0eebf0f4fcbd}, artifactAccount=custom-artifact, provenance=null, uuid=null), id=73cd74b1-4089-4a65-833c-1cb917414f51, boundArtifact=null) could not be resolved.

​

I would appreciate any help

https://redd.it/ll6amj
@r_devops

How do you secure credentials when operating CI/CD?

I know a lot of people are happy to just pass your credentials to GitHub, Circle CI etc. This is fine depending on your risk profile but those credentials often have near admin access to your cloud account.

For those of you who are not happy with providing that level of trust to a third party what do you do? Does anyone know how someone like AWS manages their credentials with CI/CD?

https://redd.it/ll0yfd
@r_devops

Sensu

Is anyone using Sensu for monitoring? It has all the DevOps concepts that I like to see, but just don't hear that much about it. Everything is New Relic or Datadog..

https://redd.it/lll6m1
@r_devops

Serverless infrastructure

Hello all,

I’m looking into learning a bit more about the underlying technology under serverless architecture.

How would a provider approach the solution, load balancers, reverse proxies, and finally the application web server?

Thank you 🙏🏽

https://redd.it/lll3wj
@r_devops

Project Manager looking for Feedback

I am currently a waterfall based project manager who is learning more modern agile project management methodology & DevOps frameworks. The feedback I am seeking is – Is there value to you as a developer if the PM or the project leader knew or was familiar with programming?

I am under no delusion that I would be able to replace a developer nor do I have the desire to. I just want to know the best way to be part of the team.

*I am thinking about taking two Code Academy Career Paths (Code Foundations Computer Science) for those who are interested.

https://redd.it/lllgzn
@r_devops

Does AWS ECS add a price overhead if you don't use Fargate?

If you manage your own ECS cluster on EC2, then is there a price overhead from using ECS?

Also:

\- How does the pricing of Fargate compare to the pricing of EC2 these days?

\- How does the pricing of Fargate with spot instances compare to the pricing of EC2 (no spot instances) these days?

Thanks!

https://redd.it/llilj6
@r_devops

CI/CD Pipeline For Library + Backend Server

Hey all, I'm fairly new to DevOps and I am curious what my options are for the following scenario.

​

TLDR; How do you setup a CICD pipeline that builds two different repos, where one is dependent on the other, and it can handle the situation where you need to push out code to both at the same time?

​

To start off, I have a library A and then a backend server B that depends on A. The code for A and B are maintained in separate git repos. I want to setup a CI/CD pipeline so that whenever I push out changes to A and it successfully builds, it will trigger B to automatically rebuild using the new version of A. Likewise, if I push out new changes to B, it will automatically grab the latest version of A and rebuild itself. I think this is a fairly typical situation. I'm using CircleCI and have an idea of how to set this up in that environment.

​

Here's what I'm curious about though. What happens when I need to make changes to both A and B, and then push out changes to both repos back to back? I will end up with some weird synchronization issues if I followed the setup described above. For example, if I open a pull request for A and trigger the pipeline, it will attempt to build the previous version of B since the current version is probably still sitting in its own pull request that has yet to be merged. On top of this, I would really only want to build once, and the above setup I described would cause the server to be built twice assuming everything else worked out.

​

Is there a common way to deal with this issue? Hopefully I have explained it well enough. I appreciate any feedback or thoughts you can give me!

​

For some extra context, I'm working with Java using Gradle as my build tool. Both the library and server are built on Spring.

https://redd.it/llhf02
@r_devops

Java classes communication using Uri

https://stackoverflow.com/q/66233294/15165716



I am trying to get java objects (object1 and object2) to communicate using URIs. I would like these objects to have a Uri that has different endpoints and to execute some code when the URI is reached. The objects could be running on different computers (but on the same network) or locally on the same machine.

An example of this could be: If object2 wants to enter object1 it could access something like "object1@localhost:80/service/enter?name=object2" . and object2 would get a reply saying {"Accept"} or {"Error:condition not met"}.

I know it is a lot easier to use socket programming but because of the scope of the project I'm working on that is not possible. Any help would be greatly appreciated.

https://redd.it/llhdon
@r_devops

I’m new to DevOps and am very confused

I’m trying to build a small project to better understand the tech for DevOps jobs and I’m hitting major bumps.

The idea: simple flask API in a docker container, simple database in a docker container, unit test the database through API endpoints, and push changes to repo if unit tests pass.

I’m struggling on how to push the code if tests pass. I was thinking of using Jenkins to build and test the app, but I really don’t know how to proceed.

Eventually I want to use ansible, kubernetes, and aws, but I’ve been searching the internet for days on how to proceed and I’m stumped.

How does this all connect together?

I seriously am in the dark and don’t know what I don’t know

https://redd.it/llgnb1
@r_devops

Which CI/CD tools you are using ?



View Poll

https://redd.it/llg6y0
@r_devops

DevOps engineer at work sugested I start with Docker, Kubernetes, and Elm. Anything else you guys would add?

I feel a bit rudderless at the moment. My working plan is to take one of my repos that is sitting on Github and try to deploy it. Then add the commercial features we expect at work: multiple instances, load balancing, etc.

Are the three techs plus this personal project all I need or would you guys like to add anything?

https://redd.it/llefyi
@r_devops

Best way to benchmark and load test an api.

Hi guys

I want to know how you guys benchmark and load test an api endpoint. Is it done depending on the language we use or are there few things which we need to know before load testing an endpoint like whats the architecture the application is hosted etc.

Thanks in advance.

https://redd.it/llv5s7
@r_devops

CI/CD Process for internal Python package

Hi everyone,

​

I am not very well versed in DevOps practices - I am a data scientist and I have good software engineering skills, but CI/CD was always something that "someone else does".

​

Recently, I've created a python package for my team to use. We're just hosting it on Github and we're expecting people to download via `pip install <github link>` like you would install any Python package from a github repo rather than PyPI.

&#x200B;

My question - what kind of CI/CD pipeline can I/should I set up for this? What's important to have - or even, what questions do I need to ask to *know* what's important to have?

&#x200B;

Thanks!!

https://redd.it/lle4us
@r_devops

Managing Microservices using Kubernetes and Docker

OSS colleagues, this Modern Container-Based DevOps program begins by guiding the user through the concept of microservices, explaining fundamentals and other components in IT that play a vital role in obtaining a microservices architecture. It then addresses how to use Git, and work with and manage containers using Docker as well as Podman on RHEL 8. The course then covers how to perform daily container management tasks, and works its way through managing container images, storage, and networking. Module 1, "Microservices Essentials Overview," introduces the microservices essentials, including what they are, why Git is so important, and how containers fit into the picture. The last lesson explains everything that's going on in containers. Module 2, "Managing Containers," explains how to work with containers, including Docker containers and Podman. Module 3, "Implementing Full Microservices with Container Orchestration Platforms," explores container orchestration platforms, which provide the perfect way of managing microservices in an enterprise environment. In this lesson Kubernetes, the most significant container orchestration platform, is also introduced.

Enroll today (individuals & teams): https://tinyurl.com/1pj3ph8z

Much career success, Lawrence E. Wilson - Online Learning Central (https://tinyurl.com/bto061zr)

https://redd.it/llbopy
@r_devops

Using Syslog for Application Logs?

&#x200B;

I am researching a log forwarding solution to aggregate all of the OS and application/services logs across all of our various systems to a single data store. Syslog/Rsyslog works great for OS logs in our system currently, but I am unsure how suitable it will be for application originated logs where the log message may be a large json string containing a serialized stack trace, etc....

I know syslog has added support for json messages but my understanding is it is basically just placing the json string in the message portion of the syslog formatted message, not json from the ground up. Also it seems like some of the syslog implementations have hard limits on message sizes and may potentially split a large message into multiple when processing. My other concerns are with configuring a large number of nodes and configuration updates, I have read configuring the syslog agents is a real pain point.

Can anyone with Syslog experience comment on using Syslog for application messages? Do you recommend it or have any success using it on your systems? Any advice would be appreciated.

Thanks

https://redd.it/llbhax
@r_devops

GCP loadbalancer monitoring, aggregated per route

Hey guys,

Would you know of a tool, SAAS preferably, which would read logs from a GCP loadbalancer and would produce stats (latency/volume/error) aggregated per routes?

By route aggregation I mean:

GET /api/users/52356

GET /api/users/1234

I'd love the tool to be able to detect those two routes are actually the same: /api/users/{\\d+}

I cannot find anything like that, so I made something like LB logs -> big query -> custom view with route parsing -> google data studio to visualize.

If it doesn't exist, should we build that? :)

https://redd.it/llyw0f
@r_devops

So, here's a question...

Is every DevOps engineer in Romania taken?

Honestly, I know the War for Talent is real, but it seems as though for each member of the DevOps community, there are at least half a dozen job offers lying around.

Anyway, there are a few projects (quite a lot, actually) we are working on and we need like lots of great DevOps engineers and maybe you could help me with a few pointers on what is truly attractive to you, when considering job opportunities. Any sort of information is priceless, right now, and greatly appreciated!

https://redd.it/llxbae
@r_devops