How would go about creating an isolated environment of ~60 .net (4.x) micro services?

So.. I'm a the only DevOps guy in the company, inherited 3 environments from my predecessor: test,qa and ofcourse prod.
Each environment is comprised out of multiple servers hosting a magnitude of ~60 types of different micro services that make up the entire system (a trading platform) .
Few of these services have multiple instances, some are IIS hosted services, some are windows hosted services.
Most can bind to the same IP address with different DNS bindings, but some require binding on entire scopes, so they have their own IP addresses.

Currently we use a rather complex but very customizable and generic build and deploy system, that was built in house on top of a commercial build runner, very similar concept to Jenkins I believe, but windows based and running from an IDE (with a Web-UI for developers to run and deploy services on these 3 environments).

I was tasked with finding a way to "pack" the entire trading platform - with all ~60 micro services and the relevant infrastructure (rmq, ) into something that can be deployed on demand per developer, so we could spin up entire environments in matter of minutes
and have a separate "dev" environment for each developer, created and destroyed on demand.

What could be a good way of achieving this?
Right now I am perusing the idea of a single windows machine, hosting all these services on it, that can be cloned on re-configured with a custom dns name via scripting.
Still not 100% sure this is the best way to go around this, and it does have some challenges
(For example: many if not most of the services should be accessible from outside the self-contained environment, so that the developers can access their APIs and test them with external tools, there are many IP addresses internally, trying to figure out a way to only expose only a single IP address on the host network and keep the rest internal to the "all in one" server.

I have 0 knowledge and experience with containers, not entirely sure if that's the best idea here.
I would love to hear your throughs on this, and how would you achieve this goal and be able to spin complete environments up/down on demand.

Thanks!!

https://redd.it/k5svoc
@r_devops

Scanning artifacts for tokens, passwords and other secrets

Hello all

In our company we want to scan artifacts that are uploaded to Nexus repository for passwords, tokens and secrets. We want to avoid situations that someone upload artifact with secret by mistake or make sure that this person made in on purpose.
I'm looking for tools to check packages for such things. It doesn't need to one swiss army knife to do it all, it can be set of tools to scan different standards like npm, mvn etc.

I'm aware that it may contains many false positives and false negatives but it may improve quality of our products

https://redd.it/k56uy1
@r_devops

Elasticsearch on Docker - file permission issues

I am trying to test LDAP in elasticsearch/kibana by running it in Docker on Windows 10.

I pulled the image down:

docker pull docker.elastic.co/elasticsearch/elasticsearch:7.10.0

And ran it:

docker run -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch:7.10.0

I did the same with Kibana and everything came up as expected. I then upgraded the elastic licence and started the 30 day trial to get the LDAP features in X-Pack.

I opened a shell to the container and updated the elasticsearch.yml file with my LDAP information and set the path to the role\_mapping.yml file. I also updated role\_mapping.yml file with the LDAP informaiton. I followed the instructions here: [https://www.elastic.co/guide/en/elasticsearch/reference/7.10/ldap-realm.html](https://www.elastic.co/guide/en/elasticsearch/reference/7.10/ldap-realm.html)

Now when I restart the elasticsearch container I get the following error then the container halts:

uncaught exception in thread [main]
java.lang.IllegalStateException: security initialization failed Likely root cause: java.security.AccessControlException: access denied ("java.io.FilePermission" "/role_mapping.yml" "read")
at java.base/java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)
at java.base/java.security.AccessController.checkPermission(AccessController.java:1036)
at java.base/java.lang.SecurityManager.checkPermission(SecurityManager.java:408) at java.base/java.lang.SecurityManager.checkRead(SecurityManager.java:747)
at java.base/sun.nio.fs.UnixPath.checkRead(UnixPath.java:810)
at java.base/sun.nio.fs.UnixFileSystemProvider.exists(UnixFileSystemProvider.java:524)
at java.base/java.nio.file.Files.exists(Files.java:2514) at org.elasticsearch.xpack.core.XPackPlugin.resolveConfigFile(XPackPlugin.java:361) at org.elasticsearch.xpack.security.authc.support.DnRoleMapper.resolveFile(DnRoleMapper.java:86)
at org.elasticsearch.xpack.security.authc.support.RoleMappingFileBootstrapCheck.create(RoleMappingFileBootstrapCheck.java:47)

I overwrote the elasticsearch.yml file by copying a version with the LDAP config commented out so I could start the container. When I did this I checked the file permissions and both elasticsearch.yml and role\_mapping.yml have the same:

-rw-rw---- 1 elasticsearch root 199 Dec 2 15:51 elasticsearch.keystore
-rwxr-xr-x 1 root root 568 Dec 2 18:36 elasticsearch.yml
-rw-rw---- 1 elasticsearch root 2301 Nov 9 21:28 jvm.options
drwxrwxr-x 2 elasticsearch root 4096 Nov 9 21:32 jvm.options.d
-rw-rw---- 1 elasticsearch root 8451 Nov 9 21:35 log4j2.properties
-rwxr-xr-x 1 elasticsearch root 502 Dec 2 20:15 role_mapping.yml
-rw-rw---- 1 elasticsearch root 197 Dec 2 20:10 roles.yml
-rw-rw---- 1 elasticsearch root 0 Nov 9 21:32 users
-rw-rw---- 1 elasticsearch root 0 Nov 9 21:32 users_roles

I even tried CHMODing the role\_mapping.yml file to 777 but I still get the same result.

I feel like I'm missing something obvious...

​

edit: formatting

https://redd.it/k5y3y2
@r_devops

Flagr – a feature flagging, A/B testing, and dynamic configuration microservice

I thought the r/devops subreddit might be interested in this project I just found!

https://github.com/checkr/flagr

If you like this, [I do a weekly roundup of open source projects that includes an interview with one of the devs you can subscribe to.](https://console.substack.com/)

https://redd.it/k614vs
@r_devops

Major Pagerduty outage

Status page - https://status.pagerduty.com/incidents/tpvmgn9nnnfb

Event Investigation Issue Affecting Global Routing Keys

https://redd.it/k665k7
@r_devops

Keycloak alternative!

I'm looking for an open source Auth server, Keycloak seems pretty good but doesn't support MongoDB database, I also found Gluu but its system requirement is a little bit high, do you guys know any good alternative that support MongoDB?

https://redd.it/k6747i
@r_devops

GitOps CI/CD tools for non-Kubernetes workloads, like Spark Structured Streaming?

Hello,

So, I'd like to have some declarative way to manage Spark Structured Streaming deployments: rollbacks, healthchecks, logs, etc. I've read about Airflow, but it seems very finely tuned to run batch applications only.

Most of deployment tools like ArgoCD, FluxCD, etc do provide a very nice workflow, but are extremely specific to Kubernetes... Do you guys know of any alternative? How are streaming jobs usually managed?

https://redd.it/k69rnp
@r_devops

gitlab-ci.yml pipeline merge branch

I'm not finding any documentation on how to do this? But maybe I'm not wording my question correctly for google. This is what I'm trying to do....

Repo branches

* dev
* master

When I run my pipeline on dev I would like to do this...

* validate
* This will do the needed checks on stuff
* If this passes, merge the dev branch with the master branch
* build
* Do build stuff in master branch

Is this something that is possible?? If so are there any examples someone could point me to?

https://redd.it/k670pk
@r_devops

Is there a good AWS tutorial that covers all the aspects and development and devops?

I started to look at Stephane Maarek AWS DevOps course. After I bought it, it suggested that I needed to have some AWS experience.

I could do both his courses. If I study something in the DevOps course and notice I need more exposure I can refer to Stephane's Developer course.

I was curious if there was a class/course/tutorial that covered building a product in AWS that taught a person developer & devops skills.

https://redd.it/k670i2
@r_devops

Learning contains AWS (basicly DevOps) with CloudAcademy or LinuxAcademy or aclodguru

Hi I wanted to learns DevOps skills and I wonder which subscription should I buy. I want to have courses with a lot of labs included (hands on labs- learning by doing).

Which one do you recommend and why?

Does this certificates matter in job market?

Thanks!

https://redd.it/k65rvr
@r_devops

Authenticated docker pulls with selfed hosted gitlab

I was wondering if anyone can point me in the right direction, we use self hosted GitLab CI/CD. (We manage the runners in EKS) With the recent Docker pull limits implemented we started occasionally hitting limits. We are planning to purchase a Docker team account but I am unsure how to actually make sure the runners are using the team account since they are currently pulling as anonymous.

I was looking into https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/ but not sure if this is the right direction...

FWIW We stand up our EKS cluster with terraform and deploy the Gitlab runners with a helm chart

https://redd.it/k61muy
@r_devops

CICD for WordPress to deploy custom plugin + knockout js, using GitLab

Hi,

Where can I find someone professional to help me set this up? I prefer to hand over the task, instead of hacking it together and experimenting myself.

I have a WordPress site which contain 2 custom things which is version controlled:

\- Some regular files such as a custom WordPress plugin and some other things. In total 5 folders and subfolders.

\- One of the folder contain a custom knockout script which needs to compile. Developers should have as easy workflow as possible, to save time.

We want to figure out a better workflow and automate as much as possible. Looking for suggestions who to hire and approximate number of hours needed?

​

\*\*Background:\*\*

Current setup is not optimal:

>! \- Developers develop fetch latest git version, and develop in local workstation. They upload it to a "dev-server" (different environment than live server) and inside of that server, they run a set of commands (to compile knockout etc), and then they check their work. If OK, then they commit the compiled code to staging branch, which using cicd script will rsync the select folders to staging server (same environment as live server). Check everything again, and repeat with master branch which rsync to live serer. !<

https://redd.it/k5y63v
@r_devops

Beginner Question: Deploying server application with Terraform and Packer

I've been developing most of my applications to run using Docker but want to have a better understanding on some other deployment patterns. I'm looking for a guide or some clarification on how to effectively use Terraform and Packer to deploy applications *without* relying on Docker. Is this possible? What is the pattern.

From my basic understanding (using AWS) is Terraform can help me make the virtual server instance (EC2) and can be defined using an AMI which can be used to create an image with a copied binary that has been compiled for deployment. But is that it? Will the application start on its own or do I need something else like Ansible to start the application?

https://redd.it/k5x7e8
@r_devops

Load distribution Kubernetes

Hello, we currently have the problem that one node has more load than the others. How can we distribute the load (Pods) better. Is there a trick for this?

\[Imgur\]([https://i.imgur.com/r8ABJww.png](https://i.imgur.com/r8ABJww.png))

https://redd.it/k5wyzx
@r_devops

Have anyone got tired of repetitive nature of the devops Job ?

I am working as devops engineer for past 5 years and have already with 6 different organisations . Everywhere it's the same.

Can anyone suggest me some alternate career in IT ? I am thinking about going in IT security as it's something which my past experience should work then I am also inclined towards programming as I want to work with those big organisations .

https://redd.it/k5vu1u
@r_devops

Diagramming tool for infrastructure that creates code

How would you feel about a tool that would let you diagram infrastructure and export as code?

[https://i.imgur.com/kBqlUkK.png](https://i.imgur.com/kBqlUkK.png)

Essentially you can drag and drop pieces, and then when you have it designed as you wish, export as code to a Terraform file. Similar to Cloudcraft, if you're familiar. Is this something you use? Pros and cons you see with this?

https://redd.it/k6iowx
@r_devops

External Code coverage tools for repositories hosted in Azure DevOps

Hi


We're looking at an external service for handling code coverage stuff ( prefer something that could also generate traceability matrixes etc ) for our pipelines. The one in Azure DevOps leaves a lot to be desired. We're looking for a tool like [codecov.io](https://codecov.io), however it doesnt seem to support having the repository itself in azure devops. We use git if that helps.


I was hoping the community had some suggestions for other tools we could have a look at.


Appreciate any help on this topic. Thanks!

https://redd.it/k6liuh
@r_devops

[Hiring] Devops expert with training delivery experience

Should have over 6-8 years of working experience in a large MNC. Should have delivered at least 4-6 training across all devops topics.

Reach out to me with your experience, sample video link, and profile so that we can discuss more details.

DM me with above details.

https://redd.it/k6mu5t
@r_devops

Unleash - an open source feature toggle service

/u/NeckbeardAaron [mentioned Unleash in my last post about Flagr.](https://www.reddit.com/r/devops/comments/k614vs/flagr_a_feature_flagging_ab_testing_and_dynamic/gek58gf/?utm_source=reddit&utm_medium=web2x&context=3) I figured I would re-post it here so others are aware of the project too! (I wasn't)

https://github.com/Unleash/unleash

If you like this, [I do a weekly roundup of open source projects that includes an interview with one of the devs you can subscribe to.](https://console.substack.com/)

https://redd.it/k6nw7w
@r_devops

Anyone else feeling burned out?

Man I was so excited about WFH and all those cool projects but now they feel meh. Project after project, it just feels like another thing to do.

How are you guys coping with this?

Before this situation I’d have been travelling and discovering small parts of my province. However, for humanity’s sake I’ve been stuck at home for the past 10 months (WFH started early for us).

https://redd.it/k6l1fv
@r_devops

Bringing up my weaknesses

I've been a DevOps Engineer for 2 years now, no other background. Just graduated and went straight into a Jr SWE on an Infrastructure team which quickly became a DevOps team. In this time I've gotten to build docker images, GitLab and Jenkins pipelines, write Helm manifests for kubernetes clusters, administer Rancher (K8s orchestrator), and write/build IaC with Terraform, Packer, and Ansible. But what I havent done a lot of is Linux debugging/administration. Being so new to the industry I've always had seniors who would take on the more difficult debugging, whether it be network connectivity errors with out of band servers or kernel issues, etc. So, I think this is something I should work on more. What recommendations do y'all have for this area?

https://redd.it/k6p56s
@r_devops