Monorepo CI/CD with GitLab and Helm

Hey everyone,

Took a stab at writing my first tech blog post about how we implemented CI/CD for our monorepo hosted in GitLab.


https://alanmasciangelo.github.io/posts/Seeking-ci-with-GitLab-Helm-and-monorepos
I tried to include some micro and macro concepts and some lessons we learned along the way. To date we have over 40 services and 100 Docker images being maintained in our monorepo. We don't have the resources for a fancy build system like the big guys but I think where we've gotten works pretty well.

Ended up being a whole lot of words but I hope you enjoy. Happy Monday!

https://redd.it/m0f39w
@r_devops

virtual machine auto-deployment/reset/revert

I've been working on some web application. I want to implement some functionality when users in the web app click on deploy the virtual machine should be deployed. When the user clicks on destroy the virtual machine should be destroyed also when user click on reset virtual machine, the virtual machine should be returned to its original state (for this, I planned to use snapshots of the virtual machine) but don't have any idea how to automate this from the web application. Can anyone suggest anything?

For this, I plan to use AWS.

Thanks

https://redd.it/m0cky2
@r_devops

Interview question - Cross Containers Access

A interviewer gave a really strange requirement. I want to know how can we create this and also will it have any real world application? Do u have any personal experience of such requirements?


So two containers are running on a system so while inside container 1 I want to check the contents of Container 2 without coming out of Container 1. how can this be done?


One solution that I have is to setup ssh access in both the containers in that way we can access the containers from one another.


Any other suitable solution?

https://redd.it/m0jmk1
@r_devops

Get the IP address of new MAAS ubuntu

I'm testing things out with MAAS. From a VM (vmware), I was able to perform a PXE boot. I'm now looking at the login screen confused.

According to their documentation, I have to ssh to the machine with the IP address. I have set up keys during the installation.

Thing is, and please excuse my very stupid question, I don't know the IP address.

In the mass webgui, I don't see any machines. Same as the DHCP pool, says it's 100% free.

How can I get the IP address? Am I doing something wrong?

https://redd.it/m0io0f
@r_devops

If you run Docker-in-Docker or even K8s-in-Docker in CI/CD, check this out ...

Running Docker-in-Docker or even K8s-in-Docker is common in CI/CD (e.g., the CI/CD job needs to run containers, but the job runs in a container itself).

However, up to now this has required the use of privileged containers or binding the host Docker socket into the container, both of which are very insecure (the containerized job can easily take over the host). Docker's DinD image and K8s.io KinD are example of approaches using privileged containers.

There is now a solution that allows rootless containers to run not just microservices, but software such as Docker, systemd, and even K8s, inside the container, securely and with full isolation.

It's a new low level container runtime (a new "runc") called Sysbox.

It works below Docker and K8s, so you launch the containers as usual:

docker run --runtime=sysbox-runc -it <any-image>

Folks in DevOps can really benefit from this for CI/CD, container-based dev environments, local K8s clusters, etc.

It's open-source, free to use, and uses cutting-edge OS-level virtualization techniques (e.g. uid shifting, procfs & sysfs emulation, seccomp syscall trapping, etc.) to do what it does.

It's not rootless Docker or Podman: with these you get rootless containers, but you can't run things like Docker or K8s inside.

Check it out, I'll be happy to answer any questions about it (I am one of the lead developers).

https://redd.it/m0nhhh
@r_devops

Vault : Enable https with OpenShift4 and helm3

I'm here in order to have some help about how to enable the https for the vault UI with openshift and helm3 with a self signed certificate.


To do that, I use helm3 and a free OpenShift 4 cluster with a Red Hat CodeReady Containers.

Currently, this is what I have done :

Add hashicorp repo :

```
helm repo add hashicorp https://helm.releases.hashicorp.com
```
Install the latest version of vault :


```
[tim@Host-002 crc-linux-1.22.0-amd64]$ helm install vault hashicorp/vault \
> --namespace vault \
> --set "global.openshift=true" \
> --set "server.dev.enabled=true"
```

Then I run `oc get pods`

```
[tim@Host-002 crc-linux-1.22.0-amd64]$ oc get pods

NAME READY STATUS RESTARTS AGE
vault-0 0/1 ContainerCreating 0 20s
vault-agent-injector-7bfb9cffc5-4tl6s 0/1 ContainerCreating 0 21s
```

I run an interactive shell session with the vault-0 pod :
```
oc rsh vault-0
```

Then I initialize Vault :

```
/ $ vault operator init --tls-skip-verify -key-shares=1 -key-threshold=1
Unseal Key 1: iE1iU5bnEsRPSkx0Jd5LWx2NMy2YH6C8bG9+Zo6/VOs=

Initial Root Token: s.xVb0DvIMQRYam7oS2C0ZsHBC

Vault initialized with 1 key shares and a key threshold of 1. Please securely
distribute the key shares printed above. When the Vault is re-sealed,
restarted, or stopped, you must supply at least 1 of these keys to unseal it
before it can start servicing requests.

Vault does not store the generated master key. Without at least 1 key to
reconstruct the master key, Vault will remain permanently sealed!
It is possible to generate new unseal keys, provided you have a quorum of
existing unseal keys shares. See "vault operator rekey" for more information.
```

Export the token :

```
export VAULT_TOKEN=s.xVb0DvIMQRYam7oS2C0ZsHBC
```

Unseal Vault :

```
/ $ vault operator unseal --tls-skip-verify iE1iU5bnEsRPSkx0Jd5LWx2NMy2YH6C8bG9+Zo6/VOs=

Key Value
--- -----

Seal Type shamir
Initialized true
Sealed false
Total Shares 1
Threshold 1
Version 1.6.2
Storage Type file
Cluster Name vault-cluster-21448fb0
Cluster ID e4d4649f-2187-4682-fbcb-4fc175d20a6b
HA Enabled false
```

I check the pods :

```
[tim@Host-002 crc-linux-1.22.0-amd64]$ oc get pods
NAME READY STATUS RESTARTS AGE
vault-0 1/1 Running 0 35m
vault-agent-injector-7f5bc979b6-p5bw6 1/1 Running 0 35m
```

 
I'm able to get the UI without **https** :

In the OpenShift console, I switch to the **Administrator** mode and this is what I've done :
- Networking part
- Routes > Create routes
- Name : vault-route
- Hostname : 192.168.130.11
- Path :
- Service : vault
- Target Port : 8200 -> 8200 (TCP)

Now, if I check the URL : https://192.168.130.11/ui :

https://nsa40.casimages.com/img/2021/03/02/210302100735266662.png

The UI is available.

 

What I've done for the https access :

I've created the directory /vault/certs in my /home and :
```
openssl genrsa 2048 > ca-key.pem
openssl req -new -x509 -nodes -days 365000 -key ca-key.pem -out ca.pem
openssl req -newkey rsa:2048 -days 365000 -nodes -keyout server-key.pem -out server-req.pem
openssl rsa -in server-key.pem -out server-key.pem
openssl x509 -req -in server-req.pem -days 365000 -CA ca.pem -CAkey ca-key.pem -set_serial 01 -out server-cert.pem
```

For the informations requested I used :
```
Country Name (2 letter code) [AU]:XX
State or Province Name (full name) [Some-State]:XXX
Locality Name (eg, city) []:XXX
Organization Name (eg, company) [Internet Widgits Pty Ltd]:XXX
Organizational Unit Name (eg, section) []:XXX
Common Name (e.g. server FQDN or YOUR name) []:192.168.130.11
```
And :
```
[tim@localhost certs]$ openssl verify -CAfile ca.pem server-cert.pem
server-cert.pem: OK
```
 
To configure https :
```
[tim@Host-002 crc-linux-1.22.0-amd64]$ oc create secret tls vault-cert

--cert=/home/vault/certs/server-cert.pem --key=/home/vault/certs/server-key.pem -n vault
secret/vault-cert created
```
```
[tim@Host-002 crc-linux-1.22.0-amd64]$ oc create secret generic pki-int-cert --form-file=ca.pem=/home/vault/certs/ca.pem -n vault
secret/pki-int-cert created
```
```
[tim@Host-002 crc-linux-1.22.0-amd64]$ oc edit statefulset.apps/vault
```
And I've updated the volumeMounts section like that :
```
volumeMounts:
- mountPath: /vault/data
name: data
- mountPath: /vault/config
name: config
- mountPath: /home/vault
name: home
- mountPath: /vault/certs
name: certs
readOnly: true

```
And the volumes section like that :
```
volumes:
- configMap:
defaultMode: 420
name: vault-config
name: config
- emptyDir: {}
name: home
- name: certs
projected:
defaultMode: 420
sources:
- secret:
name: pki-int-cert
- secret:
name: vault-cert
```

I kill the vault-0 pod to take into account the changes and I check if my pod has access to my different secrets :
```
[tim@localhost certs]$ oc rsh vault-0
/ $ ls
bin etc lib mnt proc run srv tmp var
dev home media opt root sbin sys usr vault
/ $ cd vault/
/vault $ ls
certs config data file logs
/vault $ cd certs/
/vault/certs $ ls
ca.pem tls.crt tls.key
```
Then I've edited the vault-config file like that :
```
[tim@Host-002 crc-linux-1.22.0-amd64]$ oc edit cm vault-config
```
```
apiVersion: v1
data:
extraconfig-from-values.hcl: |-
disable_mlock = true
ui = true

listener "tcp" {
tls_cert_file = "/vault/certs/tls.crt"
tls_key_file = "/vault/certs/tls.key"
tls_client_ca_file = "/vaut/certs/ca.pem"
address = "[::]:8200"
cluster_address = "[::]:8201"
}
storage "file" {
path = "/vault/data"
}
```

And I rekill my pod.

After that, if I try to use the first route created, I've this error :

https://nsa40.casimages.com/img/2021/03/02/21030210511540708.png

So I've deleted the first route and I recreate it with https :
- Networking part > Routes > Create routes
- Name : vault-route
- Hostname : 192.168.130.11
- Path :
- Service : vault
- Target Port : 8200 -> 8200 (TCP)
- Secure route enabled
- TLS Termination : Passthrough

https://zupimages.net/up/21/09/c1le.png

And if I try the url https://192.168.130.11/ui :
https://zupimages.net/up/21/09/tkad.png


I've this error... I think I missed something but I don't know what...

Someone to help me ?

Thanks a lot !

https://redd.it/m0pqkx
@r_devops

Been in “DevOps” role for 2ish years...never done it.

Hello all,

Looking for some guidance here on next steps in my career. I was offered an opportunity to transition into a Cloud Infra / DevOps role at my current company about 2 years ago. Previously, I’d been working mostly on windows endpoints and servers for the last 6-7 years. I’ve always had a passion for automation and consider myself fairly advanced at scripting in PowerShell.

My interest in DevOps really came about when I taught myself Git and started using it for my scripts. When given the opportunity to make the switch, I did so with the expectation that the team/environment would help me build on this and get exposure to all aspects of IaC, ci/cd, etc. as well as nurture my interest in coding as it pertains to infrastructure. That has not been the case at all.

I should note this is an internal IT department. We’re not shipping any code or doing CI of any kind. For the most part, we have no idea what runs on the infrastructure we manage. I’ve made attempts to bridge the gaps between our team and dev teams - trying to understand how we can make their lives easier. But there is no alignment at all. They plan, strategize, deploy, and mostly just bother me when they need a new box or something breaks. My team, mostly made up of traditional infra sys admins, has essentially no interest either. I am the “DevOps guy”, they do other stuff. It wouldn’t be a problem had I actually got some exposure or experience in how this is all supposed to work previously, besides my own reading and studying. This has been a disappointing experience.

That said, I wouldn’t say the last two years have been a waste. I’ve learned a ton about Azure as well as finally got exposure to managing Linux infrastructure, mostly Java app servers, some ha clusters, and SFTP. But, I know I’m not going to get the exposure/experience I need to truly be successful by staying in the current environment. For the most part, our version of DevOps is pipelining our image builds with and putting config management/salt scripts into Git. That’s it. The job would be done at that point.

Being someone that loves coding, I really want to understand how web app architectures work, how to scale a production environment, ship code, implement meaningful observability, the works.

My question: given the circumstances, I’m planning to take 3-6’months off to frankly do everything I just said on my own and build a portfolio before applying to a “real” DevOps-minded shop. Is this wise? How can I explain that although I had the title, I didn’t get the exposure? Any suggestions for how to make the most of the time I will have to study/transition?

Thanks a ton

https://redd.it/m0tg7y
@r_devops

Comtrya: Rust Application for Local Configuration Management / Dotfiles

Hi,

I'm working on a new tool to help simplify dotfiles and packages when bootstrapping a new machine, with the plan to support more actions to provide single machine configuration management.

It's early days, but I wanted to share a quick demo and the repository and get some initial feedback.

​

Sharing both mirrors, as I'm happy to receive issues and PR/MRs on either.

​

https://gitlab.com/rawkode/comtrya

https://github.com/rawkode/comtrya

​

DEMO VIDEO>> https://i.rawko.de/kpu7rZ85

​

I hope some of you find this useful and I'm excited to bring more features over the coming days and weeks

https://redd.it/m0ujmn
@r_devops

Do you suffer from downtime when deploying and how do you manage to solve it?

Downtime that can be caused due to lack of coordination between new and older deployed services, bugs that are discovered only after deploy or any other incidents that may harm user experience and system overall resilience, what is your strategy and how effective is it?
(Do you still suffer from downtown although taking those measures?)
Thanks.

https://redd.it/m0sbl7
@r_devops

SSH into Ubuntu MAAS

I'm testing out MAAS. I've PXE booted a VM in vCenter to install ubuntu 18.04. The machine booted up and got an IP address.

Problem is, I can't seem to SSH into it. I've made sure to import the SSH key by doing:

john john-lnx ~ $ cat ~/.ssh/id_rsa.pub
# Copy the output, and paste it in the MAAS webgui for SSH keys. (I've done that in the MAAS installation but again now for troubleshooting)

When I try to SSH into the machine, this is what I get:


john john-lnx ~ $ ssh [email protected] -v
OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n 7 Dec 2017
debug1: Reading configuration data /home/john/.ssh/config
debug1: /home/john/.ssh/config line 2: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug1: Connecting to 172.24.25.232 [172.24.25.232] port 22.
debug1: Connection established.
debug1: identity file /home/john/.ssh/id_rsa type 0
debug1: key_load_public: No such file or directory
debug1: identity file /home/john/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/john/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/john/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/john/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/john/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/john/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/john/.ssh/id_ed25519-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
debug1: match: OpenSSH_7.6p1 Ubuntu-4ubuntu0.3 pat OpenSSH* compat 0x04000000
debug1: Authenticating to 172.24.25.232:22 as 'ubuntu'
debug1: SSH2_MSG_KEXINIT sent
Connection reset by 172.24.25.232 port 22

Is this caused by the `key_load_public` it seems to look for? What did I do wrong?

https://redd.it/m13mp4
@r_devops

Ubuntu's MAAS install configuration

In usual a usual PXE installation, a preseed is given to set up things like date, timezone, language, root user, disks, etc..

I'm testing things out with MAAS, and after it installed a node, the node had already a custom configuration, sort of skipping those configurations.

In MAAS, how are those things defined? Where can I specify things like timezone, language, root user, etc?

Thanks ahead!

https://redd.it/m19f9u
@r_devops

Any good DevOps / Engineering podcasts?

Hey everyone!

Can someone recommend me some good DevOps / Engineering pod casts, that talks about various things like, different deployment strategies, different ways to do cicd, microservices etc.

https://redd.it/m19y4r
@r_devops

Attempt to visualize Nomad and Consul topology with old school tool (TcpDump)

Hi folks,

If you are working with Nomad and Consul you have noticed the lack of visualization tools for network traffic tools and solutions.

I took on this task and started a new solution called LiteArch Trafik

It cross references data between consul and docker metadata on each node and uses very small set of dependencies like: Jq, TcpDump, docker and consul API

There is LAB to launch nomad and consul cluster with Ubuntu Multipass using some bash scripting as well to try it out.

If you are looking for similar solutions give it a try and let me know what you think of it.

​

Documentation

Code

https://redd.it/m12lnp
@r_devops

Pushing new configurations



Our production environment is not connected to any other environment.

When new configuration are needed to implemented (Which happens pretty often)

The configuration files are packaged, one for Linux Nodes, the other for Windows Nodes.

And a release note is given and the engineers go thru the documents which contains instructions on how the new configurations ought to be deployed. Usually its the same but sometimes it changes, the only way we would know it, is to read the document.

Is there any way this can be automated?

i.e. A tool that reads the document and does the job automatically?

Or a way I can get the development team to packages the configuration files in a way where it can be easily deployed?

​

Idk if i asking something that is not out there, but i find what we do really ridiculous, but of course, i.e. asking the development team to package the configuration files in a different way would be a war since we have to deal with humans

But i think i am able to do it, if i push the bosses enough.

​

​

I am open to any ideas

https://redd.it/m10eej
@r_devops

I figured out a CI/CD pipeline with Github Actions and AWS CDK

Disclaimer, I'm not DevOps, just a plain ole' reggy SWE, but I believe this falls under the DevOps domain, right?

The flow is:
1. Github actions build the project; JS/CSS/HTML assets are exported
2. Github actions run cdk and deploy infra as well as upload assets
3. That's about it

Seems simple, but when it worked, it was mind blowing...

Post: JAMStack CI/CD with Lerna, NextJS, CDK, and Github Actions

https://redd.it/m0w444
@r_devops

Nginx sub URL based routing?

URLs
Locatoins
/R1/v1/dev should redirect to 10.10.10.10/v1/dev
so here /v1/dev will be part of the request that should be considered as a part of proxy_pass's URL
/v1/dev is not static value but whatever comes after R1 in location, that will be considered as proxy_pass's end URL.

/R1/v1/test should redirect to **20.20.20.20/v1/test**

is it possible to have this king of configuration of single nginx server ?

https://redd.it/m0zfon
@r_devops

New project: Event-Based Serverless Container Workflows with Direktiv

G'day DevOps!

Apologies if this is the wrong group - we posted this is r/serverless and asked for advice on other groups - someone dm'ed and suggested r/devops. Apologies if this is the wrong group! We wanted to share with you the latest creation from our team!

Direktiv is an open-source event-driven serverless container workflow engine.

Event-driven because we support the CloudEvents standard (also scheduled execution & API driven). Serverless because workflows and execution are instantiated when needed using containers or vorteil. Workflow engine because that's at its core what Direktiv is.

Direktiv was created to address 4 problems faced with workflow engines we faced:

1. Cloud agnostic: we wanted Direktiv to run on any platform, support any code and NOT be dependent on the cloud provider's services
2. Simplicity: the configuration of the workflow components should be simple more than anything else (only YAML and jq to express all states, transitions, evaluations and actions). We've modelled Direktiv's specification after the CNCF Serverless Workflow Specification with the ultimate goal to make it feature-complete and easy to implement
3. Reusable: should have the ability to reuse/standardise containerised code across workflows
4. Multi-tenanted/secure: we want to use Direktiv in a multi-tenant service provider space, which means all workflow executions have to be isolated; data access secured and isolated, and all workflows and actions are truly ephemeral.

The workflow language is VERY simple YAML primitives expressions. We're pretty confident in the engine now, so we're now focused on building standard containers to be used. You can see the progress (for now) on Docker Hub (https://hub.docker.com/search?q=vorteil&type=image)

Direktiv Github: https://github.com/vorteil/direktiv as open source

Documentation: https://docs.direktiv.io/

Beta front-end: https://wf.direktiv.io/ \- we hope to make this a commercial component of the product.

Please let us know what you think about the idea, the implementation, use-cases for it (we have a couple in mind) or some real-world examples (this is what we need help with).

I promised James (of the team members who talks a lot) that I would end the HN introduction with the lines below:

\# The Prime Direktiv:

Captain's log, stardate 47634.44. Cloud bills are high, we're dependent on dinosaur companies and we still have no standards. Forget about boldly changing anything, we just want to change SOMETHING

https://redd.it/m1jdf1
@r_devops

Git vulnerability update your versions

git vulnerability with code execution issue https://github.com/git/git/security/advisories/GHSA-8prw-h3cq-mghm update now

https://redd.it/m1j6v9
@r_devops

Best place to find AWS SRE contractors?

I'm looking to contract with a SRE that has experience setting up EKS clusters on AWS with full CI/CD (from Github), wildcard SSL, etc. I'd like to setup the ability to spin up ephemeral test environments based on PR creation as well. Would someone here be available for a contract? Or is there a better place to look?

https://redd.it/m1ihon
@r_devops

How common is it for a company to use SAAS products and the security to just object to every single external connection that SAAS provider requests?

It's a rant. But my company is trying to have a digital transformation. They have paid for every single tool in the world. But when it comes to working with SAAS products the security will simply put roadblocks for everything that the provider asks. For eg. A monitoring SAAS product we use is requesting access to our AWS account to pull metrics. However Security needs to review that request essentially delaying the work for unforeseeable future. How common is it in other companies? The previous companies I have worked in never had these issues and now I am pissed of due to these hurdles every single day.

https://redd.it/m1idgk
@r_devops