Would you guys be interested in a web app to convert docker-compose files into kubernetes files ?

Hey guys, everything is in the title. Just an idea for a side project I had and I was wondering if anyone else besides me would find this interesting.

Thanks.

https://redd.it/eb0y8v
@r_devops

Vault Token Creation Query

Hello.

I've taken responsibility for vault at my company, trying to tidy it up and bring it in line in terms of security etc. In the old world, the root token was everywhere and everyone knew it. We may as well have kept all our passwords in an excel spreadsheet!

For users who required temporary vault access, I'd create them a token with the 32 days TTL. Then keep creating new tokens for them when that expires.

However, this is incredibly manual and involves someone (me), logging into the vault as root to create a new token for them.

I'm seeking some inspiration, I have some ideas but curious about what others are doing.

Is there a way to automate this, as I want our engineers to not have access to the root token, and just have what's needed to do their job. But also be able to create their own tokens when needed.

But quite simply, the mission here is for me to not have to ever login as root again to renew all these tokens. Or, at the very least do I increase the max TTL in config to say 1 year?

Cheers

https://redd.it/eb15hz
@r_devops

Pros and cons of using azure pipelines over travis?

I am thinking of shifting my C++ project from Travis to Azure Pipelines because of the unlimited build times in Azure Pipelines. I am trying to make a comparison chart to figure out if the shift is worth the effort and also if there is any other benefit of moving to azure pipelines. I would love to hear your thoughts on this.

https://redd.it/eb0khp
@r_devops

Molecule systemd doesn't work

I'm having trouble getting systemd to work when I do molecule test. I can get it to work when I start the docker container manually on the command line. Is there some special configuration necessary for molecule that I am missing? I've followed the documentation and I'm using centos/systemd which as I've said works outside of molecule. With molecule I get a failed dbus error or it simply fails to even create the container and says that it doesn't exist. Also, is there configuration needed to use oci-systemd-hook or is it used automatically?

https://redd.it/eb1plk
@r_devops

What is the best approach to serve high bandwidth through NAT Gateway

I want design AWS architecture like this, but not sure how to handle high bandwidth (>100GB) traffic.

A kubernetes cluster with lots of microservices , both frontend and backend. An LB in front of the worker nodes. K8s replica can scale high bandwidth traffic.

My question is where should I create the Kubernetes cluster? I know there is no bandwidth constraints in Public subnet, but AWS NAT Gateway has bandwidth constraints. What is the approach by big companies to serve high bandwidth through NAT Gateway. Or should I put my K8s cluster in public subnet itself.?

Any help is appreciated .Thanks

https://redd.it/eaa7ea
@r_devops

Fargate and EC2 for ECS: Trying to understand their best use cases.

Hey there,

I am trying to figure out which of the two launch types is best, taking into account that I already have experience with managing an EC2 container cluster:

- There was a significant [price reduction](https://aws.amazon.com/blogs/compute/aws-fargate-price-reduction-up-to-50/) to Fargate pricing earlier this year.
- Amazon [has just released](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/cluster-auto-scaling.html#asg-capacity-providers) improved ECS autoscaling, where they provide cluster scale-in, scale-out and instance draining automation
- Recently they also launched [saving plans](https://aws.amazon.com/blogs/aws/aws-ecs-cluster-auto-scaling-is-now-generally-available/) that offer more flexibility than reserved instances (can also be used by Fargate)
- Fargate spot could also provide some additional savings.
- In Fargate, you only pay for the CPU and Memory that you define. But as you can't predict how it will consume CPU and Memory exactly, there will still be unused capacity, that you would have to pay in the end, similar to having EC2 container instances. Does it still make sense to go for Fargate, comparing the same average reservation rate?
- There's this nice [comparison](https://www.trek10.com/blog/fargate-pricing-vs-ec2/) which suggests that Fargate's pricing for an average reservation of 70% is similarly priced with an EC2 based cluster. Has anyone tried to do a similar calculation? Does it sound right according to your experience? It doesn't seem to take into account, other charges such as Data transfer charges related to cluster computing




Any input greatly appreciated :)

https://redd.it/eb7pnm
@r_devops

How are you guys building your containers for your code?

Hey guys.

I’m migrating an app into a container to allow me to be a little more flexible. I am curious how are you build the container with the code in your pipeline? Do you copy it in from a system path? Or do you have a run command for docker to clone the repo?

Just kind of trying different ways to see what is the best for me and would like to hear your opinions.

https://redd.it/ea53by
@r_devops

[My thoughts] Cloud-Native is not about containers, nor a synonym for Microservices.

Cloud-Native really started to be sort of a thing in about 2010 by a couple of industry thought leaders, and one of them is Paul Fremantle, who wrote about it in his blog. He doesn't talk about the tools and technologies in his blog; instead, he states that for systems to behave well on the cloud, they need to be written for the cloud & this is where the cloud-native approach began.

Containers & microservices give a boost to the cloud-native model, but that's not everything.

The tools and technologies empower today's application deployment; Cloud-native computing takes advantage of many modern methods, including PaaS, multi-cloud, microservices, agile methodology, containers, CI/CD, and DevOps. Now, we could even see its own foundation: the Cloud Native Computing Foundation (CNCF), launched in 2015 by the Linux Foundation.

According to a recent survey by CNCF, it is found that the use of Cloud Native technologies in production has grown over 200%.

https://redd.it/ea0reb
@r_devops

What is the best approach to serve high bandwidth traffic with AWS NAT Gateway

I want design AWS architecture like this, but not sure how to handle high bandwidth (>100GB) traffic.

A kubernetes cluster with lots of microservices , both frontend and backend. An LB in front of the worker nodes. K8s replica can scale high bandwidth traffic.

My question is where should I create the Kubernetes cluster? I know there is no bandwidth constraints in Public subnet, but AWS NAT Gateway has bandwidth constraints. What is the approach by big companies to serve high bandwidth through NAT Gateway. Or should I put my K8s cluster in public subnet itself.?

Any help is appreciated .Thanks

https://redd.it/eaacua
@r_devops

Does AWS come with a load balancer or do I have to build one myself?

I was wondering whether if AWS comes with a load balancer. If not, I was thinking about using NGINX but also I would like your opinions on which do you think is cheaper and easier to set up an maintain?. Opinions are welcomed.

https://redd.it/eaelgv
@r_devops

[blog] Shadow requesting for great good

[https://medium.com/carwow-product-engineering/shadow-requesting-for-great-good-92cde331363a](https://medium.com/carwow-product-engineering/shadow-requesting-for-great-good-92cde331363a)

https://redd.it/ebdaky
@r_devops

How to use multiple Docker containers to set up Jenkins agent in Jenkins pipeline?

The following snippet is an example provided by Cypress, a Javascript testing framework that I'm using. Here is the [link](https://github.com/cypress-io/cypress-example-kitchensink/blob/master/Jenkinsfile) to the Github page.

pipeline {
agent {
// this image provides everything needed to run Cypress
docker {
image 'cypress/base:10'
}
}

stages {
// first stage installs node dependencies and Cypress binary
stage('build') {
steps {
// there a few default environment variables on Jenkins
// on local Jenkins machine (assuming port 8080) see
// https://localhost:8080/pipeline-syntax/globals#env
echo "Running build ${env.BUILD_ID} on ${env.JENKINS_URL}"
sh 'npm ci'
sh 'npm run cy:verify'
}
}

stage('start local server') {
steps {
// start local server in the background
// we will shut it down in "post" command block
sh 'nohup npm run start:ci &'
}
}

// this stage runs end-to-end tests, and each agent uses the workspace
// from the previous stage
stage('cypress parallel tests') {
environment {
// we will be recording test results and video on Cypress dashboard
// to record we need to set an environment variable
// we can load the record key variable from credentials store
// see https://jenkins.io/doc/book/using/using-credentials/
CYPRESS_RECORD_KEY = credentials('cypress-example-kitchensink-record-key')
// because parallel steps share the workspace they might race to delete
// screenshots and videos folders. Tell Cypress not to delete these folders
CYPRESS_trashAssetsBeforeRuns = 'false'
}

// https://jenkins.io/doc/book/pipeline/syntax/#parallel
parallel {
// start several test jobs in parallel, and they all
// will use Cypress Dashboard to load balance any found spec files
stage('tester A') {
steps {
echo "Running build ${env.BUILD_ID}"
sh "npm run e2e:record:parallel"
}
}

// second tester runs the same command
stage('tester B') {
steps {
echo "Running build ${env.BUILD_ID}"
sh "npm run e2e:record:parallel"
}
}
}

}
}

post {
// shutdown the server running in the background
always {
echo 'Stopping local server'
sh 'pkill -f http-server'
}
}
}

My goal is to have a Jenkinsfile that is very similar to the above because I want to have parallel Cypress testing as shown in the above snippet. In the example above, the Jenkins agent is simply the official Cypress Docker image `cypress/base:10`.

agent {
// this image provides everything needed to run Cypress
docker {
image 'cypress/base:10'
}
}

However, for me to run all my tests with my own database, I need to spin up two separate Docker containers. One container contains the front-end portion of my web app and the other container contains the back-end portion of my web app.

Below is the Dockerfile for my front-end container, which is located in `my-app/docker/combined/Dockerfile`.

FROM cypress/included:3.4.1

WORKDIR /usr/src/app

COPY package*.json ./

RUN npm install

COPY . .

EXPOSE 5000

RUN npm install -g history-server nodemon

RUN npm run build-test

EXPOSE 8080

Below is the Dockerfile for my back-end container, which is located in `my-app/docker/db/Dockerfile`. All it is doing is copying some local data into the Docker container and then initialising my MongoDB dat

abase with this data.

FROM mongo:3.6.14-xenial

COPY ./dump/ /tmp/dump/

COPY mongo_restore.sh /docker-entrypoint-initdb.d/

RUN chmod 777 /docker-entrypoint-initdb.d/mongo_restore.sh

Usually, I would use `docker-compose` and the following `docker-compose.yml` file to spin up these two containers. As you can see, the front-end container called "combined" is dependent on the back-end container called "db".

version: '3'
services:
db:
build:
context: .
dockerfile: ./docker/db/Dockerfile
container_name: b-db
restart: unless-stopped
volumes:
- dbdata:/data/db
ports:
- "27017:27017"
networks:
- app-network

combined:
build:
context: .
dockerfile: ./docker/combined/Dockerfile
container_name: b-combined
restart: unless-stopped
env_file: .env
ports:
- "5000:5000"
- "8080:8080"
networks:
- app-network
depends_on:
- db

Below is the docker-compose command I would use.

docker-compose up --build

**I would like my Jenkins agent to be the** `combined` **container; however, I need the** `combined` **container to connect to my** `db` **container, which needs to be spun up. My question is, how do I achieve this in Jenkins pipelines? I've read** [this documentation](https://jenkins.io/doc/book/pipeline/syntax/)**; however, it doesn't mention anything about using multiple Dockerfiles to create a Jenkins agent. Is something like this possible and could someone please show me what my Jenkinsfile should look like in order to achieve my goal?**

https://redd.it/ebeu8d
@r_devops

Question about restricting access to GCP by IP and how that works with API's

Hi All,

I'm a sysadmin / IT Manager so this is a bit out of my depth. Please feel free to lynch me and chuck me out if I've inadvertently got the wrong group.

Basically, in my last job, we had a full tech team including CTO, HoE, etc so I never had to get involved over and above making sure they had a computer and an internet connection. They built an app hosted in AWS with a backend database for adding users that could only be accessed via the main office IP address for obvious security reasons. It was all self contained. Brilliant.

In my new job, they are just starting to build and app for generating reports from social media specific to their requirements. Locally this has all been fine, now they want to push it to GCP, again fine.... But as there is no real tech team here, just a singular front and back ender learning as they go, they are now asking for my advice around security.

My assumption is they should lock it down the same way, however if they do that will it stop the api's from the various social channels talking to the service? Or if the call is made from inside does it pass it through OK...

No idea, sorry guys, but your advice is greatly appreciated in advance

(even typing this makes me feel dumb)

https://redd.it/ebg4xw
@r_devops

Lifecycle management: versioning and vulnerability tracking of your tools, applications, containers and more

Every project has tools, applications, Docker containers and more that are used. All of these need to be regularly updated for feature completeness or security and compliance reasons. Some teams use paid systems to track this information but more than often this is checked manually now and then and in the worst-case versions are checked barely. The reason why this is such a tedious task is that most of the time manual labor is required.

To help with this I have created an open-source project that can track automatically current versions that are being used in your project. But it can also tell you if there are new versions and if your containers are vulnerable. If you are interested please have a look at [https://github.com/arminc/k8s-platform-lcm](https://github.com/arminc/k8s-platform-lcm)

I hope this tool can save you time and give you faster insights in what needs to be updated.

https://redd.it/ebjp6d
@r_devops

Using Node.js to Write Safer Bash Scripts

Hey everybody,

This is an article I've written about how and why we're wrapping our Bash CI/CD scripts with a more modern language. In our case, since the entire company programs in JS, we use Node.JS. You could just as well use Python or Java if so inclined.

https://medium.com/getvim/using-node-js-to-write-safer-bash-scripts-ad6a523a5324

https://redd.it/ebcjyn
@r_devops

Microsegmentation --> 0 trust

Interesting (albeit loaded) read.

I never thought of microsegmentation and 0 trust to go hand in hand, but it seems if leveraged correctly they can. The article says doing so may have some technical challenges, curious as to what they are.

https://redd.it/ebfnxn
@r_devops

Big Data learning curve too big? New Job

Background: I'm a Linux/Unix admin with over 5 years of experience. Been stuying a lot about DevOps metodology and its tools, such as: Docker, Kubernetes, Ansible, etc.

I've been offered a job opportunity that involves managing Cassandra, Spark and Kafka(Big Data I believe).

I don't have much knowledge about these tools(And thats what I told the company). However, they seem to really like my profile and told me that if I promise to take some courses, I'm pretty much in. I'll be studying while on the job.

I really think this could be a good opportunity to learn more about DevOps and getting some experience.

​

Do you think the learning curve is too big? This is not an entry level job. The position is ITops Architect. If no, where should I start stuying, Cassandra, Spark or Kafka. Or the three of them at the same time.

https://redd.it/eblyfk
@r_devops

OSS Scanning for Conda and RStudio/Cran

Anyone working with data scientist out there and have a need to scan R and Cran packages? As well as Conda?

I’ve been looking into Tidelift and Blackduck. Both do 75% of what we are looking for but the lack of Cran package scanning is apparent in all OSS scanning utilities.

Anyone else have any insight?

https://redd.it/eblpaq
@r_devops

Where in process to perform static analysis?

Hey /r/devops, I am in the process of implementing SonarQube in our environment to do some automated quality checking on our projects. Our release process consists of using 3 git branches, one for raw development, one for preparation of release candidates, and then a release-only branch. Would you recommend having the quality scans run for every development build? Or would it be better to scan release candidates and do remediation as part of the release process? I'm as much interested in specific advice as I am in generating discussion in this thread!

https://redd.it/ebley4
@r_devops

Short summaries of main concepts of DevOps

Tomorrow is my first interview on a DevOps position. Maybe exist short summaries and concepts of devops, CI/CD, and so on, to read, and reprat main ideas and main moments? Or frequent questions on interviews?

https://redd.it/ebhxr5
@r_devops