Weird out of memory errors

I've deployed k8s cluster into 2 node pools (one huge and one small but auto-scalable VMs) on DigitalOcean. The cluster runs an application (which has \~10 deployments) alongside with prometheus operator and AlgoCD. But last a few days our application becomes offline for a few minutes / a few times a day due to high resource usage.

The problem is that kubernetes deploys all jobs/pods on a single node, when node throws out of resource error and fails k8s reschedules all pods on another node then 2nd node throws out of resource issue then 1st node becomes online again and k8s schedules pods onto 1st node and then it throws out of resource error then 2nd node becomes online, ... some kind of weird loop of events happens.

I couldn't figured out why does this happens. I have not enough experience in k8s to fix this issue. So what could I do in this situation?

https://redd.it/kbaf3s
@r_devops

Student has a Question about Docker and K8s

Hi,

I want to learn more about DevOps and I was going to spend my winter break learning about Docker and K8. But apparently, I just found out K8 is dropping support for Docker.

​

Does anyone experienced have any recommendation on what I should learn and the path to it? I want to learn how to set up CI/CD and deploy a PERN stack website.

https://redd.it/kbb2vk
@r_devops

Is Docker still the way to go to for containers?

Hi DevOps friends,

A few days ago I chatted with a Director of a PaaS startup in hyper-growth mode and he told me that I shouldn't use Docker anymore for two reasons:

1. Kubernetes is deprecating Docker as a container runtime, see: https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.20.md#deprecation

2. He mentioned reproducibility issues with the archives (containers) and that Docker is aware of this common issue. Personal notes here: I worked for an ML/DL PaaS and never saw this issue in 3 years of works.

Instead, he suggested using OCI (https://opencontainers.org/), but actually, docker containers are OCI complaints! This is where started my confusion. Did he confuse docker runtime with containers?

From my understanding, even if you have a k8s cluster, you will not have any problem running docker containers given that they are OCI compliant.

I'm feeling a bit confused here. My guess is that we were talking about Docker (which is an entire tech stack!) from different perspectives, for me is still the best way to build and ship containers, but for him was only about the runtime.

Should we still use docker (at least to build containers)? Would love to hear what you think.

https://redd.it/kazzqi
@r_devops

What does it mean to design and build production level CI/CD pipeline from scratch ?

Apologies for such a simple question. But I always see this in job description. I am a Junior who has mix experience of bit of everything from Cloud to Kubernetes heck even some basic scripting but nothing too crazy that really set me out in the market. What are the tools and skills I need to build a production level CI/CD from scratch. ? I don't want to simply deploy "Hello world" app but something that will challenge me to up-skill myself.

https://redd.it/kbdglj
@r_devops

Frustrated of the company's decision

Hi, I am a devops engineer.

I am currently working in a company that is making a lot of profit during the coronavirus pandemic. We even had an increased in sales and retention this year.

However, recently the company decided to dismantle all the teams that are under the incubator group. The teams within this group are responsible for pilot projects or products with new POCs for about 5 years. An example of an output was a new feature that was created by a team that had automated a process that requires 20-30 employees.

I decided to leave the company because I do not see eye to eye with the company's decision. I can't do anything to voice out my opinion. We as a team automate a lot of business processes for some new products which could potentially be applied to the current products that we have.

From my perspective, if a company is making more money, should take the opportunity to take more risks to allow innovation. A company that missed such an opportunity, will lose out in the long run.

Please let me know your thoughts.

https://redd.it/kazs7d
@r_devops

The ethics of Pull Requests, being the “Reviewer”

Hi,

I wrote this blog post yesterday, the 2nd of a 3 part series. Maybe it could be interesting for this channel as well.

Link: https://werner-dijkerman.nl/2020/12/10/the-ethics-of-pull-requests-being-the-reviewer/

Please let me know your thoughts about it.

Kind regards,

Werner

https://redd.it/kb1vff
@r_devops

Is anybody can advice advanced Jenkins courses (shared library, pipelines, groovy)?

Is anybody can advice advanced Jenkins courses (shared library, pipelines, groovy)?

https://redd.it/kbgdv9
@r_devops

How do you implement API Gateway for a Microservice project?

I want to write an API Gateway for handling requests and routing them to proper services, it's kinda like a reverse proxy. But I want to add some functionalities like monitoring and analyzing requests before they reach final services. There are many people recommending tools like Kong, AWS API Gateway, Nginx while other people prefer writing one by themselves. So what are the pros and cons for using tools and writing your own?

Here are my concerns for this project:

* Scalability (easy to upgrade, add new features)
* Work well with Terraform and Heroku.
* Low cost, as much as possible (in terms of pricing and efforts)

Thank you!

https://redd.it/kbgczf
@r_devops

How do you do environment seggregation?

Currently we are running a separate Kubernetes cluster in a separate vpc for each of our envs, three dev envs, staging, sandbox and prod. There has been talk about collapsing into two vpc's and two Kubernetes cluster, just prod and non-prod. Then segregating environments based on kubernetes namespaces.

How do you divide environments?

https://redd.it/kaurvo
@r_devops

Change backend storage for hasicorp external vault

Hello,

I am using this terraform module to provision vault. Now, I want to change the backend storage. This seems to be not possible using the terraform that I am currently using ie https://github.com/terraform-google-modules/terraform-google-vault. If anyone can confirm this that will be helpful. According to the documentation we can use several backend storage storages. By default, it is currently using GCS as storage. I want to use MySQL.

https://redd.it/kaxup1
@r_devops

learn nodejs to write AWS Lambda functions

Hi,

I have a desire to learn nodejs so that I can write AWS Lambda functions. I use Python today but would like to learn node as I see plenty of AWS samples and examples in node. Looking for some pointers to simple tutorials where I can start.

Thanks

https://redd.it/kawjpf
@r_devops

how to uninstall argocd

how to uninstall argocd?

https://redd.it/kayuih
@r_devops

Do you think Kubernetes will keep on growing in DevOps land from 2021 on-wards?

Do you think Kubernetes will keep on growing in DevOps land from 2021 on-wards?

https://redd.it/kaqqxx
@r_devops

Do we have a consensus on where pipelines should stored?

Do you consider ci-cd pipeline as a part of the codebase?
Well, for public projects, especially open-source, I think it doesn't make much sense, yet Microsoft keeps Vscode build pipeline in the app repository. I am interested in the internal projects that you carry out in your company. Especially if you are using terraforming for immutable deployments. Do you keep stuff like terraform, packer, kubernetes manifest, dockerfiles and pipelines separately from your application repository?
Is there a consensus on best practice?

https://redd.it/kbnrd4
@r_devops

Thoughts on what belongs in a pipeline and what belongs in the build tool

Hello,

the question: how much should be done in locally runnable build tools vs build servers?

background:

I work on a project with an architecture similar to as follows:

- individual microservices are developed by semi-autonomous teams are part of a bigger project
- this bigger project has a provisioner which is used to install the project onto a cluster of VMs. It installs a variety of technologies such as docker, kubernetes, etc, to these VMs.
- this provisioner is released as a tarball with all necessary build artifacts, which all have fixed versions themselves. this is done because some of the install targets are not available through the public internet
- configurations which are included in the provisioner are versioned with the provisioner

this should give a rough idea of the approach, which I am sure is very common; individual sub-components product build artifacts which are orchestrated via a provisioner

the CI/CD challenge becomes how to automate things such that developers and operations engineers don't need to manually collect and version the provisioner each release. originally, this was how it was done, and it was a very slow, error prone and stressful process which resulted in lots of over time and sleepless nights.

I threw together some jenkins pipelines to automate most of it and it's been working well enough, but I'm not all that happy with it myself.

the issue is a lot of the build process exists in pipelines which are run in a jenkins server. these pipelines are under version control as well but they do an awful lot. building our compiled artifacts, dockerizing them, performing sonarqube and twistlock analysis and updating our provisioner all happens within these pipelines.

we use maven for some of our projects, and what I like about maven is it can be run locally by developers. there are plugins which can build docker images, perform sonarscanning and such. It's current quite a specialized task to update our pipelines, and I can't help but wonder had we done more in maven and less in jenkins if we might have been better off.

the issue with maven is it's got first-class support for Java, but kind of leaves our other projects out to dry. Is there a more language agnostic build tool which could be run locally that could be used to define our build pipeline? is this something other people have found is better than the Jenkins, GitHub Actions, TravisCI heavy approach that we're using now?

Interested to see what others think on this.

https://redd.it/kbr5wv
@r_devops

DevOps/SRE questions

I just wanted to share my experience. I went through couple of interviews for DevOps Engineer positions.

**Offer 1**
* Base: 89K
* Bonus: 7%
* Location: East Coast

**Offer 2**
* $61/hr on a 6 month contract
* Location: East Coast

**Offer 3**
* Base: 75K
* Bonus: 10%
* Location: South

**My Skills**
* AWS
* Python
* CICD

I did not take any offers because either they were less than what I make currently or not worth a swich because the increase is minimal. Have salaries taken a hit due to Covid? I have 2 YOE. I am hoping fellow professionals who have interviewed for similar positions can share their experience.

Stay safe everyone

https://redd.it/kbuigv
@r_devops

What factors do you use to determine your 2021 DevOps forecast ?

Example - Delivery Lead Time forecasted for the business in 2021

https://redd.it/kbuit8
@r_devops

Good Devops Resume Examples?

I graduated 2 years ago, went straight into a devops team at a SAAS, work with AWS/Terraform/Jenkins/etc. daily, but I’m being underpaid for my current position and looking to move into a higher salary devops role in Boston or remote.

I know my skills are in demand, but
since I haven’t had to put together a devops-specific resume before, is there anything important I should be adding? Or just make it clear I have experience in a modern stack and list some projects?

https://redd.it/kbzq8a
@r_devops

Resources for Python and Linux

Hello all,

I know Python and Linux skills are needed in the DevOps field but I don’t know which/what type of skills. Does anyone have resources or links to either docs/videos to learn Python and Linux specifically for DevOps? I’m trying to learn the necessities and the musts of Python & Linux just for the DevOps. Thanks in advance!

https://redd.it/kbvmw4
@r_devops

do you prefer deployments referencing specific docker images by some version/git-hash or using something like ":latest"?

I'd like to adopt a best practice with my new deployments. Currently I run with docker images in k8's where the deployment references image tags like prd or qa . Is it a better practice to deploy with specific tag names using git hash or version? When would you prefer one over the other?

https://redd.it/kbsweo
@r_devops

I'm trying to learn how to automate everything from development to production, care to chime in on how I'm doing and what to do next?

Hi r/devops, I hope all of you are safe.

I'm a software engineer that would love to transition to devops some time in the future. I figured that the best way to start is to learn how to implement the "devops" way of an application's lifecycle. As a precursor, I have developed a simple pipeline at work with Bitbucket that:

1. Upon pushing to remote, it runs the automated test suite and reports the result.
2. If something is merged to a branch of interest (like staging), the pipeline will SSH onto the relevant server, run git pull origin <branch>, and then restart Nginx.
3. It then pings a healthcheck endpoint that make sure services such as RabbitMQ or Redis are still functional.
4. A report of the whole process is then emailed to stakeholders

Nothing breathtaking, really. The servers are still provisioned and configured by hand, there's a ton of hardcoded (or not really, they're in Bitbucket's environment dashboard) stuff such as SSH keys that feels icky. But all in all, it gets the job done and I'm proud and happy to work on these kinds of solutions.

Now I have a side project in the works, and I want to use this opportunity to apply better practice with strong emphasis on automation. It is a non-SPA Django app with a Postgres database. I currently have the following things one:

1. Use Docker in development to make sure each dependency is consistent (i.e. I don't even have Postgres and the required Python version to run the app installed in my machine).
2. Use docker-compose to start both the app and the database in development
3. A simple Gitlab CI file that runs the app's test suite, utilizing Docker and docker-compose as well.

I now want to have a publicly available version of this app so that the client can test it. I added the following things:

1. A production docker-compose.yml that spins up nginx in front of the app. Also uses gunicorn to serve the app instead of Django's development server.
1. A Terraform configuration that spins up an EC2 instance and a bunch of other stuff, in which in the end I can access the instance via ssh and http
2. A couple of Ansible playbooks that:
- Install Docker
- Install Docker-Compose
- Copy the application source code to the ec2 instance using the synchronize module
- Rebuild the images and restart the container
- Collect the static assets and run the database migrations
- Create admin accounts if they are not existing yet

With this setup, I can consistently re-create the prod environment with terraform apply + ansible-playbook.

I know this setup is still pretty rudimentary so I have a bunch of questions:

1. After running terraform apply, I run terraform show to get the public ip of the created instance. I then update my /etc/ansible/hosts file. Is there a way to automate this?
2. When running ansible-playbook (or adhoc commands for that matter), I still need the instance's private key in order to connect: ansible-playbook ./initial-setup.yml --private-key=~/Keys/myprivatekey.pem. Is this a normal way of doing things? It just feels weird being tied to this key file, that I need to store it in a non-VCS storage for safekeeping.
3. What's my next step here? Do I integrate Terraform and Ansible into Gitlab CI, and run the commands above according to some trigger?

Thank you for your time to read my queries. All of this is new yet exciting to me, and I can't wait to hear your thoughts. Stay safe!

https://redd.it/kbswda
@r_devops