Git Concepts and Theory Resources

Hey All! I'm a bit new to DevOps, and I am looking to beef up my knowledge of Git and its general theory and principles, as well as different branching strategies. Wanted to see if anyone had any resources or online tutorials that they recommend?

Thank you!

https://redd.it/lrjv3k
@r_devops

Docker vs LXC for testing environments

I'm building my own CI setup for web development. I'm at the stage when I'm asking myself a question - should I use Docker or another, nested LXC/LXD hypervisor for hosting testing environments for my branches. I'm familiar with Docker, and I'm quite familiar with LXC. I'm pretty sure I'll continue to use Docker for local development, but for public testing environments LXC is kinda tempting. Maybe because it's conceptually simpler than Docker, and more similar to the production environments.

What are your experiences? Would spawning LXContainers be a better idea than spinning up Docker environments? Both seem to have everything needed for this task, but I feel like LXC will be slightly easier to maintain, I'm not sure if I'm right though. I have admittedly more experience with Docker than LXC.

And I'll answer "Why not cloud?! \~_\~" already: just learning purposes :) .

https://redd.it/lrkkmv
@r_devops

CI/CD tools similar to octopus deploy (IIS)

So one thing I like about Octpus Deploy is that it has the ability to configure IIS and it's a CI/CD tool. Are there other CI/CD tools that can do this natively aswell as if they are free and open source (nice to have but not a deal breaker). Right now we have jenkins and been writing ansible (config management) playbooks to boot strap iis configs (site, app pools, bindings, web apps, virtual directories). I can handle this all in powershell/ansible but this is more for my team of various skill levels with some of them being more of a Jr to mid-level sysadmin who work better with GUIs than straight code like powershell. Some of them execute code from the net without understanding what they are executing to fix an issue so that's what I'm working with. I'm not management, so unfortunately I'm trying to find a tech solution to a partial human issue.

We do not let our developers do this as part of their code base because they are developers at their core with no ops experience and there have been plenty of instances where we had to step in to stop them from shooting themselves in the foot. So that's another reason why specifically a CI/CD tool.

https://redd.it/lrgnux
@r_devops

Terraform EC2 post deploy configuration

Wondering if anyone can share their ideas on getting config files and installing packages on new EC2 instances provisioned using terraform.

options considered:

\- baking packages into AMI & deploying config files to EC2 instance using Terraform

\- using Terraform to run post exec hooks on the EC2 instance after deploy

\- using Ansible to deploy scripts and packages to EC2 instance after deploy

These seem to be the only ways to keep the configuration of the instance located with the IAC package, I'm a little fuzzy on how I would execute these solutions so any advice if you have done it before or think it's a good idea would be useful.

Would like to avoid deploying supporting resources like a chef or puppet server.

https://redd.it/lrn3e4
@r_devops

Would love some clarification about Kubernetes deployment to EC2

Hey all,

I am coming from a Python development background (backend) and I just got a new Backend/DevOps job offer from a company I am really interested in.

The assignment is:

>Create a Jenkins declarative pipeline to deploy a Kubernetes cluster (1 to 3 Nodes). The K8s should be deployed on top of AWS EC2 Instances and it should be used by Terraform and Ansible.

As I have absolutely zero knowledge about the technological stack mentioned above, I wanted to ask if I got it right in my head to make sure I am doing this right.

1. Create 3 Ubuntu instances on Amazon EC2 using the AWS console
2. Create a Jenkinsfile that will be used to deploy the K8s cluster to the EC2 instance

Now, few things I don't get,

1. What is the use of Terraform and Ansible in this assignment? How should I implement it?
2. All the configuration (i.e building the EC2 instances and deploying K8s) should be in the Jenkinsfile? If so, where I can find some examples related to what I need to do (Google was not very helpful, surprisingly)
3. Should I use the EC2 plugins for Jenkins?
4. What do "1 to 3 nodes" mean?

Thanks a lot!

https://redd.it/lrevfa
@r_devops

a better model for CI/CD cycling



Hello DevOp'ers !

I am having some troubles finding a better model for my CI/CD cycle.

This is mainly gitflow issue.

My current model : We have 2 bitbucket branches : master and develop.

Both are linked via webhooks to jenkins to be deployed to prod. and dev. servers.

QA team for testing features and releases on dev. server.

Team of developers for both frontend and for backend. With lots of troubles.

The problem : Team manager wants to be able to have better control over accepted/rejected features. Say : developer1 commited a feature to develop branch, developer2 added a hotfix to develop, we want QA team to easily test them both separately, and then testing both together with the least hussle possible. And afterwards easily select what commit stays and moved to master branch.

As I said, this is mainly gitflow issue.

Any suggestions ?

https://redd.it/lre433
@r_devops

Kubernetes CPU cores

Hi community,

Need some help in understanding this concept. I have a mysql pod running on a 8 cores single node k3s cluster, i have set the request to '200m' and the hard limit is set to '500m'. During benchmarking mysql using sysbench, i have noticed mysql process uses all the 8 cores during the process despite of setting the limit to 500m which is just about 6.5% of the total cpu. The idle cpu during the Benchmarking process is approx 60%. I am trying to understand, why the processes are spread across all cores , is this how it works in Kubernetes ? Can we control, the cpu cores limits as well on top of cpu time limit ?

Thanks

https://redd.it/lr9e1g
@r_devops

For IaC: Pulumi or Terraform?

I am moving toward supporting Cloud IaC and am learning new tools in the process. For those that have supported both AWS and Azure using Terraform and/or Pulumi, what was the good and the bad?

https://redd.it/lrvkc8
@r_devops

How to use a Lambda function to deploy single-tenant applications to an EKS cluster?

I am working on a pet project which will have an EKS cluster where I want to have a tool do automated deployments of single-tenant applications. Got a good part of it working but not the Lambda part.

I have a single-tenant application that will need to be deployed to an EKS cluster and I am planning on having it so there is an SQS queue that will get messages of what to deploy. The idea being a user will 'sign up' so to speak and then that would create an SNS notification that will fan out to two or three SQS queues. One being the queue to set up the database and the other being to deploy the application to the EKS cluster. Third, being maybe a status system I am debating.

My question/design issue is how can I have a Lambda function be the one to process this? I have built a rough tool that does most of what I want in a single script run locally however I am trying to figure out how to get Lambda to process it and handling the EKS security part. IE currently it is running against EKS via my local kubeconfig but that doesn't work in Lambda and I really don't want to hard code the kubeconfig into the Lambda function as that is both a security issue and an operations issue..

The way I have Lambda handle this must work across multiple Lambda functions deploying at once as well as work in Lambda functions that do stuff like delete a deployment (more than just a K8s deployment).

https://redd.it/lru0wb
@r_devops

Opinions on gruntwork.io infra code library

Hello

I am curious if anyone has or had a subscription to gruntwork.io's code repo to leverage their IaC library of TF modules and if so any feedback that you can share.

Thanks!

https://redd.it/lrt20h
@r_devops

Pods Disk Utilization

Hi, community,

How can I track the storage of the pods running in the cluster? I have metrics server and Kube-state-metrics deployed but the problem is I cannot access the web UI as it's just a test environment running behind VPN? Is there any way I could fetch this data using the command line? Are there any lightweight tools that can be used to query from CLI or any functions to query Kube-server using the command line?

Thanks a lot

https://redd.it/lrz24c
@r_devops

Did you use Loki for logs aggregation? (loki vs elk)

Hi.

I discovered https://grafana.com/oss/loki/

Did somebody use that in production or with side project?

If yes, what is your opinion and how it is compared to ELK?

https://redd.it/lrokdj
@r_devops

Running Kubecost as a Prometheus metric exporter

Hi all, one of the original authors of the kubecost project. We've just open-sourced a prometheus exporter for tracking Kubernetes cost metrics. You can read public data from cloud provider pricing APIs, map those to pod resource request statistics, and view them in prometheus. Let us know what you think:

https://github.com/kubecost/cost-model/blob/develop/kubecost-exporter.md

https://redd.it/lrmq6x
@r_devops

Free intro to Linux commandline/server course starts this Monday

This course has been running successfully now every month since February 2020 - more detail at: https://LinuxUpskillChallenge.org - daily lessons appear in the sub-reddit r/linuxupskillchallenge - which is also used for support/discussion.

Suitable whatever your background, and aims to provide that "base layer" of traditional Linux skills in a fun interactive way.

https://redd.it/ls4xtb
@r_devops

[aws-quota-checker] A tool to check your AWS account for quota utilization to prevent hitting a limit

Hey r/devops, I recently hit an AWS quota limit without prior notice (ALBs created by aws-load-balancer-controller) and was looking for a way to prevent this from happening again.

Wasn't really satisfied with the solutions I found so I built one myself. It's called [aws-quota-checker](https://github.com/brennerm/aws-quota-checker) and does exactly that. It retrieves the quota limits of your AWS account and compares them to the number of current resources (e.g. number of EC2 instances).

Here's a little demo:

$ pip install aws-quota-checker
$ aws-quota-checker check vpc_count
VPCs per region [default/eu-central-1]: 1/5 ✓

Think this may be helpful for some of you. Feel free to give it a shot and let me know if you encounter any issues or would like to see some additional feature.

https://redd.it/lrqtf2
@r_devops

How to get a job as a DevOps Engineer Event

Hey Everyone I am running an online event in March on "How to get a job as a DevOps Engineer" for anyone interested:

https://community.cncf.io/events/details/cncf-manly-presents-recruitmentops-how-to-get-a-job-as-a-devops-engineer/#/

https://redd.it/lrw72f
@r_devops

What is your DevOps/SRE dream job/dream employer?

I was thinking today about how I like my current job and company a lot, and how I think the culture is pretty solid and growing in a good direction. It’s a small startup, but I am really enjoying it and the DevOps culture is great. I used to work for a huge bank and it was the polar opposite. Although my company doesn’t have the prestige that Google, etc. has, I like it and I hope we continue to grow.

What is your DevOps dream job? Who would you want to work for in an ideal situation? Is it because of prestige? Culture? Benefits? Pay?

https://redd.it/lrm7nw
@r_devops

Volume of non-engineering time as a lead or principal?

We've all seen the ten thousand posts about how to get a start and starting out in DevOps. What about at the other end? What sort of balance are Leads and Principals seeing regarding engineering vs non-engineering time?

I am nominally a lead engineer, although serving more as a principal. I am explicitly NOT a manager - I have no formal line management responsibility. I have previous experience as a manager, can do it, but am keen not to revisit that. Without giving too much info, let's say the sector is 'established financial services', although I am exclusively working in an arms-length division doing modern cloud-native stuff. I seem to spend over 90% of my week doing management and admin. Seriously, in a week, I get less than an hour to actually spike or code anything. Most of this non-engineering time is spent in meetings or dealing with bureaucracy, not mentoring, reviewing code, working with engineers etc. Whilst the volume of meetings is an acknowledged problem in the org, here it seems to have reached ridiculous levels. I am literally on Slack/Teams/email etc all day every day. I am concerned that my own professional development and enthusiasm is grinding to a halt.

I am well aware that it is easy to lose your balance in such situations - all you can see becomes all you can see. What sort of split engineering:non-engineering time are others seeing at this level?

https://redd.it/lrnhtz
@r_devops

Advise needed on a mini PC build for learning purposes.

Hi guys,

I’m looking for some advise. I’m working forwards getting certified in various areas of computing as I’m wanting to move into a DevOps role in the next few years. I’m starting with CompTIA Network+ and then moving onto Linux Foundation certifications (Admin/ and Engineer) before then going on to Kubernetes, Docker, Cloud, etc.

I’m looking for a small computer just to practise some of these concepts, mainly networking and Linux. I currently have a M1 Mac Mini which is great for the design work I do but virtualisation isn’t quite reliable yet. I’ve been looking at an 8th Gen Intel Nuc with an i5 but I’m not sure it will be enough. Someone at my work today mentioned they use laptop CPU’s and to get a mini atx build but I’m not keen on it. It’s something I’m likely to sell on in the next year or two once the Mac M chips become more mature. It will also only be a machine for learning purposes, not for gaming or anything like that.

Ideally, I don’t want to pay anymore that around £400 but would be willing to stretch that if necessary. Any help would be hugely appreciated.

https://redd.it/lrsdv8
@r_devops

CI/CD... if you were to start over, what tools would you use?

So at work we're mainly a Jenkins shop, with some homegrown tools.

A friend (at a fast growing startup) asked for ci/cd advice, and all I know is what we're currently doing at work isn't the way to go.

What would you recommend if you could start fresh?

Environment: EKS on AWS. Mainly Java-based microservices (going to be dozens), a few lambdas, a couple node+angular web apps (behind nginx). The startup expects to grow very fast with multiple feature teams soon.

What I dislike about what we do at work is that CI & CD are disjointed. I.e., CI using Jenkins (building images to our docker registry) then a completely separate process for CD. Also deployment to stage, pre-prod, prod, etc., are completely disconnected from each other.

Ideally I'd like a tool that we can streamline an entire pipeline from dev (or at least stage) to production & DR, manage rollbacks, maybe with some approval steps along the way?

I was going to look into concourse but at a glance it seems pivotal centric and pivotal is no more...

Any thoughts / suggestions on what's currently the "state of the art"?

Thanks!!

https://redd.it/ls7lzk
@r_devops

Watch Kubernetes Experts Attempt to Fix Broken Kubernetes Clusters (Episode II)

Damn ... this weeks was really fucking tough.

Both Jason DeTiberus (@detiber) and Walid Shaari (@walidshaari) decided to "mess" with `etcd` and debugging the problem was rather challenging.

I hope you find this entertaining and helpful

I need a really strong drink after this one.

https://www.youtube.com/watch?v=JzGv36Pcq3g

https://redd.it/ls9ctm
@r_devops