For those new to DevOps, who would be interested in personal, one-on-one deep-dives on specific questions?

What?

I’m offering my time, right now two or three hours a week to answer your personal questions, one-on-one, regarding DevOps and no matter the question, I will try to answer it honestly and to the best of my abilities.

Why?


I’ve been a longtime lurker of several subreddits, r/learningprogramming, r/devops, r/azure, and r/aws (literally just created an account today). Every week there are multiple posts that ask 1) how do I actually get into DevOps, 2) what tools do I need to learn, and 3) I did this thing “x” is that DevOps? Sometimes the wiki is able to address these questions, but I also believe that sometimes people need more specifics and tailored advice. I was also a teaching assistant back in college and really enjoyed teaching, but sadly nowadays I don’t get to teach that much. Which is why I want to have these series of deep-dives to address myths or misconceptions.

About myself:

I’ve been working as a DevOps engineer for close to six years now. I’ve worked at a large, financial institution where I’ve built the official internal secret storage solution comprised of Hashicorp’s Vault and Consul. This is used in production by 10,000 engineers and 500+ applications daily. I have deep knowledge in the following, but not limited to: Terraform, AWS, Kubernetes, Jenkins, and in my own personal time I’ve used GitLabCI, Netlify, Google Cloud platform, Azure, as well as built a small, personal kubernetes cluster comprised of 5 raspberry pi’s.

What I have to offer:

DevOps is a big field and it’s going to continue to grow in importance as businesses look to reduce the time to build, test, deploy, and refactor software. Also, finding a good DevOps engineer is much harder than finding a good Frontend/Backend engineer as the pool of DevOps engineers just seems to be much smaller. I’m hoping to help change that and this seems like the first good step.

If you are interested, please upvote, leave a comment on what topics/questions you find interesting, and shoot me a dm.

https://redd.it/e75tun
@r_devops

Settings storage

I need a new strategy for storing (and retrieving) of settings files. I have a bunch of apps that need access to some settings files, this files are shared across some apps. Then there is also my dev environment locally that needs access to this settings to (but with different values eg: local db instead of prod db). I am using AWS for all the devops things so I am thinking to have the settings files in a S3 bucket so that the apps can download the files and use them. Is there a better way of doing this?

https://redd.it/e6zcpu
@r_devops

Exhausted NFS, need small scale high perf storage

I'm using NFS on an ec2 which is 1.5tb in size
There are a few servers (usually around 15-20) that write various files in many folders at a high rate)

I need a replacement for the NFS server because io and cpu can't take it
So a few questions:

1. AWS EFS, will it yield better results than NFS? Never tried it
2. Thinking of GlusterFS (through I had strange issues with it in the past)
3. Maybe Ceph ? (Never tried it)

I don't need scale, just make it work fast and reliable

Your thoughts?

https://redd.it/e6wgtp
@r_devops

Docker socket authorization best practices

Hi all!
My original question was going to be "Is using HTTPS to connect to the docker daemon with mTLS from a container running on that host more secure than simply mounting `/var/run/docker.sock`"
I answered my own question after a quick search and found https://docs.docker.com/engine/extend/plugins_authorization/
Which states:
> Docker’s out-of-the-box authorization model is all or nothing. Any user with permission to access the Docker daemon can run any Docker client command. The same is true for callers using Docker’s Engine API to contact the daemon.

So now I want to know:
How have you used and configured authorization plugins? Under what scenarios does it make sense to configure authorization Vs running another VM for applications that need access to a docker socket.
I've started using Zalenium recently and it starts sibling containers for running browsers in.
If a malicious Zalenium image got pulled, it would have complete access to docker. This of course is true for any container you're mounting the docker socket into (or presumably even if using a mTLS connection from container to host)
Would love to hear thoughts on this, thanks!

https://redd.it/e6v6df
@r_devops

Learn Containerization fundamentals with examples

Hello,

I'm a recent university graduate and started working as DevOps engineer recently. I'd like to widen my skillset to help my team better. I've had zero exposure to any DevOps tech that's used in the industry and would like to learn it now.
I'm a average with my tech skills and know basics of programming and some OS concepts. We use azure and i know few things about the platform now. I would like to learn about Containerization (mainly docker) and later touch upon kubernetes. Any resources which would help me get started. I'm not a very technical person so I'm looking for a resource which covers basics. Hands on labs would be great.

Thanks in advance.

https://redd.it/e7awob
@r_devops

Unit testing Prometheus alerts

Is there a way to test out an alert config for possible false positives? Currently, I have a bash script using promtool to test the expression with past data to see how many times the alert would be triggered. I want to build this into a PRB, but just wondering if there is another way to go about this?

https://redd.it/e79edh
@r_devops

Phoenix Project IRL

The catch here is that we dont have a Brent. Can you imagine if in the middle of the book Brent up and quit the company? There would be no book. The story would have ended there with the company being split up and sold. Thats the predicament we are in. Two of our top guys who happen to be (used to be!) contractors to our company and have managed a massive system for 16+ years and managed to hoard all of the technical know hows suddenly quit.

https://redd.it/e78psf
@r_devops

Artifactory and Python Wheels

Hi all. I use alpine-based images for most of my deployed images. This is great due to smaller size and usually smaller attack surface. The major down-side, from a python perspective, is that the binary wheels that are stored on PyPi are usually not compatible with MUSL based libraries (non-GLIBC) and need to be built from scratch for every pip install. I am already using some caching tricks and multi-stage builds in docker to help a little bit...but...I would like to use artifactory to make this process less painful.

​

I am struggling to understand the process with artifactory involved. FWIW, I use poetry for locking my packages. The general flow would be something like

* `poetry install`
* <somehow push back the built wheels to my artifactory instance>
* update the lock with hashes of the built wheels??

Theoretically any future projects with the latest hashes would be able to use these prebuilt wheels. Can anyone help walk me through how this should actually be accomplished for stage two and three of my steps above?

https://redd.it/e6ibif
@r_devops

Exporting JFrog Artifactory Bundles?

We're looking at Artifactory for solving a problem related to moving containers between completely isolated environments. The bundles feature is appealing, but I haven't found a way to Import/Export bundles, just entire systems and repositories. Is there a way that just isn't in the documentation I'm looking at or are the only two export levels system and repository?

https://redd.it/e6jv4k
@r_devops

Is there a key value parser for fluentd where you can change what the quote character should be?

Is there a key value parser for fluentd where you can change what the quote character should be?
I want to use `¤` as the quote character and my logs look like this:

```
key: ¤value with space¤, foo: ¤i can "quote"¤, bar=¤I hope nobody uses my special quote character inside a value¤
```

The built in parsers **ltsv** can't even handle values with quotes. Se the example below:

```
<parse>
@type ltsv
delimiter_pattern /\s+/
label_delimiter =
</parse>
```

The following example:
```
level=info ts=2019-12-05T07:39:24.487738226Z caller=operator.go:1050 msg="sync prometheus"
```

becomes:
```json
{
"level": "info",
"ts": "2019-12-05T07:39:24.487738226Z",
"caller": "operator.go:1050",
"msg": "\\"sync",
"prometheus\\"": null
}
```

https://redd.it/e6hggy
@r_devops

Handling multiple API version

Hey guys newbie OPS guy here,

Recently I received the task to provide support for our API and that this API will have multiple versions running at the same time as latest, current, to be discontinued ( something like this ). The first thing that came to mind was multiple infrastructures to support each of the versions, but I'm not very fond of this idea if you can share your experience and how you worked with this challenge your help is much appreciated.

thanks.

https://redd.it/e6gukz
@r_devops

Kubernetes Guided Install & Configuration

A month or two ago I was learning Kubernetes and got some really good insight/help here. I decided to blog about it and document how to do it. It was one of the learning experiences that made me decide to write a blog. I wanted to share and return the favor for anyone new coming across Kubernetes and wanting to setup a lab and test it out.

This is a multi part article that walks through setting up Kubernetes using Photon (Fusion/VMware friendly). Some of the paths/parts take you through network overlays, Kubernetes dashboards and briefly SSL.

[Intro to Kubernetes](https://blog.woohoosvcs.com/2019/10/intro-to-kubernetes/)

Apologies if this comes across as breaking rule #4. My intent is definitely not spam. Hope this is useful to someone!

https://redd.it/e7hyme
@r_devops

What are some best courses on Azure DevOps that teach you CI/CD with GitHub?

I am a full stack developer (read .NET , Angular and SQL database).
I am new to DevOps. I am primarily interested in learning only what would help me function smoothly as a developer in an organization.
I am talking creating CI/CD pipelines. Automated deployments etc of .NET and angular applications. May be also a bit of kubernetes and docker.

What video courses do I need to watch? I say video courses because i am a slow reader. So slow that i lose interest after just few minutes of reading.

https://redd.it/e7jbps
@r_devops

When do you apply your application artifact?

I have always worked under the assumption that the application artifact and configuration should be applied to the container outside of the build process at runtime. This keeps the underlying container seperate from the application and configuration allowing for easy updates of the container and or the application depending on need.

Update the application / artifact in the repo, or update the configuration parameters in the parameter store and run a new container to get the updated application / configuration keeping turn around times low.

Or build a new container to update the os and under lying libraries with out changing the artifact or configuration. Allowing us to update / fix errata without rebuilding an artifact.

This means a developer pulls an existing prod container and applies his new artifact to it and configuration is pulled from a param store. Xhe does not need to build a new container or touch a dockerfile or run a Packer build. Turn around is very short as testing a new build is as simple as running an existing container.

Am I wrong in this? Is this an anti pattern? I am having what amounts to a religious debate with a dev about this, I am willing to be "wrong" but it feels right to me..

https://redd.it/e7jeel
@r_devops

CI/CD for CI/CD

At what point, if ever, does our automation become so complex that it itself must go through a CI/CD process?

https://redd.it/e7j46k
@r_devops

what makes an application a sidecar?

Hey guys, so we have two containers running in a pod.. the idea is to have main container doing the job( receiving request) and a secondary container doing a background task of constantly checking the main app network status

i have my deployment file with the secondary container coming up first and then the main container coming after, please check code snippet below [https://gist.github.com/dapseen/51bc7e59b96d934938a4afce34c615bb](https://gist.github.com/dapseen/51bc7e59b96d934938a4afce34c615bb) Is it because of the placement of container i.e fund worker coming starting up first that makes it the main app and other container coming up after makes it secondary container.

&#x200B;

Is there a special annontation in k8 that can tell cluster or kubernetes the container to make side car

&#x200B;

Thanks

https://redd.it/e7noxn
@r_devops

What DevOps solutions (if any) would you pick if it were up to you?

I have worked as a System Administrator and Programmer for decades. To me the term DevOps means Developers and Operations working together. For years, the industry seems to think that DevOps almost exclusively means AWS/Chef/Jenkins/Docker/Kubernetes. A few years ago, I worked for a small firm which had inadvertently paid to have many of their processes encoded into ansible so that they could scale out and build thousands of copies of their one LAMP solution in different environments (which they didn't even imagine doing). One of the owners made motions to me suggesting that all I (or any other IT guy) really wanted to do was to have my way with his backside. No one could understand why they ended up this way or why they had to pay me to undo this. I build out solutions on Digital Ocean, work primarily from the shell using vim, and publish my changes into github. Other than that, I don't really feel that DevOps (aka AWS/Chef/Jenkins/Docker/Kubernetes) is particularly helpful. Folks are more interested in having a two page website with a contact button that works. So my questions is this: If it weren't for the fact that your particular company had made it their missions to migrate their legacy operations into the AWS (Google or Azure) cloud and implement DevOps tools, would you chose to operate this way on your own?

https://redd.it/e7hv9e
@r_devops

How do you manage service accounts?

I work at an older company that primarily uses Microsoft; our service accounts are just special users with passwords that never expire, as most of our tools only support LDAP.

A recent audit uncovered that teams are being reckless with these (production) credentials—sharing them publicly on Bitbucket, Sharepoint, etc.—and re-using them across many different projects and systems. Due to the latter, changing exposed passwords will likely have a significant impact... but not changing them gives bad actors access to an untold number of production systems.

How is your company managing service accounts / long-running system-to-system access?

I'd like to learn some best practices and talk to management about how we can avoid this mess in the future.

https://redd.it/e6gj5h
@r_devops

AWS cross-account CI/CD without IAM

Im interested in the best options for cross account CI/CD. I am moving from single to multi account architecture. I currently have a Jenkins instance and I am not interested in rebuilding it (Default VPC) in order to use direct connect.

My client does not want to use IAM for this.

&#x200B;

What are you guys thinking ?!

https://redd.it/e6jb8m
@r_devops

How often do you get network partitions or node failures in your clusters?

Been experimenting with distributed tools like NSQ and DGraph. My side projects are consequently quite costly per month, because I like to have my clusters simulate production-like settings in the cloud, using purely IaaS.

It got me thinking how necessary full distribution is in a real world scenario, and whether these tools offering first class support for distribution is because of "just-in-case", or to fulfil an SLA, or if because cluster failures are actually commonplace, be it nodes or networks.

Knowing the frequency of downtimes in the real world can help me or anyone interested decide how much we can get away with doing SPOF deployments for a production env, or how long, before it becomes a real potential issue.

Would also appreciate links to surveys or stats. Either there's not much on the topic, or I have shitty googling skills.

https://redd.it/e7p4lw
@r_devops