Code formatting and cleanup as part of pipeline

Hi

We are getting our pipeline set up, and want to add a phase before build to format and clean up code. Is this a good plan? We have pre-commit hooks already for our node packages which does this already, but our java and c# does not. Some of our team don't want to do this automatically as it will add a commit by our pipeline to the history, but others feel it's the only way to get all of the code in standard formatting and clean style.

Thanks.

https://redd.it/e6u9ot
@r_devops

Options for AWS cross account CI/CD no IAM

Im interested in the best options for cross account CI/CD. I am moving from single to multi account architecture. I currently have a Jenkins instance and I am not interested in rebuilding it (Default VPC) in order to use direct connect.

My client does not want to use IAM for this.

​

What are you guys thinking ?!

https://redd.it/e6m2y7
@r_devops

2020 one year challenge to get into the Cloud / DevOps space from fast food employee with basic tech knowledge to a working level.

Hello everyone. I will be challenging myself to be in the Cloud / DevOps space by EO2020.

# Current skills

1. Intermediate Python knowledge
1. Basic understanding of core data types
2. Can create a program / script given enough time
2. Extremely basic Linux knowledge
1. Understanding of commands like cd, ls, mv, I know how to switch users, log into root
3. Extremely basic Networking knowledge
1. I know a few different protocols like DNS, DHCP, etc. The basic shit. I only know these at a theoretical level, if that makes any sense. No practical knowledge working with them

# Gameplan

1. **Try** and get a helpdesk position for a better source of income, while learning. If I have to stay in fast food until EO2020, that's fine too. Fuck it.
2. Obtain a Linux Academy subscription
3. Start off by learning Linux System Administration fundamentals by following the Junior Linux System Administration course on Linux Academy which will lead to my **RHCSA**
4. Next, learn fundamental Networking, by following and understanding the [Networking course by Google](https://www.youtube.com/watch?v=QKfk7YFILws)
5. Database Administration and SQL Language Basics course on Linux Academy
6. Source Control With Git course on Linux Academy
7. DevOps Essentials course on Linux Academy
8. Using Terraform to Manage Applications and Infrastructure course on Linux Academy
9. Jenkins Quick Start course on Linux Academy
10. Certified Jenkins Engineer course on Linux Academy
11. Ansible: Setup, Configure, and Ad Hoc Commands Deep Dive on Linux Academy
12. Ansible: Playbooks Deep Dive course on Linux Academy
13. AWS Concepts course on Linux Academy
14. AWS Essentials course on Linux Academy
15. AWS Certified Solutions Architect Associate

\- Yes, throughout the year I will also be focusing on learning the processes behind DevOps, not just solely focused on tooling

\- Yes, I will be trying to put together projects

\- *"You need to get professional experience for this stuff, which will help you learn about the processes, pain points, blah blah".* Yes Mr Reddit poster, I would need to get a job to be able to understand these things at a solid level. Oh wait, in order to get a job, I need experience. Oh wait, in order to get experience I need a job. **That** is the problem a lot of the more seasoned engineers don't see on here for us new guys wanting to even learn any of this stuff / get a job in the space.


\- *"Don't just chase certs, they're meaningless".* Yes Mr Reddit poster, I understand that exp > certs any day of the week, but remember, in order for me to get experience, I need...... a job. Bingo. The only certs I'm getting are the RHCSA + AWS SAA. The Jenkins cert I will study for, **but not actually take.**

Anybody who wants to use my path as a guide, and challenge themselves as well, feel free to do so. Will provide an update at the end of the year. Worst case scenario? I don't get a job, and stay a fast food pleb with 10x more tech knowledge than I knew before. Best case scenario? That's what we're going to find out.

Peace.

https://redd.it/e6z74j
@r_devops

HTTP Availability and Polling Intervals

My question is this: Is it true to say, that polling an endpoint every 30 minutes for availability is too long? My thoughts are that if there is a minute of downtime when that poll occurs, that will count for 30 minutes of downtime (as the next check is done 30 minutes later).

Is there any best practices on polling intervals for endpoints? In fact, how often do you poll?

https://redd.it/e72lbc
@r_devops

K8s deployment/config files : how do you maintain them?

So, we've started using K8s recently and have gone live too. We've started doing releases too and realised that we have to update the Prod K8s configmaps/secrets comparing it with the UAT. Some values change for the Prod keys. We're doing the prod deployments manually. We just copy the deployment dir to a new one, change the image names and then deploy them.

I know this is far from a best practice, but we've just started out. So I'm asking around to understand what's the best practices you guys follow? We're looking at Spinnaker too but that's not in the immediate future.

https://redd.it/e72nz7
@r_devops

Need Suggestion for logging from a distributed system (Servers, UI, Sensors)

Currently I am working on a project for logging all events in a distributed system. The goal is to make the development team's job more efficient by being able to observe the full stack logs in one place. As of now, there is no efficient tracing mechanism for exceptions and events after pushing to production and staging. Our stack is mainly Python, JS, and some C++. All services are currently running on Azure app services and Azure IoT hub (for logging IoT devices' data) . Azure provides Application insight service which makes it easy to stream logs using their SDK from app services to app insight. They have dashboards that can be customized by the users to show desired charts and also it provides possibility of querying logs. I dont want to jump in and use app insight only because we are using app services in Azure. I am researching and reading about all the possibilities for different log management systems. I have used ELK ( Elasticsearch, Logstash, Kibana) in past, I am not sure if we need extra tech stack added to our technologies.

Also, I look into the Sentry if you heard about them. Their aim is for providing a better tracing for exceptions and linking the root causes of the exception together. They have SDK almost for all languages.

Another system that I looked into was Greylog which seems like is more for monitoring the server events and network and less concerned about exceptions and customized logs.

I also looked into Datadog, Splunk, and Sumo Logic.

Questions that I ask myself

\-----------------------

\- Is it better to stick with Application Insight (Azure)? Why?

\- What happens if we want to move away from Azure to another could system or even in house servers? ( Application Insight can be decoupled from Amazon app services )

\- Should we just use ELK since we can customize the metrics and provide as much as telemetry that we want?

\- Should we host our own logging management portal in house?

​

Any suggestion and questions that can help me to narrow down and get to the right solution would be highly appreciated, it would be a big help

Thanks in advance

https://redd.it/e73reg
@r_devops

I'm trying to gauge the value of containerization

I have just begun the process of evaluating containerization for the PWAs I write for various organizations. I host the apps on a CentOS server on DigitalOcean and everything is great. Would I gain any value by containerizing the apps in a Docker environment vs. letting them be stand-alone apps on the server?

https://redd.it/e72rj8
@r_devops

For those new to DevOps, who would be interested in personal, one-on-one deep-dives on specific questions?

What?

I’m offering my time, right now two or three hours a week to answer your personal questions, one-on-one, regarding DevOps and no matter the question, I will try to answer it honestly and to the best of my abilities.

Why?


I’ve been a longtime lurker of several subreddits, r/learningprogramming, r/devops, r/azure, and r/aws (literally just created an account today). Every week there are multiple posts that ask 1) how do I actually get into DevOps, 2) what tools do I need to learn, and 3) I did this thing “x” is that DevOps? Sometimes the wiki is able to address these questions, but I also believe that sometimes people need more specifics and tailored advice. I was also a teaching assistant back in college and really enjoyed teaching, but sadly nowadays I don’t get to teach that much. Which is why I want to have these series of deep-dives to address myths or misconceptions.

About myself:

I’ve been working as a DevOps engineer for close to six years now. I’ve worked at a large, financial institution where I’ve built the official internal secret storage solution comprised of Hashicorp’s Vault and Consul. This is used in production by 10,000 engineers and 500+ applications daily. I have deep knowledge in the following, but not limited to: Terraform, AWS, Kubernetes, Jenkins, and in my own personal time I’ve used GitLabCI, Netlify, Google Cloud platform, Azure, as well as built a small, personal kubernetes cluster comprised of 5 raspberry pi’s.

What I have to offer:

DevOps is a big field and it’s going to continue to grow in importance as businesses look to reduce the time to build, test, deploy, and refactor software. Also, finding a good DevOps engineer is much harder than finding a good Frontend/Backend engineer as the pool of DevOps engineers just seems to be much smaller. I’m hoping to help change that and this seems like the first good step.

If you are interested, please upvote, leave a comment on what topics/questions you find interesting, and shoot me a dm.

https://redd.it/e75tun
@r_devops

Settings storage

I need a new strategy for storing (and retrieving) of settings files. I have a bunch of apps that need access to some settings files, this files are shared across some apps. Then there is also my dev environment locally that needs access to this settings to (but with different values eg: local db instead of prod db). I am using AWS for all the devops things so I am thinking to have the settings files in a S3 bucket so that the apps can download the files and use them. Is there a better way of doing this?

https://redd.it/e6zcpu
@r_devops

Exhausted NFS, need small scale high perf storage

I'm using NFS on an ec2 which is 1.5tb in size
There are a few servers (usually around 15-20) that write various files in many folders at a high rate)

I need a replacement for the NFS server because io and cpu can't take it
So a few questions:

1. AWS EFS, will it yield better results than NFS? Never tried it
2. Thinking of GlusterFS (through I had strange issues with it in the past)
3. Maybe Ceph ? (Never tried it)

I don't need scale, just make it work fast and reliable

Your thoughts?

https://redd.it/e6wgtp
@r_devops

Docker socket authorization best practices

Hi all!
My original question was going to be "Is using HTTPS to connect to the docker daemon with mTLS from a container running on that host more secure than simply mounting `/var/run/docker.sock`"
I answered my own question after a quick search and found https://docs.docker.com/engine/extend/plugins_authorization/
Which states:
> Docker’s out-of-the-box authorization model is all or nothing. Any user with permission to access the Docker daemon can run any Docker client command. The same is true for callers using Docker’s Engine API to contact the daemon.

So now I want to know:
How have you used and configured authorization plugins? Under what scenarios does it make sense to configure authorization Vs running another VM for applications that need access to a docker socket.
I've started using Zalenium recently and it starts sibling containers for running browsers in.
If a malicious Zalenium image got pulled, it would have complete access to docker. This of course is true for any container you're mounting the docker socket into (or presumably even if using a mTLS connection from container to host)
Would love to hear thoughts on this, thanks!

https://redd.it/e6v6df
@r_devops

Learn Containerization fundamentals with examples

Hello,

I'm a recent university graduate and started working as DevOps engineer recently. I'd like to widen my skillset to help my team better. I've had zero exposure to any DevOps tech that's used in the industry and would like to learn it now.
I'm a average with my tech skills and know basics of programming and some OS concepts. We use azure and i know few things about the platform now. I would like to learn about Containerization (mainly docker) and later touch upon kubernetes. Any resources which would help me get started. I'm not a very technical person so I'm looking for a resource which covers basics. Hands on labs would be great.

Thanks in advance.

https://redd.it/e7awob
@r_devops

Unit testing Prometheus alerts

Is there a way to test out an alert config for possible false positives? Currently, I have a bash script using promtool to test the expression with past data to see how many times the alert would be triggered. I want to build this into a PRB, but just wondering if there is another way to go about this?

https://redd.it/e79edh
@r_devops

Phoenix Project IRL

The catch here is that we dont have a Brent. Can you imagine if in the middle of the book Brent up and quit the company? There would be no book. The story would have ended there with the company being split up and sold. Thats the predicament we are in. Two of our top guys who happen to be (used to be!) contractors to our company and have managed a massive system for 16+ years and managed to hoard all of the technical know hows suddenly quit.

https://redd.it/e78psf
@r_devops

Artifactory and Python Wheels

Hi all. I use alpine-based images for most of my deployed images. This is great due to smaller size and usually smaller attack surface. The major down-side, from a python perspective, is that the binary wheels that are stored on PyPi are usually not compatible with MUSL based libraries (non-GLIBC) and need to be built from scratch for every pip install. I am already using some caching tricks and multi-stage builds in docker to help a little bit...but...I would like to use artifactory to make this process less painful.

​

I am struggling to understand the process with artifactory involved. FWIW, I use poetry for locking my packages. The general flow would be something like

* `poetry install`
* <somehow push back the built wheels to my artifactory instance>
* update the lock with hashes of the built wheels??

Theoretically any future projects with the latest hashes would be able to use these prebuilt wheels. Can anyone help walk me through how this should actually be accomplished for stage two and three of my steps above?

https://redd.it/e6ibif
@r_devops

Exporting JFrog Artifactory Bundles?

We're looking at Artifactory for solving a problem related to moving containers between completely isolated environments. The bundles feature is appealing, but I haven't found a way to Import/Export bundles, just entire systems and repositories. Is there a way that just isn't in the documentation I'm looking at or are the only two export levels system and repository?

https://redd.it/e6jv4k
@r_devops

Is there a key value parser for fluentd where you can change what the quote character should be?

Is there a key value parser for fluentd where you can change what the quote character should be?
I want to use `¤` as the quote character and my logs look like this:

```
key: ¤value with space¤, foo: ¤i can "quote"¤, bar=¤I hope nobody uses my special quote character inside a value¤
```

The built in parsers **ltsv** can't even handle values with quotes. Se the example below:

```
<parse>
@type ltsv
delimiter_pattern /\s+/
label_delimiter =
</parse>
```

The following example:
```
level=info ts=2019-12-05T07:39:24.487738226Z caller=operator.go:1050 msg="sync prometheus"
```

becomes:
```json
{
"level": "info",
"ts": "2019-12-05T07:39:24.487738226Z",
"caller": "operator.go:1050",
"msg": "\\"sync",
"prometheus\\"": null
}
```

https://redd.it/e6hggy
@r_devops

Handling multiple API version

Hey guys newbie OPS guy here,

Recently I received the task to provide support for our API and that this API will have multiple versions running at the same time as latest, current, to be discontinued ( something like this ). The first thing that came to mind was multiple infrastructures to support each of the versions, but I'm not very fond of this idea if you can share your experience and how you worked with this challenge your help is much appreciated.

thanks.

https://redd.it/e6gukz
@r_devops

Kubernetes Guided Install & Configuration

A month or two ago I was learning Kubernetes and got some really good insight/help here. I decided to blog about it and document how to do it. It was one of the learning experiences that made me decide to write a blog. I wanted to share and return the favor for anyone new coming across Kubernetes and wanting to setup a lab and test it out.

This is a multi part article that walks through setting up Kubernetes using Photon (Fusion/VMware friendly). Some of the paths/parts take you through network overlays, Kubernetes dashboards and briefly SSL.

[Intro to Kubernetes](https://blog.woohoosvcs.com/2019/10/intro-to-kubernetes/)

Apologies if this comes across as breaking rule #4. My intent is definitely not spam. Hope this is useful to someone!

https://redd.it/e7hyme
@r_devops

What are some best courses on Azure DevOps that teach you CI/CD with GitHub?

I am a full stack developer (read .NET , Angular and SQL database).
I am new to DevOps. I am primarily interested in learning only what would help me function smoothly as a developer in an organization.
I am talking creating CI/CD pipelines. Automated deployments etc of .NET and angular applications. May be also a bit of kubernetes and docker.

What video courses do I need to watch? I say video courses because i am a slow reader. So slow that i lose interest after just few minutes of reading.

https://redd.it/e7jbps
@r_devops

When do you apply your application artifact?

I have always worked under the assumption that the application artifact and configuration should be applied to the container outside of the build process at runtime. This keeps the underlying container seperate from the application and configuration allowing for easy updates of the container and or the application depending on need.

Update the application / artifact in the repo, or update the configuration parameters in the parameter store and run a new container to get the updated application / configuration keeping turn around times low.

Or build a new container to update the os and under lying libraries with out changing the artifact or configuration. Allowing us to update / fix errata without rebuilding an artifact.

This means a developer pulls an existing prod container and applies his new artifact to it and configuration is pulled from a param store. Xhe does not need to build a new container or touch a dockerfile or run a Packer build. Turn around is very short as testing a new build is as simple as running an existing container.

Am I wrong in this? Is this an anti pattern? I am having what amounts to a religious debate with a dev about this, I am willing to be "wrong" but it feels right to me..

https://redd.it/e7jeel
@r_devops