I get to start a DevOps program from scratch. Suggestions???

Hi all!

I was just put in charge of some pretty big projects at my job. Think creating a data warehouse, IAM, and all scripting support to the NetOps team, to name a few.

Here is the exciting part. I get to create this DevOps environment from scratch. My boss is giving me all the tools I need. I’m already familiar with SDLC/Agile, and that’s the model I’m going with.

I want to set this up correctly so, here’s my ask: what are common mistakes made in DevOps? What are some good resource websites? I saw the YouTube Bootcamp, are there any other good trainings out there? Any suggestions?

Again, my boss is giving me free reign. He’s already pretty impressed with my pre-project reports and trusts me to do whatever’s necessary to create a good environment.

Thanks!

https://redd.it/e6n9rl
@r_devops

Keeping track of infrastructure

I have a dozen of physical hardware laying around being Jenkins slaves. These slaves are actively used by multiple departments for building and testing on real hardware(mostly microcontrollers, but also android/osx phones). \*It's currently a goddamn mess\*. I'm not joking when I say that we don't even know if we have all the machines inventoried and what machine does what because the jenkins labels are all over the place. Some machines have custom scripts running on them to fix unrelated bugs that affect other systems. It's frustrating and needs a big overhaul, without bringing down the current setups. Some of the machines are misconfigured to the point that we can't even access them anymore, others are just unstable.

As part of that we started standardizing jenkins labels, installed monitoring tools MANUALLY(ansible would be nice, if IT allowed us to use it properly lmao i'm in devops this is WIP) and are now basically going to inventory everything.

What kind of tools can be used for this? Currently it's just some random webpage/wiki that needs manual updating, which is incredibly prone to error and a Bad Idea(tm). I need something that can be deployed automatically and monitored real-time. Zabbix is an option but it doesn't seem to have a "public" view without logging in(departments should be able to look at the status, but not touch). Also requires a master/slave setup, and I'd prefer agentless if possible

Other advice is also welcome

Yes, I know normally maintaining infrastructure isn't really part of devops. I just got on the job and have legacy stuff to deal with.

https://redd.it/e6w353
@r_devops

When writing deploy/rollback scripts - pick target based on current branch. Good or bad?

We are having this discussion internally. Some say that is safer to have specific scripts, other that because its automated, having a single on that switch depending on branch is cleaner.

​

What are good practices on that? Does that even make sense that we are having this discussions?

https://redd.it/e6wzsi
@r_devops

How to group IT Monitoring Tools by Market?

Hey everyone,

Fresh graduate who wants to join an IT monitoring company as a junior Product Management. I am researching the space so as to understand the market better.

How would you categorize the different IT monitoring solutions (e.g Nagios, Zabbix, Prometheus, Influx, Netdata, Datadog,etc.)? I am trying to find the different markets (segments) that exist inside the domain.

I have some ideas, but I don't want to nudge anyone.

**Thank you so much for your time, it is highly appreciated :)**

(I just cross-posted to maximise exposure from r/sysadmin)

https://redd.it/e6yu7h
@r_devops

Code formatting and cleanup as part of pipeline

Hi

We are getting our pipeline set up, and want to add a phase before build to format and clean up code. Is this a good plan? We have pre-commit hooks already for our node packages which does this already, but our java and c# does not. Some of our team don't want to do this automatically as it will add a commit by our pipeline to the history, but others feel it's the only way to get all of the code in standard formatting and clean style.

Thanks.

https://redd.it/e6u9ot
@r_devops

Options for AWS cross account CI/CD no IAM

Im interested in the best options for cross account CI/CD. I am moving from single to multi account architecture. I currently have a Jenkins instance and I am not interested in rebuilding it (Default VPC) in order to use direct connect.

My client does not want to use IAM for this.

​

What are you guys thinking ?!

https://redd.it/e6m2y7
@r_devops

2020 one year challenge to get into the Cloud / DevOps space from fast food employee with basic tech knowledge to a working level.

Hello everyone. I will be challenging myself to be in the Cloud / DevOps space by EO2020.

# Current skills

1. Intermediate Python knowledge
1. Basic understanding of core data types
2. Can create a program / script given enough time
2. Extremely basic Linux knowledge
1. Understanding of commands like cd, ls, mv, I know how to switch users, log into root
3. Extremely basic Networking knowledge
1. I know a few different protocols like DNS, DHCP, etc. The basic shit. I only know these at a theoretical level, if that makes any sense. No practical knowledge working with them

# Gameplan

1. **Try** and get a helpdesk position for a better source of income, while learning. If I have to stay in fast food until EO2020, that's fine too. Fuck it.
2. Obtain a Linux Academy subscription
3. Start off by learning Linux System Administration fundamentals by following the Junior Linux System Administration course on Linux Academy which will lead to my **RHCSA**
4. Next, learn fundamental Networking, by following and understanding the [Networking course by Google](https://www.youtube.com/watch?v=QKfk7YFILws)
5. Database Administration and SQL Language Basics course on Linux Academy
6. Source Control With Git course on Linux Academy
7. DevOps Essentials course on Linux Academy
8. Using Terraform to Manage Applications and Infrastructure course on Linux Academy
9. Jenkins Quick Start course on Linux Academy
10. Certified Jenkins Engineer course on Linux Academy
11. Ansible: Setup, Configure, and Ad Hoc Commands Deep Dive on Linux Academy
12. Ansible: Playbooks Deep Dive course on Linux Academy
13. AWS Concepts course on Linux Academy
14. AWS Essentials course on Linux Academy
15. AWS Certified Solutions Architect Associate

\- Yes, throughout the year I will also be focusing on learning the processes behind DevOps, not just solely focused on tooling

\- Yes, I will be trying to put together projects

\- *"You need to get professional experience for this stuff, which will help you learn about the processes, pain points, blah blah".* Yes Mr Reddit poster, I would need to get a job to be able to understand these things at a solid level. Oh wait, in order to get a job, I need experience. Oh wait, in order to get experience I need a job. **That** is the problem a lot of the more seasoned engineers don't see on here for us new guys wanting to even learn any of this stuff / get a job in the space.


\- *"Don't just chase certs, they're meaningless".* Yes Mr Reddit poster, I understand that exp > certs any day of the week, but remember, in order for me to get experience, I need...... a job. Bingo. The only certs I'm getting are the RHCSA + AWS SAA. The Jenkins cert I will study for, **but not actually take.**

Anybody who wants to use my path as a guide, and challenge themselves as well, feel free to do so. Will provide an update at the end of the year. Worst case scenario? I don't get a job, and stay a fast food pleb with 10x more tech knowledge than I knew before. Best case scenario? That's what we're going to find out.

Peace.

https://redd.it/e6z74j
@r_devops

HTTP Availability and Polling Intervals

My question is this: Is it true to say, that polling an endpoint every 30 minutes for availability is too long? My thoughts are that if there is a minute of downtime when that poll occurs, that will count for 30 minutes of downtime (as the next check is done 30 minutes later).

Is there any best practices on polling intervals for endpoints? In fact, how often do you poll?

https://redd.it/e72lbc
@r_devops

K8s deployment/config files : how do you maintain them?

So, we've started using K8s recently and have gone live too. We've started doing releases too and realised that we have to update the Prod K8s configmaps/secrets comparing it with the UAT. Some values change for the Prod keys. We're doing the prod deployments manually. We just copy the deployment dir to a new one, change the image names and then deploy them.

I know this is far from a best practice, but we've just started out. So I'm asking around to understand what's the best practices you guys follow? We're looking at Spinnaker too but that's not in the immediate future.

https://redd.it/e72nz7
@r_devops

Need Suggestion for logging from a distributed system (Servers, UI, Sensors)

Currently I am working on a project for logging all events in a distributed system. The goal is to make the development team's job more efficient by being able to observe the full stack logs in one place. As of now, there is no efficient tracing mechanism for exceptions and events after pushing to production and staging. Our stack is mainly Python, JS, and some C++. All services are currently running on Azure app services and Azure IoT hub (for logging IoT devices' data) . Azure provides Application insight service which makes it easy to stream logs using their SDK from app services to app insight. They have dashboards that can be customized by the users to show desired charts and also it provides possibility of querying logs. I dont want to jump in and use app insight only because we are using app services in Azure. I am researching and reading about all the possibilities for different log management systems. I have used ELK ( Elasticsearch, Logstash, Kibana) in past, I am not sure if we need extra tech stack added to our technologies.

Also, I look into the Sentry if you heard about them. Their aim is for providing a better tracing for exceptions and linking the root causes of the exception together. They have SDK almost for all languages.

Another system that I looked into was Greylog which seems like is more for monitoring the server events and network and less concerned about exceptions and customized logs.

I also looked into Datadog, Splunk, and Sumo Logic.

Questions that I ask myself

\-----------------------

\- Is it better to stick with Application Insight (Azure)? Why?

\- What happens if we want to move away from Azure to another could system or even in house servers? ( Application Insight can be decoupled from Amazon app services )

\- Should we just use ELK since we can customize the metrics and provide as much as telemetry that we want?

\- Should we host our own logging management portal in house?

​

Any suggestion and questions that can help me to narrow down and get to the right solution would be highly appreciated, it would be a big help

Thanks in advance

https://redd.it/e73reg
@r_devops

I'm trying to gauge the value of containerization

I have just begun the process of evaluating containerization for the PWAs I write for various organizations. I host the apps on a CentOS server on DigitalOcean and everything is great. Would I gain any value by containerizing the apps in a Docker environment vs. letting them be stand-alone apps on the server?

https://redd.it/e72rj8
@r_devops

For those new to DevOps, who would be interested in personal, one-on-one deep-dives on specific questions?

What?

I’m offering my time, right now two or three hours a week to answer your personal questions, one-on-one, regarding DevOps and no matter the question, I will try to answer it honestly and to the best of my abilities.

Why?


I’ve been a longtime lurker of several subreddits, r/learningprogramming, r/devops, r/azure, and r/aws (literally just created an account today). Every week there are multiple posts that ask 1) how do I actually get into DevOps, 2) what tools do I need to learn, and 3) I did this thing “x” is that DevOps? Sometimes the wiki is able to address these questions, but I also believe that sometimes people need more specifics and tailored advice. I was also a teaching assistant back in college and really enjoyed teaching, but sadly nowadays I don’t get to teach that much. Which is why I want to have these series of deep-dives to address myths or misconceptions.

About myself:

I’ve been working as a DevOps engineer for close to six years now. I’ve worked at a large, financial institution where I’ve built the official internal secret storage solution comprised of Hashicorp’s Vault and Consul. This is used in production by 10,000 engineers and 500+ applications daily. I have deep knowledge in the following, but not limited to: Terraform, AWS, Kubernetes, Jenkins, and in my own personal time I’ve used GitLabCI, Netlify, Google Cloud platform, Azure, as well as built a small, personal kubernetes cluster comprised of 5 raspberry pi’s.

What I have to offer:

DevOps is a big field and it’s going to continue to grow in importance as businesses look to reduce the time to build, test, deploy, and refactor software. Also, finding a good DevOps engineer is much harder than finding a good Frontend/Backend engineer as the pool of DevOps engineers just seems to be much smaller. I’m hoping to help change that and this seems like the first good step.

If you are interested, please upvote, leave a comment on what topics/questions you find interesting, and shoot me a dm.

https://redd.it/e75tun
@r_devops

Settings storage

I need a new strategy for storing (and retrieving) of settings files. I have a bunch of apps that need access to some settings files, this files are shared across some apps. Then there is also my dev environment locally that needs access to this settings to (but with different values eg: local db instead of prod db). I am using AWS for all the devops things so I am thinking to have the settings files in a S3 bucket so that the apps can download the files and use them. Is there a better way of doing this?

https://redd.it/e6zcpu
@r_devops

Exhausted NFS, need small scale high perf storage

I'm using NFS on an ec2 which is 1.5tb in size
There are a few servers (usually around 15-20) that write various files in many folders at a high rate)

I need a replacement for the NFS server because io and cpu can't take it
So a few questions:

1. AWS EFS, will it yield better results than NFS? Never tried it
2. Thinking of GlusterFS (through I had strange issues with it in the past)
3. Maybe Ceph ? (Never tried it)

I don't need scale, just make it work fast and reliable

Your thoughts?

https://redd.it/e6wgtp
@r_devops

Docker socket authorization best practices

Hi all!
My original question was going to be "Is using HTTPS to connect to the docker daemon with mTLS from a container running on that host more secure than simply mounting `/var/run/docker.sock`"
I answered my own question after a quick search and found https://docs.docker.com/engine/extend/plugins_authorization/
Which states:
> Docker’s out-of-the-box authorization model is all or nothing. Any user with permission to access the Docker daemon can run any Docker client command. The same is true for callers using Docker’s Engine API to contact the daemon.

So now I want to know:
How have you used and configured authorization plugins? Under what scenarios does it make sense to configure authorization Vs running another VM for applications that need access to a docker socket.
I've started using Zalenium recently and it starts sibling containers for running browsers in.
If a malicious Zalenium image got pulled, it would have complete access to docker. This of course is true for any container you're mounting the docker socket into (or presumably even if using a mTLS connection from container to host)
Would love to hear thoughts on this, thanks!

https://redd.it/e6v6df
@r_devops

Learn Containerization fundamentals with examples

Hello,

I'm a recent university graduate and started working as DevOps engineer recently. I'd like to widen my skillset to help my team better. I've had zero exposure to any DevOps tech that's used in the industry and would like to learn it now.
I'm a average with my tech skills and know basics of programming and some OS concepts. We use azure and i know few things about the platform now. I would like to learn about Containerization (mainly docker) and later touch upon kubernetes. Any resources which would help me get started. I'm not a very technical person so I'm looking for a resource which covers basics. Hands on labs would be great.

Thanks in advance.

https://redd.it/e7awob
@r_devops

Unit testing Prometheus alerts

Is there a way to test out an alert config for possible false positives? Currently, I have a bash script using promtool to test the expression with past data to see how many times the alert would be triggered. I want to build this into a PRB, but just wondering if there is another way to go about this?

https://redd.it/e79edh
@r_devops

Phoenix Project IRL

The catch here is that we dont have a Brent. Can you imagine if in the middle of the book Brent up and quit the company? There would be no book. The story would have ended there with the company being split up and sold. Thats the predicament we are in. Two of our top guys who happen to be (used to be!) contractors to our company and have managed a massive system for 16+ years and managed to hoard all of the technical know hows suddenly quit.

https://redd.it/e78psf
@r_devops

Artifactory and Python Wheels

Hi all. I use alpine-based images for most of my deployed images. This is great due to smaller size and usually smaller attack surface. The major down-side, from a python perspective, is that the binary wheels that are stored on PyPi are usually not compatible with MUSL based libraries (non-GLIBC) and need to be built from scratch for every pip install. I am already using some caching tricks and multi-stage builds in docker to help a little bit...but...I would like to use artifactory to make this process less painful.

​

I am struggling to understand the process with artifactory involved. FWIW, I use poetry for locking my packages. The general flow would be something like

* `poetry install`
* <somehow push back the built wheels to my artifactory instance>
* update the lock with hashes of the built wheels??

Theoretically any future projects with the latest hashes would be able to use these prebuilt wheels. Can anyone help walk me through how this should actually be accomplished for stage two and three of my steps above?

https://redd.it/e6ibif
@r_devops

Exporting JFrog Artifactory Bundles?

We're looking at Artifactory for solving a problem related to moving containers between completely isolated environments. The bundles feature is appealing, but I haven't found a way to Import/Export bundles, just entire systems and repositories. Is there a way that just isn't in the documentation I'm looking at or are the only two export levels system and repository?

https://redd.it/e6jv4k
@r_devops