How to Build a Platform Team

Hello all,

I'd like to share this [blog](https://lightstep.com/blog/how-to-build-a-platform-team/) (that's part of a series) on what I've seen work and not work when trying to move to "platform" based devops. There are some very different parts of the lifecycle of building a team and ways that I've seen things go wrong.

Does this match others' experience?

What might be missing?

What tradeoffs do you all see in platform vs less centralized approaches?

Best,

1mentat

https://redd.it/dlmj7w
@r_devops

Looking for some advice!

I have recently changed job and moved into a DevOps role. The role itself is fine and it’s everything that I expected. However, this new company has a separate department for networking (NOC), Part of application sits on top of their network which we have no control over. For context the application is VOIP related so it’s very dependent on a stable network. It is also has issues with NAT, else I would simply move the whole stack into the cloud to resolve the issue but that’s simply not possible at this time.

The problem is they continue to have issues with packet loss and outages but our team seems to get most of the blame (because all regular people see is the application failing, despite it normally being the networks not being set up correctly).

They don’t seem to care about the trouble it causes our department. They also outright refuse to move with the times. A lot of there work is repeatable and could be easily done with ansible but they would rather do everything manually but then complain they never have time setup monitoring for their equipment etc...

Just wondering how you deal with this sort of situation and if anyone has experienced this sort thing before?

https://redd.it/dlvgx6
@r_devops

Starting with terraform

Hi a Jr Deployment engineer here, i am starting to use terraform with aws and some gcp for IaaC can you give me some advices or materials to start and get the right knowledge in little time? Thanks.

https://redd.it/dljdvt
@r_devops

terraform vs ansible question: ssh key management for a large number of EC2 servers

hi guys,
I'm new to ansible and terraform.
we have to frequently manage a large number of EC2 servers for ssh - each server may have a different set of users. I know what is the current set of users i want there (meaning - i may have had a set of users there yesterday...but today there's a different set of users. so delete the ones that are no longer needed).
In a lot of ways it seems like what Terraform was designed for, but i cant find any information about users/ssh creation through terraform. Everyone says this is for terraform.

But I cant seem to find any terraform documentation around "desired final state" for users/ssh.

Any help/advice/sample code would be much appreciated

https://redd.it/dlzxvu
@r_devops

InSpec profile organization with Ansible

I work at a company whose software is primarily deployed on-premise by our customers. We are transitioning to a cloud offering that will be deployed and managed by our team. Our software consists of a number of components installed on Windows compute instances. The number of machines in each customer deployment vary greatly from customer to customer.

To enable deployment of our software in the cloud, we have created a number of Ansible roles. These roles are then collected into a couple playbooks and run against a custom inventory file for each customer. We are really happy with how this is working out for us.

The next step in our cloud evolution is to audit these deployments with InSpec. We are struggling with deciding how to organize the InSpec profiles. One option we played with was to place each InSpec profile in the tests directory of the corresponding Ansible role. We came across an Ansible role that would actually run the InSpec profile. The thought was to have a task at the end of each role that runs its own InSpec profile for validation.

This approach would work well as the software is being deployed, but I'm not sure how we would re-run these profiles after (we want to regularly run the InSpec profiles for compliance and auditing purposes). I can't just re-run all the Ansible automation as its not all idempotent, and I don't believe it can all easily be made idempotent.

Any thoughts on this problem? How do you organize your InSpec profiles?

https://redd.it/dlkoeh
@r_devops

They're hiring. But do they want me, or just my code? The big issue with DevOps hiring.

This is the second time this has happened to me.

For the purposes of confidentiality, I will refer to this as CompanyX. I will also iterate that this has happened twice now.

As part of a DevOps position, I am being asked to complete a technical exercise.

That sounds sensible to me, but let me cut straight to the point - I believe that the gravity and sheer volume of the exercise leads me to believe that the DevOps role is being used as a pawn for CompanyX to obtain a technical deployment solution for their technology. Therefore, I believe that they are not actually hiring, they’re looking for code solutions.

In more depth, the technical exercise is divided into 3 main sections of requirements with around 15 sub tasks. In the shortest summary possible this involves:

* a self-contained modularisable docker image build: obfuscated, cyphonized, minified Flask-App for \*TWO\* generic GitHub Python Flask examples which is then uploaded.
* a docker-compose solution for multiple containers, exposing relevant ports for connectivity, with best practices that include volume exposure,
* a deployment of the generated image as a load balanced cluster with workers. (I would use K8), with a provisioned port. Handling extensive load of requests and automatically spinning up and down containers.
* A continuous integration pipeline with Travis to support the docker image, suitable for deployment on AWS.
* Other python scripts for optimising the workflow.
* Testing the correct balancing on a simulated load with an entire testing phase (python scripts to check)

Im not looking for solutions or correction on this, so don’t dive into what was asked of me.

But I can tell you for sure, as a seasoned DevOps engineer with a masters in computer science, the above technical exercise is not something you can just stroll through in a few hours or days. It requires a lot of requirements engineering, designing and optimisation. It also requires a lot of questions too. I’ve worked clients before on a contractual basis and full-time in a role where this sort of task would take between 4-6 months to do properly, and I would charge around £10k+ for reasons like it collides so many different technologies together and needs to be scalable for environments like the cloud. Not trivial. And before you say that DevOps roles require more than just this - maybe, but not a huge amount more than maintenance and optimisation.

Sure, as a company you should ask technical questions like this. But not on this scale. This is essentially completing the job, before you started the job. And doing it for free. If we take the perspective of CompanyX, its actually quite smart to think of a list of requirements for your tech stack, and put it out as a job interview exercise, get the solution back, integrate their solution, and then reject the candidate. But its highly immoral.

I want to know from you guys - any relevant experiences with companies’ technical exercises as such?

Does anyone have a similar story?

Does anyone have a response for something like this?

How would you tackle this?

Thankssss a lot in advance everyone :)

https://redd.it/dlysgj
@r_devops

Security tools for DevOps ecosystem

In company I work for we have multiple devops teams which work using different tool sets. Trying to ensure security of such environment was horror so I have decided to create something like hub for security tools(scanners and audit tools) which is ment to be easily accessible for any kind of CICD.
Functionalities so far:
- service discovery (integration with openstack and GCE)
- SAST testing ( integration with micro focus fortify and checkmarx)
- DAST testing (integration with acunetix and w3af)
- network scanners ( integration with Nessus 6, openvas, nexpose)
- CIS compliance (for docker and k8s)

Standard input:
- possibility to run any kind of test via API - doesn’t matter which tools are being used by security, API is the same, ( for example running SAST on every merge request or DAST during integration testing)
Standard output:
- security quality gateway, deploy or no deploy dependent on the result of provided tests.

I am really interested if anyone else would be interested in such tool, or what else should it implement so You would be interested?

https://redd.it/dm2ekm
@r_devops

A task for aspiring devops engineers

You got a developer who needs to create, build and deploy an application to an infrastructure, but doesn't want to deal with systems (no time, other priorities, should never really touch systems without experience with them).

Your job is to automate deployment of infrastructure required for creating, building and deploying the project, let's start:

This is your developer source code:

https://github.com/spring-projects/spring-petclinic

What do you do with it? Just build locally and copy binaries to some fresh VM? Let's do it the right way:

----

First - let's create a simple infrastructure:

We will need a set of Ansible playbooks for following components (don't use Ansible Galaxy directly, it's bad practice, only as inspiration):

* Gitlab

* Jenkins (from which you have to authenticate to Gitlab)

* Grafana and Prometheus

* One application dev server

* Two application production servers

* Load balancer

Obviously we are missing secrets management, authentication services, gathering logs and so on, but it was suppose to be simple.

Goal is to deploy whole thing using a single playbook. Prometheus is suppose to grab metrics from all VMs, load balancer and our application.

Now that we have our infrastructure, let's get into the meat:

* Create Gitlab project and clone Spring Petclinic repository to it
* Create two new branches: dev and prod (usually we would have staging too, but we are keeping it simple af)
* Create two Jenkins pipelines which do the following on each commit - fetches code from `dev` or `prod` branches, builds our code, runs tests, deploys to relevant dev or prod servers

When a developer pushes changes to branch `dev`, Jenkins should automatically build, test and deploy to `dev`where developer can further test the application before merging the code into `prod` from which same process takes place, but results in a production deployment.

----

This is super simplified and you can choose different technologies, but that's basically what devops is about - glueing systems and development world into something coherent and automated.

P.S. Feel free to post ideas for this task or entire new tasks, but let's keep it simple.

https://redd.it/dm3mdj
@r_devops

Managing configuration files inside of a Kubernetes cluster

First, I'm sorry if it sounds confusing, wrong or really simple, I'm still at the beginning of my devops role.

​

I'm starting to work with Kubernetes to migrate a Java application to a pod (as a first step).

My application:

Currently this application obtain it's configurations from a couple of files: a application.properties (1 per app) and client-name.properties (1 or more per app). These are simple files that have multiple configurations such as which address/port should a client use or SMTP configurations, all of the are key/value configs.

From what I saw there is configuration management such as ConfigMaps and Secrets to change simple configs inside of a pod in a key/value fashion. My problem is that I can have n clients per app and n\*(client-name.properties) files, therefore it seems that using a ConfigMap ends up as a hassle (if it's even possible).

​

Is there a (better) way to manage this multiple configurations that my app uses today?

​

Any help and comments are appreciated.

Thanks.

https://redd.it/dm4ozl
@r_devops

How to transition to IT Management?

I have about 4 years of experience and titles have ranged from Systems Engineer to DevOps Engineer. Basically doing the same shit in all of the jobs I've had after college - maintaining cloud infrastructure, writing automation scripts, building CI/CD pipelines, developing APIs and internal web applications, etc.

It bothers me that my managers don't know much about tech an are pulling in at-least $30-$50K more than me. Some of them don't even have degrees, and the only one that does is from a random no-name school. Even the helpdesk manager here makes slightly more than me.

Why do I bust my ass off constantly learning new tech to make less than a fucking helpdesk manager? And why do companies even pay helpdesk managers six figures? Do they not realize anyone can do that type of work? I'm sitting here going through hours of kubernetes training to prepare for an upcoming project and I feel like it'd be easier to just coast by in a management job where you don't have to know a god damned thing.

Looking for tips. Thanks.

https://redd.it/dm6p9k
@r_devops

Transition of System admin to DevOps easy?

I am a system administrator (3 years of experience) who works on building VDIs (Virtual Desktop infrastructure) and manages vCenters to VM Troubleshootings and etc... I have very basic knowledge on scripting.

I want to move into DevOps and wants to know how difficult would it be for a person with just 3 years of experience surviving in DevOps team? I am very passionate yet lazy at some point. Any advise from you guys will def gonna help me in acheiving my target.

https://redd.it/dmbiwo
@r_devops

How to control the state of databases in between API Tests?

I'm somewhat new to Automated API Testing and have been experimenting with doing some API (functional?/regression?) tests in SoapUI and Katalon against multiple microservices, along with using assertions to check for a valid response. However, I'm not sure how to handle the state of the database before and after a test is run.

For example, if I upload something through a POST to an endpoint and it's supposed to create 5 entries, then I can assert that those 5 were created by calling another endpoint with a GET and checking whether 5 objects are returned in the array. However, if I run that same test case again then I'll receive 10 entries (since the old 5 entries still exist) but my old assertion will fail since it's expecting 5.

I guess I can try to use DELETE calls to manually clear the state before a run but then I have to first track down the ids of what I want to delete. Or I can have some sql that runs to make sure those records are deleted but I run into the same issue, and both options feel janky. The only other option seems to be to restore the databases from dump files but that limits one to only running such tests on a test environment, which I guess might be a good thing? However, I haven't seen such support in SoapUI or Katalon for restoring database so far.

What is the best way to handle the state of databases behind microservices while running API tests whether it's on a local machine or in the cloud?

https://redd.it/dmasni
@r_devops

What do you use to backup databases in a deterministic way? Are there any off-the-shelf containers?

I'm looking for a solution to backup relational databases (only MySQL, MariaDB, Postgres). We want to run the databases in containers (database folder bind mount from host) so that we can manage everything with Docker Swarm (Kubernetes is too complex and not needed). The risks of running databases in containers require a good backup.

* Versioned backups: Most of our databases are small, so its possible and wanted to keep multiple versions (one per day in last week, one per week last year, one per month etc.).

* Report/GUI: It would be good to easily check if backups have been done without problem.

* 'Configurable by Code': We want integrate it into our standard CI/CD pipeline, so it would be good to configure it with a text file that can be managed in git. We want to update the configuration whenever this file changes, e.g. by restarting a container.

Ideas:

1. Solution (Custom solution): Run a cron job with mysqldump etc. and delete old versions that are not needed any more. Disadvantage: Needs to be thoroughly tested. Another thing I have to support, bug fix.

2. Solution (Simple image from Dockerhubg): I haven't found a good image that does the thing described above on Dockerhub or github. Can you recommend something? Is there something with a GUI or report to check if backup has been done?

3. Solution (Payed Enterprise product): This is a common problem. There must be payed fancy products. Can recommend something?

Your thoughts?

https://redd.it/dmbx2m
@r_devops

Deploying software that depends on a virtual machine for non-tech savvy users

I am developing a piece of software which consists of a Node.js application which controls a virtual machine that runs Android-x86. Also there are some pieces of the Android SDK like ADB the software depends on.

Currently there is no smart way of deployment whatsoever at all. Everything has to be installed manually. Depending on the system I also get issues regarding networking and so on from time to time. I also created another question [here](https://www.reddit.com/r/devops/comments/dja9zk/deploying_software_that_depends_on_a_virtual/) where I first thought about using Docker.

While researching if I can solve this using Docker I came up with solutions that require Kubernetes as well. I want to deploy the software to non-tech savvy users as well. So I cannot afford to have Docker and Kubernetes as dependencies due to the complexity of the installation process.

Is there any technology that I am not aware of that can simplify the setup process for the end user? The perfect scenario would be a virtual machine or container containing the whole environment that can somehow be converted to a single binary which can be installed/executed.

https://redd.it/dm60io
@r_devops

SecretHub opinions and success stories?

Hi!
I am looking for secrets management SaaS and I recently stumbled [SecretHub](https://secrethub.io/). Their Terraform integration looks extremely easy, which is really important to me.

I was wondering if anybody used them in production and, if so, how the find the product so far. It seems to me like the offered features are very basic (nothing of the sort of HashiCorp Vault), but that's not necessarily a problem for small businesses. Any opinions? Cheers

https://redd.it/dm4rps
@r_devops

On-call role in the chain of handling system failures: simple ops only?

I've shared with my team this mental model about the escalation and de-escalation path for handling system failures and there were different opinions about the role of the on-call in it. I wonder what do you guys think?

In essence, these are the stages of where/who handles system failures:

1. software has fault-tolerance, i.e. handles failure without effect on the clients
2. automated operations, i.e. automated scripts listening to health checks, supervisor programs, etc.; no human needed
3. on-call, i.e. humans, but they use very simple instructions and runbooks; no real heavy debugging, etc.
4. expert help, i.e. humans who deeply understand the system, can do heavy debugging, have access to sources, etc.

There were different opinions in the team on whether on-call is expected to be limited to very simple runbooks/checks. Ie. ops that are simple, but for some reason were not automated (maybe some have human judgment that is hard to automate or there was never time to automate).

For the context, this discussion was in a data infra team, which has both platform engineers, SREs; the team is setting up OSS software and also develops their own, mostly complex, distributed systems. Every engineer on the team participates in the on-call.

What do you guys think? Is it beneficial to limit expectation of on-call to very simple ops and that they call/delegate to particular expert in the team to handle other cases?

More details in the post, but most info is above, no need to read. [https://medium.com/@abaranau/mental-model-for-failure-handling-on-call-run-books-etc-719240ed6cc8](https://medium.com/@abaranau/mental-model-for-failure-handling-on-call-run-books-etc-719240ed6cc8)

https://redd.it/dm27c2
@r_devops

Questions on Cloud lock-in .

I'm about to start setting up cloud infrastructure, I wanted to hear more about cloud vender lock-in and how serious can it get?
Any past experience that you guys had to deal with that you might wanna share?
I'm looking into multi cloud setup, is it advisable?

https://redd.it/dm0vi7
@r_devops

CI/CD with AWS lambda and python

Hi DevOps,

How can i setup CI/CD with AWS lambda and Python.Requirement is - Need to run test case which written in python, once the test case is successful, It has deploy the code.

​

New to CI/CD Please help. Thanks

https://redd.it/dmiqz8
@r_devops

Frustration with lack of feedback after interviews

​

Let me start by saying that I am pretty happy with my current job. I have a good salary and my work has some bonus structure, pays all my remote coworking expenses and good medical benefits for my family. I'm 35 and have tons of work experience coming from tech support, sys-admin, some development work and ultimately I am now a Sr. SRE at a \~120 person startup. I didn't go to college but I've been mostly working in the field since age 16. I've been a linux guy since redhat 4.2 in 1997-8.

​

I've never had much trouble finding a job when I needed one in the past, and usually only interviewed at one or two places before taking a job. But after some layoffs occurred at my company earlier this year I thought I should probably start interviewing in case something great comes along, perhaps finding something as good as my current job or better without potential "running out of runway" issues.

​

I've done 3 rounds interviews and have been passed on by all of them so far. I know that isn't a ton, but it's still a bit disheartening. I don't even need the job, but I'm wondering wtf is going on.

​

For all 3 interviews I have had great phone screens with recruiter and then with a technical person or hiring manager. One company gave me a take-home project which I thought was unreasonable as timeboxed to 4 hours. I committed what I had in 4 hours and then worked the rest of the weekend on it and committed that as well.

​

Another company had me do a 1 hour videocall wherein I was given a very broken shell script (about 15 flaws in the script), the end result of which was a running microservice container then some problems in AWS with security groups and load balancers to make the service reachable. I was able to complete the excercise in the timeboxed window, perhaps even a few minutes early, but the guy kept feeding me hints that I probably didn't even need if he would just give me a few more minutes.

​

Another (huge) company I had what I thought was a great call (technical and soft skills)

​

All 3 companies 'passed' on whatever the final interview stage was (usually a full-day onsite etc), and none of them would provide me any useful feedback whatsoever like it was against their policy to do so.

​

I mean really, just give it straight. You didn't fix the bash problems fast enough, we are looking for someone with experience with X, or "we already knew we were hiring my cousin".

​

​

It's pretty frustrating out there. I'm just thankful for the job I have and hope it lasts for a long time, especially if we really have a recession coming down the pike.

​

That's it ..

https://redd.it/dmmjq8
@r_devops

What kind of experience did you have before you landed your DevOps job?

What kind of work experience did you have before you landed your first DevOps job? What type of work did you do prior, how many years experience, did you have any certifications?

I'm thinking of moving to DevOps after a few years working as a developer but not sure if it's even possible.

Thanks

https://redd.it/dmofx8
@r_devops

Vault: 500 errors connecting to Consul backend, but everything works fine?

Hi everyone. Sorry if this isn't the correct place to post this, but /r/hashicorp is a ghost town and I'm quite stuck.

I'm deploying brand new Consul and Vault clusters onto GKE using the vault-helm and consul-helm Helm charts. I'm able to get the two services up and I have 5 replicas of each running in the cluster.

I'm using Consul as my backend to Vault. Once Vault is unsealed, I'm able to log in and create secrets. They are successfully stored in the backend and if I log into other nodes they can all see the secrets that I created on the first node. I have auto-unseal configured and the nodes unseal themselves as they should. Everything appears to work just fine.

The problem is that, if I run "kubectl logs <vault pod name>", I get hundreds of these after a few minutes:

[WARN] storage.consul: check unable to talk with Consul backend: error="Unexpected response code: 500 (Unknown check "vault:10.26.0.131:8200:vault-sealed-check")"

Vault is not sealed; all of the nodes have auto-unseal configured using GCP KMS, which appears to work perfectly fine. Vault and Consul run in the same GKE cluster, on the same IP subnet, and can ping each other. I can see that Vault data is being saved in Consul, so I'm not sure what the error is trying to tell me. It doesn't seem to make a whole lot of sense.

Here's the "HA" section of my Vault config in the vault-helm Helm chart:

ha:
enabled: true
replicas: 5

# config is a raw string of default configuration when using a Stateful
# deployment. Default is to use a Consul for its HA storage backend.
# This should be HCL.
config: |
ui = true

listener "tcp" {
address = "[::]:8200"
cluster_address = "[::]:8201"
tls_disable = 0
tls_cert_file = "/vault/userconfig/cert-file/cert.crt"
tls_key_file = "/vault/userconfig/cert-key/cert.key"
}
storage "consul" {
address = "https://consul-consul-server:8500" (this is the name of the k8s service for Consul)
service = "vault"
path = "apps/vault/"
scheme = "http"
token = "12345678-a1b2-c3d4-e5f6-123456789012"
}

# Example configuration for using auto-unseal, using Google Cloud KMS. The
# GKMS keys must already exist, and the cluster must have a service account
# that is authorized to access GCP KMS.
seal "gcpckms" {
credentials = "/vault/userconfig/vault-auto-auth/vault-auto-auth.json"
project = "gcp-project-name"
region = "us-central1"
key_ring = "gcp-keyring-name"
crypto_key = "gcp-key-name"
}

# A disruption budget limits the number of pods of a replicated application
# that are down simultaneously from voluntary disruptions
disruptionBudget:
enabled: true

# maxUnavailable will default to (n/2)-1 where n is the number of
# replicas. If you'd like a custom value, you can specify an override here.
maxUnavailable: null

# Definition of the serviceaccount used to run Vault.
serviceaccount:
annotations: {}

# mlock prevents memory from being swapped to disk. If swap is enabled this should
# be true.
mlock:
enabled: true

Any ideas or suggestions would be greatly appreciated! Happy to gather any additional config that might help. Thanks in advance.

https://redd.it/dmh7k9
@r_devops