google/schedviz: A tool for gathering and visualizing kernel scheduling traces on Linux machines. Built with Go
https://github.com/google/schedviz

https://redd.it/dlltza
@r_devops

Good pairing between Terraform and Docker, for the non-ops person?

I'm not an ops by trade. I do however have a few side projects that require servers, domains, etc.

I was going to deploy it by hand, and my project is small enough in scale that doing it by hand would be *totally fine*. With that said, I often find that over long periods of time "by hand" results in a bunch of servers that I forget exactly how I setup. So, I'm looking to not have it be by hand.

My original plan was to manually deploy Docker on a machine to handle configuring the application environment; as I have a good amount of experience with Docker, and am not a big fan of Ansible _(though I've never tried Chef/etc)_. So my goal is to stick with Docker.

Terraform seems like a good tool for provisioning so I plan on using that to manage creation of my infra. However that leaves a bit of a gap; what would I use to configure the machines that Terraform provisions?

I'm in a bit of a weird place. I do not have a private Docker registry, nor did I plan to make one. So to bridge the gap between Terraform and my application running in a container, will I still need to use something like Ansible to configure the machines and deploy new containers?

Just seeking some advice from seasoned folks. My goal is simple. Docker, to me is simple. Terraform looks elegant, and simple. In between is where I'm a bit stuck at what will be simple; and not have me confused when I come back to it in 6 months. Advice appreciated, thank you.

https://redd.it/dl8jrv
@r_devops

Is "infrastructure" a two-pizza team?

Dear Father Bezos,

I recently converted to the Religion of Microservices, and I endeavor to follow all the commandments: microservices-shall-own-their-data, microservices-shall-own-end-to-end-delivery-of-their-business-domain, "You build it; you run it," etc. As a Believer, I know it's bad to create a "database team" and an "application team" because then I've structured the two-pizza teams based on function rather than business domain. The teams ought to include DevOps/Operations pros who know how to deploy and operate the service, just as they include developers and testers, which in other religions work in separate departments.

Selah.

Monitoring, CI/CD, Kubernetes, DB admin... as I add up everything these teams must do, I find it increasingly difficult to see how these small teams can do everything we need a production service to do. Terraform makes Infra something developers can incorporate, but getting all the routing and permissions and CI/CD integration right... I can see why some have left the Faith and keep a dedicated Operations team for these things. At what point do those things become a business domain in themselves, and how do I avoid that becoming a functional team all over again? To take just one example: Kubernetes isn't easy; won't the teams have a hard time managing it themselves?

When I ask these questions, I hear a lot of, "there's no right answer," and "it totally depends on the size and shape of the company." I suppose that's true. I guess what I'm asking, Father, is how should I go about drawing these team+service boundaries? Specifically when it comes to things like infrastructure, Kubernetes, load balancing, logging, monitoring, CI/CD, how can each team be expected to do all those things well?

Tell the Saints I said, "hi."

Sincerely,
A True Believer

https://redd.it/dl4ass
@r_devops

Vanilla Kubernetes is completely over my head...

Anyone else feel the same way? Like, I can use EKS or GCE all day long, but vanilla kubernetes built more of less from scratch? Just so fucking lost.

I'm honestly strongly considering switching careers. I started out as a Linux admin where the most difficult thing was trying to fix a package that failed to install, and I've failed upwards to the point that I've "risen to incompetence" as they say.

Might be time to go to culinary school.

https://redd.it/dlnlo4
@r_devops

Anyone here worked on Apach Nifi or zookeeper

I have a question
Can zookeeper fail over when there are only 2 nodes?

Leader and worker

To test failovers, I close down the Server that has been selected as Cluster Coordinator

In this case, zookeeper should automatically elect the remaining one server as leader. But it keeps failing and goes into continuous loop of trying to connect to the first server

Zookeeper Logs in Server 2 when leader (Server 1) went down:


-----—------------------
2019-10-22 18:44:01,135 [myid:2] - WARN [NIOWorkerThread-2:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
2019-10-22 18:44:02,925 [myid:2] - WARN [NIOWorkerThread-3:NIOServerCnxn@370] - Exception causing close of session 0x0: ZooKeeperServer not running
2019-10-22 18:44:03,320 [myid:2] - WARN [QuorumPeer[myid=2](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):QuorumCnxManager@677] -
Cannot open channel to 1 at election address ec2-server-1.compute-1.amazonaws.com/172.xx.x.x:3888
java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)

-----—------------------


Server 2 Config files:

zoo.cfg


-----—------------------
tickTime=2000
initLimit=5
syncLimit=2
dataDir=/home/ec2-user/zookeeper
clientPort=2181
server.1=ec2-server-1.compute-1.amazonaws.com:2888:3888
server.2=0.0.0.0:2888:3888

nifi.properties

nifi.cluster.is.node=true
nifi.cluster.node.address=ec2-server-2.compute-1.amazonaws.com
nifi.cluster.node.protocol.port=8082
nifi.cluster.flow.election.max.wait.time=2 mins
nifi.cluster.flow.election.max.candidates=1

# zookeeper properties, used for cluster management #
nifi.zookeeper.connect.string=localhost:2181
nifi.zookeeper.root.node=/nifi

-----—------------------

Server 1 Config files:

zoo.cfg

-----—------------------

tickTime=2000
initLimit=5
syncLimit=2
dataDir=/home/ec2-user/zookeeper
clientPort=2181
server.1=0.0.0.0:2888:3888
server.2=ec2-server-2.compute-1.amazonaws.com:2888:3888

nifi.properties

nifi.cluster.is.node=true
nifi.cluster.node.address=ec2-server-1.compute-1.amazonaws.com
nifi.cluster.node.protocol.port=8082
nifi.cluster.flow.election.max.wait.time=2 mins
nifi.cluster.flow.election.max.candidates=1

# zookeeper properties, used for cluster management #
nifi.zookeeper.connect.string=localhost:2181
nifi.zookeeper.root.node=/nifi

-----—------------------


What am I doing wrong?

https://redd.it/dlmptz
@r_devops

How to Build a Platform Team

Hello all,

I'd like to share this [blog](https://lightstep.com/blog/how-to-build-a-platform-team/) (that's part of a series) on what I've seen work and not work when trying to move to "platform" based devops. There are some very different parts of the lifecycle of building a team and ways that I've seen things go wrong.

Does this match others' experience?

What might be missing?

What tradeoffs do you all see in platform vs less centralized approaches?

Best,

1mentat

https://redd.it/dlmj7w
@r_devops

Looking for some advice!

I have recently changed job and moved into a DevOps role. The role itself is fine and it’s everything that I expected. However, this new company has a separate department for networking (NOC), Part of application sits on top of their network which we have no control over. For context the application is VOIP related so it’s very dependent on a stable network. It is also has issues with NAT, else I would simply move the whole stack into the cloud to resolve the issue but that’s simply not possible at this time.

The problem is they continue to have issues with packet loss and outages but our team seems to get most of the blame (because all regular people see is the application failing, despite it normally being the networks not being set up correctly).

They don’t seem to care about the trouble it causes our department. They also outright refuse to move with the times. A lot of there work is repeatable and could be easily done with ansible but they would rather do everything manually but then complain they never have time setup monitoring for their equipment etc...

Just wondering how you deal with this sort of situation and if anyone has experienced this sort thing before?

https://redd.it/dlvgx6
@r_devops

Starting with terraform

Hi a Jr Deployment engineer here, i am starting to use terraform with aws and some gcp for IaaC can you give me some advices or materials to start and get the right knowledge in little time? Thanks.

https://redd.it/dljdvt
@r_devops

terraform vs ansible question: ssh key management for a large number of EC2 servers

hi guys,
I'm new to ansible and terraform.
we have to frequently manage a large number of EC2 servers for ssh - each server may have a different set of users. I know what is the current set of users i want there (meaning - i may have had a set of users there yesterday...but today there's a different set of users. so delete the ones that are no longer needed).
In a lot of ways it seems like what Terraform was designed for, but i cant find any information about users/ssh creation through terraform. Everyone says this is for terraform.

But I cant seem to find any terraform documentation around "desired final state" for users/ssh.

Any help/advice/sample code would be much appreciated

https://redd.it/dlzxvu
@r_devops

InSpec profile organization with Ansible

I work at a company whose software is primarily deployed on-premise by our customers. We are transitioning to a cloud offering that will be deployed and managed by our team. Our software consists of a number of components installed on Windows compute instances. The number of machines in each customer deployment vary greatly from customer to customer.

To enable deployment of our software in the cloud, we have created a number of Ansible roles. These roles are then collected into a couple playbooks and run against a custom inventory file for each customer. We are really happy with how this is working out for us.

The next step in our cloud evolution is to audit these deployments with InSpec. We are struggling with deciding how to organize the InSpec profiles. One option we played with was to place each InSpec profile in the tests directory of the corresponding Ansible role. We came across an Ansible role that would actually run the InSpec profile. The thought was to have a task at the end of each role that runs its own InSpec profile for validation.

This approach would work well as the software is being deployed, but I'm not sure how we would re-run these profiles after (we want to regularly run the InSpec profiles for compliance and auditing purposes). I can't just re-run all the Ansible automation as its not all idempotent, and I don't believe it can all easily be made idempotent.

Any thoughts on this problem? How do you organize your InSpec profiles?

https://redd.it/dlkoeh
@r_devops

They're hiring. But do they want me, or just my code? The big issue with DevOps hiring.

This is the second time this has happened to me.

For the purposes of confidentiality, I will refer to this as CompanyX. I will also iterate that this has happened twice now.

As part of a DevOps position, I am being asked to complete a technical exercise.

That sounds sensible to me, but let me cut straight to the point - I believe that the gravity and sheer volume of the exercise leads me to believe that the DevOps role is being used as a pawn for CompanyX to obtain a technical deployment solution for their technology. Therefore, I believe that they are not actually hiring, they’re looking for code solutions.

In more depth, the technical exercise is divided into 3 main sections of requirements with around 15 sub tasks. In the shortest summary possible this involves:

* a self-contained modularisable docker image build: obfuscated, cyphonized, minified Flask-App for \*TWO\* generic GitHub Python Flask examples which is then uploaded.
* a docker-compose solution for multiple containers, exposing relevant ports for connectivity, with best practices that include volume exposure,
* a deployment of the generated image as a load balanced cluster with workers. (I would use K8), with a provisioned port. Handling extensive load of requests and automatically spinning up and down containers.
* A continuous integration pipeline with Travis to support the docker image, suitable for deployment on AWS.
* Other python scripts for optimising the workflow.
* Testing the correct balancing on a simulated load with an entire testing phase (python scripts to check)

Im not looking for solutions or correction on this, so don’t dive into what was asked of me.

But I can tell you for sure, as a seasoned DevOps engineer with a masters in computer science, the above technical exercise is not something you can just stroll through in a few hours or days. It requires a lot of requirements engineering, designing and optimisation. It also requires a lot of questions too. I’ve worked clients before on a contractual basis and full-time in a role where this sort of task would take between 4-6 months to do properly, and I would charge around £10k+ for reasons like it collides so many different technologies together and needs to be scalable for environments like the cloud. Not trivial. And before you say that DevOps roles require more than just this - maybe, but not a huge amount more than maintenance and optimisation.

Sure, as a company you should ask technical questions like this. But not on this scale. This is essentially completing the job, before you started the job. And doing it for free. If we take the perspective of CompanyX, its actually quite smart to think of a list of requirements for your tech stack, and put it out as a job interview exercise, get the solution back, integrate their solution, and then reject the candidate. But its highly immoral.

I want to know from you guys - any relevant experiences with companies’ technical exercises as such?

Does anyone have a similar story?

Does anyone have a response for something like this?

How would you tackle this?

Thankssss a lot in advance everyone :)

https://redd.it/dlysgj
@r_devops

Security tools for DevOps ecosystem

In company I work for we have multiple devops teams which work using different tool sets. Trying to ensure security of such environment was horror so I have decided to create something like hub for security tools(scanners and audit tools) which is ment to be easily accessible for any kind of CICD.
Functionalities so far:
- service discovery (integration with openstack and GCE)
- SAST testing ( integration with micro focus fortify and checkmarx)
- DAST testing (integration with acunetix and w3af)
- network scanners ( integration with Nessus 6, openvas, nexpose)
- CIS compliance (for docker and k8s)

Standard input:
- possibility to run any kind of test via API - doesn’t matter which tools are being used by security, API is the same, ( for example running SAST on every merge request or DAST during integration testing)
Standard output:
- security quality gateway, deploy or no deploy dependent on the result of provided tests.

I am really interested if anyone else would be interested in such tool, or what else should it implement so You would be interested?

https://redd.it/dm2ekm
@r_devops

A task for aspiring devops engineers

You got a developer who needs to create, build and deploy an application to an infrastructure, but doesn't want to deal with systems (no time, other priorities, should never really touch systems without experience with them).

Your job is to automate deployment of infrastructure required for creating, building and deploying the project, let's start:

This is your developer source code:

https://github.com/spring-projects/spring-petclinic

What do you do with it? Just build locally and copy binaries to some fresh VM? Let's do it the right way:

----

First - let's create a simple infrastructure:

We will need a set of Ansible playbooks for following components (don't use Ansible Galaxy directly, it's bad practice, only as inspiration):

* Gitlab

* Jenkins (from which you have to authenticate to Gitlab)

* Grafana and Prometheus

* One application dev server

* Two application production servers

* Load balancer

Obviously we are missing secrets management, authentication services, gathering logs and so on, but it was suppose to be simple.

Goal is to deploy whole thing using a single playbook. Prometheus is suppose to grab metrics from all VMs, load balancer and our application.

Now that we have our infrastructure, let's get into the meat:

* Create Gitlab project and clone Spring Petclinic repository to it
* Create two new branches: dev and prod (usually we would have staging too, but we are keeping it simple af)
* Create two Jenkins pipelines which do the following on each commit - fetches code from `dev` or `prod` branches, builds our code, runs tests, deploys to relevant dev or prod servers

When a developer pushes changes to branch `dev`, Jenkins should automatically build, test and deploy to `dev`where developer can further test the application before merging the code into `prod` from which same process takes place, but results in a production deployment.

----

This is super simplified and you can choose different technologies, but that's basically what devops is about - glueing systems and development world into something coherent and automated.

P.S. Feel free to post ideas for this task or entire new tasks, but let's keep it simple.

https://redd.it/dm3mdj
@r_devops

Managing configuration files inside of a Kubernetes cluster

First, I'm sorry if it sounds confusing, wrong or really simple, I'm still at the beginning of my devops role.

​

I'm starting to work with Kubernetes to migrate a Java application to a pod (as a first step).

My application:

Currently this application obtain it's configurations from a couple of files: a application.properties (1 per app) and client-name.properties (1 or more per app). These are simple files that have multiple configurations such as which address/port should a client use or SMTP configurations, all of the are key/value configs.

From what I saw there is configuration management such as ConfigMaps and Secrets to change simple configs inside of a pod in a key/value fashion. My problem is that I can have n clients per app and n\*(client-name.properties) files, therefore it seems that using a ConfigMap ends up as a hassle (if it's even possible).

​

Is there a (better) way to manage this multiple configurations that my app uses today?

​

Any help and comments are appreciated.

Thanks.

https://redd.it/dm4ozl
@r_devops

How to transition to IT Management?

I have about 4 years of experience and titles have ranged from Systems Engineer to DevOps Engineer. Basically doing the same shit in all of the jobs I've had after college - maintaining cloud infrastructure, writing automation scripts, building CI/CD pipelines, developing APIs and internal web applications, etc.

It bothers me that my managers don't know much about tech an are pulling in at-least $30-$50K more than me. Some of them don't even have degrees, and the only one that does is from a random no-name school. Even the helpdesk manager here makes slightly more than me.

Why do I bust my ass off constantly learning new tech to make less than a fucking helpdesk manager? And why do companies even pay helpdesk managers six figures? Do they not realize anyone can do that type of work? I'm sitting here going through hours of kubernetes training to prepare for an upcoming project and I feel like it'd be easier to just coast by in a management job where you don't have to know a god damned thing.

Looking for tips. Thanks.

https://redd.it/dm6p9k
@r_devops

Transition of System admin to DevOps easy?

I am a system administrator (3 years of experience) who works on building VDIs (Virtual Desktop infrastructure) and manages vCenters to VM Troubleshootings and etc... I have very basic knowledge on scripting.

I want to move into DevOps and wants to know how difficult would it be for a person with just 3 years of experience surviving in DevOps team? I am very passionate yet lazy at some point. Any advise from you guys will def gonna help me in acheiving my target.

https://redd.it/dmbiwo
@r_devops

How to control the state of databases in between API Tests?

I'm somewhat new to Automated API Testing and have been experimenting with doing some API (functional?/regression?) tests in SoapUI and Katalon against multiple microservices, along with using assertions to check for a valid response. However, I'm not sure how to handle the state of the database before and after a test is run.

For example, if I upload something through a POST to an endpoint and it's supposed to create 5 entries, then I can assert that those 5 were created by calling another endpoint with a GET and checking whether 5 objects are returned in the array. However, if I run that same test case again then I'll receive 10 entries (since the old 5 entries still exist) but my old assertion will fail since it's expecting 5.

I guess I can try to use DELETE calls to manually clear the state before a run but then I have to first track down the ids of what I want to delete. Or I can have some sql that runs to make sure those records are deleted but I run into the same issue, and both options feel janky. The only other option seems to be to restore the databases from dump files but that limits one to only running such tests on a test environment, which I guess might be a good thing? However, I haven't seen such support in SoapUI or Katalon for restoring database so far.

What is the best way to handle the state of databases behind microservices while running API tests whether it's on a local machine or in the cloud?

https://redd.it/dmasni
@r_devops

What do you use to backup databases in a deterministic way? Are there any off-the-shelf containers?

I'm looking for a solution to backup relational databases (only MySQL, MariaDB, Postgres). We want to run the databases in containers (database folder bind mount from host) so that we can manage everything with Docker Swarm (Kubernetes is too complex and not needed). The risks of running databases in containers require a good backup.

* Versioned backups: Most of our databases are small, so its possible and wanted to keep multiple versions (one per day in last week, one per week last year, one per month etc.).

* Report/GUI: It would be good to easily check if backups have been done without problem.

* 'Configurable by Code': We want integrate it into our standard CI/CD pipeline, so it would be good to configure it with a text file that can be managed in git. We want to update the configuration whenever this file changes, e.g. by restarting a container.

Ideas:

1. Solution (Custom solution): Run a cron job with mysqldump etc. and delete old versions that are not needed any more. Disadvantage: Needs to be thoroughly tested. Another thing I have to support, bug fix.

2. Solution (Simple image from Dockerhubg): I haven't found a good image that does the thing described above on Dockerhub or github. Can you recommend something? Is there something with a GUI or report to check if backup has been done?

3. Solution (Payed Enterprise product): This is a common problem. There must be payed fancy products. Can recommend something?

Your thoughts?

https://redd.it/dmbx2m
@r_devops

Deploying software that depends on a virtual machine for non-tech savvy users

I am developing a piece of software which consists of a Node.js application which controls a virtual machine that runs Android-x86. Also there are some pieces of the Android SDK like ADB the software depends on.

Currently there is no smart way of deployment whatsoever at all. Everything has to be installed manually. Depending on the system I also get issues regarding networking and so on from time to time. I also created another question [here](https://www.reddit.com/r/devops/comments/dja9zk/deploying_software_that_depends_on_a_virtual/) where I first thought about using Docker.

While researching if I can solve this using Docker I came up with solutions that require Kubernetes as well. I want to deploy the software to non-tech savvy users as well. So I cannot afford to have Docker and Kubernetes as dependencies due to the complexity of the installation process.

Is there any technology that I am not aware of that can simplify the setup process for the end user? The perfect scenario would be a virtual machine or container containing the whole environment that can somehow be converted to a single binary which can be installed/executed.

https://redd.it/dm60io
@r_devops

SecretHub opinions and success stories?

Hi!
I am looking for secrets management SaaS and I recently stumbled [SecretHub](https://secrethub.io/). Their Terraform integration looks extremely easy, which is really important to me.

I was wondering if anybody used them in production and, if so, how the find the product so far. It seems to me like the offered features are very basic (nothing of the sort of HashiCorp Vault), but that's not necessarily a problem for small businesses. Any opinions? Cheers

https://redd.it/dm4rps
@r_devops