How to deal with db migrations in rolling updates on managed instance groups

Hi,

i'm studying for the Google Cloud Professional Architect Certification.
For update deployments, everywhere is recommended (or mandatory) the rolling update feature (on a managed instance group, or on kubernetes pods), so some instances begins to update while others remain on the old version. This is obviously good because you have continuity and no downtime.

But, if I have, let's say, a django (or ruby) application with a managed relational database, and I need to apply a database migration (meaning a change of structure), and without it my new version breaks and with it my old version does not work, how is it possible to deal with it?

How do you keep that continuity?

https://redd.it/drl5yl
@r_devops

New DevOps platform for Game Engines!

Hello fellow humans, we’re launching our CI/CD platform that focus on the game industry get an early access right now to check it out.
[https://www.wololoci.com/](https://www.wololoci.com/)

We’re going to be interviewing at YC this winter so any feedback will be appreciated it.PD: and if you can show us some love at product hunt will be awesome

[https://www.producthunt.com/posts/wololo](https://www.producthunt.com/posts/wololo)

https://redd.it/drrvts
@r_devops

Does anyone use terraform for azure?

I know you can use Terraform to spin up different kinds of resources on Azure and I can find some info on the topic on Microsoft's website, I just don't find much on medium or on other types of blogs. Is there anywhere on here that uses Terraform to manage/ and spin up Azure infrastructure?

https://redd.it/drgjve
@r_devops

Embedded devops

I was asked to figure out a ci/cd pipeline for various embedded systems. I'm familiar with the tool chains involved however there is a huge problem that I haven't really found a good solution for.

Embedded systems need to be tested on actual hardware. This means having the board(s) plugged in.

Currently they have a few physical machine to handle this but it's not scalable and it wastes a lot of time from everybody involved.

I do have some ideas I want to try but they're kind of a long shot. If this is a solved problem I couldn't find it.

What are your experiences in this area?

https://redd.it/drehj9
@r_devops

Considering moving off Azure DevOps

For a number of years we’ve mainly been a Microsoft shop and using DevOps made sense at the time. It’s fairly easy to configure and for the most part no cost since it’s basically included with MSDN. That being said, in the last few years we’ve become a lot less Microsoft centric and considering moving to a different platform.

There are some things I don’t want to leave behind though. I really like how everything is tied together, almost like an ERP system. If you setup the boards and pipelines it’s really nice. For example, create a branch to fix a bug, associate the bug with that branch, then your commits get linked to that ticket. Later, when you merge to master and build/release, those subsequent items are linked as well. Long story short, I can create a release and very easily see what is different compared to the last one.

My question is, what is a typical setup outside DevOps that would provide that same type of functionality? We only have a handful of developers and while cost isn’t a huge factor, I really don’t want to go from paying $40/month (total) to hundreds.

https://redd.it/drw5tz
@r_devops

Issue connecting to Consul from inside container

I'm working on Dockerizing some applications and currently Consul runs locally on my laptop but eventually will be on Kubernetes. I have a Node.js application that even though I can connect and application works fine when I run the application outside the container, when I run it in a container I get this error message back

​

Error message:

Error: bad request
at create (/opt/node_app/app/node_modules/papi/lib/errors.js:14:5)
at Object.response [as Response] (/opt/node_app/app/node_modules/papi/lib/errors.js:38:15)
at IncomingMessage.<anonymous> (/opt/node_app/app/node_modules/papi/lib/client.js:592:26)
at IncomingMessage.emit (events.js:215:7)
at endReadableNT (_stream_readable.js:1183:12)
at processTicksAndRejections (internal/process/task_queues.js:80:21) {
isPapi: true,
isResponse: true,
message: 'bad request',
statusCode: 400
}

&#x200B;

Node.js code:

const CONSUL_ID = require('uuid').v4();
const ip = require('ip');
const my_IP = ip.address();
let options = {
name: service,
address: `${my_IP}`,
port: port,
id: CONSUL_ID,
check: {
ttl: '10s',
deregister_critical_service_after: '1m'
}
};
consul.agent.service.register(options, function(err) {
if (err) throw err;
console.log(`Registered service with ID of ${CONSUL_ID}`);
});

&#x200B;

I've checked and it's getting the correct IP for my laptop like it normally would if ran outside the container. It is also has the proper `service` and `CONSUL_ID` so I am good there too.

&#x200B;

Command used to start container:

docker run --net=host -p 8091:8080 -e mongo_uri=mongodb+srv://<user>:<password>@<subdomain>.mongodb.net -e CONSUL=127.0.0.1 myapp:b424cab

&#x200B;

Any ideas what might be the issue currently or suggestions on how I can make this work for both local development as well as on Kubernetes?

https://redd.it/dpvf04
@r_devops

First DevOps Interview

Hey guys. I'm having my first on site interview for a devops engineer position.

I'm surprised to have made it this far, because my experience with jenkins is basically none, and quite basic experience with Ansible. My main strengths are in programming, docker, and I guess having the mental urge to automate tasks and try new things.

That being said, what are some of the things I should look out for, or have knowledge of before hand?

I'm not expecting to get the job, however I am extremely excited to get experience and learn more through the interview.

https://redd.it/dpue2z
@r_devops

How to get listed in CNCF?

Does an early stage startup have any chance to get listed to CNCF landscape? Does it have to be a sponsor? Any recommendations on the process? Thanks.

https://redd.it/dpt697
@r_devops

generating adhoc plots

When I do system benchmarking I create stats for each time I run the benchmark/test. Is there a tool which I can post the data to to generate graphs? I prefer web based. At the moment I use python matplotlib but not all machines have particular version of Python and especially Matplot lib and its dependencies.

Ideally, Grafana would be awesome for this but I don't think I can create adhoc graphs. There is a JSON plugin but I couldn't get it to work. Are there any similar setups like that?

https://redd.it/drykec
@r_devops

Am I Doing it Wrong?

Just one month ago, I joined a team of 4 developers starting my DevOps engineer job.

Since then, I've been in a daily conflict with one of the developers, let's can call him Brett.

Brett is highly motivated freshly graduated engineer that writes a functional but garbage-quality code. I'm fine with it though, not really my current major concern as I'm still new to the project.

What really bothers me is the changes he does to infrastructure files, projects structure (monorepos here), and other stuff that don't contribute to the project functionality. He not only generate technical dept on my shoulders (writing a two lines of awk, duplicating it through the make file, and inventing a new files-based way to lock system packages in docker to solve a problem we don't have), but prevent me from cleaning the old stuff (he's not ok with removing a binary that counts lines of code from repository, and don't really have a problem with running migration automatically every time a container starts).

I know devops is mainly about collaboration and closing the gap between operations and development, that's what I'm trying to do when I automate the devs manual tasks into a simple make command, but I believe Brett is trying to limit my contributions, and forces bad practices that will inevitably lead to confusion for our next recruits, and he writes scripts that don't really complicate simple tasks and hard to maintain.

We talked, we agree with the business owner on a decision making process, but today after we argued about changes he made, he was very stubborn, another colleague agreed with him and approved the changes, then Brett merged his work.

While I'm still working on my own branch, he copies some of my changes to his work and commit it, and till now I've yet to merge my branch due to constant arguments he made against my work and conflict his copying results that add additional rebasing work I have to do.

Feeling disrespected, paralyzed, and not fit with the team, I offered that I resign, and the business owner replied by scheduling a call tomorrow on which I'll resign unless Brett limits his contribution to the core project. But I guess I'm going against the devops way here.

What am I doing wrong as a devops engineer? is it my approach that causes the conflict? any suggestions? Have you find yourself in a conflict with other developers when trying to bring better practices to the team and how did you deal with it?

https://redd.it/dpsafj
@r_devops

How to avoid the 403: FORBIDDEN due to the IAM service?

Account, users, role, resource, policy... when you are using IAM for the first time, it is not easy to cope with all these new denominations and concepts. I wrote an article that explains [what is IAM and how it works](https://www.padok.fr/en/blog/aws-iam), with concrete examples. Check it out and give me your opinion.

https://redd.it/dpp6iw
@r_devops

Easy way to create internal web tools

Hello, I do some devops work for my company, mostly involving various powershell and SQL scripts that I run to do some of our internal tasks or stat gathering. In order to present these stats I have been using an internal website that I have made completely manually with php. Since I do not want to focus on the web development side of things and would rather work on other things, I was wondering what a good tool would be that I can use to easily develop some fairly simple sites for things like presenting stats from in a table that connects to sql, triggering JSON file creations etc.

&#x200B;

Basically, I just want an easier way of doing the web side of things so that I don't have to re-learn and mess around with PHP every time I want to add something to our internal tools, since I am not a "web developer" per se. What are some decent options that I can learn to unload the web dev side onto a tool?

https://redd.it/ds1bvv
@r_devops

Given a blank slate, tool recommendations?

Hi Guys, so I have 8 months of DevOps experience previously working as a desktop engineer for two and a half years.Starting a new role in the education sector, recently they brought their IT department which previously worked in different projects/faculties and now they have all been brought together.So this is where I come in to help dictate the application development process by designing and maintaining the infrastructure to build an efficient route to live regardless of the project. So this is essentially a blank slate.

As a very high level overview

So my ideal setup would be creating a production environment for whatever project, replicating the stack and using that as UAT. This would be done through Terraform and Ansible. The Dev estate I am not too sure how this is going to be setup yet but I was thinking about learning docker and k8s to bring up a dev estate.

The way I would look into setting up a CI/CD pipeline using Jenknnis and actioned through Azure DevOps as this is what they use, I would ensure code pushed from the DEV branch is deployed to UAT and tested then deployed to replicate production servers and once tested DNS is switched over to the new servers and the old ones are brought down.

The way how the code is deployed is through building the codebase on the same server Jenkins would be setup on, sending a tar'd file of the code to AWS CodeDeploy to be taken to the next environment. 

Through caveats include no changes or access will be allowed on production or UAT servers, any requested changes will be done through terraform and ansible to keep things consistent.

Personally with my limited experience I have used Terraform and ansible and I have felt this provides alot of the work to have that infrastructure as code system in place. At the same time I understand there are tools like chef, puppet and salt which would probably make the job easier but that is where my question is what would you guys want and how would you want to implement it. I should finally mention we seem to be using Azure DevOps as the main tool to setup and manage the pipelines.

https://redd.it/ds2fhn
@r_devops

Testing Pulumi infrastructure as code written in Python

Most of you are already aware of [Pulumi](https://www.pulumi.com/), an open source tool that allows writing infrastructure as code using high-level programming languages (JavaScript, Python, Go, and soon C#) instead of a domain-specific language.

Pulumi has an official integration testing library written in Go: https://godoc.org/github.com/pulumi/pulumi/pkg/testing/integration. However, if are developing in Python, you probably want to write your tests in Python as well. To that end, I'd like to share a Python integration testing framework that I've been working on: https://github.com/bincyber/pitfall

If you find this project useful, I would love to get your feedback on usability, any bugs encountered, or missing features that should be added.

https://redd.it/ds46nh
@r_devops

Best way to upgrade Kuberenetes without breaking anything? (EKS)

We have jenkins on EKS which needs to upgraded b/c AWS is not supporting our version anymore 1.11. Upgrading will probably break several pipelines. Has anyone had any experience with this or recommend a way to tackle it?

https://redd.it/ds3sb3
@r_devops

Why collaboration technology is critical for GitOps

“Collaboration is key to this entire GitOps process,” . “Infrastructure teams, development teams, even management, project management, security, and business stakeholders, all need to collaborate together to produce this code in a fast and efficient manner.”

[https://about.gitlab.com/blog/2019/11/04/gitlab-for-gitops-prt-1/](https://about.gitlab.com/blog/2019/11/04/gitlab-for-gitops-prt-1/)

Code: [https://gitlab.com/gitops-demo](https://gitlab.com/gitops-demo)

https://redd.it/ds3de2
@r_devops

Tutorial: Install vagrant-libvirt on macOS to Manage Remote Libvirt Hosts

I've written a tutorial on how to get [vagrant-libvirt](https://github.com/vagrant-libvirt/vagrant-libvirt) running on macOS to manage remote libvirt based virtualization hosts using Vagrant running on my Mac. It's mostly based on work done in [an issue](https://github.com/vagrant-libvirt/vagrant-libvirt/issues/497) over at the `vagrant-libvirt` repository.

Maybe someone will find it as useful as I have. :) Any feedback will be greatly appreciated!

https://lunar.computer/posts/vagrant-libvirt-macos/

https://redd.it/ds6zl2
@r_devops

Running Containers in the Cloud - What's Your Solution of Choice?

If you were starting a greenfield project today and planned to utilize containers, what would be your solution of choice? Assume a microservices architecture with multiple services (yes I know it is 'bad' to start with microservices), all stateless applications.

These are some of the options I can think of off the top of my head.

All the managed kubernetes offerings (aks, eks, gke).

Cloud vendor specific solutions like Azure Container Instances, AWS ECS (fargate or ec2 backed), GCP App Engine Flexible Environment.

Roll your own kubernetes cluster either manually or via a tool like rancher.

Run containers on vms with container runtime installed (gross).

I ask this because we are starting a project soon and a lot of the team is familiar with AWS. But I'm not personally a huge fan of ECS (and I'm not sure it is still relevant), and I'm not sure I trust the AWS EKS offering. Interested to hear peoples thoughts!

&#x200B;

I think I would lean toward GKE as kubernetes and containers have kind of been their baby. I also prefer the GCP platform, I feel like they have a much better developer experience than AWS or Azure. Things just seem simpler. AWS and Azure have too many relics from the days of on prem data centers. GCP abstracts a lot of that complication away.

https://redd.it/drz57o
@r_devops

Anyone else feel like this subreddit has really gone downhill? Is there anything we can do?

Maybe I'm just a major asshole. I try not to be. But I feel the same way about here as I do about /r/sysadmin anymore. Every time I come here anymore I feel like it's just an echo chamber of "Terraform, Kubernetes, and Azure DevOps". And everything is a nail for those three hammers. And no one is doing anything other than really vanilla infrastructure as code. Like that's the end-all and be-all of DevOps.


I also feel like the vast vast vast majority of conversations going on here are about greenfield new stuff. And a bunch of people that aren't all that close to production are like "yeah, Azure DevOps is great. go with that. we got our little hello world app up in like an hour". Like, I know people can't just wave a wand and choose where they work. It's not anyone's fault if they work in an old school slower moving shop. But maybe don't evangelize stuff if you're not using it in an extremely high velocity production scenario? (ie dozens of devs deploying to production many times a day)


Like I said, maybe I'm just an asshole. But it would be cool to talk about stuff like solving real business use cases with creative solutions. And not just "oh you want to automate creating servers? use this tool". Maybe there needs to be another subreddit for (what I, as a piece of shit call) "real" DevOps? Like, no company is making it to their IPO because an engineer moved their shit to Kubernetes. So can we just talk about tools a little less? Or, really, a *lot* fucking less?

https://redd.it/ds7vbc
@r_devops

Grafana Annotations in pie chart

Hi Everyone

We were exploring grafana annotations published by Jenkins, and they are showing up in graphical charts. But annotation markings were invisible when we changed the graph to pie mode.

Is it possible to have annotations on top of pie charts?

Regards

https://redd.it/ds9gas
@r_devops