DevOps in an IoT shop?

I've been doing DevOps in SaaS shops for some time now, and now I'm in an IoT shop. Now that I've been there a month, I now know that it's two different, but related things.

I could run down a long list of things that make the difference, but it boils down to this: There is a hesitancy to have other people do things for you. Databases are on VMs, no autoscaling, things like that.

Anyhow, anyone have any advice on IoT DevOps?

https://redd.it/obrxru
@r_devops

GCP Autoscaling group and Terraform - prevent downtime when changing template

I had to swap out an autoscale group's template image today and caused a blip whereby the WWW servers in the autoscale group were destroyed before new ones were up and running. I need to prevent that. I thought I had chosen the right lifecycle options but I guess I chose wrong.

This is

* GCP
* Terraform
* GCP instance group manager
* GCP instance template
* GCP disk image

The update policy -- which is what I thought would control the behavior I'm concerned with -- is as follows:

update_policy {
type = "PROACTIVE"
instance_redistribution_type = "PROACTIVE"
minimal_action = "REPLACE"
max_surge_fixed = 12
max_unavailable_fixed = 6
min_ready_sec = 600
}

Any guidance is appreciated. This issue was hard to Google for. I'm sure it is in the docs but I've read and reread 'em and I think I need to ping off a human now.

https://redd.it/obso0n
@r_devops

What is the number 1 thing that you learned from the quarantine?

It's been a long year and looking back many of us faced a new kind of situation. What is the number one thing that you learned related to quarantine and working from home situation?

https://redd.it/obq2ps
@r_devops

AWS MicroService Deployment

Trying to deploy my SpringBoot microservice app with an AWS Postgres DB backend. What's the easiest no-brainer way to build a CICD pipeline for this thing to get auto-deployed on code commit?

I've heard GitLab CICD -> AWS for deployment is good or should I go for a pure AWS solution with AWS CodeDeploy, pipeline, commit etc?

https://redd.it/oc12xm
@r_devops

How long does it take for a Linux or Win Sysadmin to transfer is skill to Azure/Aws and become well versed in it ? Do you believe Powershell is more powerful than Bash ?

(excuse my english french guy from canada here)

I am learning Windows Server with Vmware workstation and Linux too... after I am solid in both I will learn the Cloud not sure between Azure or Aws ... I wanted to know this and any advices I can get :

1) How long does it take for a Windows Sysadmin to transfer is skill and learn Azure or become well versed in it ? (intermediate - senior level) like powershell, managing dc, file share... im mostly interested in O365, desktop management things like that and others AZ Services ...You guys know how much it cost to learn Azure or if there is some MS plans for student or learning it without paying huge fee's ? kind of like building virtual network scenario in VMware to test and study different topics but with Azure ? everywhere I go on MS website they charge for everything mostly.. so what's the best path to learn ?

2) Is it possible to be near intermediate or senior in both tech Stack Linux AND Windows with automation, cloud services learning it all etc or it's not worth it ? For example Powershell is starting to be cross platform but no Linux admin I talk too will ever consider using it to manage production env over puppet/ansible/bash python etc but Winadmins say it's superior to bash so what to give what to think of it ?

3) I live in a very remote place in north Quebec... I thinking about going back to programming after 25 years and push some Javascript/React and become a full 100% remote programmer... from your humble opinion is it easier to be 100% remote as a dev or as a sysadmin ? should I go back to programming or aims more for a devops kind of role ? (I don't really care I love both side programming sysadmin was doing some C++ back in the days) the problem is more geography for me than anything else I can study 10hr's a day for 2 years if I really want too, have all the time in the world thing is how to use it the most efficient way

4) Do you believe Powershell is more powerful than Bash ? MS seem to want to go full support of different OS with Powershell... would you manage large network or datacenter who run Linux with Powershell or stick with ansible, puppet, bash/python etc or whatever your company and scenario use ? Powershell seem to be used more and more on Linux with new cmd

Thank you very much for your time greatly appreciated if there is any RHCE MCSE experts out there for advices

https://redd.it/oc2miv
@r_devops

Reliable email forwarding from a domain email to personal gmail account?

Apologies if this isn't the right sub to ask this - it's an email deliverability question.

I have small-business clients that set up a site on a domain, but they use Gmail for all their email. Therefore, addresses like "[email protected]" need to be forwarded to "[email protected]"

Most web hosting services allow email forwarding, however - due to tighter spam/spoofing restrictions - many begin to fail (perhaps a trust issue with the shared hosting MX servers?) Note this is after setting up SPF, DKIM and DMARC all correctly, but they still fail.

I've got around this by setting them up with Google Workspace and using Routing to handle email forwarding. This works as the domain's MX servers are now in Google's ecosystem and so forwarding a domain address to a Gmail isn't a problem.

But as my clients are small businesses, the extra monthly cost of Workspace isn't great, just to solve this one deliverability issue.

So.. sorry for the long-winded explanation... my question is: Is there another way to solve deliverability from domain address to Gmail in a cheaper way? Are there other, cheaper email services I can use, whose MX servers can reliably forward to Gmail?

Thanks for any suggestions!

https://redd.it/oc2zs1
@r_devops

Separate or same pipeline for infrastructure and application deployment

Our company is trying to write CI/CD pipeline for infrastructure and vm based application deployment. Is it best practice to have everything in 1 pipeline or should we separate infrastructure and application deployment into 2 pipelines? Any website out there that goes through pros and cons for each?

https://redd.it/oc487z
@r_devops

How can i improve AWS cloudwatch alerts?

Historically Ive used datadog for all my projects to handle alarms to slack which would typically include:


* A quick description of the problem
* A graph
* Links to documentation or known remediation steps

The company Ive now moved to are tied into cloudwatch, I am trying to get them to migrate but that would be a huge project to move years of monitoring.

Cloudwatch can send alarms with SNS and their chatbot to slack to cover the first two points. But I cannot get it to send documentation or links. These are attached to the alarm description but cloudwatch only sends the title.

https://redd.it/oc5u2m
@r_devops

Exposing our applications with GCLB and Istio @ BlaBlaCar

Hello there,

BlaBlaCar is a major actor in CarPooling in Europe, Russia and Brasil.

I wrote a quick article on how we setup our public facing application on GKE cluster with Google Load Balancer. We also have setup certificates with Cert-Manager and we have HTTPS redirection on GCLB.

We rely on Istio to do that with Gateway and VirtualService. Nothing fancy on that part we use the regular resources and setup. We only have on patch done with EnvoyFilter to fix the number of networks hops the Istio Gateway is trusting.

I explain how we have done it and why we do it like that. Our main goal is to let our service teams to be able to do everything on there side.

https://medium.com/blablacar/exposing-our-applications-with-gclb-and-istio-3b635813ab12#860e

​

Feel free to ask me any question.

https://redd.it/oc5xgv
@r_devops

Not sure how to start

Not sure if this is the right place to post.

I work at a Hospital 5k employees, no official development team. BUT we have people writing lots of custom scripts, code and even a few internal tooling websites (the DBAs, random ops guys with coding talent, etc). Works great..... EXCEPT, we have things with no built in authentication, on an unsupported OS, no one knows how it works, the guy who wrote it didn't document anything at all, holds people's socials in a database, can't even find his source, and he got laid off over a year ago. All we know is HR says they get an excel sheet from finance, type ABC data into the site, and they think it integrates with some important downstream system, but they don't know how/why/etc so they just keep doing it.

Ignoring the specifics of that problem.

What I'd like to do is attempt to wrangle all the "not developer" developers, and get them to put all their code in git, and any long running process in a docker container with tests and the whole 9 yards. Nobody has any experience with docker including me (outside some personal home projects), minimal experience with git (I do use git for all my stuff), and no professional experience working with actual developers.

I see kubernetes and all these big fancy orchestration tools. But I don't think I'll get buy in from my peers or management if I tell people they now should learn another 2/3 layers of abstraction.

If I wanted to get my company started with docker, for small apps (<30 concurrent users), and try to get them to embrace internally developed tools. How do I start without over engineering the hell out of everything?

I have read docker tutorials and setup and got stuff running in it. But it feels like I just cobbled together another half assed mess, whenever I look at making it more centralized/controlled, it seems like there's another 3 layers of tools, abstraction, and whatnot. I need something in the middle so I don't scare away my colleagues and can get some sort of buy in. Maybe it's a dumb goal for my org, but I'd like to try something.

Books, articles, videos, specific tools/software or your own advice would be appreciated.


Tldr;

My medium sized company has non-existent code development practice, no experience with devops, docker, etc. Imagine being in IT at the year 2000, migrating to virtual servers was the last "new" thing we did. How do I start them down the path of devops without scaring them with all the modern tooling?

https://redd.it/oc6pqe
@r_devops

HashiCorp Vault Plugin for TOTP

After mulling over my options for authentication in my home lab, I decided the most comprehensive solution is Vault. However, it doesn't support 2FA for the OSS version. Luckily, you can write plugins!

https://github.com/thequailman/vault-plugin-auth-usertotp

vault-plugin-auth-usertotp is an auth method plugin for HashiCorp Vault. Create user accounts, add TOTP tokens (user supplied pin + TOTP), and have peace of mind using 2FA.

This plugin is also a drop-in replacement for the native userpass auth method, so stop using that and use this instead!

https://redd.it/oc8vor
@r_devops

Looking for recommendation for automated database configuration management

So we have RDS which can be managed via Terraform/Cloudformation.

I am looking for your recommendations for automating the inside of the database.

For example:

\- I need to create users for AWS IAM auth to work: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.IAMDBAuth.DBAccounts.html#UsingWithRDS.IAMDBAuth.DBAccounts.MySQL

\- I would need to be able to create a new user in the database whenever a new Developer joins the team that needs access to a DB.

Any tips on how to automate this?

https://redd.it/oc988o
@r_devops

Is CDKTF the future?

I just got into Terraform for some work stuff, and it feels kinda clunky.

It might be because I'm mostly a developer, so I'm more used to reason in terms of imperative languages, but I really cannot get into the frame of mind of how terraform is supposed to work.

Anyway...I found out about CDK, which basically it's terraform but in Python, Go and Javascript. I did not look into it a lot yet, and it seems kinda green (still in beta).

I'd like some experienced opinion on whether you think CDK will catch on, and if it actually makes sense to switch, or if I should just bear with it and eventually pure terraform will be better once you "get it".

Thanks!

https://redd.it/ocankq
@r_devops

Any online assessment tools to check hands-on DevOps skills of candidates?

Folks, are there any online coding/assessment tools available to test practical AWS/Azure/GCP/Kubernetes/DevOps hands-on skills of candidates in a cloud environment/sandbox? Not MCQ type questions, but real troubleshooting, configurations etc.?

https://redd.it/ocbc6x
@r_devops

Review Apps - What Platforms Offer Them?

For the last few years I've used Heroku Pipelines for lightweight dev ops for SPA / Node applications. It's pretty fantastic in that it's reasonably easy to setup and it's opinionated in a good way. I've only used it without Docker images but I'm told it works with them as well.

But recently I joined a company that is using GCP with docker containers and basically zero dev ops. They have literally been pushing docker images up to a Kubernetes cluster which was also setup manually. I'm converting all of this to Kubernetes files but that will just get us a repeatable environment, not review apps.

My question is what are the best options for getting Review Apps up and running? I'm looking at GitLab but it's pretty flakey so far and my googles for review apps aren't showing much other than for Azure and it feels a little weird to use Azure to deploy to GCP.

Re: load, there's basically zero load on these servers. Someone just went ham and convinced them they needed all of this infrastructure for hypothetical spikes but it's a B2B company and new customers can't even self-deploy so that's not likely.

https://redd.it/oce4w0
@r_devops

E(F)K stack within ecs

I was curious if my plan to implement a basic centralized logging solution for my small startup sounds fine. I see most tutorials out there are running the EFK stack in K8s but I'd rather not since we don't use K8's anywhere.

Our product is generally deployed in two sets, the core product in AWS(ECS) and a small physical IoT device given to customers on-site(basically a linux box).

I want to collect logs from all these locations and create a central place to monitor them.

For our products ECS stack, I was planning on just adding a fluentd container to run within the cluster to collect and send off the logs to Elasticsearch.

For the small IoT box I was planning on also just running the fluentd container or the td-agent(Haven't determined when to use one or the other)


So my plan is:

1. Create ECS Fargate Cluster with two services - (Kibana and Elasticsearch)
2. Create ALB/ACM & Route 53 record for Kibana - (So the team can access dev-logs.mydomain.com)
3. Create a custom fluentd docker image - (So I can load in the custom fluentd.conf file that contains creds to authenticate and send logs to the remote elasticsearch server, contains any needed fluentd plugins, etc)
4. Run this fluentd docker container in our stack.
5. At this point things should just work?

I was hoping to get feedback if this sounds like a good plan, would this be manageable for a 1 man "ops" department or if there is a simpler / better solution.

https://redd.it/ocf4wm
@r_devops

Is 1 year experience too little to take on a job role where I'd be the only DevOps Engineer?

Looking to make a switch, spoke to the manager and the developers have been managing pipelines and infrastructure, they need an experienced devops engineer to handle everything.

I've been working with a team of Devops engineers and a tech lead, It might be a lot of responsibility and trial and error, but could be worth it.

When did you realize you did not need continuous help from someone, and if you did would you say a role like this (only DevOps engineer) is for someone with more experience?

https://redd.it/ocgjdg
@r_devops

Jaeger gRPC tracing and plugins

I have used Jaeger to trace Rest APIs for quite some time and had no problem with it, but recently my team realized that all of our recently deployed gRPC servers were not having their failed calls being traced as errors. I mean, we can see the event "error" and its message, but the error icon does not appear and it end up looking like a successful call in the UI. Any one has any ideia about it? I haven't find much reference on the internet and the closest thing that I got was the grpc-plugin. To be honest, I have no idea of the use cases for the Jaeger plugins

https://redd.it/ockynm
@r_devops

I built a new visualisation tool for AWS Infrastructure - Quadzig

Hi Reddit!

I am a Site Reliability Engineer by profession and I built Quadzig as a new tool to visualize, understand & search AWS resources. All of Quadzig's visualizations and search functionalities are built to deal well with multi AWS account setups. This means you get a single pane of glass to visualize & search your resources across all your AWS Accounts.

The product is available both as a SaaS offering(with a free tier!) and as a self-hosted solution on AWS Marketplace(if you have stricter compliance requirements).

Reference Links:

1. Quick Video demo of the product - https://www.quadzig.io/#video
2. SaaS Pricing - https://quadzig.io/#pricing
3. AWS Marketplace Offerings - https://aws.amazon.com/marketplace/seller-profile?id=92018d4d-dd3e-4942-8287-282a98a0e377

I would love it if you could give Quadzig a try and let me know your thoughts. I will be around to answer any questions.

PS: If you subscribe to AWS Marketplace offering to evaluate Quadzig, drop me a line at [email protected] and mention this Reddit post and get 100% off on first month of usage. :)

https://redd.it/ockqw7
@r_devops

newb packer ansible copy error

I want to learn the ansible provisioner for packer using vmware-iso.

I'm trying a very simple file copy with ansible during packer vmware template build.

> - name: Install test file
> copy:
> src: /root/provisioning/packer/testfile
> dest: /var/tmp/

I keep getting this error:

vsphere-iso: fatal: 127.0.0.1: FAILED! => {"changed": false, "msg": "Source /root/provisioning/packer/testfile not found"}

The file exists on the ansible controller.

> #cat /root/provisioning/packer/testfile
> this is a test

What am I missing?

https://redd.it/oclqv0
@r_devops

Will DevSecOps replace DevOps in future?

Hii everyone, I read few articles about DevSecOps and I got a bit confused. The thing I could get from the article was that, DevSecOps is a bit more advanced version of DrvOps where security testing and measure are being automated with the Software Development pipelines, but we also take care of security in DevOps ri8?

I don't understand what exactly it means and will it take over DevOps in future, or the two terms will combine in future. Can anyone help me with this ?

https://redd.it/oclczh
@r_devops