How are people executing communications/notifications for a large, distributed engineering organization?

I work in an engineering organization with \~1000 engineers across 100 teams that is globally distributed. These teams collectively build/maintain/depend on an incredible amount of AWS infrastructure, software, and other resources.

A consistent issue that I see affect our organization revolves around how to structure communications when dependencies/policies/resources change in some way.

* How do we notify and capture the attention of teams that own or depend on these resources when they change? How do we notify people about a breaking change or a policy update?
* How do we even know that teams have seen an announcement?

I'm looking for a tool that could help us streamline this process. We could build something ourselves but I want to ask around to see if anyone else has dealt with this problem before committing to any course of action.

**Existing Tools:**

* We have an existing application that allows teams to self-manage which projects/repos are owned by the team, who is on the team, Slack channels, email list, JIRA board, etc as a reference for the rest of the organization.
* Email list for all engineering members.
* Engineering Slack announcement channel

**Potential Requirements:**

The ideal tool would let us create entities/targets based on our existing team metadata and allow us to send targeted notifications to each team.

* We would like a single entry point that any engineering team can use to send these notifications.
* We would like teams to be able to designate where they would like different types of notifications to be routed. For example, an “Action Required” type of notification could be set to generate a JIRA ticket in a team’s backlog.
* We would like to be able to keep a history of these notifications and refer back to them.
* We would like some notifications to contain “acknowledgement” mechanisms if desired to allow tracking how teams have responded to notifications.

https://redd.it/eemqri
@r_devops

GCP-Nuke v0.1.0 alpha - the GCP project cleanup tool with added radiation

Link: [https://github.com/arehmandev/gcp-nuke](https://github.com/arehmandev/gcp-nuke)

Hi all. Any Google Cloud users here wanting to cleanup projects? Maybe even do a dry-run to see whats taking up those bills?

AWS-nuke seemed to be incredibly popular and mature, so after many nights of rolling around dreaming of GCP support tickets ([Shared VPC ghost subnets? Anyone?](https://cloud.google.com/vpc/docs/deprovisioning-shared-vpc)) I decided to do something about it.

I've written a few CLIs in my time, but nothing prepared me for the unreliability of a delete API operation claiming it was complete, and a dependent resource claiming it still exists. As such, I've added retries, dependency checks, and tried to make each resource as easy as possible to contribute to. There is a roadmap still to improve and add robust testing, and many code reviews I'm sure to come, but its a start.

As of now it supports compute and GKE resources (GCE instances, disks, instance groups, instance templates, GKE clusters).

Let me know what you think, I'm always on the lookout to improve my Go development.

Cheers.

https://redd.it/eelfpl
@r_devops

Boss wants to fire after hours call service, what can we move to?

We currently pay for a small company to answer the phones after hours and escalate to a technician if needed or just take a message. They've been unreliable lately and we want to see about automating it. We are looking for an answering service that could do these things below, what products or suggestions would you have for this?

​

* During after hours we generally just forward our whole company phone \*72 etc to the call center, instead we would forward to this new service.
* IVR Answer and forward to one of four duty phones or straight to after hours voicemail for non urgent stuff.
* Duty phone numbers need to be easily changed or even better scheduled in advance specifying changeover times
* Calls should be recorded
* Metadata should be recorded, from number, duration, IVR selections made etc
* If weekly and/or daily summaries could be sent to a couple people that would be great
* Ideally less than $100/mo.

FreePBX came to mind but that seems like a very high initial investment in labor in getting it running. There's probably some cloud service out there doing what I need I've never heard of, what do you say?

https://redd.it/eemgjh
@r_devops

[Need Advice] Setting up a VPN for my kubernetes cluster without a loadbalancer

First off I want to say that I am very new to devops and I have just started learning about a lot of things. My case is as follows:

I want to add a VPN to my cluster so that I can visit services that are exposed using a ClusterIP service. OVPN has a tutorial / helm chart on how to do this, however these services are installed as Loadbalancers. For the sake of just development I do NOT want to spend money on a loadbalancer if I can avoid this.

I have tried using a nodeport but I have yet to get that working.

Any advice would be greatly appreciated.

https://redd.it/eem5it
@r_devops

Testing within DevOps

Hello, I know this is a bit of a taboo subject with DevOps engineers but my DevOps team has recently brought in a QA/Automation engineer into the team ( Me :) ), I was wondering if there are any tips or tools that you recommend me looking into? I can't find much on testing within DevOps but I have made a start at things like Pester, Chaos Monkey and Ansible.

https://redd.it/eellt6
@r_devops

Trying to learn too much too quick, what should I prioritize?

Hi all, I just would like a bit of advice. I currently work in DevOps for my company. I was the first iteration of a DevOps and the only one however I feel my skill set is seriously lacking and I want to try and skill up so that I'm prepared a bit more for the future and what tasks/projects that may bring.

I got moved into this position due to a strong ops/infrastructure/release background but I had zero dev knowledge/experience. So I started trying to learn Python, C#, Powershell. I didn't get too far with any of them as I kept splitting my time and resources between them but I learnt enough Python so that I could build a monitoring system using Python/Selenium/Prometheus and Grafana and just the barebones of PowerShell so that I could automate a few boring tasks/know how to google and implement scripts I found.

All my experience is On-Prem so I started trying to learn AWS and Azure via online courses and playing around with them but the problem I was finding is the Azure stuff was changing too quickly and because I was spreading myself thin I was struggling to keep up. Also, I only know Windows so I started trying to pick up Linux etc.

Then on top of that because literally everything I see about DevOps is to do with containers so I started trying to learn about Docker and Kubernetes. The problem I am facing is I don't really know what to prioritise, my work situation is very strange at the moment so currently there isn't any expansion for moving forward with technology and my free time isn't massive and I feel the last few months I haven't really learnt much because I hit a point where it felt like I wasn't absorbing information.

​

So coming up to the new year I'd like to prioritise what I should learn so I can set some achievable goals so I can actually progress things. Any help or advice would be greatly appreciated.


Merry Christmas :)

https://redd.it/eeiywa
@r_devops

Cloud-Native Logging And Monitoring Pattern

**Folks,**

Since the early ages of mainframes and punch-card programming, there have always been gauges for monitoring the system’s temperature and power among other things.

Here is This week Article [Cloud-Native Logging And Monitoring Pattern](https://www.magalix.com/blog/cloud-native-logging-and-monitoring-pattern).

**What's inside:**

* The Need For Observability in The Cloud-Native World.
* Cloud-Native Logging/Kubernetes Logging.
* Cloud-Native Metrics.
* The Pull-Based Method/The Push-Based Method.

[https://www.magalix.com/blog/cloud-native-logging-and-monitoring-pattern](https://www.magalix.com/blog/cloud-native-logging-and-monitoring-pattern)

https://redd.it/ef2jty
@r_devops

Need some experienced devs / hackers


Hey guys, I’m looking for some very experienced devs / hackers to work on a project with serious money within it, drop me a message if you want work on it

https://redd.it/eejxsj
@r_devops

GDPR for operating systems

As I value privacy a lot along with respect the GDPR law, why are operating systems not complying to this law? When I log into a linux server, the first thing it says is: `Last login: ... <ip-address>`. It actually keeps track of my ip address. I actually never took time to further investigate what it does with your IP address, but assume the server got hacked, there is sensitive login information from all the devs who logged in to the server.

Why is this law not respected by operating systems? Is it necessary to do so?

https://redd.it/eedwv4
@r_devops

When should you use Terraform over provider API?

If a service has an API and is also a Terraform provider, how do you determine which to use? If you already built python classes and toolkits that utilizes the API, should you migrate to Terraform?

https://redd.it/edcbq9
@r_devops

Where to host for eastern Europe and China

Hi everyone, I'm a dev ops engineer in the US with experience in mainly GCP and some aws. I'm relatively new to things like site speed but I'm learning about latency and how optimizing for regions in the US is important.

I'm working on a project that would be primarily servicing eastern Europe and China. I see hosting in China is possible on GCP and aws but I can't seem to find a cloud platform in eastern Europe. Anyone able to point me in the right direction?

https://redd.it/edbs59
@r_devops

Database schema changes

I'm current working with the LAMP stack and extrapolating everything into microservices and implementing immutable infrastructure. The one issue I'm having conceptually is how to manage database schema changes via atomic / immutable infrastructure.


Currently our websites will experience downtime when updating the software where database schema changes occur and I was hoping to get some advice on how to avoid this using containerisation.

Thanks

https://redd.it/edbdjz
@r_devops

Is this a devops role? What should I be learning?

Hi everyone, I'm starting an internship in a few weeks and I'm trying to study some concepts in advance so I won't be completely lost when I start. The responsibilities seem to be related to what a devops position would have, but it's on the "Tools" team and I'm not sure of what I should be learning before I start. Some of the job responsibilities on the description are:

- Create internal tooling frameworks that help teams simplify environment management and monitoring.

- Utilize CI/CD and other tools to help other engineers to self-service day to day operations.

- Explore image & container-based build, testing, and deployment strategies.

- Investigate new ways to automate deployments, canary analysis, and rollbacks.

I'm a complete beginner to all of these concepts with no experience regarding CI/CD pipelines, containers, and automation deployments. Currently I'm thinking about starting this [devops learning path on Linux Academy] (https://linuxacademy.com/learning-path/junior-devops-engineer-entry-level/) but skipping the Linux Foundation course because that seems less relevant. Is that a good idea? If not, any suggestions on better places I could get started would be greatly appreciated.

Merry Christmas!

https://redd.it/efct7w
@r_devops

Intro to Salt Stack Article

Since my Kubernetes article was perceived as helpful and well received, I thought I would share my Intro to Salt Stack. It's a multi article series on my journey to learning Salt Stack. I have learned quite a bit from this subreddit and wanted to share back.

I know we have some seasoned devops in here but also aspiring devops and those wanting to get a taste of it. You can easily follow the article to setup a lab.

Tell me what you think. Downvote if it is not useful but tell me why you downvoted if you do. I want to start a conversation even if its constructive criticism! :-)

https://blog.woohoosvcs.com/2019/11/introduction-to-saltstack/

https://redd.it/edargq
@r_devops

Automated test tool to help in CI/CD

I am new to this CI/CD thing. I know the concept is not new and it is as old as computer science itself but the buzz is new. I learned that for CI to work, you need some kind of automated tool that will test your new code and will tell if it passes or not so you can integrate it with main branch.

My question is can you point me to any such kind of tools that will automate this task? I mean let's say I am working on a project which has three modules; Login, Register, Generate Invoice. If I write a code for login, register and generating invoice than how will that tool be able to test this functionality?

https://redd.it/ed98t5
@r_devops

Migrating from Amazon Web Services (AWS) to Google Cloud Platform (GCP)

Here's a detailed article on how our DevOps team reduced downtime to a mere fraction and could introduce new features faster. [Give it a read.](https://blog.hike.in/migration-to-google-cloud-platform-gcp-b92036429a22)

https://redd.it/ed7349
@r_devops

Are there any good books (or book chapters) or articles on the theory and maths for performance testing?

Most load testing tools produce reports with mean, max, and percentile details. For example [this](https://jmeter.apache.org/usermanual/generating-dashboard.html) is a JMeter report.

But:

* How should we interpret the details in this report?
* What would be the success criteria?
* How can I make sure that my test is actually relevant and capturing the right details?

Are there any good books, papers, or articles that cover these topics?

https://redd.it/ed1p4c
@r_devops

I purchased a domain name from name.com and hosting my site at aws lightsail, where could I configure a forwarding email address?

Say I want an email address like [email protected], which only forwards it to my other account on gmail, do I need to set it up with name.com or with aws? Since AWS is providing nameserver?


Solution : If anyone else, faces the same issue, I used yandex mail service, and added mx record, spf record, and dkim signatures in lisghtsail's domain configuration.

https://redd.it/eczhuh
@r_devops

RollingUpdate Deployment on AKS is killing all POD's

Hi, i'm a DevOps Engineer and i have a issue with the RollingUpdate Deployment on AKS (Azure). When I try to execute a a pipeline in the stage of deploy that execute a RollingUpdate Strategy deployment on AKS the process kill all the POD's.

With this code:

`spec:`
`progressDeadlineSeconds: 60`
`strategy:`
`type: RollingUpdate`
`rollingUpdate:`
`maxUnavailable: 1`
`maxSurge: 1`

&#x200B;

I see this topic on Stack but I didn't find anything that works
[https://stackoverflow.com/questions/46369100/kubernetes-rolling-update-killing-off-old-pod-without-bringing-up-new-one](https://stackoverflow.com/questions/46369100/kubernetes-rolling-update-killing-off-old-pod-without-bringing-up-new-one)

https://redd.it/ecv3qm
@r_devops

Things to have in a resume for someone who wants to switch from developer to DevOps engineer

I have worked as a developer for more than 3 years. For the last 1 year I have been working with AWS, learning architecture and CICD in cloud. I will go for the solutions architect associate exam very soon. Now I want to apply for DevOps related jobs as I find it more interesting. I want to know what are the dos and don'ts for the resume. Thanks in advance. 🙂

https://redd.it/ecoy41
@r_devops

Serverless approach or Kubernetes Cluster Approach for Banking System

Hi , I am new to serverless architecture. I am about to create a banking application. Is it a good idea to develop my whole system using serverless architecture (using AWS SAM) or should it be a combination of both Serverless and Kubernetes Cluster. Thanks in advance

https://redd.it/ecor5c
@r_devops