Simple explanation of what Hadoop is and how it works?

Sorry but I am not a backend expert but I do know the basics. So I stumbled upon something called Hadoop that is being used by big companies such as Spotify but I have searched google but I have not yet found a simple explanation about what it is and how it works.

https://redd.it/e9u01q
@r_devops

Free RevDeBug Test Suite License

Hello DevOps

I'm a network technician and won a license for RevDeBug program.

I wont be using it, so here is the key: E80217DCA5EC4F70

https://redd.it/ea3yfw
@r_devops

Consul KV or Vault KV?

So currently we have some microservices deployed with consul for service discovery and vault for a pki store. I'm not the devops man so I'm curious about both services kv stores. If I want to provide config values to a service, does it make more sense to use consuls kv store or vaults? And what is the difference in application between them?

https://redd.it/ea6dtw
@r_devops

Could I consolidate VPC stacks by importing a VPC value from another template?

I have to consolidate two vpcstacks...one is 3000+ lines, the other is 300+ lines. I did not create these templates, and they are not super modular...as a quick and dirty, could I import the 3000 line vpc template's vpcid into the smaller vpc template and call it a day? What would be the easiest way to do this? :) Thanks.

https://redd.it/ea5t4b
@r_devops

How do you deal with snacking at work, is it a problem ?

As Developers I know that it is not very easy to stay in shape and lose weight. Most people have one problem that can be difficult to deal with and that is having snacks around. I'd like to offer some tips that I personally use for this problem.

1. **Bring your own snacks that are healthy**
2. **Identify the times of the day where you get the hungriest and prone to snacking.** Schedule either a time for those particular times to go for a brief walk or eat healthy snacks that you bring to work that are filling (Oatmeal/fruit or a protein source is my favorite.)
3. **Get out of habit of going to the snack area.** Once you are in the habit of no longer going to the snack area, you will not be faced with this urgency. This will take couple weeks get used to. Make a commitment starting today and try to stay away for 2 weeks straight from the snack bar!
4. **Communicate to people around you** to stop offering snacks because you have decided to take care of your health! They might even hold you accountable and create more leverage!
5. **Implement a version of Intermittent fasting**. Once you decide that you won't eat until a certain period of time and the eating is just simply out of question, it will also mean that you won't be looking for snacks.
6. **Drink tea or coffee through out the day** (Tea is better since it doesn't have so much caffeine.) you will notice that it suppresses your appetite

**Also let me know if this is a challenge for you and what are some of the strategies you use ?**

https://redd.it/eabqr2
@r_devops

Any thoughts on New Relic One?

Sounds like it going to be the default launch page in early 2020. Anyone play around with it and have any thoughts?

https://redd.it/ea51dv
@r_devops

Anyone handling terraform exports in a clever way?

My situation is we run a fairly large scale enterprise cloud project (round 140 different applications/teams) which is very very enterprise.

We built each appstack entirely isolated which means all states are separated and secure in each teams own s3 bucket. Due to terraform very unencrypted state file.

Now it would be very handy to have remote state access between stacks sometimes, but we don’t do this, because of the security aspect.

Does anyone have an elegant way to handle this? I’m assuming some kind of pipeline -> send exports to KV store or something to allow up to date sharing of data we wish to share between stacks.

https://redd.it/ea7u63
@r_devops

What's the simplest way to run some of your code in a "cluster" fashion?

TLDR: How do I make a bit of my code work in a lightweight cluster-like fashion?

## Background

I'm using a mix of devops technologies (rancher, k8s, docker, chef, terraform, etc). Very multi cloud, multi-tenant, multi-region, etc.

Occasionally, I want some bit of my code to run in HA-style across all my clouds, regions, clusters, etc.

I can code this up myself (using some type of message bus, REST API, etc), but in the end I suspect I'll end up my own clustering software, which sounds like reinventing the wheel.

Is there a dead-simple way to take a bit of my code and "run it as a cluster"?

* All instances talk to each other
* There is one leader
* All requests are forwarded to the leader
* The leader handles all the requests & sends results back to nodes
* Data can be stored in a distributed manor

Basically like how hashicorp vault and others do clustering.

I'm imagining something like

in a loop:
if cluster.am_i_the_leader()
request = cluster.get_next_request()
result = parse(request)
cluster.save_data(result)
else
result = cluster.forward_request_to_leader(request)

Has anyone else done this for their code? Do you use a programming library/module, or use something like an etcd cluster via API?

https://redd.it/ea5go9
@r_devops

Anybody else using Zoho in AU/NZ?

Mail stopped coming in yesterday about, 0600 UTC by the look of it.

452 4.3.1 Temporary System Error

(so, no it's not a DNS issue, well not on my domain).

I also had a complaint from somebody who was using IMAP that it suddenly stopped working.

Zoho support chat is down, rang AU support number and was only given to leave a message.

Anybody else using the `mx.zoho.com.au` `mx2.zoho.com.au` and `mx3.zoho.com.au` similarly afflicted, or is this isolated to me?

I can login to Zoho app, Webmail, and even send emails from it without issue, but all receiving emails rejected.

https://redd.it/eaq62x
@r_devops

Terraform Emterprise changing pricing model

We've been on Terrafom Enterprise for a little less than a year and initially purchased a 25 user license, deployed using their suggested best practices which has resulted in 400+ workspaces. Overall we've been very happy happy with their service and software.

Then this week we had a call with their sales team that did not go well. We've been informed that licensing is moving from per user to per workspace... with each workspace license being $1000. This means a possible 10x increase in costs.

I figured I'd get the word out in case anyone else was about to walk into a similar situation.

https://redd.it/eaq8bh
@r_devops

Is this truly a good method to get into DevOps, or is there a better way?

Currently a Sys Admin. My company is looking into Azure and I think this is a great opportunity to take the reigns over and get involved in the Cloud field.

I've been studying for AWS for a bit, but kinda stopped.

Touching Azure made me wanna get into the DevOps field again. [This Reddit post](https://www.reddit.com/r/devops/comments/e6z74j/2020_one_year_challenge_to_get_into_the_cloud) that was on the front page of this subreddit seemed like a good way to actually get into it, is that true? Or do you have a better way of getting into it that isn't really addressed? Thanks!

https://redd.it/eaq2q6
@r_devops

Building Docker Images with Terraform

I just finished a boilerplate for dynamically deploying multiple cloud functions with terraform, I’m more interested in deploying to cloud run however. Is it possible to use Terraform to build an image from a Dockerfile, push that image to gcr and trigger the new image on cloud run?

Looking through the Terraform Docker provider, it looks like this might not be possible...

https://redd.it/eaou80
@r_devops

Gasper: A Platform as a Service (PaaS) for deploying and managing applications and databases

GitHub: [https://github.com/sdslabs/gasper](https://github.com/sdslabs/gasper)

Gasper is a Platform as a Service (PaaS) used for deploying and managing applications and databases in any cloud topology

It comprises of everything you need to build your own cloud all packaged with ❤️in a **single binary**

You can think of it as an open-source WIP alternative to [Heroku](https://www.heroku.com/) with super-easy setup procedures and high scalability and reliability

https://redd.it/ealcua
@r_devops

Given the tools, how do you incentivize your developers to make great dashboards/increase system observability?

A system dashboard is really valuable for stakeholders, new teammates and on-call personnel - but I realize that developers themselves don't have any real incentives to make great dashboards.. They often do not need an abstract level of overview - they already feel and navigate in more natural, low level metrics like logging, error handling software, health checks or simply just by looking at your system input and outcome with a certain level of cultural know-how.

I really like Google's approach with SLOs, which tries to align the incentives more between devs and ops in regards to reliability. Are you aware of any similar dynamics when it comes to observability?

https://redd.it/eak0a6
@r_devops

Given the tools, how do you incentivize your developers to make great dashboards/increase system observability?

A system dashboard is really valuable for stakeholders, new teammates and on-call personnel - but I realize that developers themselves don't have any real incentives to make great dashboards.. They often do not need an abstract level of overview - they already feel and navigate in more natural, low level metrics like logging, error handling software, health checks or simply just by looking at your system input and outcome with a certain level of cultural know-how.

I really like Google's approach with SLOs, which tries to align the incentives more between devs and ops in regards to reliability. Are you aware of any similar dynamics when it comes to observability?

https://redd.it/eak0a6
@r_devops

I'm confused - Build Stage for CI/CD with Php (not compiled language?)

Can someone please explain how a language that is not compiled (PHP) have a build stage in CI/CD?

[https://thecodingmachine.io/a-practical-guide-to-continuous-integration-in-php](https://thecodingmachine.io/a-practical-guide-to-continuous-integration-in-php)

https://redd.it/eawehk
@r_devops

Advice wanted for working abroad

Hi r/devops. I'm a fairly junior DevOps engineer who's been working in tech for just over two years and doing DevOps role for about a year. I'm from and currently live in the UK. The prospect of moving to another country with lots of opportunities in tech is something that really excites me and am seriously considering. Canada is somewhere I'm finding especially desirable. However I have some worries, especially in regards to experience.

To give some context, I'll just talk about what I've been through and where I'm currently at. I finished my Masters and started off as an entry-level developer in October 2017. I started to get more interested in DevOps work when as part of that role, I was tasked with writing Terraform and helping to solutionize some AWS infrastructure challenges. It really seemed to come to me very naturally and there was something about it that made me really attracted to this kind of work - never really figured out why.

It took some time for my company to move me into a DevOps role, but eventually I was moved onto another project and was fortunate enough to work with one of our most senior DevOps engineers. During my time on this project and under this guys tutelage, I have used Openshift to deploy our services, I've created and maintained many of our CI/CD jobs in Atlassian Bamboo (often leveraging Docker as part of that), I've written lots of bash scripts with a bit of Python thrown in, I've become more familiar with Linux and I've set up monitoring alerts in New Relic.

I have had very good peer feedback since I've been doing this role. Recently the senior DevOps guy rolled off, so I was tasked with being the lead (and indeed only) DevOps engineer on the project. There's more high pressure situations and it was certainly a bit scary. But again, the feedback I've had has been very good and I'm pleased with how I've settled into the role, and I'm pleased to know that I'm trusted well enough to take on this responsibility.

My main frustration is that there's a lot of decent tech that I haven't had the opportunity to use for a while now. To give some examples, I haven't done any cloud platform work since my previous project (over a year). I haven't written any SQL since I was at Uni or indeed deployed a SQL service, as we are lumped with using a really awful database on my current project. I've also not had the opportunity to work with Jenkins for CI/CD. I've also only ever worked with JavaScript/NodeJS applications as well, so I wouldn't know much at all about Maven/Gradle builds with Java for example.

So sorry that was a bit me-me-me but hopefully it gives some context! :) Can anyone offer any advice for someone in this position? This would be the first time I've ever at looked at living in another country as well, so applying for jobs abroad, applying for work permits etc. is also all new territory for me. So any advice on that area as well would also be really useful.

Thanks for reading :)

https://redd.it/eajbav
@r_devops

Convert Any Cloud Linux Instance To PostgreSQL Server.

Please checkout my blog on converting any cloud instance to psql server.

Link: [https://medium.com/@sumitkukade/convert-any-cloud-linux-instance-to-postgresql-server-ffc9faecbeae](https://medium.com/@sumitkukade/convert-any-cloud-linux-instance-to-postgresql-server-ffc9faecbeae)

https://redd.it/eaxuc6
@r_devops

[Question] How to deploy Angular App to Azure Web Apps?

My code is on Github and want to deploy via CI/CD, triggered on the master branch.

I have followed multiple how-tos and have found so far, that there are two ways:

1. Use Kudu to automatically deploy and run the app. Not my goal. I want to run the Angular App serverless, i.e. the dist folder that is generated via ng build [—prod].

2. Run a build+release pipeline. This seems more like what I intend, but I keep failing at the deployment stage.

All tutorials I have found online seem to be outdated or straight up didn’t work for me.

Am I missing anything? The only guidelines that are not frankensteined seem to be guided towards server-ful node apps.

https://redd.it/eaykks
@r_devops

Need some advice regarding Terraform

This might turn into a wall of text, but hear me out.


First, some background info for clarity:
We use terraform extensively at my current job, along with Azure for our cloud provider. We use the open source version and use a storage account in Azure for the back end/state file storage and have automated the entire plan/apply (with approval) using Jenkins. Everything works pretty well for the most part and there are few issues overall.


Now, onto the issue:
The way we use terraform currently is on a per-subscription basis (i.e. nonprod subscription has a got repo with all of the Infrastructure for that subscription in it, same for prod). They don't use workspaces. There are literally hundreds, if not more, resources defined in each of these state files.

Now I am relatively new to both the team and to terraform, but this struck me as a problem for a number of reasons. My main concerns were security (giving someone access to create Infrastructure means essentially giving them access to ALL the infrastructure in a given environment), lack of scalability, and blast radius (if someone makes a mistake, it could potentially affect all the infrastructure in a given environment). I expressed my concerns with the team and got some pushback. I suspected there may have been a lack of understanding as to what workspaces are and how they work, so I put together a poc to help clarify my points.

This is where I am confused and maybe I'm just not understanding or am way off base here. The team thinks that we don't need to use workspaces because we aren't "big enough" to justify their use. They feel that a single state file that contains all the infrastructure is the right way to proceed because as you build, it's always revalidating the entire infrastructure (with terraform plan) and if someone changes something manually in the portal, we find out right away. They have also argue that it would take way too much effort to rewrite everything to accommodate workspaces.

So, we are in the process now of upgrading from terraform 0.11 to 0.12 and are in a position where we are going to have to rewrite a lot of our terraform code as part of this effort. I suggested again that we consider splitting the code our into separate repos (even putting the terraform code in the same repo as the application that it supports) and using workspaces. Again, I was met with the same resistance. I even offered a compromise: rewrite the old code using their old method (since that was considered easier by the team) and any new infrastructure could be done using my proposal. They also disagreed with that and instead want to push on with their method and "revisit" workspaces after they finish upgrading to 0.12.


So my question to you all is this: am I not thinking about terraform the right way? I have read the documentation from hashicorps site and thought I understood their best practices, but my team seems to think that their documentation regarding best practices is "too vague" or that it doesn't apply to us because we aren't at that scale yet. I believe we should build a solution that can scale with the needs of the business, but I feel less and less certain after each of these discussions. We also want to get other teams to accept terraform as the primary method of creating infrastructure, so I'm not sure if I am approaching this the right way or not.

Sorry for the long read but really wanted to get this off my chest. Any advice is appreciated!

https://redd.it/eaxzki
@r_devops

Do you still run a reverse proxy behind an ALB?

We’re migrating from ELB to ALB. Now that we have some basic intelligent routing and we know it has strong connection management, do you still run a reverse proxy behind an ALB and why?

For reference, I’m managing ELB -> Apache -> JVM today. While rewrites are nice, it’s also easy for us to move / correct rewrites back up stream in the service. We keep domain (and https) redirects outside this setup. All that is left is proxy / connection management.

https://redd.it/eah1eg
@r_devops