Reddit DevOps
268 subscribers
1 photo
31K links
Reddit DevOps. #devops
Thanks @reddit2telegram and @r_channels
Download Telegram
What is more stressful? DevOps or Software coding? (Software Developers)

Just curious.

https://redd.it/ge9jim
@r_devops
The basics of SQL injection and testing for them as an ops engineer

Hi, I wrote a post about the basics of SQL injection because I wanted to get to the bottom of it.
I'm mainly an Ops guy, and recently with all these "SecOps" talks, I thought I should get myself familiar with the basics and how to test my company's applications.

It's also an attempt (probably futile) of having developers pay "wholesome" attention to their own applications. Maybe some will...

[https://dev.to/prodopsio/sql-injection-for-developers-2pi](https://dev.to/prodopsio/sql-injection-for-developers-2pi)

Would love to hear any thoughts and feedback.

Cheers.

https://redd.it/ge1gm1
@r_devops
Very new to Jenkins and maven builds. Please help out

We have a project in git that is built either as snapshot or maven release. They are uploaded to different directories in our artifactory. We have the option to set profiles before build is triggered. I can see when a maven release is done, the profile name is tagged along to all uploaded artifacts. However, in snapshot builds, this does not occur. What can I do?

https://redd.it/ge142v
@r_devops
What are the best AWS devops courses?

What are the best AWS devops courses? I am not sure why, but most AWS courses are not enough hands-on. Ideally, everything we see on the screen should be put in code so that we don't need to be familiar with the UI on different AWS services and deploy everything to AWS and configure it by entering a simple command line to execute some scripts. Is there any course like that or is it too much to ask?

https://redd.it/gebcg3
@r_devops
Real DevOps vs Wanting to do DevOps (upper management faults)

I work at a company that doesn't really do DevOps. We basically support Corporate (internal tools, email, etc) and Production (customer / revenue generating stuff.) with some automation.

When I look at DevOps job posting, I mainly see that it concentrates on Production (based on the above assumption) and NOT corporate back end stuff. For me it's very frustrating when management wants us to do pipelines and Inf as Code but has no plan to hand off the corporate stuff because the other team that should be doing it doesn't take it on and it falls back to me. So I get told to do IaC and I am currently trying to create a Terraform Module, but then I get pulled into stupid crap with Office 365 because no one else manages it. Has anyone been in this situation and how did you get management to realize if they are really serious about IaC, they need to start leading how teams should be developed?

https://redd.it/ge08m8
@r_devops
[Article] Putting Security into The IaC Pipeline

Everyone talks about shifting security "left".. but how can you do that and what does it mean? Well in essence it's building security checks/policies/controls into the development life-cycle as early as possible, rather than bolting on security tools in production. It means validating code templates and doing security checks as early and often as possible. One great example of that is scanning IaC templates for misconfigurations and issues before build cloud infrastructure. Here is an article that I wrote on this topic and would like some additional feedback if possible for sure:

[https://medium.com/@fernando0stc/putting-security-into-the-iac-pipeline-4de98f88ad24?source=friends\_link&sk=d628526e4758cde258901447c7453f4b](https://medium.com/@fernando0stc/putting-security-into-the-iac-pipeline-4de98f88ad24?source=friends_link&sk=d628526e4758cde258901447c7453f4b)

https://redd.it/ge8bsg
@r_devops
Let me tell you why I think GCP is better than AWS and you tell me where you agree or disagree.

​

>This post is not just a rant about my predilection for GCP. I genuinely want to read your opinions, especially those of you who have used both platforms like me (I have about two years of experience with each platform). And if you feel I have said something that you think it's wrong or not factually accurate, also please let me know, I am happy to be educated or informed in a civil
and constructive manner. I have no affiliation with Google whatsoever, this is entirely my opinion based on my perceptions of the merits of both platforms. I am also conscious that there are more tools and services that I could ever expect to use in a lifetime so it is entirely plausible I am missing areas where AWS by far outshines GCP due to my limited experience or missing even more areas where GCP is better.

​

* Where do you agree?
* Where do you disagree?
* Can you provide examples of your own where AWS or GCP was better than the other?

​

**Ikea for Cars**

If AWS and GCP were both car companies and you wanted to purchase a car, AWS would give you the wheel, a chunky verbose manual and the keys and then tell you to go to twenty different shops they also own to get the rest of the components and ask you to put them together yourself the best you can. Sure, maybe you can hire a service and get tools to automate this part, but it still falls on you to assemble these components together and maintain the automation.

The experience of GCP on the other hand is more like collecting the car keys and driving off from the parking lot, with the option of dismantling and customising the car if you wish, but the default is a fully built functioning car so you can achieve your objectives, which is driving around, not assemble the car.

My first experience working with AWS, before I had much to compare it to, was brief and I didn’t like it; I felt the interface and the way tools and settings were organised was counter-intuitive and weird. 

For example assigning a static ip to a server was just bizarre, I kept looking for ways to assign the static ip without knowing that it was meant to be called elastic ip and hidden away in a separate set of menus. Then these elastic ips were part of a different pool of ips than the ones that were assigned dynamically. To my dismay I had to stop a production server to change the ip and also change the DNS pointing to that new ip, this was because my predecessor hadn’t assigned an static ip to the server, my bet is that he probably gave up after ten minutes trying to figure out that it was called an elastic ip.

My second experience working with AWS was after a year and a half working with GCP and now by comparison I really couldn’t stand AWS, it took me a few months to get accustomed back to use it and I remember that in my first few weeks I actually considered quitting and just accepting roles with GCP.

It’s not that AWS is harder to use than GCP, it’s that it is needlessly hard; a disjointed, sprawl of infrastructure primitives with poor cohesion between them. A challenge is nice, a confusing mess is not, and the problem with AWS is that a large part of your working hours will be spent untangling their documentation and weeding through features and products to find what you want, rather than focusing on cool interesting challenges.

Let’s just go over a few of the things that make AWS such a pain to use and how it compares with GCP. 

**Accounts vs Projects**

One of the first differences that strikes you when going from GCP to AWS is accounts vs projects. In GCP you have one master account/project that you can use to manage the rest of your projects, people log in with the google account and then you can set permissions to any project however you want. So you can have a dev project, a production project, etc. All of this works out of the box and there is absolutely nothing additional for you to do. 

In AWS you have accounts, and each account has a separate set of users. There are
ways to connect these accounts so your user has permissions on other accounts. One way of doing this is creating a master users account and then adding roles that can be assumed in all other accounts by this master account. 

This is not only a pain to set up, it’s very painful to use as well. For example when using terraform scripts you need to coordinate multiple roles across several modules if you need to work across multiple accounts. 

**Command Line Interface Tools (CLI tools)**

Let’s just compare what you have to do in order to use GCP cli compared to AWS provided we are using 2FA and a couple of different projects/accounts. 

In GCP after you [install the Google SDK](https://cloud.google.com/sdk/docs/quickstart-linux), all you need to do is run gcloud init, which redirects you in the browser to a Google login page. Here you can login with your two factor authentication (which if you have an android phone is as easy as unlocking the phone and pressing okay) and you are done. Your login session is attached to your Google session so when you kill this session you are logged out— very simple.

In AWS you need to create a token that you can use to login with your CLI, simple enough, right? But now we want to use two factor auth, and this is where the fun begins.

After you login with your token you then need to [create a script](https://github.com/asagage/aws-mfa-script) to give you a 12 hour session, and you need to do this every day, because there is no way to extend this.

Okay, but that’s not a big deal, you say, after all it’s just a code that you need to input once a day and you can get on with your day after that.

But wait, there is more! If you need to assume roles in another account, you need to create yet [another script](https://github.com/Integralist/Shell-Scripts/blob/master/aws-cli-assumerole.sh) that creates another profile for you to use.

That’s one step plus two scripts, plus many steps in between. And sure, you can automate much of this or use someone else’s tools you find online (that you most likely will need to tweak), but why? Why do we have to do so much work to use AWS? Why can’t AWS abstract away this pain away from you in the way that Google has done?

**Web User Interface**

If using the CLI is too painful for you, you can always log in to the portal and use their user interface, although I don’t recommend you do this for everything, in fact I recommend you use it the least possible and only for reference and to check status of your services.

AWS interface looks like it was designed by a lonesome alien living in an asteroid who once saw a documentary about humans clicking with a mouse. It is confusing, counterintuitive, messy and extremely overcrowded. 

I can’t even count the times I’ve gotten lost or stumped in the AWS console, sometimes over the most stupid details, like missing that there was a next button hidden on a weird corner. Or trying to use search bars that can only search prefixes (WTF?)

But the biggest frustration I have from the AWS console is how you are always overwhelmed with scores of settings and options you need to fill in before actually provisioning anything.

One example that comes to mind is when someone at work said we should use codebuild/codedeploy to replace Jenkins for ECS deployments. The first engineer tried, he got stuck, the second engineer tried, he got stuck, I tried for hours and I got stuck… in the end I just gave up for lack of wanting to spend any more time on a tool that doesn’t seem to be that popular for CI/CD that I thought was meant to make life easier. 

Amazon seems to be particularly terrible at interfaces in almost all of their products though. For example in my Smart TV the Netflix app works flawlessly and is intuitive to use whereas the Amazon Prime app is an abomination, you are constantly accidentally pressing the wrong button or getting lost or the subtitles are often out of sync. 

In a rant that a Google engineer who had worked at Amazon [wrote a while back](https://gist.github.com/chitchcock/1281611) he explained the issue with
Amazon and Bezos not understanding interfaces (or is it human interaction?)

>Jeff Bezos is an infamous micro-manager. He micro-manages every single pixel of Amazon’s retail site. He hired Larry Tesler, Apple’s Chief Scientist and probably the very most famous and respected human-computer interaction expert in the entire world, and then ignored every goddamn thing Larry said for three years until Larry finally — wisely — left the company. Larry would do these big usability studies and demonstrate beyond any shred of doubt that nobody can understand that frigging website, but Bezos just couldn’t let go of those pixels, all those millions of semantics-packed pixels on the landing page. They were like millions of his own precious children. So they’re all still there, and Larry is not.

GCP’s user interface is on the other hand very intuitive to use and whenever you want to provision anything you are given [sane defaults](https://en.wikipedia.org/wiki/Convention_over_configuration) so you can deploy anything in a couple of clicks, I have never gotten lost using GCP or needed to consult a million pages of documentation to find out what I needed to do.

This however does not mean that GCP is taking away from you the power to configure things to an intricate detail, it just means they are giving you an example of a working configuration that you can then tweak to your purposes.

There are also other things you can do from the UI in GCP that either work really badly in AWS or are non-existent. For example you can easily open a terminal and ssh into any instance you have spun (provided you set permissions for it) and it works really well.

Another feature you have in GCP that I absolutely LOVE is the ability to view the CLI command that would do whatever settings you have in the console. That makes learning the cli so much easier, it’s far better than scouring the net for examples on how to do anything or trying to make sense of AWS’s gorgeous documentation…

**Documentation**

You can forgive the documentation in AWS being a nightmare to navigate for being a mere reflection of the confusing mess that is trying to describe. Whenever you are trying to solve a simple problem it is far too often you end up drowning in reference pages, the experience is like asking for a glass of water and being hosed down with a fire hydrant.

Great documentation is contextual, not referential. If you wanted to learn how to cook a dish, you don’t want someone to point you to a list of ingredients, you want a recipe describing how to use them, and this is where AWS documentation too often fails; it exhaustively describes everything that they have, but they don’t always do a good job at putting the documentation into context.

To be perfectly fair to whoever is tasked to document anything in AWS, it is a lot harder to document something that’s confusing and messy than something that’s simple to use. Extensive and overly verbose documentation is often a sign of complicated and over convoluted software or processes, so in this sense Google Cloud already has an advantage to begin with.

The documentation in GCP is generally clear and concise, and while it may not always be perfect I generally found it useful and to the point. If you want other good examples of great documentation look at DigitalOcean — they are great. 

**GKE vs EKS**

If your intent is to use Kubernetes, don’t even bother with AWS, their implementation is so bad I can’t even comprehend how they have the gall to call it managed, especially when compared with GCP

In GCP if you want to spin a cluster, no problem, just a couple of clicks and you are there. The defaults are easy and sane and the entire product feels very cohesive with all the ugly, tedious bits abstracted away from your experience. 

With GKE you don’t need to join the nodes, you don’t need to plan for an upgrade of these nodes either, it’s done automatically or with a couple of easy clicks, and this does not mean you are sacrificing complexity. You can customise a lot, but when presented with sane, simple defaults, it’s a
lot easier to understand a product that when being overwhelmed with a barrage of options and trying to figure out how everything fits together as it’s the case with EKS.

Spinning an EKS cluster gives you essentially a brick. You have to spin your own nodes on the side and make sure they connect with the master, which a lot of work for you to do on top of the promise of “managed”

And yes, I know that there are official terraform modules that take care of most of this work for you and make the job a lot easier and there is also a tool called eksctl developed by weave works which is great, but these aim to simplify a complex solution that should have been abstracted away by AWS by design, not rely on others to make sense of the mess with complex scripts and tools.

Even if you use those tools to create your automation on top of AWS, the fact remains that there are a lot of moving parts underneath that you will always be responsible to orchestrate and make sure that are working and up to date. eksctl for example uses cloudformation templates in the background.

**Product Overload**

At the time of writing this, there are 169 AWS products compared to 90 in GCP. AWS has been around for longer and therefore they have more offering, and in good Amazon spirits, they constantly and aggressively are expanding this offering to give you more of what you may need (and a lot of what you don’t need)

This sounds like a good thing, until you start seeing the amount of half cooked products or [near duplicates they have](https://www.forbes.com/sites/janakirammsv/2018/01/08/aws-service-sprawl-starts-to-hurt-the-cloud-ecosystem/#20aee5455c1f). One good example is Parameter Store and Secret Manager, the latter offers almost identical functionality except for a couple of extra features that you need to pay extra for, which begs the question, why not consolidate these products to avoid confusion and time wasted by the users investigating which one they should be using?

GCP on the other hand has fewer products but the ones they have (at least in my experience) feel more complete and well integrated with the rest of the ecosystem, and choosing one product over other doesn’t become an agonising choice that requires extensive research (okay, you still need to research, but not nearly as much)

I used to mock Apple in the past for how limiting they were and how very few features they had compared to Windows and Linux Distros until I started using a Macbook, it was then that it became so clear to me that having an opinionated approach on a few products and tighter integration of the various components often yields a far superior and more stable experience which is similar to the experience that I am having with GCP vs AWS. GCP gives you less, but what it gives you is far better integrated, simple to use and works better than its AWS counterpart — so unless you have a very compelling feature in AWS that you are missing in GCP, you should seriously consider picking GCP.

**AWS is a lot more Expensive**

AWS charges substantially more for their services than GCP does, but it also has another hidden cost attached to it; expertise and manpower. 

With GCP, a relatively inexperienced engineer in platforming tools can pick it up and get his work done in a relatively short time because most of the tedious tasks of piecing all the parts together have been done by Google already. Also using GCP is substantially more intuitive so even if you have no previous experience on cloud platforms you will pick it up very quickly.

A task that may take you a day or less to do in GCP, you may spend a week to do the same thing in AWS. One example I can give here is Endpoints. I was working with a terraform cluster installation and I wanted to restrict outbound traffic to the internet. The problem is that if you do this then you are also cutting off traffic to AWS, in order to address this problem you need to set up endpoints. Endpoints essentially allow you to connect to AWS via the AWS intranet as opposed to the internet (Don’t ask me why cloud providers don’t do this
by default, it makes no sense to me).

So simple enough, I just add these endpoints and then my job is done. Problem is that I was working with a cluster provisioner in terraform with a lot of moving parts and using multiple AWS services and you cannot set up an endpoint that applies to all AWS services, you can only do one endpoint per service and I had to do a lot of digging trying to figure out exactly all the services that the provisioner was using and add endpoints for each one of them, every time I added an endpoint, I found out I had to add another endpoint, I ended up adding about five of them and then I found out that a couple of the services that I was using didn’t have endpoints for them, so in the end I just had to allow outgoing traffic via a NAT. 

Out of curiosity I investigated how to do this in Google Cloud because I had never done it before, just to compare how difficult it would be in comparison to AWS and I wasn’t surprised to find out that you can accomplish the same thing just by [clicking on a checkbox](https://cloud.google.com/vpc/docs/configure-private-google-access) or activating a setting and it applies to all services Also, doing this in GCP is free whereas in AWS you have to [pay for each endpoint](https://aws.amazon.com/privatelink/pricing/)

The above is just one example, but I have found that generally any task that I want to do in AWS requires far more energy and effort to do than GCP, meaning you are probably going to need to hire far more engineers and need more time if you are using AWS than if you are using GCP. 

**The Cost of interrupted Flow**

Another significant cost to your organisation if you decide to use AWS is the continuous interrupted flow. [Flow](https://en.wikipedia.org/wiki/Flow_%28psychology%29) is the state where you ideally want your engineers to be a good portion of their time at your company, not only they will be much happier, they will also be a lot more productive. 

The problem with using AWS is that because everything is so confusing and complicated to use you will (at least at the beginning or when you need to apply a significant change or embark on another project) have to spend a lot of time reading documentation and testing to figure out how things work, and the irritating thing is that it won’t be fun experimentation, it will be tedious and trivial issues that should not exist, like the endpoint issue I described above. 

**Performance**

I am not going to do extensive testing in both platforms and post benchmarks for this article since it’s a lot more work than I want to spend on this but I’ll just say that in my experience I felt that performance was almost always better in GCP, for example copying from instances to buckets in GCP is INSANELY fast, I remember being shocked by this because in a previous job I had to do a lot of hourly backups to buckets of large chunks of data in AWS and I always felt the copying was slow, but this was not the case at all for GCP.

This is a bit old but some articles by [this guy](https://thehftguy.com/2016/06/15/gce-vs-aws-in-2016-why-you-should-never-use-amazon/) may still apply today.

**So what’s better about AWS?**

As I mentioned I think that AWS certainly offers a lot more features and products than GCP does, and you may benefit from some of them. You can certainly do more with AWS, there is no contest here. If for example you need a truck with a server inside or a computer sent over to your office so you can dump your data inside and return it to Amazon, then AWS is for you. AWS also has more flexibility in terms of location of your data centres. Other than that… I would chose GCP any day, and I think GCP will cover the vast majority of your cases.

**But wait, there are lots of third party tools to automate AWS**

Yes, like the aforementioned eksctl, some of them do an amazing job at this but they are still third party tools. I firmly believe AWS needs to work a lot on their abstraction of needless complexity.

**So if GCP is so much better, why so many more people use AWS?**

AWS has been around a lot longer.
Also the amount of engineers who are certified in AWS is substantially larger as well. When people start looking around to get into DevOps everyone is shouting AWS as must have on your CV so everyone is scurrying to get certified. 

Imagine if you are an engineer with 5 years of experience in AWS with lots of money and effort spent on AWS certifications and you are tasked to do infrastructure at greenfield project, what are you going to chose? Not GCP. 

On the flipside it seems a lot more companies are making the switch to GCP. I’ve been hearing from previous colleagues who told me that applied for some roles who were migrating to GCP and when they mentioned they didn’t have any experience in AWS their reply was “No one has” So it may just be a good time to learn GCP for a change. 

So based on that account, and given that everyone is racing to get AWS certifications, you may just be better off doing the opposite and take on GCP because you may against less competition. 

**But this AWS complexity is creating so many jobs for us!**

Yes, and it does sound like I am shooting myself on the foot by posting this because this is my job but I do have to admit that AWS does create a lot more jobs for DevOps, SRE, Sysadmin, Platform Engineers etc. due to this complexity and the lack of desire for the developers themselves wanting to tackle this complexity. 

But then again I embrace change, if I have to learn other skills so be it, and I still think there is a lot of room to grow a career with GCP, it’s just that your work will hinge more on the interesting stuff, with tedious cloud platform bits abstracted away. 

**So I should pick GCP then?**

No, you should pick whatever fits your needs. If you are a small company or an independent developer I totally recommend giving these two a miss and go with DigitalOcean or Linode or some of the smaller companies which are even easier to use and will cover your most basic needs for less money. 

AWS is still my second choice as an enterprise cloud provider after GCP. I know there is also Azure, but many years of using Microsoft crap products (with the exception of VScode) really has put me off using anything they make, so I always to try to give Azure jobs a miss if I can help it, so I have no opinion on their platform other than a healthy dislike for the company behind it ;-). 

https://redd.it/ge79zy
@r_devops
bazel build docker: Y/N?

I'm looking at setting up a system which uses Docker containers for reproducible test runs, which will be, if all goes as planned, used for backward compatibility for as much as a decade. (We have some hardware pipelines to support, and they don't update what they install on the hardware for years.)

I haven't used Docker much in the past, and found some strongly-worded claims saying that using the `docker build` executable is a bad idea. Their primary suggested alternative was https://github.com/bazelbuild/rules_docker.

In brief, their criticisms were:

* The `docker` executable demands root access for things that shouldn't need root access. Running any subcommand of it without sudo can still execute some side processes as root.

* `bazel build docker_*`does not need root to build/get/publish/modify images. It does not have a `run` subcommand.

* Reproducibility: Layers of a Docker container have random identifiers, making them less than perfectly reproducible

* bazel-created containers label layers with a SHA hash of the layer contents, making them deterministic


These seem like good points, particularly the first one, but there don't seem to be enough users of the bazel version for anyone to have written the counterpoint. (At least, nowhere I can find.) Anyone here know it?

https://redd.it/ge5pwp
@r_devops
How do you deal with Freeze changes?

At our org, we are currently on a "change freeze" because of CV19 reasons. During this time, all deployments to our production infrastructure has have been paused, which has resulted in a huge discrepancy between our staging and our production environments, when normally, they would only be at most one or two changes in difference. Currently, our deployment strategy consists of just updating a deployment in k8s with the latest build, but since, alot of our k8s manifests have been updated (additional sidecars, additional supporting features), which while work, and have been declaratively defined, I'm worried about everything going to shit. Am I just overthinking everything, or should I go about the post-freeze mass-deployment another way?

https://redd.it/ge4hp8
@r_devops
your favorite kubernetes 3rd party software?

I have been using tigera heavily for securing security groups on EKS, and was wondering if you guys pay for any 3rd party vendors in your kubernetes environment?

https://redd.it/ge440y
@r_devops
Working Through a CI/CD Pipeline

I've been working in the "DevOps" space for a decade now and when people ask about our deployment practices I get a wide range of reactions from people. So I finally wrote up the stages of CI/CD pipeline. Starting from manual deployments to a single server, to full infrastructure deployments. Feel free to let me know if I missed a section or have more questions about moving from one step to the next.

https://medium.com/devops-dudes/working-through-a-ci-cd-pipeline-2a75525eb486

https://redd.it/ge2s2b
@r_devops
How to tell Helm which repo & chart is going to install?

I'm following Helm's document [here](https://helm.sh/docs/intro/using_helm/#helm-install-installing-a-package) to install a package, a.k.a. a helm chart:

$ helm install happy-panda stable/mariadb

It's really convenient to have a ready-to-use package. But in case of a package doesn't belong to \`stable\` repo, for example \`appmesh-grafana\` then I have to do more than this:

$ helm repo add aws https://aws.github.io/eks-charts
$ helm install aws/appmesh-grafana --version 0.1.0

Therefore it forces me surf around the internet to find repo url, package name, etc. I don't want to do it every single time!

Is there any way to specify the repo and chart to install using \`Chart.yaml\`? I prefer Chart.yaml than shell script.

https://redd.it/ge2p2v
@r_devops
Windows 10 Pro question - managing RDP sessions

This is a super n00b question:

I have 16 windows 10 pro systems used for long running data preparation processes (think image type conversion and data extractions). There are two employees, one in the US, one in Israel that work on these conversions, and they use their time zone difference to their advantage. Today, they use a single blanket account to RDP into the server and start a conversion. They grab each others sessions to keep the process running (the tools aren't fond of running as a service).

From a security perspective, this is awful. We have no way to audit actions on the servers, and the data is somewhat sensitive.

I would prefer to give them domain accounts, but then they can't grab each others sessions.

Do you guys have a method for managing this scenario? Should I upgrade them to windows server 2016 and use RDP session shadowing? Other ideas?

https://redd.it/ge2k2n
@r_devops
SRE vs Architect

Hey all,

I’m a current security engineer trying to figure out where I can grow into my next role.

I am versed in the world & tools of devops so SRE sounds like a good idea but unsure if SRE is a lateral move vs a vertical one. Does an SRE compare to architect in terms of day to day tasks and certifications?

If anyone can also point me to some certs a SRE or architect should be getting I’d appreciate it. B

https://redd.it/ge033z
@r_devops