I am a jr. web developer looking to move to Devops. Career Advice

I'm currently working as a jr. web developer mostly using html and css not so good with javascript or typescript. On my team I build the required webpages using html, css and my other team members will do the functionality part of the webpages using javascript. I am looking to move to devops as it involves much to little of programming and also I know how important devops is as part of a software development lifecycle so I am not worried about future in devops.

I wanted to move to devops because I am not so good with programming but can handle identifying and fixing bugs and devops mostly involves working using tools (this is what I heard from couple of my friends in devops role) correct if my understanding is wrong on this.

As most of the members here are on devops, it would be helpful for me if you could give me some insights on this.

https://redd.it/kqdgnk
@r_devops

IaaS providers in Europe

Hi All,

​

​

I'm currently looking for an IaaS provider which should be based in Europe and have European datacenters.

Would prefer not dedicated servers but at least a public cloud/VM-based solution.

The goal is for a small one-man startup launching a SaaS product.

So, based on the above, immediately out of scope are: Azure, Google Cloud, AWS, DigitalOcean, Vultr, Linode. As I said: European.

What I experienced myself until now:

\- OVH: have tried their public cloud last year - absolutely terrible with regard to stability and uptime.

\- Hetzner: tried a dedicated server there in the past, not impressed.

Others I've found until now:

\- Scaleway: looks as bad as OVH, would rather stay away from them.

\- Exoscale: looks ok I guess, can't really find a lot of info on their quality.

\- Cloudsigma: looks ok I guess, can't really find a lot of info on their quality.

\- Ionos: seems like a lot of negative info on them online?

\- Leaseweb: also looks like a lot of negative info on them online?

​

So, my 2 questions:

\- of those listed above (not OVH, not Hetzner, not Scaleway): anyone here has any positive experiences with them that they would care to share?

\- are there any other relevant European providers I missed and some here have good experience with?

​

Thanks!

https://redd.it/kqa4bc
@r_devops

Slack is down on the first work day of 2021

Looks like we're going straight into Episode 1 of the new season of 2020

https://redd.it/kqc3ga
@r_devops

What's your thought about AWS having downtime today again?

Seems like AWS had the second in a row downtime https://downdetector.com/status/aws-amazon-web-services/

Seems like like Notion, Slack, Zoom was affected by that. Seems like solutions like CAST AI https://resources.cast.ai/blog/when-aws-sinks-why-sink-with-it or not independent but still even Anthos might be a solution

https://redd.it/kqi7md
@r_devops

WebMap : A Python tool used to automate the execution of the following tools : Nmap , Nikto and Dirsearch but also to automate the report generation during a Web Penetration Testing

I am very excited to showcase my new python script : WebMap

https://github.com/Anteste/WebMap

All started from a small idea: how to automate a Web Penetration Testing .

I used to run every tool in his own terminal and it was taking a lot of time, but now with this tool you can execute all of them just with a simple command : ./webmap.py

This project is free and Open Source so use it as you want and if you have any suggestion you can submit a pull request 😉

https://redd.it/kqk645
@r_devops

DevOps Adoption

Has anyone here followed the Three ways from the DevOps Handbook by Gene Kim?

https://redd.it/kqlwx2
@r_devops

Official Salary Sharing thread for devops :: Jan 2021

Crediting this thread from /r/cscareerquestions that gets posted monthly December Salary Sharing Thread for Experienced Devs

I like to keep up to date with the current state of salaries/compensation across the world. Feel free to share your information below.

This thread is aimed at anyone from entry > Sr level DevOps/SRE/Infra engineers.

Please only post an offer if you're including hard numbers, but feel free to use a throwaway account if you're concerned about anonymity. You can also generalize some of your answers (e.g. "Biotech company" or "Hideously Overvalued Unicorn"), or add fields if you feel something is particularly relevant.

Education:
Prior Experience:
$Internship
$RealJob
Company/Industry:
Title:
Tenure length:
Location:
Salary:
Relocation/Signing Bonus:
Stock and/or recurring bonuses:
Total comp:


Note that you only really need to include the relocation/signing bonus into the total comp if it was a recent thing. Also, while the primary purpose of these threads is obviously to share compensation info, discussion is also encouraged.

The format here is slightly unusual, so please make sure to post under the appropriate top-level thread, which are: US High/Medium/Low CoL, Western Europe, Eastern Europe, Latin America, Aus/NZ, Canada, Asia, or Other.

If you don't work in the US, you can ignore the rest of this post. To determine cost of living buckets, I used this site: https://www.bestplaces.net/

If the principal city of your metro is not in the reference list below, go to bestplaces, type in the name of the principal city (or city where you work in if there's no such thing), and then click "Cost of Living" in the left sidebar. The buckets are based on the Overall number: Low: < 100, Medium: >= 100, < 150, High: >= 150. (last updated Dec. 2019)

High CoL: NYC, LA, DC, SF Bay Area, Seattle, Boston, San Diego

Medium CoL: Orlando, Tampa, Philadelphia, Dallas, Phoenix, Chicago, Miami, Atlanta, Riverside, Minneapolis, Denver, Portland, Sacramento, Las Vegas, Austin, Raleigh

Low CoL: Houston, Detroit, St. Louis, Baltimore, Charlotte, San Antonio, Pittsburgh, Cincinnati, Kansas City

https://redd.it/kqo29h
@r_devops

A structure for infrastructure repos

After working over many iterations of how to organize Infrastructure as Code in repositories, I've landed on the following general-purpose structure.

It took a lot of experimenting with different forms (many of them more complex and "fancy") to finally end on this. It's as simple as I could make it while also making it easy to reason about and solve some general problems of organizing content.

I've used it with many tools on very large-scale infrastructure, in monorepos and individual repos. I give Terraform as an example, but you can use it for deploying/maintaining all kinds of things.

# env/

This is almost exactly the same as a Unix system's etc/ directory. The difference is env/ describes environments. Know how the 12 Factor App says your configuration should be stored in "the environment"? That's this.

It's also hierarchical. Each directory is intended to "inherit" configuration from a parent directory.

The general structure I follow is env/{environment-label}/{region-label}/ .The {environment-label} typically contains the vendor, product name, and account name.Each directory is composed of json config files.

Example:

terraform plan \
-var-file env/aws-myproduct-nonprod/terraform.tfvars.json \
-var-file env/aws-myproduct-nonprod/us-east-2/terraform.tfvars.json

If you're deploying changes to just us-east-2, you can still inherit the variables that apply to all of nonprod. Your configuration is DRY, but you didn't need to do any "templating" shenanigans.

For regionless infrastructure, I use env/aws-myproduct-nonprod/all/{iam,route53,acm}/.You want to deploy regionless stuff separate from region-specific.

The default config file in a directory may be terraform.tfvars.json, and if needed provide overrides like override.auto.tfvars.json . Finally you have a root-module-name.tfvars.json file.In this way you have configuration for each of account-alias, nonprod, us-east-2, and root-module.

This way you can separate and re-use configuration at each level by just passing several -var-file options to Terraform.

To deploy all this, I recommend cd ing into a specific env/ directory and running a generic deployment command, like make plan-terraform or something. Your Makefile has the relative paths back to each config file. It would preserve those paths (like using readlink -f) , then change to one of the app/ directories below, and run terraform plan , like so:


frontend_conf=$(shell readlink -f webserver.tfvars.json)
region_conf=$(shell readlink -f terraform.tfvars.json)
account_conf=$(shell readlink -f ../terraform.tfvars.json)
plan:
cd ../../../app/tf-web-frontend/ && \
terraform plan \
-var-file $(account_conf) \
-var-file $(region_conf) \
-var-file $(frontend_conf)

The point behind this is so anybody can deploy anything without needing to actually know anything about how the deploy works. You just change to a directory and run make plan. (Ideally from a Docker container with pinned versions of tools for this repo)

Also, it's important not to reference configuration across hierarchies. Only reference configuration from your current hierarchical tree/level or below. Otherwise you get into dependency issues across environments/regions/accounts. If you have to reference something in some other environment, use something like a Terraform remote state data source.

# app/

This folder is used to store "apps". Think of them as a complete "application", like a Python module, or C program. They are intended to be "run" the same way you would "run" any other application. They have default configuration (that is overridden by whatever's passed in from env/), they take options/arguments/input, and produce output.

Basically this is where a Terraform root module would go - but not a sub-module (see below). All kinds of things can be "apps": Packer configs, Makefiles, Shell scripts, etc. Each directory should be its own complete component.

And no "apps" should depend upon other

"apps", so that there's no weird dependency issues, and you can easily reason about how each works. If you need to re-use code between multiple "apps", you put it in the next directory.

&#x200B;

# lib/

This directory is the same as your standard Unix "lib/" directory: libraries of code. No default configuration at all. It's intended only to be re-usable components that are used in "apps".

So this is where you put Terraform sub-modules, and anything else you need to keep DRY. You can depend on other lib/ directories from a lib/ directory, but keep it to a bare minimum. It's better to link to multiple lib/ folders from your app/ . This helps reduce dependency conflicts and makes it easier to reason about components.

&#x200B;

# bin/

Nothing fancy here. Shell scripts used as part of your CI/CD process, or wrappers to run tools with the above hierarchy. You can also keep them in app/ for more simplicity.

https://redd.it/kqfl5e
@r_devops

Cloudformation templates

I am using Cloudformation templates to build a Ec2 instance(https://pastebin.com/MYuc0UU1). Its very time confusing to stop and restart the Ec2 image afer I make a change to the .yml file. This is all for a AMI image creation.

&#x200B;

Is there a way to spin up my ec2 instance and execute the .yml file to ease my testing?

https://redd.it/kqli0l
@r_devops

Building an uptime dashboard for a distributed system

We have a product for which we would like to create a dashboard to show

the historic uptime and display any service outages or issues.

&#x200B;

Our service is a Rails app that is running on a handful of servers and depends on

other components like database, cache, S3 storage and some other supporting services

both internal and vendor provided

&#x200B;

Currently we are running health checks on rails app only and providing their results

in an html page. Since the system is constructed using many cooperating parts the

current html page many times says that the system is up and available but some customers

can be experiencing issues in production. It happens fairly frequently like a few times a week.

&#x200B;

We would like to build a better solution beyond polling port 80 alone and provide more

details like which component of the system is having issues and how customers are being

impacted like whether system is degraded and operating with limited functionality or

completely down.

&#x200B;

The system is running in AWS and monitoring individual components like EC2, EBS volumes,

RDS instances, Kinesis etc is very low level only engineers can consume that type of info.

&#x200B;

We like a dashboard that can be consumed by customers and executives who might not know

or care about all the components/services that the final product is built with but are very

interested in uptime.

&#x200B;

An open source solution is highly preferred and we are willing to invest engineering effort to put it together if there is nothing out there already.

&#x200B;

https://redd.it/kql7hf
@r_devops

Help:Automating installation of PostgreSQL & keycloak in Ubuntu & Windows


I am on my first job.My first task is this:Manager is asking me to automate installation & configuration of PostgreSQL & keycloak in Ubuntu & Windows.He asks for a design document.I am completely new to devops and job environment.I don't know where to start.I need to finish this today :(.
What tools are available for this types of tasks?
Should I write script (batch/shell) or should I use some tools?
Please help me get started!
Any link,advice will be very helpful! Thanks
--Noob devops guy

https://redd.it/kqrc70
@r_devops

Hosting options: Gatsby+Flask

Hi everyone!

I'm working on a Gatsby project with a Flask backend and I'm starting to think about the viability of this combination in terms of hosting options before it's "too late" to reconsider.

Heroku was the first option that came to mind and they do indeed seem to support Flask.

Even so, do you reckon it will be a smooth experience deploying a Gatsby+Flask website to Heroku?

Looking forward to hearing your insights!

P.S. The reason I've chosen Flask is twofold: learning something new and not having to reimplement the working Python web scraper I already have in JavaScript.

https://redd.it/kqqwm7
@r_devops

Solution for cloud assets inventory management.

Hi,

I am looking for solution that will help me manage the inventory of our cloud assets in Azure / AWS. R&D, QA, Solution architects, sales all open assets for POC \\ development \\ testing etc... some forget to terminate the resource after its not needed any more. i tried using tagging but its not clear enough, not centralized enough and I find it hard to enforce users who open new assets to follow the tagging policy (although i see now that AWS does provide such a thing), I thought of using a shared excel sheet where the user would fill the details, but i am sure there is a better solution

I to be able to know who is the owner of the asset

what is for? RnD / POC / Testing etc...

creation date

end date - does it have a date that it could be shutdown or stopped.

get alerts on assets the need to be closed

etc...

Thanks!

https://redd.it/kqu9sb
@r_devops

How do you set up a virtual machine with a box located on a Google Drive using Google Drive File Stream?

&#x200B;

Could not open the medium 'F:\My Drive\VM\macOS Catalina Virtual Disk Image.vmdk'.
VMDK: error writing extent header in 'F:\My Drive\VM\macOS Catalina Virtual Disk Image.vmdk' (VERRNOMEMORY).
VD: error VERRNOMEMORY opening image file 'F:\My Drive\VM\macOS Catalina Virtual Disk Image.vmdk' (VERRNOMEMORY).

I was trying to use a virtual hard disk file in order to create a virtual machine, but it gave me this error. Is it possible to make it work?

https://redd.it/kqp3jb
@r_devops

Do I need to configure ssl certs on nginx itself if nginx is inside ec2 instance, which in turn is on loadbalancer which can only be accessed using https?

note: nginx routes traffic to the app inside the same ec2 instance.

I have the ssl certs applied to the application load balancer

https://redd.it/kqjcxx
@r_devops

Tool: Product Score Card

My team manages several LOBs, each having their own Products (I define a Product as not only the Application, but all the ancillary stuff required to make it Operationalized - backups, replication, defined as IaC using Terraform, having Azure DevOps Pipeline for deployment, etc.)

One thing I would like to do, is be able to choose certain metrics that we could use to measure to and be able to give a "Score" of a Product".

For example, lets say we have Products that have Services using Azure KeyVault (the fact that the Service is using a KeyVault could be scored a 1/1 for that metric), but say we are introducing a new "feature" where all KeyVault's should be using Private Endpoint. Now all of our legacy Services would be getting a 0/1 for that new metric until they have been switched to using Private Endpoint.

Also, we have a centralized repository for our Azure DevOps Pipelines base templates, which are versioned and referenced by various Services. So if a Service is using the latest version they get a high score, but as we release newer versions of the pipeline and they continue to use the older version, their score starts to decline.

I was curious if there was a tool for doing something such as this? Bonus points if it can be automated to doing certain checks on the specific metric.

I had began tinkering with developing something myself, but, as with tools like Terraform, it would be nice if there was already a product with a set standard.

https://redd.it/kqf5nw
@r_devops

How To Drastically Optimize Your Software Team’s Workflow

Optimizing your software team’s workflow hinges on the tools you use, an analysis of where you are, and being agile in your forward progression. Read more here.

https://www.codemotion.com/magazine/dev-hub/devops-engineer/optimize-software-workflow/

https://redd.it/kqxw2j
@r_devops

Any experiences or opinions on using Sentry.io vs Azure App Insights

It doesn't appear anyone has written a comparison article so far. I'm interested to know if people have used Azure App Insights and Sentry.io for error tracking and how they compare. We're happily using Sentry.io for error capture from an app running on AWS, looking to build a totally separate app in Azure and I can see that Application Insights can do exceptions/failures management too (along with a lot of other stuff).

https://redd.it/kqzfgk
@r_devops

Tool for inserting csv for MySQL DB

I want to insert a csv file to MySQL DB after doing some basic validation. LOAD DATA INFILE is candidate, but here I need to perform some basic validations.

Column data types in csv are right data type.
Number of columns are same as db table and csv.

Should I write a custom golang or python script? or I can use open-source tool for this?

I am newbie devops.
Ref -1
Ref -2

https://redd.it/kqxoy7
@r_devops

ZAP seems to incorrectly report path traversal vulnerability in Angular app

I'm running OWASP ZAP as part of an automated CI/CD process. I am doing a spider and active scan. The report showed that there is a Path Traversal vulnerability.

This is an Angular 2 site and the javascript application is downloaded and runs in the local web browser so there wouldn't be anything revealed on the server.

\------------------------------------------------Alert Detail
High (Medium) Path Traversal
URL https://localhost:8088/Mydir/login
Method POST
Parameter usr
Attack login
Instances 1
CWE Id 22
WASC Id 33
Source ID 1

\----------------------------------------------------------Request
POST https://localhost:8082/Mydir/login HTTP/1.1 User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:84.0) Gecko/20100101 Firefox/84.0 Accept: application/json, text/plain, */* Accept-Language: en-US Content-Type: application/json Content-Length: 28 Origin: https://localhost Connection: keep-alive Referer: https://localhost/Frontend/ Host: localhost:8088

\----------------------------------------------------------Response
HTTP/1.1 200 Access-Control-Allow-Origin: * Access-Control-Allow-Methods: GET, POST, PUT, DELETE, OPTIONS Access-Control-Max-Age: 3600 Access-Control-Allow-Headers: * Access-Control-Expose-Headers: xsrf-token Access-Control-Expose-Headers: xsrf-token X-Content-Type-Options: nosniff X-XSS-Protection: 1; mode=block Cache-Control: no-cache, no-store, max-age=0, must-revalidate Pragma: no-cache Expires: 0 X-Frame-Options: DENY Content-Type: application/json Date: Tue, 05 Jan 2021 08:41:50 GMT Keep-Alive: timeout=60 Connection: keep-alive ----------------------------------------------------------------

Please let me know if you need any other information.

https://redd.it/kqxl6r
@r_devops