Insight About starting into a New Environment

Hello there !

I will be starting a new job as a DevOps Engineer in a month (Pretty exited/scared !).

This will be my first "real" DevOps experience. I am living in France , and studied IT for 3 year , with an "Internship" system , I studied 20% of the time , and was working in a company the other 80%.

So this will not be my first working experience , nor my first DevOps related Job.

But I will work in a relatively young Company and I will be the only DevOps.

As said before the company is very young (created about 6 month ago I think) , and my job will be to setup the baseline of the future architecture , everything need to be done , they currently have a test environment on a handful of EC2 , but nothing more and everything need to be set ( the Architecture , CI/CD,Monitoring, ect...)

So I am here to get insight of real SRE/DevOps engineer (whatever you are called in your company !) , that went down the same path (or not , I am eager to learn , and any insight is good to have !)

What are for you , the bid DO's and DON'T while thinking and building a new architecture from scratch ? Or what are for you, the best practices ? Do you have any resources (Book , article , or else) that you consider usefull in this situation ?

https://redd.it/evohki
@r_devops

How to automate Github releases with drone

I recently switched to automatic Github releases and wanted to share the journey.

Hope it helps :)

[https://blog.nicco.io/2020/01/29/automate-github-releases-with-drone/](https://blog.nicco.io/2020/01/29/automate-github-releases-with-drone/)

https://redd.it/evnhay
@r_devops

JMX APM solutions recommendations?

Hi. Can you show me some free or paid solutions to collect application metrics from jmx or javaagent? We are currently running on NewRelic, but it\`s pricey. We run our apps within tomcat and spring on VMs in GCE. We have plans to migrate all the stuff to GKE, so it would be great if the metrics system supports this. Moreover, we are switching our logs to Stackdriver, so it would be nice if the APM supports this.

Right now I\`m looking at Apache Skywalking, BindPlane(supports Stackdriver), Elastic APM, DataDog.

https://redd.it/evlptj
@r_devops

Improving Packer builds - or use something else

I'm a former Linux-sysadmin transitioning to Cloud-DevOps in my current role of the last 12 months. 20 or so developers in the company with limited DevOps experience/mindset between us all. Core product is a niche java application.

Main CI/CD is jenkins in AWS using EC2s for testing along with a couple of legacy onprem servers.

We fundamentally use Packer to (manually) build Centos and windows 2008/2012 AMIs for the development teams. The linux AMIs are configured via Packer-Puppet provisioner (changing to Ansible w/local-exec soon). The Windows AMIs are configured in Packer via PS + scripts. A huge number of dependencies (testing frameworks, applications etc) are baked into the AMIs, so the AMIs can get quite big (50+ GB). The AMIs are plugged into AWS jenkins-EC2 and all our testing is done from there. When it works, it seems to work well enough.

Because the AMIs are so big, manually troubleshooting packer-build issues with them is tedious and time-consuming and taking up an obscene amount of time. I try things like testing with virtualbox-iso first, or RDP’ing into the currently running packer-build to see what’s going on and so on - but again it is a very manual process.

Specific problems are:

1. Packer is used manually - it needs to be run manually from the command line whenever the AMI’s need updating. We do have a Jenkins-script that can invoke the packer build, but it has to be manually triggered.
2. Windows AMIs seem a nightmare to debug issues with.
3. It seems to be extremely tedious to debug even simple issues with Packer.
4. The Packer - AMI build process is not part of any CI/CD - it’s all done manually.


Specific Questions I have are:

1. What alternatives exist to build AMIs?
2. Should we be in the ‘Packer Build AMI’ mindset at all? Could/Should we be using CodeDeploy/Other tech ?
3. How do others handle big AMI builds ?
4. Can we split the Packer-build into more manageable, discrete building blocks ? Has anyone done this with Jenkins-Pipelines ?
5. What tools/tech is available to automate and test Packer-builds ? I know of serverspec, but it seems to be more for the ‘test the configuration of the AMI itself’ rather than ‘build and test the Packer-AMI building process’ ?

We are basically at the limit of what we can achieve manually. There’s no room for any further manual builds or AMIs or so on. We need to drastically automate Packer, or consider an entirely different way for this process.

https://redd.it/evkzm0
@r_devops

Development environment workflow suggestion/recommendation

hey folks.

I was looking for some suggestions for a development workflow that I have been exploring for some time now.

For my work, I tend to setup a lot of different environments for a number of things to test. Specifically, things like:


* load balancing
* reverse proxy
* CDN testing methodolgies
* web server
* app server (nodeJS, python primarily)

Depending on what I am working on, I could spin up as few as 1 VM, to as many as 6-8, depending on what is needed. Ideally, these environments would be things that I could put together, destroy and share with other team members as needed. Also, the need to deploy this to a infrastructure (cloud, vsphere) is also another important aspect I need to take a look at.

I have been exploring using hashicorps tool suite. Specifically, testing and using:

* packer
* vagrant
* terraform
* consul
* vault

I have started to build packer images that work with several providers. Works great and gives me flexibility. It gets me what I need going, then I use ansible to come in and do additional configuration as needed.
My initial workflow was something like: packer -> ansible -> terraform

I still need to spin up VM's to test obviously, so this is where I started to explore vagrant.

What i wanted to ask the folks here is maybe for some suggestions/recommendations or things to consider for a optimal workflow.

I am just trying to visualize and put together the entire workflow. I just started to test vagrant, but trying to see how that would fit in correctly.

On a side note, I do use docker quite a bit, but think I could extend and make my environment better with some suggestions.

Hoping that makes sense.

Much appreciated my friends!

​

Cheers,

TCG

https://redd.it/evtoic
@r_devops

Whats new or viable in Configuration Management outside Puppet, Chef, Ansible or SaltStack?

Is anybody working on configuration management systems outside of the old quadfecta? Is anybody trying anything new? OpsMop shut down because of lack of interest, but whats new, exciting or even viable outside of the typical four?

https://redd.it/evucfw
@r_devops

Is there a good tool that scans you datacenter to catalog where PII information is stored?

Is there any product that scans your datacenter and tries to catalog all places where PII is stored? Things like a list of databases/tables, pipelines with messages containing PII, logs, server filesystems, etc. We are happy to give the tool read only credentials/user permission to each db/server/cloud service in order to create this sort of list.

Is there a tool/product that exists for this type of job? I am not looking for PII scanning of user facing tools but specifically for all things created by the engineering organization. Thanks.

https://redd.it/evtfi3
@r_devops

How I Organize Terraform Modules: OFF the Beaten Path

I wrote up my somewhat non-standard approach to organizing terraform modules, on my company's blog


[https://blog.smartlogic.io/how-i-organize-terraform-modules-off-the-beaten-path](https://blog.smartlogic.io/how-i-organize-terraform-modules-off-the-beaten-path)

​

I've found this approach really useful for managing the complexity of large and \~living (in the sense of frequently changing code) modules. And mostly, coworkers seemed to have liked it too. As I mention in passing in the post, I mostly use this approach for larger helm charts as well.

But I'm curious what y'all think of it? Anybody else go about it this way? Are there downsides I am overlooking?


Anybody else do something that is neither like what I describe as standard herein nor like my approach?


Thanks :-)

https://redd.it/evsf3q
@r_devops

Jenkins in Kubernetes - Running Docker Builds?

Hello all --

We are moving to Jenkins in K8s wherein different teams can 'own' their own instance of Jenkins and configure builds as they please. I've created some basic agent pods that allow teams to do the basics, but I'm struggling to figure out how we can perform docker tasks easily.

It seems our only options are to run dind (which appears to require root?) or run a container with docker client and mount the running docker socket from the K8s worker node (also requires root). Our cluster has a pretty strict security policy which disables containers from running as root.

Does anyone have any workarounds for this? Really trying to avoid static VMs for agents.

Thanks!

https://redd.it/evylfn
@r_devops

Need user control with AWS EC2 instances?

Here's a guide on doing it with jumpcloud:

[https://medium.com/teamzerolabs/secure-and-manage-ssh-access-with-ldap-sssd-and-jumpcloud-5dd7156a337](https://medium.com/teamzerolabs/secure-and-manage-ssh-access-with-ldap-sssd-and-jumpcloud-5dd7156a337)

We have been deploying this pattern with several clients and it definitely is better than adding everyone's pub keys to ec2s by hand.

This will also cover making ami images to make sure they self-enroll with jumpcloud ldap as well

https://redd.it/evvsck
@r_devops

Redirect a URL but only if coming from specific origin

Running Jenkins behind NGINX. Trying to come up with a way to redirect to Blue Ocean UI (`/blue`) when hitting the main page, but still allow access to the classic interface when specifically requested.
Right now I have something like
```
location = / { return 302 $request_uri/blue; }
```

But I want to only do that if the request doesn't from `/blue.*`.
Maybe some kind of conditional on the value of `Referer`?

Thanks for any help.

https://redd.it/evr9vg
@r_devops

Interview Project

Hello Everyone!

I'm in the final stages on an interview process with F5 and they have assigned me the task of creating a simple application and putting it into Github within containers....

Problem is I have a networking background and that's it.

I am looking for any info on how I could go about doing this. I have a friend who can assist me in creating the application so I'm mainly focused on the Docker side of the problem. Thanks in advance to anyone who assists

https://redd.it/evzupw
@r_devops

Replacing Fabric for updates

Hey folks,

Ive been using fabric1 for updates on my servers running Django (ssh to server, git pull and restart uwsgi). Ive got 3 severs behind load balancer, so the process is I stop traffic to one, update it with my fabric script then turn it on again.. and so on. Is there any tool thats not overly complicated to configure that would allow me the same workflow?

https://redd.it/ew1bij
@r_devops

[loooking for recommendation] Devops conference

Hello all,

I work as a devops cloud platform engineer junior. I have basics of networking and automation (I started working with Puppet some time ago).

I got an opportunity to attend any devops related conference from my employee all around the world.

I prefer conferences about networking and automation for beginners, ideally in Europe.

Cost does not matter, could you please recommend me some conferences? ☺️

https://redd.it/ew2old
@r_devops

strategy for building ec2 instances

I am new to aws. I am building few hundred large instances. Most will be the same expect for one. At the moment I am using a shell script to create the instances. Its laborious.

To create a AMI, I use a stock AWS image, login to it and run bunch of commands and create my personal AMI. To run bunch of those commands, is it better to use Ansible or some other tool? Once the AMI is done, I launch them thru Ansible ec2 module. It works great. I am wondering if there is a better way to do it. I want to make sure I am not overlooking anything.

https://redd.it/ew4cpk
@r_devops

Experience with RunMyJobs

Just wanted to share my experience, we recently bought ourselves a subscription on the "industry standard" cloud based workload automation solution, "RunMyJobs" (TBH, I had never heard of it before).

I'm genuinely impressed. This solution supports integration of a whole lot of platforms out of the box and has agents for pretty much every platform out there (ranging from HP-UX, AIX to Solaris to Linux etc) has support for pretty much every scripting language out there and support Java (called RedwoodScript in the product, which is basically Java with some magic added), Web-Services (SOAP sadly, not REST).

I have been using this for a couple of weeks now and converted the majority of my cron jobs, batch processes, Docker deployments and am currently working on migrating a bunch of SAP transactions to it .. so far, we're really impressed with its capabilities.

A downside, which we are struggling with is the limited alerting capabilities that are available by default. Only email seems to be there out of the box, however it should be possible to integrate / alert to other services through RedwoodScript.. still figuring this one out.

https://redd.it/ewa05m
@r_devops

Thoughts on CSPM tools??



https://redd.it/ewa0mb
@r_devops

Looking for insights on private domains, public domains and TLS on AWS.

Hello DevOps colleagues,

I am asking for insights about private domains, public domains and SSL on AWS.
Our entire AWS setup is completely internal - no public subnets, no public IPs. The VPCs/subnets are connected via TGW+VPN with our company network for access and internet. We currently have a private Route53 domain for each env, lets call it "env.my-private-domain". Each of the 5 envs as it's own AWS acc.

Our company DNS is forwarding "env.my-private-domain" to the corresponding AWS acc and it's working fine for now. But we will have to use TLS in the future. :)

We are running internal workloads on Fargate. Users will have to connect from the company network to webservices hosted on Fargate. Wether we run SSL end to end or terminate SSL inside AWS via LB or NGINX is not a problem yet. Because we need to have the option to use TLS first.


Going towards TLS leads us to two options:

A) Keep the private domain "env.my-private-domain"

But this gives us new problems.

* We could use AWS ACM to build a private CA, but it costs 400$ a month + for every cert. Each env is a seperate AWS account with would lead us to 5x400$ per month just for a CA in each env. -> This is too expensive for us.

* Host our own CA with Vault. -> We want to avoid the hassle of managing Vault and hosting our own CA.

B) Switch to our public company domain "mycompany.com".

I am not complete sure if this would work. We ask internal IT to allow us to use a subdomain like "department.mycompany.com". We create a Route53 domain for each env like "dev.department.mycompany.com". We ask our internal IT to forward domains like "dev.department.mycompany.com" to our corresponding Route53 Inbound Resolvers. From here we would like to use Let's Encrypt wildcard certs. I think it should work because we using a public domain and should be able to prove that we own mycompany.com. Also Let's Encrypt started supporting wildcard certs in 2018.

We think about such a setup. We use a wildcard Let's Encrypt cert for every env. Something like "*.dev.department.mycompany.com"
So we can use the same wildcard cert for multiple different services/versions in a single env:

Wildcard Let's Encrypt *.dev.department.mycompany.com

* v1-serviceA.dev.department.mycompany.com
* v2-serviceA.dev.department.mycompany.com
* v1-serviceB.dev.department.mycompany.com
....

Wildcard Let's Encrypt *.staging.department.mycompany.com

* v1-serviceA.staging.department.mycompany.com
* v2-serviceA.staging.department.mycompany.com
* v1-serviceB.staging.department.mycompany.com
....

Do you any remarks or insights for option B ? Thank you in advance :)

With best regards

https://redd.it/ew3twx
@r_devops

Automate Building Docker Base Images for Ruby and Rails applications

Automate Building Docker Base Images for Ruby and Rails applications

[https://medium.com/faun/build-docker-base-images-for-rails-2258ea2f11dc](https://medium.com/faun/build-docker-base-images-for-rails-2258ea2f11dc)

https://redd.it/ew20z6
@r_devops

University Concentration

Hi all,

Currently working part time in IT support (internal support + automation) and studying a Masters in IT. I've built some simple web apps, and have done a lot of scripting (python, bash, powershell) on the job with Azure InTune deployment.

With my background, my question is which should I focus on for a career in devops, a concentration in networking and infrastructure or software engineering?

https://redd.it/ew1oon
@r_devops

[Looking for Recomendation] Orchestrated Toolchains and automation

Hi all!

I was wondering what is the normal/standar for toolchain orchestration and automatic tasks.

the kind of Alarms -> Open Tickets -> Remediation -> Close ticket or escalate in case it doesn't do anything. and such...

I'm starting with learning the Flow Designer of xMatters. We are sending data from Dynatrace and CA UIM and testing creating a ticket in BMC Helix and executing remediations in our environment with CA RA and such.

Does anyone have something similar? What was the approach?

Thanks for any insight / examples or alternatives!

https://redd.it/ewffmw
@r_devops