High availability, loadbalancing. Achieving fast host eviction. Basically how to quickly remove/add unhealthy/healthy hosts?

My understanding of HA and LB is that these two does not assume each other.

High availability service does not have to be loadbalanced, and vice versa, loadbalanced service does not imply it is highly available.

Now, for a long time I cannot grasp how big systems and smart people actually achieve both efficiently?

Nowadays it is easy to create highly available loadbalanced environment that on the first sight works.

I'll take AWS for example but cloud is cloud.

Take ALB add to autoscaling group that spans multiple AZs. Set some health check and you're good to go.

But, health checks occur every 5sec (min) and default values are even bigger. That will lead us to not so highly available case. We have 5 seconds of a requests routing to service being down.

How is more effective loadbalanced HA service remediation is achieved really?

https://redd.it/eoqwpj
@r_devops

What is the best tool for the release management for Kubernetes in a small startup?

In our small startup (6 developers) and we use Google Kubernetes Engine + CircleCi (which deploys docker images to K8S with simple bash scripts). We have \~10 services, but we are growing. We use a simplified git-flow model. W plan to use in the future: terraform for infrastructure, helm charts for managing deployments.

We want to have a possibility:

1) (now) to track our releases in every environment

2) (now) manually rollback / automatically rollback after seeing some alerts in grafana. Rollback must apply Django migrations first and then use the image from the previous deployment.

3) (now) release a set of services in one go

4) (future) use some more advanced development techniques like canary deployment

5) (future) have some metrics regarding deployments (eg. build time)

​

There are plenty of tools which could be used for this job and it's hard to choose

1) Spinnaker -> it is said that it needs 3 engineers to maintain it, so probably it's too complex.

2) Jenkins -> it's more CI/CD tool.

3) Harness/Argo/Google Deployment/Team City/Code Fresh -> maybe one of those?

4) Write a simple inhouse solution in Django -> the most flexible one, but also the most error-prone.

What do you recommend?

https://redd.it/eonysq
@r_devops

As a DevOps Engineer in your current role, how do you interact with AWS, via the Web Management Console, AWS CLI, or AWS SDK? Is each employee given a specific account with granular IAM roles for the specific part of the infrastructure they are responsible for?

Also what core technologies of AWS do you work with the most?

https://redd.it/eos8q9
@r_devops

Potential job offer, largely a Microsoft shop. Wrong to lose interest over this?

Hi DevOps,

I'm currently working as a DevOps Engineer on the US east coast. Largely working on Linux machines. Been doing so for about three years. My knowledge has grown ten fold since I started and I'm loving it. Love writing infrastructure as code, loved building our deployment pipeline, love using the slew of tools available to us. 99% AWS.

Fast forward to recently and an old colleague references me for their company. Very good company, west coast, huge salary bump (I'm well below average right now, part of why I was interested in moving). I did very well in the interview and have been offered to be flown in to proceed. But they're largely a Microsoft shop. Azure is fine, I have used it. But my fear is they're 99% on Windows servers and I know most of the tools are built with that as an afterthought. This is turning me off to them, but I can't decide if that's "pettiness" as I know terraform, Kubernetes, Jenkins, and most other tools are supported on it.

Ignorance on purpose isn't good either, and it's likely that many positives would come from it, but I feel skeptical about moving away from an industry standard and having to deal with another learning curve.

They're also "in the process of integrating" Kubernetes so they haven't utilized any containerization until now.

On the positive side the position would involve an Angular app, its companion app in Ionic, and using Redis for SQL which sounds really interesting and would broaden my spectrum quite a bit.

All in all I'm not trying to shut myself out from outside of my comfort zone, but I'd also like to stay in the spectrum of what is standard and will make it easier to move when I decide to.

Is it normal to be concerned about this?

https://redd.it/eoq5dt
@r_devops

Can't find an old post on a great way to build Value Chains

Hey!
This isn't directly related to devops and is a little more on the BI/Operations Research side of things.

But a while ago in a conversation in the comments, someone linked a site which described a really cool method for mapping value chains and a dependency graph.

So I'll try and describe it and, hopefully, someone will get what I'm talking about and link it.

Forgive my ramblings and thanks for taking the moment to read this, I'm after a solid hour of googling.

Essentially, it places all parts of the organization on a graph, where the parts on the left are "commodities" and the ones farther right are less so. And then like a normal dependency graph, you connect the parts of the organization that are dependent on each other. This lets you see what part of the organization you need to work on automating and what parts you don't, and plan strategies to achieve the change you want.

The example he gave was based on the film industry, where the Production team is dependent on the Actors Guild (which isn't a commodity and therefore more expensive). So instead of applying pressure on the Actors Guild, the Production team moves another part of the organization that they have more influence over, and through them apply pressure on the Actors Guild.

The authors name was something like "McOwens"


Thanks again!

https://redd.it/eorfjf
@r_devops

Slack app that integrates GitHub and Circle CI to boost collaboration

We developed this app for ourselves to help us not forget about code reviews and notify us about failed and successful builds - but without spammy behavior like other integrations.

It’s really a collaboration booster for us as it takes away the manual hustle of checking the CI and GitHub to see if the build is done and if someone assigned you for a review or if your review is done.

We thought this could help others too, so right now we are looking for just a few companies for our private beta to see if it can solve their problems too.

If you can relate to these problems I will be more than happy to share more with you.
Cheers

https://redd.it/eom6ju
@r_devops

What’s the difference between DevOps and System Admin role?

I’ve been reading around and I don’t really know much about DevOps. Some say it’s a culture, some say it’s a job, and others just write a ton of technologies to learn like AWS, docker, Git, Linux, shell scripting, etc. Is DevOps glorified IT support role? Or is it a systems admin role? Or is it a systems admin role turned into a developer role? I’m not sure what dev ops is anymore based on all the info on this sub. I’m currently a 3rd year CS student very interested in technology and working as a software engineer and all the info on this sub confuses me.

https://redd.it/eow5hy
@r_devops

I Want To Get A Deeper Understanding Of DevOps.

Hello, Everyone !!

I want to get a deeper understanding of the philosophy, tools of DevOps, and learn to put them to use. I have a the basic concepts and understanding of the more popular tools ( Jenkins, TrqavisCI, Chef ,Ansible,Dockers,Kubernetes, Terraform) that doesn't mean I can use them or atleast not yet, but I'd like to. If anyone could suggest books to read, forums or site to look at, magazines, and any education site that would really be appreciated ( a career building or entry lvl online course). I'm not sure if I want to do this as a profession, but I find it fascinating and would like to know before completely diving into this avenue of IT.

Thanks everyone !!!

https://redd.it/eox7di
@r_devops

Is expecting programmers to do dev ops the new normal?

I've had a few interviews recently, and I am being bombarded by questions about continuous integration, deployment, azure, aws... I'm a programmer, applying for programmer roles.

Is this the new normal, or is this just my podunk city? Have you guys worked in places where they had their coders play dev ops? I imagine it's a horrendous idea for obvious reasons. I respect the dev ops skill set and the role they play, but I respect it enough to know I can't do a professional quality job with it.

https://redd.it/eowr7g
@r_devops

NetEng moving to NetOps: best andible/Jenkins books for the money?

Read the Phoenix project and understanding the core concepts of cicd, agile, scrum etc. Now my .org has picked ansible with Jenkins as the toolset of choice for systems/network automation. Read the howtos on red hats site and understand concepts but really need a project book of things to practice and try to get practical experience. What do you senior folks recommend?

https://redd.it/eovn35
@r_devops

CI/CD with Jenkins and Helm

I'm working on setting up some CI/CD pipelines and am wondering how others handle it when it comes to Jenkins and Helm charts. I initally was going to use Spinnaker (for the CD part) though couldn't get ECR to work (too many hours to justify) so back to Jenkins for now.

​

I am wondering how you do your CI/CD pipelines when using Jenkins and Helm charts, I have the ECR build part working (been for a while) and am now wondering how to do the CD portion. Mainly that currently I don't see a way to have a manual judgement process or similar. Best I came up with was using params and when checks.

https://redd.it/eovh59
@r_devops

Ultimate microservices architecture, Spring vs lambda ?

Hi everyone,

I have been studying event-base microservices architecture for a while now and I am still hesitant between using a traditional approach with Spring or an AWS lambda approach using an event fork pipeline and the serverless framework as backbone for the lambda + specific resources (api gw etc.)

If anyone has been building microservices using the two way could give his feedbacks, I would greatly greatly appreciate.

Thanks a lot.

Jonathan.

https://redd.it/eov5p9
@r_devops

Transaction vs Service

Hello All,

Evaluating multiple APM products. They all seem to blur the concepts of services and business transactions. How would you classify the difference?

AppDynamics seems to push that a service represents ONLY by the performance of the endpoint of a service while a Business Transaction is a collection of calls across potentially multiple services to perform a some operation for the applicaiton.

Dynatrace seems to classify a transaction as each individual call made against a service.

DataDog seems to view a service as one or multiple endpoints that work together in some way and a span is the execution of a logical unit of work within or across services.

NewRelic seems to focus on transactions which it defines as one logical unit of work in a software application.

Does anyone know of a good resource to understand the differences? If I have a group of 10 containers each running one process presenting itself in kubernetes as a service and a rest endpoint that when called will need to touch all 10 processes to complete its task: would you say the rest endpoint is a service? or is the collection of all 10 processes the service?

https://redd.it/eov452
@r_devops

Cloud cost overruns with IaC?

I’ve been talking to some folks about when they experienced sizable cloud cost overruns, and any processes or tools they used to deal with it? (notably in a software dev org)

Personally I’ve experienced setting the wrong Azure Search tier in my Terraform script and tripling my monthly bill unexpectedly.

Any war stories anyone wants to share, especially when using Infrastructure-as-Code? Any best practices on how you’ve dealt with the problem?

https://redd.it/eou0c5
@r_devops

Resources regarding cost of various IPC among microservices?

I'm having trouble finding studies that share performance profiles of REST vs RPC among microservices. Has anyone come across such information that can be shared?

https://redd.it/eon7ug
@r_devops

DevOps pipeline vulnerability scanning

I am building some prototype DevOps pipelines to demonstrate the toolchain used in a typical production work flow. I have code running through static code analysis (sonarqube) , container image vulnerability scanning with clay and OpenSCAP scanner. I am wondering how do you handle the code and container vulnerabilities inside the pipeline stages. Do you fail the pipeline stage based on a number of new bugs/cves or manual promotion based on eyeballing? It seems pretty hard to automatically fail/pass stages for vulnerability scanning without manual inspection of the scan results.

https://redd.it/eomhsp
@r_devops

Managing monolithic repositories

Hey /r/devops,

some time ago I asked here how you manage your monolithic repos with subtree splits. Especially the tag syncing was a problem I now solved with a little tool we use in our CI/CD-Pipelines.

You can find my previous question [here](https://www.reddit.com/r/devops/comments/cw3q3o/how_do_you_manage_monorepos_with_subtree_splits/).



The [monorepo-operator](https://github.com/SimonBaeumer/monorepo-operator) is written in go and can be used on linux. It can manage your repos, syncs them and allows you to perform batch tasks on all repos concurrently.

Hopefully this helps other people facing the same problems. If you have any suggestions feel free to give me feedback.

https://redd.it/ep2mrg
@r_devops

How dangerous is it to work with a server that does not have sophisticated firewalls?

I am just starting to learn all the DevOps necessary to deploy my web apps.

My AWS ec-2 are all behind a security group that only allows SSH access from my static IP home address (just got one recently). I have made one DO droplet be accessible via a password (less than 10 letters) so I can screw around with it in class and when im not home. I still didn't learn how to do what I do with security groups in AWS (IP restrictions)on DO.

How dangerous is it? Can my machine ever get internet STD's from SSHing to an instance that is open like that? I read on SO that in mere minutes of my server going on the air, I can expect thousands of brute force attacks. But how can they find me so quick? It takes google weeks to index new domains. How are attackers notified of new instances so quick?

https://redd.it/eom1qx
@r_devops

CircleCI vs Gitlab

Hi all,

I know these topics come up frequently but I havent seen a specific comparison that doesnt end up as a discussion about 50 other CI tools too :)


I'm currently using Gitlab, very happy with it overall (coming from Jenkins originally) but my new client has developers who want to use Circle (and have experience with Circle), particularly for its out of the box macOS build integration (though we can solve that via gitlab with a macstadium runner in any case)


I'm not wedded to Gitlab and I know that CircleCI is a known-good tool, but I can't see any specifically great reasons to use Circle over Gitlab. The only one seems to be Github integration, which Gitlab does quite poorly.

​

Does anyone have any compelling generic reasons for why I should use Circle over Gitlab?

​

Thanks

https://redd.it/eojm6u
@r_devops

nfstream: a flexible network data analysis framework

nfstream is a Python package providing fast, flexible, and expressive data structures designed to make working with online or offline network data both easy and intuitive. It aims to be the fundamental high-level building block for doing practical, real world network data analysis in Python. Additionally, it has the broader goal of becoming a common network data processing framework for researchers providing data reproducibility across experiments. Feedbacks and Contributions are welcome!

[https://github.com/aouinizied/nfstream](https://github.com/aouinizied/nfstream)

https://redd.it/eokhi4
@r_devops