Reddit DevOps
274 subscribers
66 photos
32.1K links
Reddit DevOps. #devops
Thanks @reddit2telegram and @r_channels
Download Telegram
Canary over K8S - which tool and technique?

We have the need for Canary - get some more confidence before full-blown deployment. What are your recommended tools, techniques and practices? My current Hypothesis is using Istio and a static canary - simply check the next version over 1% of the traffic. Not ideal, but a lean way to go. Thoughts?

https://redd.it/eo29k2
@r_devops
How did/do you automate your Change Management process and evidence or controls for audit and stakeholders?

I'm curious about how your DevOps team went about meeting almost classic Change Management requirements and audit requirements. Im pretty stuck on where to start with this tbh.

https://redd.it/eo1lfg
@r_devops
Where and how are application secrets backed up?

SSH keys, passwords and passphrases, master passwords for passwords managers, API keys etc. Where and how are they all backed up? If my PC cached on fire right now, how would I decrypt data, or SSH to key secured servers?

1. Where do you backup? Only locally on external flash drives, or in the cloud?
2. How do you backup? Backing up to a cloud is meaningless if you do it in a way that exposes you to MITM.
3. In what format do you backup? Plain text, or encrypted (in which case - where do you backup the key you use to encrypt your secrets)? Should you ever bother with obfuscation, or is it ok to backup server private key as "server\_private.pem"?

https://redd.it/eo00y8
@r_devops
Aiming to create k8s/cloudnative app, advice needed!

Hello!

My company wants to partially rewrite their product with Kubernetes and AWS in mind.

The thing is that the part we are aiming to rewrite is a 2-service monolithic platform based on Java and tomcat. On its own it's fairly complicated.

The main app wakes up in 70s and serves as a frontend to tightly coupled underlying service serving customer customizations.
It's design was created few years ago with on-prem deployments and consultancy in mind. Since then it grew beyond understanding of a mere developers and only few most experienced devs are 100% aware of what exactly is there.

From automation perspective we did what we could given how much ppl we have.
We have helmcharts for each major component. Those helmcharts are deployed out of helmfiles we keep outside of chart repo.
We also pull more and more devs into our playground with various trainings and enablement, but it goes slow.
They doesn't have much time because of management mentality of "feature gives money, cleaning the mess doesn't".

Monitoring is totally different thing for next post. Let's just say, that it's there but it definitely could've been better.

As you can see it's doesn't fit well in the cloudnative spirit.
Moreover language change can't be the case. Java stays.

What materials can I show to main architect describing things like:

* Java and HPA in Kubernetes

* How important app upstart times are

* How to measure and compare metrics like resources consumed per request handled in Java world?

* How to split monolithic apps like this (where is the sane boundary between too little microservices and too much)

* What is the safest approach to tiering such app?

* How can we laverage service meshes for circuit breaking and other things that devs formerly have been including in the apps?

* How AWS can come in hand? S3, IAM are obvious ,what else is there?

https://redd.it/enzz62
@r_devops
Github Actions vs CircleCI

I was wondering if anyone has had some experience working/trying out both systems, and what they felt were the biggest weaknesses of them?

Also want to see if someone has used Github Actions for multiple repos, and how they like that whole process.

Thanks!

https://redd.it/enztfs
@r_devops
Measuring CI/CD

Hello there

obligatory: first time posting here

Now that we have this cleared I'd like to ask you all a question about how/is it possible to measure stuff in continuous integration / continuous deployment process.

Why I'm asking?

I'm writing a big assignment for my university, I have to implement a CI/CD process and measure it's effect. I can do this in my own dev env but luckily I've found a company that would like me to implement them a CI/CD pipeline (I'll be paid for that) and my professor is very excited about the possible outcome of this collaboration and my findings about it.

The plan I have is rather simple

1. create some hooks on gitlab
2. build docker images and run tests whenever a merge request is created.
3. automate the deployment after test runs succesfully
4. leave the actual decision whether to deploy/wait to someone who pushes the button

What can I measure here? My initial ideas are:

1. how much time people spent on manual integration & deployment vs how much time it take if automated
2. track how much $$$ a company can save (thanks to findings from pt1)
3. track the history of failed deployments (due to human error) and track this after automation
4. short questionnaire for developers to check their happiness before & after(???)

These seem interesting things to track but I feel like there are is some more technical data out there that can be measured in the process

We all know that automated process IS better, but I need to know HOW it is better, in what ways and how much. I assume this won't only be good for my grades but in the future if I'd like to justify the time spent on implementing and automating things in the company workflow.

Does anyone know anything about the above topic or can you point me in the right direction (books, articles, some scientific papers maybe)?

https://redd.it/eopu51
@r_devops
How to achieve Ops self-service?

What have you implemented to handle incoming Ops requests for things like creating accounts, repos, projects etc. FYI, this is mainly around the Atlassian stack.

Today, this always seems to result in a ton of back and forth around missing, wrong or misunderstood info.

Looking forward to resolving this headache!

https://redd.it/eoosvf
@r_devops
cassowary - modern load testing tool

Hi /r/Devops,

I've written a load testing tool called cassowary, from the beginning it was just used to solve a problem I had, but I've polished it up in the last couple of weeks. Functionality wise it's quite close to ab but has modern features like exporting metrics to Prometheus and a file slurp mode where URL paths can be defined in a file (that way the load test can hit several underlying microservices). Also it's written in Go so the project has cross-platform binaries. Anyway hope someone can find it useful, link below:

[https://github.com/rogerwelin/cassowary](https://github.com/rogerwelin/cassowary)

https://redd.it/eoq8r1
@r_devops
High availability, loadbalancing. Achieving fast host eviction. Basically how to quickly remove/add unhealthy/healthy hosts?

My understanding of HA and LB is that these two does not assume each other.

High availability service does not have to be loadbalanced, and vice versa, loadbalanced service does not imply it is highly available.

Now, for a long time I cannot grasp how big systems and smart people actually achieve both efficiently?

Nowadays it is easy to create highly available loadbalanced environment that on the first sight works.

I'll take AWS for example but cloud is cloud.

Take ALB add to autoscaling group that spans multiple AZs. Set some health check and you're good to go.

But, health checks occur every 5sec (min) and default values are even bigger. That will lead us to not so highly available case. We have 5 seconds of a requests routing to service being down.

How is more effective loadbalanced HA service remediation is achieved really?

https://redd.it/eoqwpj
@r_devops
What is the best tool for the release management for Kubernetes in a small startup?

In our small startup (6 developers) and we use Google Kubernetes Engine + CircleCi (which deploys docker images to K8S with simple bash scripts). We have \~10 services, but we are growing. We use a simplified git-flow model. W plan to use in the future: terraform for infrastructure, helm charts for managing deployments.

We want to have a possibility:

1) (now) to track our releases in every environment

2) (now) manually rollback / automatically rollback after seeing some alerts in grafana. Rollback must apply Django migrations first and then use the image from the previous deployment.

3) (now) release a set of services in one go

4) (future) use some more advanced development techniques like canary deployment

5) (future) have some metrics regarding deployments (eg. build time)

​

There are plenty of tools which could be used for this job and it's hard to choose

1) Spinnaker -> it is said that it needs 3 engineers to maintain it, so probably it's too complex.

2) Jenkins -> it's more CI/CD tool.

3) Harness/Argo/Google Deployment/Team City/Code Fresh -> maybe one of those?

4) Write a simple inhouse solution in Django -> the most flexible one, but also the most error-prone.

What do you recommend?

https://redd.it/eonysq
@r_devops
As a DevOps Engineer in your current role, how do you interact with AWS, via the Web Management Console, AWS CLI, or AWS SDK? Is each employee given a specific account with granular IAM roles for the specific part of the infrastructure they are responsible for?

Also what core technologies of AWS do you work with the most?

https://redd.it/eos8q9
@r_devops
Potential job offer, largely a Microsoft shop. Wrong to lose interest over this?

Hi DevOps,

I'm currently working as a DevOps Engineer on the US east coast. Largely working on Linux machines. Been doing so for about three years. My knowledge has grown ten fold since I started and I'm loving it. Love writing infrastructure as code, loved building our deployment pipeline, love using the slew of tools available to us. 99% AWS.

Fast forward to recently and an old colleague references me for their company. Very good company, west coast, huge salary bump (I'm well below average right now, part of why I was interested in moving). I did very well in the interview and have been offered to be flown in to proceed. But they're largely a Microsoft shop. Azure is fine, I have used it. But my fear is they're 99% on Windows servers and I know most of the tools are built with that as an afterthought. This is turning me off to them, but I can't decide if that's "pettiness" as I know terraform, Kubernetes, Jenkins, and most other tools are supported on it.

Ignorance on purpose isn't good either, and it's likely that many positives would come from it, but I feel skeptical about moving away from an industry standard and having to deal with another learning curve.

They're also "in the process of integrating" Kubernetes so they haven't utilized any containerization until now.

On the positive side the position would involve an Angular app, its companion app in Ionic, and using Redis for SQL which sounds really interesting and would broaden my spectrum quite a bit.

All in all I'm not trying to shut myself out from outside of my comfort zone, but I'd also like to stay in the spectrum of what is standard and will make it easier to move when I decide to.

Is it normal to be concerned about this?

https://redd.it/eoq5dt
@r_devops
Can't find an old post on a great way to build Value Chains

Hey!
This isn't directly related to devops and is a little more on the BI/Operations Research side of things.

But a while ago in a conversation in the comments, someone linked a site which described a really cool method for mapping value chains and a dependency graph.

So I'll try and describe it and, hopefully, someone will get what I'm talking about and link it.

Forgive my ramblings and thanks for taking the moment to read this, I'm after a solid hour of googling.

Essentially, it places all parts of the organization on a graph, where the parts on the left are "commodities" and the ones farther right are less so. And then like a normal dependency graph, you connect the parts of the organization that are dependent on each other. This lets you see what part of the organization you need to work on automating and what parts you don't, and plan strategies to achieve the change you want.

The example he gave was based on the film industry, where the Production team is dependent on the Actors Guild (which isn't a commodity and therefore more expensive). So instead of applying pressure on the Actors Guild, the Production team moves another part of the organization that they have more influence over, and through them apply pressure on the Actors Guild.

The authors name was something like "McOwens"


Thanks again!

https://redd.it/eorfjf
@r_devops
Slack app that integrates GitHub and Circle CI to boost collaboration

We developed this app for ourselves to help us not forget about code reviews and notify us about failed and successful builds - but without spammy behavior like other integrations.

It’s really a collaboration booster for us as it takes away the manual hustle of checking the CI and GitHub to see if the build is done and if someone assigned you for a review or if your review is done.

We thought this could help others too, so right now we are looking for just a few companies for our private beta to see if it can solve their problems too.

If you can relate to these problems I will be more than happy to share more with you.
Cheers

https://redd.it/eom6ju
@r_devops
What’s the difference between DevOps and System Admin role?

I’ve been reading around and I don’t really know much about DevOps. Some say it’s a culture, some say it’s a job, and others just write a ton of technologies to learn like AWS, docker, Git, Linux, shell scripting, etc. Is DevOps glorified IT support role? Or is it a systems admin role? Or is it a systems admin role turned into a developer role? I’m not sure what dev ops is anymore based on all the info on this sub. I’m currently a 3rd year CS student very interested in technology and working as a software engineer and all the info on this sub confuses me.

https://redd.it/eow5hy
@r_devops
I Want To Get A Deeper Understanding Of DevOps.

Hello, Everyone !!

I want to get a deeper understanding of the philosophy, tools of DevOps, and learn to put them to use. I have a the basic concepts and understanding of the more popular tools ( Jenkins, TrqavisCI, Chef ,Ansible,Dockers,Kubernetes, Terraform) that doesn't mean I can use them or atleast not yet, but I'd like to. If anyone could suggest books to read, forums or site to look at, magazines, and any education site that would really be appreciated ( a career building or entry lvl online course). I'm not sure if I want to do this as a profession, but I find it fascinating and would like to know before completely diving into this avenue of IT.

Thanks everyone !!!

https://redd.it/eox7di
@r_devops
Is expecting programmers to do dev ops the new normal?

I've had a few interviews recently, and I am being bombarded by questions about continuous integration, deployment, azure, aws... I'm a programmer, applying for programmer roles.

Is this the new normal, or is this just my podunk city? Have you guys worked in places where they had their coders play dev ops? I imagine it's a horrendous idea for obvious reasons. I respect the dev ops skill set and the role they play, but I respect it enough to know I can't do a professional quality job with it.

https://redd.it/eowr7g
@r_devops
NetEng moving to NetOps: best andible/Jenkins books for the money?

Read the Phoenix project and understanding the core concepts of cicd, agile, scrum etc. Now my .org has picked ansible with Jenkins as the toolset of choice for systems/network automation. Read the howtos on red hats site and understand concepts but really need a project book of things to practice and try to get practical experience. What do you senior folks recommend?

https://redd.it/eovn35
@r_devops
CI/CD with Jenkins and Helm

I'm working on setting up some CI/CD pipelines and am wondering how others handle it when it comes to Jenkins and Helm charts. I initally was going to use Spinnaker (for the CD part) though couldn't get ECR to work (too many hours to justify) so back to Jenkins for now.

​

I am wondering how you do your CI/CD pipelines when using Jenkins and Helm charts, I have the ECR build part working (been for a while) and am now wondering how to do the CD portion. Mainly that currently I don't see a way to have a manual judgement process or similar. Best I came up with was using params and when checks.

https://redd.it/eovh59
@r_devops
Ultimate microservices architecture, Spring vs lambda ?

Hi everyone,

I have been studying event-base microservices architecture for a while now and I am still hesitant between using a traditional approach with Spring or an AWS lambda approach using an event fork pipeline and the serverless framework as backbone for the lambda + specific resources (api gw etc.)

If anyone has been building microservices using the two way could give his feedbacks, I would greatly greatly appreciate.

Thanks a lot.

Jonathan.

https://redd.it/eov5p9
@r_devops