Software development management

Hi, I am a third year business student from Rotterdam. For my intership at an IT company I was asking myself... is it any useful to implement software development management processes(like DevOps) on a team of only 2.5 FTE developers. They mostly work on internal software like PSA and incident management systems. Like to know your opinions

https://redd.it/eob0vk
@r_devops

Reducing costs from S3 using Nginx Proxy?

I work at a video streaming company focused on the sports market, currently moving from on-prem storage solutions to AWS. Mostly, it’s because we want to focus on building great things, instead of having to worry about servers and in this specific case scalability. That’s why we chose to move from running our own storage servers to using S3 instead. Right now, we got it all set up and it all works fine, but we are struggling with the higher price tag. Although we embraced ourselves that we would get those monthly fees, I can not wrap my head around the fact that the data traffic is so darn expensive.

In order to reduce this, we setup a server with nginx, acting as a proxy for S3. Since we already had the hardware and bandwidth, so this made sense in order to get the costs down a bit on traffic out from S3.

It sure did get the costs down, but when I look into my logs I see that we have a cache miss ratio of about 30 %, which seems strange since most of the video we deliver is only relevant for about a week until it falls into oblivion (highlights and other sports related content only has a high viewing ratio for like 24 hours until people don’t care anymore).

I’m don’t really know what I am grasping for here, just hope that there is someone else who has stumbled onto this problem and might have a neat solution to it. As I said before, we are currently using nginx with proxy_cache, having the cache size limit to 1300G and a proxy_cache_valid for 200 OK set to 864h. Also worth to note is that we have enabled proxy_cache_lock, in order to only fetch the requested file once.

Anyone having any great ideas?

https://redd.it/eobx5d
@r_devops

Big consulting firm vs small dev group

Just wondering if anybody has done devops for both a large consulting firm and for a small tight knit dev group and can compare them.

I've been on a small dev group for several years now, and am kinda getting burned out and needing a change. In my current situation, I am constantly too busy leaving little time for personal growth and bigger projects, largely stuck on day to day stuff. We talk about big picture things a lot and it is exciting, but we never get to it. Also, the tiny on-call rotation comes around to me way too frequently.

A recruiter from a consulting firm contacted me the other day, and was considering the change. The pros of it seem to be focus on the technologies, and less BS to deal with. The cons being regular travel. What else am I missing and not considering?

https://redd.it/eo8j31
@r_devops

Ruby or Python more prevalent in DevOps? What are some must-know libraries for each that you utilize on a daily basis?



https://redd.it/eoegc0
@r_devops

Asterisk: command not found

Whereis asterisk
/etc/asterisk
I added to env variable with export PATH=$PATH:/etc/asterisk and source env variable file.
Still getting the same error.
I have installed it and wanted to access the asterisk gui, The port is configuration is all good.
Can’t reload asterisk cli. Googled the issue but still no help.

https://redd.it/eoe63h
@r_devops

Feel like Recruiter's make Job Application Process way more complicated?

I have applied to a bunch of job listings for a DevOps Position, and I get a swarm of email's and phone calls, which is totally fine, but because the Recruiter's themselves work for another company, and email you back from that companies email address, it makes it so hard to figure out which job posting they are talking about.

I'm sure it is just as confusing to them as it is for us, so I'm not blaming them or anything.

I want to make this process as smooth as possible, and in between the emails and calendars, it becomes really stressful trying to discern what is what who is who? Am I the only one?

Its 2020, there should be a much better way to go about the job application process already. Like software that keeps track of each application, and automatically creates a calendar for you, by interpreting the emails (AI stuff), and auto-responds based on the initial availability input you provide.

Say, I'm available Mon-Friday 12:00 noon to 5:00 pm, I applied to 100 job postings a month. It should read back the positive feedback, and auto-respond to the interviewer based on first-come-first-server basis with a specific time and date, and auto-populate a calendar for you. That would be so freaking neat!

I'm sure that will eventually happen, but it's taking forever lol.

In the meantime, how do you guys handle the process? I want to make this as less stressful as possible and avoid the complication altogether.

Is there a better way already and I'm not aware of?

https://redd.it/eodgdp
@r_devops

Ideal solution to separate config/data files from source code -Java

Hi everyone!

Hope you guys are doing great :)

I would like your suggestions regarding solving the following problem

I have a Git repository which contains Test Scripts written in Java as well as some config and test data files stored in Yaml and .properties file . The Test Scripts / code doesn’t that often . On the contrary the config and test data changes quite often hence there are way too many commits because of test data /config changes . Hence I am thinking of separating out the test data and config files from the test scripts / code and somehow find a way to access these files during runtime .

Also, if this helps ,I am using Maven .

What would be the best way to do this ? Also the another reason for this is to give a layman ability to access these test data files so that they can modify these files without worrying about the code .

I would prefer an open source solution . If I can host a service on my private network that would be the best as I would have to go through lot of formalities to use public cloud based solution .

Thanks in advance !
Adi

https://redd.it/eo8hve
@r_devops

In Terraform I'm getting a Malformed Terraform Policy Document Error on a seemingly perfect piece of code. What am i doing wrong?

MalformedPolicyDocument: Partition "aws" is not valid for resource "arn:aws:logs:\*:\*:\*

on modules/openshift/05-roles.tf line 26, in resource "aws\_iam\_policy" "openshift-policy-forward-logs":

26: resource "aws\_iam\_policy" "openshift-policy-forward-logs" {

Here is the offending code snippet:

// This policy allows an instance to forward logs to CloudWatch, and
// create the Log Stream or Log Group if it doesn't exist.
resource "aws_iam_policy" "openshift-policy-forward-logs" {
name = "openshift-instance-forward-logs"
path = "/"
description = "Allows an instance to forward logs to CloudWatch"

policy = <<EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents",
"logs:DescribeLogStreams"
],
"Resource": [
"arn:aws:logs:*:*:*"
]
}
]
}
EOF
}

https://redd.it/eo7pwj
@r_devops

How are you using RANCHER for cluster MGT?

We recently provisioned rancher on our cluster, came with great alert and monitoring tool, any other things you use rancher for?

&#x200B;

Kindly share.

https://redd.it/eo3rku
@r_devops

Canary over K8S - which tool and technique?

We have the need for Canary - get some more confidence before full-blown deployment. What are your recommended tools, techniques and practices? My current Hypothesis is using Istio and a static canary - simply check the next version over 1% of the traffic. Not ideal, but a lean way to go. Thoughts?

https://redd.it/eo29k2
@r_devops

How did/do you automate your Change Management process and evidence or controls for audit and stakeholders?

I'm curious about how your DevOps team went about meeting almost classic Change Management requirements and audit requirements. Im pretty stuck on where to start with this tbh.

https://redd.it/eo1lfg
@r_devops

Where and how are application secrets backed up?

SSH keys, passwords and passphrases, master passwords for passwords managers, API keys etc. Where and how are they all backed up? If my PC cached on fire right now, how would I decrypt data, or SSH to key secured servers?

1. Where do you backup? Only locally on external flash drives, or in the cloud?
2. How do you backup? Backing up to a cloud is meaningless if you do it in a way that exposes you to MITM.
3. In what format do you backup? Plain text, or encrypted (in which case - where do you backup the key you use to encrypt your secrets)? Should you ever bother with obfuscation, or is it ok to backup server private key as "server\_private.pem"?

https://redd.it/eo00y8
@r_devops

Aiming to create k8s/cloudnative app, advice needed!

Hello!

My company wants to partially rewrite their product with Kubernetes and AWS in mind.

The thing is that the part we are aiming to rewrite is a 2-service monolithic platform based on Java and tomcat. On its own it's fairly complicated.

The main app wakes up in 70s and serves as a frontend to tightly coupled underlying service serving customer customizations.
It's design was created few years ago with on-prem deployments and consultancy in mind. Since then it grew beyond understanding of a mere developers and only few most experienced devs are 100% aware of what exactly is there.

From automation perspective we did what we could given how much ppl we have.
We have helmcharts for each major component. Those helmcharts are deployed out of helmfiles we keep outside of chart repo.
We also pull more and more devs into our playground with various trainings and enablement, but it goes slow.
They doesn't have much time because of management mentality of "feature gives money, cleaning the mess doesn't".

Monitoring is totally different thing for next post. Let's just say, that it's there but it definitely could've been better.

As you can see it's doesn't fit well in the cloudnative spirit.
Moreover language change can't be the case. Java stays.

What materials can I show to main architect describing things like:

* Java and HPA in Kubernetes

* How important app upstart times are

* How to measure and compare metrics like resources consumed per request handled in Java world?

* How to split monolithic apps like this (where is the sane boundary between too little microservices and too much)

* What is the safest approach to tiering such app?

* How can we laverage service meshes for circuit breaking and other things that devs formerly have been including in the apps?

* How AWS can come in hand? S3, IAM are obvious ,what else is there?

https://redd.it/enzz62
@r_devops

Github Actions vs CircleCI

I was wondering if anyone has had some experience working/trying out both systems, and what they felt were the biggest weaknesses of them?

Also want to see if someone has used Github Actions for multiple repos, and how they like that whole process.

Thanks!

https://redd.it/enztfs
@r_devops

Measuring CI/CD

Hello there

obligatory: first time posting here

Now that we have this cleared I'd like to ask you all a question about how/is it possible to measure stuff in continuous integration / continuous deployment process.

Why I'm asking?

I'm writing a big assignment for my university, I have to implement a CI/CD process and measure it's effect. I can do this in my own dev env but luckily I've found a company that would like me to implement them a CI/CD pipeline (I'll be paid for that) and my professor is very excited about the possible outcome of this collaboration and my findings about it.

The plan I have is rather simple

1. create some hooks on gitlab
2. build docker images and run tests whenever a merge request is created.
3. automate the deployment after test runs succesfully
4. leave the actual decision whether to deploy/wait to someone who pushes the button

What can I measure here? My initial ideas are:

1. how much time people spent on manual integration & deployment vs how much time it take if automated
2. track how much $$$ a company can save (thanks to findings from pt1)
3. track the history of failed deployments (due to human error) and track this after automation
4. short questionnaire for developers to check their happiness before & after(???)

These seem interesting things to track but I feel like there are is some more technical data out there that can be measured in the process

We all know that automated process IS better, but I need to know HOW it is better, in what ways and how much. I assume this won't only be good for my grades but in the future if I'd like to justify the time spent on implementing and automating things in the company workflow.

Does anyone know anything about the above topic or can you point me in the right direction (books, articles, some scientific papers maybe)?

https://redd.it/eopu51
@r_devops

How to achieve Ops self-service?

What have you implemented to handle incoming Ops requests for things like creating accounts, repos, projects etc. FYI, this is mainly around the Atlassian stack.

Today, this always seems to result in a ton of back and forth around missing, wrong or misunderstood info.

Looking forward to resolving this headache!

https://redd.it/eoosvf
@r_devops

cassowary - modern load testing tool

Hi /r/Devops,

I've written a load testing tool called cassowary, from the beginning it was just used to solve a problem I had, but I've polished it up in the last couple of weeks. Functionality wise it's quite close to ab but has modern features like exporting metrics to Prometheus and a file slurp mode where URL paths can be defined in a file (that way the load test can hit several underlying microservices). Also it's written in Go so the project has cross-platform binaries. Anyway hope someone can find it useful, link below:

[https://github.com/rogerwelin/cassowary](https://github.com/rogerwelin/cassowary)

https://redd.it/eoq8r1
@r_devops

High availability, loadbalancing. Achieving fast host eviction. Basically how to quickly remove/add unhealthy/healthy hosts?

My understanding of HA and LB is that these two does not assume each other.

High availability service does not have to be loadbalanced, and vice versa, loadbalanced service does not imply it is highly available.

Now, for a long time I cannot grasp how big systems and smart people actually achieve both efficiently?

Nowadays it is easy to create highly available loadbalanced environment that on the first sight works.

I'll take AWS for example but cloud is cloud.

Take ALB add to autoscaling group that spans multiple AZs. Set some health check and you're good to go.

But, health checks occur every 5sec (min) and default values are even bigger. That will lead us to not so highly available case. We have 5 seconds of a requests routing to service being down.

How is more effective loadbalanced HA service remediation is achieved really?

https://redd.it/eoqwpj
@r_devops

What is the best tool for the release management for Kubernetes in a small startup?

In our small startup (6 developers) and we use Google Kubernetes Engine + CircleCi (which deploys docker images to K8S with simple bash scripts). We have \~10 services, but we are growing. We use a simplified git-flow model. W plan to use in the future: terraform for infrastructure, helm charts for managing deployments.

We want to have a possibility:

1) (now) to track our releases in every environment

2) (now) manually rollback / automatically rollback after seeing some alerts in grafana. Rollback must apply Django migrations first and then use the image from the previous deployment.

3) (now) release a set of services in one go

4) (future) use some more advanced development techniques like canary deployment

5) (future) have some metrics regarding deployments (eg. build time)

&#x200B;

There are plenty of tools which could be used for this job and it's hard to choose

1) Spinnaker -> it is said that it needs 3 engineers to maintain it, so probably it's too complex.

2) Jenkins -> it's more CI/CD tool.

3) Harness/Argo/Google Deployment/Team City/Code Fresh -> maybe one of those?

4) Write a simple inhouse solution in Django -> the most flexible one, but also the most error-prone.

What do you recommend?

https://redd.it/eonysq
@r_devops

As a DevOps Engineer in your current role, how do you interact with AWS, via the Web Management Console, AWS CLI, or AWS SDK? Is each employee given a specific account with granular IAM roles for the specific part of the infrastructure they are responsible for?

Also what core technologies of AWS do you work with the most?

https://redd.it/eos8q9
@r_devops