Is 80% of what you do at a large company just navigating how to get things done in the organization?

I swear if all we had to do was create dockerfiles, CICD pipelines, and helm charts to deploy people's code, it would be easy, but we spend 80% of the time working around firewall rules, company proxy ssl certs, submitting jira tickets, backlog grooming, planning poker, security patching and regular meetings to get other departments to make simple changes, like adding a dns record or upgrading a ruby gem.

Is this normal devops work in a large organization?

https://redd.it/y2a2p4
@r_devops

Overnight - how do you handle automatic escalation when site goes down?

We had a situation where site went down right after midnight - We didnt find out or fixed until the morning.
Yes there were 100 emails and slack messages but aint anyone gonnna be checking them at midnight.

small team - 2 op guys - cannot blame them for missing it.

how do you handle escalation and notifications overnight?

https://redd.it/y2c8lz
@r_devops

Most stories I develop against have the requirements changed multiple times during sprint.. is this normal?

I'm pulling my hair out here.. I receive the problem in the story, and then my solution is coded to spec. Once I get into my PR review it's then decided there's "x,y,z" ways better/more optimal ways to perform these various tasks.

I mean I get there is typically more than one way to solve a problem, but this seems out of the ordinary versus my past positions I've held. Typically, if you have a working solution, and all tests pass you're all set.

Is the point of Agile to move fast, and push out features, and then create a backlog of stories to potentially optimize in the future?

/rant

https://redd.it/y27gx9
@r_devops

Investigating DevOps and the role of a DevOps Engineer - What are the best sources of research?

Howdy,

I have been tasked by senior management to spend some time learning more about DevOps (as a topic) and DevOps engineering. It was suggested that I get a copy of 'The DevOps Handbook' and begin my research with this book. Is this a definitive source? Are there other sources you can share to further my education and understanding of DevOps and how to implement within existing teams (development, IT, Support, Sales, etc)?


Thanks in advance for the help!

https://redd.it/y29eqe
@r_devops

Is the Vault Development tier more than enough for 1 production environment + 2 UAT?

Is the Vault Development tier more than enough for 2 Production Kubernetes clusters + 2 UAT Kubernetes clusters?

I seem to be missing something here, but with using Vault + Terraform, the Development tier pricing for Vault is more than enough for several environments. After all the only time we'll be interacting with Vault is when we are updating our clusters through terraform.

Or will the Kubernetes cluster itself connect to Vault every time it needs to access a deployed secret?

https://redd.it/y22toj
@r_devops

Going to full in into AWS or become 50% 50% among AWS and Azure

I have started looking for a position in DevOps, and I already passed the AWS Developer Associate certificate, but I was kind of thinking about where to go next - should I go my way with AWS and pass DevOps certification (to get a hand on CodeBuild, CodeDeploy, CodePipeline, ECS) or should I go and start also working with Azure? The main thing why I'm asking this - is as I have seen many positions asking specifically for Azure knowledge , therefore by getting practice with Azure I could also look for this position

https://redd.it/y2l52f
@r_devops

Virtual hackfest for developers - big cash prizes!



Hey guys since this group is all about developers, and this opportunity to win loads seems like a perfect fit for developers.

Remotebase is hosting the largest virtual hackathon with exciting challenges and prizes. Its a simple challenge of "building the remote world" and the crash prizes are amazing!!! I am definitely participating. Lets compete with one another ;) Oh and we can also make teams and win big.

I am leaving a link to register if anyone's interested. https://share.hsforms.com/10P3Qx8yGQd2TRxM6VuvmWwcxb53

You can also head to their website for more info.

https://redd.it/y2myi6
@r_devops

Project: Kubernetes Cluster on Oracle Free Tier using Terraform & Ansible

Hi there. I recently built a cluster on Oracle Cloud Free Tier using Terraform & Ansible and would like to share the yaml files with the community. This can be great for anyone starting to dip into Kubernetes and needs a cloud hosted cluster to play with.

https://github.com/solamarpreet/kubernetes-on-oci

​

P.S There are ansible files for both Microk8s and K3s so you can pick the distribution you prefer. I recommend Microk8s. Let me know if you have anymore queries.

https://redd.it/y2qao8
@r_devops

trial period ended after a month because a no documentation, bad communication, knowledge silo boss didn't like my pointing out Plesk and pet VPSes aren't scalable. I was hired to "audit and modernize", and I feel scammed.

he's keeping the externalized backup system I put in place though, the hypocrite.

I have a feeling this hire wasn't his decision in the first place, since his team and founding partner all said they were happy with what I was talking about...

I'll have to do the whole interviews song and dance again, I guess.

anyone hiring an experienced SRE / infrastructure engineer (remote EU) ? :D

https://redd.it/y2vdz5
@r_devops

The feature request for a "allow-failure" option on GitHub Actions now almost has one thousand thumbs up

Posted 2,5 years ago, the feature request for a "allow-failure" option on GitHub Actions now almost has one thousand thumbs up.

* [Please support something like "allow-failure" for a given job](https://github.com/actions/toolkit/issues/399)

https://redd.it/y2t6b9
@r_devops

How do you manage your patching for cloud VMs (like EC2 instances)?

Interested to hear how people manage their cloud instance patching for security and software updates. My current company uses Packer to create AMIs and deploys the EC2 instances with Terraform. My main questions are the following:

​

1. How do you identify when a patch is required on your deployed instances?
2. How do you then update your AMIs?
3. What tools/processes do you use for the above and do you automate it?

​

Thanks, really appreciate any experiences anyone has with this that they can share.

https://redd.it/y2yxr6
@r_devops

Results of the study: How stressful is working in DevOps?

We wanted to thank Reddit users who participated in our Stress survey a few months ago. Thanks to you, we created [this report on Stress in IT (focusing on DevOps)](https://spacelift.io/blog/are-it-jobs-stressful). We hope you will find it interesting! Let us know your feedback. Few highlights:

* DevOps is the 2nd most stressed IT area.
* 60% of DevOps choose to blame themselves as a way to deal with stress. No other group was revealed to be so self-critical.
* Playing video games is the most popular stress coping mechanism for DevOps professionals, with almost 62% admitting it.
* 39% of DevOps engineers admit that work-related stress impacts their personal lives. This is the highest percentage in all segments we surveyed.
* DevOps are also the ones who take off the most time.

https://redd.it/y30kjs
@r_devops

Coding challenge for a DevOps role

I got a coding challenge for a DevOps role with the usual CI/CD pipelines building, container orchestration with K8S, etc etc. So full on focus on the Operations side.

They are a Software Dev Hub and this will be the first DevOps guy in their team so no one really knows the Ops side.

So they have given me a backend coding challenge, like what?

I can code a bit but not that good since it isn't my focus and like true IT pro I first tried to Google to see if there was a solution to make the problem go away (coding challenge) before actually doing the work and I found the solution online.

​

Now I am conflicted. Should I act like I wrote the code and do some refactoring and writing the logic differently (which I don't wanna do) or tell them truthfully that I found the solution and just build some Ops solution around the backend code.

I was thinking about designing the architecture of K8S for microservices around this coding challenge to make it more fitting for the role I am applying for and telling them I found the solution online already so I just added the infrastructure.

​

My question is if I should proceed with the latter since I am uncertain if they will even appreciate - let's say - my DevOps way of thinking. Or would they lash out and feel like I cheated the challenge and not proceed with me.

​

Personally I think that what I did is way more suited as a challenge for a DevOps engineer instead of a backend coding challenge.

https://redd.it/y326vd
@r_devops

Running Cypress Tests in Parallel Using Docker Compose

https://webapp.io/blog/running-cypress-tests-in-parallel-using-docker-compose/

https://redd.it/y3407l
@r_devops

Get another AWS certification or learn something else?

I have a AWS SAA cert already.

I went through the roadmap already to see what skills I would need but I would like to know if it would make sense to get the SOA cert or move on to terraform, K8, or Python.

Either way, in which order should I study them in?

If it makes sense to go ahead and get the AWS SOA, should I study Python next? K8 (CKA)? Or Terraform?

I'm probably not going to get Terraform certified but will be using the exam outline to study.

https://redd.it/y388te
@r_devops

24/7 services and global teams

I support a B2B application that is deployed in the cloud across multiple regions and supports customers around the world. Our team is located in several areas around the world. Do any of you work like that? What challenges and best practices do you have?

My observations: To form great working relationships you have to be online together at least occasionally. Trying to make it work exclusively with asynchronous email, chat, JIRA comments etc is not very effective at building rapport, getting things done efficiently or solving problems.

To remedy the lack of overlap, we try to have a common meeting at some odd hour, trying to rotate whose time zone gets the most advantage. This helps a ton and fortunately all sides have been willing to work together on this. But I know from discussions w/ individuals this practice, long-term, begins to weigh heavily on them. It is the regular loss of personal time that starts to get on your nerves.

My remarks here don't even begin to cover being on-call which is a whole different monster headache in and of itself. So I'm not trying to find solutions for that here.

In my case, I guess a big factor is that we are a medium sized application which supports a DevOps team of only so many people and we must find a way to work closely together. Perhaps larger applications with bigger revenue may have better ideas to solve this. What are your experiences?

https://redd.it/y35o0e
@r_devops

Productive infrastructure for a small agency to manage multiple products?

Assume you are in charge of a tiny dev agency (< 5 developers). Those are developers who don't want to spend a ton of time maintaining a DevOps infrastructure.

Your agency develops and maintains multiple small products. Let's say 20 of them. Products vary from simple static Next.js apps to full-blown things with frontend in Angular, authentication service implemented with golang, business logic service written in Elixir, C++ service that handles some stuff, multiple databases, and queues like rabbitmq.

Some products change rapidly and new features are rolled out frequently. Sometimes multiple times a day. All products live on the same dedicated server atm.

When some product grows - you branch it off to a separate server. Note that we're not interested in infrastructure for a single product here. The focus here is managing multiple products on a single dedicated server.

The whole team is comfortable with Linux & terminal. No one has production experience with Nomad, K8s, or dokku, but everybody is just tired of docker swarm, Nginx configs, let's encrypt, etc.

I'd love to have a couple of declarative config files on GitHub that would sync the server state to declaration changes through Actions. For example:

>nvim config.cfg

>
>traefik instance {}
.... other apps
>
>app ProductH {
>
>service X is a docker app:0.3 from local docker registry
service Y is a docker postgres:11, password "whatever", data_dir "/whatever"
service Z is some executable pulled down from github release/1.2.19
service Z can talk to service X
service X can talk to service Y
inject service Y postgres secrets to service X environment
inject service X address to service Z environment as serviceX.service.local
expose service X to global traefik under "product.xyz" domain, ssl: true, allow wss: true
}
..... other apps

>
>:wq
plan
deploy to local
push to github/infrarepo/release
deploy to producton

Sounds similar to HashiCorp. I thought about and even tried setting up Ansible + nomad + consul + vault + traefik, but making it production-ready and maintaining seems to be not as easy as everybody believes afaiu, even on a single node.


What kind of infrastructure would you advise those developers to set up?

Would Ansible + Dokku work great for that? I know most things won't be declarative, but for the sake of productivity, I'm ok with it.

Is it worth investing in making the HashiCorp stack work smoothly? How much pain should I be ready to handle if I aim to self-host Vault?

https://redd.it/y37o8b
@r_devops

Who forgets their EC2 machines running ?(spoiler alert - me)

Hey folks,
As an engineering manager (and also as IC) I found myself struggling with EC2 machines that are left up and running long after the work was completed.
So who forgets their machines up in the air?

I'll appreciate honest answers. We're all humans after all :)

View Poll

https://redd.it/y36ebx
@r_devops

"Paved Road" Internal PaaS

Has anyone heard the phase “paved road” before? Apparently it was coined at Netflix to describe their internal PaaS. I know other large enterprises have similar platforms internally.

Have any of you built or used systems like this? I may be shepherded down this path and would love to know what works and what doesn’t, or even better glaring mistakes to watch out for.

Can anyone provide some wisdom?

https://redd.it/y3fcoz
@r_devops

Who's running Crossplane in Production?

Curious to see how Crossplane has been performing for people in production environments. Who's using it, how they're using it (GitOps through ArgoCD?), and at what scale.

Overall it seems like a really interesting project trying to handle (what seems like) a lot of things. Also, I wanted to see if any one had thoughts or where the project is still immature compared to a similar tool like Terraform or AWS CDK.

https://redd.it/y3fo9p
@r_devops

Which software does UI search the best?

Are you able to search and find what you're looking for easily? Can you search across multiple rows/columns in a table with a single search? Ex. If you're searching an IP inventory list, could you search a specific IP and find all resources associated with that IP? overlapping CIDRs?

Are you able to search by things like regions or compartment from that same search bar?

Does Azure have search powered by Bing? Is it worthless?

https://redd.it/y3hosd
@r_devops