24/7 services and global teams

I support a B2B application that is deployed in the cloud across multiple regions and supports customers around the world. Our team is located in several areas around the world. Do any of you work like that? What challenges and best practices do you have?

My observations: To form great working relationships you have to be online together at least occasionally. Trying to make it work exclusively with asynchronous email, chat, JIRA comments etc is not very effective at building rapport, getting things done efficiently or solving problems.

To remedy the lack of overlap, we try to have a common meeting at some odd hour, trying to rotate whose time zone gets the most advantage. This helps a ton and fortunately all sides have been willing to work together on this. But I know from discussions w/ individuals this practice, long-term, begins to weigh heavily on them. It is the regular loss of personal time that starts to get on your nerves.

My remarks here don't even begin to cover being on-call which is a whole different monster headache in and of itself. So I'm not trying to find solutions for that here.

In my case, I guess a big factor is that we are a medium sized application which supports a DevOps team of only so many people and we must find a way to work closely together. Perhaps larger applications with bigger revenue may have better ideas to solve this. What are your experiences?

https://redd.it/y35o0e
@r_devops

Productive infrastructure for a small agency to manage multiple products?

Assume you are in charge of a tiny dev agency (< 5 developers). Those are developers who don't want to spend a ton of time maintaining a DevOps infrastructure.

Your agency develops and maintains multiple small products. Let's say 20 of them. Products vary from simple static Next.js apps to full-blown things with frontend in Angular, authentication service implemented with golang, business logic service written in Elixir, C++ service that handles some stuff, multiple databases, and queues like rabbitmq.

Some products change rapidly and new features are rolled out frequently. Sometimes multiple times a day. All products live on the same dedicated server atm.

When some product grows - you branch it off to a separate server. Note that we're not interested in infrastructure for a single product here. The focus here is managing multiple products on a single dedicated server.

The whole team is comfortable with Linux & terminal. No one has production experience with Nomad, K8s, or dokku, but everybody is just tired of docker swarm, Nginx configs, let's encrypt, etc.

I'd love to have a couple of declarative config files on GitHub that would sync the server state to declaration changes through Actions. For example:

>nvim config.cfg

>
>traefik instance {}
.... other apps
>
>app ProductH {
>
>service X is a docker app:0.3 from local docker registry
service Y is a docker postgres:11, password "whatever", data_dir "/whatever"
service Z is some executable pulled down from github release/1.2.19
service Z can talk to service X
service X can talk to service Y
inject service Y postgres secrets to service X environment
inject service X address to service Z environment as serviceX.service.local
expose service X to global traefik under "product.xyz" domain, ssl: true, allow wss: true
}
..... other apps

>
>:wq
plan
deploy to local
push to github/infrarepo/release
deploy to producton

Sounds similar to HashiCorp. I thought about and even tried setting up Ansible + nomad + consul + vault + traefik, but making it production-ready and maintaining seems to be not as easy as everybody believes afaiu, even on a single node.


What kind of infrastructure would you advise those developers to set up?

Would Ansible + Dokku work great for that? I know most things won't be declarative, but for the sake of productivity, I'm ok with it.

Is it worth investing in making the HashiCorp stack work smoothly? How much pain should I be ready to handle if I aim to self-host Vault?

https://redd.it/y37o8b
@r_devops

Who forgets their EC2 machines running ?(spoiler alert - me)

Hey folks,
As an engineering manager (and also as IC) I found myself struggling with EC2 machines that are left up and running long after the work was completed.
So who forgets their machines up in the air?

I'll appreciate honest answers. We're all humans after all :)

View Poll

https://redd.it/y36ebx
@r_devops

"Paved Road" Internal PaaS

Has anyone heard the phase “paved road” before? Apparently it was coined at Netflix to describe their internal PaaS. I know other large enterprises have similar platforms internally.

Have any of you built or used systems like this? I may be shepherded down this path and would love to know what works and what doesn’t, or even better glaring mistakes to watch out for.

Can anyone provide some wisdom?

https://redd.it/y3fcoz
@r_devops

Who's running Crossplane in Production?

Curious to see how Crossplane has been performing for people in production environments. Who's using it, how they're using it (GitOps through ArgoCD?), and at what scale.

Overall it seems like a really interesting project trying to handle (what seems like) a lot of things. Also, I wanted to see if any one had thoughts or where the project is still immature compared to a similar tool like Terraform or AWS CDK.

https://redd.it/y3fo9p
@r_devops

Which software does UI search the best?

Are you able to search and find what you're looking for easily? Can you search across multiple rows/columns in a table with a single search? Ex. If you're searching an IP inventory list, could you search a specific IP and find all resources associated with that IP? overlapping CIDRs?

Are you able to search by things like regions or compartment from that same search bar?

Does Azure have search powered by Bing? Is it worthless?

https://redd.it/y3hosd
@r_devops

How to save on ALB costs when deploying a simple fargate container on AWS?

Context: All the stuff I'm doing in AWS is for personal/education/demonstration purposes, so all costs are coming out of my pocket for no business/monetary gain.

I have a simple app running in a container (.25 vcpu/.5 mem). I have some cicd where a push to the main branch triggers a container image build and upload to ECR and then updates the task def with the new image. The cost of everything up to this point is pretty acceptable to me.

Next, I want to expose my container ( just a react frontend) to the internet, and apparently the best way to do that is to create an ALB, and Target Group, then associate those tasks to the TG, then the TG to the ALB. The problem is the single ALB would cost me nearly $20/mo to run. That's waaaay too much.

At that price, I'll just spawn a t3a.medium on a 3 year up front reservation and then host a bunch of containers on that with apache/nginx as a load balancer and expose the ec2 directly.

Is there any way around this ALB to save on costs? I'd like to be able to demonstrate a cicd pipeline that builds and deploys containers, and also show off my dockerfile stuff.

https://redd.it/y3jk4o
@r_devops

Hacktoberfest Open-Source DevOps Virtual Event (An Invite to r/DevOps!)

Hello r/DevOps friends!

We at the Apache DevLake team are hosting a Hacktoberfest event today focused on open-source DevOps, and I'd love to have you join us! We have some great speakers that I think you'll enjoy.

Hope to see many of you there, thanks for your support and friendship!

https://merico-dev.zoom.us/webinar/register/WN\_TnYZy7JDRmWJIhA9V3Fycg

https://redd.it/y3fbeg
@r_devops

What are the best, well-maintained Github Action library for publishing to npm, deploying to production, building and testing?

What are the best, well-maintained Github Action library for publishing to npm, deploying to production, building and testing? For some reason, there are thousands of such library and it seems once in a while they break and it can be a pain in the ass to fix them, so I was wondering if I should use a library so that it's easier to fix them when they stop working.

https://redd.it/y3j1dd
@r_devops

If an organisation does not use DevOps - what do they use instead to develop and release software?

Hi,


I am only familiar with working with DevOps to develop and release software. Organizations who are yet to adopt DevOps, what do they use instead? I can't imagine how organisations can cope otherwise.


Is it that they might use GitHub or AzureDevOps, but have yet to implement proper controls and teams around it? So they are not properly implementing the DevOps tools, just partly utilising them.


It would be great to get your feedback and experience on the adoption of DevOps.


Thank you

https://redd.it/y3negp
@r_devops

Any Platform to Excercise DevOps methods like we get for programming languages like CodeChef or HackerRank??

I was applying for a job and they asked me to do a assessment and redirected to codility page there i choose DevOps Nginx as a skill and the test was really good. So i check the site and it was for companies only to provide assessments for hire. Any other sites which i can use to sharp my DevOps Skills guys??

Thank You!!!

https://redd.it/y3s8p2
@r_devops

recently graduated software engineering student doing infrastructure - i like it, but am i shooting myself in the foot in the long run by doing less complex programming?

i really do enjoy the process of devops and creating scaling infrastructure to ensure everything is smooth and running.

however, looking at my other fellow colleagues who are doing more "programming"/software architecture/mathy stuff i can't help but worry that i might be shooting myself in the foot in the long run - especially since i want to become better at C++ (but at the same time i can totally understand the importance of having a good, stable pipeline)

i basically feel like an IT dude (for developers) working in linux and editing yaml files (although it is growing in complexity) and we will be soon transitioning towards a cloud environment.

any suggestions on how i can keep my "software engineering" skills sharp?

https://redd.it/y3sld5
@r_devops

Circle CI How can i block merging in github until successful deployment check

Hi all!

So my problem is I want people to not be able to merge into master branch until circleCi checks that their deployment would not fail. Im really new to configuring circle Ci, we have many different folders and we're working with salesforce. So my question is where to start? how do i make those checks? each job can take up to 10 min so how should i approach it to not make it to time consuming to check? How do i configure it?

https://redd.it/y3tmo8
@r_devops

In reality, how possible is an always releasable trunk branch?

With the intention to use trunk based development with one of its requirements being the trunk branch to always be in a releasable state, how likely is this actually in practise?

Having the target branch of any PRs be always releasable requires a suite of tests being run against it and blocking the PR on any failures. This is fine, but to cover all cases this test suite must surely be very extensive and take much longer than is recommended to run. How do we create excellent test coverage at this point whilst also not blocking PRs for many hours whilst it runs unit, integration, end to end browser testing , screenshot tests and more?

The two requirements for tests to be both extensive and fast seem to contradict each other when slower browser tests is surely also a necessity.

Have people found much success when migrating an existing project reliant on manual testing to this philosophy whilst maintaining usual business requirements?

https://redd.it/y3xub4
@r_devops

C# Make - Cake v2.3.0 released

Version 2.3.0 of the .NET based build orchestration tool Cake has been released with new Command aliases, new .NET Workload aliases, improved global caching of scripts, bug fixes, and more. 🚀🍰 More info in the release blog post:

https://cakebuild.net/blog/2022/10/cake-v2.3.0-released

https://redd.it/y3z7nb
@r_devops

Evaluating cloud computing

Hi all,

I am an information systems management student, currently writing my dissertation about cloud computing adoption for European and US SMEs.

The goal of the dissertation is to potentially create a new cloud onboarding strategy that companies can use when deciding to move their operations from on-premise to the cloud.

Part of the research consists of investigating the organization's experience through the use of an online survey.

If you have been part of a digital transformation that included the adoption of cloud services I would really appreciate if you could spare 15 minutes to share your insights in the following Google Forms Survey: https://forms.gle/9cdNFwJX3DVHCuYG8

Thank you.

https://redd.it/y40cnb
@r_devops

Personal liability in the event of a breach

I was curious what your thoughts are regarding the level of liability devops engineers would have in the event their org was breached. What got me thinking about this is the recent verdict on the Uber CISO : https://www.darkreading.com/attacks-breaches/what-the-uber-breach-verdict-means-for-cisos-in-the-us. Now that guy clearly veered of into active cover up behavior that none of us here would likely succumb to.

Obviously, as a devops engineer, I'm way down the totem pole and but if for example there's a breach and I know my org isn't following the regulations on timely disclosure, does this now mean that I have to automatically consider becoming a whistleblower just so I don't go to jail when the feds come knocking?

https://redd.it/y3xjvm
@r_devops

How should data population/cleanup scripts be ran in production?

I know, how dare I suggest running a script in production. I am a DevOps Engineer, I should never condone such craziness. But the truth is, sometimes a new table is created and needs to be populated with data, or data in an existing table needs to be cleaned up.

In my situation, this is a somewhat common occurrence and the scripts are written by engineers who obviously don't have access to production. We do have proper dev/qa/staging environments where the scripts can be tested. And these are big tables, so the scripts can take hours or even days to run.

What's the best way to allow developers to run these data population/cleanup scripts in production without compromising on security?

https://redd.it/y3yuw4
@r_devops

Best Training Platform?

Hi There,

Over the next 2 years. I want to improve my knowledge of DevOps and all things related.

I currently work in SEO and handle website migrations from one platform to another and have a friend that works in this industry and I want to pivot to this role as it sounds more interesting and I genuinely enjoy talking to him about his job and I'd like to do it.

Anyways.... I need to learn and get qualifications before I apply for junior roles.

What are the best learning platforms for someone looking to get into DevOps?

What about:-

Cloud Guru?
Cloud Academy?
AWS Training Platform?
Udemy?


I am happy to invest money and was originally looking at Cloud Guru but has some bad reviews so don't want to sink money into something that isn't worth it.

So I am turning to a wider group for guidance. Any help would be appreciated.

https://redd.it/y43m0u
@r_devops

In an ideal DevOps setting, where is the people in the loop?

Once you have a DevOps environment, it seems like if you successfully automate the workflows, then only the coding itself and the actual user are the only people in the loop. Is this true? I'm assuming that the pipeline is already established, so I'm not counting that person.

https://redd.it/y42ncb
@r_devops

Is there an Industry Standard CI/CD tool? (or some contenders for that title?)

I've worked a little bit with Azure Devops CI/CD and a little bit with CircleCI CI/CD. Is there an industry standard CI/CD tool yet? Or a few that I should keep my eye on? I'm wondering where I should spend my time learning.

https://redd.it/y4aipe
@r_devops