Cheap CDN option for serving 50TB of video traffic in South America?

Hello,

I'm working on a project for a non-profit doing education via a video course online. They have a project which will require them to get a lot of people through their video course which will end up being about 50TB of video downloads when all is said and done (+/- 20%).

I've been looking at CDN options and so far the cheapest I can find is using DigitalOcean spaces (S3 clone) w/ built in CDN which will be $0.01 per GB of bandwidth transferred so about $500 for the 50TB (not bad!).

The downside with DigitalOcean is the CDN PoP locations aren't close to where the end users will be (in South America) and I worry about latency and playback start rate for the videos...

Cloudflare has closer PoP locations but their sales people are quoting me $5k/month minimum with 1 year contract which would be a starting amount of $50k and not something the non-profit can afford right now.

​

Are there any other CDN solutions for serving the 50TB of video (and in general for hosting video for fairly cheap) with good PoP locations in South America I might be overlooking?

https://redd.it/sd9idn
@r_devops

Can't connect to MariaDB from a container

So I deployed an app from a container, based on Alpine. It's supposed to connect to a baremetal MariaDB on a different host, but it just won't do that.

* Connect to MariaDB from the Docker host (i.e., outside container) -- works
* ping to MariaDB from inside the container -- works
* `curl https://ifconfig.me` from inside the container -- works
* Connect to MariaDB from inside the container -- timeout

I don't know what else to do at the moment.

Additional info:

* It's part of a 2-node swarm
* Host OS is Ubuntu 20.04
* I'm managing the swarm using Swarmpit

I'd really appreciate any help in troubleshooting this issue.

https://redd.it/sdouyp
@r_devops

Switching to Sr Cloud Ops Engineer from SRE

I start my new job as a Sr Cloud Ops Engineer next month. Right now I am a SRE with 5 years of experience in AWS, IAC, serverless, Jenkins, etc. To my understanding the new job will be working with app teams on diagnosing their cloud environments and CI/CD pipelines. Feeling under prepared for the new job and am quite frankly nervous as this is a big jump in my career. Does anyone have any tips for somebody transitioning to a senior operations role?

https://redd.it/sdgrmd
@r_devops

𝐄𝐯𝐞𝐫𝐲𝐭𝐡𝐢𝐧𝐠 𝐘𝐨𝐮 𝐍𝐞𝐞𝐝 𝐭𝐨 𝐊𝐧𝐨𝐰 𝐀𝐛𝐨𝐮𝐭 𝐘𝐀𝐌𝐋

Please check out my post in Better Programming-𝐄𝐯𝐞𝐫𝐲𝐭𝐡𝐢𝐧𝐠 𝐘𝐨𝐮 𝐍𝐞𝐞𝐝 𝐭𝐨 𝐊𝐧𝐨𝐰 𝐀𝐛𝐨𝐮𝐭 𝐘𝐀𝐌𝐋
* YAML Stands for 𝐘𝐀𝐌𝐋 𝐀𝐢𝐧’𝐭 𝐌𝐚𝐫𝐤𝐮𝐩 𝐋𝐚𝐧𝐠𝐮𝐚𝐠𝐞.
* YAML is similar to 𝐉𝐒𝐎𝐍 𝐨𝐫 𝐗𝐌𝐋..
* YAML is used to write 𝐜𝐨𝐧𝐟𝐢𝐠𝐮𝐫𝐚𝐭𝐢𝐨𝐧 𝐟𝐢𝐥𝐞𝐬.
* YAML is used by 𝐃𝐨𝐜𝐤𝐞𝐫, 𝐊𝐮𝐛𝐞𝐫𝐧𝐞𝐭𝐞𝐬, 𝐀𝐖𝐒 𝐜𝐥𝐨𝐮𝐝𝐟𝐨𝐫𝐦𝐚𝐭𝐢𝐨𝐧, 𝐉𝐞𝐧𝐤𝐢𝐧𝐬, 𝐀𝐧𝐬𝐢𝐛𝐥𝐞, and several other tools.

https://betterprogramming.pub/everything-you-need-to-know-about-yaml-fdbb7acf6db6

https://redd.it/sdrkxf
@r_devops

Contract Negotiation for On-Call Compensation

I'm nearing the end of the interview process for a DevOps Engineer position and they've indicated that there are on-call responsibilities for the role. I am trying to gauge what fair compensation is and what to take into consideration. So far I have:

Stand-by compensation
Per incident compensation
Company cell phone
Service level agreement (how quickly to call-back)

Curious about what other people have in their current roles, not sure what fair market rate is for this.

In the past, I worked in a position where I had a company cell phone, got paid for stand-by (an extra 1/hr per weekday, 2/hr weekend), and got paid normal OT for time spent on an incident (round up to the nearest hour). Most of the time I would end up with enough OT to get time and half on the extra hours.

Bonus points: also curious about severance for these roles, especially when getting into the $150k+ salary range!

https://redd.it/sdcxno
@r_devops

IAM Policy to restict usrs do destory only instances that they own

Hi guys, I used [CloudCustodian to set up a Lambda function](https://cloudcustodian.io/docs/aws/examples/ec2-auto-tag-user.html) that adds a tag (CreatorName) to any newly created instance.

This part works quite well.

I'm now attempting to create an IAM policy that would allow only users that has its value in the EC2-instance CreatorName tag, do delete the machine.

This is the policy:

{
"Version":"2012-10-17",
"Statement":[
{
"Effect":"Allow",
"Action":[
"ec2:TerminateInstances"
],
"Condition":{
"StringEquals":{
"aws:ResourceTag/CreatorName":"${aws:username}"
}
},
"Resource":"arn:aws:ec2:<redacted>:<redacted>:instance/*"
}
]
}

This simply does not work.

I have a hunch as to why, users login to AWS via SAML, so they're in the "SAML federated users" status.

This leads me to believe that variable ${aws:username} in the above template doesn't actually correspond to my login name.

So in example, the action is actually carried out by a user named 'admin', where my user (TEseSKal) is just the 'principal', right?

Here's the CloudTrails audit:

{
"eventVersion":"1.08",
"userIdentity":{
"type":"AssumedRole",
"principalId":"<redacted>:<redacted>",
"arn":"arn:aws:sts::<redacted>:assumed-role/Admin/<redacted>",
"accountId":"<redacted>",
"accessKeyId":"<redacted>",
"sessionContext":{
"sessionIssuer":{
"type":"Role",
"principalId":"<redacted>",
"arn":"arn:aws:iam::<redacted>:role/Admin",
"accountId":"<redacted>",
"userName":"Admin"
},
"webIdFederationData":{

},
"attributes":{
"creationDate":"2022-01-26T22:01:24Z",
"mfaAuthenticated":"false"
}
}
},
"eventTime":"2022-01-26T22:01:51Z",
"eventSource":"ec2.amazonaws.com",
"eventName":"TerminateInstances",
"awsRegion":"<redacted>",
"sourceIPAddress":"<redacted>",
"userAgent":"console.ec2.amazonaws.com",
"requestParameters":{
"instancesSet":{
"items":[
{
"instanceId":"<redacted>"
}
]
}
},
"readOnly":false,
"eventType":"AwsApiCall",
"managementEvent":true,
"recipientAccountId":"<redacted>",
"eventCategory":"Management",
"sessionCredentialFromConsole":"true"
}

So, am I correct in this assumption?

If so, is there way to make the policy take into account the principal, and not the user?

I Googled it but couldn't make any meaningful progress.

https://redd.it/sdi2pt
@r_devops

What don't you like about Heroku and PaaS ?

I plan to build a new PaaS alternative to Heroku, and cheaper.

Can you tell me what don't you like about Heroku and other PaaS ?
Which features do you like ? And which ones would you want see on a cloud platform ?
And finally what is your use case of Heroku ? What do you build on it ?

Thanks.

https://redd.it/sdtx0c
@r_devops

How do you deploy SqlServer schema changes from git?

A lot of our developers have been using RedGate SQL Source Control to check in and manage their schema changes across releases for both monolithic and non-monolithic systems. In general this schema is maintained within the same repo as the codebase and it is released prior to the code release(after passing backwards compatibility testing).

Given that we are using RedGate already something like SQL Compare would probably work but it looks like the SQL Compare CLI requires a far more expensive package to use and we don't really want to have to host a Redgate Deploy server ourselves as we have been trying to eliminate self-hosted dev tooling from our stack out of maintenance concerns.

Does anyone know of a decent way of taking SQL schema and deploying it via CLI diffs that doesn't require some crazy self-hosted or costly licensing? We are using github actions for all new development so something that would be easy to tie into that would be helpful.

https://redd.it/sdaavx
@r_devops

Open-source cloud cost policies?

I wanted to share an interesting learning from the open source project https://github.com/infracost/infracost/ and see what people think about writing policies for cloud costs using things like Open Policy Agent and HashiCorp Sentinel. For example: https://github.com/infracost/actions/tree/master/examples/opa

I worked on that project because the people who are purchasing cloud resources are not shown costs upfront, so they don’t know how much the resources will cost before launching them. My assumption was that because this is open source and engineers are flying blind, they (the engineers) would pull it into their workflow. Actually, something different is happening:

The engineers are not pulling it in - it turns out to be the senior DevOps, SREs and platform teams. One of their challenges is figuring out how their small team of 7 people can fulfill the infrastructure requirements of hundreds of engineers. To solve this, they have created and put in place processes for engineers to provision infrastructure when they need. Now they want to implement cost policies and guardrails so that these hundreds of engineers don’t blow past all budgets. For example, if a change will result in a higher than 15% increase, leave a warning. If a change results in a >25% increase in costs, block the change till a team lead has reviewed it.

This has two implications for us. First, we need to create an output that isn’t only used by humans but is also digested into other systems to make further decisions. The second is the people we have been speaking to are not our end users. We need to figure out how we can get introduced to our end users, and create a different set of questions for each persona.

So I'd love to hear how you think about policies and guardrails for containing cloud costs.

https://redd.it/sda490
@r_devops

This image will help to get started with Kafka hands-on and understand its concepts

https://github.com/bluxmit/alnoda-workspaces/tree/main/workspaces/kafka-workspace

https://redd.it/sdwtbz
@r_devops

Need help researching and specifying company devops strategy

I work in a small company, ~20 employees of which we are only 3 people (soon more) in the development department. I am responsible for the devops side of things along side full stack development, as we grow, I hope to be able to focus on devops.

I am currently researching the area more in depth, in order to write out an initial draft of considerations and descriptions of our near-future and long-term devops strategy. Below I have drafted the headlines I intend to describe, with initial thoughts and questions I have for the sections, please tell if I am missing any:

# Workflows

This is the section I am most in doubt about how to approach.

I intend to describe branching and release strategy based off trunk based development. Any other resources would be good as well to help build a deeper understanding.

Here I will also describe continues integration, delivery and deployment. I feel I have an intuition about these, but I really need some resources to read to better my understanding of especially how to handle the integration part.

# Infrastructure
We have our own server, mainly because of a great need for lots and lots of disk space (we are using about ~120TB at the moment, wav files in multiple iterations for > 25K titles takes up a lot of space).

The server itself is managed by another company, I am only controlling the already created virtual machines (ubuntu).

We are currently hosting our application through a self hosted docker swarm, but I am thinking that we would be better of utilizing some managed kubernetes cloud, instead of continuing to host our own swarm as the complexity rises, it is getting more and more difficult to manage. Kubernetes in the cloud should also give us higher scalability and maintainability. But because of our massive need for data storage, I don't think a pure cloud solution is feasible, or am I missing some details?

# Monitoring
Humio WIP

# Security
WIP - I need resources for what I need to consider here.

https://redd.it/sdwx6p
@r_devops

Having a difficult time splitting traffic for one domain via cloudfront

I'm moving a legacy config from on-prem to AWS. The site was originally PHP and a new react platform was later developed on the same domain name. Basically what's happening is there's a single nginx server which sends routes for the new react platform to the react app and the rest get handled by PHP.

I'm trying to accomplish something similar in AWS. At first I thought I could use an ALB and split traffic to cloudfront and the PHP stuff to another target, but it looks like you cannot send traffic from the ALB to cloudfront (aside from a redirect).

So I did a bit more research and it seems that the recommended way is to use Cloudfront first with multiple origins and redirect traffic based on behaviours.

I understand how this works but I'm having a lot of trouble making it all work the way I want it to, mostly because react is a single page index.html. We have other single react apps hosted in cloudfront/s3 and this is easy to deal with by setting default root to index.html and setting up 404 and 403 error in cloudfront to redirect to index.html. Both the default root and the error pages apply everywhere though, it's not per origin. So if I set an index.html default root for example all requests use that.

I'm wondering if anybody has done something similar before and if you've found a working solution to split traffic like this with Cloudfront with a react site. Can it be achieved with the s3 bucket not having hosting enabled or without bringing in any additional cloudfront/lambda functions to modify the request etc.?

origin #1 : cloudfront > oai > s3 (hosting disabled).

origin #2 : cloudfront > ALB > internal IP of PHP web server


For behaviours I have setup react routes first so /react-route goes to the s3 origin for example and the default catchall (the very last rule) is the * catchall and directs the rest of traffic to PHP web server.

https://redd.it/sdyoxa
@r_devops

Is Kubernetes useful outside of Cloud environments?

Hi! I'm currently working on redistributing services from one server to another group of servers. Kubernetes sounded like a useful tool for this, since it would allow me to place every server inside the cluster and manage them quite easily. So i started reading and practicing kubernetes, but every example that showed up involved a cloud. Now i'm a little confused, is kube really useful for my problem?

https://redd.it/se035a
@r_devops

How Infrastructure as Code Should Feel

More and more IaC seems to be the default approach to provisioning cloud infrastructure. But with that there is a risk that it is implemented in a "paint by numbers" way, just something else to tick off when starting a new project. In this blog post I don't detail how to implement infrastructure as code, nor do I evangelize the benefits of implementing it, instead I describe how infrastructure as code should feel for those who already have it and hopefully provide a path back to Nirvana for anybody who isn’t realising the benefits it can bring.

How Infrastructure as Code Should Feel

https://redd.it/se269e
@r_devops

Seeking advice, recommendation

Hey guys,

I'm building a fairly simple/lightweight private app for a BigCommerce store.

Naturally, the app needs to be hosted and so I was looking for some recommendations, preferably AWS.

I'm virtually certain we would be able to stay within the confines of the free tier and I'm oscillating between Amazon EC2 and AWS Lambda.


Thanks for any feedback!

https://redd.it/se3enq
@r_devops

Came back to Devops after 10 years, so much changed but Jenkins is still the default CI/CD?!

Hi,

I'm a pretty experienced developer but new to modern Devops (used to do Devops but been out of the game for years), and I've been trying to choose a CI/CD tool. With so many other changes in the stack over the past few years, I was surprised that the default choice for CI/CD is still … Jenkins.

Several of my friends in DevOps told me that they started with Jenkins, switched to a commercial solution that seemed better, and then came back to Jenkins.

What I like about Jenkins:

Easy to get started -- has a good configuration UI, can ignore advanced features until you need them.
Powerful enough for complex projects, includes CaC.
Big community and lots of people writing good plugins.

Points against Jenkins:

You need to write your own build scripts.
You need to learn Groovy to use its CaC.
The UI just shows logs of your jobs. For example, when I build an environment using Terraform, I wish the UI showed me the results visually. (You can get this information from the Terraform logs, but it’s not seamless.)

If you've abandoned Jenkins, what made you do it?

If you've abandoned a commercial solution, what made you come back to Jenkins?

https://redd.it/se4ww6
@r_devops

On premise RTS confusion

I'm working on a real-time IoT system which will be deployed on premise, on a single virtual machine, scaling and high availability is not a concern here, the actual device is our critical part while the BE is more of a "nice to have". Also there will be a small amount of devices (<50) and our backend logic is not really that complex, we have like 6 subsystems

We need to support full duplex communication between browser UI and those devices. Our backend is running on NestJS. Communication with UI or devices is fairly straight forward, but where I'm struggling right now is deciding on how to make the communication backbone.
In cloud environment I'd use a message broker for it, that way logic is nicely decoupled and we have a nice buffer and a pub/sub interface

here, i'm not quite sure what is the optimal (or somewhat optimal) solution because of the following factors:

* people working on the project are quite inexperienced, so anything too complex would backfire on our ETA
* the virtual machine running this will not be that great, let's say it has 8GB RAM (maybe I'm overthinking on this part, but installing some software might hog too much of the resources)

So the question is:


* does it make sense to add something like RabbitMQ in here? (devices are using gRPC http2)
* do I just go Redis pub/sub?
* or just good old observer pattern?

https://redd.it/se68zg
@r_devops

Can anyone give an ELI5 of this article?

https://medium.com/@cfatechblog/bare-metal-k8s-clustering-at-chick-fil-a-scale-7b0607bd3541

I just started and want to get a better grasp of the DevOps world. This article is really interesting, but I feel like I don't understand how they use the technology.

Could anyone provide me with some information about how it works?

https://redd.it/se7jca
@r_devops

Learning Devops - Need help

Hey people,

&#x200B;

Why some of the microservices get service endpoint automatically but some not?

Cluster was created by terraform with elb and private and public subnets, I'm making use also in external-dns to manage public dns zone for my domain application.Cluster based on AWS EKS.

https://imgur.com/t020i6F

I attached a picture for your reference.

https://redd.it/se8l0b
@r_devops

It looks like docker networking is kind of sticked to the order or containers boot

I had 2 cases where it is definatelly a fault of the order containers boot

First happened when I ran

`docker network create some-shared-network`

and created two, or more projects with docker-compose that reused the external network to communicate with each other

This didn't work after I restart my machine and is mostly related to order of spinned up containers, service B requires service A to start first to be visible

Now I had similar problem with my selfhosted Jira software that could not communicate with the database which all were in the same stack (in the same non-external network)

I had to scale down the app service to 0 instances and then scale it up to previous values... recreating the stack didn't help... and suddenly it noticed the presence of the database...

Docker, what the heck

https://redd.it/se95az
@r_devops

Buddy: It just .Works

A few months ago my team and I set out to replace an existing WordPress site with a Gatsby.js PWA. We originally had a shared hosting plan, but as our Gatsby site became more and more fledged out, deployments to this hosting provider became increasingly difficult. Our original hosting platform was geared more towards WordPress hosting and did not come with CI/CD customization out of the box, so we ended up getting our own dedicated server on Cloudways – and that’s where Buddy comes into the picture. The perfect “middleman,” Buddy is the seamless fit for our Gatsby application – our first pipeline began with a staging environment and it involves 3 steps: as soon as the associated branch on GitHub receives a new push, Buddy prepares our environment by fetching and uploading the new files to our server. Finally, according to our package.json we are able to trigger node installations and a Gatsby build process to deploy our site. I just sit back and watch the logs of the pipeline to make sure all is well, and Buddy will just do its thing. It just works. Gone are the days of having to SSH into your server and manually doing everything yourself! And the best part? The free tier Buddy offers is more than generous enough to suit your every needs. Highly recommend checking them out – it’s worked wonders on someone like me who is more front-end-oriented and is quite new to DevOps.

https://redd.it/se9qoa
@r_devops