Nomad Routing Question

I've been reading up on Nomad, trying to gauge how it works, and I have a question about the network.

Most documents recommend using a service mesh like Consul, but my question is: Does it route the traffic through Consul itself or is it just for service discovery?


I.e., is it: User Request -> Load Balancer -> Consul (on lb) -> Consul (on host) -> Web Service Container?


Or is it: User Request -> Load Balancer -> Web Service Container, where the Consul plugin for, say HAProxy just tells it which hosts it should route to?

https://redd.it/sdbpxk
@r_devops

Anyone experienced with squid proxy?

Have deployed 3.5 squid proxy to act as a firewall and a transparent proxy ( end user doesn’t know there’s a proxy) it works fine for the most part but some dev have complained that they get errors like server retuned error in unknown format ( they have been using salesforce bulkapi through java clients)
I check the logs and nothing is there on squid side. It happens only very rarely, same job goes through upon reruns. Scratching my head as I am not able to reproduce this problem. Does anyone have any suggestions?
Thanks

https://redd.it/sdhkxo
@r_devops

GH Actions + AWS - Is Terraform even needed?

Hey there,

So long story short, there is no DevOps guy on my team yet, so I'm helping with researching and basic setup in the meanwhile.

We are moving to a container based solution for one of our applications, and I was suggested to look into Terraform.

While I do understand what Terraform is for, describing a infrastructure on a special syntax, and let the TF do its magic to achieve that state by manipulating a remote env - in this case AWS - I'm having a bit of a trouble to understand why should I use it when there is a GH Action to deploy a container on AWS directly.

Could anybody provide a bit of insight of pros/cons of using or not using Terraform in this scenario? TIA

https://redd.it/sdi8ir
@r_devops

Kubernetes Cluster deployment using tekton

So we would like a way to be able to easily deploy new Kubernetes clusters whenever we would like all with the same configuration so we can easily deploy our apps, and are thinking of doing that in tekton. So you would run the tekton pipeline locally in minikube, and steps would be taken to install all the dependencies onto the (possibly bare metal) server, I suppose through an ssh connection.

Is this a good idea? Is there a standard way to do something like this? Thanks

BTW, I'm an inexperienced junior developer. So if this is not the right approach please let me know

https://redd.it/sdi2yx
@r_devops

Advice needed: creating CICD pipeline per code or manually

Hi guys, hope you can help me, give me an advice or have some good ideas. Currently we have lets say a pipeline generator where we can create new AWS codepipelines with just creating a small config file which includes the naming, source repo… works pretty fine, written in Typescript and uses CDK. We have 4 pipeline „types“ which differs on the stages and trigger type (s3, ecr, codecommit…). Now more and more developers ask for a custom pipeline because they want different stages, approval steps and so on instead of those pipelines we currently can create via the cicd pipeline generator.

PS: the cdk code of the generator is not written by us it was an external company.

Would you extend the generator to add more pipeline „types“ even if only one project need this custom pipeline, most of the requests are completely different use cases and so the pipelines.

Also creating a pipeline with the generator and modify the codepipeline manually to meet the requirements was a question from the manager. It would work but because the pipeline would be created with CDK so in the end Cloudformation and messing around manually is never a good idea, what you think?

Or would you manually create those custom pipelines? but than it would not be IaC :/ What management force us.

What you think?

https://redd.it/sdkzwn
@r_devops

Cheap CDN option for serving 50TB of video traffic in South America?

Hello,

I'm working on a project for a non-profit doing education via a video course online. They have a project which will require them to get a lot of people through their video course which will end up being about 50TB of video downloads when all is said and done (+/- 20%).

I've been looking at CDN options and so far the cheapest I can find is using DigitalOcean spaces (S3 clone) w/ built in CDN which will be $0.01 per GB of bandwidth transferred so about $500 for the 50TB (not bad!).

The downside with DigitalOcean is the CDN PoP locations aren't close to where the end users will be (in South America) and I worry about latency and playback start rate for the videos...

Cloudflare has closer PoP locations but their sales people are quoting me $5k/month minimum with 1 year contract which would be a starting amount of $50k and not something the non-profit can afford right now.

​

Are there any other CDN solutions for serving the 50TB of video (and in general for hosting video for fairly cheap) with good PoP locations in South America I might be overlooking?

https://redd.it/sd9idn
@r_devops

Cheap CDN option for serving 50TB of video traffic in South America?

Hello,

I'm working on a project for a non-profit doing education via a video course online. They have a project which will require them to get a lot of people through their video course which will end up being about 50TB of video downloads when all is said and done (+/- 20%).

I've been looking at CDN options and so far the cheapest I can find is using DigitalOcean spaces (S3 clone) w/ built in CDN which will be $0.01 per GB of bandwidth transferred so about $500 for the 50TB (not bad!).

The downside with DigitalOcean is the CDN PoP locations aren't close to where the end users will be (in South America) and I worry about latency and playback start rate for the videos...

Cloudflare has closer PoP locations but their sales people are quoting me $5k/month minimum with 1 year contract which would be a starting amount of $50k and not something the non-profit can afford right now.

​

Are there any other CDN solutions for serving the 50TB of video (and in general for hosting video for fairly cheap) with good PoP locations in South America I might be overlooking?

https://redd.it/sd9idn
@r_devops

Can't connect to MariaDB from a container

So I deployed an app from a container, based on Alpine. It's supposed to connect to a baremetal MariaDB on a different host, but it just won't do that.

* Connect to MariaDB from the Docker host (i.e., outside container) -- works
* ping to MariaDB from inside the container -- works
* `curl https://ifconfig.me` from inside the container -- works
* Connect to MariaDB from inside the container -- timeout

I don't know what else to do at the moment.

Additional info:

* It's part of a 2-node swarm
* Host OS is Ubuntu 20.04
* I'm managing the swarm using Swarmpit

I'd really appreciate any help in troubleshooting this issue.

https://redd.it/sdouyp
@r_devops

Switching to Sr Cloud Ops Engineer from SRE

I start my new job as a Sr Cloud Ops Engineer next month. Right now I am a SRE with 5 years of experience in AWS, IAC, serverless, Jenkins, etc. To my understanding the new job will be working with app teams on diagnosing their cloud environments and CI/CD pipelines. Feeling under prepared for the new job and am quite frankly nervous as this is a big jump in my career. Does anyone have any tips for somebody transitioning to a senior operations role?

https://redd.it/sdgrmd
@r_devops

𝐄𝐯𝐞𝐫𝐲𝐭𝐡𝐢𝐧𝐠 𝐘𝐨𝐮 𝐍𝐞𝐞𝐝 𝐭𝐨 𝐊𝐧𝐨𝐰 𝐀𝐛𝐨𝐮𝐭 𝐘𝐀𝐌𝐋

Please check out my post in Better Programming-𝐄𝐯𝐞𝐫𝐲𝐭𝐡𝐢𝐧𝐠 𝐘𝐨𝐮 𝐍𝐞𝐞𝐝 𝐭𝐨 𝐊𝐧𝐨𝐰 𝐀𝐛𝐨𝐮𝐭 𝐘𝐀𝐌𝐋
* YAML Stands for 𝐘𝐀𝐌𝐋 𝐀𝐢𝐧’𝐭 𝐌𝐚𝐫𝐤𝐮𝐩 𝐋𝐚𝐧𝐠𝐮𝐚𝐠𝐞.
* YAML is similar to 𝐉𝐒𝐎𝐍 𝐨𝐫 𝐗𝐌𝐋..
* YAML is used to write 𝐜𝐨𝐧𝐟𝐢𝐠𝐮𝐫𝐚𝐭𝐢𝐨𝐧 𝐟𝐢𝐥𝐞𝐬.
* YAML is used by 𝐃𝐨𝐜𝐤𝐞𝐫, 𝐊𝐮𝐛𝐞𝐫𝐧𝐞𝐭𝐞𝐬, 𝐀𝐖𝐒 𝐜𝐥𝐨𝐮𝐝𝐟𝐨𝐫𝐦𝐚𝐭𝐢𝐨𝐧, 𝐉𝐞𝐧𝐤𝐢𝐧𝐬, 𝐀𝐧𝐬𝐢𝐛𝐥𝐞, and several other tools.

https://betterprogramming.pub/everything-you-need-to-know-about-yaml-fdbb7acf6db6

https://redd.it/sdrkxf
@r_devops

Contract Negotiation for On-Call Compensation

I'm nearing the end of the interview process for a DevOps Engineer position and they've indicated that there are on-call responsibilities for the role. I am trying to gauge what fair compensation is and what to take into consideration. So far I have:

Stand-by compensation
Per incident compensation
Company cell phone
Service level agreement (how quickly to call-back)

Curious about what other people have in their current roles, not sure what fair market rate is for this.

In the past, I worked in a position where I had a company cell phone, got paid for stand-by (an extra 1/hr per weekday, 2/hr weekend), and got paid normal OT for time spent on an incident (round up to the nearest hour). Most of the time I would end up with enough OT to get time and half on the extra hours.

Bonus points: also curious about severance for these roles, especially when getting into the $150k+ salary range!

https://redd.it/sdcxno
@r_devops

IAM Policy to restict usrs do destory only instances that they own

Hi guys, I used [CloudCustodian to set up a Lambda function](https://cloudcustodian.io/docs/aws/examples/ec2-auto-tag-user.html) that adds a tag (CreatorName) to any newly created instance.

This part works quite well.

I'm now attempting to create an IAM policy that would allow only users that has its value in the EC2-instance CreatorName tag, do delete the machine.

This is the policy:

{
"Version":"2012-10-17",
"Statement":[
{
"Effect":"Allow",
"Action":[
"ec2:TerminateInstances"
],
"Condition":{
"StringEquals":{
"aws:ResourceTag/CreatorName":"${aws:username}"
}
},
"Resource":"arn:aws:ec2:<redacted>:<redacted>:instance/*"
}
]
}

This simply does not work.

I have a hunch as to why, users login to AWS via SAML, so they're in the "SAML federated users" status.

This leads me to believe that variable ${aws:username} in the above template doesn't actually correspond to my login name.

So in example, the action is actually carried out by a user named 'admin', where my user (TEseSKal) is just the 'principal', right?

Here's the CloudTrails audit:

{
"eventVersion":"1.08",
"userIdentity":{
"type":"AssumedRole",
"principalId":"<redacted>:<redacted>",
"arn":"arn:aws:sts::<redacted>:assumed-role/Admin/<redacted>",
"accountId":"<redacted>",
"accessKeyId":"<redacted>",
"sessionContext":{
"sessionIssuer":{
"type":"Role",
"principalId":"<redacted>",
"arn":"arn:aws:iam::<redacted>:role/Admin",
"accountId":"<redacted>",
"userName":"Admin"
},
"webIdFederationData":{

},
"attributes":{
"creationDate":"2022-01-26T22:01:24Z",
"mfaAuthenticated":"false"
}
}
},
"eventTime":"2022-01-26T22:01:51Z",
"eventSource":"ec2.amazonaws.com",
"eventName":"TerminateInstances",
"awsRegion":"<redacted>",
"sourceIPAddress":"<redacted>",
"userAgent":"console.ec2.amazonaws.com",
"requestParameters":{
"instancesSet":{
"items":[
{
"instanceId":"<redacted>"
}
]
}
},
"readOnly":false,
"eventType":"AwsApiCall",
"managementEvent":true,
"recipientAccountId":"<redacted>",
"eventCategory":"Management",
"sessionCredentialFromConsole":"true"
}

So, am I correct in this assumption?

If so, is there way to make the policy take into account the principal, and not the user?

I Googled it but couldn't make any meaningful progress.

https://redd.it/sdi2pt
@r_devops

What don't you like about Heroku and PaaS ?

I plan to build a new PaaS alternative to Heroku, and cheaper.

Can you tell me what don't you like about Heroku and other PaaS ?
Which features do you like ? And which ones would you want see on a cloud platform ?
And finally what is your use case of Heroku ? What do you build on it ?

Thanks.

https://redd.it/sdtx0c
@r_devops

How do you deploy SqlServer schema changes from git?

A lot of our developers have been using RedGate SQL Source Control to check in and manage their schema changes across releases for both monolithic and non-monolithic systems. In general this schema is maintained within the same repo as the codebase and it is released prior to the code release(after passing backwards compatibility testing).

Given that we are using RedGate already something like SQL Compare would probably work but it looks like the SQL Compare CLI requires a far more expensive package to use and we don't really want to have to host a Redgate Deploy server ourselves as we have been trying to eliminate self-hosted dev tooling from our stack out of maintenance concerns.

Does anyone know of a decent way of taking SQL schema and deploying it via CLI diffs that doesn't require some crazy self-hosted or costly licensing? We are using github actions for all new development so something that would be easy to tie into that would be helpful.

https://redd.it/sdaavx
@r_devops

Open-source cloud cost policies?

I wanted to share an interesting learning from the open source project https://github.com/infracost/infracost/ and see what people think about writing policies for cloud costs using things like Open Policy Agent and HashiCorp Sentinel. For example: https://github.com/infracost/actions/tree/master/examples/opa

I worked on that project because the people who are purchasing cloud resources are not shown costs upfront, so they don’t know how much the resources will cost before launching them. My assumption was that because this is open source and engineers are flying blind, they (the engineers) would pull it into their workflow. Actually, something different is happening:

The engineers are not pulling it in - it turns out to be the senior DevOps, SREs and platform teams. One of their challenges is figuring out how their small team of 7 people can fulfill the infrastructure requirements of hundreds of engineers. To solve this, they have created and put in place processes for engineers to provision infrastructure when they need. Now they want to implement cost policies and guardrails so that these hundreds of engineers don’t blow past all budgets. For example, if a change will result in a higher than 15% increase, leave a warning. If a change results in a >25% increase in costs, block the change till a team lead has reviewed it.

This has two implications for us. First, we need to create an output that isn’t only used by humans but is also digested into other systems to make further decisions. The second is the people we have been speaking to are not our end users. We need to figure out how we can get introduced to our end users, and create a different set of questions for each persona.

So I'd love to hear how you think about policies and guardrails for containing cloud costs.

https://redd.it/sda490
@r_devops

This image will help to get started with Kafka hands-on and understand its concepts

https://github.com/bluxmit/alnoda-workspaces/tree/main/workspaces/kafka-workspace

https://redd.it/sdwtbz
@r_devops

Need help researching and specifying company devops strategy

I work in a small company, ~20 employees of which we are only 3 people (soon more) in the development department. I am responsible for the devops side of things along side full stack development, as we grow, I hope to be able to focus on devops.

I am currently researching the area more in depth, in order to write out an initial draft of considerations and descriptions of our near-future and long-term devops strategy. Below I have drafted the headlines I intend to describe, with initial thoughts and questions I have for the sections, please tell if I am missing any:

# Workflows

This is the section I am most in doubt about how to approach.

I intend to describe branching and release strategy based off trunk based development. Any other resources would be good as well to help build a deeper understanding.

Here I will also describe continues integration, delivery and deployment. I feel I have an intuition about these, but I really need some resources to read to better my understanding of especially how to handle the integration part.

# Infrastructure
We have our own server, mainly because of a great need for lots and lots of disk space (we are using about ~120TB at the moment, wav files in multiple iterations for > 25K titles takes up a lot of space).

The server itself is managed by another company, I am only controlling the already created virtual machines (ubuntu).

We are currently hosting our application through a self hosted docker swarm, but I am thinking that we would be better of utilizing some managed kubernetes cloud, instead of continuing to host our own swarm as the complexity rises, it is getting more and more difficult to manage. Kubernetes in the cloud should also give us higher scalability and maintainability. But because of our massive need for data storage, I don't think a pure cloud solution is feasible, or am I missing some details?

# Monitoring
Humio WIP

# Security
WIP - I need resources for what I need to consider here.

https://redd.it/sdwx6p
@r_devops

Having a difficult time splitting traffic for one domain via cloudfront

I'm moving a legacy config from on-prem to AWS. The site was originally PHP and a new react platform was later developed on the same domain name. Basically what's happening is there's a single nginx server which sends routes for the new react platform to the react app and the rest get handled by PHP.

I'm trying to accomplish something similar in AWS. At first I thought I could use an ALB and split traffic to cloudfront and the PHP stuff to another target, but it looks like you cannot send traffic from the ALB to cloudfront (aside from a redirect).

So I did a bit more research and it seems that the recommended way is to use Cloudfront first with multiple origins and redirect traffic based on behaviours.

I understand how this works but I'm having a lot of trouble making it all work the way I want it to, mostly because react is a single page index.html. We have other single react apps hosted in cloudfront/s3 and this is easy to deal with by setting default root to index.html and setting up 404 and 403 error in cloudfront to redirect to index.html. Both the default root and the error pages apply everywhere though, it's not per origin. So if I set an index.html default root for example all requests use that.

I'm wondering if anybody has done something similar before and if you've found a working solution to split traffic like this with Cloudfront with a react site. Can it be achieved with the s3 bucket not having hosting enabled or without bringing in any additional cloudfront/lambda functions to modify the request etc.?

origin #1 : cloudfront > oai > s3 (hosting disabled).

origin #2 : cloudfront > ALB > internal IP of PHP web server


For behaviours I have setup react routes first so /react-route goes to the s3 origin for example and the default catchall (the very last rule) is the * catchall and directs the rest of traffic to PHP web server.

https://redd.it/sdyoxa
@r_devops

Is Kubernetes useful outside of Cloud environments?

Hi! I'm currently working on redistributing services from one server to another group of servers. Kubernetes sounded like a useful tool for this, since it would allow me to place every server inside the cluster and manage them quite easily. So i started reading and practicing kubernetes, but every example that showed up involved a cloud. Now i'm a little confused, is kube really useful for my problem?

https://redd.it/se035a
@r_devops

How Infrastructure as Code Should Feel

More and more IaC seems to be the default approach to provisioning cloud infrastructure. But with that there is a risk that it is implemented in a "paint by numbers" way, just something else to tick off when starting a new project. In this blog post I don't detail how to implement infrastructure as code, nor do I evangelize the benefits of implementing it, instead I describe how infrastructure as code should feel for those who already have it and hopefully provide a path back to Nirvana for anybody who isn’t realising the benefits it can bring.

How Infrastructure as Code Should Feel

https://redd.it/se269e
@r_devops

Seeking advice, recommendation

Hey guys,

I'm building a fairly simple/lightweight private app for a BigCommerce store.

Naturally, the app needs to be hosted and so I was looking for some recommendations, preferably AWS.

I'm virtually certain we would be able to stay within the confines of the free tier and I'm oscillating between Amazon EC2 and AWS Lambda.


Thanks for any feedback!

https://redd.it/se3enq
@r_devops