Integrate AWS SSO with Github

I am trying to integate AWS SSO with GitHub.

I have added the Sign On URL, Issuer URL and Public certificate from SSO on Github.

But I am stuck in the Application MetaData section in SSO.

For "Application SAML metadata file" I uploaded the file which I downloaded from above "AWS SSO SAML metadata file"

But when I Save Changes. It says "An error occurred, SPSSODescriptor is null."

I even tried entering meta values manually but I can't find ACS URL in the XML.

I am just confused about what to fill in ACS Url and Application SAML audience.

Any help?
Thanks!

https://redd.it/qfkifg
@r_devops

Cheapest Image/Video storage for my scenario?

Hey all,

I've been suggested to use Backblaze b2, but I noticed a flaw,

My goal to upload a file to a media storage and then store that uploaded file's URL in my database automatically with my API so I can refer to it. Backblaze b2 does not support this, I have to manually go to my storage, find my image link and then store it in my database.

I am deploying with Heroku. I am only using this so user can change their profile picture. It will not require any downloads.

I've heard Wasabi is the next cheapest option? Does that suit my case?

​

​

Here is a past post I made relevant to the subject on this subreddit :)! :

https://www.reddit.com/r/devops/comments/pr9dzh/cheapest\_imagevideo\_cloud\_storage\_for\_2021/

https://redd.it/qfjhuf
@r_devops

AWS secrets manager - Fargate: Can PHP FPM read the environment variables for root?

We currently have a PHP application running on EC2 (PHP7, php-fpm and Nginx) and we use .env file to store our credentials, and RDS is our database.

We dockerized the application and we want to move the application from EC2 to farmgate. We want to stop using .env file to store credentials. We want to use the AWS secrets manager instead.

There are two ways of implementing the AWS secrets manager, one way is to call the AWS rest API like this, but the DevOps prefers to export AWS secrets manager credentials to fargate as Linux environment variables like this I think

The problem is that, if you're running PHP from the terminal, it has access to the root user, so if you do getenv() you can get the correct value, if you run The PHP from the browser, it can only access its own enviroment, which are under www-data.

I asked the Dockerfile maintainers about this and I was told that their images can read the enviroment variables, here, I think the problem is that, the variables are exposed to root but not to www-data

Can I make php-fpm read the linux enviroment variables for the root user? If not, I was thinking of generating .env file dynamically every time the container runs by doing this

RUN printenv | sed -e 's/=/="/' -e 's/$/"/' > .env

The problem with this, is that I need to run that command every time a user adds a secret to the secrets manager and I don't want to recreate the container every time that happens. Any solution to that problem?

https://redd.it/qfms2z
@r_devops

More Experience In Environment Without a Lot of Need?

Not sure if this question is ok here so if not let me know and i'll remove it. So basically i'm an Azure cloud engineer and would like to get more devops experience but the current environment and business in general doesn't seem to have a big need for devops. I'm familiar with docker and ansible and a few of the technologies but only because i use them for personal use at home. We do utilize a good amount of services in Azure but i'm having trouble finding spots where devops would be helpful. For instance we only have about 15 vm's and only spin up a new one maybe every other month or so. I just don't see terraform being very useful for that. Also we don't really have any in house or custom apps. No in house developers. So no real need for app services or even container services or AKS. We have a few small apps but they are all built by a different team using power apps and power automate. I think i'm going at least start looking at ARM templates (or terraform) for different types of vm's to standardize that for future vm's. Any other suggestions of things to look into? I don't want to force new technologies just for the sake of having them if there is no real benefit. Just curious if anyone had any suggestions. It may just be this is an Azure environment that wouldn't benefit from devops tech. Sorry for the long post. Thanks for reading.

https://redd.it/qfowuo
@r_devops

ECS Fargate vs EKS

So, my startup is using ECS Fargate to run applications and it's going really well, it satisfies all our requirements. I mean, its fargate so we barely have to do anything, serverless is just amazing.

But if this is the case then why do people need Kubernetes, I mean apart from the fact that we can SSH into our application and that it's cloud native, I don't see any advantage of using Kubernetes.

I know I might have missed any points but that's point of this post, I want to learn more about this.

I have worked on Kubetnetes(not much though) and I found it pretty interesting and challenging but the startup I work for uses ECS Fargate and tbh it's kinda boring, and I feel like it stopped my growth.

https://redd.it/qfotkk
@r_devops

Learning devops hand-on, out of ideas

How do you do fellow devops engineers?

I come from a backend developer background and trying to learn devops to expand my skillset. I was trying to implement gitops principles but soon ran out of ideas.

What I have done till now:

1. Declared my infrastructure (GKE) using pulumi
2. Declared helm charts for ingress-nginx, cert-manager, external-dns
3. Deployed argocd and defined applications to deploy the above declared helm charts
4. Everything is automated and made easily recreatable

Now what? I was thinking maybe deploy Prometheus and grafana and configure it, can someone suggest me a better path? I want to learn how industry level infrastructure works.

If anyone's willing to go skim through my code to suggest changes:

infrastructure: https://github.com/jellyfish-squadddd/infrastructure

gitops: https://github.com/jellyfish-squadddd/gitops


PS: I do know that some of my secrets are exposed rn, I will fix it by using mozilla/sops tomorrow.

Thanks!!

https://redd.it/qfphn5
@r_devops

Resources on writing Python tests for CI/CD

I’m currently getting my feet wet with and learning DevOps, and read that part of the CI/CD process is to write tests that are automatically executed when code is pushed. Then, if the tests pass, the code is deployed to production.

So I’ve been working on a pet project that is pretty basic: an AWS Lambda function written in Python, API Gateway, and DynamoDB table on the back end, and a static web page with some JavaScript to fetch the API on the front end. I’ve got all this working through SAM and GitHub Actions, but I’m curious as to what tests I should be writing for QA for something this simple. I don’t think I’m grasping it enough and after pondering it for a couple days I wanted to ask if anyone here could point me in the right direction.

The Python is pretty simple: increment a value in the table then return that value. What kind of tests should I write for this??

https://redd.it/qfrn9p
@r_devops

Quality paid DevOps resources or classes

It's near the end of the year and I have a pretty large training budget I need to take advantage of. I was looking for some recommendations on good paid resources or classes. I come from a fairly heavy cloud infrastructure, cyber, and automation background but I'm starting to do more work with containerized environments (Kubernetes variants in both AWS/Azure and a lot of supporting toolsets luke Terraform, helm, docker, Jenkins, Gitlab etc.). I guess I'm looking for some quality resources beyond the typical Linkedin Leaning crap or low-quality Udemy courses and cost isn't an issue. I'm not against certs but I don't really want to manage renewing more certifications beyond what I have already. I appreciate any recommendations!

https://redd.it/qfopsa
@r_devops

Are we really doing CI/CD?.

In this post, We discuss the various spectrum of deployments and what Tailwinds observe with its customers.

https://medium.com/tailwinds-navigator/kubernetes-tip-is-ci-cd-really-popular-83887e1737e0

Do read and share your views/comments.

https://redd.it/qfz3dq
@r_devops

Holon Hackathon 2021

Hey all!

Wanted to invite you to a Hackathon I'm organising at the moment, sponsored by Filecoin and IPFS

Up to 55k in FIL to be won!

Check out this link for all the details https://taikai.network/en/holon/challenges/holon-innovate/ overview

https://redd.it/qfzfbt
@r_devops

Integrate GitHub with AWS SSO

I am trying to integrate GitHub into AWS SSO.

In the "Application Metadata" section that contains Application ACS URL and Application SAML audience. According to this link their values are different (I am taking AssertionConsumerService and EntityDescriptor) and when I upload the metadata file from AWS SSO page, it gives some different values.

​

When I copy values from that link, I get "Your request included an invalid SAML response" error.

When I upload the meta data file, I get "It's not you, it's us. We couldn't complete your request right now. Please try again later." error

​

I am confused about the attribute mapping part too, I have just mapped the default "Subject" to "${user:subject}".

I login on my github from email and on my aws with my username, I think attribute mapping is used for this but not really sure how to do it in console.

​

Not really sure what I am doing wrong here, need help.

Thanks!

https://redd.it/qg1vxz
@r_devops

Can some devOps people give me some advice regarding my SQL databases for a small project? (Microsoft SQL server)

Hi All,

I'm going to be developing a small Android application (by myself) for work and am wanting some advice or direction regarding the devOps side of things. The app should be relatively straight-forward and a separate.

This project will require an API + database with a few tables, nothing major. I'm quite experienced with SQL, however I'm not sure how to go about setting my development and production environments such that it doesn't wreck me later on.

I'm using Microsoft SQL server developer and SSMS for development, but will be using Microsoft SQL server Express for deployment.

I understand that I should use some SQL script, probably in source control, for creating the tables. It sounds like I should also have an easy way to push changes from the development database to production database, but I don't really know what that means or how to achieve it.

Some advice and direction would be very appreciated because I feel quite unsure about the how to proceed. I don't believe I require anything too fancy.

I'd be glad to answer questions in the chat if this isn't clear.

Thanks

https://redd.it/qg0iw9
@r_devops

Terraform & Makefiles - Better options?

Seeing a lot of people using Makefiles with Terraform for CI/CD integration and smoother deployments.

Does anyone have a better alternative or is this the current best practice?

Been using the AWS CDK extensively and just coming back to using Terraforming a new role after a long absence.

https://redd.it/qg8e36
@r_devops

Interview next week

I've a interview next week for a devops position(new grad). I've been told there will be coding challenges. Most of my experience are on docker, kubernetes, cicd and I've a basic understanding of python. So, I'm not sure if I will be making a fool out of myself at the interview. Is there anyway i can prepare myself in a week for python?

https://redd.it/qg9fxw
@r_devops

Add new package to poetry.lock file

I have an existing repository that already has a poetry.lock file and a pyproject.toml. I would like to add numpy to the list of packages. I tried using

>poetry add numpy

But it the doesn't work because I have to do this change locally and my local env doesn't have the python version required in the poetry.lock file. In the end I am running this code on a server so I don't want to install a different python version.

Is there any other way to add a new package ? I tried adding

>numpy = "1.21.1" # in the toml file

But when I build the project I get an error that there is no version matching numpy.

https://redd.it/qgf0ja
@r_devops

Looking to Automate AD User Administration

Hi. We maintain a sandboxed network (let's call it a training lab) that has a 2016 DC and a mix of Windows and RHEL (mostly 7) hosts. I'm looking to create a web page someplace to allow other admins/operators to administer general users in the domain, including such tasks as: adding, changing, removing. We're looking to do this for two reasons: 1) we're not satisfied with Active Directory Users and Computers (ASUC) tool is it lacks some UNIX integration functionality, and 2) we want to minimize potential for human error (e.g., picking the next unused uidnumber).

So...being that the network is air-gapped, and everything needs to be transferred in by DVD, can someone suggest native Windows/Linux tools that would be good for accomplishing this? We have IIS and apache, so the page could be hosted on either. Python and powershell are natively available to script in, but there are obviously other candidates. I'd be interested in hearing what others think/may have done in this regard.

https://redd.it/qgepkd
@r_devops

Free intro to Linux commandline/server course starting 1 November 2021


This free month-long course is re-starting again on the first Monday of next month.

This course has been running successfully now every month since February 2020 - more detail at: https://LinuxUpskillChallenge.org - daily lessons appear in the sub-reddit r/linuxupskillchallenge - which is also used for support/discussion.

Suitable whatever your background, and aims to provide that "base layer" of traditional Linux skills in a fun interactive way.

Any feedback very welcome.

https://redd.it/qgeibn
@r_devops

Github Enterprise Cloud Security Configuration

Hi Folks - is there any resource for recomemnded configuration of GitHub Enterprise Cloud security?

https://redd.it/qgj45f
@r_devops

Synapse Project Discussion K8S,Nomad Alternative

Hi guys, i want to start a discussion about my new pet project, Synapse it will be something like kubernetes or docker-swarm but easier and simpler to use, so here is the why first i am "reinventing the wheel" instead of using already made wheels.

Kubernetes: Too resource hungry, over bloated for small projects and homelab use.
Docker-Swarm: No new features, and i am having weird things happens in my cluster like images not updating correctly some services being auto deleted and no auto scale.
Nomad: The better of the three, light weight, nice UI, but i really really hate the allocation resource thingy, you have to specify the minimum ram and CPU mhz (i dunno why the fuck would someone allocate apps in mhz)

Ok the ranting is done, now i will explain how Synapse will do differently than the others.

First i will explain the concept, then i will explain the technical parts of the project.

Concept:

I want to make it a UI first, config file second approach, so everything will be available in the UI to manage and see the status of the nodes, servers and etc.
Master/server will handle container allocation, UI, ReverseProxy(With load balancing), Wireguard connection between nodes, basically an all in one package.
If nodes and master is not on the same lan network(which will be most cases) the communication between them will be made exclusively through the VPN connection, for example you can host the master on a VPS which has access to the public internet, and it will manage and route traffic through the vpn connection to the machines in your home.
When you create a Service/Job you will be able to specify if you want to prioritize RAM, Disk or CPU and the scheduler will try to find the machine in your cluster that has the best of the three options and deploy there, but if needed will use the other nodes as well.
Actively move services around to optimize the hardware usage, for example if in NodeA exists ProcessA which uses 30% of cpu, ProcessB which uses 1% and ProcessC which uses 5%, and NodeB has 4 process which are barely using cpu, Synapse will move ProcessB and ProcessC to NodeB so ProcessA can utilize the NodeA cpu to the maximum
I will try to implement all the parallel updating features, health check features that k8s,nomad,docker-swarm to synapse as well

OBS: I am a C# lover so the whole stack will be in C#

Technical aspect:

Server will be one project which will use Blazor server-side for the user interface, YARP for the reverse proxy and load balancer, LettuceEncrypt for the auto SSL goodness, Docket.Dotnet for the docker communication API.
It will only support containers, no binaries or anything the like.
Will use LiteDB to save configuration and stats.
Communication between server and nodes will be made through a custom tcp socket for bi directional communication.
Probably the server will not be able to ru on High Availabity mode, if someone disagrees please tell me why and how i can make it HA.

Please tell me your opinions, hate on me on creating another new "wheel", any kind of feedback will be welcome, this is the first pet project that got me excited so i want to make it as user friendly and as feature rich as i can for personal use and for the community as well.

I have no plans to monetize it, it will be open source and i will accept any contribution, for now it is too early in the development and i will make the repository public next week probably.

And please ask away if you didnt understand parts of the project, i will try to explain it better.

TLDR: I am making an alternative to K8S, Nomad and Docker-Swarm, which will be easier and pain free to use on small-medium scale projects and homelab use and i want to discuss it so i can better make the project.

https://redd.it/qgjzjc
@r_devops

Which CI tool do you use/prefer at your work ?

We have number of CI/CD tools available now. And they provide competitive features to attract customers. Which CI tool our DevOps community use/prefer at their work ?

View Poll

https://redd.it/qgndeq
@r_devops

How do you debug an issue between varnish and Apache?

centos@ip-172-35-25-65 ~$ varnishlog
0 CLI - Rd ping
0 CLI - Wr 200 19 PONG 1635280998 1.0
0 CLI - Rd ping
0 CLI - Wr 200 19 PONG 1635281001 1.0
10 SessionOpen c 127.0.0.2 55870 127.0.0.2:80
10 ReqStart c 127.0.0.2 55870 894208400
10 RxRequest c GET
10 RxURL c /
10 RxProtocol c HTTP/1.0
10 RxHeader c X-Real-IP: 198.95.75.75
10 RxHeader c X-Forwarded-For: 198.95.75.75
10 RxHeader c X-Forwarded-Proto: https
10 RxHeader c X-Forwarded-Port: 80
10 RxHeader c Host: staging03.cherry.com
10 RxHeader c Connection: close
10 RxHeader c Cache-Control: max-age=0
10 RxHeader c Authorization: Basic aGc6am9objEyMw==
10 RxHeader c Upgrade-Insecure-Requests: 1
10 RxHeader c User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.54 Safari/537.36
10 RxHeader c Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,/;q=0.8,application/signed-exchange;v=b3;q=0.9
10 RxHeader c Accept-Encoding: gzip, deflate
10 RxHeader c Accept-Language: en-US,en;q=0.9,fr;q=0.8
10 RxHeader c Cookie: ajsanonymousid=%22424f4cd9-cbbc-4ead-83b1-273cb21cf453%22; fbp=fb.1.1630002144579.2012566540; qca=P0-1416512434-1630002144589; edwvts=708154457303700204; gid=GA1.2.1572498662.1635275261; ajsuserid=%224543534%40mimpi99.com%22; gclau=1.1.
10 VCLcall c recv pass
10 VCLcall c hash
10 Hash c /
10 Hash c staging03.cherry.com
10 Hash c 80
10 Hash c ajsanonymousid=%22424f4cd9-cbbc-4ead-83b1-273cb21cf453%22; fbp=fb.1.1630002144579.2012566540; qca=P0-1416512434-1630002144589; edwvts=708154457303700204; gid=GA1.2.1572498662.1635275261; ajsuserid=%224543534%40mimpi99.com%22; gclau=1.1.1880042
10 VCLreturn c hash
10 VCLcall c pass pass
10 FetchError c no backend connection
10 VCLcall c error deliver
10 VCLcall c deliver deliver
10 TxProtocol c HTTP/1.1
10 TxStatus c 503
10 TxResponse c Service Unavailable
10 TxHeader c Server: Varnish
10 TxHeader c Content-Type: text/html; charset=utf-8
10 TxHeader c Retry-After: 5
10 TxHeader c Content-Length: 392
10 TxHeader c Accept-Ranges: bytes
10 TxHeader c Date: Tue, 26 Oct 2021 20:43:23 GMT
10 TxHeader c X-Varnish: 894208400
10 TxHeader c Via: 1.1 varnish
10 TxHeader c Connection: close
10 TxHeader c X-Age: 0
10 TxHeader c X-Cache: MISS
10 Length c 392
10 ReqEnd c 894208400 1635281003.852778196 1635281003.852984428 0.000073195 0.000165701 0.000040531
10 SessionClose c error
10 StatSess c 127.0.0.2 55870 0 1 1 0 1 0 273 392
0 CLI - Rd ping
0 CLI - Wr 200 19 PONG 1635281004 1.0
0 CLI - Rd ping
0 CLI - Wr 200 19 PONG 1635281007 1.0
0 CLI - Rd ping
0 CLI - Wr 200 19 PONG 1635281010 1.0
0 CLI - Rd ping
0 CLI - Wr 200 19 PONG 1635281013 1.0


I tried to log what was happening when I got from the client side:


Error 503 Service Unavailable
Service Unavailable

Guru Meditation:
XID: 894208400

​

Now, I thought it was because of Apache not running, because when I close varnish I get a 502 gateway error from nginx. Anyway, I read the error