Authentication error using API Gateway

Hello,

I have an AWS API that calls a Lambda function, both built by SAM. When I test it via the console, it is works. When I hit it via a browser, I get:

{"message":"Missing Authentication Token"}

I have tried to use the following URL:

https://XXXXxf6cil.execute-api.us-east-1.amazonaws.com/Prod/POST/[email protected]
https://XXXXxf6cil.execute-api.us-east-1.amazonaws.com/Prod/[email protected]
https://XXXXxf6cil.execute-api.us-east-1.amazonaws.com/[email protected]

The URL for my production stage is:

https://XXXXxf6cil.execute-api.us-east-1.amazonaws.com/Prod

When I go to perform a (successful) test of API via the console, the request is what I have below. So basically, I thought that I could add the request on to the end of the production stage URL and use that?

/films?[email protected]

​

I haven't configured any authentication so I suspect the problem is elsewhere? There resource policy on the lambda function looks like it has been generated and applied correctly by SAM. This (films/POST) is currently the only method for this API. If I look at the Method Request, Authorization is set to "None".

https://redd.it/rce80r
@r_devops

AWS vs. Azure for dockerized auto-scaling applications

Hi there,

Our client has proposed that we should look over switching from Azure to AWS, but I want to make sure that this is a good choice. The applications are Python based (Django) Applications that run in Docker, some are just web-servers, some are for handling CRON-tabs. Served via NGINX and uWSGI and CI taken care of by Github actions.

I have been working with Azure for about one month, and I have run into so many quirks and issues that I've almost lost my sense of sanity sitting with their solutions. It's half baked, documentation isn't very good and error logging is catastrophic at times. Short and sweet - I hate it wholeheartedly and want to throw it into the fires of hell every other day, so I'm inclined to go towards AWS - but wanted to check with you who maybe have experience from both/the particular experience of our stack in AWS who can tell how it works for you.

Looking forward reading your replies. Thanks.

https://redd.it/ra7ihh
@r_devops

Don't Just Track Your Machine Learning Experiments, Version Them - Distributed Versioning vs. Centralized Tracking for ML Experiments with DVC and Git

Machine learning experiments often get split between Git for code and experiment tracking tools for meta-information - because Git can't manage or compare all that experiment meta-information, but it is still better for code.

The following guide explains how to apply DVC for ML experiment versioning that combines experiment tracking and version control: **[Don't Just Track Your ML Experiments, Version Them](https://dvc.org/blog/ml-experiment-versioning)** - Instead of managing these separately, keep everything in one place and get the benefits of both, like:

* **Experiments as code:** Track meta-information in the repository and version it like code.
* **Versioned reproducibility:** Save and restore experiment state, and track changes to only execute what's new.
* **Distributed experiments:** Organize locally and choose what to share, reusing your existing repo setup.

Experiment versioning treats experiments as code. It saves all metrics, hyperparameters, and artifact information in text files that can be versioned by Git, which becomes a store for experiment meta-information. The article above shows how with DVC tool, you can push experiments just like Git branches, giving you flexibility to share experiment you choose.

https://redd.it/rcgaio
@r_devops

Best DevOps courses (between beginner and advanced)?

I heard good things about Stephane Maarek's DevOps course, but I'm afraid it's a little too specific for me, since it includes AWS only, and the IaC part is with AWS CloudFormation only. But if I can apply the things I learn from that when I'm creating Terraform scripts for Azure, then that's totally fine with me.
And I also already completed Stephane's AWS Developer course. Oh and one more thing, sometimes I felt like the Developer course was too focused on teaching exactly enough to pass the Certification exam, and it skipped explaining some things more in-depth.


But if anyone can suggest a really good, possibly more generic course on DevOps topic, I'd be very grateful. Also it's okay if it's several seperate courses on topics that are neccessary for a DevOps engineer. I have motivation but am having trouble deciding where to start and how..

https://redd.it/rchk6q
@r_devops

Jenkins pipeline for npm project for code deployment to sandbox

Hi can anyone plz help me out with jenkins pipeline for deployment of 2repos in bitbucket to sandbox environment using npm commands

https://redd.it/rciq8l
@r_devops

Git and GitHub Tutorial – Version Control for Beginners

If you're a beginner developer and want to learn about Git and GitHub then this article is for you.

This tutorial will help you understand what Git and version control are, the basic Git commands you need to know, how you can use its features to boost your work efficiency, and how to extend these features using GitHub.

Check it out: https://www.freecodecamp.org/news/git-and-github-for-beginners/

https://redd.it/rchs9r
@r_devops

Is Kubernetes Betamax?

The other day I was reading through some old Reddit posts about the basic question ‘should I use Kubernetes at all?’ and found this post from Reddit user u/trg0819 where they posit that Kubernetes matters because it is the industry standard.

>Betamax in theory did everything VHS did. But one made its way to be the industry standard, and people still using Betamax after that point were not well regarded.

His point is extremely valid: there’s value to using what everyone else is using, totally outside of its individual merits. In that way, Kubernetes is VHS all the way. Every major cloud platform supports Kubernetes, and 50–80% of containerized applications running in those platforms are using it (this article is old and the current rate is probably higher).

But I still have to ask, is Kubernetes Betamax?

I’ll admit that my analogy requires an understanding of an audio-visual format war from nearly 40 years ago, but bear with me: Sony’s Betamax was a superior video format at the time of its release. Betamax creates a denser image in pure analog, with more vertical lines in each frame. You can watch a comparison here.

So why is there a 0.0% chance that your grandparents’ basement is filled with Betamax tapes? There are a number of theories about price and availability, but one simple answer is that the most complex answer is not the best answer for everyone. The cost and complexity of Betamax players, combined with their relatively short play time, meant they lacked mass-market appeal.

How is this connected to Kubernetes? I don’t think anyone could argue that Kubernetes is a simple tool. Last year I saw people hailing a ‘simple troubleshooting guide’ that felt like… anything but. https://learnk8s.io/troubleshooting-deployments

And as we see a severe talent crunch, with teams unable to hire experienced Kubernetes experts, it seems reasonable to say that Kubernetes may just be too complex for mass-market adoption.

Even though most teams doing distributed workloads are using Kubernetes, it’s still true that most of the web requests that are handled are not handled by a distributed workload, so Kubernetes is the default tool for the job, the job is still not fully wide-spread.

Here’s one of the secrets of ‘failed’ formats like Betamax and DAT tape and others: They lived on for a very long time as tools for professionals.

The reason people aren’t using Kubernetes, why they’re sticking with tools that don’t involve distributed frameworks like serverless, is not because Kubernetes can’t do what they need, it’s because they’re intimidated by a tool that, as the original poster said, feels like overkill.

Okay, so in the real world at this point the analogy breaks down. I’ll mention here that tools and services like Mirantis are popular for a reason: a lot of teams are finding a huge expertise barrier to creating distributed workloads in production, and they need both tools and experts to them across that initial barrier.

https://redd.it/rcl7su
@r_devops

How common it is to fail builds due to security vulnerabilities?

We had a fairly mature DevSecOps practice in the previous company I worked for. We had static & dynamic application security testing, as well as container security and software composition analysis tools.

We broke build and deployment pipelines when high severity vulnerabilities were identified, but the inability to release hot fixes to the code impeded the development velocity.

I decided to develop an aging threshold mechanism that allows developers to exclude specific vulnerabilities in a text file, but the caveat was that the pipeline always checked if the vulnerability is aged over 2 weeks. If it was the case, no more exceptions were allowed to deploy.

On top of it, we had a policy to re-deploy the containers every week, so when a deployment failed, it notified the relevant teams that the deployment failed (we didn't have it in the build process though).

Which portions of these practices are adopted in your companies?

https://redd.it/rcl6cs
@r_devops

Anyone have experience with Sumo Logic for log aggregation and analytics?

We currently use Splunk however our renewal is coming up. With an ongoing cloud migration and for lack of a better term “digital transformation”, a few friends have suggested checking out Sumo Logic as a replacement. We are also evaluating Solar Winds and Data Dog. Any feedback on the platform would be helpful. Thanks!

https://redd.it/rcmpiv
@r_devops

Let's make faster GitLab CI/CD pipelines

In my article, I wrote about a bunch of tips to make your GitLab CI/CD pipelines very fast:

https://blog.nimbleways.com/let-s-make-faster-gitlab-ci-cd-pipelines/

Here is the code for everything in the article

https://gitlab.com/daoudi.mohammed/gitlab-faster-pipelines/-/tree/main

I made patches for all the commits. If you want to try it yourself:

git reset --hard 054bc48b
git apply patches/...

https://redd.it/rcnz5x
@r_devops

GoLang for DevOps in general, Lambda Functions/Automation in particular.

Hello folks, I have been using Python and NodeJS here and there to do different kind of stuff on Lambda functions such as inserting something into a DB or zipping and unzipping a file in S3 etc. I don’t find myself excelling at neither of them. I have been digging into GoLang for sometime now and I just love it! I want to excel at it and use it in my day to day life as a DevOps engineer. I’m also not very sure how to expose and teach myself more GoLang that it would help me in my rather specific purpose.

With my limited time I want to be as effective and time efficient as possible therefore kindly looking for your recommendations on how to proceed.

I’d appreciate if you skip a comparison between Python and Go as I want to follow what I’m excited about as its a great motivation to excel at something. Otherwise, I’m very open to your suggestions.

https://redd.it/rcp4pw
@r_devops

Host multiple websites on a single VPS with Docker

I plan to host many Node/React demo projects on a single VPS, single domain for all, each on a different subdomain. They will have very low traffic so performance and scaling aren't focus, only requirement is to be always online. Each project will have it's own Dockerfiles and docker-compose, some of them maybe CI/CD pipelines.

I identified 2 possible paths:

1. use Dokku, then CI isn't possible and project's Docker configs wont be used, but the upside is not spending any time on configuration, functional application source is the only requirement
2. second option is to use Nginx container in front, and then separate containers for applications, upside is the flexibility for custom configuration and optional CI, downside is additional time and effort for each app setup

How would you approach this, and have you done something similar already?

https://redd.it/rcq3bo
@r_devops

Anyone know of a service to proxy third party API access? Mainly, something that can centralize third party API token access into a "service" that I could maintain global authentication for?

The use case is basically to rehost access to third party API services (Maybe, 20-30+ public API services that require an API token, and are hard to rotate when leakage occurs). Developers use these API services regularly in production, development environments, and even locally. And controlling access to these API keys is cumbersome at best, and insecure at worst.

I'd like to find something that can either act as a transparent proxy with some sort of basic authentication which modifies headers and adds API tokens dynamically, so that various developers and applications never actually have access to these global private API tokens.

Or, something that rehosts the API services to a new URL on the self hosted (or SAAS) service itself. IE - Instead of doing an API request with a token against example.com/api/test, you'd do it to rehostmyapi.com/example.com/test or something with a basic authentication parameter that I can control, and the API token itself doesn't need to change.

Is this something that exists? I can certainly develop a project like this to fit my needs, but I'd rather see if I can leverage something (even if paid) to reduce workload and maintenance.

I'd like to find something that fits this specific criteria - I understand that I can likely rewrite my applications to consume data from services into a centralized format, but a project like this does not integrate easily with current development processes in the near-ish future, so a stop gap solution along these lines would go a very long way in modernizing my infrastructure and application development patterns.

https://redd.it/rcqee9
@r_devops

How and when do you split your Terraform project into multiple small projects?

In my current project it takes several minutes to run a terraform plan, which is very limiting for development. Usually I'll make my changes manually in the console and then recreate it in terraform after I know all the details.

Also having multiple DevOps engineers make changes in parallel is always frustrating. If I apply the changes I'm testing on my branch, it prevents my coworker from testing any changes and vice-versa.

I guess both of these problems could be solved by breaking the project into smaller projects, but that seems fragile because so many of the resources depend and reference each other, especially the networking components.

What has been your experience with this?

https://redd.it/rctzpk
@r_devops

Submit a GET request to a REST API endpoint - Interact with web services - Ansible module uri

How to retrieve a JSON list of users via a GET request to a REST API web service HTTPS endpoint from a remote Linux host in a few lines of Ansible code.

[https://youtu.be/U92t0h9Cw8Q](https://youtu.be/U92t0h9Cw8Q)

---
# API https://reqres.in/api/users?page=2
- name: uri module demo
hosts: all
become: false
vars:
server: "https://reqres.in"
endpoint: "/api/users?page=2"
tasks:
- name: list users
ansible.builtin.uri:
url: "{{ server }}{{ endpoint }}"
method: GET
status_code: 200
timeout: 30
register: result

- name: debug
ansible.builtin.debug:
var: result.json.data


\#ansible #webservice #rest #api #uri #get

https://redd.it/rclrzd
@r_devops

how to setup CICD pipeline for containerize nodejs multi service project. GitLab

Hi, I have two different repos. One for backend and one for the frontend. They are both containerize. Currently I am using docker-compose to run the application. However, I am having issues creating CICD gitlab pipeline and how to implement it using local runners. Requesting guidance on resources or basic idea on how I should go about implementing it. Right now I am just building it but later on the test cases will also be run in the pipeline. Thanks in advance :-)

https://redd.it/rcf65j
@r_devops

How is AI Accelerating DevOps in 2021

DevOps helps deliver and manufacture products faster than the regular process. DevOps is the unification of development and operations, which helps businesses by optimization of their services. However, DevOps can become faster and more efficient with the help of AI, and some even claim that in the future, DevOps will be fully driven by AI.
readmore...

https://redd.it/ranw6p
@r_devops

GCP Certificate lists

Hello guys, I was using AWS, now I am starting GCP. I have a question about the GCP certificate manager. On AWS has ACM, where I uploaded my certificates and listed them. But I cannot find an alternative this on GCP there is a Certificate Manager, which if am I right I can create my own CM and then can request for certs. But where I can find my all listed certificates? For example, if I want to attach some certificate to the load balancer how can attach it? I need a direct request or I can pick from somewhere a certificate for attaching? Thanks!!!

https://redd.it/ra5vuu
@r_devops

I Have no idea what I'm doing and I start a New role in a Month.

I quit my old job because I had been coasting for two years at that point, performance reviews finally caught up with me, and I could see the writing on the wall. Because the labor market is so tight and my previous company carries some prestige, I miraculously got a pretty good offer in some kind of cloud ops role.

So I applied to DevOps positions because the last time I remember being engaged at my job I was in a similar role. I'm beginning to think that had more to do with friendly coworkers and being a bright-eyed newgrad not yet burned out. For reference, I transitioned into webdev thereafter and struggled ever since. I pretty much gave up and no one noticed for a while. So even though I should be a seasoned webdev, I basically don't know any frameworks (theoretically I should know Springboot and Angular), still get intimidated by even the simplest directory structures, and I even struggle with basic code management stuff like using maven/IDEs/git. Basically, I suck with every tool I was supposed to be using.

I suck at everything that isn't telegraphed leetcode. I have a formal CS education so I'm fairly decent at anything that's just implementing raw data structure/algorithms without a real use case attached. If you wanted me to code up an A* algorithm or implement a B-tree, I probably could. You might think surely a University education would mean I know my way around a Linux shell... and like I know how to navigate a directory to submit an assignment. I have an AWS ASA certification from way back when. I fiddled with Terraform way back when. I technically work with splunk though I couldn't write up a functioning query if prompted. I've written a Jenkins pipeline like one time and always with a lot of help from internal tools.

I'm reading through the Google SRE book right now and there's so much shit they just glance over like it's fundamental 101 stuff. And almost every time I'm like "Fuck me, I don't know even know what OAuth2 is besides what my coworker does in Postman before the other api calls". I didn't learn security in school and it doesn't come naturally to me. Honestly, my old company actually had fairly good security protocols but to me it all just looked like pointless code bloat and impenetrable bureaucratic procedures. Now I'm supposed to be one of the guys charged with that stuff. Oops.

https://redd.it/r9zkfv
@r_devops

DuploCloud 10x faster than Terraform?

Saw this company at AWS Re:Invent this year claiming 10x faster than Terraform, with I guess instant compliance framework mapping for SOC2. We're looking to get our SOC2 and PCI soon. Does anyone use their software? https://duplocloud.com/

https://redd.it/rd0xf8
@r_devops

How close are your pre-prod and prod setups?

When we talk about pre-production setup for testing, the expectation is to have a platform as close as production to replicate prod behaviour. But I think it is really difficult to maintain that integrity in a large organization. With over 100+ microservices, you have to almost double the cost.

Also if there are deployments, you need to check if test build is deployed or release can be deployed.

Wanted to understand how do you guys maintain your pre-prod environments? And how close is it your production platform?

https://redd.it/rd1lxh
@r_devops