Digger - get instant URLs and Terraform for your microservices on AWS

Hey r/devops! We've been working on this for a while, built an alpha and looking for feedback.


Developers today have great tools to quickly launch small projects without thinking of infrastructure (Firebase, Vercel, Heroku). But these tools don't work for teams. Big tech companies that can afford dedicated platform teams tend to build self-service tools for developers on top of AWS / Azure / GCP to launch new services and manage environments. But smaller teams who can't afford it are out of luck. If they have DevOps expertise in the team then they'll write a lot of repetitive Terraform, and if they don't they'll often struggle for weeks learning all the AWS concepts and make lots of mistakes.


We thought this is wrong, and built Digger


Digger manages your cloud account, allows to create apps and microservices from templates (can be custom), generates and runs Terraform, and manages environments. So developers get modern Vercel-like experience while DevOps engineers still retain full control. Starting on AWS with Digger is just as simple as on Heroku, but cheaper and you get a future-proof stack with DevOps best practices.


What do you think?

https://redd.it/kvd989
@r_devops

Pyroscope - continuous profiling tool to help debug performance issues. Would love some feedback!

Hi r/devops

At my last job I had to deal a lot with performance issues on the backend and I found profiling tools to be very helpful in figuring out where the bottlenecks occur. But the problem is that it’s often pretty hard to replicate exact situations that happen in production environment. So I figured why not profile my apps 24/7 in production — that’s how Pyroscope was born.

It’s open source and it currently works with Go, Python and Ruby apps. Here’s the link: https://github.com/pyroscope-io/pyroscope

Would love to get any feedback or hear from you all on how you do performance analysis for your apps.

https://redd.it/kvdqgd
@r_devops

How Canada's Exposure Notification App (Covid Alert) runs on AWS infrastructure.

AWS wrote a blog post regarding the infrastructure that I'm working on.

https://aws.amazon.com/blogs/publicsector/keeping-canadians-safe-protecting-privacy-covid-alert-app/

You can check out the code here:
https://github.com/cds-snc/covid-alert-server

And the terraform for our staging environment here:

https://github.com/cds-snc/covid-alert-server-staging-terraform

https://redd.it/kvfifq
@r_devops

Logging Platform

Everything generates logs, but not everything needs to be retained.

Is there a logging platform/option that will allow logs to stream, retain for a nominal amount of time, let us say 1 hour. If there are issues (insert detection pattern here - security, 5xx, other defined patterns) retain the logs of that stream for say 10 minutes before the event and 10 minutes after and then apply the log retention rule above.

Essentially you'd not have the entire log, just the snippets from when there was a detected issue and one would not have to worry about the retention based pricing?

https://redd.it/kveioo
@r_devops

Pushing More, Smaller Rocks as a Team

OK, I've read about Amazon releasing deployments every day and I'd love to take that concept and get more projects in concurrent pipelines, with staggered releases so that we're moving faster, breaking very little and recovering quickly too.

Working on building out a workflow from development through testing and customer feedback.

If you've been able to deploy more frequently than weekly what are some bottlenecks have you found and best practices to remedy them?

We're now acquainted with working remotely, should we stay remote? Have you found slow-downs from it?

Have any lesser-known tools that have helped you out to achieve releasing 2-3+ times per week?

https://redd.it/kvgg2v
@r_devops

Google Cloud Run, Terraform Cloud and GitHub actions for continuous delivery of a Ruby on Rails application

In this video, we are going to learn how to deploy an application in Cloud Run, how to use GitHub actions, Terraform Cloud, and how to integrate everything together in a real example: https://www.youtube.com/watch?v=JxBSXbmf4F0&feature=youtu.be

https://redd.it/kuy6bt
@r_devops

dEVOPS when company doesnt develop software from scratch?

Hi guys, just wondering if you guys have heard of, or have experience seeing devops principles and its automation practices being implemented outside companies that produce software. For example I work for a medium sized grocer that sells through online and traditional brick & mortar channels. We changed focus to Cloud SAS offerings and building integrations mainly and now are trying to implement a devops culture/practice. Have any of you heard of or experienced successes/challenges in this kind of scenario? Maybe not directly mapping to mine but similar context at least?

https://redd.it/kuxxek
@r_devops

Getting error while allowing accounts and roles in Terraform for GCP



Hi All,

I am trying to allocate the roles to the user in the Terraform file in a GCP project, but I am getting the below error . Please let me know if you have a better way for doing this.

Error: Request "Create IAM Members roles/compute.networkAdmin user:[email protected] for \\"project \\\\\\"vibrant-mantis-296207\\\\\\"\\"" returned error: Batch request and retried single request "Create IAM Members roles/compute.networkAdmin user:[email protected] for \\"project \\\\\\"vibrant-mantis-296207\\\\\\"\\"" both failed. Final error: Error applying IAM policy for project "vibrant-mantis-296207": Error setting IAM policy for project "vibrant-mantis-296207": googleapi: Error 403: Policy update access denied., forbidden

I used the below piece of code :

module "projects_iam_bindings" {
source  = "terraform-google-modules/iam/google//modules/projects_iam"
version = "\~> 6.4"
projects = ["vibrant-mantis-296207"\]
bindings = {
"roles/storage.admin" = [
"user:[email protected]",
    \]
"roles/compute.networkAdmin" = [
"user:[email protected]",
    \]

https://redd.it/kuuw7y
@r_devops

Sock Shop alternatives ?

Hello everyone

I am looking for a sample cloud native applications (especially Spring boot - React) to practice my devops skills on.

I tried Sock Shop by weave but I found it a little bit overwhelming. Do you know any well coded apps to host on my machine and practice on ?

It does not have to contain infrastructure files, just the application is enough for me.

https://redd.it/kvno1w
@r_devops

AWS Lightsail now supports containers

Can something as simple as AWS Lightsail with containers fit anyone but teams working on small projects?

\>>> https://youtu.be/CWXrW2rgego

https://redd.it/kvsm9b
@r_devops

Does it make sense to create Test Plans for WIndows Updates and infrastructure testing?

I hope this doesn't sound ridiculous, I started as a SysAdmin and I'm trying to integrate more into our Dev team. We use Azure Dev Ops and I'm trying to see if it's worth while to have a Test Plan (I'm guessing this would be the best option for my idea) to check the following items:

If Windows updates were applied successfully
If Machines are reachable after reboot.
If services restarted correctly post reboot
If PowerShell Pester Tests passed
If Product related Tests pass
If Maintenance Window/Update Items are complete

While including time of items, and reaching back to Work Items.

It's a bit weird trying to add our Ops team into Azure DevOps, as our code and processes normally doesn't have a build/release, the closest thing to bugs would be errors in server configurations or server down issues, and when creating a Test Plan it's not like I can really run some of these tests. From what I see I could use this functionality more as a check list than an actual Test Case.

Am I wrong? Or does anyone have any input they'd like to share?

Thanks for your advice!

https://redd.it/kvu6rs
@r_devops

tracking infrastructure drift

Hi folks,being big users of terraform in our team, we've often been faced with infrastructure drift (when your infrastructure code doesn't match your actual resources). So we built this free and open source CLI with the goal of helping manage all kinds of drifts, and we'd love some feedback :) Here's why we built this tool :https://driftctl.com/2020/12/22/announcing-driftctl

https://redd.it/kvtqtl
@r_devops

What happened to Parler is a reminder to be cloud agnostic.

I'm not a user of Parler or follow politics, like many we have all our eggs in the AWS basket and what happened to Parler is a reminder that AWS can pull the plug on your account for whatever reason they seem fit, they hold the keys.

https://redd.it/kvtluf
@r_devops

Intrusion Detection/Prevention within Kubernetes?

Does any prior work exist to monitor the connections between pods, namespaces and or the Internet to characterize normal behavior and alert on abnormalities? For example, I'd like to know if pods that traditionally only talk to microservice X start suddenly chatting up microservice Y so I can check if that's expected new behavior. Similarly, if a pod suddenly decided it aught to start scanning ports I'd like this to be immediately seen and reflected.

https://redd.it/kv9gfp
@r_devops

Question

Hello guys I hope everyone is doing well. I have a question if any one can answer that would be great so I can’t catch up the concept of build code what does the build do for example if you are using Jenkins for CI/CD when they say build test etc. I don’t understand the term build does it gather the dependencies for the code to work on a environment or does it mean something else thanks in advance.

https://redd.it/kvd3l2
@r_devops

Dependency Tracking - Is there an wasy way?

Hi everyone. I hope this is the correct sub where to post.

I'm a junior developer, working on a project where the architect just quit, and to be honest, I have no clue whom to ask about this, so, I'm very sorry if I sound a bit naive.

I tried to google and skim through reddit posts, but to be honest, this time I don't even know what to look for.

Anyway, our project looks like a "revamped" monolith, that have been split through different maven projects, that get deployed on multiple wildfly instances.

Our problem, is that the number of modules are growing, and we're slowly losing track of all the dependencies between them.

Since I have no clue about who's supposed to take care of this aspect of the project (Management? Developers?) neither what are the best practices or tools available to avoid this kind of stuff, could you give me any hints?

Just for the sake of example, let's say we have the A1.war module.

B.war, C.war, and D.war, all uses A.war as dependency.So, everytime something changes on A's interfaces, I have to rebuild B, C and D.

It happened before, that we forgot to redeploy a module or two, and that's kinda sad. So, i'm looking for something that could...I don't really know, scan the POM and draw a dependency graph? Or some book that explain the concept of "how should thos kind of problems we avoided"?

Anything really. Any info or pointing in any direction is more than welcome.

Thanks everyone!

https://redd.it/kw0167
@r_devops

Migrating from Docker Swarm, looking for real rootless replacement

My first priority is to reduce the security holes that docker has with it default root configuration. So far its 2021 and Docker itself does not work with rootless mode under swarm that would need to scale (no overlay network for rootless daemon).

https://redd.it/kw1604
@r_devops

Solved a problem I was having with file sharing sites..

I had an annoying problem with a company project where everyone in the team and different departments was using a different file sharing site. Box, Dropbox, Google Drive, that kind of thing. Makes it really hard to have security policies for data across multiple platforms like that... especially with file sharing.

Thought I'd share the solution I found -- it securely federates and combines all the different file sharing sites together into _one_ manageable metasite without actually copying the data. Even has a google/edge extension so it's convenient.

Its called https://www.dokkio.com .. free for now it seems as they are in beta, and looks like they can handle individual and up to Enterprise.

https://redd.it/kv99j8
@r_devops

Icinga2 vs Instana for monitoring

What are your thoughts?
Instana is pretty great since it comes with easy out of the box everything.
Looking into ICinga2 I like how much power I have with Python Scripts but I been trying to set up the same ecosystem that Instana provides and I'm finding that to be much harder.

https://redd.it/kv7ujy
@r_devops

Kaholo feedback CI/CD

Has anyone here used kaholo for CI/CD?

Any feedback welcome!

https://redd.it/kv6xhr
@r_devops

Modern CI/CD pipeline for front end projects

Here is what we do in my company:

precommit and prepush \#git hooks are used to catch issues before they are pushed upstream.

`precommit` runs only on staged files (takes few seconds)
prepush runs eslint typescript and unit tests (takes up to 20 seconds)

Every time a commit is pushed:

1 ) We build a #docker image & bundle cypress and other development dependencies. This allows us to run all subsequent tasks using the same Docker image.

It is fast. Takes 2-4 minutes. 🏎

2 ) We run 5 tasks concurrently to validate our build.

ESLint
TypeScript
jest Unit tests
cypress Integration tests
Fetch, validate & compile GraphQL schema

2 ) For every commit, we deploy a review app.

Review app:

Allows anyone to preview what is being developed.
Allows anyone to preview our storybook.
Allows to leave visual reviews (WIP)

3 ) Before changes can be merged to the main branch, we use GitLab to mandate at least 1 review from the team.

In addition, we use GitLab review system to advise who is the best person to review the code based on which files have changed.

5 ) When changes are merged to the main branch, we automatically deploy to production.

We use argocd to implement gitops. This means that we have a detail log of everything that has been deployed, and in case of a critical error, reverting is as simple as "git revert" Receipt

6 ) Finally, we push changes regularly to the main branch. Small incremental updates, dozens of times a day.

This means that if things break, they are typically small things and easy to revert / patch.

We use feature flags to hide any WIP features. 🏳

Originally posted:

https://twitter.com/kuizinas/status/1349177926105792514

https://redd.it/kw6pn3
@r_devops