Development for Infrastructure in 2021

Hey All,

One of the trends I saw in 2020 was that infrastructure pros and sysadmins need to code. Sure, we’ve seen the “just knowing how to code” part for years now, but something changed drastically.

The need to “think” like a developer and understand development concepts.

I started out my career as a Sysadmin and moved into the development space later on. I thought it was really interesting and I sort of saw the “writing on the wall”.

So, how can one break into the whole “think like a developer” thing?

1. First is theory. You’re going to hear phrases like “immutable vs mutable” and “idempotent”. If you’re new to development, this may sound like gibberish.
2. The second is source control. If you’re already writing code or plan on deploying application code, you’ll need to understand source control.
3. The third is code editors and IDEs. Believe it or not, people are still writing code in Notepad. A good code editor and change the game for you.
4. The fourth is understanding how to write code like a developer. You may write small PowerShell or bash scripts, but how about taking it a step further with things like unit testing?
5. Step five is CICD. In the beginning, CICD was targeted towards application code. Now, it's almost a requirement for any organization to use CICD to deploy infrastructure code.

Last month I posted a thread here that got a lot of love. It was about what I expect in the future and a series to help people get there.

Going From Infrastructure to Developer Is A Reality : devops (reddit.com)

I just finished up the series and it's called "Development for Infrastructure". If you're interested, definitely feel free to check out the playlist!

https://www.youtube.com/watch?v=u-0T-JN0GZc&list=PL8iDDHqmj1oWbbqlVwdZT9dElTyd9JRb4

https://redd.it/kv4qrw
@r_devops

Collect Custom Metrics in AKS

Custom metrics are a large part of monitoring software. I wrote a blog post on how you can define and query custom metrics in the Azure Kubernetes Service (AKS). Read more here!

It utilizes Prometheus metrics under the covers, so creating a new one or implementation an existing Prometheus exporter is necessary.

https://redd.it/kv5fh4
@r_devops

Wide Scale Deployment with/without internet

So we have approximately 800 or so “kiosk” devices in the field (Windows) - some have internet connectivity, some do not; are only connected via VPN to our server infrastructure.

We currently push updates to this software via ManageEngine Desktop Central, which does work, and works quite well, however we want to go the next step and automate from a pipeline to these deployments.

From the DesktopCentral API, there doesn’t seem to be any way to register a new package programmatically, and/or schedule deployments.

So I guess the question is 2 fold: does anyone know if there are secondary api’s or ways to control DesktopCentral programmatically, or alternatively, can anyone recommend a good deployment tool that can handle internet and non internet connected endpoints, programmatic control, and doesn’t cost stupid amounts of money.

Thanks!

https://redd.it/kv2a88
@r_devops

Using GPT-3 for plain language incident root cause from logs

Disclosure: I work for Zebrium.

We are seeing some pretty cool results using OpenAI GPT-3 with a summary of incident log events (generated by ML), to produce a plain language incident root cause description. Read the blog here.

https://redd.it/kva8a3
@r_devops

Any team leaders or managers out there?

If you are one, can you confirm my suspicions?

I've noticed a theme the more I chat with people about DevOps, cloud-native, K8s etc. It seems like people issues are tougher than technical issues.

They accept that the tech will constantly change and make their life tough. But the harder part seems to be getting the team aware of the change and to drive movement in affected activities.

Essentially, making sure all bases are covered seems to be an increasingly difficult moving target.

Is there any substance to this?

https://redd.it/kuz43q
@r_devops

Digger - get instant URLs and Terraform for your microservices on AWS

Hey r/devops! We've been working on this for a while, built an alpha and looking for feedback.


Developers today have great tools to quickly launch small projects without thinking of infrastructure (Firebase, Vercel, Heroku). But these tools don't work for teams. Big tech companies that can afford dedicated platform teams tend to build self-service tools for developers on top of AWS / Azure / GCP to launch new services and manage environments. But smaller teams who can't afford it are out of luck. If they have DevOps expertise in the team then they'll write a lot of repetitive Terraform, and if they don't they'll often struggle for weeks learning all the AWS concepts and make lots of mistakes.


We thought this is wrong, and built Digger


Digger manages your cloud account, allows to create apps and microservices from templates (can be custom), generates and runs Terraform, and manages environments. So developers get modern Vercel-like experience while DevOps engineers still retain full control. Starting on AWS with Digger is just as simple as on Heroku, but cheaper and you get a future-proof stack with DevOps best practices.


What do you think?

https://redd.it/kvd989
@r_devops

Pyroscope - continuous profiling tool to help debug performance issues. Would love some feedback!

Hi r/devops

At my last job I had to deal a lot with performance issues on the backend and I found profiling tools to be very helpful in figuring out where the bottlenecks occur. But the problem is that it’s often pretty hard to replicate exact situations that happen in production environment. So I figured why not profile my apps 24/7 in production — that’s how Pyroscope was born.

It’s open source and it currently works with Go, Python and Ruby apps. Here’s the link: https://github.com/pyroscope-io/pyroscope

Would love to get any feedback or hear from you all on how you do performance analysis for your apps.

https://redd.it/kvdqgd
@r_devops

How Canada's Exposure Notification App (Covid Alert) runs on AWS infrastructure.

AWS wrote a blog post regarding the infrastructure that I'm working on.

https://aws.amazon.com/blogs/publicsector/keeping-canadians-safe-protecting-privacy-covid-alert-app/

You can check out the code here:
https://github.com/cds-snc/covid-alert-server

And the terraform for our staging environment here:

https://github.com/cds-snc/covid-alert-server-staging-terraform

https://redd.it/kvfifq
@r_devops

Logging Platform

Everything generates logs, but not everything needs to be retained.

Is there a logging platform/option that will allow logs to stream, retain for a nominal amount of time, let us say 1 hour. If there are issues (insert detection pattern here - security, 5xx, other defined patterns) retain the logs of that stream for say 10 minutes before the event and 10 minutes after and then apply the log retention rule above.

Essentially you'd not have the entire log, just the snippets from when there was a detected issue and one would not have to worry about the retention based pricing?

https://redd.it/kveioo
@r_devops

Pushing More, Smaller Rocks as a Team

OK, I've read about Amazon releasing deployments every day and I'd love to take that concept and get more projects in concurrent pipelines, with staggered releases so that we're moving faster, breaking very little and recovering quickly too.

Working on building out a workflow from development through testing and customer feedback.

If you've been able to deploy more frequently than weekly what are some bottlenecks have you found and best practices to remedy them?

We're now acquainted with working remotely, should we stay remote? Have you found slow-downs from it?

Have any lesser-known tools that have helped you out to achieve releasing 2-3+ times per week?

https://redd.it/kvgg2v
@r_devops

Google Cloud Run, Terraform Cloud and GitHub actions for continuous delivery of a Ruby on Rails application

In this video, we are going to learn how to deploy an application in Cloud Run, how to use GitHub actions, Terraform Cloud, and how to integrate everything together in a real example: https://www.youtube.com/watch?v=JxBSXbmf4F0&feature=youtu.be

https://redd.it/kuy6bt
@r_devops

dEVOPS when company doesnt develop software from scratch?

Hi guys, just wondering if you guys have heard of, or have experience seeing devops principles and its automation practices being implemented outside companies that produce software. For example I work for a medium sized grocer that sells through online and traditional brick & mortar channels. We changed focus to Cloud SAS offerings and building integrations mainly and now are trying to implement a devops culture/practice. Have any of you heard of or experienced successes/challenges in this kind of scenario? Maybe not directly mapping to mine but similar context at least?

https://redd.it/kuxxek
@r_devops

Getting error while allowing accounts and roles in Terraform for GCP



Hi All,

I am trying to allocate the roles to the user in the Terraform file in a GCP project, but I am getting the below error . Please let me know if you have a better way for doing this.

Error: Request "Create IAM Members roles/compute.networkAdmin user:[email protected] for \\"project \\\\\\"vibrant-mantis-296207\\\\\\"\\"" returned error: Batch request and retried single request "Create IAM Members roles/compute.networkAdmin user:[email protected] for \\"project \\\\\\"vibrant-mantis-296207\\\\\\"\\"" both failed. Final error: Error applying IAM policy for project "vibrant-mantis-296207": Error setting IAM policy for project "vibrant-mantis-296207": googleapi: Error 403: Policy update access denied., forbidden

I used the below piece of code :

module "projects_iam_bindings" {
source  = "terraform-google-modules/iam/google//modules/projects_iam"
version = "\~> 6.4"
projects = ["vibrant-mantis-296207"\]
bindings = {
"roles/storage.admin" = [
"user:[email protected]",
    \]
"roles/compute.networkAdmin" = [
"user:[email protected]",
    \]

https://redd.it/kuuw7y
@r_devops

Sock Shop alternatives ?

Hello everyone

I am looking for a sample cloud native applications (especially Spring boot - React) to practice my devops skills on.

I tried Sock Shop by weave but I found it a little bit overwhelming. Do you know any well coded apps to host on my machine and practice on ?

It does not have to contain infrastructure files, just the application is enough for me.

https://redd.it/kvno1w
@r_devops

AWS Lightsail now supports containers

Can something as simple as AWS Lightsail with containers fit anyone but teams working on small projects?

\>>> https://youtu.be/CWXrW2rgego

https://redd.it/kvsm9b
@r_devops

Does it make sense to create Test Plans for WIndows Updates and infrastructure testing?

I hope this doesn't sound ridiculous, I started as a SysAdmin and I'm trying to integrate more into our Dev team. We use Azure Dev Ops and I'm trying to see if it's worth while to have a Test Plan (I'm guessing this would be the best option for my idea) to check the following items:

If Windows updates were applied successfully
If Machines are reachable after reboot.
If services restarted correctly post reboot
If PowerShell Pester Tests passed
If Product related Tests pass
If Maintenance Window/Update Items are complete

While including time of items, and reaching back to Work Items.

It's a bit weird trying to add our Ops team into Azure DevOps, as our code and processes normally doesn't have a build/release, the closest thing to bugs would be errors in server configurations or server down issues, and when creating a Test Plan it's not like I can really run some of these tests. From what I see I could use this functionality more as a check list than an actual Test Case.

Am I wrong? Or does anyone have any input they'd like to share?

Thanks for your advice!

https://redd.it/kvu6rs
@r_devops

tracking infrastructure drift

Hi folks,being big users of terraform in our team, we've often been faced with infrastructure drift (when your infrastructure code doesn't match your actual resources). So we built this free and open source CLI with the goal of helping manage all kinds of drifts, and we'd love some feedback :) Here's why we built this tool :https://driftctl.com/2020/12/22/announcing-driftctl

https://redd.it/kvtqtl
@r_devops

What happened to Parler is a reminder to be cloud agnostic.

I'm not a user of Parler or follow politics, like many we have all our eggs in the AWS basket and what happened to Parler is a reminder that AWS can pull the plug on your account for whatever reason they seem fit, they hold the keys.

https://redd.it/kvtluf
@r_devops

Intrusion Detection/Prevention within Kubernetes?

Does any prior work exist to monitor the connections between pods, namespaces and or the Internet to characterize normal behavior and alert on abnormalities? For example, I'd like to know if pods that traditionally only talk to microservice X start suddenly chatting up microservice Y so I can check if that's expected new behavior. Similarly, if a pod suddenly decided it aught to start scanning ports I'd like this to be immediately seen and reflected.

https://redd.it/kv9gfp
@r_devops

Question

Hello guys I hope everyone is doing well. I have a question if any one can answer that would be great so I can’t catch up the concept of build code what does the build do for example if you are using Jenkins for CI/CD when they say build test etc. I don’t understand the term build does it gather the dependencies for the code to work on a environment or does it mean something else thanks in advance.

https://redd.it/kvd3l2
@r_devops

Dependency Tracking - Is there an wasy way?

Hi everyone. I hope this is the correct sub where to post.

I'm a junior developer, working on a project where the architect just quit, and to be honest, I have no clue whom to ask about this, so, I'm very sorry if I sound a bit naive.

I tried to google and skim through reddit posts, but to be honest, this time I don't even know what to look for.

Anyway, our project looks like a "revamped" monolith, that have been split through different maven projects, that get deployed on multiple wildfly instances.

Our problem, is that the number of modules are growing, and we're slowly losing track of all the dependencies between them.

Since I have no clue about who's supposed to take care of this aspect of the project (Management? Developers?) neither what are the best practices or tools available to avoid this kind of stuff, could you give me any hints?

Just for the sake of example, let's say we have the A1.war module.

B.war, C.war, and D.war, all uses A.war as dependency.So, everytime something changes on A's interfaces, I have to rebuild B, C and D.

It happened before, that we forgot to redeploy a module or two, and that's kinda sad. So, i'm looking for something that could...I don't really know, scan the POM and draw a dependency graph? Or some book that explain the concept of "how should thos kind of problems we avoided"?

Anything really. Any info or pointing in any direction is more than welcome.

Thanks everyone!

https://redd.it/kw0167
@r_devops