SFTP -> MySQL ETL Service

I have a task that requires grabbing customer csv files from a SFTP server, transforming it, and loading it into a MySQL table.

Looking for suggestions of services out there. Something like stitcherdata but I need to output into a MySQL table.

Not looking to custom code this, we currently do this but we are to the point where we'd like to offload this to a service where it is their core competency.

https://redd.it/ldhhuu
@r_devops

Beginner Question, Are there any useful tutorials on deploying simple web apps (that use both client and server) hosted on the Google app engine

I am completely new to the creation of a web apps, and only have a limited amount of experience making small programs in python and java on my local drive.

The webpage needs to pull in stock data from APIs, based on the users selection (keyed into a text box) perform a computation on it using a python script, and display the output of the computation on the webpage.

I am aware that I need client side and server side scripts, from what I have learned so far I need Javascript on the front end and Python on the back end, but thats as far as my knowledge goes, it sounds basic but I'm suffering from an information overload and wondering if someone could point towards some resources on make a web app that can be deployed on GAE for the above criteria?

Its small, its only going to be used by 1 or 2 people at a time, so the simplest solution with least overhead would be the best!


Again really sorry for the beginner question.

https://redd.it/ldihea
@r_devops

Managing health checks at scale

We used health checks to monitor cron jobs, but it didn't scale, as we had to manually register each job. So I adapted a script from the healthchecks.io maintainer to automatically register the job, and it's been working great.

The article: https://dev.to/15five/managing-health-checks-at-scale-4731

The script: https://gist.github.com/caleb15/1a817ef5e58e8a8caf65190cff33806e#file-healthcheck-sh

Enjoy!

https://redd.it/lddc1u
@r_devops

anyone here have experience with slalom build or simply being a devops engineer/consultant at a consulting firm? Trying to understand the pros and cons

hoping for some insight into slalom build, but otherwise just doing devops in a consulting firm, whether as an engineer or consultant

https://redd.it/ldpqnc
@r_devops

Kubernetes cluster diagram

Hi, I'm a system engineer junior, and I'm on my first big project with cloud and k8s. Nowadays I practice those on my local computer, or in the free tier.

I Haven't seen that big clusters, as our production cluster with a lot of namespaces and \~50 pods and services in each other namespace. I want to draw a diagram explained that cluster. I have a tool: draw.io.

​

My question is; Are there any tips and tricks or good practices to draw a huge cluster like this? Do I have to draw all different pods and services and deployments? Can anyone show me something like this?

https://redd.it/ld83z1
@r_devops

Flow to send mail when workitem is assigned

Hello i am trying to create a flow which will send a mail if workitem is assigned to any users. The mail will be sent to the one who created the work item or user which mail about the issue

https://redd.it/ld481u
@r_devops

How to create alerts in ELK

Hi there,

I am aware of x-pack edition of elastic stack comes with the alerting option ? Does the open source edition of the stack ELK provides the option for alerts ? If yes, how we configure alerts in the ELK stack ?

Thanks

https://redd.it/ld3lad
@r_devops

Alerts in ELK Stack

Hi there,

I am aware of x-pack edition of elastic stack comes with the alerting option ? Does the open source edition of the stack ELK provides the option for alerts ? If yes, how we configure alerts in the ELK stack ?

Thanks

https://redd.it/ld35gh
@r_devops

IaC Snippets - Your input would be amazing

We are beginning to provide IaC snippets for common resources like Docker Containers, S3 Buckets, and DynamoDB Tables in Commandeer.

https://docs.getcommandeer.com/blog/latest/introducing-iac-snippets-for-ansible-docker-compose-serverless-framework-and-terraform/

I'd appreciate any input you might have on any ways we can make this beneficial for your day to day work.

https://redd.it/ldydke
@r_devops

What are your indicators of a system in distress?

When looking at a system, what do you use in consideration to determine if it is in distress? Is it the heap usage? CPU usage?

https://redd.it/le41m4
@r_devops

Free alternative to Dynatrace Pure Path

Hi,

We use Dynatrace in our project for APM needs and i find the pure path feature really helpful to analyze API or service performance. I was wondering is there a free alternative to check for pure path which i can use to analyze my side projects. I can use Prometheus for basic application monitoring n alerting or micrometer.io to check for my application methods , but is there any open source product where i can view end to end flow n time taken for an api call

https://redd.it/le4xji
@r_devops

How can Stackoverflow make you a better developer

We all use SO on a daily basis. I’ve found that participating by asking and answering questions to be very effective for my own coding skills. I wrote a short article on why I think you should try it and improve your programming skills.

https://devopsian.net/posts/so-make-you-better-dev/

Would be glad to hear your opinions.

https://redd.it/le1dwz
@r_devops

How to connect front end to back end?

I want to do a project wherein I am using ML algorithm and train data and get a prediction. I want to take input from my html web page and give output on it. Possibly also a visualization. How to make this happen?

https://redd.it/le0tq3
@r_devops

Gathering interest in podcast or YT channel idea

My friend and I work on two different teams on the same software project (he's a Scrum Master, I'm a Dev Lead) and we've had some success at getting our organization to move toward adopting DevOps and Lean principles for software development.

Our organization is an enterprise IT company with a mix of projects and technologies that were acquired or developed in house. Our particular project has a spotlight on it, and we've been successful over the last year in helping our management become data aware.

We've started to work with other parts of our company to move them in that direction as well, and have learned a lot doing so.

We would like to disseminate our knowledge and thought that some kind of periodic podcast or video channel would be a great way to capture our experiences. Would anyone here find this helpful?

From lurking in this community for a while, it does seem as though the content is more focused on the technology while our idea is more about how to move the organization, so if there is a better place to ask, we'd love to know.

Anyway, looking forward to seeing your comments and ideas.

https://redd.it/ldzdxx
@r_devops

Automate Manual SQL runs

I recently started working with our DBAs to automate some of their workflows. One of the problems I've been tasked with is automating some manual SQL requests they run often. Anything outside of the normal migrations, like adding customer account data or bulk updating columns. Currently our DBAs run these manually but I would like to have a review / approval process before running along with syntax checking and output verification. Any suggestions for how to do this?

https://redd.it/ldyi02
@r_devops

Want to share Interview Preparation Courses

I have organized some of the best interview preparation courses like:

1. AlgoExpert
2. SystemsExpert
3. Epic React Pro by Kent C. Dodds
4. Grokking OOD
5. Grokking The Coding Interview
6. Coderust: Hacking The Coding Interview
7. Grokking Dynamic Programming Patterns
8. Grokking the System Design Interview
9. ZeroToMastery: Master the Coding Interview Big Tech (FAANG) Interviews
10. Gaurav Sen: System Design
11. TechSeries dev: AlgoPro, Tech Interview Pro
12. BackToBackSWE
13. CodeWithMosh
14. InterviewCake
15. InterviewCamp
16. Applied Course
17. InterviewEspresso
18. SimpleProgrammer

And some other courses. DM me if you are interested to have these courses.

https://redd.it/ldy45j
@r_devops

What DevOps KPIs do you track?

Hey folks. I am curious what are some key indicators do you track to have an understanding of how well your organization is doing with DevOps? That is if you could pick 3 metrics that would tell where you should focus and optimize your delivery pipelines, what would they be?

I would also appreciate any links to some tools that could help with such insights.

Cheers!

https://redd.it/lea2nd
@r_devops

How Can I Do Continous Deployment On My VPS With Docker?

I am trying to step up my devops chops by trying to implement a CI/CD (deployment) pipeline for my personal website.

On the CI side, I am working on implementing gitflow, automated testing, and pushing new Docker images on successful merge to master via Github Actions.

As far as the CD side, I am currently hosting my site on a Vultr VPS. I would like to set up a system that checks for a new Docker image for the site, and if one exists, it should pull it down and restart my website (probably via docker-compose) with the new Docker image. I could probably just write a cronjob that runs every few minutes and checks for a new image/redeploys but that seems inefficient and clunky.

I am aware that something like this could easily be implemented using AWS ECS/ECR but I think that would be overkill for my small site. Does there exist some tool that could do the "polling" for a new image and redeployment in a more streamlined way?

Any help/feedback would be appreciated.

https://redd.it/le9v8f
@r_devops

Building Python for multi-Linux distro deployment

We’re building some system-level applications that have to interface and configure Unix users/groups, other system tools like mail servers, nginx, etc.

The team is planning on writing the apps in Python, and distributing them via RPMs and Debs. Their reasoning for using system packages is primarily so that we can pin dependencies on the system. To me, this sounds fairly complicated, as we’d have to package the code up twice and test deployment across multiple versions of the multiple OSes.

I’ve mostly worked on ephemeral systems that were either configured with Ansible/Puppet that would configure the system + deploy applications, or baked Docker containers. Does the system package approach make more sense on non-ephemeral hosts? Is there a better alternative for Python apps? I’ve looked into something like Pyinstaller that can package the Python app into an executable, would that still be Linux OS-dependent?

Thanks!

https://redd.it/le8o13
@r_devops

Bare metal servers provisioning software

I am searching for a not very complicated system for provisioning bare metal servers (mostly Supermicro).

The steps for the new server deployment at the moment are:

1. Add server to a bootstrap network segment
2. Install base OS on the new server (Ubuntu)
3. Make some tweaks via ansible (add user/keys, some packages, iptables rules, etc)
4. Move server to the prod network segment and transfer to app admins.

Items 1,3,4 already automated with Ansible, now time to OS install process.

I'm looking to maas.io, but it looks some overkill for my purpose.

I would appreciate some advice.

Thanks!

https://redd.it/le76oz
@r_devops

Automate the initializing and unsealing of HashiCorp Vault without public cloud infrastructure

Hey everyone! Here's a small project I worked on recently that I thought people here might find interesting: https://github.com/jace-ys/vault-init.

It draws inspiration from https://github.com/kelseyhightower/vault-init, but doesn't rely on any public cloud infrastructure for the encrypting and storing of Vault's root tokens and unseal keys. You would typically use this if you do not have access to public cloud infrastructure, or if your Vault deployment must operate entirely on-prem.

Hope someone here finds it useful, and let me know if you have any issues/feedback!

https://redd.it/ldud7w
@r_devops