Moving database by buckets of data

We are planning to move from one no-sql database to another. Our customer data is stored in buckets like user, access, documents, etc.

We have a migration tool that extracts keys/values from a bucket into an intermediate json and the same tool can be then used to push the keys into the other no-sql db.

But these buckets for one customer are so huge that it will take 10 days to extract all their data into this intermediate file and another 10 days to push into the new db. Even if we do bucket by bucket, each bucket can take 5-7 days. Some buckets have more data than the others.

I was looking at AWS batch and that's what we need. Use the command as a batch process and let AWS spin up 100s of instances and execute them. But how to tell one process to extract between 1-1000, second to extract from 1000-2000, etc.

Is there a way to index all the contents of the bucket and let AWS batch split into say 100 ec2 instances and each instance does 1000 at a time in parallel?

https://redd.it/kkmxun
@r_devops

Are you seeing more Python than Go?

Hey All,

Wondering if you're seeing this same trend. Essentially I'm seeing far more Python than Go being used from an automation/cloud development perspective.

Why do you think that is?

https://redd.it/kkl0mx
@r_devops

CD for WordPress on a VM?

Hi all,

I am looking for ideas for CD for a WordPress instance on a VM. Usually we stop apache. Copy existing directories to a backup folder locally. Copy over new contents. Start apache. Test the website with the ew content.

Sometimes we may even need to take a backup of the dB on rds.

Any suggestions on how we can do it for WordPress apache. I know it's easier to do with a docker image but this is different.

https://redd.it/kkj7gv
@r_devops

Is FedoraCore OS good alternative to the CentOS?

We recently came know about CentOS moving rolling release mode. Then there was two recent articles on Fedora Magazine about Fedora CoreOS.

[https://fedoramagazine.org/deploy-fedora-coreos-with-terraform/](https://fedoramagazine.org/deploy-fedora-coreos-with-terraform/)
https://fedoramagazine.org/getting-started-with-fedora-coreos/

I know CoreOS is an acquisition of Red Hat.

But cannot we use Fedora CoreOS in the place of CentOS? Anyone using Fedora CoreOS please share their experience.

https://redd.it/kke1ht
@r_devops

(CDK vs Pulumi) vs Terraform

Hey everyone.

I've recently been hired to take lead on the IaaC side of a small company. I'm a big time Terraform guy and was initially going to go with that to provision our infrastructure. However, no one else on the team is very familiar with it, but they're all good with Python. I was wondering what people thought about CDK vs Pulumi and how the better one compares to Terraform (outside of the obvious imperative vs declarative stuff).

https://redd.it/kl7t9u
@r_devops

Configuration management for deployed applications/microservices

Hi /r/devops!


How do you handle application configuration for multiple applications across several environments? Let's say you have 100 microservices, every microservice needs to have separate configuration for database connection string, some message broker like Azure Service Bus and endpoints to other 3rd party APIs. All of them vary between different environment (QA, Staging, Production). What is the best way to handle those in a bigger number?

https://redd.it/kl99dd
@r_devops

Musings around a Dockerfile for Jekyll

If you like cool stories about how an engineer, faced with an impossible problem, overcame all odds and solved it, this post is not for you. This is a story of how I spent a non-trivial amount of time, how I hit a couple of walls, and how I nearly came back to square one. Why do I write it?

The first reason is for me: I want to document my journey so if I ever think about trying again in the future, I’ll have some arguments against it. Second, in opposition to the actual zeitgeist, you learn a lot more by failing than by succeeding. Writing about one’s learning "bury" it deeper into memory.

https://blog.frankel.ch/musings-dockerfile-jekyll/

https://redd.it/kl7oqx
@r_devops

Ops Tools Marketing Bullshit Dictionary

I feel that often the marketing is dishonest and/or feels like the product or service is shoved down my throat. Do you feel the same way?

Here are some examples of bullshit and what in my view should be the response:

https://ilya-sher.org/2020/12/27/ops-tools-marketing-bullshit-dictionary/

https://redd.it/kkz1sh
@r_devops

What are the tools/language/service I should know to become a Junior DevOps Engineer?

Hi, I'm a web developer and I like to change my path to DevOps Engineer for 2021.

Thank you!

https://redd.it/kkxrqt
@r_devops

On-Call in Area with Severe Weather

I'm giving consideration to buying a home in an area with some severe storms during the winter months.

However, working in this field, I'm required to participate in my company's on-call rotation, and respond to incidents. With this being the case, I need to make sure that I maintain network connectivity at all times (including power outages).

I can use a generator to maintain my home's power, but if the power is out for an extended period (8+ hours), the network infrastructure may also fail.

Redundant network infrastructure could work (cable for normal usage, satellite during wide-spread outages); I think the combination of a generator + cable + satellite (as a fallback) should get me the level of reliability that I need.

Has anyone done this before? Am I going down a bad road?

My apologies if this post doesn't have enough relation to DevOps!

https://redd.it/kkqmgr
@r_devops

What is expected from DevOps lead ?

I am experienced dev with decent programming knowledge of Python / C-C++ / Hack and SQL along with Linux. Last two years I have been working with team designing kubernetes based platform for very large e-commerce organization. I acquired fair bit of knowledge about containerization / reconciliation design pattern / release management / ci-cd / monitoring.

I have a year of experience leading a team of 3 Jr developers along with talking to various stakeholders including client in service based startup.

I recently accepted a position as DevOps lead for a small company which is using azure services ( service fabric / .Net / SQL server ). Fairly small team, I will be the first devOps specialist.

I wanted to ask more seasoned DevOps Engineer and tech leads what do you think my first steps should be to make most of this opportunity?

https://redd.it/kldq4l
@r_devops

Coming to DevOps? Learn how to code.

From two main ways to enter the modern infrastructure development world, the ops to dev way might be the more challenging one. One critical skill that you have to acquire coming to DevOps is how to write code. In this video, I explain why: https://youtu.be/etMiQiLJY6M

https://redd.it/klm6p7
@r_devops

Are you involved in DevOps? Take this survey to share your experiences with AWS , CI/CD processes, cloudmonitoring and you can win a new smartphone, gaming laptop, licenses, Amazon vouchers, testing gear and more!

https://survey.developereconomics.com/name/de20?utm\_medium=some\_org&utm\_source=reddit&utm\_campaign=de20

https://redd.it/klm1jf
@r_devops

What self hosted Git solution are you using?

I'm looking for a quick and easy self hosted Git solution just for my own private projects at home. What would you guys suggest? I've been looking at https://gitea.io/en-us/ which seems to be quite easy to setup/use. I have Gitlab in work, but it seems to be a bit too big and overkill for my simple requirements of writing some code and tracking it a local repo.

https://redd.it/klqfla
@r_devops

Does ArgoCD supports auto rollback?

I'm using ArgoCD for continuous deployment. I'm pretty new to DevOps so I don't know which tool has which features. Does ArgoCD have a feature to rollback if latest deployment have health issues?

Here's example situation: Sometimes I'm committing bad code to master branch that gets deployed to our production cluster and crashes our main API. I read somewhere that CD tools usually have a feature that rolls back to previous deployment if some part of the application become offline or during increased error rates. Does Argo CD solves that problem or not?

\> I know, i know I have to test code before putting it into production. But our small startup team doesn't have enough time to spend on writing unit, integration tests. So we have to break things and move fast.

Thanks

https://redd.it/klo5cg
@r_devops

How can I self study to apply to a DevOps engineer position?

My current kind of work is mainly on Jenkins, doing some simple CI stuff like writing groovy, bash, python, some basic linux troubleshooting and no more. I know DevOps has a broad range of tools and concepts and I don’t know how to self study it.
So how do you guys self study DevOps? How is your progress?

https://redd.it/klpvm2
@r_devops

DevOps Interviews (Europe)

Hi all,

what is your experience regarding Interviews for DevOps positions in the last months?

All companies I interviewed with asked for AWS/GCP and Terraform. Seems at the end your just guy who writes 40 hours a week Terraform code.

I have a CKA Cert, CKS is scheduled this week, plus heavy K8s, GitLab CI/CD, VMware, Vault, Ansible, Salt, Salt-Cloud, Prometheus, Grafana, Loki experience, but all on-prem.

It is rlly frustrating as it seems clicking around in AWS and writing Terraform is more valued than run your own Infrastructure or your own K8S Cluster.

https://redd.it/klws08
@r_devops

GitLab - what's so special?

Guys, I have never tried out GitLab before. We currently have a Jenkins implementation (on-premise) with a GitHub code repo, Jfrog Artifactory and a bunch of other open source tools for scanning vulnerabilities, testing etc. We are still relatively new in our DevOps journey (20% of total workloads) and don't really have a vision on our LT strategy here (we do have some folks who are wedded to their tools, though).

Thought it made sense to ask those who use GitLab about what they find special about it? And how did they come to evolve their long term vision for how they develop and deploy code (and what's realistic)?

https://redd.it/km593t
@r_devops

View errors in sentry.io?

I have a project using Sentry.io. In the dashboard it shows "40 errors" and when I click on that it just shows me a list of "issues" which only shows the initial "it works" event.

How can I view the actual errors that are taking place?

https://redd.it/km1ics
@r_devops

Future of DevOps

Hi All,

I am a fresher and was offered DevOps role which I accepted. Please enlighten me about the demand for DevOps engineers, salary growth I can expect after 2-5 years and the difference between SRE and DevOps.

Should I try to switch to developer role or DevOps is as rewarding as development?

Also, if anyone can share learning path or material, that would be helpful.

Thanks in advance.

https://redd.it/klurdu
@r_devops

Are there any online services that provides Win-10 VM's with an nvidia GPU? (for debugging).

Not sure if this is the right place to ask, but figured it was worth a shot!

Does anyone know of any services that provide Windows 10 VM's and also specify which GPU (if any) they use under the hood? (Or possibly even provide an option to choose from different GPU configurations)

I have an AMD graphics card, but I need to do some debugging on a Windows machine with an NVidia card. Only need to install visual studio on it, and transfer a few files to it. So don't need a lot in terms of perforamnce.

Thanks!

https://redd.it/kly9sz
@r_devops