If you're a Data Engineer working on Azure, you'll likely encounter Synapse or Databricks.
I acknowledge that there are other solutions like HDInsights (which is somewhat outdated), Microsoft Fabric (still quite new and lacking a clear engineering focus for data engineering), or Azure Data Explorer, which represents a different paradigm.
Let's focus on Synapse and Databricks.
So, which one should you choose?
Based on my experience, engineers tend to prefer Databricks, whereas organizations often opt for Synapse and expect engineers to manage it. There are numerous arguments in favor of each product. Both are accessible through the same Azure portal and have considerable overlap.
This post aims to highlight the differences between the two, and I've summarized these in the accompanying picture.
Which one do you prefer?
#dataengineering #surfalytics #synapse #databricks #azure #azuredataengineer
https://www.linkedin.com/posts/dmitryanoshin_dataengineering-surfalytics-synapse-activity-7143313309646094336-oigm
I acknowledge that there are other solutions like HDInsights (which is somewhat outdated), Microsoft Fabric (still quite new and lacking a clear engineering focus for data engineering), or Azure Data Explorer, which represents a different paradigm.
Let's focus on Synapse and Databricks.
So, which one should you choose?
Based on my experience, engineers tend to prefer Databricks, whereas organizations often opt for Synapse and expect engineers to manage it. There are numerous arguments in favor of each product. Both are accessible through the same Azure portal and have considerable overlap.
This post aims to highlight the differences between the two, and I've summarized these in the accompanying picture.
Which one do you prefer?
#dataengineering #surfalytics #synapse #databricks #azure #azuredataengineer
https://www.linkedin.com/posts/dmitryanoshin_dataengineering-surfalytics-synapse-activity-7143313309646094336-oigm
🔥3🤔1
What is the most cost-effective solution for data engineering?
The answer doesn't lie in a specific vendor or technology, but in understanding the use cases and setting up optimal infrastructure. This involves leveraging best practices in software and appropriately sizing hardware. In essence, there's a strong correlation between a team's skill set for a particular stack and cost-tracking initiatives.
Often, open source solutions can be more expensive than commercial offerings. Consider a scenario where a team of 10 data engineers struggles with infrastructure configuration and daily on-call duties for an open-source solution to keep pipelines operational, versus just 2 data engineers efficiently managing the same with a commercial offering.
For fun, I tried to find evidence that Snowflake is cheaper than Databricks, but I couldn't. This seems like a great niche for Snowflake's SEO optimization.
I suspect Snowflake should cost roughly the same for the same compute instance, but with Data Warehouse as a Service, there's often less focus on right optimization and query patterns.
Have you seen any evidence that Snowflake is cheaper than Databricks? How would you choose the right solution?
https://www.linkedin.com/posts/dmitryanoshin_dataengineering-dataanalytics-surfalytics-activity-7143663492347117568-JrB1
The answer doesn't lie in a specific vendor or technology, but in understanding the use cases and setting up optimal infrastructure. This involves leveraging best practices in software and appropriately sizing hardware. In essence, there's a strong correlation between a team's skill set for a particular stack and cost-tracking initiatives.
Often, open source solutions can be more expensive than commercial offerings. Consider a scenario where a team of 10 data engineers struggles with infrastructure configuration and daily on-call duties for an open-source solution to keep pipelines operational, versus just 2 data engineers efficiently managing the same with a commercial offering.
For fun, I tried to find evidence that Snowflake is cheaper than Databricks, but I couldn't. This seems like a great niche for Snowflake's SEO optimization.
I suspect Snowflake should cost roughly the same for the same compute instance, but with Data Warehouse as a Service, there's often less focus on right optimization and query patterns.
Have you seen any evidence that Snowflake is cheaper than Databricks? How would you choose the right solution?
https://www.linkedin.com/posts/dmitryanoshin_dataengineering-dataanalytics-surfalytics-activity-7143663492347117568-JrB1
Linkedin
🏄♂️ Dmitry Anoshin on LinkedIn: #dataengineering #dataanalytics #surfalytics #snowflake #databricks
What is the most cost-effective solution for data engineering?
The answer doesn't lie in a specific vendor or technology, but in understanding the use cases…
The answer doesn't lie in a specific vendor or technology, but in understanding the use cases…
❤5🔥2🐳1🫡1
Hey and Happy New Year!✨ We’ve got some super exciting news for you!
🌊🏄 Join our Surfalytics Bootcamp in Tofino this Summer!
📅 Dates: Tue,Aug 20 - Sun,Aug 25, 2024
📍 Location: The picturesque Tofino, a prime surfing destination
What we offer:
🏄♀️ Professional surf training for all skill levels
🏕️ Breathtaking scenery and unforgettable experiences
🤝 Meet and connect with fellow data enthusiasts. Discuss the latest trends in the data industry, sharp your analytics and engineering skills, and prepare for next season of tech interviews.
❗ Limited spots available
🔥 Book your spot now and catch the wave of a lifetime! #SurfTofino #surfalytics
https://www.instagram.com/reel/C16RY7auF_z/?igsh=NzNndWsxaXRzaTh3
🌊🏄 Join our Surfalytics Bootcamp in Tofino this Summer!
📅 Dates: Tue,Aug 20 - Sun,Aug 25, 2024
📍 Location: The picturesque Tofino, a prime surfing destination
What we offer:
🏄♀️ Professional surf training for all skill levels
🏕️ Breathtaking scenery and unforgettable experiences
🤝 Meet and connect with fellow data enthusiasts. Discuss the latest trends in the data industry, sharp your analytics and engineering skills, and prepare for next season of tech interviews.
❗ Limited spots available
🔥 Book your spot now and catch the wave of a lifetime! #SurfTofino #surfalytics
https://www.instagram.com/reel/C16RY7auF_z/?igsh=NzNndWsxaXRzaTh3
❤🔥6🍾2