Наверняка видели кучу подобных картинок! Вот это мне понравилась из недавних 😁
https://www.linkedin.com/posts/mr-deepak-bhardwaj_dataarchitecture-dataplatform-machinelearning-activity-7238762147990618112-RnrP
Data Platform Architecture: Data Flow and Modelling
The landscape of modern data architecture is intricate. Yet, mastering its core components is essential for building robust, scalable, and efficient data platforms. Let’s break down the essentials:
🔘 𝐏𝐥𝐚𝐭𝐟𝐨𝐫𝐦 𝐅𝐨𝐮𝐧𝐝𝐚𝐭𝐢𝐨𝐧
↳ Access management, encryption, backup/restore, computing, storage, networking, and monitoring form the backbone of your platform, ensuring resilience, scalability, and availability.
🔘 𝐃𝐚𝐭𝐚 𝐒𝐨𝐮𝐫𝐜𝐞𝐬
↳ Operational Databases & Files: Collection of raw data from various systems and sources.
🔘 𝐃𝐚𝐭𝐚 𝐋𝐚𝐤𝐞 (𝐒𝐭𝐚𝐠𝐢𝐧𝐠 𝐀𝐫𝐞𝐚)
↳ Landing Zone: Temporary storage for raw and transient data before processing.
↳ Persistent Staging Area (PSA): Long-term storage for historical data, auditing, and retention.
🔘 𝐋𝐚𝐤𝐞 𝐇𝐨𝐮𝐬𝐞 (𝐃𝐚𝐭𝐚 𝐕𝐚𝐮𝐥𝐭𝐬)
↳ Raw Data Vault: Source-aligned structure for tracking, time-travel, and high-volume writes.
↳ Business Data Vault: Business-aligned structure optimised for query performance and data aggregation.
🔘 𝐃𝐚𝐭𝐚 𝐖𝐚𝐫𝐞𝐡𝐨𝐮𝐬𝐞 (𝐃𝐚𝐭𝐚 𝐌𝐚𝐫𝐭𝐬)
↳ Subject-Oriented Design: Organised around specific business functions, often using star schemas.
↳ Aggregated and Summarised Data: Stores aggregated data to simplify analysis and reporting.
↳ Optimised for Query Performance: Ensures fast and efficient data retrieval for end-users.
🔘 𝐄𝐯𝐞𝐧𝐭 𝐁𝐮𝐬 & 𝐒𝐭𝐫𝐞𝐚𝐦 𝐀𝐧𝐚𝐥𝐲𝐭𝐢𝐜𝐬
↳ Real-time data processing is critical. Stream Analytics provides immediate insights, while the Event Bus connects systems, apps, and IoT devices.
🔘 𝐌𝐚𝐜𝐡𝐢𝐧𝐞 𝐋𝐞𝐚𝐫𝐧𝐢𝐧𝐠 𝐈𝐧𝐭𝐞𝐠𝐫𝐚𝐭𝐢𝐨𝐧
↳ Data feeds into feature stores, enabling the training and utilisation of machine learning models, resulting in more innovative, adaptive systems.
🔘 𝐆𝐨𝐯𝐞𝐫𝐧𝐚𝐧𝐜𝐞 & 𝐒𝐞𝐜𝐮𝐫𝐢𝐭𝐲
↳ Policies, data quality, cataloguing, and security measures like encryption and access management ensure that data is safe, compliant, and valuable.
#DataArchitecture #DataPlatform #DataWarehouse #DataLake #BigData
#DataPlatformArchitecture #DataFlow #DataModelling
https://www.linkedin.com/posts/mr-deepak-bhardwaj_dataarchitecture-dataplatform-machinelearning-activity-7238762147990618112-RnrP
Data Platform Architecture: Data Flow and Modelling
The landscape of modern data architecture is intricate. Yet, mastering its core components is essential for building robust, scalable, and efficient data platforms. Let’s break down the essentials:
🔘 𝐏𝐥𝐚𝐭𝐟𝐨𝐫𝐦 𝐅𝐨𝐮𝐧𝐝𝐚𝐭𝐢𝐨𝐧
↳ Access management, encryption, backup/restore, computing, storage, networking, and monitoring form the backbone of your platform, ensuring resilience, scalability, and availability.
🔘 𝐃𝐚𝐭𝐚 𝐒𝐨𝐮𝐫𝐜𝐞𝐬
↳ Operational Databases & Files: Collection of raw data from various systems and sources.
🔘 𝐃𝐚𝐭𝐚 𝐋𝐚𝐤𝐞 (𝐒𝐭𝐚𝐠𝐢𝐧𝐠 𝐀𝐫𝐞𝐚)
↳ Landing Zone: Temporary storage for raw and transient data before processing.
↳ Persistent Staging Area (PSA): Long-term storage for historical data, auditing, and retention.
🔘 𝐋𝐚𝐤𝐞 𝐇𝐨𝐮𝐬𝐞 (𝐃𝐚𝐭𝐚 𝐕𝐚𝐮𝐥𝐭𝐬)
↳ Raw Data Vault: Source-aligned structure for tracking, time-travel, and high-volume writes.
↳ Business Data Vault: Business-aligned structure optimised for query performance and data aggregation.
🔘 𝐃𝐚𝐭𝐚 𝐖𝐚𝐫𝐞𝐡𝐨𝐮𝐬𝐞 (𝐃𝐚𝐭𝐚 𝐌𝐚𝐫𝐭𝐬)
↳ Subject-Oriented Design: Organised around specific business functions, often using star schemas.
↳ Aggregated and Summarised Data: Stores aggregated data to simplify analysis and reporting.
↳ Optimised for Query Performance: Ensures fast and efficient data retrieval for end-users.
🔘 𝐄𝐯𝐞𝐧𝐭 𝐁𝐮𝐬 & 𝐒𝐭𝐫𝐞𝐚𝐦 𝐀𝐧𝐚𝐥𝐲𝐭𝐢𝐜𝐬
↳ Real-time data processing is critical. Stream Analytics provides immediate insights, while the Event Bus connects systems, apps, and IoT devices.
🔘 𝐌𝐚𝐜𝐡𝐢𝐧𝐞 𝐋𝐞𝐚𝐫𝐧𝐢𝐧𝐠 𝐈𝐧𝐭𝐞𝐠𝐫𝐚𝐭𝐢𝐨𝐧
↳ Data feeds into feature stores, enabling the training and utilisation of machine learning models, resulting in more innovative, adaptive systems.
🔘 𝐆𝐨𝐯𝐞𝐫𝐧𝐚𝐧𝐜𝐞 & 𝐒𝐞𝐜𝐮𝐫𝐢𝐭𝐲
↳ Policies, data quality, cataloguing, and security measures like encryption and access management ensure that data is safe, compliant, and valuable.
#DataArchitecture #DataPlatform #DataWarehouse #DataLake #BigData
#DataPlatformArchitecture #DataFlow #DataModelling
Please open Telegram to view this post
VIEW IN TELEGRAM