DataExpert-io/data-engineer-handbook
This is a repo with links to everything you'd ever want to learn about data engineering
Language:Jupyter Notebook
Total stars: 24911
Stars trend:
#jupyternotebook
#apachespark, #awesome, #bigdata, #data, #dataengineering, #sql
This is a repo with links to everything you'd ever want to learn about data engineering
Language:Jupyter Notebook
Total stars: 24911
Stars trend:
22 Jan 2025
9am ▏ +1
10am ▊ +6
11am ▉ +7
12pm ▊ +6
1pm █▏ +9
2pm █▍ +11
3pm █ +8
4pm █▏ +9
5pm █ +8
6pm ▍ +3
7pm █ +8#jupyternotebook
#apachespark, #awesome, #bigdata, #data, #dataengineering, #sql
🔥1
D4Vinci/Scrapling
🕷️ Undetectable, Lightning-Fast, and Adaptive Web Scraping for Python
Language:Python
Total stars: 2301
Stars trend:
#python
#ai, #aiscraping, #automation, #crawler, #crawling, #crawlingpython, #data, #dataextraction, #hacktoberfest, #playwright, #python, #python3, #scraping, #selectors, #stealth, #webscraper, #webscraping, #webscrapingpython, #webscraping, #xpath
🕷️ Undetectable, Lightning-Fast, and Adaptive Web Scraping for Python
Language:Python
Total stars: 2301
Stars trend:
12 Feb 2025
6am ▍ +3
7am ▎ +2
8am ▋ +5
9am ▍ +3
10am ▊ +6
11am ▉ +7
12pm █▉ +15
1pm █▌ +12
2pm █▉ +15
3pm █▌ +12
4pm ▊ +6
5pm █▋ +13#python
#ai, #aiscraping, #automation, #crawler, #crawling, #crawlingpython, #data, #dataextraction, #hacktoberfest, #playwright, #python, #python3, #scraping, #selectors, #stealth, #webscraper, #webscraping, #webscrapingpython, #webscraping, #xpath
sinaptik-ai/pandas-ai
Chat with your database or your datalake (SQL, CSV, parquet). PandasAI makes data analysis conversational using LLMs and RAG.
Language:Python
Total stars: 17113
Stars trend:
#python
#ai, #csv, #data, #dataanalysis, #datascience, #datavisualization, #database, #datalake, #gpt4, #llm, #pandas, #sql, #texttosql
Chat with your database or your datalake (SQL, CSV, parquet). PandasAI makes data analysis conversational using LLMs and RAG.
Language:Python
Total stars: 17113
Stars trend:
2 Mar 2025
5am ██ +16
6am █▍ +11
7am █▏ +9
8am ▉ +7
9am █ +8
10am ▉ +7
11am ▊ +6
12pm █ +8
1pm █ +8
2pm █▌ +12
3pm ▍ +3
4pm █▍ +11#python
#ai, #csv, #data, #dataanalysis, #datascience, #datavisualization, #database, #datalake, #gpt4, #llm, #pandas, #sql, #texttosql
cocoindex-io/cocoindex
ETL framework to turn your data AI-ready - with realtime incremental updates and support custom logic like lego.
Language:Rust
Total stars: 672
Stars trend:
#rust
#ai, #changedatacapture, #data, #dataengineering, #dataindexing, #datainfrastructure, #dataprocessing, #dataflow, #etl, #helpwanted, #indexing, #knowledgegraph, #llm, #pipeline, #python, #rag, #realtime, #rust, #semanticsearch, #streaming
ETL framework to turn your data AI-ready - with realtime incremental updates and support custom logic like lego.
Language:Rust
Total stars: 672
Stars trend:
20 Apr 2025
3pm █▉ +15
4pm ▍ +3
5pm ▋ +5
6pm ▉ +7
7pm ▋ +5
8pm ▋ +5
9pm ▊ +6
10pm ▊ +6
11pm █ +8
21 Apr 2025
12am ▍ +3
1am ▉ +7
2am ▉ +7#rust
#ai, #changedatacapture, #data, #dataengineering, #dataindexing, #datainfrastructure, #dataprocessing, #dataflow, #etl, #helpwanted, #indexing, #knowledgegraph, #llm, #pipeline, #python, #rag, #realtime, #rust, #semanticsearch, #streaming
spiceai/spiceai
A portable accelerated data query and LLM-inference engine, written in Rust, for data-grounded AI apps and agents.
Language:Rust
Total stars: 2249
Stars trend:
#rust
#artificialintelligence, #data, #developers, #infrastructure, #machinelearning, #sql, #timeseries
A portable accelerated data query and LLM-inference engine, written in Rust, for data-grounded AI apps and agents.
Language:Rust
Total stars: 2249
Stars trend:
22 Apr 2025
12pm ▏ +1
1pm █ +8
2pm ▉ +7
3pm █▌ +12
4pm █▊ +14
5pm █▎ +10
6pm ▋ +5
7pm ▊ +6
8pm ▊ +6
9pm █▋ +13#rust
#artificialintelligence, #data, #developers, #infrastructure, #machinelearning, #sql, #timeseries
meta-llama/synthetic-data-kit
Tool for generating high quality Synthetic datasets
Language:Python
Total stars: 104
Stars trend:
#python
#data, #generation, #llm, #python, #synthetic
Tool for generating high quality Synthetic datasets
Language:Python
Total stars: 104
Stars trend:
29 Apr 2025
4pm █▋ +13
5pm █▋ +13
6pm ▍ +3
7pm ▊ +6
8pm ▋ +5
9pm ██▎ +18
10pm █ +8
11pm ▋ +5
30 Apr 2025
12am ▋ +5
1am ▎ +2
2am ▉ +7
3am ▊ +6#python
#data, #generation, #llm, #python, #synthetic
cocoindex-io/cocoindex
Real-time data transformation framework for AI. Ultra performant, with incremental processing.
Language:Rust
Total stars: 1459
Stars trend:
#rust
#ai, #changedatacapture, #data, #dataengineering, #dataindexing, #datainfrastructure, #dataprocessing, #dataflow, #etl, #helpwanted, #indexing, #knowledgegraph, #llm, #pipeline, #python, #rag, #realtime, #rust, #semanticsearch, #streaming
Real-time data transformation framework for AI. Ultra performant, with incremental processing.
Language:Rust
Total stars: 1459
Stars trend:
20 May 2025
1am ▊ +6
2am ▍ +3
3am ▊ +6
4am █▎ +10
5am █▎ +10
6am ▋ +5
7am █▎ +10
8am ▎ +2
9am ▌ +4
10am █▎ +10
11am ▉ +7
12pm █ +8#rust
#ai, #changedatacapture, #data, #dataengineering, #dataindexing, #datainfrastructure, #dataprocessing, #dataflow, #etl, #helpwanted, #indexing, #knowledgegraph, #llm, #pipeline, #python, #rag, #realtime, #rust, #semanticsearch, #streaming
speedyapply/2025-AI-College-Jobs
2025 AI/ML internship & new graduate job list updated daily
Language:
Total stars: 1330
Stars trend:
#ai, #applications, #artificialintelligence, #college, #data, #datascience, #fulltime, #hiring, #intern, #internship, #internships, #interviews, #jobbboard, #jobs, #junior, #machinelearning, #ml, #newgrad, #quant, #university
2025 AI/ML internship & new graduate job list updated daily
Language:
Total stars: 1330
Stars trend:
31 May 2025
11pm ▏ +1
1 Jun 2025
12am +0
1am ▋ +5
2am ▌ +4
3am ▉ +7
4am █▌ +12
5am █▌ +12
6am █▉ +15
7am ▋ +5
8am ▌ +4
9am ▉ +7
10am ▉ +7#ai, #applications, #artificialintelligence, #college, #data, #datascience, #fulltime, #hiring, #intern, #internship, #internships, #interviews, #jobbboard, #jobs, #junior, #machinelearning, #ml, #newgrad, #quant, #university
DataExpert-io/data-engineer-handbook
This is a repo with links to everything you'd ever want to learn about data engineering
Language:Jupyter Notebook
Total stars: 28322
Stars trend:
#jupyternotebook
#apachespark, #awesome, #bigdata, #data, #dataengineering, #sql
This is a repo with links to everything you'd ever want to learn about data engineering
Language:Jupyter Notebook
Total stars: 28322
Stars trend:
1 Jun 2025
6pm ▏ +1
7pm ▏ +1
8pm ▏ +1
9pm +0
10pm +0
11pm ▊ +6
2 Jun 2025
12am ████ +32
1am █▋ +13
2am █▉ +15
3am █▋ +13#jupyternotebook
#apachespark, #awesome, #bigdata, #data, #dataengineering, #sql
D4Vinci/Scrapling
🕷️ An undetectable, powerful, flexible, high-performance Python library to make Web Scraping Easy and Effortless as it should be!
Language:Python
Total stars: 5620
Stars trend:
#python
#ai, #aiscraping, #automation, #crawler, #crawling, #crawlingpython, #data, #dataextraction, #hacktoberfest, #playwright, #python, #python3, #scraping, #selectors, #stealth, #webscraper, #webscraping, #webscrapingpython, #webscraping, #xpath
🕷️ An undetectable, powerful, flexible, high-performance Python library to make Web Scraping Easy and Effortless as it should be!
Language:Python
Total stars: 5620
Stars trend:
7 Jul 2025
8pm ██▏ +17
9pm █▎ +10
10pm █▋ +13
11pm █▋ +13
8 Jul 2025
12am █▎ +10
1am █▌ +12#python
#ai, #aiscraping, #automation, #crawler, #crawling, #crawlingpython, #data, #dataextraction, #hacktoberfest, #playwright, #python, #python3, #scraping, #selectors, #stealth, #webscraper, #webscraping, #webscrapingpython, #webscraping, #xpath
mendableai/firecrawl
🔥 Turn entire websites into LLM-ready markdown or structured data. Scrape, crawl and extract with a single API.
Language:TypeScript
Total stars: 43926
Stars trend:
#typescript
#ai, #aiscraping, #crawler, #data, #htmltomarkdown, #llm, #markdown, #rag, #scraper, #scraping, #webcrawler, #webscraping
🔥 Turn entire websites into LLM-ready markdown or structured data. Scrape, crawl and extract with a single API.
Language:TypeScript
Total stars: 43926
Stars trend:
30 Jul 2025
11pm ▏ +1
31 Jul 2025
12am ▉ +7
1am ▌ +4
2am █▎ +10
3am █▎ +10
4am ▌ +4
5am █▏ +9
6am █▍ +11
7am █▏ +9
8am ▋ +5
9am █▏ +9
10am █▍ +11#typescript
#ai, #aiscraping, #crawler, #data, #htmltomarkdown, #llm, #markdown, #rag, #scraper, #scraping, #webcrawler, #webscraping
any4ai/AnyCrawl
AnyCrawl 🚀: A Node.js/TypeScript crawler that turns websites into LLM-ready data and extracts structured SERP results from Google/Bing/Baidu/etc. Native multi-threading for bulk processing.
Language:TypeScript
Total stars: 1201
Stars trend:
#typescript
#aiscraping, #aitools, #crawl, #data, #htmltomarkdown, #rag, #scrape, #scraping, #serp, #webscraper
AnyCrawl 🚀: A Node.js/TypeScript crawler that turns websites into LLM-ready data and extracts structured SERP results from Google/Bing/Baidu/etc. Native multi-threading for bulk processing.
Language:TypeScript
Total stars: 1201
Stars trend:
31 Jul 2025
4pm █ +8
5pm █▍ +11
6pm ▉ +7
7pm ▋ +5
8pm █▍ +11
9pm █▏ +9
10pm ▉ +7
11pm █ +8
1 Aug 2025
12am █▌ +12
1am ▊ +6#typescript
#aiscraping, #aitools, #crawl, #data, #htmltomarkdown, #rag, #scrape, #scraping, #serp, #webscraper
firecrawl/firecrawl
The Web Data API for AI - Turn entire websites into LLM-ready markdown or structured data 🔥
Language:TypeScript
Total stars: 49183
Stars trend:
#typescript
#ai, #aiscraping, #crawler, #data, #htmltomarkdown, #llm, #markdown, #rag, #scraper, #scraping, #webcrawler, #webscraping
The Web Data API for AI - Turn entire websites into LLM-ready markdown or structured data 🔥
Language:TypeScript
Total stars: 49183
Stars trend:
19 Aug 2025
1am ▋ +5
2am ▋ +5
3am ▌ +4
4am ▌ +4
5am ▉ +7
6am █▌ +12
7am █▏ +9
8am █▎ +10
9am ▌ +4
10am ▊ +6
11am █ +8
12pm █▎ +10#typescript
#ai, #aiscraping, #crawler, #data, #htmltomarkdown, #llm, #markdown, #rag, #scraper, #scraping, #webcrawler, #webscraping
ruc-datalab/DeepAnalyze
DeepAnalyze is the first agentic LLM for autonomous data science.
Language:Python
Total stars: 172
Stars trend:
#python
#agent, #agentic, #agenticai, #chatbot, #data, #dataanalysis, #dataengineering, #datascience, #datavisualization, #gpt, #llm
DeepAnalyze is the first agentic LLM for autonomous data science.
Language:Python
Total stars: 172
Stars trend:
21 Oct 2025
12pm ▏ +1
1pm ▏ +1
2pm ▎ +2
3pm ▊ +6
4pm ▎ +2
5pm ▏ +1
6pm +0
7pm █ +8
8pm ▌ +4
9pm ▎ +2
10pm ▎ +2
11pm ▍ +3#python
#agent, #agentic, #agenticai, #chatbot, #data, #dataanalysis, #dataengineering, #datascience, #datavisualization, #gpt, #llm
deepnote/deepnote
Deepnote is a drop-in replacement for Jupyter with an AI-first design, sleek UI, new blocks, and native data integrations. Use Python, R, and SQL locally in your favorite IDE, then scale to Deepnote cloud for real-time collaboration, Deepnote agent, and deployable data apps. https://deepnote.com/
Language:TypeScript
Total stars: 278
Stars trend:
#typescript
#artificialintelligence, #data, #dataanalysis, #datascience, #datavisualization, #deepnote, #eda, #jupyter, #jupyterhub, #jupyterlab, #machinelearning, #notebooks, #python, #r, #sql
Deepnote is a drop-in replacement for Jupyter with an AI-first design, sleek UI, new blocks, and native data integrations. Use Python, R, and SQL locally in your favorite IDE, then scale to Deepnote cloud for real-time collaboration, Deepnote agent, and deployable data apps. https://deepnote.com/
Language:TypeScript
Total stars: 278
Stars trend:
4 Nov 2025
1pm ▏ +1
2pm ▌ +4
3pm ▉ +7
4pm █▌ +12
5pm █▉ +15
6pm ██▋ +21
7pm ███▍ +27#typescript
#artificialintelligence, #data, #dataanalysis, #datascience, #datavisualization, #deepnote, #eda, #jupyter, #jupyterhub, #jupyterlab, #machinelearning, #notebooks, #python, #r, #sql
bytewax/awesome-public-real-time-datasets
A list of publicly available datasets with real-time data maintained by the team at bytewax.io
Language:
Total stars: 1480
Stars trend:
#awesomelist, #data, #datascience, #datavisualization, #datasets, #realtime, #streaming
A list of publicly available datasets with real-time data maintained by the team at bytewax.io
Language:
Total stars: 1480
Stars trend:
13 Nov 2025
5pm ▏ +1
6pm +0
7pm ▏ +1
8pm +0
9pm +0
10pm ▏ +1
11pm ▏ +1
14 Nov 2025
12am ▎ +2
1am ▎ +2
2am █▏ +9
3am █▎ +10
4am ▍ +3#awesomelist, #data, #datascience, #datavisualization, #datasets, #realtime, #streaming
nkzw-tech/fate
fate is a modern data client for React.
Language:TypeScript
Total stars: 125
Stars trend:
#typescript
#async, #data, #react
fate is a modern data client for React.
Language:TypeScript
Total stars: 125
Stars trend:
9 Dec 2025
2pm ▏ +1
3pm ▊ +6
4pm ▎ +2
5pm ▎ +2
6pm ▎ +2
7pm ▏ +1
8pm ▏ +1
9pm ▎ +2
10pm ▎ +2
11pm +0
10 Dec 2025
12am ▊ +6#typescript
#async, #data, #react
metabase/metabase
The easy-to-use open source Business Intelligence and Embedded Analytics tool that lets everyone work with data :bar_chart:
Language:Clojure
Total stars: 45129
Stars trend:
#clojure
#analytics, #bi, #businessintelligence, #businessintelligence, #clojure, #dashboard, #data, #dataanalysis, #datavisualization, #database, #metabase, #mysql, #postgres, #postgresql, #reporting, #slack, #sqleditor, #visualization
The easy-to-use open source Business Intelligence and Embedded Analytics tool that lets everyone work with data :bar_chart:
Language:Clojure
Total stars: 45129
Stars trend:
19 Dec 2025
10am ▎ +2
11am ▋ +5
12pm ▎ +2
1pm ▏ +1
2pm ▍ +3
3pm ▉ +7
4pm ▋ +5
5pm ▏ +1
6pm ▌ +4
7pm ▏ +1
8pm ▎ +2
9pm ▎ +2#clojure
#analytics, #bi, #businessintelligence, #businessintelligence, #clojure, #dashboard, #data, #dataanalysis, #datavisualization, #database, #metabase, #mysql, #postgres, #postgresql, #reporting, #slack, #sqleditor, #visualization