This repository contains a collection of everything needed to work with libraries related to AI and LLM.

More than 120 libraries, sorted by stages of LLM development:

→ Training, fine-tuning, and evaluation of LLM models
→ Integration and deployment of applications with LLM and RAG
→ Fast and scalable model launching
→ Working with data: extraction, structuring, and synthetic generation
→ Creating autonomous agents based on LLM
→ Prompt optimization and ensuring safe use in production

🌟 link: https://github.com/Shubhamsaboo/awesome-llm-apps

👉 @codeprogrammer

🥇 This repo is like gold for every data scientist!

✅ Just open your browser; a ton of interactive exercises and real experiences await you. Any question about statistics, probability, Python, or machine learning, you'll get the answer right there! With code, charts, even animations. This way, you don't waste time, and what you learn really sticks in your mind!

⬅️ Data science statistics and probability topics
⬅️ Clustering
⬅️ Principal Component Analysis (PCA)
⬅️ Bagging and Boosting techniques
⬅️ Linear regression
⬅️ Neural networks and more...


┌ 📂 Int Data Science Python Dash
└ 🐱 GitHub-Repos

👉 @codeprogrammer

Want to learn Python quickly and from scratch? Then here’s what you need — CodeEasy: Python Essentials

🔹Explains complex things in simple words
🔹Based on a real story with tasks throughout the plot
🔹Free start

Ready to begin? Click https://codeeasy.io/course/python-essentials 🌟

👉 @DataScience4

🎁⏳These 6 steps make every future post on LLMs instantly clear and meaningful.

Learn exactly where Web Scraping, Tokenization, RLHF, Transformer Architectures, ONNX Optimization, Causal Language Modeling, Gradient Clipping, Adaptive Learning, Supervised Fine-Tuning, RLAIF, TensorRT Inference, and more fit into the LLM pipeline.

﹌﹌﹌﹌﹌﹌﹌﹌﹌

》 𝗕𝘂𝗶𝗹𝗱𝗶𝗻𝗴 𝗟𝗟𝗠𝘀: 𝗧𝗵𝗲 𝟲 𝗘𝘀𝘀𝗲𝗻𝘁𝗶𝗮𝗹 𝗦𝘁𝗲𝗽𝘀

✸ 1️⃣ Data Collection (Web Scraping & Curation)

☆ Web Scraping: Gather data from books, research papers, Wikipedia, GitHub, Reddit, and more using Scrapy, BeautifulSoup, Selenium, and APIs.

☆ Filtering & Cleaning: Remove duplicates, spam, broken HTML, and filter biased, copyrighted, or inappropriate content.

☆ Dataset Structuring: Tokenize text using BPE, SentencePiece, or Unigram; add metadata like source, timestamp, and quality rating.

✸ 2️⃣ Preprocessing & Tokenization

☆ Tokenization: Convert text into numerical tokens using SentencePiece or GPT’s BPE tokenizer.

☆ Data Formatting: Structure datasets into JSON, TFRecord, or Hugging Face formats; use Sharding for parallel processing.

✸ 3️⃣ Model Architecture & Pretraining

☆ Architecture Selection: Choose a Transformer-based model (GPT, T5, LLaMA, Falcon) and define parameter size (7B–175B).

☆ Compute & Infrastructure: Train on GPUs/TPUs (A100, H100, TPU v4/v5) with PyTorch, JAX, DeepSpeed, and Megatron-LM.

☆ Pretraining: Use Causal Language Modeling (CLM) with Cross-Entropy Loss, Gradient Checkpointing, and Parallelization (FSDP, ZeRO).

☆ Optimizations: Apply Mixed Precision (FP16/BF16), Gradient Clipping, and Adaptive Learning Rate Schedulers for efficiency.

✸ 4️⃣ Model Alignment (Fine-Tuning & RLHF)

☆ Supervised Fine-Tuning (SFT): Train on high-quality human-annotated datasets (InstructGPT, Alpaca, Dolly).

☆ Reinforcement Learning from Human Feedback (RLHF): Generate responses, rank outputs, train a Reward Model (PPO), and refine using Proximal Policy Optimization (PPO).

☆ Safety & Constitutional AI: Apply RLAIF, adversarial training, and bias filtering.

✸ 5️⃣ Deployment & Optimization

☆ Compression & Quantization: Reduce model size with GPTQ, AWQ, LLM.int8(), and Knowledge Distillation.

☆ API Serving & Scaling: Deploy with vLLM, Triton Inference Server, TensorRT, ONNX, and Ray Serve for efficient inference.

☆ Monitoring & Continuous Learning: Track performance, latency, and hallucinations;

✸ 6️⃣Evaluation & Benchmarking

☆ Performance Testing: Validate using HumanEval, HELM, OpenAI Eval, MMLU, ARC, and MT-Bench.
≣≣≣≣≣≣≣≣≣≣≣≣≣≣≣≣≣≣≣≣≣≣≣≣≣≣

https://t.iss.one/DataScienceM ⭐️

html-to-markdown

A modern, fully typed Python library for converting HTML to Markdown. This library is a completely rewritten fork of markdownify with a modernized codebase, strict type safety and support for Python 3.9+.

Features:
⭐️ Full HTML5 Support: Comprehensive support for all modern HTML5 elements including semantic, form, table, ruby, interactive, structural, SVG, and math elements
⭐️ Enhanced Table Support: Advanced handling of merged cells with rowspan/colspan support for better table representation
⭐️ Type Safety: Strict MyPy adherence with comprehensive type hints
Metadata Extraction: Automatic extraction of document metadata (title, meta tags) as comment headers
⭐️ Streaming Support: Memory-efficient processing for large documents with progress callbacks
⭐️ Highlight Support: Multiple styles for highlighted text (<mark> elements)
⭐️ Task List Support: Converts HTML checkboxes to GitHub-compatible task list syntax

nstallation

pip install html-to-markdown

Optional lxml Parser
For improved performance, you can install with the optional lxml parser:

pip install html-to-markdown[lxml]

The lxml parser offers:

🆘 ~30% faster HTML parsing compared to the default html.parser
🆘 Better handling of malformed HTML
🆘 More robust parsing for complex documents

Quick Start
Convert HTML to Markdown with a single function call:

from html_to_markdown import convert_to_markdown

html = """
<!DOCTYPE html>
<html>
<head>
    <title>Sample Document</title>
    <meta name="description" content="A sample HTML document">
</head>
<body>
    <article>
        <h1>Welcome</h1>
        <p>This is a <strong>sample</strong> with a <a href="https://example.com">link</a>.</p>
        <p>Here's some <mark>highlighted text</mark> and a task list:</p>
        <ul>
            <li><input type="checkbox" checked> Completed task</li>
            <li><input type="checkbox"> Pending task</li>
        </ul>
    </article>
</body>
</html>
"""

markdown = convert_to_markdown(html)
print(markdown)

Working with BeautifulSoup:
If you need more control over HTML parsing, you can pass a pre-configured BeautifulSoup instance:

from bs4 import BeautifulSoup
from html_to_markdown import convert_to_markdown

# Configure BeautifulSoup with your preferred parser
soup = BeautifulSoup(html, "lxml")  # Note: lxml requires additional installation
markdown = convert_to_markdown(soup)

Github: https://github.com/Goldziher/html-to-markdown

https://t.iss.one/DataScienceN ⭐️

LangExtract

A Python library for extracting structured information from unstructured text using LLMs with precise source grounding and interactive visualization.

GitHub: https://github.com/google/langextract

https://t.iss.one/DataScience4 🖕

This channels is for Programmers, Coders, Software Engineers.

0️⃣ Python
1️⃣ Data Science
2️⃣ Machine Learning
3️⃣ Data Visualization
4️⃣ Artificial Intelligence
5️⃣ Data Analysis
6️⃣ Statistics
7️⃣ Deep Learning
8️⃣ programming Languages

✅ https://t.iss.one/addlist/8_rRW2scgfRhOTc0

✅ https://t.iss.one/Codeprogrammer

✅ open-source alternative to Perplexity.

✅ Real-time web search with Firecrawl API
✅ Advanced answers with GPT-4o-mini
✅ Every sentence with reference and source
✅ Automatic stock display with TradingView


┌ 🔍 Fireplexity
├ 🥵 Website
└ 🐱 GitHub-Repos

https://t.iss.one/DataScienceN 🌟

🧱 AI now generates worlds in the style of Minecraft — presenting the GameFactory model

Researchers trained the model on 70 hours of Minecraft gameplay and achieved impressive results: 
GameFactory can create procedural game worlds — from volcanoes to cherry blossom forests, just like in the iconic simulator.

🔥 Want your own endless world? Just set the parameters.

🟠 Examples and code — at the link: https://yujiwen.github.io/gamefactory/

🟠Github: https://github.com/KwaiVGI/GameFactory

https://t.iss.one/DataScienceN 🌟

python-docx: Create and Modify Word Documents #python

python-docx is a Python library for reading, creating, and updating Microsoft Word 2007+ (.docx) files.

Installation

pip install python-docx

Example

from docx import Document

document = Document()
document.add_paragraph("It was a dark and stormy night.")
<docx.text.paragraph.Paragraph object at 0x10f19e760>
document.save("dark-and-stormy.docx")

document = Document("dark-and-stormy.docx")
document.paragraphs[0].text
'It was a dark and stormy night.'

https://t.iss.one/DataScienceN 🚗

Data scientists, this is for you — I dug up LeetCode for DS

DataLemur — a powerful platform that collects real interview problems from Tesla, Facebook, Twitter, Microsoft, and other top companies

Inside: practical tasks on SQL, statistics, Python, and ML. You can filter by difficulty level and company

Top-notch for those preparing for interviews for Data Scientist / Data Analyst roles. Get it here 🍯

👉 https://t.iss.one/DataScienceN 👍