🌟 MiraData: Large, long-duration video dataset with structured annotations.

When training generative models, the training dataset plays an important role in the quality of reference of ready-made models.
One of the good sources can be MiraData from Tencent - a ready-made dataset with a total video duration of 16 thousand hours, designed for training models for generating text in videos. It includes long videos (average 72.1 seconds) with high motion intensity and detailed structured annotations (average 318 words per video).

To assess the quality of the dataset, a system of MiraBench benchmarks was even specially created, consisting of 17 metrics that evaluate temporal consistency, movement in the frame, video quality, and other parameters. According to their results, MiroData outperforms other well-known datasets available in open sources, which mainly consist of short videos with floating quality and short descriptions.

🟡 Project page
🟡 Arxiv
🤗 Hugging Face
🖥 GitHub

#Text2Video #Dataset #ML

https://t.iss.one/DataScienceT ⭐️

🌟 Mamba Vision: An effective alternative to transformers for computer vision

Mamba Vision is an implementation of the Mamba architecture using Selective State Space Models (SSM) in image processing from Nvidia Lab.

MambaVision demonstrates more efficient use of computing resources compared to traditional transformer-based architectures (VIT and Swin), and the use of SSM opens up new ways of extracting and processing visual features. The proposed architecture shows good scalability, maintaining efficiency as the model size increases.
MambaVision is applicable to a variety of computer vision tasks, including image classification and semantic segmentation.

The project is in its early stages and its effectiveness on real-world computer vision tasks has yet to be fully assessed.
At the moment, it has only been used in the image classification task.

🖼 Family of MambaVision Pretrained (ImageNet-1K) models (direct download from Google Drive):

MambaVision-T (32M)
MambaVision-T2 (35M)
MambaVision-S (50M)
MambaVision-B (98M)
MambaVision-L (228M)
MambaVision-L2 (241M)

⚠️ Licensing:

For non-commercial projects: CC-BY-NC-SA-4.0
For commercial use: request via form

🖥 Github
🟡 Arxiv

#MambaVision #ML

https://t.iss.one/DataScienceT ⭐️

🌟 DG-Mesh: Constructing high-quality polygonal meshes from monocular video.

DG-Mesh reconstructs a high-quality dynamic 3D vertex-matched mesh from monocular video. The pipeline uses 3D Gaussian wavelets to represent dynamic scenes and differentiable algorithms to construct polygons.

DG-Mesh allows you to track the movement of vertices, simplifying the texturing of dynamic objects.
The method is memory efficient and fully differentiable, allowing optimization of the target object's 3D mesh directly.

The Github repository contains code for local training using datasets:

- D-NeRF
- DG-Mesh
- NeuralActor
- Custom dataset , shot on Iphone 14 Pro and processed in Record3D, RealityCheck and masked in DEVA.

🖥 Local launch:

conda create -n dg-mesh python=3.9
conda activate dg-mesh
conda install pytorch torchvision torcaudio pytorch-cuda=11.8 -c pytorch -c nvidia

#Install nvdiffrast
pip install git+https://github.com/NVlabs/tiny-cuda-nn#subdirectory=bindings/torch
pip install git+https://github.com/NVlabs/nvdiffrast/

# Install pytorch3d
export FORCE_CUDA=1
conda install -c fvcore -c iopath -c conda-forge fvcore iopath -y
pip install "git+https://github.com/facebookresearch/pytorch3d.git"

# Clone this repository
git clone https://github.com/Isabella98Liu/DG-Mesh.git
cd DG-Mesh

# Install submodules
pip install dgmesh/submodules/diff-gaussian-rasterization
pip install dgmesh/submodules/simple-knn

# Install other dependencies
pip install -r requirements.txt

🟡 Project page
🖥 GitHub
🟡 Arxiv

#Video2Mesh #3D #ML #NeRF

https://t.iss.one/DataScienceT ⭐️