🔥 Data Pyramid for Embodied Manipulation

💡 The paper proposes a data pyramid framework for embodied manipulation, which organizes data sources into five complementary categories: real-robot data, UMl-style data, egocentric and exocentric data, simulation data, and general vision-language data. The framework is designed to address the challenges of embodied agents, which require data that couples observations with physical states and actions. The authors analyze recent embodied foundation models through the lens of their data recipes, examining how different sources are selected, aligned, and mixed during training. They relate data composition to capabilities in perception, reasoning, planning, action generation, and world prediction. The paper also discusses six open challenges, including building large-scale tactile datasets, collecting failure and recovery data, developing scalable data-collection pipelines, aligning actions across embodiments, leveraging egocentric data for dexterous manipulation, and designing principled data recipes for robot learning. The proposed data pyramid framework provides a foundation for the design of next-generation embodied systems, which can learn to perceive, reason, and act in complex environments. The authors hope that this work will pave the way for the development of more advanced embodied agents that can learn from diverse data sources and improve their performance in various tasks.

📅 Published on Jul 27

🔗 Links:
• GitHub: https://github.com/huggingface
• arXiv: https://arxiv.org/abs/2607.24744
• PDF: https://arxiv.org/pdf/2607.24744
• Project Page: https://jasper-aaa.github.io/embodied-data-pyramid/

━━━━━━━━━━━━━━━━━━━━━━━━
📢 By: https://t.iss.one/PaperNexus

#EmbodiedManipulation #RobotLearning #MultimodalData #EmbodiedAI #FoundationModels

🔥 OmniVAE: An Audio-Video VAE with Cross-Modal Alignment for Joint Generation

💡 The paper introduces OmniVAE, a joint audio-video variational autoencoder that learns fine-grained semantic alignment between audio and video latent representations. Recent generative models have moved beyond silent video or standalone audio synthesis towards joint generation of synchronized audio and video, but this remains challenging due to the fundamental structural differences between the two modalities. Most existing methods use audio and video VAEs trained separately, resulting in a lack of cross-modal alignment, which leaves the downstream generative model to learn cross-modal synchronization from scratch.

OmniVAE addresses this issue by jointly training an audio-video VAE that captures temporal-semantic correspondence and aligns the two latent spaces. It uses a segment-level audio-video contrastive objective to achieve this alignment and also distills features from pre-trained modality-specific semantic encoders into each modality, improving the downstream learnability of both latent spaces.

The results of extensive experiments show that OmniVAE consistently improves the learnability of the latent spaces, translating into higher generation quality and more accurate cross-modal synchronization in downstream text-to-audio-video generation tasks. The findings underscore the importance of learning unified representations as a foundation for omnimodal modeling. Overall, OmniVAE provides a novel approach to joint audio-video generation, enabling more effective and synchronized generation of audio and video content.

📅 Published on Jul 26

🔗 Links:
• GitHub: https://github.com/huggingface
• arXiv: https://arxiv.org/abs/2607.23855
• PDF: https://arxiv.org/pdf/2607.23855
• Project Page: https://openmoss.ai/OmniVAE.github.io/

🤖 Models citing this paper:
• https://huggingface.co/OpenMOSS-Team/OmniVAE

━━━━━━━━━━━━━━━━━━━━━━━━
📢 By: https://t.iss.one/PaperNexus

#AudioVideoGeneration #CrossModalAlignment #VariationalAutoencoders #MultimodalLearning #JointGenerationModels

🔥 GNM Head: A Generative aNthropometric Model of the human head

💡 The paper introduces a new parametric model called the Generative Anthropometric Model of the human head, or GNM Head. The model is designed to address the limitations of existing publicly available models, which typically only capture the outer geometry of the head and ignore internal structures such as the eyes and mouth. These existing models also often suffer from reduced geometric quality due to low-fidelity input data sets.

The GNM Head model is built on an extensive database of high-resolution 3D scans combined with high-quality anatomy-specific artist-made samples. The model encompasses the head, face, neck, eyeballs, teeth, and tongue, and includes specialized sub-models for the ocular and intra-oral structures.

The paper details the data provenance, model architecture, and performance of the GNM Head model, including its ability to fit target 3D face scans. The results show that the GNM Head model is a significant improvement over existing models, offering a more comprehensive and accurate representation of the human head.

To foster community innovation, the complete GNM framework is made publicly available. The GNM Head model has the potential to serve as a crucial conditioning signal within generative large vision models, allowing for tight spatial control of generated imagery. Overall, the paper contributes a new and improved parametric model of the human head, which can be used in a variety of applications, including computer vision, graphics, and animation.

📅 Published on Jul 26

🔗 Links:
• GitHub: https://github.com/huggingface
• arXiv: https://arxiv.org/abs/2607.23687
• PDF: https://arxiv.org/pdf/2607.23687

━━━━━━━━━━━━━━━━━━━━━━━━
📢 By: https://t.iss.one/PaperNexus

#GenerativeModeling #AnthropometricAnalysis #3DHeadModeling #FacialReconstruction #HumanComputerInteraction

🔥 Harness Handbook: Making Evolving Agent Harnesses Readable,Navigable, and Editable

💡 The paper discusses the challenges of evolving agent harnesses, which are critical components of modern AI systems. As models, APIs, environments, and requirements change, harnesses must be continuously modified. However, this is difficult because production harnesses are large, tightly coupled, and behaviorally distributed, making it hard to identify the code locations that implement specific behaviors.

The authors introduce the Harness Handbook, a behavior-centric representation that is synthesized automatically from a harness code base via static analysis and LLM-assisted structuring. The Handbook links each behavior to its corresponding source code, making it easier to navigate and edit the harness.

The authors also introduce Behavior-Guided Progressive Disclosure BGPD, a method that guides agents from high-level behaviors to relevant implementation details and verifies candidate locations against the current source code.

The results show that Handbook-Assisted planning improves behavior localization and edit-plan quality while using fewer planner tokens. The largest gains are seen on scattered sites, rarely executed paths, and cross-module interactions. The paper concludes that evolving complex agent systems depends not only on generating edits but also on determining where those edits should be made.

Overall, the paper provides a solution to the problem of evolving agent harnesses by introducing a behavior-centric representation and a method for guiding agents to relevant implementation details. The results demonstrate the effectiveness of the proposed approach in improving behavior localization and edit-plan quality.

📅 Published on Jul 14

🔗 Links:
• GitHub: https://github.com/huggingface
• arXiv: https://arxiv.org/abs/2607.13285
• PDF: https://arxiv.org/pdf/2607.13285
• Project Page: https://ruhan-wang.github.io/Harness-Handbook/

━━━━━━━━━━━━━━━━━━━━━━━━
📢 By: https://t.iss.one/PaperNexus

#EvolvingAgentHarnesses #ArtificialIntelligenceSystems #HarnessHandbook #BehaviorCentricDesign #AIModelMaintenance

🔥 Mage-VL: An Efficient Codec-Native Streaming Multimodal Foundation Model

💡 The paper presents Mage-VL, an efficient codec-native streaming multimodal foundation model for real-time multimodal understanding and interaction. Standard vision-language models suffer from Moravec's paradox, exceling at complex offline visual reasoning but struggling with simple streaming perception tasks and processing them inefficiently. To address this, the authors propose a custom tokenizer, Mage-ViT, which replaces uniform frame sampling with selective encoding of dynamic, entropy-rich regions using motion vectors and residual energy across sparse anchor and predicted frames. This approach reduces visual token consumption by over 75 percent while preserving spatiotemporal context.

Mage-ViT is trained from scratch on approximately 560 million unlabeled images and 100 million unlabeled video frames, and it matches or outperforms flagship encoders trained on billions of image-text pairs. The authors also establish AI4AI data pipelines encompassing prompt-code joint optimization for multimodal captioning and AI-driven performance diagnosis to guide training recipes.

Furthermore, through a bio-inspired dual-system architecture, consisting of a lightweight System 1 event gate and a causal System 2 decoder, Mage-VL enables proactive streaming perception. Extensive evaluations show that Mage-VL-4B matches Qwen3-VL-4B on static tasks while achieving strong gains in video understanding and 2D/3D spatial reasoning, with up to a 3.5 times wall-clock inference speedup, and comprehensively surpasses the 15B Phi-4-reasoning-vision baseline.

The paper delivers seven key empirical findings covering pre-training data efficiency, variable-resolution scaling, codec system acceleration, Video QA SF redundancy, motion-spatial synergy, AI4AI data pipelines, and Zero-Vision SFT for multimodal RL. Overall, the paper presents a novel approach to multimodal understanding and interaction, with significant improvements in efficiency, accuracy, and speed.

📅 Published on Jul 27

🔗 Links:
• GitHub: https://github.com/huggingface
• arXiv: https://arxiv.org/abs/2607.24904
• PDF: https://arxiv.org/pdf/2607.24904
• Project Page: https://microsoft.github.io/Mage

🤖 Models citing this paper:
• https://huggingface.co/microsoft/Mage-VL

🚀 Spaces citing this paper:
• https://huggingface.co/spaces/microsoft/mage-vl-demo

━━━━━━━━━━━━━━━━━━━━━━━━
📢 By: https://t.iss.one/PaperNexus

#MultimodalFoundationModels #CodecNativeStreaming #VisionLanguageModels #RealTimeMultimodalUnderstanding #EfficientMultimodalProcessing

🔥 Pass the Baton: Trajectory-Relayed On-Policy Distillation

💡 The paper addresses the issue of prefix failure in on-policy distillation, where the student model commits to a wrong reasoning direction and subsequent generations build on this deviation, producing misguided continuations. The authors identify a teacher-student continuation asymmetry on failed prefixes, where the teacher tends to redirect while the student continues along the original direction. They propose Relay On-Policy Distillation, which converts this asymmetry into a label-free handoff trigger.

During training, Relay On-Policy Distillation constructs relay trajectories by letting the teacher briefly take over at detected trigger points to produce a teacher leg, after which the student resumes and is optimized on the resulting trajectory. A limited relay budget concentrates intervention on critical early positions while limiting departure from the student policy.

The method is evaluated on eight mathematical reasoning benchmarks with a Qwen 3-4B-Instruction-2507 teacher and Qwen 3-0.6B/1.7B-Non-Thinking students. The results show that Relay On-Policy Distillation achieves the best or second-best results on every benchmark, outperforming standard on-policy distillation by 5.73 percent and the strongest baseline Fast On-Policy Distillation by 1.49 percent on average for 1.7B, with consistent gains at 0.6B. Additionally, the training trajectory length is reduced by over 50 percent.

Overall, the paper proposes a novel method to address prefix failure in on-policy distillation, achieving significant improvements in performance and efficiency.

📅 Published on Jul 28

🔗 Links:
• GitHub: https://github.com/huggingface
• arXiv: https://arxiv.org/abs/2607.26057
• PDF: https://arxiv.org/pdf/2607.26057
• Project Page: https://zju-real.github.io/Relay-OPD/

━━━━━━━━━━━━━━━━━━━━━━━━
📢 By: https://t.iss.one/PaperNexus

#OnPolicyDistillation #TrajectoryRelayedLearning #PrefixFailureMitigation #TeacherStudentContinuationAsymmetry #RelayOnPolicyDistillation

🔥 MODUS: Decoder-Only Any-to-Any Modeling of Diverse Modalities

💡 The paper introduces MODUS, a novel any-to-any modeling approach that enables the prediction of any modality from any combination of others within a single network. Existing any-to-any models are typically trained from scratch using encoder-decoder or diffusion architectures, which can impact their performance and prevent them from leveraging strong pre-trained decoder-only models. In contrast, MODUS treats all modalities symmetrically and supports arbitrary modalities as inputs and outputs without modality-specific heads, losses, or task pipelines. This is achieved through a decoder-only architecture that allows every modality to be both an input and an output of the same model. The resulting model, MODUS, can support a range of applications, such as chained generation through intermediate modalities or cross-modal self-verification by scoring the model's own outputs with another generated modality. The authors demonstrate that MODUS exhibits strong out-of-the-box performance and is competitive with specialist and multi-task baseline models using a single model across various benchmarks. Overall, MODUS provides a flexible and effective approach to any-to-any modeling, and all materials are open-sourced for further research and development.

📅 Published on Jul 28

🔗 Links:
• GitHub: https://github.com/huggingface
• arXiv: https://arxiv.org/abs/2607.25948
• PDF: https://arxiv.org/pdf/2607.25948
• Project Page: https://modus-multimodal.epfl.ch/

📊 Datasets citing this paper:
• https://huggingface.co/datasets/epfl-vilab-modus/MODUS-15Modality

━━━━━━━━━━━━━━━━━━━━━━━━
📢 By: https://t.iss.one/PaperNexus

#MultimodalLearning #DecoderOnlyArchitectures #AnyToAnyModeling #ModalityTransfer #CrossModalPrediction

🔥 ReDesign: Recovering Editable Design Structures from Images via Agentic Decomposition

💡 The paper ReDesign presents a novel approach to recovering editable design structures from images, a common and costly bottleneck in modern design workflows. The problem is challenging because it requires recovering multiple attributes such as typography, vector geometry, colors, grouping, and layer ordering. The proposed method, ReDesign, uses an agentic framework that grows an editable layer hierarchy by selecting and composing specialized tools across modalities. To ensure reliability despite imperfect tool outputs, the framework introduces a verification mechanism at each expansion step, providing local accept, prune, or retry feedback that prevents error accumulation and avoids large-scale reruns.

The authors evaluate the method's editability at scale using the Figma Edit Replay Benchmark, consisting of 909 raw Figma files and 14796 controlled edit instructions that replay edits on reconstructed outputs. The results show that ReDesign achieves strong visual fidelity while delivering the highest editability across layout, color, and text edits, outperforming layered decomposition baselines and serial tool use pipelines. The paper's contributions include the introduction of the ReDesign framework, the Figma Edit Replay Benchmark, and the demonstration of the method's effectiveness in recovering editable design structures from images. Overall, the paper presents a significant advancement in the field of design recovery and editing, with potential applications in various design workflows.

📅 Published on Jul 28

🔗 Links:
• GitHub: https://github.com/huggingface
• arXiv: https://arxiv.org/abs/2607.25565
• PDF: https://arxiv.org/pdf/2607.25565
• Project Page: https://jintae-00.github.io/ReDesign/

📊 Datasets citing this paper:
• https://huggingface.co/datasets/Jintae-Park/ReDesign-Figma909

━━━━━━━━━━━━━━━━━━━━━━━━
📢 By: https://t.iss.one/PaperNexus

#ComputerVision #GraphicDesignAutomation #ImageProcessing #VectorGraphicsRecovery #DesignStructureExtraction

🔥 CodeNib: A Multi-View Data System for Serving Repository Context to Coding Agents

💡 The paper introduces CodeNib, a multi-view data system designed to provide repository context to coding agents. The problem addressed is that coding agents repeatedly search, navigate, and retain context from evolving repositories, but disconnected indexes, language servers, and task-local histories force repeated discovery and obscure life cycle costs. CodeNib builds reusable lexical, dense, and structural views per repository commit, maps outputs to repository-relative source ranges, maintains selected views across edits, and serves ranked search, symbol navigation, and bounded context through one runtime.

The method involves creating a system that can efficiently serve context to coding agents by building and maintaining multiple views of the repository. The system is evaluated across 100 snapshots, mapping quality-cost frontiers across the repository-context life cycle. The results show that when outputs match an independent rebuild, graph and vector updates are 8.7 times and 25.4 times faster at the median.

The system is also evaluated on a static-navigation subset matching normalized live-server locations, where the median per-request live/static latency ratio is 4.7 times. Additionally, the results show that selected context policies preserve localization with 50-87 percent fewer trajectory tokens than paired grep/read. Overall, the results support multi-view repository-context serving with explicit, operation-specific validity boundaries, providing a solution to the problem of efficiently serving context to coding agents.

📅 Published on Jul 28

🔗 Links:
• GitHub: https://github.com/huggingface
• arXiv: https://arxiv.org/abs/2607.25431
• PDF: https://arxiv.org/pdf/2607.25431
• Project Page: https://codenib.ai

📊 Datasets citing this paper:
• https://huggingface.co/datasets/sysevol-ai/codenib-synthesis

━━━━━━━━━━━━━━━━━━━━━━━━
📢 By: https://t.iss.one/PaperNexus

#CodeAnalysisTools #MultiViewDataSystems #CodingAgents #RepositoryContext #SoftwareDevelopmentTools

🔥 TurboVLA: Real-Time Vision-Language-Action Model at 32 Hz on an RTX 4090 with <1 GB VRAM

💡 The paper introduces TurboVLA, a new vision-language-action model that achieves real-time performance at 32 Hz on an RTX 4090 with less than 1 GB VRAM. The conventional approach to vision-language-action models involves using a large language model as the central interface between perception and action, which incurs substantial computational and memory overhead. In contrast, TurboVLA reformulates the conventional pathway as a direct vision-language-action mapping, where visual observations and language instructions are independently encoded and directly exchanged through lightweight bidirectional vision-language interaction. This simplified design constructs task-conditioned representations directly from visual and linguistic features, significantly reducing computational and memory costs.

The TurboVLA model achieves 97.7 percent average success with only 0.2 billion parameters, 31.2 ms inference latency, and 0.9 GB inference VRAM on a consumer-grade RTX 4090, matching or outperforming substantially larger vision-language-action policies. The results establish TurboVLA as a simple and effective alternative to the prevailing language-centric vision-language-action paradigm, offering a new perspective on how vision, language, and action can be connected for efficient robotic manipulation. The code for TurboVLA is available online, making it accessible for further research and development. Overall, the paper presents a significant contribution to the field of vision-language-action modeling, demonstrating the potential for real-time and efficient robotic manipulation using a novel and simplified approach.

📅 Published on Jul 29

🔗 Links:
• GitHub: https://github.com/huggingface
• arXiv: https://arxiv.org/abs/2607.27205
• PDF: https://arxiv.org/pdf/2607.27205
• Project Page: https://h-embodvis.github.io/TurboVLA/

━━━━━━━━━━━━━━━━━━━━━━━━
📢 By: https://t.iss.one/PaperNexus

#VisionLanguageAction #RealTimeAI #EfficientDeepLearning #VisionLanguageInteraction #LowMemoryComputing