Google presents NATURAL PLAN: Benchmarking LLMs on Natural Language Planning

Presents a realistic planning benchmark with Trip Planning, Meeting Planning, and Calendar Scheduling, which is surprisingly challenging for SotA models

GOLD project is a groundbreaking collaboration between MintNeuro, Amber Therapeutics, and Imperial College London.

This £800k Innovate UK-funded initiative marks another significant leap towards mission to revolutionise neural implants with advanced semiconductor technology.

With GOLD, researchers are addressing the challenges of current implantable neurotechnologies by creating modular, medical-grade chips that minimise size and power consumption, ultimately reducing patient risk.

Omega is a plugin that uses LLM to assist users with bioimage processing and analysis tasks

The instructions for using #Omega are available on GitHub.

Check the high-resolution videos here.

#Omega is a first step towards the vision of turning bioimage analysis into a true dialogue between mind and machine.

Omega can perform tasks like processing images, segmenting cell nuclei, counting them, creating detailed reports, controlling the viewer parameters, downloading image files, and more — all through natural language commands!

Imagine asking #Omega to "segment cell nuclei in the selected image" and following up with post-analysis that would have otherwise required code! It can handle both 2D and 3D images with ease leveraging popular packages such as #StarDist or #CellPose. Here’s an example in 2D.

#Omega can segment nuclei in 3D images too! Watch it in action.

#Omega can devise step-by-step strategies for image processing tasks.

One of #Omega’s standout features is its ability to create custom widgets on demand. For instance, it can generate a widget to filter segments based on their area, tailored to your specific needs.

#Omega can create complex widgets, like this one for max color projection of a 3D stack. The hue variation represents the depth within the stack.

The AI-augmented code editor in #Omega allows for easy reuse and modification of generated code. It supports code highlighting, completion, and even automatic error fixing.

#Omega’s code editor also allows sharing code and widgets across machines on the same network, facilitating collaboration.

#Omega can ‘see’! Using advanced vision capabilities, Omega can interpret images directly within the napari viewer, enhancing its ability to decide the best approach for image processing tasks like segmentation.

Your Language Agents Already Know How to Achieve High-level Goals

SelfGoal, a framework to enhance an LLM-based agent's capabilities to achieve high-level goals.

LLMs can perform well at basic tasks but struggle to achieve high-level goals without access to detailed instructions or retraining.

The proposed approach adaptively breaks down a high-level goal into a tree structure of practical subgoals during interaction with the environment.

Improves performance on various tasks, including competitive, cooperative, and deferred feedback environments.

Overall, when solving complex problems, having the goals represented as an external tree structure helps the LLM agent. This is similar to a previous paper I featured where an external dynamic buffer of thought templates was used instead to aid the LLM in solving harder problems.

🚀 ChatGPT 4o is going to be integrated into Apple Intelligence. When the system knows you need a large language model that’s more current and has more info it will go out to ChatGPT.

Andrej Karpathy, co-founder of OpenAI: we're quickly heading into a world where you can open up your phone and just say stuff. It talks back and it knows you. And it just works. Super exciting and as a user, quite looking forward to it.

Actually, really liked the Apple Intelligence announcement. It must be a very exciting time at Apple as they layer AI on top of the entire OS. A few of the major themes.

Step 1 Multimodal I/O. Enable text/audio/image/video capability, both read and write. These are the native human APIs, so to speak.

Step 2 Agentic. Allow all parts of the OS and apps to inter-operate via "function calling"; kernel process LLM that can schedule and coordinate work across them given user queries.

Step 3 Frictionless. Fully integrate these features in a highly frictionless, fast, "always on", and contextual way. No going around copy pasting information, prompt engineering, or etc. Adapt the UI accordingly.

Step 4 Initiative. Don't perform a task given a prompt, anticipate the prompt, suggest, initiate.

Step 5 Delegation hierarchy. Move as much intelligence as you can on device (Apple Silicon very helpful and well-suited), but allow optional dispatch of work to cloud.

Step 6 Modularity. Allow the OS to access and support an entire and growing ecosystem of LLMs (e.g. ChatGPT announcement).

Step 7 Privacy.

Apple strongly advocated for on-device AI

Today, we know they will run ~3B LLM on device (Mac, iPhone, iPad) using fine-tuned LoRA Adapters for different tasks, claiming to outperform other 7B and 3B LLMs!

What we know:

📱 On-device 3B with grouped-query-attention, activation and embedding quantization running on the neural engine

📊 iPhone 15 Pro 0.6 ms time-to-first-token with 30 tokens/second latency

❌ Server model size and details are unknown

🔄 Dynamically load, cache, and swap LoRA adapter models as needed

📈 On-device model 49K vocab size, while the server model 100K

🛠️ Used Rejection Sampling and descent policy optimization and a leave-one-out advantage for RLHF

📝 Used synthetic data generation for tasks like summaries

🔍 750 evaluation samples for each production use case

🇺🇸 Seems to be an English model, this would explain the limitation to US only

🧠 Used Apple's AXLearn framework (JAX) and FSP to train on TPUs and GPUs

⚡ 3B + Adapter outperforms Phi-3 mini on summarization

🏆 3B + Adapter achieves 78.7% on IFEval > Mistral 7B; Server Model matches GPT-4-Turbo

Apple use fine-tuned adapters for different tasks to outperform much bigger and stronger models.

Sword Health raises $130M and its valuation soars to $3B

Sword health an AI-powered virtual physical therapy startup, has raised $30 million and let employees sell $100 million worth of equity to new and existing investors, including Khosla Ventures.

The round brings the nine-year-old company’s valuation to $3 billion, a 50% increase from the $2 billion value it garnered in its Series D in November 2021.

ARC PRIZE: A $1M+ competition towards open AGI progress.

Fitbit Research Breakthrough!

Google Deepmind released a new study reveals FitBit interventions can not only boost health but also slash healthcare costs for those who are inactive.

By increasing physical activity, these programs help prevent chronic diseases and lead to long-term savings.

Luma AI dropped a Sora-like AI video generator called Dream Machine.

But unlike Sora or KLING, it's completely open access to the public.

Iteratively enhanced LLM outputs outperform OpenAI GPT-4 Omni on AlpacaEval 2.0, MT-Bench, and FLASK

Mixture-of-Agents (MoA) uses multiple LLMs in a layered architecture to iteratively enhance the generation quality.

MoA:
- Select multiple LLMs with varying strengths
- Create a multi-layer architecture where each layer consists of multiple LLMs.
- Each new layer uses the outputs from previous layers as auxiliary information to generate its response.

Insights:

1. MoA models achieve superior performance with a 65.1% score on AlpacaEval 2.0 compared to 57.5% by GPT-4 Omni.
2. LLM tends to generate better responses when presented with outputs from other models.
3. Layered aggregation of outputs from multiple LLMs leads to better responses.
4. Selection of LLMs for each MoA layer is crucial for Performance and Diversity.
5. MoA-Lite variant outperforms GPT-4 Turbo by 4%, being twice as cost-efficient.
6. Aggregation of model responses can lead to a high Time to First Token (TTFT).
7. Could be interesting for synthetic data generation and evaluation.

GitHub.

For folks working on ARC-AGI: here's a repository of sequences of actions taken by a human when solving a task, collected by Simon Strandgaard. Shows different solving strategies, and could serve as inspiration.

Who said just Nvidia was making money from the AI boom?OpenAI has more than doubled its annualized revenue to $3.4 billion in the past six months or so, OpenAI CEO Sam Altman has told staff, a sign that growth in the ChatGPT developer’s business is accelerating despite intensifying competition.

Can LLMs invent better ways to train LLMs?

Sakana AI release DiscoPOP: a new SOTA preference optimization algorithm that was discovered and written by an LLM!

This method leverages LLMs to propose and implement new preference optimization algorithms.

Then train models with those algorithms and evaluate their performance, providing feedback to the LLM. By repeating this process for multiple generations in an evolutionary loop, the LLM discovers many highly-performant and novel preference optimization objectives!

GitHub.

Model.

Researchers from Microsoft, Google, MIT, and Oxford released new research presenting DenseAV.

It's an AI algorithm that learns the meaning of words and sounds by watching videos, and can help better understand animal communication and discover language.

Did you know you can generate high-quality synthetic datasets at home, matching GPT-4 generated datasets?

Self-synthesis is a new method that uses an “empty” user message to generate a large-scale scale instruction dataset using Llama 3 70B. 🔥

Implementation:

1. Select fine-tuned/trained LLM like Llama 3 70B

2. Create left-side truncated templates before the user messages start (<|start_header_id|>user\n<|end_header_id|>)

3. Generate & prompt LLM to generate a synthetic user turn and assistant turn.

4. Filter generated samples using LLM to ensure high quality and diversity

Insights:

- Generated 4 million instruction pairs, filtered to 300k high-quality pairs
- Outperforms or matches other open LLMs fine-tuned on GPT-4 data
- Matches or outperforms Llama-3-8B-Instruct on some benchmarks.
- Used Llama 3 8B to categorize and classify samples and sentence transformers for similarity matching.

It can be used for multi-turn data generation

The prompts used to generate and filter the data are in the paper

Creating 1,000 high-quality & filtered samples cost ~$1.1
The dataset is released under CC BY-NC.

Models and datasets.

Nvidia released Nemotron 340B, an open LLM matching GPT-4
They also released a Technical Report on how they trained it and what's special about it!

Implementation

Pretraining: 2-phase pretraining, first trained on 8T and then continued on 1T higher quality tokens and Instruction data with a steeper slope of learning rate decay.

Fine-tuning: First fine-tuned on 800K coding samples, followed by 200K diverse task samples.

RLHF: Applied Direct Preference Optimization (DPO) followed by Reward-aware Preference Optimization (RPO) on multiple iterations.

Insights

- 98% of data used in post-training was synthetically generated
- Pretraining data: English data (70%), Multilingual data (15%), Source code (15%).
- Trained on 6144 H100 GPUs with 8-way TP, 12-way PP with interleaving and DP to achieve ~42% MFU
- Adjusting data distribution and learning rate decay in the 2 pretraining phase improves model quality.
-Only used 20k Human annotated data mostly for Reward Modeling
- Focused on task diversity, topic diversity and instruction diversity
- Used an iterative approach for “response” generation starting with Mixtral then switching to Nemotron models
- Detailed Synthetic Data pipeline instruction including all prompts to generate data.

It's finally here. Q* rings true. Tiny LLMs are as good at math as a frontier model.

By using the same techniques Google used to solve Go (MTCS and backprop), Llama8B gets 96.7% on math benchmark GSM8K!

That’s better than GPT-4, Claude and Gemini, with 200x less parameters

The World Federation of Exchanges has published a report on 'Demystifying Tokenisation Embracing the Future.'

HIghlights

🔸Potential of DLT and Tokenisation: Distributed ledger technology (#DLT), initially developed for #cryptocurrencies like #Bitcoin, has the potential to transform traditional #finance through tokenisation, which digitizes ownership rights of various assets on a #distributedledger.

🔸Natural Evolution in Finance: #Tokenisation is seen as a natural progression in the financial industry, following the move from paper share certificates to dematerialisation, while retaining core principles like ownership and regulatory compliance.

🔸Benefits of Tokenisation: #Tokenisation can offer a range of benefits across various industries and asset classes. One of the advantages of tokenisation is fractional ownership. Fractional ownership allows multiple investors to own a share of an asset. Fractional ownership also makes these become more easily tradeable and accessible which can lead to increased liquidity

🔸Cautious Perspective on Benefits: While tokenisation offers new opportunities, some benefits such as 24/7 trading and same-day settlement are overemphasized. Challenges include conflicts of interest in disintermediated models and potential issues with market liquidity and trading costs.

🔸Challenges and Barriers: Despite being around for over 15 years, tokenisation hasn't gained widespread adoption in traditional markets due to high transaction volumes, interoperability challenges, high implementation costs, and regulatory uncertainties.

🔸 Critical Examination Needed: The paper concludes that while tokenisation has many potential benefits, they need to be critically examined. Transitioning to a tokenised system requires significant upfront investment without clear benefits in already efficient markets like equity.

Fei Fei Li argues that further advancements in AI will require spatial intelligence; seeing and talking are not enough, AI needs to do in order to learn.