18 years ago we taught a petri-dish of neurons to fly an F-22

Scientists took embryonic cortical hemispheres from rats, dissolved the connective tissue, then deposited a solution of neurons and glial cells on an array of micro-electrodes, and connected it to a simulation.

Paper

Auto-Regressive Next-Token Predictors are Universal Learners

Summary: “This new paper trains extremely simple linear(!) and shallow MLP networks to get competitive ppl on language modeling and 4 digit multiplication tasks! The claim seems to be that much of the magic is in auto-regressive objective, not the architecture.”

GREAT: A measure of task difficulty based on minimum-needed pondering length: “We introduce a new complexity measure -- length complexity -- which measures the number of intermediate tokens in a CoT sequence required to approximate some target function, and analyze the interplay between length complexity and other notions of complexity.”

“Finally, we show experimentally that simple next-token predictors, such as linear networks and shallow Multi-Layer Perceptrons (MLPs), display non-trivial performance on text generation and arithmetic tasks. Our results demonstrate that the power of language models can be attributed, to a great extent, to the auto-regressive next-token training scheme, and not necessarily to a particular choice of architecture.”

= Obvious to anyone with a brain for a long time, as its directly analogous to the tape-usage vs simplicity tradeoff well-known to exist in simple Turing machines since the 1930s.

Unfortunately many involved in AI don’t seem to have a brain, so great that someone finally did a paper on this. (Though FWIW, do recall a paper from last year that also exhaustively demonstrated this, but was rejected from conferences for dumb reasons.)

= If you’re still telling the AI to respond as briefly as possible, when asking hard questions, without giving it any space to think before answering — then you’re a confirmed moron.

Let the AIs think!

BTW: “much of the magic is in auto-regressive objective, not the architecture” = “more resources is all you need” = Bitter Lesson confirmed yet again.

Arxiv Link

Visual example of thinking-length vs accuracy tradeoff

See how their very simple MLP model beats both GPT-3.5 and GPT-4 on a simple multiplication question (1394x8618=),

-- in large part due to their simple MLP model spending a far longer time thinking through the answer to the question in its output. See above how much longer the MLPs output is.

This also somewhat explains the drop in quality on OpenAI's models as they've “upgraded” their models to give terser outputs.

Many problems are simply too hard to instantly know the answer to, but can be trivially figured out given enough time to think.

Let the AIs think!

Arxiv Link

Bard knows

8====D~ ({})

JapanGPT: An LLM to reflect Japanese values

“Certainly Japanese LLMs are getting much better, but they are far behind GPT-4,” says Passaglia, a physicist at the University of Tokyo who studies Japanese language models. But there is no reason in principle, he says, that a Japanese LLM couldn’t equal or surpass GPT-4 in future. “This is not technically insurmountable, but just a question of resources.”

Article