Some guys seriously thinking ChatGPT can decode LZ4 compression 😂

But is it as crazy as it sounds? Mixed.

Gwern’s adds:

“A fun idea, but it seems like it ought to backfire: a Transformer model is feedforward, so it only has so many layers to 'think'; if it's spending those layers decoding compressed strings (which is amazing if it can), it doesn't have any 'time' to think about whatever abstract form it decodes the inputs to, and it definitely doesn't have time to combine a de facto huge context window to reason over it.
Now, a more interesting idea might be to see if you can train a Transformer on compressed strings to make it native, and maybe tradeoff some width for depth. There's some slight analogies in image generation to training on Fourier components (like JPEG) rather than pixels. But AFAIK no one has experimented with heavy duty compression inputs for regular natural language generation.”