LLMs Evolve Beyond Function Approximation to True Computation
TL;DR. New insights clarify that large language models are transitioning from simple function approximators to systems capable of inherent algebraic compositionality and general computation. - Early AI models, like Word2Vec, struggled with variable-size inputs and lacked true compositional power, limiting their computational scope. - Transformers addressed these limitations by enabling representations that scale with input, facilitating more complex and abstract operations within LLMs. - This redefinition positions LLMs as extending traditional computational models into informal language rules, blurring the lines of strict logical definitions.
- LLMs are increasingly demonstrating capabilities beyond mere function approximation, moving towards genuine computation.
- The advancement from fixed-size input limitations to variable-size handling via Transformers unlocked significant computational potential in neural networks.
- The computational nature of LLMs can be understood as an extension of formal language rules to informal language, challenging classical definitions of computation.
Sources
- AI Is Computation — gabrielpickard.com