Beyond Next-Token Prediction

Toward Provenance-Aware Autoregressive Learning

Abstract

Modern language models are trained by predicting the next token in a sequence. This objective has proven remarkably effective, producing systems capable of reasoning, programming, translation, and scientific assistance from a single underlying learning rule. Yet the data presented to these systems has already undergone substantial compression. Human knowledge is generated through conversations distributed across people and time, while pretraining corpora largely consist of documents in which those conversations have been flattened into text.

This paper explores an architectural extension that preserves more of the original learning process without abandoning autoregressive training. The prediction target remains the next token, but every token is conditioned on the source and timestamp of the Thought to which it belongs. Rather than modeling isolated documents, the model learns over temporally grounded discourse. After pretraining, the same conversational representation continues through FourThought, allowing the model to participate in, predict, and continually learn from ongoing civic dialogue rather than requiring a fundamentally different alignment interface.

The result is not a replacement for autoregressive language modeling, but an extension in which provenance and time become first-class components of the latent representation, enabling continual learning across sovereign communities.

1. Documents Are Already Compressions

Large language models are rarely trained on conversations as they originally occurred.

Scientific papers are trained as papers rather than as years of discussion, experimentation, disagreement, prediction, revision, and publication.

Books are trained as completed manuscripts rather than as drafts, correspondence, editorial decisions, and responses from readers.

Online discussions are often reduced to linear text despite originating as conversations among many participants unfolding across time.

The resulting corpus is extraordinarily valuable, but it is also highly compressed. It preserves language while discarding much of the process that generated it.

Current language models therefore learn the products of civilization’s learning process rather than the process itself.

2. Autoregression Remains Unchanged

This proposal does not replace autoregressive language modeling.

The model is still trained through next-token prediction.

The change is not the objective.

The change is the representation presented to the objective.

Instead of treating every token as belonging only to an ordered sequence, each token also inherits information from the Thought in which it appears.

That Thought has an identifiable source.

It has a timestamp.

It exists within an ongoing conversation.

The model therefore predicts the next token while conditioning on three distinct forms of context:

• the preceding language,
• who is speaking,
• and when the statement occurs.

Autoregression remains the primitive.

The conversational object over which autoregression operates becomes richer.

3. Extending the Context Window

Traditional transformers represent order through positional embeddings.

They answer a local question:

Where does this token occur within the current sequence?

Many learning problems require an additional question:

When did this occur in the world?

Timestamp embeddings provide that information independently of sequence position. Source embeddings identify who produced the associated Thought.

Together these representations allow the context window to contain observations drawn from different participants across different moments in time while remaining within a standard transformer architecture.

The model no longer observes only the order of words.

It also observes the temporal structure of the conversation producing those words.

4. From Pretraining to Continual Learning

The frontier model learns by predicting the continuation of temporally grounded conversations.

Deployment should preserve that representation rather than replacing it with an unrelated alignment interface.

FourThought provides a structured dialectic through which the same conversational objects continue to exist after deployment.

People stake Thoughts.

The model stakes Thoughts.

People respond.

The model predicts those responses before they occur.

As new Thoughts are staked and time passes, the discrepancy between predicted and observed community responses becomes an ongoing learning signal.

Pretraining and deployment therefore share the same underlying representation.

The model first learns from historical discourse.

It then continues learning from living discourse.

5. Provenance During Inference

Modern language models increasingly provide citations after generating responses.

This proposal instead incorporates provenance into inference itself.

Every Thought already possesses an identifiable source and timestamp.

Those representations become part of the attention computation rather than metadata attached afterward.

The resulting output is therefore conditioned not only on semantic similarity but also on the provenance of the information contributing to the response.

Inference becomes provenance-aware by construction.

6. Local Learning and Shared Learning

Each Iris learns within a local community.

It develops local parameters through interaction with that community’s FourThought ledger while remaining aligned with the objectives defined by that community.

Learning does not terminate after deployment.

Nor does it require a single continually trained global model.

Instead, independent Iris instances continue learning locally while contributing distilled symbolic and representational knowledge to a shared coordination substrate.

Future Iris instances begin from that accumulated prior before continuing their own local learning.

The architecture therefore combines continual local adaptation with continual civilizational accumulation.

7. Discussion

Modern language models learn extraordinarily well from text.

This work asks whether they can also learn from the process that continually produces text.

The proposal does not replace next-token prediction.

It extends the context available to that objective through source and temporal representations, then preserves that same conversational structure after deployment through FourThought.

The central claim is modest.

Language is not generated by isolated documents.

It is generated by conversations unfolding across people and time.

Representing more of that process during both pretraining and deployment may provide a more natural foundation for continual learning than treating alignment as a stage separate from language modeling itself.