← Back to Literature

The Context Window: Positional vs Timestamp Embeddings

by Speaker John Ash

The Context Window: Positional Embeddings vs Timestamp Embeddings

Positional Embeddings

In a traditional transformer, the context window is a fixed sequence of slots.

Slot 1 → Position 1
Slot 2 → Position 2
Slot 3 → Position 3
...
Slot N → Position N

Each slot is one discrete quantum. You cannot subdivide a slot. You cannot put something between position 2 and position 3. The number of slots IS the context window size — if you have 1024 slots, you can see 1024 tokens, period.

The slots are ordered. Position 1 comes before position 2. The model knows this because each position maps to a fixed embedding vector. The distance between position 1 and position 2 is the same as the distance between position 999 and position 1000 — one slot.

The resolution and range are locked together. If your context window is 1024 tokens and your data is a book, you can see 1024 words. If you want to see more of the book, you need more slots. If you want finer granularity within those 1024 words, you can’t have it — one slot, one token.

The context window is a grid. Fixed size, fixed spacing, fixed resolution.

Timestamp Embeddings

With timestamp embeddings, the context window is a set of containers.

Container 1 → any timestamp
Container 2 → any timestamp
Container 3 → any timestamp
...
Container N → any timestamp

Each container can hold an observation from any point in time. Day 1 in container 47. Day 750 in container 3. Day 500 in container 12. The order of the containers does not matter. The timestamp encoding tells the model when each observation is from, independent of which container it sits in.

This means:

The temporal range is decoupled from the number of containers. 100 containers can span 10 years or 10 milliseconds. The range is determined by what timestamps you put into the containers, not how many containers you have.

The temporal resolution is determined by the frequency bands, not the number of containers. The learnable frequency bands in the TimestampEncoding span 13 orders of magnitude — from sub-second to multi-decade. The encoding can distinguish two observations 0.001 seconds apart just as easily as two observations 10 years apart, because different frequency bands activate at different scales. Resolution is a property of the encoding, not the container count.

The density can be non-uniform. You can cluster 50 containers tightly around one time period and scatter the other 50 across years. You can have gaps. You can have bursts. The model doesn’t care — each container just says “here is an observation, and here is when it happened.”

The containers are unordered. In a positional embedding transformer, shuffling the positions destroys the sequence. With timestamp embeddings, shuffling the containers changes nothing — the temporal information is in the encoding, not the container index.

The Subdivision Property

A positional context window is like a ruler with fixed markings. 1, 2, 3, 4, 5. You cannot add a marking at 2.5. You cannot zoom in on the space between 3 and 4.

A timestamp context window is like having N pins you can place anywhere on an infinite timeline. You choose where to put them. You can cluster them, spread them, mix dense and sparse regions. The pins are your containers. The timeline is continuous.

This is continuously subdivisible. Given 100 containers:

In all three cases, the number of containers is the same. The temporal range and resolution are completely different. The encoding handles all of them because the frequency bands cover all scales simultaneously.

Voices: One Per Container vs Many

A traditional transformer has one voice per slot — one token, one position. A language model: one word per position. A time series model (time_transformer_proof.py): one value per timestamp. The context window is a sequence of single observations across time.

But there’s nothing requiring one voice per container. At a given timestamp, multiple sources can each make a prediction. Each source is a voice. Each voice gets its own container, and all containers at the same timestamp share the same timestamp encoding but have different source embeddings and different values.

This is the trust transformer’s situation. At one moment in time, N sources speak simultaneously. Each source-at-a-timestamp is one container. The model currently uses N containers — all from the same timestamp. Self-attention operates across the N voices, letting them interact. But the window sees only one moment.

Where We Are Now

time_transformer_proof.py — one voice, many timestamps. The context window is a sequence of single values across time. Each container holds one observation from a different moment. Self-attention operates across time. The model learns temporal patterns by attending over the sequence.

trust_v7.py — many voices, one timestamp. The context window is N source predictions at a single moment. Each container holds one source’s prediction. Self-attention operates across sources. The model must learn temporal patterns entirely through the timestamp encoding on each token, because there is no temporal variation in the sequence — all tokens share the same timestamp.

The context window size in both cases is finite and fixed. But what fills it is different: one is a timeline of single voices, the other is a snapshot of many voices.

Where We’re Going

Many voices, many timestamps. K timestamps × N sources = K×N containers. Each container knows when it’s from (timestamp encoding) and who said it (source embedding). Self-attention operates across both time and source simultaneously.

Container 1  → Source A, Day 100, prediction value
Container 2  → Source B, Day 100, prediction value
Container 3  → Source C, Day 100, prediction value
Container 4  → Source A, Day 200, prediction value
Container 5  → Source B, Day 200, prediction value
Container 6  → Source C, Day 200, prediction value
Container 7  → Source A, Day 350, prediction value
...

The containers are still unordered — the timestamp encoding and source embedding identify each one, not its position. The window is still finite — K×N containers, not infinite.

The key difference from the current architecture: the model can now produce a prediction for each of the K timestamps, and loss is computed across all K targets. The cycle structure — source A is expert at Day 100, source B at Day 200, source A again at Day 350 — becomes visible within a single training example. Currently, the model must piece together the cycle across thousands of independent single-timestamp examples over many epochs.

This is analogous to a chat that occurred over many days. Each message has its own timestamp, its own speaker, and they all sit in the same context window. The model knows when each message was sent and who sent it. It can attend across the full conversation.