RoPE (Su et al., arXiv:2104.09864, April 2021) made long-context models possible by encoding position as a rotation in the Q and K space, allowing the dot product to depend only on relative position. YaRN (Peng et al., arXiv:2309.00071, November 2023) extended this with NTK-aware scaling — adjusting the rotation base frequency selectively for different attention head dimensions — to reach 128K tokens without full retraining. Ring attention (Liu et al., arXiv:2310.01889, October 2023) shards the sequence across GPUs to make million-token training tractable. The honest assessment: every model claiming 1M or more context tokens still degrades measurably on needle-in-a-haystack evals at 500K+ on real documents outside the training distribution. The capability exists; the reliability does not. Long context is not a substitute for retrieval discipline — it is a complement. For DealLens, this is the architectural decision that determines your serving cost: a 128K-context model that handles most VC memos natively costs 4× more in KV-cache than a 32K model paired with a strong retriever.
RoPE extension — YaRN and NTK scaling
Standard RoPE applies rotation matrices to Q and K with frequency base b=10000: position m is encoded by rotating by angle m/(b^(2i/d)) for each dimension pair i. Higher-frequency components (small i) encode fine-grained relative position; lower-frequency components (large i) encode coarse structure. Extending the context beyond training length causes the high-frequency components to see position values they were never trained on, degrading retrieval precision at long range.
Position interpolation (Chen et al., arXiv:2306.15595, June 2023) rescales positions by L_train/L_new — compressing the position space so the model stays within its trained range. This works for mild extensions (2-4x) but degrades at 8x+ because the resolution loss for high-frequency components is too large. YaRN/NTK-aware scaling (Peng et al., arXiv:2309.00071, November 2023) selectively scales only the low-frequency RoPE dimensions while leaving high-frequency ones at or near their trained resolution, recovering precision at long distances.
For DealLens, the practical implication: a Llama 3 70B model extended from 8K to 128K via YaRN will reliably retrieve content within the first 32K tokens but may degrade for content past 64K. For memos up to 50K tokens (a typical 10-page VC memo with appendices), 128K context is reliable. For 200K+ token due-diligence packages, hybrid retrieval (VectorDB for first-pass, then native context for final synthesis) is the more reliable architecture.
Ring attention — sharding the sequence
Ring attention (Liu et al., arXiv:2310.01889, October 2023) shards the input sequence across N GPUs, each holding a contiguous chunk. During attention computation, each GPU computes the local QK^T for its chunk's queries against all keys, communicating key and value shards in a ring pattern — each GPU sends its K, V to the next while simultaneously receiving from the previous. After N steps, each GPU has computed its full attention output against all sequence positions. The communication complexity is O(N * seq/N * d_model) = O(seq * d_model) per step — the same order as a single matmul, making ring attention communication-compute balanced at high GPU counts.
The practical consequence: million-token training is now feasible with ring attention on 64-128 H100 GPUs. Google and Meta both use variants of sequence parallelism for their long-context training runs. For the JHU humanoid capstone, ring attention is not relevant to inference (the VLA runs on-robot, not on a cluster) — but it is relevant if you want to train a custom long-context VLM that processes multi-camera, multi-frame visual histories.
The KV-cache memory wall
KV-cache memory scales linearly with context length: bytes = 2 * num_layers * num_kv_heads * head_dim * seq_len * dtype_bytes. For Llama 3 70B at BF16 with GQA G=8: 2 * 80 * 8 * 128 * seq * 2 = 327,680 * seq bytes. At seq=128K, that is ~42 GB — more than half an A100 80GB. At seq=1M, it is 327 GB — exceeding a single DGX H100's HBM. This is the memory wall that makes serving 1M-token models expensive regardless of context quality.
The mitigation strategies: (1) quantize the KV-cache to INT8 or FP8 — 2x memory reduction with negligible quality loss, (2) layer-wise KV-cache dropping (discard KVs from earlier layers after a threshold), (3) PagedAttention (vLLM) virtualizes the KV-cache into variable-size pages to reduce fragmentation, (4) speculative decoding reduces the number of decoding steps, reducing per-query KV-cache duration. For DealLens at 128K context, KV-cache INT8 quantization is the first intervention — it halves serving cost with essentially zero quality impact.
Retrieval vs context — the right tool for each job
Long context and retrieval solve different problems. Long context excels at tasks requiring reasoning over a complete document (e.g., reading a 100-page due-diligence package and synthesizing a memo) — tasks where the retriever's chunking would destroy the semantic continuity needed for the answer. Retrieval excels at tasks requiring lookup from a large corpus (e.g., finding the three most relevant precedent deals from 10K historical memos) — tasks where no single context window can hold the full corpus.
The correct architecture for DealLens is hybrid: a vector retriever (BM25 + dense) for first-pass candidate selection across 10K+ memos, followed by native-context reading of the top 5-10 retrieved documents (32K tokens each) for synthesis. Running all 10K memos through a 128K-context model directly is 1000x more expensive and retrieves no additional quality from documents ranked below the top 10. Chapter 20 (Advanced RAG and evals) covers the retrieval side in depth.
Models trained at 128K tokens do not uniformly retrieve information from the full context. The 'lost in the middle' phenomenon (Liu et al., 2023) shows that retrieval accuracy is highest at the beginning and end of the context window and lowest in the middle. Designing inputs to exploit this matters.