RS-005VALIDATEDPublished on September 17, 2026
Memory-Bounded LRU Caches for Vector Embedding Pipelines
Investigating deterministic cache eviction policies for multi-dimensional float32 embeddings in resource-constrained container environments.
01Problem Statement & Research Question
Core Research Question:Can an intrusive doubly-linked list LRU cache bound vector memory consumption within ±2% of a fixed heap budget?
Embedding vectors (e.g. 768 or 1536 floats per token block) rapidly exhaust serverless memory limits if cached naively without hard byte-size bounds.
02Hypothesis
Tracking exact buffer byte allocations during cache insertion guarantees deterministic eviction before heap garbage collection pauses trigger.
03Architecture & Methodology
Doubly linked list combined with hash map indexing.
Methodology Overview:
Simulated 100,000 embedding retrieval calls with power-law query distributions under a 64MB memory quota.
04Implementation & Experiments
TypeScript LRU with typed ArrayBuffer byte tracking.
05Empirical Findings
- Maintained 84.6% cache hit ratio under Zipfian distribution while staying strictly within 64MB.
- Eliminated node out-of-memory crashes under sustained 5,000 req/s loads.
06Known Limitations
- Does not persist across cold restarts unless backed by an external redis/valkey tier.
07Reproduction & Usage
Reproduction Command:
Run embedding cache tests in src/domain/dsa/__tests__/LruCache.test.tsEmbedded in Ask the Lab semantic retrieval pipeline.