Pinecone

Official Blog - pinecone.io/blog/cascading… ConstBERT Paper - "Efficient Constant-Space Multi-Vector Retrieval" arxiv.org/abs/2504.01818

🧵 4/n. 📊 How the numbers stack up On MSMARCO, ConstBERT32 keeps MRR 39.04 while shrinking the index from 22 GB to 11 GB. As a reranker, ConstBERT32 pushes TREC DL nDCG@10 to 74.4, matching ColBERT yet running in under 6 ms when paired with ESPLADE first‑stage retrieval.

🧵 3/n. The picture lays out a search pipeline that trims a huge document pool to a tiny list that an LLM can read. A sparse model and a dense embedding model each grab about 1000 likely matches from a corpus that holds 10M-100M records. Their two hit lists are blended, then a…

🧵 2/n. Why constant space matters Every document now carries the same vector count, so the index grows linearly with corpus size rather than document length. Fixed length lets the database pack vectors into cache‑friendly blocks, which improves paging and SIMD throughput, and…

ConstBERT proves smart pooling beats brute token storage. Fixed‑vector pooling keeps multi‑vector power without the memory bill. ConstBERT, from @pinecone cuts multi‑vector index size by about 50% yet keeps top‑tier ranking scores like nDCG@10 74.4 on TREC DL, once it is used…

🧑‍💻 Want to learn the essentials of retrieval-augmented generation (RAG) this weekend? We've got you covered. Start here ➡️ pinecone.io/learn/retrieva… 👷‍♀️ Then build a simple RAG pipeline with hybrid search: colab.research.google.com/github/pinecon… 🛠️ Try a more advanced RAG implementation or a…

☕️ Are you a Java developer tasked with bringing accuracy and relevancy to your AI responses in an existing enterprise application? RAG is an approach to ground your AI responses in accurate and up to date knowledge, whether that be your company's proprietary data or real-time…

🚀 Exciting News! 🚀 I'm thrilled to be delivering the keynote "Sparse Retrieval in the Age of RAG" at the LiveRAG Challenge during sigir-2025! Join me on July 17 at 14:30 to explore how sparse retrieval techniques can power faithful, efficient, and interpretable RAG systems.

Whether you're passionate about IR, RAG, or cutting-edge AI, I can't wait to dive into the future of retrieval together. Mark your calendars and see you there! @pinecone #AI #RAG #InformationRetrieval #SIGIR2025 #LiveRAGChallenge

🚀 DIME (Dimension Importance Estimation) Originally introduced by @guglielm0f et al., 2024 to pinpoint which embedding dimensions matter for each query, DIME showed that masking noisy coordinates can lift dense-retrieval quality.

What we're adding at @pinecone: - Formalisation as a denoising step – framing DIME as a light-weight projection into a cleaner sub-space. - Research-grade code & full benchmarks – open-source scripts for PRF-DIME and LLM-DIME on MS MARCO, TREC-DL '19/'20, DL-HD, Robust '04

Our founder and CEO @EdoLiberty will be at TechCrunch Disrupt. You won't want to miss this.

Want to help shape Pinecone’s onboarding & index creation? We’re interviewing users seriously exploring or building with Pinecone. Book a slot with our team to share your insights in a 15–30 min interview & get a $50 Amazon gift card! Details below: x.com/JesiraeDong/st…

Wanted to build semantic search, retrieval-augmented generation, or agentic RAG apps, but got stuck at chunking? Take a look at our article on chunking strategies to get your bearings! You'll learn why chunking is so important, a bunch of different strategies for chunking, and…