TREC Tip-of-the-Tongue Track

Hello TREC-ToTers! We have released the test queries for the TREC 2025 Tip-of-the-Tongue (TREC-ToT) Track. Please see the guidelines for more information: trec-tot.github.io/guidelines. Run submission deadline will tentatively be in August. #TREC2025 #TRECToT #TREC2025ToT

📣📣📣

Do not forget to participate in the #TREC2025 Tip-of-the-Tongue (ToT) Track :) The corpus and baselines (with run files) are now available and easily accessible via the ir_datasets API and the HuggingFace Datasets API. More details are available at: trec-tot.github.io/guidelines

Hello TREC-ToTers! 👋🏽 Excited to announce the release of TREC 2025 Tip-of-the-Tongue (TREC-ToT) Track guidelines: trec-tot.github.io/guidelines. We will release test queries in July and run submission deadline will be in August. #TRECToT #TREC2025ToT Please register to participate:

Please share the word and register to participate! #TREC2025ToT

These approaches powered the TREC 2024 TOT track test queries and will continue into the 2025 track (trec-tot.github.io). Joyful collaboration with Yifan He, @841io, Jaime Arguello, and @UnderdogGeek! #SIGIR #TREC #TOT

📂 Our Code & Data 🔗LLM-Elicitation: github.com/kimdanny/llm-t… 🔗Human query collection interface with visual stimuli set: github.com/kimdanny/human…

⚡️Multi-Domain Coverage Combining both methods allows TOT query evaluation in multiple domains. We tested simulated evaluation in Movie, Landmark, and Person domains. Moreover, we build a broader, more inclusive TOT test collection.

Solution2️⃣: Human-Elicitation We designed an interface with visual prompts to induce a TOT state in human participants. Their queries closely match authentic TOT queries and captures genuine TOT experiences in a controlled setting.

Solution1️⃣: LLM-elicitation TOT user simulator to produce synthetic queries. Results show high system rank correlation and linguistic similarity compared to real queries. This scalable simulated evaluation method overcomes data scarcity by simulating new queries on demand.

🤔Why the Problem? TOT query data collection relies heavily on community question answering websites (e.g., Reddit). This causes data availability issues and domain bias (e.g., most TOT queries end up being about movies or books).

👅Tip-of-the-Tongue (TOT) search is a complex form of known-item search, shaped by the expression of partial recall, personal context, and uncertain memories. However, TOT research has long been hindered by the scarcity of high-quality TOT queries.

🚨New Breakthrough in Tip-of-the-Tongue (TOT) Retrieval Research! We address data limitations and offer a fresh evaluation method for the TOT complex queries. Curious how TREC TOT track test queries are created? Check out this thread🧵 and our paper📄: arxiv.org/abs/2502.17776

Gentle reminder: Run submission deadline for #TREC2024 Tip-of-the-Tongue (ToT) track is this Friday end of day (AoE).

*IMPORTANT ANNOUNCEMENT* We are extending the #TREC Tip-of-the-Tongue (ToT) Track run submission deadline to October 4th (**10 days from today**). Registration and submission details can be found in the track guidelines: trec-tot.github.io/guidelines #TREC2024 #TRECToT #TREC2024ToT

Dear TREC-ToTers, less than two weeks to go! #TREC2024 #TRECToT #TREC2024ToT 👉🏽 Run submission deadline: **September 23rd** 👉🏽 Submission form: ir.nist.gov/evalbase/conf/… 👉🏽 Guidelines: trec-tot.github.io/guidelines 👉🏽 You need to register to participate: trec-tot.github.io/guidelines#reg…

We have released the test queries for TREC 2024 Tip-of-the-Tongue Track! #TRECToT #TREC2024ToT 🟢 Guidelines: trec-tot.github.io/guidelines 🟢 Run submission deadline: September 23rd Let's gooooooo!

Hello TREC-TOTers! Gentle reminders for #TREC2024 ToT Track: 🟢 Guidelines + train/dev data already available @ trec-tot.github.io/guidelines 🟢 Test queries will be released later this month 🟢 Submission deadline will be in September Watch this space for more announcements! #TRECToT

🟡 Announcing the release of TREC 2024 Tip-of-the-Tongue (TREC-ToT) Track guidelines: trec-tot.github.io/guidelines. We will release test queries in August and run submission deadline will be in September. #TRECToT #TREC2024ToT Please register to participate: trec-tot.github.io/guidelines#reg…