Ian Gemp
@drimgemp
Research scientist @deepmind. It's all multiagent.
I just published the story of how I created the world’s first No-Limit Holdem poker solver and made $500k by age 23 medium.com/@olegostroumov… I had to keep the story secret since 2013, but now you can read how I went from near broke to reshaping world's toughest poker games
📄Convex Markov games – @drimgemp et al If you can 'convexify' MDPs, so you can do for Markov games. These two papers lay out a general framework + algorithms for the zero-sum version. 🔗openreview.net/pdf?id=yIfCq03… 🔗openreview.net/pdf?id=dSJo5X5… 5/n
How should we rank generalist agents on a wide set of benchmarks and tasks? Honored to get the AAMAS best paper award for SCO, a scheme based on voting theory which minimizes the mistakes in predicting agent comparisons based on the evaluation data. arxiv.org/abs/2411.00119
@AAMASconf 2025 was very special for us! We had the opportunity to present a tutorial on general evaluation of AI agents, and we got a best paper award! Congrats to Marc, @kateslarson, @qberthet, @drimgemp and the rest of the team!
If you're attending @AAMASconf 2025 and are interested in general evaluation of AI agents, you should check our tutorial on May 19th! The website is here sites.google.com/view/aamas2025…, including some draft notes! Co-organized with Marc Lanctot, @drimgemp, and @kateslarson 1/2
Frontier models are often compared on crowdsourced user prompts - user prompts can be low-quality, biased and redundant, making "performance on average" hard to trust. Come find us at #ICLR2025 to discuss game-theoretic evaluation (shorturl.at/0QtBj)! See you in Singapore
If you're attending @AAMASconf 2025 and are interested in general evaluation of AI agents, you should check our tutorial on May 19th! The website is here sites.google.com/view/aamas2025…, including some draft notes! Co-organized with Marc Lanctot, @drimgemp, and @kateslarson 1/2
Thanks John Schultz from @GoogleDeepMind for the wonderful talk. Mastering Board Games by External and Internal Planning with Language Models youtu.be/JyxE_GE8noc x.com/weballergy/sta…
Title: Mastering Board Games by External and Internal Planning with Language Models Speaker: John Schultz, Deepmind Time: Jan 16, 2-3 pm EST Pls mark your calendar!
We did - Joint work with @marnezhurina @LuciaCKun @mehdidc @laion_ai - no wonder this bunch led straight to Wonderland. Code: github.com/LAION-AI/AIW Homepage: marianna13.github.io/aiw/ Paper: arxiv.org/abs/2406.02061
Yet another opportunity to point out that reasoning abilities and common sense should not be confused with an ability to store and approximately retrieve many facts.
📌 This paper investigates the dramatic breakdown of state-of-the-art LLMs' reasoning capabilities when confronted with a simple common sense problem called the "Alice In Wonderland (AIW) problem". This is despite their strong performance on standardized reasoning benchmarks.…
AI often struggles to give correct and coherent answers to questions, but could game theory clue these models in? Mimicking a cryptic messaging game, MIT CSAIL’s “Consensus Game” improves the reliability of language models. To do this, one part of the AI system generates…
Super cool to see our work on game theory + LLMs mentioned in Quanta magazine! Thanks Steve Nadis for covering it along with Athul and team! x.com/drimgemp/statu…
I’m very bullish on game theory x language models. Both for improving multi-agentic reasoning and for improving language models themselves. So many exciting directions! Thankful for the coverage of our work with @Yikang_Shen, @gabrfarina and @jacobandreas!
Great session chaired by @CarloDeramo! 🤓🧑💻 🔹Improving Convergence and Generalization... @BoZhao__ @gowerrobert @RobinSFWalters @yuqirose 🔹Meta Continual Learning Revisited... Y.Wu, Y.Wei, + 🔹Approximating Nash Equilibria... @drimgemp @MarrisLuke +
Our work on approximating Nash equilibria via stochastic optimization received an honorable mention!
Announcing the #ICLR2024 Outstanding Paper Awards: blog.iclr.cc/2024/05/06/icl… Shoutout to the awards committee: @eunsolc, @katjahofmann, @liu_mingyu, @nanjiang_cs, @guennemann, @optiML, @tkipf, @CevherLIONS
Stoked to announce an Agentic Markets workshop @agenticmarkets at #ICML 2024! @icmlconf 📇 Details: sites.google.com/view/amw-2024/… ✏️ Call for papers: due May 17th 📅 Conference: July 26/27th featuring GOAT speakers like Tuomas Sandholm, Gillian Hadfield, @drimgemp @KonstDaskalakis…
📢Excited for our UMD MARL talk @ Mar 26, 12:00 pm ET📢/--by Ian Gemp @drimgemp, Research Scientist, @GoogleDeepMind on "Approximating Nash Equilibria in Normal-Form Games" in-person: IRB-5165 virtually: sites.google.com/view/universit…… @johnpdickerson @ml_umd #RL #AI #MultiAgentAI
Calling all #GameTheory and #MARL enthusiasts! This Friday (15 Mar) @drimgemp will be talking about ‘Approximating Nash Equilibria via Stochastic Optimization’. Come check it out!
What do haggling, debate, and convincing your kids to go to bed all have in common with Poker? With #LLMs, we map them all onto the framework of #gametheory; we then generate conversational strategies using the same methods that beat top Poker pros. arxiv.org/abs/2402.01704
Tired of using FID for evaluating generative models? Come to our #NeurIPS2023 poster on FLS, a new complete metric for generative models that also penalizes overfitting! neurips.cc/virtual/2023/p… github.com/marcojira/fls @bose_joey @drimgemp Chongli Qin @yorambac @gauthier_gidel
How can metrics for evaluating generative models take into account generalization? In our new paper, we propose a new sample-based metric to address exactly this challenge: the Feature Likelihood Score (FLS). Paper: arxiv.org/abs/2302.04440 Github: github.com/marcojira/fls 1/12
It's back! We're accepting internship applications again! @GoogleDeepMind Looking forward to working again with many incredible junior researchers (& engineers)! Please reach out with any questions! deepmind.com/student-resear…
AI systems give us recommendations every day, from videos to watch to products to buy. What if they could suggest ways to cooperate? To achieve this, we trained a neural network to promote collaboration between humans in a game. Here are the results. 🧵 dpmd.ai/44CNE5n