Mengxuan Sun

Excited to share that our scoping review has been accepted after four rounds of rigorous peer review! It’s great to see real-world use cases of NLP improving cancer care, not just from technology side. Surprised how few studies actually evaluate these tools in clinical settings.

Nice discussion of robustness issues in health AI (input perturbations, missing data, domain shift, etc). Useful for other domains as well nature.com/articles/s4174…

Ever experienced cancer? We are doing a research study about using Artificial Intelligence to improve quality of life questionnaires. We would like to invite you to complete a short questionnaire. forms.office.com/e/rKxHjS3NW4 Find more information: cancerresearchuk.org/about-cancer/f…

New blog: People do not understand how LLMs can/cannot help them People will make much better use of LLMs if they understand what the technology can and can not do. Unfortunately many people have little understanding of this; I make a few suggestions ehudreiter.com/2025/03/13/peo…

New blog: Do LLM coding benchmarks measure real-world utility? LLM coding benchmarks do not measure real-world utility. I contrast what is measured by SWE-bench with what is measured by a recent study of real-world utility in software development. ehudreiter.com/2025/01/13/do-…

When it comes to applying LLM to different domains, it's difficult for researchers to fully consider what problems exist and what we need to evaluate. Qualitative evaluation studies are helpful.

Thanks Ehud for this blog. Qualitative methods for LLM evaluation need more attention. We did a focus group study with doctors, patients, NHS IT and stakeholders, let them discuss a use case. They give us a lot of interesting insights that have not been discovered.

New blog: How AI can help reform UK NHS ehudreiter.com/2024/09/23/how… The UK government wants to reform the UK health system by digitisation, shifting care to communities, and focusing on prevention. I think there is a lot of potential for AI to help with this...

Want to deploy language models to real time clinical applications? Check out our stimulation studies on the potential impacts

Talked with a friend who is also doing an interdisciplinary PhD and based on the domain department rather than tech we used to work We both feel real world issues are more complicated than we think. But understanding real world without tech could be a key to make tech more useful

Do you work on human evaluation of NLP systems or related topics? We just released the call for papers for The 4th Workshop on Human Evaluation of NLP Systems (HumEval’24)! :) Submission deadline 11 March 2024! Website: humeval.github.io (1/4)

Discover problems in actual scenarios and understand the needs of doctors and patients is important. Use LLMs to be a helper to solve problems, rather than replacing a doctor.

Some explosions use ChatGPT to explain medical notes. Really interesting to know this from medical experts I got a new comment from a medical phd who said ChatGPT give too many “maybe” answers when making diagnosis/suggestions. This may lead patients to a heavier mental burden

New blog: ChatGPT error or human error? ehudreiter.com/2023/10/19/cha…

It’s really interesting to see how ChatGPT responds in this use case. Surprisingly ChatGPT almost replied “You are not alone in this journey” in each prompt to comfort patients.

New blog: Evaluating chatGPT ehudreiter.com/2023/04/04/eva…

New blog: Does chatGPT make leaderboards less meaningful? ehudreiter.com/2023/03/27/cha…