Publications

A searchable list of some of my publications is below. You can also access my publications from the following sites.

My ORCID is ORCID iD icon

Publications:

1.

Tobi Olatunji, Charles Nimo, Abraham Owodunni, Tassallah Abdullahi, Emmanuel Ayodele, Mardhiyah Sanni, Chinemelu Aka, Folafunmi Omofoye, Foutse Yuehgoh, Timothy Faniran, Bonaventure F. P. Dossou, Moshood Yekini, Jonas Kemp, Katherine Heller, Jude Chidubem Omeke, Chidi Asuzu, Naome A. Etori, Aimérou Ndiaye, Ifeoma Okoh, Evans Doe Ocansey, Wendy Kinara, Michael Best, Irfan Essa, Stephen Edward Moore, Chris Fourie, Mercy Nyamewaa Asiedu

AfriMed-QA: A Pan-African, Multi-Specialty, Medical Question-Answering Benchmark Dataset Best Paper Proceedings Article

In: Proceedings of Annual Meeting of the Association for Computational Linguistics (ACL), 2025.

Abstract | Links | BibTeX | Tags: ACL, computational linguistics, Dataset, machine learning, Socal Impact Award

@inproceedings{2025-Olatunji-APMMQBD,

title = {AfriMed-QA: A Pan-African, Multi-Specialty, Medical Question-Answering Benchmark Dataset},

author = {Tobi Olatunji and Charles Nimo and Abraham Owodunni and Tassallah Abdullahi and Emmanuel Ayodele and Mardhiyah Sanni and Chinemelu Aka and Folafunmi Omofoye and Foutse Yuehgoh and Timothy Faniran and Bonaventure F. P. Dossou and Moshood Yekini and Jonas Kemp and Katherine Heller and Jude Chidubem Omeke and Chidi Asuzu and Naome A. Etori and Aimérou Ndiaye and Ifeoma Okoh and Evans Doe Ocansey and Wendy Kinara and Michael Best and Irfan Essa and Stephen Edward Moore and Chris Fourie and Mercy Nyamewaa Asiedu},

url = {https://afrimedqa.com/

https://aclanthology.org/2025.acl-long.96.pdf

https://arxiv.org/abs/2411.15640

https://2025.aclweb.org/program/awards/

https://www.cc.gatech.edu/news/new-dataset-makes-health-chatbots-googles-medgemma-more-mindful-african-contexts

},

doi = {10.48550/arXiv.2411.15640},

year  = {2025},

date = {2025-07-31},

urldate = {2025-07-31},

booktitle = {Proceedings of Annual Meeting of the Association for Computational Linguistics (ACL)},

abstract = {Recent advancements in large language model(LLM) performance on medical multiple choice question (MCQ) benchmarks have stimulated interest from healthcare providers and patients globally. Particularly in low-and middle-income countries (LMICs) facing acute physician shortages and lack of specialists, LLMs offer a potentially scalable pathway to enhance healthcare access and reduce costs. However, their effectiveness in the Global South, especially across the African continent, remains to be established. In this work, we introduce AfriMed-QA, the first large scale Pan-African English multi-specialty medical Question-Answering (QA) dataset, 15,000 questions (open and closed-ended) sourced from over 60 medical schools across 16 countries, covering 32 medical specialties. We further evaluate 30 LLMs across multiple axes including correctness and demographic bias. Our findings show significant performance variation across specialties and geographies, MCQ performance clearly lags USMLE (MedQA). We find that biomedical LLMs underperform general models and smaller edge-friendly LLMs struggle to achieve a passing score. Interestingly, human evaluations show a consistent consumer preference for LLM answers and explanations when compared with clinician answers.

},

keywords = {ACL, computational linguistics, Dataset, machine learning, Socal Impact Award},

pubstate = {published},

tppubtype = {inproceedings}

}