Medical question answering (QA) systems have become a popular research topic in medical information services due to their importance for professionals, physicians and patients. The aim of this article is to explore medical question answering papers focusing on the datasets, methods and application in deep learning framework. We searched three databases: PubMed, IEEE explore and Google scholar, and forward and backward citations from 2022 up to 2025. Our first search results 88 studies, 40 of which are included for in-depth analysis following the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) methodology. We anticipate several challenges including language, contextual and cultural difference for QA tasks.

• Question answering

• Health care

• Machine learning

• Systematic review

Desta BB, Asfaw TT, Abate ST, Gebremeskel GB, Taye AA, et al. (2026) A Systematic Review of Question Answering Systems in Medical Domain: Transformer Based Approaches. Comput Sci Eng 5(1): 1015.

Question-answering (QA) systems are a vital application within the field of Natural Language Processing (NLP), designed to provide accurate answers to specific user queries [1]. Question Answering (QA) has evolved significantly over the past several decades using different approaches. Domain specific medical question answering system processes questions asked by physicians and provide short answers. It has shown a rapid progress due to the publication of large-scale medical QA datasets such as MedRedQA [2], MedSQuAD [3], MedExQA [4], and domain specific models like BioBERT [5], ClinicalBERT [6], SciBERT [7]. Medical QA serves as a strategic tool for continuous professional development and wider access to expert medical knowledge. In recent years, the advent of advanced artificial intelligence (AI) technologies like large Language Models (LLMs), has opened new avenues for enhancing healthcare services.

Question answering (QA) systems can be open-domain and closed-domain. The open-domain QA systems [8], rely on a general vocabularies and large-scale knowledge bases to handle diverse information needs. In contrast, closed-domain QA systems focus on a specific field and use a domain-specific knowledge base [9]. For instance, healthcare question answering systems generate responses are related only to medical terminology and healthcare topics [10].

We conducted a literature review on studies conducted from 2023 to 2025 to characterize the current state of research in medical QAS. Although several researchers have reviewed QA more broadly, none of them focused exclusively on medical QA. To the best of our knowledge, this is the first scoping review specifically dedicated to medical QA systems. Forty papers are included in our systematic review after applying the inclusion and exclusion criteria. We provide a comprehensive overview of the datasets, methodologies and evaluation metrics based on the selected papers. Finally, we discussed the challenges and opportunities to provide a comprehensive understanding of the medical QAS.

Contributions

- We systematically review medical QAs studies and categorized them into three modules:

o Domain based (consumers, education, or legal domain & Medical domain)

o Geographical based (Ethiopian, American, Chinese) and

o Language based (Amharic, English, German, and etc)

- We synthesize about the methods, datasets and evaluation metrics from forty medical QAs papers. The challenges, opportunities and open research issues described in this review will enable future researchers who is interested in studying the medical QA system.

Research questions formulated to address the objectives of this SLR, covering three primary queries spanning from Research Question 1 (RQ1) to Research Question 3 (RQ3) as shown in Table 1.

Table 1: Research questions

Research Questions	Significance
What are the most commonly used datasets in medical question answering system?	To identify studies who collect QA datasets and use existing datasets.
What are the methodologies used in medical QA systems?	Identify methods used for Medical Question Answering system
What are the evaluation metrics used in medical question answering systems?	To identify which evaluation metrics are commonly used for medical QA systems

Question Answering Approaches

Early QA systems built on rule-based systems, whereas modern QA systems rely on deep learning, Transformers models, which have a capable of performing reasoning, summarization, and dialogue.

Deep Learning Approaches (2015–2018): Neural architectures such as Convolutional Neural Networks (CNNs) and Bidirectional Long Short-Term Memory (BiLSTM) networks models learn distributed semantic representations. Deep learning models improve semantic matching and answer extraction tasks by learning contextual relationships within biomedical text [11]. Additionally, attention mechanisms enable them to focus selectively on the most relevant parts of the input sequence. Despite these advancements, deep learning based QA systems depend on large annotated datasets, which are often difficult to obtain in the medical domain. This is due to experts are needed for annotation. They also struggled with long-range dependencies during text comprehension.

Transformer-Based Models (2018–2021): The emergence of the Transformer models such as BERT revolutionized QA research by introducing deep contextual language representations. Domain-specific biomedical model including BioBERT [5], ClinicalBERT [6], SciBERT [7], and PubMedBERT [2], pre-train on millions of biomedical abstracts, clinical narratives, and scientific publications. These models understand medical terminology and clinical context more effectively than earlier systems.

We conducted a systematic review of QA research following the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) methodology. We searched PubMed, IEEE xplore, Google scholar, and forward and backward citations from 2023 up to 2025 using the following search strategy syntax:

(“Question answering” OR “Medical question answering”) AND (“deep learning” or ‘Generative AI’ or ‘transformer based’)

The search string strategy developed in accordance with the journal database. The Boolean operator “OR” and “AND” operators were used during searching. The ‘‘AND’’ operator was used to find only those papers that address all key dimensions simultaneously, while the ‘‘OR’’ operator expanded the scope by including synonymous terms and variations. The following table illustrates the search details of the study (Table 2 and Table 3).

Table 2: Data sources locations

Databases	URL
IEEE xplore	http://ieeexplore.ieee.org/
Google Scholar:	http://scholar.google.com/
PubMed	http://pubmed.ncbi.nlm.gov

Table 3: Search strategy

Databases	Search terms
Google scholar	(question answering or Medical question answering) AND (transformer based or generative AI or learning or deep learning)
IEEE explore	(question answering or Medical question answering) AND (transformer based or generative AI or learning or deep learning)
PubMed	(question answering or Medical question answering) AND (transformer based or generative AI or learning or deep learning)

The inclusion and exclusion criteria applied for three primary reasons. First, to reduce the number of papers intentionally since there may be similar articles appearing across the journal databases. Secondly, to eliminate a survey and preprint articles. The third reason is to filter articles so that included articles are within the topic of medical domain. Figure 1 indicates the general workflow of the review.

Figure 1: Higher-level workflow of the Search strategies for systematic literature review

Snowballing Approach

The snowballing method presented in [12], was used for retrieving research articles in conjunction with database searches. Articles retrieved from the various digital databases (Primary studies) support getting additional publications using the reference list (backward snowballing-BSB) and citations (forward snowballing FSB). The approach helped in touching all the relevant articles needed for this study without missing some key publication.

Study selection criteria

88 papers identified by initial searching the database with the search term. Before screening, we exclude four articles from IEEE Xplore, four article from PubMed, and six articles from google scholar due to incorrect article types. Two other articles removed because they were duplicates. This process resulted 72 remaining papers.

Then, to select the remaining studies, we used a 2-step screening process. The first step involved reading the abstracts and titles of all the papers, including only papers that were about medical QA. We removed irrelevant papers that focused on “consumer questions” and “general questions”. In the second stage of screening, a full-text review conducted to screen the papers further. We have read the full text of these papers and excluded additional 32 papers that did not relate to our research topic. After this process, we remain with 40 potential papers, which is the focus of our review. The detail inclusion and exclusion criteria are included in Table 4.

Table 4: Inclusion and exclusion criteria

Type	Criteria
Inclusion	Articles related to medical QA Published in peer reviewed conference proceedings and journals Journals written in English language. Articles based of deep learning & transformer based Approaches Articles containing the abstract and the text
Exclusion	Studies other than text modalities Non-research papers (such as PhD dissertations and books). Survey articles, preprint and systematic review articles Key notes, symposium papers and tutorials.

As shown in Table 5, we obtained 30 articles from google scholar, 28 articles from IEEE xplore and 30 articles from PubMed initially. Despite some articles were published on arXiv, we retrieved them via Google Scholar rather than through direct search on the arXiv platform. After removing papers with exclusion criteria, 40 article papers were remained for final evaluation.

Table 5: Initial Search Result

No	Database	Before screening	After screening
1	Google scholar	30	12
2	IEEE xplore	28	12
3	PubMed	30	16

Figure 2 presents the PRISMA [13], flow diagram, describing the step wise procedure for papers to be included in the review process.

Figure 2: A detail Overview of the review process for medical QAS based on the PRISMA guidelines

Discussion of the results

We included 40 studies and identified themes, including question answering, answer selection, question generation, datasets and evaluation methods. Amharics’ questions used to train and evaluate QA systems were restricted to certain sources, types and complexity levels. No system communicated confidence levels in the answers. Many studies suffered from high risks of bias and applicability concerns. Only 8 studies completely satisfied any criterion for clinical utility, and only 7 reported user evaluations. Most systems were built with limited input from clinicians.

We perform a discussion of medical QA studies by tabulating descriptions of each study’s methods, datasets, evaluation metrics, and findings. We systematically divide the papers based on the publication year of the papers as shown in Figure 3. Among 40 medical QAS studies, 14 papers in 2023, 15 papers in 2024 and 11 papers in 2025 were published.

Figure 3: Yearly distribution of publications included in the analysis (blue), with excluded literature reviews shown for reference (orange). Only forty papers published between 2023 and 2025 were included.

RQ1: What are the methods most commonly used for medical question answering?

To address this question, a comprehensive reading have been conducted. Primarily, the integration of artificial intelligence into healthcare has shown immense potential to improve both the quality and efficiency of medical services. Recent research has emphasized the rise of deep learning based QA systems in healthcare domain. Deep learning methods have significantly advanced QA systems by enabling models to interpret and generate answers with greater accuracy [14]. These models excel at encoding questions and context passages, identifying relevant answer spans, and producing coherent, making them central to contemporary QA system design.

Nineteen (19) studies used pre-trained large language models (LLMs) such as BERT, Llama-2-7b-hf [15], and GPT [16], within their QA pipelines. These models applied to identify candidate answer spans from passages based on the given questions. 17 papers were using RNN type models for the QAS. Four studies reported that fine-tuning pre-trained LLMs on medical data improved performance compared to general-domain LLMs alone. Generative models have improved the interpretation of medical texts, aiding clinical decision-making and the automation of key healthcare processes [17]. Table 6 highlights the summary of recent QA models.

Table 6: Summary of studies that uses recent methodologies.

Title	Method	Gaps
Fine-Tuning LLaMa2 with LoRA and Quantization for Medical Question Answering[18]	LLaMa2-7B	efficiency
Quang Nguyen et al (2022)[19]	Llama-3-70B, Gemma-2-27B, and Mixtral	Complexity
Santoshi et al. (2022)[16]	BERT and GPT-2 Model	interpretability
Baivab Das et al,2024[20]	BioBERT	Interpretability
Vladika et al,2024 [10]	DeBERTa v3model	Interpretability
Munazza Zaib et al,2025	BioBERT	Language dependent
Zahra et al (2025)[21]	Biomedical BERT	Language dependent
Karan et al[22]	large language models(Med-PaLM 2)	Scalability issues
Soonchan et al (2025)[23]	large language models (LLMs)	Hallucination
Xiang et al (2025)[24]	Transformer models	Language dependent
Alaleh et al 2023[25]	AI models	Language dependent
Han et al (2023)[26]	Large Language Model	hallucination
Ziyang et al (2025) [27]	Temporal-Sensitive Retrieval-Augmented Generation	Generate garbage value
Abdelrahman 2024[20]	RNN and Encoder Decoder Models	Scalability
Ying Li et al[28]	Seq2Seq model	Challenged in long sequences

We categorize the reviewed paper into three basic modules

- Application domain based and

- Geographical basis

Full medical question answering system on different domains (Medical, education and tourism)

Yu sun 2022 [33], research on intelligent question answering framework of open education based on knowledge graph and natural language processing. They embed the intelligent question and answer module into the learning space of OUC University to give the learners recent knowledge. This improve learners’ interest in learning and promote the construction of learning society.

Xiangxiang et al. [41], proposed a question answering model for lung cancer based on knowledge graph. Lung cancer is the high-incidence and high-risk disease in the world. In order to prevent and treat lung cancer, a lung question and answer systems developed based on a knowledge graph. They used lung cancer medical case dataset to develop the knowledge graph of lung cancer, and the problem dataset constructed from the template and enhanced the data volume and diversity of data. By analyzing the questions through the model, the answer, the entity and relationships obtained by querying from the knowledge graph.

Huang et al., introduced ClinicalBERT [6], for information retrieval and enhancing the generation of clinical notes. Root et al. discussed the LLM for automated medical coding to reduce the time and effort for coding clinical notes. Liu et al. showed the ability of GPT-3 in generating medical literature, showing the potential to synthesize complex medical information.

Chinese medical QA dataset, containing approximately 26 million QA pairs. This dataset supports both retrieval and generative QA tasks. Benchmarks indicate that existing models perform well below desired thresholds, emphasizing the dataset’s challenging nature even for current pre-trained language models. A study in uses a BERT model and provide a QA model which gives an exact match and F1 score of 35.11% and 70.96% respectively obtained.

A study by Wonjin et al., proposed biomedical extractive question answering. They investigate the question distribution across the general and biomedical domains and conclude that biomedical domains are likely require list-type answers (multiple answers) than factoid-type answers (single answer). Thus, they propose a sequence tagging approach for BioEQA, which is a multi-span extraction setting. Their experiment results outperform the existing baselines using BioASQ 7b and 8b list-type questions. The BioASQ QA dataset contains factoid, list, yes/no and summarization questions along with passages and answers.

Haoran Yu [30], evaluate various LLM models for medical question answering system. The sentence-5 model combined with Mistral 7B demonstrated superior performance with a precision score of 0.762. They argue that the enhanced capabilities of the model is due to its advanced pre-training techniques, robust architecture, and effective prompt construction methodologies. They provide a description on how to train and fine-tune LLMs using MedQuAD dataset. Lee et al proposed BioBERT [5], model for question answering and named entity recognition.

Ahcene et al. [29], proposed a Transformer-Based Question Answering Model for the Biomedical Domain using COVID-19 QA dataset. They use two transformer models namely BERT and RoBERTa models. The RoBERTa model achieved an Exact Match (EM)/F1 score of 0.38/0.64, respectively.

TANDA is one of the text-based QA models which uses BERT and RoBERTa for modeling the contextual relation between question-and-answer sentences. As transformers are used in these pre-trained models for modeling the contextual representation of each given token by using the multi-head attention mechanism, we classify these models as interaction-based approaches. The pre-trained word embedding from ELMo and BERT language models to transformer’s encoder for modeling the input words and, and fine-tuned pre-trained transformer-based language models including BERT and RoBERTa for answer selection task.

Ruilin et al, [32], proposed attention-based BiGRU over BRCNN for Chinese medical question answering in the development of smart medicine. Their algorithm first inputs the word vector generated by BERT into the attention module. The second technique takes the attention and word vector of the sentence into BiGRU together, and the sequence information of the sentences obtained via the interaction of attention and word vector. The sequence information of sentence used as input of multi-scale CNNs to obtain n-gram information of different size of sentences. At the end, the best answer to the question selected by calculating the cosine similarity between the question and the answer. The proposed model achieved an accuracy of 0.5% and 0.2% using cMEDQA and cMedQA2 datasets.

Lei et al. [42], proposed medical education and artificial intelligence. Computer assisted medical diagnosis technology is used in the field of medical assistance to assist doctors in making diagnostic decisions. As the number of patients increases, the diagnostic pressure on doctors increases and more efficient computer-aided medical diagnosis technology. The performance test results when the diagnostic end threshold of the medical reasoning module is 0.8.

Geographical based focused on language specific models

Languages like Amharic and Geez pose specific QA difficulties. They have a rich morphology and less digital text. Tilahun et al. [43], present deep learning based Amharic question answering using 2628 question-answer pairs curated from Amharic Wikipedia dumps. They test an end-to-end QA and reading comprehension models. Their experimental result shows f1 score of 69.58% and 71.74% using baseline models and retriever-reader approach respectively. Their model follows a retriever reader (reading-comprehension) pipelined QA model.

A study in [42], propose medical Q&A intelligent interaction system using an active interactive intelligent Q&A. The model testing results show that the accuracy of the multilevel interactive answer selection model on the SNLI dataset is 89.2% with out the use of external resources. The ablation test results showed that the overall accuracy of the model was 89.64%, and the accuracy decreased to 88.56% after deleting word level features. After deleting sentence level features, the accuracy decreased to 88.97%.

Hailemariam et al. [44], showed that translating English training data into Amharic (“translate-train”) can bootstrap ODQA. They created AmaSQuAD by machine translating SQuAD 2.0 to Amharic and fine-tuning QA models. This approach achieved substantial gains: 52.50% to 52.66% increases in exact match over English-trained DPR and BM25 baselines.

Jing zhang [31], proposed a similarity algorithm for English intelligent question answering system. They perform a sentence similarity algorithm based on WordNet. Their method used for the design of the intelligent English question answering system with realization. They use 500 repeated sentences divided into 5 groups and the correct rate of each test set group is finally calculated. The results have shown that the average accuracy of the entire QA answering system has 75% similarity score.

The authors in [34], propose multilingual BERT model for a Vietnamese QA system and found that BERT model still has certainly limitation in term of time and precision to return a Vietnamese answer. They propose a multilingual BERT for Vietnam question answering system about tourism. Their method combined BERT and knowledge graph to enhance accurately and find quickly for an answer. We experimented our crafted QA data about Vietnam tourism on three models such as LSTM, BERT and BERT+vnKG. As a result, their model outperformed two previous models in terms of accuracy and time.

Liang et al. [40], proposed a multitask approach for Chinese question generation and question answering using a multilingual T5(mT5) and UniLM transformer model. They perform answer extraction, question generation and question answering tasks as a multitasking using one model. They designed as an end-to-end question generation and question answering from Chinese texts. The experiment result indicate that multitask QG and QA approach has greater performance result than individual tasks. They scored a Bleu of 51.7% vs 49.7% for question generation tasks and 71.5% vs 69.8% f1 score respectively using mT5-small model.

A study on Malayalam Question Answering (QA) [11], use a subset of five tasks from the Facebook BAbI dataset and propose a hybrid of Convolutional Neural Networks (CNN) and Bidirectional Long Short-Term Memory (Bi-LSTM) networks. CNN is first used to extract high-level semantic features from the text, capturing important local patterns, while the Bi-LSTM layer subsequently models contextual dependencies and sequential relationships. A Soft Max activation function is applied to predict the correct answers for the corresponding questions. The approach effectively addresses the problem of capturing both semantic and contextual information in a morphologically rich. The experimental results demonstrate that the hybrid CNN-BiLSTM model achieves 91% accuracy across all five tasks.

Chinese medical QA dataset, containing approximately 26 million QA pairs. This dataset supports both retrieval and generative QA tasks. Benchmarks indicate that existing models perform well below desired thresholds, emphasizing the dataset’s challenging nature even for current pre-trained language models. A study in uses a BERT model and provide a QA model which gives an exact match and F1 score of 35.11% and 70.96% respectively obtained.

A study in the work proposed an approach for persian question classification. Their study used two methods for feature representation, including word2vec and weighted TF-IDF. They used a modified TF-IDF where the TF-IDF representation was multiplied by the word2vec embedding. The proposed approach used SVM and Neural Network as classification algorithms and compared between different feature representations. The highest results were obtained using the neural network and features represented by TF-IDF multiplied by word2vec.

Indexing and retrieval are a basic steps in A RAG system, used to provide the context needed by the LLM for answer generation. Zaha [45], uses indexed information to align the input question with ontology concepts. He performs a vector similarity search to retrieve questions by embedding the input question, as a vector and comparing it to the vectors of pre-indexed questions.

Other categories

Nine (9) studies reported QA systems for answer selection spanning a broad range of biomedical topics. Most QA systems sourced answers from the unfiltered medical literature such as PubMed, which includes clinical practice guidelines, primary research articles, laboratory science, and epidemiology.

The paper by Xinghang [35], proposes pre-trained language model for selecting the best answer from many user-generated responses in community forums. The authors introduce a question–answer cross-attention network (QAN), which uses a pre-trained encoder to represent questions and answers, then applies cross-attention. This architecture achieves state-of-the-art performance on benchmark datasets using SemEval 2015 and 2017. They employ large language models (LLMs) to enhance answer selection by generating external knowledge and improving prompts. However, combining deep models with LLM-based knowledge introduces challenges such as hallucination risk, prompt sensitivity, and efficiency problems like computational cost.

SIYU 2024 [36], introduces TR-BERT, a novel method for answer ranking in QA communities using Zhihu and CQA dataset. Their model combines a long-text matching technique with a pre-trained language model. This ranking method selects the noise and extracts question-answer features. They contribute by designing a new framework to process long-text data and developing the TR-BERT model for answer ranking. Experiments showed TR-BERT outperform baseline models, as evidenced by enhanced test results.

Eman [37], proposed a hybrid model framework by combining logical rules with neural networks using natural question dataset. Their hybrid approach outperforms both neural-based models and rule-based reasoning in accuracy, precision, recall and F1 score. The rule-only approach showed 95% precision, 50% recall, F1 score of 60% and 70% accuracy. The neural only approach achieved a recall of 80% with accuracy and F1 score of 85%, a precision of 90% that indicates the model performs well. The hybrid framework outperformed both baselines with 90% accuracy, F1 score of 90%, and a precision of 95% and a recall of 80%.

Abdelrahman [20], automate the process of generating qualified answers for questions to create a digital doctor. The internet is full of forums that allow people to ask some specific questions and get great answers for them. Browsing these questions in order to locate one similar with the new one. They train an end-to-end model using the framework of RNN and the encoder decoder to generate answers for medical issues. Their model was trained and evaluated using the online data including WebMD,HealthTap and eHealthForums and iCliniq.

Juee Gosavi [38], proposed a method to effectively rank answers which are most relevant and best from historical archives based on similar queries found. Their system contains two components, one contains training samples with positive, negative and neutral classes and other retrieves similar questions to posed questions. Their objective is to rank answer candidates based on pair wise comparison and trained model. Stack overflow, health Tap more and more people now use these web forums to get answers to their questions. The main task of the proposed system effectively rank answers, which are most relevant, and best from historical archives based on similar queries.

Maria Mahbub et al. [39], provides a question answering for drug information extraction from clinical notes. Their aim is to upgrade the accurate and efficient detection of people with inject drugs. Their framework contains two steps: (1) Generating standard QA dataset, (2) developing, and testing the QA model. They demonstrate the QA model’s ability to extract IDU-related information and achieves 78.03% F1 score along with 85.38% precision and 79.02% recall scores Figure 4.

Figure 4: Type of medical QA datasets

The authors in [34], investigate the employment of the pre-trained BERT language model to tackle question generation tasks. They introduce three neural architectures built on top of BERT for question generation tasks. The first one is a straightforward BERT employment, which reveals the defects of directly using BERT for text generation. Accordingly, we propose another two models by restructuring their BERT employment into a sequential manner for taking information from previous decoded results. Experiment results show that their best model yields state-of-the-art performance of BLEU 4 score of 22.17.

RQ2: What are the question answering datasets most commonly used in medical question answering?

All the studies utilized datasets containing questions with predefined answers. The domain of each dataset in the QA system can be effectively and directly influences model accuracy and the reliability of evaluation results. Our analysis focuses the textual datasets only and to facilitate further research, we have provided references for accessing the publicly available datasets.

Most studies relied on datasets consisting of questions paired with known answers. These datasets served for training and evaluation of document retrieval and answer extraction models. There are also studies lacked clear descriptions of their answer data or evaluation procedures, including insufficient details about answer sources.

Many data sets have opened to support questions and answering system development in the medical domain. The topical focus of each dataset determines the domain in which a QA system effectively applied. Nentidis et al. [48], proposed a BioASQ dataset which is a large-scale question answering challenge for yes/no, factoid, list and summary questions. Creating training data for question answering (QA) systems typically requires significant effort and expertise.

Let see some of the datasets used in the studies

HealthFc: HealthFc [30], dataset use only evidence retrieved from knowledge bases Wikipedia, PubMed, and google search. The results indicate that evidence retrieved from Wikipedia achieved an F1 prediction score of 74.5%, PubMed 73.0, and Google scholar 75.5. For datasets with complex biomedical claims, PubMed outperformed Wikipedia. This indicate that their dataset is better suited for verification over Wikipedia to focus on common health concerns. HealthFC is a novel fact-checking dataset for verifying claims to health related queries.

MultiSpanQA: MultiSpanQA [49], consists of questions that require answers to be extracted as multiple discontinuous spans from a text passage. The dataset created by extracting raw questions and Wikipedia spans from the Natural Questions [38], dataset and then re-annotating them to identify multi-span answers. The dataset consists of over 6,000 multi-span questions in its basic version and expands to 19,608 examples, including unanswerable questions. EM and F1 scores used as the evaluation metrics.

QAMPARI: QAMPARI [50], which stands for “questions with many answers over multiple paragraphs” is a crowdsourced dataset containing 63,911 questions where answers are entities distributed in several Wikipedia paragraphs. All questions had at least five answers, with an average of 13. Samples generated automatically from Wikidata and Wikipedia. The questions validated to ensure that they could be answered using Wikipedia passages.

ConcurrentQA: ConcurrentQA [51], is a multi-hop crowdsourced dataset of 18,439 questions spanning Wikipedia passages in the public domain and private domain. Wikipedia in the public domain and emails in the private domain, The format, privacy settings, noise, entity distributions, length of emails, and Wikipedia passages differ in several ways, making this dataset challenging. The system evaluated using the EM and F1 scores. The most prevalent evaluation metrics employed are EM and F1 scores.

The work in [27], present ChronoQA-a large-scale dataset tailored for evaluating temporal-sensitive RAG systems. It achieves 100% temporal relevance and it supports both single- and multi-document scenarios, mirroring real-world demands for temporal alignment and logical consistency across sources. Built from over 300,000 news articles published between 2019 and 2024, and comprises 5,176 high-quality question-answer pairs, generated via a robust multi-stage pipeline. The pipeline combines LLM-based extraction, structured question synthesis, and rigorous validation. The most commonly used datasets are MedQA, emrKBQA MedMCQA, BioASQ and MedSQuAD.

Some datasets like emrQA [52], and emrKBQA [53], rely on template-based datasets that have used the available expert annotations of the n2c2 repository to generate large-scale QA datasets using semi-automated methods. DrugEHRQA [23], and MedAlign [49], datasets are the only multimodal QA datasets that uses both structured data and unstructured clinical notes to answer questions. Generally, the medical question answering dataset summarized into two modalities shown in Figure 5.

Figure 5: Number of papers with each category of domain, method, question, answer source, and answer type.

RQ3: What are the evaluation metrics used in biomedical question answering systems?

Questions and Answers (QA) evaluation methods fall into two categories: automated assessment and human-centered evaluation. The most commonly used automatic evaluation metrics include BLEU, ROUGE-L, METEOR, EM and F1 score. However, the human-based evaluation standardized because each expert has a different level of understanding. Table 7 shows the evaluation methods used by each study. Only 19 studies evaluated their design, system usability, or the relevance of the answer to the question.

Table 7: List of included papers in the systematic review

Studies	Categories	Reference
QA	Application Domain based	[5,29-34]
	Geographical	[35-38]
		([39,40])

The type of QA task would determine the evaluation metrics used. For QA with machine reading comprehension tasks (for example, in QA over clinical notes), exact match and F1 score are the most popular metric for evaluation. Exact match refers to the percentage of predictions that exactly match the ground truth answers. Some metrics still involve new techniques such as answer selection score which measure the correctness and completeness of the answer [56].

Most studies considered the accuracy of answers for evaluation (Table 8).

Table 8: Group of papers according to evaluation methods

Metrics	Description	References
BLEU, and METEOR	Evaluate n-gram overlap or semantic similarity.	[54][18]
Exact Match(EM)	Measures if the generated answer matches the gold answer exactly.	[55]
Accuracy	Overall correctness of predictions.	([46][21][22][19] [16][23][24][26] [27][47][45][31] [15])
Precision	Measures the accuracy of positive predictions	[27][47][47][47] [45][31][30][30] [30][15])
Recall	Measures the completeness of positive predictions	([46][21][22][19])
F1 score	Improve the final macro F1 score up to 10%.	[10]

Some assessed the degree to which the words in the answer match the reference, i.e. accuracy, precision, recall, F1 with respect to words (e.g. ROUGE) or correct entire answers (e.g. yes/no or factoids), numbers of answers/questions. While ROUGE [54,18] or BLEU [54,18] quantify the degree of similarity between candidate answers and the reference sentence, they are unable to account for re-phrasings. Of the models that assessed accuracy/correctness, two used internal cross-validation, while 22 evaluated on an independent dataset.automatic question-answering task are promising for low resource languages. However, they still generating factual errors for low resource languages. This attributed due to LLM’s large parameter size and low resource language’s smaller datasets. This review has several limitations.

Data availability and quality: The issue of data availability and quality is a challenge in deep learning model training. Several research papers rely on a limited dataset, which can lead to overfitting and poor generalization to new types of questions. Many specific domains, such as medical or legal fields, lack large-scale and high-quality annotated QA datasets. The data used for QA often sourced from the web is usually noisy or unstructured, introducing errors that negatively affect information retrieval. This can limit the ability of developed models to generalize across different types of queries.

Question understanding: Questions are often ambiguous, with a single query potentially having multiple interpretations depending on the context in which it appears. Complex question types such as multi-hop questions require reasoning across several documents. Furthermore, synonymy and paraphrasing creates a burden in the process, as variations in wording (for example, “Who founded Microsoft?” versus “Who established Microsoft?”) can change the underlying intent of the question.

Contextual and semantic understanding: Extractive models challenge the long context dependencies, especially during the relevant information distributed across multiple documents. This limitation hinders their ability to identify and extract the most relevant answer spans. While model detect syntactic similarities, they frequently fail to capture deeper semantic relationships. This leads to the extraction of answers that are grammatically correct but semantically irrelevant. Knowledge retrieval: Knowledge retrieval poses another major challenge in question answering systems, especially in retrieval-augmented generation (RAG)-based architectures. The quality of retrieval plays a crucial role, as retrieval errors directly influence the accuracy of the final answer. Moreover, pre-trained retrievers often struggle to adapt to specialized domains such as medicine, where terminology nuances differ significantly from general domain data.

Opportunities: The available domain specific pre trained models, tokenizers, and large-scale corpora, which facilitates the development and performance of QA systems, are currently important. According to [57], QA systems powered by LLMs achieve performance that is comparable to that of human specialists in solving complex clinical questions. This capability is valuable in scenarios where rapid, precise access to scientific evidence can influence patient safety and care quality. Additionally, integrating biomedical QA into clinical platforms support service quality, facilitate remote education and fosters personalized care.

Pre-trained language models denotes the most recent QA methods that focused on transformer model architecture. However, they require both large-scale pre-training data sets and require vast computational resources to train the models.

We reviewed different medical question answering studies. By reviewing 40 papers, this paper answers the three basic research questions, and a qualitative analysis of literature conducted using a systematic literature review. A review of the literature clearly shows that question-answering methods have evolved over time. Starting with rule-based systems, this evolution has moved on to classical machine learning algorithms, deep learning architectures, and Transformer based pre-trained language models. This transition process has brought about significant changes not only in the methods used but also in data representation, and modeling capabilities. We provide a summary of the evaluation metrics, a datasets, and challenges of medical question answering methods. Medical QA research has several challenges such as the limited availability of medical QA datasets as well as in generating realistic medical QA datasets, construction of a large training dataset in medical question answering task is costly because of the need for domain experts. Our findings indicated that transformer models provide promising outcomes for health care tasks. Transformer-based QA models achieve the highest accuracy and F1 scores in most of the question answering studies reviewed. This study will assist future researchers in focusing on areas of medical QA that have possible future research directions.

1. Budur E, Özçelik R, Soylu D, Khattab O, Güngör T, Potts C. Building efficient and effective OpenQA systems for low-resource languages. Knowledge-Based Syst. 2024; 302: 112243.
2. Nguyen V, Karimi S, Rybinski M, Xing Z. MedRedQA for Medical Consumer Question Answering : Dataset, Tasks, and Neural Baselines. 2023; 1: 629-648.
3. Ben Abacha A, Demner-Fushman D. A question-entailment approach to question answering. BMC Bioinformatics. 2019; 20: 511.
4. Kim Y, Wu J, Abdulle Y, Wu H. MedExQA : Medical Question Answering Benchmark with Multiple Explanations. 2024: 167-181.
5. Lee J, Yoon W, Kim S, Kim D, Kim S, Ho So C. BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics. 2020; 36: 1234-1240.
6. Huang K, Altosaar J, Ranganath R, ClinicalBERT : Modeling Clinical Notes and Predicting Hospital Readmission. 2019; 1.
7. Sengupta S, Heaton C, Cui S, Sarkar S, Mitra P. “Towards Efficient Methods in Medical Question Answering using Knowledge Graph Embeddings”.
8. Seo M, Lee J, Tom K, Parikh A, Farhadi A, Hajishirzi H. Real-time open-domain question answering with dense_sparse phrase index. ACL. 2019.
9. Tran P, Nguyen D, Tran HA, Nguyen T, Tran T, Building a Closed-Domain Question Answering System for a Low-Resource Language. ACM Trans. 2023; 22: 1-14.
10. Vladika J, Matthes F. Improving Health Question Answering with Reliable and Time-Aware Evidence Retrieval. 2021: 4752-4763
11. Bibin PA, Babu Anto P. Malayalam Question Answering System Based on a Deep Learning Hybrid Model of CNN and Bi-LSTM Approach. Int J Mod Educ Comput Sci. 2024; 16: 39-55.
12. Badampudi K, Petersen D, Experiences from using snowballing and database searches in systematic literature studies Categories and Subject Descriptors. n Proc 19th Int Conf Eval Assess. 2015.
13. Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021; 372: n71.
14. Calijorne Soares MA, Parreiras FS. A literature review on question answering techniques, paradigms and systems. Comput Inf Sci. 2020; 32: 635-646.
15. KV, Mishra A. Dataset for legal question answering system in the Indian judiciary context. Data Brief. 2025; 60: 111647.
16. Kumari S, Pushphavati TP. Question Answering and Text Generation Using BERT and GPT-2 Model. Computational Methods and Data Engineering. 2023: 93-110.
17. Vega Figueroa E, Inteligencia Artificial Generativa e Investigación Científica. 2023.
18. Bui TT, Truong D, Ngo L. Optimizing LLaMa2 Fine-Tuning with LoRA and Quantization for Medical Question Answering, in 10th International Conference on the Development of Biomedical Engineering in Vietnam. Springer Nature. 2025: 473-487.
19. Nguyen Q, Nguyen DA, Dang K, Liu S, Wang SY, Woof WA, et al. Advancing Question-Answering in Ophthalmology with Retrieval-Augmented Generation: Benchmarking Open-Source and Proprietary Large Language Models. Transl Vis Sci Technol. 2025; 14: 18.
20. Abdallah A, Kasem M. Automated Question Answer medical model based on Deep Learning Technology. 2020.
21. Bahadoran Z, Mirmiran P, Kashfi K, Ghasemi A. Biomedical Research: Formulating a Well-Built and Worth-Answering Research Question. Addict Health. 2025; 17: 1564.
22. Singhal K, Tu T, Gottweis J, Sayres R, Wulczyn E, Amin M, et al. Toward expert-level medical question answering with large language models. Nat Med. 2025; 31: 943-950.
23. Kwon S, Hur S, Jang B. IDQuAD: Infectious disease question and answering dataset. PLoS One. 2025; 20: e0333075.
24. Zhu X, Khan M, Taleb-Ahmed A, Othmani A. Advancing medical question answering with a knowledge embedding transformer. PLoS One. 2025; 20: e0329606.
25. Hamidi A, Roberts K. Evaluation of AI Chatbots for Patient-Specific EHR Questions. 2023.
26. Yang H, Li M, Zhou H, Xiao Y, Fang Q, Zhou S, et al. Large Language 

    Model Synergy for Ensemble Learning in Medical Question Answering: Design and Evaluation Study. J Med Internet Res. 2025; 27: e70080.

27. Chen Z, Min E, Zhao X, Li Y, Jia X, Liao J, et al. A Question Answering Dataset for Temporal-Sensitive Retrieval-Augmented Generation. Sci Data. 2025; 12: 1855.
28. Li Y, Abisado MB. The Development of a Medical Question-answering System Based on Deep Learning in 2025 International Conference on Artificial Intelligence and Digital Ethics (ICAIDE). 2025: 305-308.
29. Haddouche A, Rabia I, Aid A. Transformer-Based Question Answering Model for the Biomedical Domain. 2023 5th Int Conf Pattern Anal Intell. 2023: 1-6.
30. Yu H, Yu C, Wang Z, Zou D, Qin H. Enhancing Healthcare Through Large Language Models: A Study on Medical Question Answering. 2024 IEEE 6th Int Conf Power Intell Comput Syst ICPICS. 2024: 895-900.
31. Zhang J. Application Research of Similarity Algorithm in the Design of English Intelligent Question Answering System. 2022 IEEE 2nd Int Conf Mob Networks Wirel Commun. 2024: 1-4.
32. Qi R, Li H, Zhang Q. Attention-based BRCNN for Chinese Medical Question Answering. 2023 IEEE Int Conf Bioinforma Biomed. 2023: 3933-3937.
33. Sun Y, Bao Y, He L. Research on Intelligent Question Answering Framework of Open Education based on Knowledge. 2022 Euro-Asia Conf Front Comput Sci Inf Technol pp. 2022: 137-140.
34. Do P, Phan THV, Gupta BB. Developing a Vietnamese Tourism Question Answering System Using Knowledge Graph and Deep Learning. ACM Trans Asian Low-Resource Lang Inf Process. 2021; 20: 1-10
35. Hu X. Enhancing Answer Selection in Community Question Answering with Pre-trained and Large Language Models. 2023: 1-24.
36. Sun S, Wang Y, Cheng J, Xiao Z, Zheng D, Hao X. Improving the Performance of Answers Ranking in Q&A Communities: A Long-Text Matching Technique and Pre-Trained Model. IEEE Access. 2025; 13: 4188-4200.
37. Alduweib E, Alromima W, Otoum N. Rule-Augmented Neural Networks for Improved Answer Validation in Question Answering Systems. 2025 1st Int Conf Comput Intell Approaches Appl. 2025.
38. Gosavi J, Jagdale BN. Answer Selection In Community Question Answering Portals. IEEE Punecon. 2018: 1-6.
39. Mahbub M, Goethert I, Danciu I, Knight K, Srinivasan S, Tamang S, et al. Question-answering system extracts information on injection drug use from clinical notes. Commun Med (Lond). 2024; 4: 61.
40. Deng L, Li J, Qi P, Liu Z, Zhang L. MCQG : Multitask Approach for Chinese Question Generation and Question Answering. 2022 14th Int Conf Signal Process Syst. 2022: 726-729.
41. Shao X, Dai C, Hu K. Research on Question Answering of Lung Cancer Based on Knowledge Graph. 2022 IEEE Int Conf Bioinforma Biomed. 2022: 3692-3696.
42. Chen L. Medical education and artificial intelligence : Question answering for medical questions based on intelligent interaction. Concurrency Computat Pract Exper. 2024; 36: e8079.
43. Abedissa T, Usbeck R, Assabie Y. AmQA: Amharic Question Answering Dataset. 2022.
44. Hailemariam ND, Guda B, Tefferi T. AmaSQuAD: A Benchmark for Amharic Extractive Question Answering. 2025.
45. Deng X, Tsai YK, Ganguly S, Tran D, Quek HY, Wilson A. Question Answering System for Smart Cities and Urban Planning With the World Avatar. 2025; 7: e70009.
46. Aurpa TT, Apu SH, Akter F, Rifat RK, Habib A. NOIRBETTIK: A reading comprehension based multiple choice question answering dataset in Bangla language. Data Br. 2025; 59: 111395.
47. Saeed AM. Building a benchmark dataset for the Kurdish news question answering. Data Brief. 2024; 57:110916.
48. Nentidis A, Katsimpras G, Krithara A, López SL, FarréMaduell E, Gasco L, et al. overview of BioASQ 2023: The Eleventh BioASQ Challenge on Large-Scale Biomedical Semantic Indexing and Question Answering. 2023: 227-250.
49. Li H, Tomko M, Vasardani M, Baldwin T. Multi Span QA: A Dataset for Multi-Span Question Answering. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2022: 1250-1260.
50. Amouyal S, Wolfson T, Rubin O, Yoran O, Herzig J, Berant J. QAMPARI}: A Benchmark for Open-domain Questions with Many Answers. Proceedings of the Third Workshop on Natural Language Generation, Evaluation, and Metrics (GEM). Assoc Comput Linguist. 2023: 97-110.
51. Arora S, Lewis P, Fan A, Kahn J, Re C. Reasoning over Public and Private Data in Retrieval-Based Systems. Trans Assoc Comput Linguist. 2023; 11: 902-921.
52. Bardhan J, Roberts K, Wang DZ. Question Answering for Electronic Health Records: Scoping Review of Datasets and Models. J Med Internet Res. 2024; 26: e53636.
53. Raghavan P, Liang JJ, Mahajan D, Chandra R, Szolovits P. emrKBQA: A Clinical Knowledge-Base Question Answering Dataset, in Proceedings of the 20th Workshop on Biomedical Language Processing. Assoc Comput Linguist. 2021: 64-73.
54. Kim Y, Wu J, Abdulle Y, Wu H. MedExQA: Medical Question Answering Benchmark with Multiple Explanations. 2024: 167-181.
55. Lu Y, Yan J, Qi Z, Ge Z, Du Y. Contextual embedding and model weighting by fusing domain knowledge on Biomedical Question Answering. Association for Computing Machinery. 2022; 1: 1-4.
56. Feng M, Bing X, Glass MR, Wang L, Zhou B. Applying deep learning to answer selection: A study and an open task. 2015: 813.
57. Singhal K, Tu T, Gottweis J, Sayres R, Wulczyn E, Hou L, et al. Towards expert-level medical question answering with large language models. ArXiv. 2023.
58. Root J, Ahn DS. Incentives and Efficiency in Constrained Allocation Mechanisms. 2025: 1-42.
59. Liu T, Tan W, Tang X, Chen J, Cao D. Adaptive Energy Management for Real Driving Conditions via Transfer Reinforcement Learning. 2020.