<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE article PUBLIC "-//NLM//DTD Journal Publishing DTD v2.3 20070202//EN" "journalpublishing.dtd">
<article xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink" article-type="research-article">
  <front>
    <journal-meta>
      <journal-id journal-id-type="nlm-ta">reapress</journal-id>
      <journal-id journal-id-type="publisher-id">null</journal-id>
      <journal-title>reapress</journal-title><issn pub-type="ppub">3042-3090</issn><issn pub-type="epub">3042-3090</issn><publisher>
      	<publisher-name>reapress</publisher-name>
      </publisher>
    </journal-meta>
    <article-meta>
      <article-id pub-id-type="doi">https://doi.org/10.22105/kmisj.vi.129</article-id>
      <article-categories>
        <subj-group subj-group-type="heading">
          <subject>Research Article</subject>
        </subj-group>
        <subj-group><subject> Retrieval-augmented generation, Healthcare Supply Chain, Medical item classification, Dense retrieval, BM25, Large language models, Retrieval-augmented generation evaluation</subject></subj-group>
      </article-categories>
      <title-group>
        <article-title>RAG Pipeline Accuracy Benchmarking in Healthcare Supply Chain: Medical Item Classification Tasks</article-title><subtitle>RAG Pipeline Accuracy Benchmarking in Healthcare Supply Chain: Medical Item Classification Tasks</subtitle></title-group>
      <contrib-group><contrib contrib-type="author">
	<name name-style="western">
	<surname> Nasir</surname>
		<given-names>Syed Mohammad</given-names>
	</name>
	<aff>Huron Eurasia India Private Limited, Bengaluru, Karnataka, India.</aff>
	</contrib><contrib contrib-type="author">
	<name name-style="western">
	<surname>Pavate</surname>
		<given-names>Aruna </given-names>
	</name>
	<aff>Thakur College of Engineering and Technology, Mumbai, Maharashtra.</aff>
	</contrib><contrib contrib-type="author">
	<name name-style="western">
	<surname>Yusuf</surname>
		<given-names>Ansari </given-names>
	</name>
	<aff>Thakur College of Engineering and Technology, Mumbai, Maharashtra.</aff>
	</contrib></contrib-group>		
      <pub-date pub-type="ppub">
        <month>06</month>
        <year>2026</year>
      </pub-date>
      <pub-date pub-type="epub">
        <day>10</day>
        <month>06</month>
        <year>2026</year>
      </pub-date>
      <volume>3</volume>
      <issue>2</issue>
      <permissions>
        <copyright-statement>© 2026 reapress</copyright-statement>
        <copyright-year>2026</copyright-year>
        <license license-type="open-access" xlink:href="http://creativecommons.org/licenses/by/2.5/"><p>This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.</p></license>
      </permissions>
      <related-article related-article-type="companion" vol="2" page="e235" id="RA1" ext-link-type="pmc">
			<article-title>RAG Pipeline Accuracy Benchmarking in Healthcare Supply Chain: Medical Item Classification Tasks</article-title>
      </related-article>
	  <abstract abstract-type="toc">
		<p>
			The range of medical products within healthcare Supply Chains is extensive and must be precisely classified into hierarchical categories for purchasing, inventory control, expenditure analysis, and regulatory purposes. Traditional classification methods may not suit the short, mixed nature of product descriptions, or they may produce undue or hallucinated labels (e.g., Large Language Models (LLMs)). Retrieval-Augmented Generation (RAG) introduces a way to ground LLM outputs using an external source of domain knowledge. This work proposes a benchmark for RAG-based pipelines for the hierarchical classification of medical supplies. The performance evaluation of a three-level taxonomy with 214 leaf-level classification nodes was conducted on a benchmark dataset comprising 950 descriptions of medical supply items. We evaluated four pipelining settings by combining dense and BM25 retrieval with two instruction-tuned generation models. Retrieval quality was assessed with MAP@5, Recall@5, NDCG@5, Context Relevance Score (CRS), and Context Coverage (CCO), and the E2E quality was measured by hierarchical classification accuracy, macro-F1, and Answer Fidelity (AF). Dense retrieval outperformed all other metrics in terms of retrieval scores. The dense-retrieval models also yield higher hierarchical classification accuracy, with the best configuration achieving 87.3% for the subcategory. The MAP@5 is strongly correlated with subcategory accuracy, as indicated by partial correlation analysis. A human assessment of 240 misclassified queries revealed context dilution, label hallucination, retrieval collapse, and formatting inconsistency as the main failure patterns. The results provide a rigorous framework for assessing and refining RAG pipelines for domain-specific health care supply classification. 
		</p>
		</abstract>
    </article-meta>
  </front>
  <body></body>
  <back>
    <ack>
      <p>null</p>
    </ack>
  </back>
</article>