Developing a High-Fidelity Medical Diagnostic Assistant using RAG

1. Executive Summary

This project involved the development of an AI-powered Medical Assistant designed to alleviate information overload for healthcare professionals. By leveraging Retrieval-Augmented Generation (RAG), the system ingested the 19th Edition of the Merck Manual (a 4,000+ page medical reference) to provide evidence-based answers to clinical queries.

The solution utilized Mistral-7B for inference and ClinicalBERT for domain-specific embeddings, successfully achieving a high degree of groundedness in diagnostic outputs while strictly preventing hallucinations on unsupported topics.

2. Business Context & Problem Statement

The Challenge

Healthcare professionals operate in a high-pressure environment where they must process vast volumes of data to deliver accurate diagnoses. The primary challenges identified were:

• Information Overload: Clinicians struggle to sift through extensive research papers and manuals quickly.
• Need for Speed: Emergencies require time-sensitive decisions regarding drug information, treatment plans, and critical care protocols.
• Risk of Error: Generic AI models often "hallucinate" medical facts, creating a safety risk if relied upon for patient care.

The Objective

The goal was to create a functional AI prototype that could streamline decision-making by answering specific clinical questions (e.g., "What are the first-line options for rheumatoid arthritis?") using only trusted, renowned medical manuals as the knowledge base.

3. Technical Architecture

The system was architected as a closed-loop RAG pipeline to ensure data provenance and accuracy.

Tech Stack

• Large Language Model (LLM): TheBloke/Mistral-7B-Instruct-v0.2-GGUF (Quantized for GPU efficiency).
• Embeddings Model: medicalai/ClinicalBERT (Selected for superior performance on medical terminology).
• Vector Database: ChromaDB for storing and retrieving high-dimensional vector embeddings.
• Orchestration: LangChain for chaining prompt templates, retrieval, and generation.
• Document Loading: PyMuPDFLoader for parsing the 4,114-page PDF source document.

Data Pipeline Implementation

1. Ingestion: The system loaded the Merck Manual of Diagnosis & Therapy, 19th Edition.
2. Chunking: The text was split using RecursiveCharacterTextSplitter with a chunk size of 512 tokens and an overlap of 20 tokens to maintain context across section breaks.
3. Vectorization: The medicalai/ClinicalBERT model converted text chunks into 768-dimensional vectors stored in ChromaDB.

4. Methodology: Prompt Engineering & Tuning

To ensure the model acted as a reliable medical assistant, rigorous prompt engineering and parameter tuning were applied.

System Prompt Design

The system was strictly constrained to avoid external knowledge. The core instruction was:

"You must only provide answers that are directly supported by the Merck Manual, 19th Edition... If a question cannot be answered using the Merck Manual 19th Edition, respond with: 'I'm sorry, I can only provide information that is available in the Merck Manual...'".

Hyperparameter Tuning

The team tested various configurations to optimize response quality:

• Temperature: Tested values of 0.2, 0.5, and 0.8. Lower temperatures (0.2–0.5) yielded more factual, concise answers suitable for medical queries.
• Retrieval (k): Tested retrieving 3 vs. 5 document chunks. k=3 provided focused answers, while k=5 offered broader context for complex queries.
• Top_p / Top_k: Adjusted to balance determinism and fluency.

5. Key Results and Evaluation

The system was evaluated using an "LLM-as-a-judge" framework, where the model rated its own outputs on a scale of 1–5 for Groundedness (adherence to source) and Relevance (answering the prompt).

6. Conclusion and Future Recommendations

Project Impact

The project successfully demonstrated that a specialized RAG architecture could standardize care practices and reduce information overload by providing instant, source-backed medical answers.

Recommendations for Production

• Expand Knowledge Base: The Merck Manual 19th Edition lacked specific sepsis protocols. Integrating updated guidelines and drug databases is essential for comprehensive coverage.
• Hybrid RAG: Implement a fallback mechanism where, if the primary source lacks data, the system can search a secondary tier of approved medical literature.
• Automated Evaluation: Continue using the automated scoring system to monitor performance as new documents are ingested.