Medprompt: The Ultimate AI Prompting Technique Revealed

Recently, renowned AI scholar and Stanford University professor Andrew Ng highly recommended Microsoft’s Medprompt paper published in November 2023. The study explores whether generalist foundation models equipped with advanced prompting techniques can outperform specialized fine-tuned models, specifically in the medical domain.

Ng’s Key Insights on Improving Generative AI

Professor Ng observed that the following four approaches can effectively enhance the output of generative AI models:

1. Write quick, simple prompts and see how they perform.
2. Based on the shortcomings of the output, iteratively enrich the prompt. This often leads to longer, more detailed prompts, even “mega-prompts.”
3. If this is still insufficient, consider few-shot or multi-shot learning (if applicable), or less frequent fine-tuning.
4. If the desired results are still not achieved, decompose the task into subtasks and apply an agent workflow.

Medprompt: Prompting vs. Fine-Tuning

Microsoft researchers designed an experiment to compare the performance of a fine-tuned specialized model (Google’s Med-PaLM 2) against a generalist foundation model (GPT-4) equipped with the Medprompt prompting framework.

Medprompt cleverly combines dynamic few-shot example selection, autonomously generated chains-of-thought, and select-and-shuffle ensemble techniques. This enabled GPT-4 to comprehensively outperform state-of-the-art specialized models like Med-PaLM 2 on nine major medical knowledge assessment datasets, including MultiMedQA.

Notably, on the MedQA dataset (based on the US Medical Licensing Exam), Medprompt boosted GPT-4’s accuracy to 90.2%, surpassing the 90% threshold for the first time and exceeding the previous best performance by 27%.

Key Components of Medprompt

1. Dynamic Few-Shot Selection: Instead of using fixed expert-curated examples, Medprompt dynamically selects a small number of the most relevant training examples as context for each test question. This avoids the limitations of fixed examples that cannot adapt to all test cases.
2. Autonomously Generated Chains-of-Thought: Medprompt leverages GPT-4’s ability to autonomously generate detailed step-by-step reasoning as exemplars. Experiments showed that model-generated chains-of-thought are more fine-grained and effective at harnessing GPT-4’s reasoning capabilities compared to expert-designed ones.
3. Select-and-Shuffle Ensemble: This technique reduces the model’s preference for option order while increasing result robustness. By generating answers with different option permutations and selecting the most frequent response, it improves reliability.

Ablation Study: Understanding Medprompt’s Performance Gains

The authors conducted an ablation study to analyze the contribution of each component to the final performance:

• Upgrading from GPT-3.5 to GPT-4 improved accuracy by 2.2% on MedQA, showing the importance of more powerful language models.
• Adding 5 randomly selected few-shot examples further increased accuracy by 3.4%, confirming the value of context learning.
• Introducing GPT-4 generated chains-of-thought yielded the most significant 3.4% performance boost, validating the effectiveness of autonomous reasoning steps.
• Using kNN-based dynamic selection of relevant few-shot examples added another 0.8%, outperforming random selection.
• Finally, the select-and-shuffle ensemble strategy seeking consistency across answer permutations lifted accuracy by 2.1% to the final 90.2% result, reducing bias and enhancing reliability.

Broader Applicability Beyond Medicine

Although named Medprompt, the techniques are not limited to the medical domain. Researchers applied the method to knowledge tests in electrical engineering, machine learning, philosophy, accounting, law, nursing, and other fields, observing similarly significant performance gains. This demonstrates the broad potential of Medprompt’s prompting techniques.

By skillfully combining these core technologies, Medprompt transformed GPT-4 into an outstanding medical knowledge expert. The study provides valuable insights for designing effective prompting pipelines, encouraging the use of language models’ self-knowledge generation and reasoning abilities, coupled with dynamic example selection and ensemble methods for continuous optimization. As we further explore and validate these techniques across various domains, we can unlock the full potential of language models for specialized vertical applications.

I also wrote a Medprompt System prompt based on the paper’s method, and the output is indeed different. Here is the output of GPT 4o without this method:

When I asked the question again using the Medprompt method, the result was: