clickBrick prompt engineering: optimizing large language model performance in clinical psychiatry

Résumé fourni par la source

Prompt engineering has the potential to enhance large language models' (LLM) ability to solve tasks through improved in-context learning. In clinical research, the use of LLMs has shown expert-level performance for a variety of tasks ranging from pathology slide classification to identifying suicidality. We introduce clickBrick, a modular prompt-engineering framework, and rigorously test its effectiveness. Here, we explore the effects of increasingly structuring prompts with the clickBrick framework for a comprehensive psychopathological assessment of 100 index patients from psychiatric electronic health records. We compare the performance of two locally-run LLMs against an expert-labeled ground truth for a variety of successively built-up prompts for the extraction of 12 transdiagnostic psychopathological criteria. Potential clinical value was explored by training linear support vector machines on outputs from the strongest and weakest prompts to predict discharge ICD-10 main diagnoses for a historical sample of 1692 patients. We could reliably extract information across 12 distinct psychopathological classification tasks from unstructured clinical text with balanced accuracies spanning 71% to 94%. Across tasks, we observed a substantially improved extraction accuracy (between +19% and +36%) using clickBrick for the most reactive model. The comparison unveiled great variations between prompts with a reasoning prompt performing best in 7 out of 12 domains. Clinical value and internal validity were approximated by downstream classification of eventual psychiatric diagnoses for 1,692 patients. Here, clickBrick led to an improvement in overall classification accuracy from 71% to 76%. ClickBrick prompt engineering, i.e., iterative, expert-led design and testing, is critical for unlocking LLMs' clinical potential. The framework offers a reproducible, explainable pathway for deploying trustworthy generative AI across mental health and other clinical fields.

Ce résumé expose les affirmations des auteurs. BNTIC ne l’interprète pas comme une validation indépendante des résultats.