Measuring Pointwise 𝒱-Usable Information In-Context-ly-Usable Information In-Context-ly

Résumé fourni par la source

In-context learning (ICL) is a new learning paradigm that has gained popularity along with the development of large language models.In this work, we adapt a recently proposed hardness metric, pointwise V-usable information (PVI), to an in-context version (in-context PVI).Compared to the original PVI, in-context PVI is more efficient in that it requires only a few exemplars and does not require fine-tuning.We conducted a comprehensive empirical analysis to evaluate the reliability of in-context PVI.Our findings indicate that in-context PVI estimates exhibit similar characteristics to the original PVI.Specific to the in-context setting, we show that in-context PVI estimates remain consistent across different exemplar selections and numbers of shots.The variance of in-context PVI estimates across different exemplar selections is insignificant, which suggests that incontext estimates PVI are stable.Furthermore, we demonstrate how in-context PVI can be employed to identify challenging instances.Our work highlights the potential of in-context PVI and provides new insights into the capabilities of ICL. 1

Ce résumé expose les affirmations des auteurs. BNTIC ne l’interprète pas comme une validation indépendante des résultats.