A Model of errors in transformers

Le résumé fourni par la source

This dataset accompanies the paper "A model of errors in transformers." Filenames The data is provided in 25 csv files. Each filename comprises a model name and a task name, as specified in the main text. The acronyms for models are the same as those used in the paper: flash=Gemini 2.5 Flash; pro=Gemini 2.5 Pro; deepseek=DeepSeek R1. The acronyms for tasks are as follows 1) List Reversal -- reversal2) Nested Linear Transformations -- nlt3) Dynamic Programming -- dp 4) Tower of Hanoi -- hanoi 5) Vanilla Addition -- vanaddition6) Addition with algorithm -- algoaddition7) Binary Addition -- binaddition 8) Multiplication -- multiplication9) Multiplication with polynomial intermediaries -- polymult For instance, data for the accuracy of Gemini Flash on the Dynamic Programming task is contained in flash_dp.csv File Contents Each row in the .csv file summarizes the result of a prompt given to the model. In each .csv file, the first two columns are "length" and "accuracy". The "length" corresponds to the parameter, c, defined in the paper. The "accuracy" is 1 if the model obtained the right result for that particular prompt and 0 otherwise. The other columns vary across the .csv files and specify the data given to the model in the prompt. The column headings should be self-explanatory. These columns can be thought of as specifying the sequences, ${\cal S}^{\rm in}$, ${\cal S}^{\rm out}$ and $\widetilde{\cal S}^{\rm out}$ as defined in the paper. For instance, the file flash_vanaddition.csv has four additional columns titled, "Number 1", "Number 2", "True Sum" and "Model Sum", corresponding to the two numbers that were provided in the prompt, the true sum of the numbers and the sum generated by the model respectively. The mean accuracy for a given c is computed by collecting all rows with that value of c and dividing the number of rows with accuracy = 1 (R) with the total number of rows (N): $a = {R \over N}$ Error bars can be generated using the formula provided in Appendix C. Data adjustments As explained in Appendix C.2, in some cases small adjustments were made to the data before plotting. These adjustments can be easily identified and undone if required. For example, in the file flash_vanaddition.csv, for some data points the complexity parameter was shifted by 1. In such cases, the "length" provided in column 1 will differ from the number of digits in "Number 1" by 1. ("Number 2" always has the same number of digits as "Number 1".) This adjustment can be undone by simply ignoring the first column and directly considering the length of the numbers.

Ce résumé expose les affirmations des auteurs. BNTIC ne l’interprète pas comme une validation indépendante des résultats.