Reproduction Package for ATIQSER 2026 submission "DOIchecker: A Tool for Validating Bibliographies of Research Papers"

Résumé fourni par la source

Reproduction Package for the Paper “doi-checker: A Tool for Validating Bibliographies of Research Papers” Abstract This artifact is the reproduction package for the paper “doi-checker: A Tool for Validating Bibliographies of Research Papers”, submitted to ATIQSER 2026, co-located with ASE 2026. The paper presents doi-checker, which extracts the references from a paper PDF and verifies the DOIs they print against Crossref and DataCite, and compares it with four other open-source reference checkers over 3888 papers published at CAV, FSE, ISSTA, PLDI, POPL and TACAS between 2020 and 2026. The artifact contains the five tools at the commits that produced the results, the caches and database mirrors that make a corpus-scale run feasible, the analysis that turns raw results into every number in the paper, and the original results of the paper — so the analysis can be checked without re-running anything. It also contains 2500 of the 3888 papers: every one whose licence permits redistribution. The rest cannot legally be bundled, and make corpus fetches them from the publishers. See The corpus and its licences — this affects what a re-run can cover, and nothing else. Everything installs and runs with no network at all. There are three ways to run the experiments, cheapest first: a smoke test over 3 papers, a subset of 949 papers (~1 h at JOBS=16), and the 2500 bundled papers (~6 h). Tested in the VM Run end to end from this archive unzipped, on a freshly imported, unmodified CAV 26 VM whose network adapter had been removed entirely (--nic1 none), so nothing could reach the internet even by accident. Every entry in the archive except this file is byte-identical to the one those runs were made from: Step Result Time ./setup_vm.sh podman installed offline, GROBID image loaded, all checks pass 54 s ./install.sh 5 virtualenvs built from bundled wheels, caches re-keyed to 100% coverage of the bundled corpus 48 s make smoketest OFFLINE=1 overall: PASS; every reference count matches the paper, and doi-checker matches it in full 3 min 20 s, or 31 s with NO_GROBID=1 make compare-paper RESULTS_DIR=experiments/results-paper identical to the paper — all 1277 macros the paper typesets regenerate byte-for-byte 20 s make check-corpus 2500 present, 1388 not bundled, 0 missing 2 s The one check that cannot be run from the archive is re-cutting the corpus out of the publishers’ volumes, because those volumes are not bundled — 5 GB, and needed only to redo the split. It was verified in the development repository instead: re-cutting the 65 CAV 2020 papers reproduced all 65 byte-identically in 46 s. From the package, make proceedings fetches the volumes first; see Fetching the rest. Where to find things Requirements disk, memory, and why the artifact does not fit inside the VM TL;DR the whole reproduction path, copy-pasteable What to Check, and How the paper’s five claims, and the command for each Contents what is in the package, and how big Experiments the three benchmark sets Smoke Test exactly what it should print Results comparing your run with the paper’s Running Offline what makes this work without a network, and the one thing it costs The corpus and its licences why 2500 of 3888 papers ship, and how to get the rest Information for Reuse other papers, and adding a sixth tool Provenance and Licences commits and terms Rebuilding this Package the reproducible build Known Issues what can go wrong, and what it means Requirements The tools are compiled for x86_64 and will not run on ARM. Any x86_64 Linux with Python 3.11+ works; the CAV 26 VM (Ubuntu 24.04.4, user and password cav, Python 3.12.3) is what it was tested on. Disk 9.2 GB archive, 19.3 GB unpacked, ~0.6 GB for a full run’s results Memory 4 GB; 8 GB is comfortable if you also run GROBID Cores any — this work is network- and I/O-bound, not CPU-bound Network none, for everything in this README The CAV 26 VM’s own disk is 25 GB with about 12 GB free, so the artifact does not fit inside it. Unpack it on the host and mount it into the VM as a shared folder, as in the TL;DR below. That is the route that was tested. The VM ships build-essential, git, curl, make, python3-full, Java and Rust, but no container engine, which GROBID needs. setup_vm.sh installs podman and the shared libraries the prebuilt hallucinator binary needs from the bundled vendor/ directory, then loads the GROBID image — all without a network. It uses sudo for apt and nothing else. Nothing is installed system-wide beyond those packages: each Python tool gets a private virtualenv (the Rust one ships as a binary), and every cache, database and result stays inside the artifact directory. If podman cannot be installed, add NO_GROBID=1 to any run target: the bundled extraction caches cover every paper in the corpus, so nothing needs re-extracting. Only checking a new PDF needs GROBID. TL;DR Step 0, on the host. Unpack the archive and give the VM the directory as a shared folder, with the VM shut down: unzip doi-checker-artifact.zip -d ~/cav-artifact # 19.3 GB, on the host VBoxManage import cav2026.ova # if you have not already VBoxManage sharedfolder add cav2026 --name artifact \ --hostpath ~/cav-artifact --automount # Not optional: every Python virtualenv contains a symlink, and vboxsf refuses # to create one unless the host says it may. There is no equivalent setting in # the VirtualBox GUI. Without this, ./install.sh stops with # Error: [Errno 1] Operation not permitted: 'lib' -> '.../.venv/lib64' VBoxManage setextradata cav2026 \ VBoxInternal2/SharedFoldersEnableSymlinksCreate/artifact 1 # The 1 vCPU / 4 GB default works but is slow, and GROBID wants headroom. VBoxManage modifyvm cav2026 --memory 8192 --cpus 4 VBoxManage startvm cav2026 Steps 1–4, inside the VM, in a terminal at /media/sf_artifact/doi-checker-artifact. Every command in this README is run from there. # 1. Set up. setup_vm.sh asks for the sudo password once; on this VM it is `cav`. ./setup_vm.sh # -> VM is ready. Next: ./install.sh ./install.sh # -> ... refchecker bibliographies 2499 / 2500 (100.0%) # 2. Check the analysis reproduces the paper, without running any experiment. # 25 seconds, and the most direct check in the artifact. make compare-paper RESULTS_DIR=experiments/results-paper # -> identical to the paper # 3. Check the pipeline itself works, on three papers. # ~40 min the first time (nothing has been read off the 19 GB yet), ~5 min after. make smoketest OFFLINE=1 # -> overall: PASS # 4. Optionally re-run more of the experiment, then compare again. make subset-experiments OFFLINE=1 # 949 papers, ~1 h at JOBS=16 make data-analysis make compare-paper make help lists every target. If you cannot use a shared folder Give the VM a bigger disk. VirtualBox cannot resize a VMDK, so clone it to VDI first (VBoxManage clonemedium), resize that, attach it, and grow the partition inside the VM with growpart and resize2fs. Make the artifact smaller. Deleting the two database mirrors brings it to 5.5 GB, which fits — but it changes the results, so read Running on a small disk first. What to Check, and How The paper asks five research questions. All five are answered by macros in data-analysis/output/macros.tex, which is the single generated file the paper \inputs — it hard-codes no figure of its own, so every number and every table cell in the evaluation comes from that file. make compare-paper therefore checks all five claims at once. They do not all reproduce equally well from an offline run, and it is worth knowing which is which before starting: Paper Question Comes from Offline? RQ1 Coverage what fraction of references print a DOI, by venue and year the doi column of references.csv yes — extraction only RQ2 Validity how many printed DOIs are invalid or non-resolvable doi-checker’s invalid_doi flags no — see the one difference RQ3 Integrity how many DOIs resolve but disagree with the cited entry doi-checker’s author_mismatch / title_mismatch flags yes — from cached registry records RQ4 Accuracy

Ce résumé expose les affirmations des auteurs. BNTIC ne l’interprète pas comme une validation indépendante des résultats.