Robust and accurate multimodal information extraction from tobacco box labels using MLDP

Le résumé fourni par la source

With the advancement of smart re-baking factories and the accelerating digital transformation of agriculture, tobacco re-baking is facing an urgent need to shift from manual inspection to intelligent automated systems for quality traceability and information management. However, traditional cigarette box label information extraction methods still suffer from limited efficiency, insufficient accuracy, and poor robustness under real industrial conditions. To address these challenges, we propose MLDP (Multimodal Learning for Document Processing), a multimodal framework tailored for robust information extraction from tobacco box labels.MLDP integrates textual and visual information in a unified framework. For the text modality, multiple DeBERTa-based variants, including BiLSTM-DeBERTa, Multi-Sample Dropout-DeBERTa, Distil-DeBERTa, Two-Stage Dropout-DeBERTa, and LongMax-Dropout-DeBERTa, are combined through an Optuna-driven weighted ensemble strategy to balance model diversity and computational efficiency. For the image modality, PaddleOCR is adopted for text detection and recognition, and a Levenshtein distance-based correction mechanism is introduced to reduce OCR errors, especially in numeric fields. To further enhance cross-modal complementarity, MLDP employs a dynamic weighted fusion strategy together with a decision matrix for modality-level error correction, improving extraction robustness under noisy and complex industrial conditions.Experiments on a real-world multimodal tobacco box label dataset show that MLDP achieves a Micro-F1 of 96.51%, outperforming the text-only baseline (MLDP-T) and image-only baseline (MLDP-V) by 4.34% and 5.73%, respectively. Compared with classical document understanding baselines, including LayoutLMv3, UDOP, and Donut, and recent advanced document parsing or multimodal baselines such as MinerU2.5, PaddleOCR-VL, and Qwen2.5-VL, MLDP achieves the best overall performance and provides a favorable accuracy-efficiency trade-off.In addition, on the DocBank benchmark, MLDP yields 1.18%-1.55% gains on non-textual elements, demonstrating promising generalization ability beyond the target industrial scenario. Nevertheless, the current framework still depends on OCR quality and has been validated mainly on tobacco box labels, so its robustness under more diverse industrial document distributions requires further study. Furthermore, we construct and open-source an industrial tobacco box label multimodal dataset containing 9,719 graphically aligned samples, providing a useful benchmark resource for this task.

Ce résumé expose les affirmations des auteurs. BNTIC ne l’interprète pas comme une validation indépendante des résultats.