FOOLSDEDIT: Deceptively Steering Your Edits Towards Targeted Attribute-Aware Distribution

Le résumé fourni par la source

Guided image synthesis methods, like SDEdit based on the diffusion model, excel at creating realistic images from user inputs such as stroke paintings. However, existing efforts mainly focus on image quality, often overlooking a key point: the diffusion model represents a data distribution, not individual images. This introduces a low but critical chance of generating images that contradict user intentions, raising ethical concerns. For example, a user inputting a stroke painting with female characteristics might, with some probability, get male faces from SDEdit. To expose this potential vulnerability, we propose the Targeted Attribute Generative Attack (TAGA), whose objective is to force SDEdit to generate data distributions aligned with a specified attribute (i.e.,targeted attribute like male), without changing the attribute of the input image. Empirical studies reveal that traditional adversarial noise struggles to achieve TAGA, while natural perturbations such as exposure and motion blur can easily influence attributes of the generated images. Inspired by the observation, we design attack methodFOOLSDEDITto achieve effective TAGA against SDEdit. It aims to search for an optimized strategy to execute attacks within a weighted graph-based attack architecture, which is formulated to model diverse strategies derived from both exposure and motion blur perturbations. Comprehensive experiments on two commonly used datasets and three social attributes present thatFOOLSDEDITforces SDEdit to generate targeted attribute-aware distributions, achieving significantly more effective TAGA than the baselines. We also validated empirically thatFOOLSDEDITcould induce bias in downstream tasks of SDEdit which rely on the generated data under attack. Our work reveals critical vulnerabilities in diffusion-based image generation models and paves the way for future research on model auditing and bias mitigation.

Ce résumé expose les affirmations des auteurs. BNTIC ne l’interprète pas comme une validation indépendante des résultats.