OpenBias: Open-Set Bias Detection in Text-to-Image Generative Models

Moreno D'IncàElia PeruzzoMassimiliano ManciniDejia XuVidit GoelXingqian XuZhangyang WangHumphrey ShiNicu Sebe

article2024CVPR97 citations

Proposes OpenBias, a framework combining large language models and vision question answering to automatically discover and quantify unexpected open-set biases in text-to-image generative models without relying on predefined attribute lists.

Listen

As text-to-image generative models are increasingly deployed into real-world consumer and commercial applications, ensuring their fairness and safety has become a major operational concern. These models often inherit and perpetuate systemic biases present in their training data. Traditional auditing techniques rely on predefined, closed sets of sensitive attributes—such as adult gender, age, and race—leaving a wide range of domain-specific, contextual, or previously unconsidered biases entirely undetected.

The article introduces and evaluates OpenBias, an automated framework designed to discover, assess, and quantify biases in text-to-image models in an open-set manner, without relying on predefined lists of target concepts or specialized manual training datasets.

OpenBias operates via a three-stage automated pipeline that treats the generative model as a black box. First, a large language model analyzes a dataset of text prompts to propose potential biases, associated categories, and visual inspection questions, while filtering out concepts explicitly mentioned in the original prompts. Second, the target generative model synthesizes images conditioned on these prompts across multiple random seeds. Third, a vision question answering model inspects the generated images to answer the bias-related questions, enabling the quantification of bias severity via normalized entropy scores. The framework was evaluated across text prompts from the COCO and Flickr30k datasets and tested on several widely used text-to-image systems, specifically Stable Diffusion 1.5, 2, and XL.

The findings confirm that OpenBias closely matches established closed-set benchmarks and human judgment while uncovering previously unstudied biases. When evaluated on standard demographic categories, the selected vision question answering model (LLaVA-1.5-13B) showed high agreement with FairFace classifiers across real and synthetic images. In a user study spanning 2,200 human evaluations, OpenBias demonstrated a low Absolute Mean Error of 0.15 against human-rated bias intensity and agreed with human judgment on the primary bias direction in 67% of cases. Crucially, the system exposed extensive novel biases beyond traditional categories, including strong defaults for commercial brands (such as exclusively generating Apple logos for laptops), specific animal breeds, object colors, and socioeconomic stereotyping (such as pairing non-white children with impoverished settings). Context-aware evaluations revealed that newer models like Stable Diffusion XL exhibit subtle bias amplification compared to earlier iterations.

These results demonstrate that bias in generative models extends far beyond traditional demographic categories and often depends heavily on surrounding contextual text. Organizations deploying generative AI risk perpetuating brand favoritism, harmful social stereotypes, and unrepresentative visual outputs if auditing is limited to traditional closed-set metrics. Because OpenBias operates modularly and externally without requiring internal model modifications, it offers an effective mechanism to expand current compliance, safety, and bias mitigation protocols.

Practitioners and decision-makers should integrate open-set auditing pipelines into existing model evaluation workflows before deploying text-to-image systems to the public. However, stakeholders should recognize that OpenBias relies on foundation models (LLaMA-2 and LLaVA-1.5) that may carry their own inherent biases or perceptual limitations. Additionally, the systematic role of prompt context requires deeper formal exploration. Overall, the methodology provides high confidence for identifying emerging generative risks and serves as a modular foundation that can readily incorporate more advanced evaluation models as they emerge.

Picsart SHI Labs University of Texas at Austin University of Trento detection algorithm generative model language model open-set bias detection text-to-image generative models OpenBias

arXiv: 2404.07990 Picsart-AI-Research/OpenBias

Paper: SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis, Dustin Podell et al. (2024). Introduces the architecture and training advances of Stable Diffusion XL, one of the primary generative systems audited for subtle bias amplification in OpenBias.
Paper: Red Teaming Language Models with Language Models, Ethan Perez et al. (2022). Establishes the methodology of using automated language models to red-team and probe target AI systems, which OpenBias adapts to multimodal bias discovery.
Paper: Taxonomizing and Measuring Representational Harms: A Look at Image Tagging, Jared Katzman et al. (2023). Develops a foundational taxonomy for categorizing and quantifying representational harms in visual AI systems that informs the conceptual scope of open-set bias evaluation.
Paper: Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification, Joy Buolamwini et al. (2018). Demonstrates the benchmark closed-set demographic auditing methodology against which OpenBias compares and expands open-set detection.
Paper: A Survey on Bias and Fairness in Machine Learning, Ninareh Mehrabi et al. (2019). Provides a comprehensive survey of the definitions, sources, and propagation of algorithmic bias across machine learning pipelines.

Paper: Discovering and Mitigating Visual Biases Through Keyword Explanation, Younghyun Kim et al. (2024). Extends the discovery of open-ended visual biases by extracting and validating interpretable text keywords to guide active bias mitigation in vision models.
Paper: VIEScore: Towards Explainable Metrics for Conditional Image Synthesis Evaluation, Max Ku et al. (2024). Applies multimodal large language models to construct explainable, metric-driven evaluations for conditional image synthesis and generation quality.
Paper: Position: TrustLLM: Trustworthiness in Large Language Models, Yue Huang et al. (2024). Broadens the evaluation of foundation model fairness and safety into a unified benchmark covering multiple trustworthiness dimensions across modern AI systems.
Paper: FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts, Yichen Gong et al. (2025). Explores multimodal safety vulnerabilities in large vision-language models, the class of models relied upon as visual evaluators within automated auditing pipelines.

Cover for OpenBias: Open-Set Bias Detection in Text-to-Image Generative Models

Abstract

Text-to-image generative models are becoming increasingly popular and accessible to the general public. As these models see large-scale deployments, it is necessary to deeply investigate their safety and fairness to not disseminate and perpetuate any kind of biases. However, existing works focus on detecting closed sets of biases defined a priori, limiting the studies to well-known concepts. In this paper, we tackle the challenge of open-set bias detection in text-to-image generative models presenting OpenBias, a new pipeline that identifies and quantifies the severity of biases agnostically, without access to any precompiled set. OpenBias has three stages. In the first phase, we leverage a Large Language Model (LLM) to propose biases given a set of captions. Secondly, the target generative model produces images using the same set of captions. Lastly, a Vision Question Answering model recognizes the presence and extent of the previously proposed biases. We study the behavior of Stable Diffusion 1.5, 2, and XL emphasizing new biases, never investigated before. Via quantitative experiments, we demonstrate that OpenBias agrees with current closed-set bias detection methods and human judgement.

Topics

detection algorithm generative model language model large language models question answering text-to-image model

Keywords

open-set bias detection text-to-image generative models OpenBias bias quantification bias assessment bias mitigation Visual Question Answering large language models social biases

1. Introduction
2. Related work
3. OpenBias
3.1. Bias Proposals
3.2. Bias Assessment and Quantification
3.2.1 Context-Aware Bias
3.2.2 Context-Free Bias
3.2.3 Bias Quantification and Ranking
4. Experiments
4.1. Pipeline Implementation
4.2. Quantitative Results
5. Findings
6. Limitations
7. Conclusions
References

Knowls

Knowl 1 — OpenBias Pipeline for Open-Set Bias Detection
model/method
OpenBias is an automated, open-set framework designed to discover, evaluate, and quantify biases in text-to-image (T2I) generative models without requiring a predefined, closed-set list of bias attributes or annotated training data.

The pipeline operates across three main stages:
1. Bias Proposal Phase: Given a set of real input captions $\mathcal{T}$ , a Large Language Model (LLM) extracts potential caption-specific biases, corresponding discrete candidate attribute classes, and diagnostic natural-language questions.
2. Image Synthesis Phase: Conditioned on the prompts where potential biases were identified, the target black-box text-to-image generator $G$ generates batches of synthetic images across multiple random noise seeds.
3. Bias Assessment and Quantification Phase: A Vision Question Answering (VQA) foundation model evaluates the generated images using the caption-specific diagnostic questions, selecting predictions from the candidate attribute classes to establish class probability distributions and compute quantitative bias severity scores.
Knowl 2 — LLM-Driven Bias Proposal and Two-Stage Filtering
algorithm
To construct a domain-specific knowledge base of candidate biases $\mathcal{B}$ from a collection of raw captions $\mathcal{T}$ , OpenBias queries a Large Language Model via in-context learning to extract bias triplets, followed by a two-stage filtering process to eliminate explicit attributes:
Input: Dataset of textual captions $\mathcal{T}$ , Large Language Model $\mathtt{LLM}$ , Knowledge Graph $\mathtt{ConceptNet}$
Output: Database $\mathcal{D}_b$ of valid caption-question pairs for each bias $b \in \mathcal{B}$
Initialize database $\mathcal{D}_b \leftarrow \emptyset$ for each proposed bias
for each caption $t \in \mathcal{T}$ do
    Prompt $\mathtt{LLM}$ with $t$ using in-context demonstrations
    Receive triplets $\mathcal{L}_t = \{(b_i^t, \mathcal{C}_i^t, q_i^t)\}_{i=1}^{n_t}$
    where $b_i^t$ is bias name, $\mathcal{C}_i^t$ is candidate classes, $q_i^t$ is identification question
    for each triplet $(b, \mathcal{C}, q) \in \mathcal{L}_t$ do
        Query $\mathtt{LLM}$ whether the answer to $q$ is explicitly mentioned in $t$
        if $\mathtt{LLM}$ determines answer is explicit then
            continue
        Query $\mathtt{ConceptNet}$ for synonyms of all classes in $\mathcal{C}$
        if caption $t$ contains any class $c \in \mathcal{C}$ or its synonyms then
            continue
        $\mathcal{D}_b \leftarrow \mathcal{D}_b \cup \{(t, q)\}
Knowl 3 — Context-Aware and Context-Free Bias Probability Formulation
equation
Given a target text-to-image generator $G$ , a bias concept $b$ with candidate classes $\mathcal{C}_b$ , and a set of caption-question pairs $\mathcal{D}_b$ , image batches $\mathcal{I}_b^t = \{G(t, s) \mid s \in S\}$ are synthesized for each prompt $t$ over a set of $N = |S|$ sampled random noise vectors $S$ . For each image $I \in \mathcal{I}_b^t$ , a Vision Question Answering model predicts the attribute class $\hat{c} = \mathtt{VQA}(I, q, \mathcal{C}_b) \in \mathcal{C}_b$ .

The context-aware bias probability measures the class distribution specifically under the prompt context $t$ : $p(c \mid t, \mathcal{C}_b, \mathcal{D}_b) = \frac{1}{|\mathcal{I}_b^t|} \sum_{I \in \mathcal{I}_b^t} \mathbf{1}(\hat{c} = c)$ where $\mathbf{1}(\cdot)$ denotes the indicator function.

The context-free bias probability aggregates predictions across all filtered captions $\mathcal{D}_b$ associated with bias $b$ to characterize the generator's global bias: $p(c \mid \mathcal{C}_b, \mathcal{D}_b) = \frac{1}{|\mathcal{D}_b|} \sum_{(t, q) \in \mathcal{D}_b} p(c \mid t, \mathcal{C}_b, \mathcal{D}_b)$
Knowl 4 — Normalized Entropy Bias Severity Score
equation
Following the fairness principle that an unbiased generator produces a uniform distribution over candidate classes $\mathcal{C}_b$ for class-agnostic prompts, OpenBias quantifies bias severity by calculating the entropy of the predicted class distribution and normalizing it by the maximum possible entropy $\log(|\mathcal{C}_b|)$ .

The bias severity metric $\bar{\mathcal{H}}_b$ is defined as: $\bar{\mathcal{H}}_b = 1 + \frac{\sum_{c \in \mathcal{C}_b} \log p(c \mid \mathcal{C}_b, \mathcal{D}_b)}{\log(|\mathcal{C}_b|)}$

The resulting metric is bounded in the interval $\bar{\mathcal{H}}_b \in [0, 1]$ , where $\bar{\mathcal{H}}_b = 0$ corresponds to a completely balanced (uniform) generation across all candidate classes, and $\bar{\mathcal{H}}_b = 1$ indicates total bias toward a single dominant class.

Knowl 5 — Zero-Shot VQA Backbone Selection on Synthetic Images

data/table

To select the optimal VQA backbone for bias assessment, six vision-language and VQA models were benchmarked zero-shot on images synthesized by Stable Diffusion XL using single-person filtered COCO captions. Predictions for closed-set attributes (Gender, Age, Race) were evaluated against the FairFace classifier as the ground truth:

Model	Gender		Age		Race
	Acc.	F1	Acc.	F1	Acc.	F1
CLIP-L	91.43	75.46	58.96	45.77	36.02	33.60
OFA-Large	93.03	83.07	53.79	41.72	24.61	21.22
mPLUG-Large	93.03	82.81	61.37	52.74	21.46	23.26
BLIP-Large	92.23	82.18	48.61	31.29	36.22	35.52
LLaVA-1.5-7B	92.03	82.33	66.54	62.16	55.71	42.80
LLaVA-1.5-13B	92.83	83.21	72.27	70.00	55.91	44.33

LLaVA-1.5-13B demonstrated superior overall performance across all three evaluation dimensions—achieving 92.83% accuracy (83.21 F1) on gender, 72.27% accuracy (70.00 F1) on age, and 55.91% accuracy (44.33 F1) on race—leading to its adoption as the default VQA backbone for OpenBias.

Knowl 6 — Distributional Alignment Between OpenBias VQA and FairFace

data/table

The alignment between LLaVA-1.5-13B predictions and the FairFace classifier was evaluated via Kullback-Leibler (KL) divergence across real images and synthetic images produced by three Stable Diffusion versions (SD-1.5, SD-2, SD-XL) on Flickr30k and COCO datasets:

Model	Flickr 30k			COCO
	Gender	Age	Race	Gender	Age	Race
Real Images	0.000	0.032	0.030	0.000	0.041	0.028
SD-1.5	0.072	0.032	0.052	0.075	0.028	0.092
SD-2	0.036	0.069	0.047	0.060	0.045	0.105
SD-XL	0.006	0.028	0.180	0.002	0.027	0.184

The consistently low KL divergence scores confirm that the zero-shot VQA predictions closely reproduce the probability distributions of dedicated attribute classifiers across both real and synthetic image distributions.

Knowl 7 — Human Agreement with OpenBias Severity and Bias Direction
empirical result
In a crowdsourced user study involving 55 unique participants and 2,200 valid responses across 15 diverse person- and object-related bias categories (spanning 390 synthetic images generated context-aware with 10 images per caption):
- Participants scored bias intensity from 0 ("No bias") to 10 and selected the majority class direction.
- The Absolute Mean Error (AME) between human-perceived bias intensity and the OpenBias normalized severity score was $\text{AME} = 0.15$ .
- OpenBias agreed with the human majority vote regarding bias direction (majority class) in 67% of cases.
- Strong alignment was observed across both social biases ("Person age", "Person gender", "Person emotion") and object attributes ("Vehicle type", "Train color").
Knowl 8 — Divergence Between Context-Aware and Context-Free Biases
empirical result
Comparing context-aware versus context-free bias measurements on Stable Diffusion XL demonstrates that generative biases can manifest differently depending on context:
- For concepts such as "motorcycle type", the context-aware bias intensity is substantially higher than the context-free score. This indicates that while the model does not globally favor a single motorcycle class across all prompts, it deterministically generates a specific motorcycle style for a given contextual background.
- Conversely, for concepts such as "bed type", the model exhibits high bias intensity in both context-aware and context-free settings, indicating a global collapse where the generator unconditionally favors the same bed design regardless of prompt variation.
Knowl 9 — Discovery of Novel Non-Social and Intersection-Specific Biases in Stable Diffusion
empirical result
Applying OpenBias to Stable Diffusion variants (SD 1.5, SD 2, SD-XL) revealed multiple unstudied domain-specific and non-social biases:
- Object and Animal Biases: SD-XL exhibits brand bias by repeatedly rendering Apple logos on generic prompts like "a photo of a person on a laptop in a coffee shop", generates predominantly yellow trains for "a train zips down the railway", and defaults to quarter horses.
- Stereotypical Person and Scene Associations: When generating "a small child hurrying toward a bus on a dirt road", SD-XL disproportionately depicts Black children in economically disadvantaged environments. For generic professional titles such as "traffic officer", the model defaults exclusively to male figures.
- Model Evolution: SD-XL shows a measurable amplification in the intensity of several biases compared to SD 1.5 and SD 2.
Knowl 10 — Methodological Limitations of OpenBias
limitation
OpenBias is subject to several specific limitations:
1. Propagated Foundation Model Biases: The framework relies on LLaMA-2-7B for proposal extraction and LLaVA-1.5-13B for image querying; latent biases, hallucinations, or failure modes in either foundation model can lead to omitted biases or skewed measurement.
2. Qualitative Context Evaluation: The comparison between context-free and context-aware settings remains largely observational and qualitative, lacking a formal causal model to separate context influence from intrinsic model priors.
3. Closed-Set Framing of Complex Demographics: Due to the discrete label requirements of VQA scoring, complex demographic attributes (e.g., gender, race) are modeled as discrete categorical sets during measurement.

Coverage note — None was omitted. All contributed methodologies, equations, algorithmic steps, quantitative benchmark tables, human study statistics, and key empirical findings are fully represented.

References

1.Sharat Agarwal, Sumanyu Muku, Saket Anand, and Chetan Arora. Does data repair lead to fair models? curating contextually fair data to reduce model bias. In WACV, 2022.
2.Omri Avrahami, Thomas Hayes, Oran Gafni, Sonal Gupta, Yaniv Taigman, Devi Parikh, Dani Lischinski, Ohad Fried, and Xi Yin. Spatext: Spatio-textual representation for controllable image generation. In CVPR, 2023.
3.Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. In NeurIPS, 2016.
4.Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint, 2021.
5.Manuel Brack, Felix Friedrich, Dominik Hintersdorf, Lukas Struppek, Patrick Schramowski, and Kristian Kersting. Sega: Instructing text-to-image models using semantic guidance. In NeurIPS, 2023.
6.Manuel Brack, Felix Friedrich, Patrick Schramowski, and Kristian Kersting. Mitigating inappropriateness in image generation: Can there be value in reflecting the world’s ugliness? arXiv preprint, 2023.
7.Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. In CVPR, 2023.
8.Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In NeurIPS, 2020.
9.Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, 2021.
10.Jun Chen, Deyao Zhu, Kilichbek Haydarov, Xiang Li, and Mohamed Elhoseiny. Video chatcaptioner: Towards the enriched spatiotemporal descriptions. arXiv preprint, 2023.
11.Jaemin Cho, Abhay Zala, and Mohit Bansal. Dall-eval: Probing the reasoning skills and social biases of text-to-image generation models. In ICCV, 2023.
12.Moreno D’Incà, Christos Tzelepis, Ioannis Patras, and Nicu Sebe. Improving fairness using vision-language driven image augmentation. In WACV, 2024.
13.Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
14.Dave Epstein, Allan Jabri, Ben Poole, Alexei A Efros, and Aleksander Holynski. Diffusion self-guidance for controllable image generation. In NeurIPS, 2023.
15.Felix Friedrich, Manuel Brack, Lukas Struppek, Dominik Hintersdorf, Patrick Schramowski, Sasha Luccioni, and Kristian Kersting. Fair diffusion: Instructing text-to-image generation models on fairness. arXiv preprint, 2023.
16.Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H. Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint, 2022.
17.Isabel O Gallegos, Ryan A Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, and Nesreen K Ahmed. Bias and fairness in large language models: A survey. arXiv preprint, 2023.
18.Vidit Goel, Elia Peruzzo, Yifan Jiang, Dejia Xu, Xingqian Xu, Nicu Sebe, Trevor Darrell, Zhangyang Wang, and Humphrey Shi. Pair-diffusion: A comprehensive multimodal object-level image editor. arXiv preprint, 2023.
19.Tanmay Gupta and Aniruddha Kembhavi. Visual programming: Compositional visual reasoning without training. In CVPR, 2023.
20.Lisa Anne Hendricks, Kaylee Burns, Kate Saenko, Trevor Darrell, and Anna Rohrbach. Women also snowboard: Overcoming bias in captioning models. In ECCV, 2018.
21.Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-prompt image editing with cross attention control. In ICLR, 2022.
22.Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021.
23.Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang, Mari Ostendorf, Ranjay Krishna, and Noah A. Smith. Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question answering. In ICCV, 2023.
24.Lianghua Huang, Di Chen, Yu Liu, Yujun Shen, Deli Zhao, and Jingren Zhou. Composer: Creative and controllable image synthesis with composable conditions. In ICML, 2023.
25.Sangwon Jung, Sanghyuk Chun, and Taesup Moon. Learning fair classifiers with partially annotated group labels. In CVPR, 2022.
26.Kimmo Karkkainen and Jungseock Joo. Fairface: Face attribute dataset for balanced race, gender, and age for bias measurement and mitigation. In WACV, 2021.
27.Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In CVPR, 2019.
28.Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Alias-free generative adversarial networks. In NeurIPS, 2021.
29.Patrik Joslin Kenfack, Kamil Sabbagh, Adín Ramírez Rivera, and Adil Khan. Repfair-gan: Mitigating representation bias in gans using gradient clipping. arXiv preprint, 2022.
30.Younghyun Kim, Sangwoo Mo, Minkyu Kim, Kyungmin Lee, Jaeho Lee, and Jinwoo Shin. Bias-to-text: Debiasing unknown visual biases through language interpretation. arXiv preprint, 2023.
31.Chenliang Li, Haiyang Xu, Junfeng Tian, Wei Wang, Ming Yan, Bin Bi, Jiabo Ye, Hehong Chen, Guohai Xu, Zheng Cao, et al. mPLUG: Effective and efficient vision-language learning by cross-modal skip-connections. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, 2022.
32.Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In ICML, 2022.
33.Tsung-Yi Lin, Michael Maire, Serge J. Belongie, Lubomir D. Bourdev, Ross B. Girshick, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. In ECCV, 2014.
34.Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In NeurIPS 2023 Workshop on Instruction Tuning and Instruction Following, 2023.
35.Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurIPS, 2023.
36.Ranjita Naik and Besmira Nushi. Social biases through the text-to-image generation lens. In Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society, 2023.
37.Junhyun Nam, Hyuntak Cha, Sungsoo Ahn, Jaeho Lee, and Jinwoo Shin. Learning from failure: De-biasing classifier from biased classifier. NeurIPS, 2020.
38.Roberto Navigli, Simone Conia, and Björn Ross. Biases in large language models: Origins, inventory, and discussion. ACM Journal of Data and Information Quality, 2023.
39.Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. In International Conference on Machine Learning, 2022.
40.Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. In Transactions on Machine Learning Research, 2023.
41.Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. SDXL: Improving latent diffusion models for high-resolution image synthesis. In ICLR, 2024.
42.Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
43.Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint, 2022.
44.Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
45.Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In CVPR, 2023.
46.Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. NeurIPS, 2022.
47.Yash Savani, Colin White, and Naveen Sundar Govindarajulu. Intra-processing methods for debiasing neural networks. In NeurIPS, 2020.
48.Robyn Speer, Joshua Chin, and Catherine Havasi. Conceptnet 5.5: An open multilingual graph of general knowledge. In AAAI, 2017.
49.Hongjin SU, Jungo Kasai, Chen Henry Wu, Weijia Shi, Tianlu Wang, Jiayi Xin, Rui Zhang, Mari Ostendorf, Luke Zettlemoyer, Noah A. Smith, and Tao Yu. Selective annotation makes language models better few-shot learners. In ICLR, 2023.
50.Xingzhe Su, Yi Ren, Wenwen Qiang, Zeen Song, Hang Gao, Fengge Wu, and Changwen Zheng. Unbiased image synthesis via manifold-driven sampling in diffusion models. arXiv preprint, 2023.
51.Sanjay Subramanian, William Merrill, Trevor Darrell, Matt Gardner, Sameer Singh, and Anna Rohrbach. Reclip: A strong zero-shot baseline for referring expression comprehension. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, 2022.
52.Dídac Surís, Sachit Menon, and Carl Vondrick. Vipergpt: Visual inference via python execution for reasoning. In ICCV, 2023.
53.Shuhan Tan, Yujun Shen, and Bolei Zhou. Improving the fairness of deep generative models without retraining. arXiv preprint, 2021.
54.Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint, 2023.
55.Sahil Verma and Julia Rubin. Fairness definitions explained. In Proceedings of the international workshop on software fairness, 2018.
56.Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework. In ICML, 2022.
57.Zeyu Wang, Klint Qinami, Ioannis Christos Karakozis, Kyle Genova, Prem Nair, Kenji Hata, and Olga Russakovsky. Towards fairness in visual recognition: Effective strategies for bias mitigation. In CVPR, 2020.
58.Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In NeurIPS, 2022.
59.Allen R Wilcox. Indices of qualitative variation. Technical report, Oak Ridge National Lab., Tenn., 1967.
60.Depeng Xu, Shuhan Yuan, Lu Zhang, and Xintao Wu. Fairgan: Fairness-aware generative adversarial networks. In 2018 IEEE International Conference on Big Data (Big Data). IEEE, 2018.
61.Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2014.
62.Cheng Zhang, Xuanbai Chen, Siqi Chai, Chen Henry Wu, Dmitry Lagun, Thabo Beeler, and Fernando De la Torre. Itigen: Inclusive text-to-image generation. In ICCV, 2023.
63.Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In ICCV, 2023.
64.Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. Men also like shopping: Reducing gender bias amplification using corpus-level constraints. In EMNLP, 2017.
65.Deyao Zhu, Jun Chen, Kilichbek Haydarov, Xiaoqian Shen, Wenxuan Zhang, and Mohamed Elhoseiny. Chatgpt asks, blip-2 answers: Automatic questioning towards enriched visual descriptions. In Transactions on Machine Learning Research, 2023.

Citation

MLA

D'Incà, M., et al. “OpenBias: Open-set Bias Detection in Text-to-Image Generative Models”. arXiv, 2024, http://arxiv.org/abs/2404.07990v2.

APA

D'Incà, M., Peruzzo, E., Mancini, M., Xu, D., Goel, V., Xu, X., Wang, Z., Shi, H., & Sebe, N. (2024). OpenBias: Open-set Bias Detection in Text-to-Image Generative Models. arXiv. http://arxiv.org/abs/2404.07990v2

Chicago

D'Incà, M., E. Peruzzo, M. Mancini, et al. 2024. “OpenBias: Open-set Bias Detection in Text-to-Image Generative Models”. arXiv. http://arxiv.org/abs/2404.07990v2.

Harvard

D'Incà, M. et al. (2024) “OpenBias: Open-set Bias Detection in Text-to-Image Generative Models”, arXiv [Preprint]. Available at: http://arxiv.org/abs/2404.07990v2.

Vancouver

1. D'Incà M, Peruzzo E, Mancini M, Xu D, Goel V, Xu X, Wang Z, Shi H, Sebe N (2024) OpenBias: Open-set Bias Detection in Text-to-Image Generative Models. arXiv

BibTeX

@article{dinca2024openbias,
  title = {OpenBias: Open-set Bias Detection in Text-to-Image Generative Models},
  author = {D'Incà, Moreno and Peruzzo, Elia and Mancini, Massimiliano and Xu, Dejia and Goel, Vidit and Xu, Xingqian and Wang, Zhangyang and Shi, Humphrey and Sebe, Nicu},
  year = {2024},
  journal = {arXiv},
  url = {http://arxiv.org/abs/2404.07990v2},
  eprint = {2404.07990}
}

Metadata:arXiv

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: IEEE

OpenBias: Open-Set Bias Detection in Text-to-Image Generative Models

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — OpenBias Pipeline for Open-Set Bias Detection

Knowl 2 — LLM-Driven Bias Proposal and Two-Stage Filtering

Knowl 3 — Context-Aware and Context-Free Bias Probability Formulation

Knowl 4 — Normalized Entropy Bias Severity Score

Knowl 5 — Zero-Shot VQA Backbone Selection on Synthetic Images

Knowl 6 — Distributional Alignment Between OpenBias VQA and FairFace

Knowl 7 — Human Agreement with OpenBias Severity and Bias Direction

Knowl 8 — Divergence Between Context-Aware and Context-Free Biases

Knowl 9 — Discovery of Novel Non-Social and Intersection-Specific Biases in Stable Diffusion

Knowl 10 — Methodological Limitations of OpenBias

References

Citation

BibTeX

Source Code

Access the Paper