Directed Diffusion: Direct Control of Object Placement through Attention Guidance

Wan-Duo Kurt MaAvisek LahiriJohn P. LewisThomas LeungW. Bastiaan Kleijn

article2024AAAI91 citations

Introduces a training-free technique for text-to-image diffusion models that directly guides cross-attention maps during early denoising steps to accurately place multiple specified objects within user-defined bounding boxes.

Listen

Modern text-to-image artificial intelligence systems can generate high-quality visual content from descriptive prompts, but they routinely struggle to compose complex scenes involving multiple objects in precise arrangements. Because simple text descriptions cannot reliably specify spatial layouts, creators face tedious and costly trial-and-error cycles. This limitation presents a major barrier for practical storytelling applications, such as illustrated books and storyboards, where characters and objects must maintain intentional positional relationships.

The article introduces and evaluates Directed Diffusion, a lightweight method designed to give creators intuitive, high-level control over where multiple objects appear in synthesized images. The technique works on existing, pre-trained image generation systems without requiring expensive model retraining or fine-tuning.

The researchers developed an approach that operates during the early stages of the image generation process, when general layouts are established. By adjusting internal cross-attention maps—the representations that connect specific words in a prompt to corresponding image regions—the system steers object generation toward user-defined bounding boxes. To evaluate the method, the authors conducted comparative experiments against standard Stable Diffusion and existing state-of-the-art layout control methods, measuring prompt fidelity using standardized text-image similarity scores across single-object and complex multi-object scenes.

The evaluation revealed several key findings. First, Directed Diffusion successfully places specified objects within user-defined bounding boxes while producing natural contextual interactions, such as realistic shadows, lighting, and occlusions with the background. Second, the method achieved quantitative prompt-alignment scores (CLIP similarity) on par with or exceeding competing approaches, reaching 0.824 in complex scene composition compared to 0.821 for baseline Stable Diffusion and 0.791 for Composable Diffusion. Third, the authors demonstrated a companion technique, Placement Finetuning, which enables users to reposition an object within an existing scene while preserving the object's identity and background consistency without retraining.

These findings demonstrate that high-level spatial control can be achieved with minimal computational overhead, requiring only a few lines of code added to standard image generation pipelines. For organizations and creative teams, this significantly reduces the time and computing costs associated with repeated prompt engineering and eliminates the heavy infrastructure requirements of training specialized models from scratch.

The authors recommend integrating Directed Diffusion into creative production workflows for static visual storytelling formats, such as comic books and illustrated literature. Practitioners should pair this positional framework with complementary identity-preservation tools to maintain consistent character appearances across multiple narrative scenes.

The method inherits certain baseline limitations from underlying diffusion models, including occasional generation failures that still require seed exploration, as well as grammatical parsing limitations in language understanding models. Confidence in the reported results is high for static, multi-object image composition, though the authors caution that substantial technical advances remain necessary before these capabilities can be extended to video production.

Google NVIDIA Victoria University of Wellington attention guidance object placement text-guided diffusion

arXiv: 2302.13153

Paper: Prompt-to-Prompt Image Editing with Cross Attention Control, Amir Hertz et al. (2022). This foundational work establishes how manipulating cross-attention maps in diffusion models enables spatial and semantic control over generated visual elements, providing the underlying mechanism adapted by Directed Diffusion.
Paper: What the DAAM: Interpreting Stable Diffusion Using Cross Attention, Raphael Tang et al. (2023). This paper demonstrates how cross-attention heat maps bind specific prompt tokens to spatial regions in Stable Diffusion, providing the core interpretability insight that enables bounding-box attention steering.
Paper: Adding Conditional Control to Text-to-Image Diffusion Models, Lvmin Zhang et al. (2023). Understanding ControlNet's learned structural conditioning highlights the motivation for training-free, lightweight spatial steering alternatives like Directed Diffusion.
Paper: T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models, Chong Mou et al. (2023). This study demonstrates that spatial layout guidance is most critical in early diffusion steps, a principle directly utilized by Directed Diffusion's early-stage cross-attention guidance.
Paper: T2I-CompBench: A Comprehensive Benchmark for Open-world Compositional Text-to-image Generation, Kaiyi Huang et al. (2023). This benchmark details the specific failure modes of text-to-image models in spatial positioning and multi-object binding that Directed Diffusion explicitly targets.

Paper: MIGC: Multi-Instance Generation Controller for Text-to-Image Synthesis, Dewei Zhou et al. (2024). Extending bounding-box spatial guidance, MIGC introduces instance-level attention controllers and shading aggregation to prevent attribute leakage in complex multi-object layouts.
Paper: Towards Understanding Cross and Self-Attention in Stable Diffusion for Text-Guided Image Editing, Bingyan Liu et al. (2024). This work deepens the theoretical understanding of what cross-attention layers encode and demonstrates how to refine attention manipulation for tuning-free localized editing.
Paper: Intelligent Grimm - Open-ended Visual Storytelling via Latent Diffusion Models, Chang Liu et al. (2024). Building upon spatial object steering and identity preservation for storytelling, StoryGen applies context modules to maintain multi-frame character and scene consistency across narrative sequences.
Paper: Initno: Boosting Text-to-Image Diffusion Models via Initial Noise Optimization, Xiefan Guo et al. (2024). Complementing attention guidance during denoising, INITNO addresses multi-subject omission and blending by optimizing initial noise distributions prior to generation.
Paper: Everything in Its Place: Benchmarking Spatial Intelligence of Text-to-Image Models, Zengbin Wang et al. (2026). This benchmark provides a comprehensive framework to evaluate spatial intelligence and multi-object reasoning across generative models, evaluating the capabilities Directed Diffusion seeks to improve.

Cover for Directed Diffusion: Direct Control of Object Placement through Attention Guidance

Abstract

Text-guided diffusion models such as DALL-E 2, Imagen, eDiff-I, and Stable Diffusion are able to generate an effectively endless variety of images given only a short text prompt describing the desired image content. In many cases the images are of very high quality. However, these models often struggle to compose scenes containing several key objects such as characters in specified positional relationships. The missing capability to “direct” the placement of characters and objects both within and across images is crucial in storytelling, as recognized in the literature on film and animation theory. In this work, we take a particularly straightforward approach to provide the needed direction. Drawing on the observation that the cross-attention maps for prompt words reflect the spatial layout of objects denoted by those words, we introduce an optimization objective that produces “activation” at desired positions in these cross-attention maps. The resulting approach is a step toward generalizing the applicability of text-guided diffusion models beyond single images to collections of related images, as in storybooks. Directed Diffusion provides easy high-level positional control over multiple objects, while making use of an existing pre-trained model and maintaining a coherent blend between the positioned objects and the background. Moreover, it requires only a few lines to implement.

Keywords

attention guidance object placement text-guided diffusion cross-attention layers composable diffusion scene compositing latent diffusion Stable Diffusion image editing

1 Introduction
2 Related Work
3 Method
3.1 Pipeline
3.2 Cross-Attention Map Guidance
4 Applications
4.1 Scene compositing
4.2 Placement Finetuning
5 Experiments and Comparisons
5.1 Comparison: Scene compositing
5.2 Comparison: One Object
5.3 Placement finetuning
5.4 Comparison: Two Objects
5.5 Comparison: object interactions from prompt verbs
6 Limitations and Conclusion
7 Acknowledgements
References
8 Implementation
9 Analysis and Ablation
10 Placement finetuning
11 Scene compositing
12 Qualitative Evaluations
13 Quantitative Evaluation and Limitations
14 Societal Impact
14.1 Scene compositing: A white church under lightning in the pink sky
14.2 Scene compositing: Mystical trees next to a dark magical pond
14.3 Scene compositing: A castle in a forest with grainy fog
14.4 Scene compositing: cherry blossoms next to the lake and a mountain
14.5 One Object: bonfire next to a man
14.6 One Object: A white horse in front of the erupting volcano
14.7 One Object: A gravestone next to a man in red shirt
14.8 Two Objects: A white dog chasing a red sphere
14.9 Two Objects: A red cube above the blue sphere in the supermarket
14.10 Two Objects: MultiDiffusion
14.11 Intersection over Union

Knowls

Knowl 1 — Directed Diffusion Framework and Two-Stage Denoising Pipeline
model/method
Directed Diffusion (DD) provides spatial control over the placement of multiple objects in text-to-image diffusion models without requiring model retraining, fine-tuning, or per-word weight parameter tuning. The model takes a text prompt $\mathcal{P}$ and spatial region information $\mathcal{R} = \{\mathcal{B}, \mathcal{I}\}$ , where $\mathcal{B}$ specifies bounding boxes for desired object positions and $\mathcal{I} \subset \{1, \dots, |\mathcal{P}|\}$ denotes the token indices of corresponding directed words.

The reverse diffusion process spanning timesteps $t \in \{T, \dots, 0\}$ is partitioned into two distinct phases:
1. Attention Editing Stage ( $t \in [T, T-N)$ ): Modifies the cross-attention maps during the initial $N$ denoising steps (typically $N=10$ ). In this stage, neural activation within the bounding boxes $\mathcal{B}$ is amplified while surrounding activations are suppressed via optimization, establishing the coarse spatial layout and rough shape of directed objects early in synthesis.
2. Conventional Denoising Stage ( $t \in [T-N, 0]$ ): Resumes standard Latent Diffusion Model denoising with classifier-free guidance without further attention manipulation. This stage refines object details, textures, and contextual physical interactions (such as shadows and illumination) to seamlessly integrate placed objects into the background.
Knowl 2 — Target Cross-Attention Map Construction with Spatial Masks
model/method
In Directed Diffusion, target cross-attention maps $\mathbf{D}^{(i)}$ are constructed for each directed token index $i$ by modifying the existing cross-attention slice $\mathbf{A}^{(i)}$ using a weaken-mask $\mathbf{W}(\mathcal{B}')$ and a strengthen-mask $\mathbf{S}(\mathcal{B})$ .

Given a bounding box $\mathcal{B}$ with normalized coordinates $b = (b_{\text{left}}, b_{\text{right}}, b_{\text{top}}, b_{\text{bottom}}) \in [0, 1]^4$ on an attention map of spatial resolution $w \times h$ , the masks are defined coordinate-wise as:

$\mathbf{W}(\mathcal{B}')_{xy} = \begin{cases} c, & (x, y) \in \mathcal{B}' \\ 1, & \text{otherwise} \end{cases}$

$\mathbf{S}(\mathcal{B})_{xy} = \begin{cases} f(x, y), & (x, y) \in \mathcal{B} \\ 0, & \text{otherwise} \end{cases}$

where $\mathcal{B}'$ is the complement of region $\mathcal{B}$ , $c < 1$ is an attenuation scalar that suppresses attention outside the bounding box, and $f(x, y)$ is a 2D Gaussian activation window centered in $\mathcal{B}$ with standard deviations $\sigma_x = b_w / 2$ and $\sigma_y = b_h / 2$ . Here, $b_w = \lceil (b_{\text{right}} - b_{\text{left}}) \times w \rceil$ and $b_h = \lceil (b_{\text{top}} - b_{\text{bottom}}) \times h \rceil$ are the width and height of $\mathcal{B}$ .

The target attention map is then formed by:

$\mathbf{D}^{(i)} = \mathbf{A}^{(i)} \odot \mathbf{W}(\mathcal{B}') + \mathbf{S}(\mathcal{B})$

where $\odot$ represents element-wise multiplication.
Knowl 3 — Cross-Attention Optimization via Trailing Attention Maps
model/method
In Stable Diffusion, the CLIP text encoder produces fixed-length token representations $|W| = 77$ . For a prompt of length $|\mathcal{P}|$ , the cross-attention tensor contains prompt maps $\mathbf{A}^{(i)}$ for $i \le |\mathcal{P}|$ and trailing attention maps $\mathbf{A}^{(i)}$ for indices $i \in \mathcal{T} = \{|\mathcal{P}|+1, \dots, |W|\}$ corresponding to unused padding tokens.

To avoid instabilities and out-of-distribution latents caused by direct attention injection, Directed Diffusion optimizes a weight vector $\mathbf{a}_t \in \mathbb{R}^{|W|-|\mathcal{P}|}$ over the trailing attention maps at timestep $t$ to steer the directed prompt cross-attention maps $\mathbf{A}_{t-1}^{(i)}$ at step $t-1$ toward target maps $\mathbf{D}^{(i)}$ :

$\mathbf{a}^* = \arg\min_{\mathbf{a}_t} \sum_{i \in \mathcal{I}} \left\| \mathbf{A}_{t-1}^{(i)}\left(\mathbf{A}_t^{(|\mathcal{P}|+1:|W|)} \odot \mathbf{a}_t\right) - \mathbf{D}^{(i)} \right\|_2^2$

where $\mathcal{I}$ is the set of directed prompt word indices, $\odot$ denotes element-wise scaling of each trailing map by its corresponding weight in $\mathbf{a}_t$ , and $\mathbf{A}_{t-1}^{(i)}(\cdot)$ denotes the indirect functional dependency of the attention map at time $t-1$ on the re-weighted trailing maps applied during the denoising step at time $t$ . The optimized weights $\mathbf{a}^*$ reweight the trailing maps to influence U-Net conditioning at step $t$ .
Knowl 4 — Directed Diffusion Cross-Attention Editing Algorithm
algorithm
The cross-attention editing algorithm modifies the forward pass through the cross-attention layers of the denoising U-Net during the attention editing stage:
procedure DDCrossAttnEdit(DM, P, B)
    Input: Denoising model DM with latent z_t, prompt P, and bounding box specification B
    Output: Updated latent z_t
    for each layer l in layer(DM(z_t, P)) do
        if type(l) in CrossAttn then
            Compute cross-attention A = Softmax(Q_l(z_t) * K_l(P)^T / sqrt(d))
            for each token index i in T union I do
                Compute target D^(i) = A^(i) * W(B') + S(B)
            end for
            Optimize weight vector a* = argmin_a L_a
            Update trailing attention slices A'^(|P|+1:77) = D^(|P|+1:77) * a*
            Update latent z_t = l(z_t, A' * V_l(P))
        else
            Update latent z_t = l(z_t)
        end if
    end for
    return z_t
Here, $Q_l, K_l, V_l$ denote the query, key, and value projection layers in cross-attention layer $l$ , $\mathcal{I}$ denotes the directed prompt token indices, $\mathcal{T} = \{|\mathcal{P}|+1, \dots, 77\}$ denotes trailing attention map indices, $W(\mathcal{B}')$ and $S(\mathcal{B})$ are weakening and strengthening masks, and $L_a$ is the trailing map optimization loss.
Knowl 5 — Multi-Object Scene Compositing via Latent Interpolation
model/method
When directing more than two bounding boxes simultaneously, direct multi-box attention editing can become unreliable. Directed Diffusion resolves this by independently generating latents for each directed object and performing linear spatial interpolation during the initial $N$ denoising steps.

For $R$ directed objects with bounding boxes $\mathcal{B}_r$ ( $r \in \{1, \dots, R\}$ ), individual denoising paths record the intermediate latents $\mathbf{z}_t^{(r)}$ . The combined scene latent $\mathbf{z}_t$ conditioned on the global prompt $\mathcal{P}$ is updated for spatial coordinates $(x, y) \in \mathcal{B}_r$ during timesteps $t \in [T, T-N]$ according to:

$\mathbf{z}_t(x, y) := \frac{1}{R} \sum_{r=1}^R \left[ w_r \mathbf{z}_t(x, y) + (1 - w_r) \mathbf{z}_t^{(r)}(x, y) \right]$

where $w_r \in \mathbb{R}$ is a weighting scalar (typically set to $0.1$ ) and $N \approx 10$ . Subsequent denoising steps ( $t < T-N$ ) follow standard diffusion without interpolation, allowing the model to produce unified lighting and contextual interactions across all objects.
Knowl 6 — Placement Finetuning for Identity-Preserving Object Repositioning
model/method
Placement Finetuning (PF) reposition an object within an already generated image while preserving its synthesized identity and visual characteristics without model retraining.

The algorithm comprises three components:
1. Segmentation and Spatial Transformation: An object mask $\mathbf{M}_o$ is extracted by thresholding the final $t=0$ cross-attention map and clipping to bounding box $\mathcal{B}$ . At step $T-N$ , a spatial translation transformation $\mathbf{X}(\cdot)$ (implemented via cyclic tensor shifting) is applied to the stored latent $\mathbf{z}_{T-N}$ and mask $\mathbf{M}_o$ , producing translated mask $\mathbf{X}(\mathbf{M}_o)$ and translated latent $\mathbf{X}(\mathbf{z}_{T-N})$ .
2. Background Inpainting: The hole left by moving the foreground object is initialized using the transformed latent $\mathbf{X}(\mathbf{z}_T)$ masked by $\mathbf{M}_o$ against the background complement $\neg \mathbf{M}_o$ , followed by a single noise-and-denoise step to synthesize background texture.
3. Latent Blending: For remaining timesteps $t \in \{T-N-1, \dots, 0\}$ , the latent is composited at each step via:
$\mathbf{z}'_t := \mathbf{z}'_t \odot \neg \mathbf{X}(\mathbf{M}_o) + \mathbf{X}(\mathbf{z}_t) \odot \mathbf{X}(\mathbf{M}_o)$

followed by standard denoising. Setting $N \approx 10$ provides a trade-off where larger $N$ preserves foreground and background identity while smaller $N$ increases contextual interaction.
Knowl 7 — Success-at-k (SS@k) Evaluation Protocol
experimental setup
To model realistic generative text-to-image usage where users iterate over random seeds rather than relying on a single output, the Success-at- $k$ (SS@ $k$ ) protocol generates $k$ candidate images using consecutive random seeds beginning from an initial seed. The highest-quality image among the $k$ candidates is subjectively selected for qualitative and quantitative evaluation. In the evaluations of Directed Diffusion, $k=12$ (SS@12) is used across all comparative benchmarks.

Knowl 8 — Quantitative Evaluation of Text-Image Alignment Across Placement Benchmarks

data/table

Text-image alignment was quantitatively assessed using CLIP similarity scores between prompt embeddings and generated images under the SS@12 protocol across three categories: Scene Composition (SceneComp), single-object placement (OneMask), and two-object placement (TwoMasks).

Directed Diffusion (DD) outperforms GLIGEN and Composable Diffusion (CD) on multi-object benchmarks (achieving 0.824 on SceneComp and 0.851 on TwoMasks), while remaining competitive with standard Stable Diffusion (SD) and GLIGEN on single-mask generation (0.807 vs 0.834 for SD and 0.810 for GLIGEN).

Knowl 9 — Limitations of Directed Diffusion
limitation
Directed Diffusion exhibits several limitations:
1. Dependency on Base Diffusion Model Robustness: It inherits baseline Stable Diffusion failure modes, including seed sensitivity where particular seeds yield distorted object anatomy, missing limbs, or omitted prompt subjects.
2. Sensitivity to the Editing Step Hyperparameter: The number of active editing steps $N$ must be manually specified. Excessively large $N$ suppresses natural contextual interaction (e.g., lighting and shadows) with the scene, while overly small $N$ fails to enforce positional placement.
3. Scalability with Simultaneous Bounding Boxes: Direct attention guidance becomes unreliable when guiding more than two bounding boxes simultaneously without employing multi-pass latent interpolation.
4. Text Encoder Semantic and Grammatical Constraints: Due to CLIP's limited understanding of complex grammar, prepositional bindings, and transitive verbs, specifying action relationships between directed objects (e.g., distinguishing 'a dog chasing a ball' from 'a dog and a ball') remains inconsistent despite accurate spatial positioning.

Coverage note — Visual qualitative figure demonstrations from the paper were omitted as standalone knowls because their core conceptual and empirical claims are fully captured in the methodology, experimental setup, evaluation table, and limitation knowls.

References

1.Arijon, D. 1976. Grammar of the Film Language. Focal Press.
2.Avrahami, O.; Fried, O.; and Lischinski, D. 2022. Blended Latent Diffusion. CoRR, abs/2206.02779.
3.Avrahami, O.; Hayes, T.; Gafni, O.; Gupta, S.; Taigman, Y.; Parikh, D.; Lischinski, D.; Fried, O.; and Yin, X. 2022. SpaText: Spatio-Textual Representation for Controllable Image Generation. CoRR, abs/2211.14305.
4.Balaji, Y.; Nah, S.; Huang, X.; Vahdat, A.; Song, J.; Kreis, K.; Aittala, M.; Aila, T.; Laine, S.; Catanzaro, B.; Karras, T.; and Liu, M. 2022. eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers. CoRR, abs/2211.01324.
5.Bar-Tal, O.; Yariv, L.; Lipman, Y.; and Dekel, T. 2023. MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generation. CoRR, abs/2302.08113.
6.Chang, H.; Zhang, H.; Barber, J.; Maschinot, A.; Lezama, J.; Jiang, L.; Yang, M.; Murphy, K.; Freeman, W. T.; Rubinstein, M.; Li, Y.; and Krishnan, D. 2023. Muse: Text-To-Image Generation via Masked Generative Transformers. CoRR, abs/2301.00704.
7.Feng, W.; He, X.; Fu, T.; Jampani, V.; Akula, A. R.; Narayana, P.; Basu, S.; Wang, X. E.; and Wang, W. Y. 2022. Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesis. CoRR, abs/2212.05032.
8.Gal, R.; Alaluf, Y.; Atzmon, Y.; Patashnik, O.; Bermano, A. H.; Chechik, G.; and Cohen-Or, D. 2022. An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion. CoRR, abs/2208.01618.
9.Hessel, J.; Holtzman, A.; Forbes, M.; Bras, R. L.; and Choi, Y. 2021. CLIPScore: A Reference-free Evaluation Metric for Image Captioning. In EMNLP.
10.Ho, J.; Jain, A.; and Abbeel, P. 2020. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33.
11.Ho, J.; and Salimans, T. 2021. Classifier-free diffusion guidance. In NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications.
12.Hyv"arinen, A. 2005. Estimation of Non-Normalized Statistical Models by Score Matching. J. Mach. Learning Research, 6: 695–709.
13.Kingma, D. P.; and Welling, M. 2013. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114.
14.Li, Y.; Liu, H.; Wu, Q.; Mu, F.; Yang, J.; Gao, J.; Li, C.; and Lee, Y. J. 2023. GLIGEN: Open-Set Grounded Text-to-Image Generation. CoRR, abs/2301.07093.
15.Liew, J. H.; Yan, H.; Zhou, D.; and Feng, J. 2022. MagicMix: Semantic Mixing with Diffusion Models. CoRR, abs/2210.16056.
16.Liu, N.; Li, S.; Du, Y.; Torralba, A.; and Tenenbaum, J. B. 2022. Compositional Visual Generation with Composable Diffusion Models. In ECCV.
17.Ma, W.-D. K.; Lewis, J. P.; Lahiri, A.; Leung, T.; and Kleijn, W. B. 2023. Directed Diffusion: Direct Control of Object Placement through Attention Guidance. arXiv:2302.13153.
18.Meng, C.; He, Y.; Song, Y.; Song, J.; Wu, J.; Zhu, J.; and Ermon, S. 2022. SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations. In Int. Conf. on Learning Representations (ICLR).
19.Nichol, A. Q.; Dhariwal, P.; Ramesh, A.; Shyam, P.; Mishkin, P.; McGrew, B.; Sutskever, I.; and Chen, M. 2022. GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models. In ICML.
20.Park, D. H.; Luo, G.; Toste, C.; Azadi, S.; Liu, X.; Karalashvili, M.; Rohrbach, A.; and Darrell, T. 2022. Shape-Guided Diffusion with Inside-Outside Attention.
21.Patashnik, O.; Wu, Z.; Shechtman, E.; Cohen-Or, D.; and Lischinski, D. 2021. Styleclip: Text-driven manipulation of stylegan imagery. In ICCV.
22.Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; Krueger, G.; and Sutskever, I. 2021. Learning Transferable Visual Models From Natural Language Supervision. In Proc. ICML.
23.Ramesh, A.; Dhariwal, P.; Nichol, A.; Chu, C.; and Chen, M. 2022. Hierarchical Text-Conditional Image Generation with CLIP Latents. CoRR, abs/2204.06125.
24.Rombach, R.; Blattmann, A.; Lorenz, D.; Esser, P.; and Ommer, B. 2022. High-Resolution Image Synthesis With Latent Diffusion Models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 10684–10695.
25.Ruiz, N.; Li, Y.; Jampani, V.; Pritch, Y.; Rubinstein, M.; and Aberman, K. 2022. DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation. CoRR, abs/2208.12242.
26.Saharia, C.; Chan, W.; Saxena, S.; Li, L.; Whang, J.; Denton, E.; Ghasemipour, S. K. S.; Ayan, B. K.; Mahdavi, S. S.; Lopes, R. G.; Salimans, T.; Ho, J.; Fleet, D. J.; and Norouzi, M. 2022. Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding. CoRR, abs/2205.11487.
27.Smith, E. 2022. A Traveler’s Guide to the Latent Space.
28.Sohl-Dickstein, J.; Weiss, E.; Maheswaranathan, N.; and Ganguli, S. 2015. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning.
29.Song, J.; Meng, C.; and Ermon, S. 2021. Denoising Diffusion Implicit Models. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021.
30.Song, Y.; and Ermon, S. 2019. Generative modeling by estimating gradients of the data distribution. In NeurIPS, volume 32.
31.StabilityAI. 2023. Stable Diffusion 2.1 Demo. https://huggingface.co/spaces/stabilityai/stable-diffusion. Accessed: 2024-01-01.
32.Thomas, F.; and Johnston, O. 1981. Disney animation : the illusion of life. Abbeville Press New York, 1st ed. edition. ISBN 0896592332 0896592324.
33.Weng, L. 2021. What are diffusion models?
34.Xie, J.; Li, Y.; Huang, Y.; Liu, H.; Zhang, W.; Zheng, Y.; and Shou, M. Z. 2023. BoxDiff: Text-to-Image Synthesis with Training-Free Box-Constrained Diffusion. CoRR, abs/2307.10816.
35.Yu, J.; Xu, Y.; Koh, J. Y.; Luong, T.; Baid, G.; Wang, Z.; Vasudevan, V.; Ku, A.; Yang, Y.; Ayan, B. K.; Hutchinson, B.; Han, W.; Parekh, Z.; Li, X.; Zhang, H.; Baldridge, J.; and Wu, Y. 2022. Parti: Pathways Autoregressive Text-to-Image model.
36.Zhang, L.; and Agrawala, M. 2023. Adding Conditional Control to Text-to-Image Diffusion Models. arXiv:2302.05543.

Citation

MLA

Ma, W.-D. K., et al. “Directed Diffusion: Direct Control of Object Placement Through Attention Guidance”. arXiv, 2023, http://arxiv.org/abs/2302.13153v3.

APA

Ma, W.-D. K., Lewis, J. P., Lahiri, A., Leung, T., & Kleijn, W. B. (2023). Directed Diffusion: Direct Control of Object Placement through Attention Guidance. arXiv. http://arxiv.org/abs/2302.13153v3

Chicago

Ma, W.-D. K., J. P. Lewis, A. Lahiri, T. Leung, and W. B. Kleijn. 2023. “Directed Diffusion: Direct Control of Object Placement Through Attention Guidance”. arXiv. http://arxiv.org/abs/2302.13153v3.

Harvard

Ma, W.-D.K. et al. (2023) “Directed Diffusion: Direct Control of Object Placement through Attention Guidance”, arXiv [Preprint]. Available at: http://arxiv.org/abs/2302.13153v3.

Vancouver

1. Ma W-DK, Lewis JP, Lahiri A, Leung T, Kleijn WB (2023) Directed Diffusion: Direct Control of Object Placement through Attention Guidance. arXiv

BibTeX

@article{ma2023directed,
  title = {Directed Diffusion: Direct Control of Object Placement through Attention Guidance},
  author = {Ma, Wan-Duo Kurt and Lewis, J. P. and Lahiri, Avisek and Leung, Thomas and Kleijn, W. Bastiaan},
  year = {2023},
  journal = {arXiv},
  url = {http://arxiv.org/abs/2302.13153v3},
  eprint = {2302.13153}
}

Metadata:arXiv

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

Eval Type	SD	GLIGEN	CD	DD
SceneComp	0.821	0.802	0.791	0.824
OneMask	0.834	0.810	-	0.807
TwoMasks	0.842	0.802	-	0.851

Directed Diffusion: Direct Control of Object Placement through Attention Guidance

Abstract

Keywords

Table of Contents

Knowls

Knowl 1 — Directed Diffusion Framework and Two-Stage Denoising Pipeline

Knowl 2 — Target Cross-Attention Map Construction with Spatial Masks

Knowl 3 — Cross-Attention Optimization via Trailing Attention Maps

Knowl 4 — Directed Diffusion Cross-Attention Editing Algorithm

Knowl 5 — Multi-Object Scene Compositing via Latent Interpolation

Knowl 6 — Placement Finetuning for Identity-Preserving Object Repositioning

Knowl 7 — Success-at-k (SS@k) Evaluation Protocol

Knowl 8 — Quantitative Evaluation of Text-Image Alignment Across Placement Benchmarks

Knowl 9 — Limitations of Directed Diffusion

References

Citation

BibTeX

Access the Paper