Towards Transformer-Based Aligned Generation with Self-Coherence Guidance

Code
Task Image

Our method directly optimizes the cross-attention maps in Transformer-based diffusion models, significantly enhancing the model's performance in coarse-grained attribute binding and further improving fine-grained attribute and style binding. For instance, our approach enables precise control over the color of an apple’s flesh and stem as well as the style of two distinct concepts.

Abstract

We introduce a novel, training-free approach for enhancing alignment in Transformer-based Text-Guided Diffusion Models (TGDMs). Existing TGDMs often struggle to generate semantically aligned images, particularly when dealing with complex text prompts or multi-concept attribute binding challenges.Previous U-Net-based methods primarily optimized the latent space, but their direct application to Transformer-based architectures has shown limited effectiveness. Our method addresses these challenges by directly optimizing cross-attention maps during the generation process. Specifically, we introduce Self-Coherence Guidance, a method that dynamically refines attention maps using masks derived from previous denoising steps, ensuring precise alignment without additional training. To validate our approach, we constructed more challenging benchmarks for evaluating coarse-grained attribute binding, fine-grained attribute binding, and style binding. Experimental results demonstrate the superior performance of our method, significantly surpassing other state-of-the-art methods across all evaluated tasks.

Type Image

(a) Overview of our method. Given a prompt, we extract the corresponding concept masks and use these masks to directly guide the attribute or style maps. (b) For fine-grained attribute binding, we extract masks by planning the proportions using LLMs. (c) For coarse-grained attribute binding and style binding, we directly apply clustering methods to extract the corresponding masks.

Task Image

Qualitative analysis of our method compared to other SOTA methods.Our approach consistently generates high-quality images with superior alignment across coarse-grained attribute binding, fine-grained attribute binding, and style binding tasks..

Third Image

During the generation process, the attention entropy across different layers in U-Net and DiT architectures reflects the semantic richness, with lower attention entropy indicating greater semantic information.The visualized token corresponds to the word "balloon".