DiffusionGPT: Revolutionizing Text-to-Image Generation with LLM-Driven Model Selection

DiffusionGPT is an innovative text-to-image generation system that addresses the limitations of current stable diffusion models. It leverages Large Language Models (LLMs) to create a unified framework capable of handling diverse input prompts and integrating domain-expert models. This system aims to overcome challenges such as model limitations in specific domains and constraints in prompt types, offering a versatile solution for high-quality image generation.

DiffusionGPT consists of several key components: 1. Large Language Model (LLM): Acts as the core controller, guiding the entire workflow. 2. Prompt Parse Agent: Analyzes and extracts salient information from input prompts. 3. Tree-of-Thought (ToT) Structure: Organizes various generative models based on prior knowledge. 4. Model Selection Agent: Utilizes human feedback and advantage databases to select the most suitable model. 5. Prompt Extension Agent: Enhances input prompts to improve generation quality. 6. Domain-Expert Generative Models: A diverse range of models sourced from open-source communities.

The DiffusionGPT workflow consists of four main steps: 1. Prompt Parse: The LLM analyzes the input prompt and extracts core content. 2. Tree-of-Thought Model Building and Searching: Constructs and searches a model tree to identify candidate models. 3. Model Selection with Human Feedback: Selects the most suitable model using advantage databases and human preferences. 4. Execution of Generation: Utilizes the chosen model to generate high-quality images, incorporating prompt extension for improved results.

DiffusionGPT offers several advantages over traditional text-to-image generation methods: 1. Versatility: Handles diverse prompt types, including prompt-based, instruction-based, inspiration-based, and hypothesis-based inputs. 2. Improved Semantic Alignment: Generates images that better capture the overall semantic information of input prompts. 3. Enhanced Quality: Produces more detailed and accurate images, especially for human-related objects. 4. Flexibility: Easily integrates new models and adapts to different domains. 5. Human-Aligned: Incorporates human feedback to improve model selection and output quality.

Experiments demonstrate the effectiveness of DiffusionGPT: 1. Qualitative Results: Visual comparisons show improved semantic alignment and image aesthetics compared to baseline models like SD1.5 and SDXL. 2. Quantitative Results: DiffusionGPT outperforms baseline models in terms of image-reward and aesthetic scores. 3. User Study: Human evaluators consistently prefer images generated by DiffusionGPT over baseline models. 4. Ablation Studies: Demonstrate the effectiveness of the Tree-of-Thought structure, human feedback, and prompt extension components.

While DiffusionGPT shows promising results, there are areas for future improvement: 1. Feedback-Driven Optimization: Incorporating feedback directly into the LLM optimization process. 2. Expansion of Model Candidates: Enriching the model generation space with more diverse models. 3. Beyond Text-to-Image Tasks: Applying the DiffusionGPT framework to other tasks such as controllable generation, style migration, and attribute editing. Limitations include the need for a large model library and potential biases in human feedback. Ongoing research aims to address these challenges and further improve the system's performance and versatility.