ControlNet for SDXL - better control over AI generated images in Stable Diffusion XL (best)

[LechM]

The first implementation of ControlNet for the best free (open source) text-to-image model, Stable Diffusion XL (SDXL), can be found at https://github.com/huggingface/diffu...les/controlnet. ControlNet is a very popular addition to Stable Diffusion AI image generation, allowing for conditional inputs such as edge maps.

SDXL v1.0 was released two weeks ago, marking a significant upgrade from previous versions like Stable Diffusion 1.5 and 2.0. Although many still prefer the commercial product, MidJourney, ControlNet now makes it feasible to better control image generation and run it locally. MidJourney uses an awkward Discord-based interface. For those looking to run SDXL locally, there are several popular front ends available:
https://github.com/AUTOMATIC1111/stable-diffusion-webui,
https://github.com/invoke-ai/InvokeAI,
https://github.com/comfyanonymous/ComfyUI,
https://github.com/vladmandic/automatic.

Images created using the older version of ControlNet for Stable Diffusion 1.5:

[LechM]

The SDXL ControlNet with depth conditioning is now also available. You can take a photo, extract a depth map from it, and then use it to fill in the content. I've found that to work really well.



https://huggingface.co/diffusers/con...depth-sdxl-1.0