4 mins read

Image to Drawings with Stable Diffusion XL and ComfyUI

Introduction

The ability to convert everyday images into black and white drawings can be a potential avenue for creative expression and business opportunities. The power of Stable Diffusion XL, paired with a custom ComfyUI workflow, has opened new possibilities for image to drawings transformations. This article explores the possibilities of image-to-sketch transformation, with a succinct a step-by-step guide to leverage this technology.


Use Cases and Monetization Ideas

  • Creative Projects: Artists and designers can use this technology to quickly generate sketch concepts from existing images, accelerating the creative process.
  • Education: In educational settings, this tool can help students understand the basics of sketching and design.
  • Merchandising: Businesses can create custom sketches from product images for marketing materials or merchandise designs.
  • Personalization Services: Offering personalized sketch services for special occasions like anniversaries or birthdays.

Setting Up the Workflow

The workflow is used in ComfyUI so will need to download it first from the official repository. If you have never used ComfyUI before, check this guide to begin. After that, you can download the actual workflow (a json file) and open the UI. After you loaded the workflow, you are ready to proceed. I recommend to install also ComfyUI manager to make things easier.

Preparing the Tools

The final workflow with all the custom nodes installed

To start, you need to download two custom nodes: ClipInterrogate.py and ResizeAspectratio.py. Place them in the custom_nodes folder of your ComfyUI installation. Next, to install these nodes, open your terminal in the ComfyUI folder and run:

ComfyUI_windows_portable\python_embeded\python.exe -s -m pip install clip-interrogator==0.6.0

Additional Node Installation

You might have noticed a message and some red nodes in your workflow. For these additional custom nodes, use the ComfyUI manager. It automatically locates and prompts you to install any missing nodes. Otherwise you can look for the missing nodes and put them as well in the custom_nodes folder.

Setting Up the IP Adapter

The IP Adapter is also needed for this workflow. Download the ip-adapter_sdxl.safeteensor from huggingface.co/h94/IP-Adapter and place it in the ComfyUI_IPAdapter_plus/models/ folder of the custom node.

Executing the Transformation

Image Selection and Processing

Choose an image and press ‘queue prompt’. Note that the first run will take much longer due to the download of models like blip2-2.7b, which are around 15GB in size.

After this first set up, each image take much less to generate, but it still requires several minutes on my 3060 GPU with 12 GB of VRAM.

Showcasing Results

In the final section of this article, I’ll share some examples I created using this workflow. You’ll see side-by-side comparisons of the original images and their sketch transformations. The results are impressive, but remember, tweaking parameters or trying other models can further refine your outputs.

It work better with bright images, it has some difficulties with arts that are in the shadow or too dark. Sometimes it also adds artifacts, but this is just the first step. Customization is always the kind, and the more you experiment with different parameters or model, the better the results will be for your specific needs.

Conclusion

The synergy of Stable Diffusion XL and ComfyUI’s custom workflow offers an accessible and powerful tool for transforming images into sketches. Whether for artistic, educational, or commercial purposes, this technology paves the way for endless creative possibilities.

Source: https://www.reddit.com/r/StableDiffusion/comments/17qql62/transforming_a_photo_into_a_black_and_white/

One thought on “Image to Drawings with Stable Diffusion XL and ComfyUI

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.