Sketch-to-Image Generation¶

The sketch-to-image generation module provides functionality to convert vector sketches into realistic images using various AI-driven image synthesis methods.

Source: tests/unit/test_sketch2img.py

Available Methods¶

The module currently supports the following methods:

`sketchkit.sketch2image.controlnet.ControlNetModel`([...])	ControlNet model for scribble-conditioned sketch-to-image conversion using SDXL.
`sketchkit.sketch2image.pix2pix.Pix2PixModel`([...])	InstructPix2Pix model for direct sketch-to-image translation using SDXL.
`sketchkit.sketch2image.t2i_adapter.T2IAdapterModel`([...])	T2I-Adapter model for edge/sketch-conditioned sketch-to-image conversion using SDXL.
`sketchkit.sketch2image.nano_banana2.NanoBanana2Model`([...])	Nano Banana 2 model using Google Gemini API.

ControlNet¶

The method implements Adding Conditional Control to Text-to-Image Diffusion Models (ControlNet) from ICCV 2023 using a scribble-based model (sd-controlnet-scribble). It tightly follows the spatial structure of the input sketch to generate a corresponding realistic image.

Output: Returns a PIL.Image (RGB mode).

Pix2Pix¶

The method implements InstructPix2Pix: Learning to Follow Image Editing Instructions (instruct-pix2pix) from CVPR 2023. It treats the generation as an image translation task, following instructions to convert the sketch context into a fully rendered image.

Output: Returns a PIL.Image (RGB mode).

T2I-Adapter¶

The method implements T2I-Adapter: Learning Adapters to Dig Out More Controllable Ability for Text-to-Image Diffusion Models from AAAI 2024 using a Canny edge adapter (t2iadapter_canny_sd15v2). It provides a lightweight way to guide the Stable Diffusion generation using the edges derived from the sketch.

Output: Returns a PIL.Image (RGB mode).

Nano Banana 2¶

This method uses Google’s Gemini 3.1 Flash Image Preview API for sketch-to-image generation. It leverages cloud-based multimodal AI to transform sketches into realistic images directly through API calls.

Requirements:

Get API key from: https://aistudio.google.com/
Add API key to ~/.sketchkit/config: GOOGLE_API_KEY="your_api_key"

Output: Returns a PIL.Image (RGB mode).

Code¶

The main interface is the ImageGenerator class.

from sketchkit.sketch2image import ImageGenerator
from sketchkit.core import Sketch

# Load an input (Sketch object or Image)
# sketch = ... 

# 1) ControlNet: Scribble-conditioned generation
cnet_gen = ImageGenerator(method="controlnet", device="cuda")
img_cnet = cnet_gen.run(
    sketch, 
    prompt="a realistic photo of a cat", 
    size=512
)
img_cnet.save("cat_real_cnet.png")

# 2) Pix2Pix: Instruction-based translation
pix_gen = ImageGenerator(method="pix2pix", device="cuda")
img_pix = pix_gen.run(
    sketch, 
    prompt="turn this sketch into a realistic photo", 
    size=512,
    image_guidance_scale=1.2
)
img_pix.save("cat_real_pix2pix.png")

# 3) T2I-Adapter: Edge-guided generation
adapter_gen = ImageGenerator(method="t2i_adapter", device="cuda")
img_adapter = adapter_gen.run(
    sketch, 
    prompt="cinematic lighting, highly detailed", 
    size=512
)
img_adapter.save("cat_real_adapter.png")

# 4) Nano Banana 2: Google Gemini API
nano_gen = ImageGenerator(method="nano_banana2")
img_nano = nano_gen.run(
    sketch,
    prompt="a realistic photo of a cat in a fancy restaurant",
    size=1024
)
img_nano.save("cat_real_nano.png")

Usage Notes¶

Constructor

ImageGenerator(method="controlnet", device="cuda", **kwargs)
- method: backend name. Supported values are "controlnet", "pix2pix", "t2i_adapter", and "nano_banana2".
- device: execution device, e.g., "cuda" or "cpu" (not used for "nano_banana2").

Run Method

run(input_data, prompt, size=None, input_size=None, **kwargs)
- input_data: accepts a Sketch object, PIL.Image, or numpy.ndarray.
- prompt: text description guiding the generation (default: "a realistic photo").
- size: output resolution. Accepts an integer (square) or (width, height) tuple.
- stroke_width (int): Width of lines when rasterizing vector sketches (default: 3).
- num_inference_steps (int): Diffusion steps (varies by model, usually 20-50).
- guidance_scale (float): Text prompt influence (default: 7.5).
- controlnet_conditioning_scale (float): Strength of ControlNet (default: 1.0).
- adapter_conditioning_scale (float): Strength of T2I-Adapter (default: 1.0).
- image_guidance_scale (float): Strength of input image influence for Pix2Pix (default: 1.0).
- seed (int): Random seed for reproducibility.