sketchkit.sketch2image package

Subpackages

Submodules

sketchkit.sketch2image.image_generator module

ImageGenerator for sketch-to-image conversion tasks.

This module provides the core ImageGenerator class, which acts as a unified interface for converting vector sketches or rasterized images into realistic images using various AI models (ControlNet, T2I-Adapter, InstructPix2Pix).

class sketchkit.sketch2image.image_generator.ImageGenerator(method: str = 'controlnet', device: str = 'cuda', **kwargs: Any)[source]

Bases: object

A class for sketch-to-image conversion.

This class provides functionality to convert vector sketches to realistic images using various AI methods. It automatically handles the preprocessing of inputs (such as rasterizing vector sketches or applying edge detection) and routes the request to the appropriate generative model.

method

The conversion method that is used (e.g., ‘controlnet’).

Type:

str

device

The computation device (‘cuda’ or ‘cpu’).

Type:

str

_model

The loaded AI model instance.

_prepare_control_image(image: ndarray | Image) → Image[source]

Standardizes the control image format to a PIL Image.

Parameters:

image (Union[np.ndarray, Image.Image]) – The raw input image array or PIL object.

Returns:

The standardized PIL Image in grayscale (‘L’) or RGB mode depending on input.

Return type:

Image.Image

Raises:

TypeError – If the input type is unsupported.

_prepare_t2i_adapter_image(image: ndarray | Image) → Image[source]

Applies Canny edge detection specifically for the T2I-Adapter model.

Parameters:

image (Union[np.ndarray, Image.Image]) – The input raster image.

Returns:

A PIL Image containing the extracted edges in grayscale (‘L’) mode.

Return type:

Image.Image

_run_controlnet(control_image: Image, prompt: str, size: int | Tuple[int, int] | None = None, **kwargs: Any) → Image[source]

Executes the ControlNet model for image generation.

Parameters:

  • control_image (Image.Image) – The conditioning sketch image.

  • prompt (str) – Text prompt.

  • size (Union[int, Tuple[int, int], None]) – Output size.

  • **kwargs (Any) – Additional parameters.

Returns:

The generated image.

Return type:

Image.Image

_run_nano_banana2(control_image: Image, prompt: str, size: int | Tuple[int, int] | None = None, **kwargs) → Image[source]

Execute Nano Banana 2 (Google Gemini) model.

_run_pix2pix(control_image: Image, prompt: str, size: int | Tuple[int, int] | None = None, **kwargs: Any) → Image[source]

Executes the InstructPix2Pix model for direct image translation.

Parameters:

  • control_image (Image.Image) – The input image to be translated.

  • prompt (str) – Instruction prompt (e.g., “make it a photo”).

  • size (Union[int, Tuple[int, int], None]) – Output size.

  • **kwargs (Any) – Additional parameters.

Returns:

The generated image.

Return type:

Image.Image

_run_t2i_adapter(control_image: Image, prompt: str, size: int | Tuple[int, int] | None = None, **kwargs: Any) → Image[source]

Executes the T2I-Adapter model for image generation.

Parameters:

  • control_image (Image.Image) – The conditioning sketch/edge image.

  • prompt (str) – Text prompt.

  • size (Union[int, Tuple[int, int], None]) – Output size.

  • **kwargs (Any) – Additional parameters.

Returns:

The generated image.

Return type:

Image.Image

_run_with_image(image: Image | ndarray, prompt: str, size: int | Tuple[int, int] | None, input_size: Tuple[int, int] | None, **kwargs: Any) → Image[source]

Processes a raster image (PIL or numpy array) for sketch-to-image conversion.

Parameters:

  • image (Union[Image.Image, np.ndarray]) – The input raster image.

  • prompt (str) – Textual prompt guiding the generation.

  • size (Optional[Union[int, Tuple[int, int]]]) – Output image size.

  • input_size (Optional[Tuple[int, int]]) – Target size to resize the input image.

  • **kwargs (Any) – Additional generation parameters.

Returns:

The generated image.

Return type:

Image.Image

Raises:

  • ValueError – If the numpy array shape is not supported.

  • TypeError – If the input type is unknown.

_run_with_sketch(sketch: Sketch, prompt: str, size: int | Tuple[int, int] | None, input_size: Tuple[int, int] | None, **kwargs: Any) → Image[source]

Processes a vector Sketch object for sketch-to-image conversion.

Parameters:

  • sketch (Sketch) – The input vector sketch.

  • prompt (str) – Textual prompt guiding the generation.

  • size (Optional[Union[int, Tuple[int, int]]]) – Target output dimensions.

  • input_size (Optional[Tuple[int, int]]) – Resolution for rasterizing the sketch.

  • **kwargs (Any) – Additional pipeline parameters.

Returns:

The generated image.

Return type:

Image.Image

_sketch_to_raster(sketch: Sketch, size: int | Tuple[int, int] = 1024, background_color: Tuple[float, float, float] = (1, 1, 1), stroke_width: int = 3, fit_canvas: bool = True) → ndarray[source]

Converts a vector Sketch to a rasterized numpy array.

Parameters:

  • sketch (Sketch) – The vector sketch to render.

  • size (Union[int, Tuple[int, int]]) – Canvas dimensions. Defaults to 1024.

  • background_color (Tuple[float, float, float]) – RGB background color. Defaults to white.

  • stroke_width (int) – Thickness of the rendered strokes. Defaults to 3.

  • fit_canvas (bool) – Whether to scale paths to fit the canvas. Defaults to True.

Returns:

The rasterized image as a numpy array.

Return type:

np.ndarray

run(input_data: Sketch | Image | ndarray, prompt: str = 'a realistic photo', size: int | Tuple[int, int] | None = None, input_size: Tuple[int, int] | None = None, **kwargs: Any) → Image[source]

Converts the input sketch or image to a realistic image.

Parameters:

  • input_data (Union[Sketch, Image.Image, np.ndarray]) – The input representation. Can be: - Sketch: A vector sketch object. - Image.Image: A PIL Image object. - np.ndarray: A Numpy array representing an image.

  • prompt (str) – Text description guiding the image generation. Defaults to “a realistic photo”.

  • size (Optional[Union[int, Tuple[int, int]]]) – Output image size (int for square, tuple for rectangle). If None, it infers the target size from the input.

  • input_size (Optional[Tuple[int, int]]) – Target dimensions (width, height) to resize the input to before processing.

  • **kwargs (Any) – Keyword arguments for Generation Control: - stroke_width (int): Stroke width for rasterization. Defaults to 3. - fit_canvas (bool): Whether to fit sketch to canvas. Defaults to True. - num_inference_steps (int): Number of denoising steps. Defaults to 30. - guidance_scale (float): CFG scale for text guidance. Defaults to 7.5. - controlnet_conditioning_scale (float): Strength of ControlNet. Defaults to 1.0. - adapter_conditioning_scale (float): Strength of T2I-Adapter. Defaults to 1.0. - image_guidance_scale (float): Strength of InstructPix2Pix image guidance. Defaults to 1.5. - negative_prompt (str): Prompt detailing what to avoid. - seed (int): Random seed for reproducibility.

Returns:

The generated realistic image.

Return type:

Image.Image

Raises:

TypeError – If the input_data is not of a supported type.

Module contents

Sketch-to-Image Conversions

This module provides a unified API for converting vector sketches into realistic images.

class sketchkit.sketch2image.ImageGenerator(method: str = 'controlnet', device: str = 'cuda', **kwargs: Any)[source]

Bases: object

A class for sketch-to-image conversion.

This class provides functionality to convert vector sketches to realistic images using various AI methods. It automatically handles the preprocessing of inputs (such as rasterizing vector sketches or applying edge detection) and routes the request to the appropriate generative model.

method

The conversion method that is used (e.g., ‘controlnet’).

Type:

str

device

The computation device (‘cuda’ or ‘cpu’).

Type:

str

_model

The loaded AI model instance.

_prepare_control_image(image: ndarray | Image) → Image[source]

Standardizes the control image format to a PIL Image.

Parameters:

image (Union[np.ndarray, Image.Image]) – The raw input image array or PIL object.

Returns:

The standardized PIL Image in grayscale (‘L’) or RGB mode depending on input.

Return type:

Image.Image

Raises:

TypeError – If the input type is unsupported.

_prepare_t2i_adapter_image(image: ndarray | Image) → Image[source]

Applies Canny edge detection specifically for the T2I-Adapter model.

Parameters:

image (Union[np.ndarray, Image.Image]) – The input raster image.

Returns:

A PIL Image containing the extracted edges in grayscale (‘L’) mode.

Return type:

Image.Image

_run_controlnet(control_image: Image, prompt: str, size: int | Tuple[int, int] | None = None, **kwargs: Any) → Image[source]

Executes the ControlNet model for image generation.

Parameters:

  • control_image (Image.Image) – The conditioning sketch image.

  • prompt (str) – Text prompt.

  • size (Union[int, Tuple[int, int], None]) – Output size.

  • **kwargs (Any) – Additional parameters.

Returns:

The generated image.

Return type:

Image.Image

_run_nano_banana2(control_image: Image, prompt: str, size: int | Tuple[int, int] | None = None, **kwargs) → Image[source]

Execute Nano Banana 2 (Google Gemini) model.

_run_pix2pix(control_image: Image, prompt: str, size: int | Tuple[int, int] | None = None, **kwargs: Any) → Image[source]

Executes the InstructPix2Pix model for direct image translation.

Parameters:

  • control_image (Image.Image) – The input image to be translated.

  • prompt (str) – Instruction prompt (e.g., “make it a photo”).

  • size (Union[int, Tuple[int, int], None]) – Output size.

  • **kwargs (Any) – Additional parameters.

Returns:

The generated image.

Return type:

Image.Image

_run_t2i_adapter(control_image: Image, prompt: str, size: int | Tuple[int, int] | None = None, **kwargs: Any) → Image[source]

Executes the T2I-Adapter model for image generation.

Parameters:

  • control_image (Image.Image) – The conditioning sketch/edge image.

  • prompt (str) – Text prompt.

  • size (Union[int, Tuple[int, int], None]) – Output size.

  • **kwargs (Any) – Additional parameters.

Returns:

The generated image.

Return type:

Image.Image

_run_with_image(image: Image | ndarray, prompt: str, size: int | Tuple[int, int] | None, input_size: Tuple[int, int] | None, **kwargs: Any) → Image[source]

Processes a raster image (PIL or numpy array) for sketch-to-image conversion.

Parameters:

  • image (Union[Image.Image, np.ndarray]) – The input raster image.

  • prompt (str) – Textual prompt guiding the generation.

  • size (Optional[Union[int, Tuple[int, int]]]) – Output image size.

  • input_size (Optional[Tuple[int, int]]) – Target size to resize the input image.

  • **kwargs (Any) – Additional generation parameters.

Returns:

The generated image.

Return type:

Image.Image

Raises:

  • ValueError – If the numpy array shape is not supported.

  • TypeError – If the input type is unknown.

_run_with_sketch(sketch: Sketch, prompt: str, size: int | Tuple[int, int] | None, input_size: Tuple[int, int] | None, **kwargs: Any) → Image[source]

Processes a vector Sketch object for sketch-to-image conversion.

Parameters:

  • sketch (Sketch) – The input vector sketch.

  • prompt (str) – Textual prompt guiding the generation.

  • size (Optional[Union[int, Tuple[int, int]]]) – Target output dimensions.

  • input_size (Optional[Tuple[int, int]]) – Resolution for rasterizing the sketch.

  • **kwargs (Any) – Additional pipeline parameters.

Returns:

The generated image.

Return type:

Image.Image

_sketch_to_raster(sketch: Sketch, size: int | Tuple[int, int] = 1024, background_color: Tuple[float, float, float] = (1, 1, 1), stroke_width: int = 3, fit_canvas: bool = True) → ndarray[source]

Converts a vector Sketch to a rasterized numpy array.

Parameters:

  • sketch (Sketch) – The vector sketch to render.

  • size (Union[int, Tuple[int, int]]) – Canvas dimensions. Defaults to 1024.

  • background_color (Tuple[float, float, float]) – RGB background color. Defaults to white.

  • stroke_width (int) – Thickness of the rendered strokes. Defaults to 3.

  • fit_canvas (bool) – Whether to scale paths to fit the canvas. Defaults to True.

Returns:

The rasterized image as a numpy array.

Return type:

np.ndarray

run(input_data: Sketch | Image | ndarray, prompt: str = 'a realistic photo', size: int | Tuple[int, int] | None = None, input_size: Tuple[int, int] | None = None, **kwargs: Any) → Image[source]

Converts the input sketch or image to a realistic image.

Parameters:

  • input_data (Union[Sketch, Image.Image, np.ndarray]) – The input representation. Can be: - Sketch: A vector sketch object. - Image.Image: A PIL Image object. - np.ndarray: A Numpy array representing an image.

  • prompt (str) – Text description guiding the image generation. Defaults to “a realistic photo”.

  • size (Optional[Union[int, Tuple[int, int]]]) – Output image size (int for square, tuple for rectangle). If None, it infers the target size from the input.

  • input_size (Optional[Tuple[int, int]]) – Target dimensions (width, height) to resize the input to before processing.

  • **kwargs (Any) – Keyword arguments for Generation Control: - stroke_width (int): Stroke width for rasterization. Defaults to 3. - fit_canvas (bool): Whether to fit sketch to canvas. Defaults to True. - num_inference_steps (int): Number of denoising steps. Defaults to 30. - guidance_scale (float): CFG scale for text guidance. Defaults to 7.5. - controlnet_conditioning_scale (float): Strength of ControlNet. Defaults to 1.0. - adapter_conditioning_scale (float): Strength of T2I-Adapter. Defaults to 1.0. - image_guidance_scale (float): Strength of InstructPix2Pix image guidance. Defaults to 1.5. - negative_prompt (str): Prompt detailing what to avoid. - seed (int): Random seed for reproducibility.

Returns:

The generated realistic image.

Return type:

Image.Image

Raises:

TypeError – If the input_data is not of a supported type.