sketchkit.sketch2image.pix2pix package

Module contents

Pix2Pix Model Implementation for Sketch-to-Image Conversion (SDXL Version)

This module encapsulates the diffusers StableDiffusionXLInstructPix2PixPipeline for generating images via instructional prompts.

class sketchkit.sketch2image.pix2pix.Pix2PixModel(device: str = 'cuda', model_name: str = 'diffusers/sdxl-instructpix2pix-768', vae_model: str = 'madebyollin/sdxl-vae-fp16-fix', torch_dtype: dtype = torch.float16, safety_checker: bool = False, **kwargs: Any)[source]

Bases: object

InstructPix2Pix model for direct sketch-to-image translation using SDXL.

pipeline

The loaded diffusers pipeline.

Type:

StableDiffusionXLInstructPix2PixPipeline

model_name

The name/path of the loaded InstructPix2Pix model.

Type:

str

device

Computation device (‘cuda’ or ‘cpu’).

Type:

str

generate(control_image: Image, prompt: str, size: int | Tuple[int, int] | None = None, negative_prompt: str = 'longbody, lowres, bad anatomy, bad hands, missing fingers, extra digit, fewer digits, cropped, worst quality, low quality', num_inference_steps: int = 30, guidance_scale: float = 7.5, image_guidance_scale: float = 1.5, seed: int | None = None, **kwargs: Any) → Image[source]

Generates an image from a sketch using instruction-based translation with SDXL.

Parameters:

  • control_image (Image.Image) – The input image to be translated.

  • prompt (str) – Text instruction (e.g., “turn it into a realistic photo”).

  • size (Optional[Union[int, Tuple[int, int]]]) – Target resolution.

  • negative_prompt (str) – Text prompt detailing what to avoid.

  • num_inference_steps (int) – Number of denoising steps. Defaults to 30.

  • guidance_scale (float) – CFG scale for text guidance. Defaults to 7.5.

  • image_guidance_scale (float) – Scale for how closely to follow the input image structure. Defaults to 1.5.

  • seed (Optional[int]) – Random seed for deterministic generation.

  • **kwargs (Any) – Additional arguments for the pipeline.

Returns:

The generated translated image.

Return type:

Image.Image

Raises:

RuntimeError – If image generation fails during pipeline execution.