Discussion between hanugm and Neil Slater

Feeds

07:56

Q: What are all the inputs that support diversity of images in text to image generation?

For this question, consider the stable diffusion model. For a given text embedding, Stable Diffusion can generate diverse images. In this context, 'diversity' refers to the variation among the images generated, meaning that several images with the same semantic content (given by the text embeddin...

hanugm

@NeilSlater So, are you saying that the 'nosie vector' is the only input that leads to diversity in images generated? Then where does the diversity emerge? Is it in some other hyperparatemeters? If yes,a re they related to neural network or some other?

Neil Slater

Are you asking about parameters other than the text prompt. E.g. choice of sampler, cfg scale etc? Or can those be considered fixed, like the prompt. Also are you asking about gamut of possible outputs related to what the different noise values produce, or about the image varying at all?

hanugm

@NeilSlater I am investigating the factors that contribute to image diversity in the Stable Diffusion model beyond the random noise vector, assuming other settings like the sampler are fixed. My focus is on understanding which additional inputs or some other model related values can influence the diversity of the generated images from the same text embedding under a constant configuration.

Neil Slater

I'm still not understanding what you are getting at with "diversity". Do you mean just that the image is different - i.e. the pixels are different colours, or are you referring to the limits of what differences can be realised (e.g. if you ran the same settings with random noise a million times with the prompt "cat", would one of the cats be surfing, or doing something else unusual that typically you would need to prompt for)?

hanugm

@NeilSlater Diversity here is generation from all the possible images that can be obtained for a fixed text embedding in constant configuration. As you mentioned 'gamut of possible outputs' in fixed configuration.

Neil Slater

07:56

Still not quite there. Are you interested in that it changes at all, or in the practical limits of what those changes can be? Are you asking if there are other factors beyond noise and settings. Or are you asking "can a cat prompt generate a dog sometimes"?

hanugm

@NeilSlater I am interested in those factors, like random noise, that help in generating different images from a fixed text embedding in a fixed configuration.

Neil Slater

OK. Got there. There are none. You can fix the seed and see this, the same image is generated every time (there are some caveats, because different GPUs may behave differently). If that's not what you mean, please explain?

hanugm

@NeilSlater I think I am looking for this only.

Neil Slater

OK.

hanugm

import torch
from diffusers import StableDiffusionPipeline, EulerDiscreteScheduler

model_id = "stabilityai/stable-diffusion-2"

# Use the Euler scheduler here instead
scheduler = EulerDiscreteScheduler.from_pretrained(model_id, subfolder="scheduler")
pipe = StableDiffusionPipeline.from_pretrained(model_id, scheduler=scheduler, torch_dtype=torch.float16)
pipe = pipe.to("cuda")

tokenizer = pipe.tokenizer
text_encoder = pipe.text_encoder
print(tokenizer)
#print(text_encoder)

prompt = "cat"

# Step 1: Tokenize the prompt

It is generating different images when I run several times, so I am thinking whether random noise is the only contributor for that since all remaining are fixed I think @NeilSlater

Neil Slater

07:58

It is possible to inject more noise vectors. Stable DIffusion is open source. If you used something like ComfyUI, or scripts, then you could add noise to almost any of the stages

hanugm

@NeilSlater Oh, then in my case, there may be none other that noise vector and possible GPU related dependency.

Neil Slater

Yes, pretty sure if you set the seed, it should be the same image each time. Although not familiar with where/how it would be set in that code.

It's pretty common to set a seed then increment it to generate e.g. 10 images that you can then compare when changing other things such as the prompt or one of teh many parameters, to be able to see more clearly what effect they are having

I was kind of hoping you were asking about the limits/gamut of the model - i.e. just how much variance can you expect given you can vary the initial noise seed. I think that is an interesting question - for which I don't have a great answer.

However, this is more about practical control and expectations on a project?

Whilst "what is the potential gamut of a diffusion model" is more philosophical

5 hours later…

hanugm

13:15

Sure @NeilSlater, I will go through them in detail, thanks for the insights.

last day (15 days later) »

all rooms