how and why are generators?

A username

Jun 19, 2021 12:41

How does Vyxal use Generators exactly?

lyxal

well

that is a good question

and I'll give you a hopefully good answer

The story of how and why Vyxal has a custom generator class

A username

Wait, custom?

lyxal

Generator

github.com/Lyxal/Vyxal/blob/master/Vyxal.py#L144

A username

I honestly thought that was a builtin.

lyxal

If you let me explain you'll see what I mean

Way back in the early days of Vyxal, there was only numbers, strings and lists

If you tried to do things with itertools/mapping/anything that would usually be lazily evaluated by python, it had to be converted to a list first

However, this meant that programs involving things like getting all the permutations of a largeish string and then filtering based on a predicate would take a long time

An optimal solution would be to just use the items returned by the mapping/itertools stuff/whatever as-is

however, there's just one problem with that

a map is a different type to itertools.<whatever> which is yet a different type to the built-in generator type

meaning that organising type cohesion would be an absolute pain

also, things like indexing would still face the same problem of having to convert the entire generator into a list

so I thought "hey what if I make a general wrapper for all the many types of generators"

"that way, I only have to deal with one type of class"

"and I can create custom methods for it too"

So I made the Generator class

Doing so also has the benefits of making user-defined generators play nice as well

because I can control how the instance is created using __init__

A username

Jun 19, 2021 12:49

This... is something I need to implement, but it might take a while to do so.

lyxal

That's why we have a custom Generator class

now for how it's used.

Anytime something that would otherwise be lazily-evaluated is returned, it's wrapped in Generator

A username

Infinite lists for one, like ∞ and ÞF.

lyxal

yeah

those are wrapped in Generator too

but also things like Generator(itertools.zip_longest(*iterable(lhs), fillvalue=rhs)

Generators have two main attributes that need considering: generated and gen

generated is a list of all the values that have been, well, generated so far by the generator

this is mostly so that generated values are retained somewhere

gen is the actual generator object that is being wrapped

Now seems like a good time to properly address issue 82 on the repo

A username

@lyxal On one hand, I don't have to deal with itertools, on the other hand, I don't have itertools.

@lyxal The corpus is not exactly useful.

lyxal

> Generators causing problems because they aren't lists #82

A username

Jun 19, 2021 12:53

Oh yeah

lyxal

By address, I mean "explain what the problem is in more detail"

A username

I saw that

"This is something that will take more than a quick fix - it would require a complete reworking of how the Generator class works."

lyxal

Yeah

Let me explain why

Say you want to map , to a list

that's easy enough, you just print each item because you know when the list ends

it's finite, so it can be evaluated as-is

A username

But if you've got something like ∞ (which btw no one uses, check the corpus)

lyxal

Does that make sense? (I'm not done yet, I'm just taking a moment to make sure you're still tracking along)

A username

Jun 19, 2021 12:55

I understand so far, yes

lyxal

2 mins ago, by A username

@lyxal The corpus is not exactly useful.

@Ausername good

now say you instead want to map , to an infinite list

your first idea might be to try and call list on the generator

the problem with that is that you'll be waiting a while for the list conversion to finish

because it's, well, infinite

meaning that the printing will never actually happen

pxeger

why would you need to call list on it?

what's wrong with just iterating over it?

A username

Can you overload comparison for Generators?

pxeger

Python's builtin map does that already

lyxal

@pxeger because if you're doing something like ƛ,;, it's returning a map object

pxeger

Jun 19, 2021 12:59

ok, so then have the interpreter automatically try to exhaust any map objects it gets if it's supposed to print them?

lyxal

problem is that because it's lazily evaluated, when it goes to be printed, it applies the mapping and tries to print the list item at the same time

@pxeger it's an infinite map though

it goes on forever

it uses yield

lyxal.pythonanywhere.com/…

A username

How does Husk handle it?

lyxal

Idk

I've never read the husk source

and I don't know haskell

A username

@lyxal Oh true

Just asking because one of Husk's main concepts is infinite lists.

Infinity is a pain.

pxeger

by "exhaust", I don't mean "create a list", I mean for x in generator: pass

just force every item to be evaluated

and if the generator is a map, then force every item to have the function applied to it

lyxal

Jun 19, 2021 13:01

that already happens

pxeger

well what's the problem then?

lyxal

and so long as you don't try to print anything inside a generator, you're good to go

the problem is printing inside a generator

it's to do with the way that printing a generator works

at the end of execution, it goes through each item and applies whatever chain of functions has been mapped/filtered/whatevered

and then once it has it's result, it prints it

followed by a | to indicate the next item

however, if during the mapping/filtering/whatevering, the value is printed, it still prints the list item

here's another way of thinking about it

pxeger

oh, so each item gets printed twice

lyxal

yes

if i call print on a finite list, it can print the whole list and still halt

pxeger

you need a separate "foreach" and "map" then. Map returns the generator after the result of the function, Foreach returns nothing

lyxal

Jun 19, 2021 13:04

the problem is when I call print on an infinite list

@pxeger how would you print a map object?

a finite map object

how would you print it as a list?

A username

What if you have a hardcoded Generator size limit that can be modified with flags?

lyxal

@Ausername I've considered that

2 mins ago, by lyxal

how would you print it as a list?

the obvious answer would be print(list(map(fn, iter)))

pxeger

@lyxal The same way you currently do. Python 2: print(map(print, range(10))) prints everything twice too. But if you want to map , over an infinite list, you should expect to get everything twice. If you don't want it twice, then either use a "forcibly exhaust iterator but return nothing" builtin, or use foreach instead of map.

A username

By printing the first (limit) items...

lyxal

@pxeger But we're using py3

35 secs ago, by lyxal

the obvious answer would be print(list(map(fn, iter)))

however, that doesn't work on infinite lists

because calling list on an infinite list will never halt

it'll keep on going and going

never stopping

so how is one to print an infinite list?

pxeger

Jun 19, 2021 13:08

@lyxal well it's the same, but instead it's print("[", end=""); for i in map(print, range(10)): print(i, end=", ")

and range(10) can just as well be an infinite list

lyxal

    def _print(self, end="\n"):
        main = self.generated
        try:
            f = next(self)
            # If we're still going, there's stuff in main that needs printing before printing the generator
            VY_print("⟨", end="")
            for i in range(len(main)):
                VY_print(main[i], end="|"*(i >= len(main)))
            while True:
                try:
                    f = next(self)
                    VY_print("|", end="")
                    VY_print(f, end="")

pxeger

yes

lyxal

that's what I currently do

pxeger

look I really don't see this as a bug. It's expected behaviour with a map builtin

lyxal

and that has the problem of duplicate printing

@pxeger neither do I really

pxeger

Jun 19, 2021 13:09

Who said it was a bug?

A username

"It's not a bug, it's a feature!"

lyxal

@pxeger github.com/Lyxal/Vyxal/issues/82

pxeger

yes I see

but why did you respond to this in a way that makes it seem like it is a valid bug

lyxal

is it a bug?

pxeger

Tell the issue creator to use foreach instead of map

it is not a bug

lyxal

Jun 19, 2021 13:10

does it need fixing?

pxeger

no

lyxal

ah

see my understanding of infinity isn't exactly the best

Vyxal

Participants