Rabbit packaging

Rydwolf Programs

Feb 22, 2023 15:14

@Ginger So what's the elevator pitch for this system

Ginger

this system will likely just be for Lua in general

Rydwolf Programs

What makes it different from the existing ones

Seggan

and please, whatever you do, not in python :P

Ginger

@Seggan ~~time for scala~~

@RydwolfPrograms just a sec

Seggan

@Ginger rol and lua imports are very different

Rydwolf Programs

Feb 22, 2023 15:15

A someone designing an interpreted language, I'd like some ideas to make idea soup with

Seggan

lua is literally just #include, rol has metadata as a comment in the compiled file

Ginger

hmm

Seggan

meta as in what functions it has, types, classes, whatever

example

Ginger

A directory of source files with a metadata file in the root is a module. A directory containing multiple modules is a module tree. The interpreter takes a path to a module tree as input, and looks for a "main" module in the tree, running it if it exists. There can only be one main module per module tree.
Module trees can be packaged into a "bundle" file, which the interpreter can also take a path to. These bundles can then be further packaged into an "application bundle", which is a native executable containing the bundle data. When run it extracts the bundle data, installs the required in…

@RydwolfPrograms ^

Rydwolf Programs

What is the alternative to this?

I don't know much about interpreted language packaging

parz

Feb 22, 2023 15:23

@Ginger I think you should check on Vyxal

Rydwolf Programs

That's mostly what I'm curious about; how does this system improve upon what exists, what does it do differently, and why

Given how much time you've spent hating how Python does it, you probably have good rationalizations for every part of the first paragraph, and I want to hear those :p

Ginger

I certainly do :b

this entire scheme is meant to fix the issues I've seen with Python

so basically Python's biggest issues are 1. for a long time modules didn't have actual, deterministic dependencies and 2. there are like 20 different ways to install modules (system python, virtualenv python, os package manager, pip) so there are myriad opportunities for package conflicts and dependency issues

this system fixes that

the biggest restriction is that the interpreter can never access modules of any kind outside of the module tree it's currently running, so you always know exactly what modules are available

Rydwolf Programs

In your system or Python's

Ginger

mine

Rydwolf Programs

One flaw with that is that you need multiple copies of libraries

Ginger

Feb 22, 2023 15:29

I know that, but tbh I don't really think it's actually an issue; most libraries aren't very big, and for the ones that are (*cough*tensorflow) I'm working on an alternate solution

Rydwolf Programs

Maybe not a super important one, but it could waste a lot of disk space for big libraries you use often (e.g., ML stuff)

Ginger

brb, I'll tell you more once I get back

Rydwolf Programs

The way I've been considering is symlinks

pxeger

Fortunately, the charging one has been solved now that we've all standardized on mini-USB. Or is it micro-USB? Shit.

Rydwolf Programs

You have a per-system or per-user place where libs get dumped, and when installing something, if it's already there, it gets symlinked to (and this would be recursive, so if you install goat_recognizer and sheep_recognizer, and they both rely on image_recognizer, you still don't install two copies)

Seggan

Feb 22, 2023 15:32

@pxeger the charging one is actually being standardized now finally with usb-c

Rydwolf Programs

And we have Unicode now

If anything that's proof that creating that 15th standard is still a good idea

Because it might actually be better :p

Seggan

@RydwolfPrograms utf-8 or utf-16? :P

pxeger

I think the biggest problem with module/packaging/build systems is their complexity, and that that is because people are continually adding more layers of abstraction to solve problems with existing solutions

Rydwolf Programs

I use UTF-7 exclusively

You know it's good when the first word after "is" in the Wikipedia article is "obscure" :p

Oh wait it's "obsolete"

I cannot read

pxeger

It's actually "an"

Rydwolf Programs

Feb 22, 2023 15:37

Yeah but "when the first word after 'an' in..." doesn't really make as much sense

Ginger

@pxeger nah; this system isn't supposed to cover everyone's use cases

@RydwolfPrograms I think I've figured out how to fix that issue

so, suppose we have two module trees, tree A (which is a big and chunky library) and tree B (which wants to use something from tree A)

normally, tree B can't access anything from tree A, or vice versa

but what tree A can do is this: in its main module, it can use a special keyword to "expose" certain members

these members must be defined in the main module, but they can use code from any other modules in the tree

Rydwolf Programs

I don't think that fixes it

Ginger

@RydwolfPrograms chill, I'm not done yet

tree B can then use a special import statement to "link" tree A, and access only the exposed members from tree A

the thing about imports in this language is this: unlike in Python, they are guaranteed to work

if I import a module, I know with 100% certainty that the build system will make it available to me (provided the module doesn't crash when loaded)

but this special import statement is not a guarantee; it can fail with an error if tree A isn't available, or fails to load, or some other weird issue

also, each tree keeps its own internal dependencies

Rydwolf Programs

@Ginger What's the purpose of this special import statement?

Ginger

if (some module in) tree A and tree B both depend on different versions of module C, they will both access their local copy of module C

@RydwolfPrograms what do you mean?

Rydwolf Programs

Feb 22, 2023 15:47

Why would I use it instead of a normal one

Ginger

normal imports can only access modules within the module tree; this statement can only access exposed members of the main module of another module tree

Rydwolf Programs

That's neat but doesn't really fix the issue of having to install the same library a zillion times

Seggan

@Ginger seems like any static language

Ginger

If the library is so big that it wouldn't make sense for it to be its own module, it should be made into a module tree

that way, other module trees can use only the one copy of it

lemme make a flowchart

user

@Seggan Ginger said it was for Lua, which probably doesn't make guarantees about its imports working

Ginger

Feb 22, 2023 15:52

> Finger

golfer named finger

Seggan

finger? :P

@user but rol and lua have very different module systems

it wont really work well for both

user

Edited

@Seggan Where does Rol come into this?

Oh Ginger wants to add it to Rol

Seggan

39 mins ago, by Ginger

can I try to implement the system I just detailed for packaging?

user

Yeah I don't see why you'd do something extra like this for a statically typed language

Rydwolf Programs

@Ginger What layer is analogous to a dependency/module/library/package

E.g., I run ginger install bigrational

Is bigrational a module that goes in my module tree, or a module tree that goes in my bundle?

And if bigrational depends on a biginteger module tree, would that also go in my bundle?

Or is it nested in some way

And now what if biginteger is a module instead of a module tree, and I have six module trees in my bundle that all use the same version of it

Ginger

Feb 22, 2023 16:01

here's a flowchart

Rydwolf Programs

I like flowcharts

Okay so when would I use exposed members

What's a concrete example

Ginger

@RydwolfPrograms module trees and bundles are two forms of the same thing; module trees contain modules which contain code

@RydwolfPrograms Suppose Tree B is tensorflow, and Trees A and C are projects that require ML stuff

Rydwolf Programs

Wait which way do the arrows go here?

Ginger

am typing, one sec

Rydwolf Programs

Oh wait arrows point to dependencies

Ginger

Feb 22, 2023 16:04

so normally one of the "other module" boxes in Trees A and C would be tensorflow (and therefore there would be two copies of it and all of its dependencies) but by making tensorflow its own tree there only has to be one copy of it

Rydwolf Programs

Is this process transparent to A and C? From what you've said I get the impression they have to specify that it's an exposed member of another tree

Which kinda defeats the point

Ginger

wdym?

Rydwolf Programs

Since it would mean the library needs to know what libraries it'll be used alongside

Ginger

to A and C tensorflow looks like just another module

Rydwolf Programs

Okay good

So this sounds like it's just the standard dependency graph sorta thing, but with the addition of the concept of a bundle

Ginger

Feb 22, 2023 16:09

the only difference would be with the import statement syntax; you'd probably do something like import tree tensorflow instead of import tensorflow

Rydwolf Programs

Why do I need to distinguish between trees and modules

Ginger

@RydwolfPrograms just to clarify: "bundle" means "module tree that's been packaged into a single file"

Rydwolf Programs

yeah

Ginger

@RydwolfPrograms what do you mean? like why is the syntax different?

Rydwolf Programs

Yeah

Ginger

Feb 22, 2023 16:10

I suppose it doesn't have to be

Rydwolf Programs

I think a lot of my confusion comes from the term "module tree"

What is the actual structure of things within it. I assume it's not actually a tree, right?

Ginger

but if it's different there's less ambiguity; remember, importing a tree can fail with a "not found" error, but importing a module cannot because modules are guaranteed to be present by the build system

@RydwolfPrograms no, it's a flat directory of modules

Rydwolf Programs

Yeah but dependency wise

Ginger

I can make a diagram if you want

Rydwolf Programs

Why is it called a tree if it's not a tree

Ginger

Feb 22, 2023 16:11

¯\_(ಠ_ಠ)_/¯

maybe a better name would be "group"

Rydwolf Programs

Feels like a module should be called a module and a module tree should be called a package or something

Ginger

because that's really what it is: a group of modules with some extra metadata on top

now, I've got something that I need to be working on, so you can ask me more questions but I might be slow to answer q:

Rydwolf Programs

And you wouldn't typically download individual modules right

Seggan

@RydwolfPrograms packages are packages in rol as in package rol or package seggan.is.the.best

Rydwolf Programs

A library would be a module tree, and all the modules within a module tree would typically be written by the same person?

Ginger

Feb 22, 2023 16:14

@RydwolfPrograms correct (that'd be taken care of by the build system while you're developing)

@RydwolfPrograms that's a way to do it, but not the way I had in mind; I'll tell you more in a bit

Rydwolf Programs

@Ginger Wait so you would have external modules as dependencies?

(I also have to go do school stuff, we can resume this later ig :p)

Jacob

Feb 22, 2023 16:25

@Ginger i dont know a ton abt package management but i know that the JS package installer PNPM uses symlinks and makes installing packages you've downloaded before really quick

Ginger

blech symlinks

Adám

@cairdcoinheringaahing @WheatWizard can you give chat.stackexchange.com/users/570753/user53691 access to chat.stackexchange.com/rooms/52405/the-apl-orchard?

caird coinheringaahin g

Done

Adám

thanks

Simd

Feb 22, 2023 16:43

CMQ: I have a decreasing function f over the reals, is there a built in way to find where f(x)=0.5 in python? It's just binary search but I don't think bisect can do it.

Jacob

substitute?

Simd

@Jacob I mean the x such that f(x)=0.5

Given the function is decreasing you could use binary search

Jacob

well it depends on f

Simd

@Jacob right but assume f is just given by some code

You are not going to do any math. We are just going to search for it

Jacob

hmm ok

well im not familiar enough with python

Simd

Feb 22, 2023 16:49

Which language could you do it in?

Aiden Chow

@Simd probably some builtin in mathematica :P

Ginger

@Simd why do you always have these mathematics problems?

mathcat

cause maths is cool!

New Posts

0

Q: Elevator movement order

There is a building with an infinite number of floors and you are the only person in an elevator that can lift an infinite number of people. You go home, but on one of the floors people enter the elevator and enter their destination floors. Your program should handle the floor order correctly Yo...

code-golf

Aiden Chow

@Simd well i wrote some crude code that i think does what you want: tio.run/…

not exact, it should be accurate up to the tolerance value

Simd

Feb 22, 2023 16:59

@AidenChow yes! I just feel there just be some module that has this covered

@Ginger it's not really math. Just search

@AidenChow always :)

@mathcat yes!

Aiden Chow

@Simd im not aware of one, but theres probably a math module out there that does this though

Simd

@AidenChow would be cool to know what it is

Aiden Chow

maybe sympy has smth, idk

Simd

@AidenChow hmm .. Isn't it more like scipy or numpy? It's not symbolic math

It really is just binary search

Aiden Chow

@Simd oh yea, its more like searching, not actual math. probably scipy then, numpy is more for arrays and things like that right?

Simd

Feb 22, 2023 17:04

Yes but I can't find it

mathcat

Feb 22, 2023 17:22

Another maths question; What kinds of positive integers can be written as (a^2+b^2)-(c^2+d^2), where a, b, c and d are positive integers too.

Do they have any common properties / can they be categorised into groups?

@mathcat *and different from each other

Aiden Chow

Feb 22, 2023 17:43

@mathcat interesting question, maybe we can split as (a^2-c^2)+(b^2-d^2) and maybe (a+c)(a-c)+(b+d)(b-d) and see if theres anything there?

m90

All of them!
(n+1)²+1²-n²-4² = 2n-14 (valid for n≥5) gives the evens.
(n+1)²+1²-n²-5² = 2n-23 (valid for n≥6) gives the odds.

Aiden Chow

damn

mathcat

... wow

Aiden Chow

smart

mathcat

@AidenChow that's pretty clever

@m90 I'll try to figure out how you ended up with this :d

Rydwolf Programs

Feb 22, 2023 17:45

@Ginger Symlinks are pog

If I made a programming language, it would use symlinks

Aiden Chow

@mathcat probably they notice that (n+1)^2-n^2 is 2n+1 and it was pretty easy from there

Ginger

@RydwolfPrograms okay so

Libraries are modules, large libraries and full applications are module groups/bundles

mathcat

@AidenChow I'm not on the same page - wouldn't that only work for consecutive a, b?

Rydwolf Programs

@Ginger That feels kind of weird

Why do you do it that way, out of curiosity?

Aiden Chow

@mathcat its a specific construction that can result in all positive integers, so any (a,b,c,d) that follow that form will be able to get all positive integers

Ginger

Feb 22, 2023 17:49

this way, applications can depend on specific versions of libraries they need without worrying about other applications having installed incompatible versions, and large libraries only have to be installed once

mathcat

@AidenChow ah, nice.

Rydwolf Programs

@Ginger So it seems like this is basically the single-destination-for-all-installed-libraries approach, but scoped per-application

Ginger

in essence, yes

it's Python virtualenvs but better (hopefully)

Rydwolf Programs

But I still don't see why differentiating between modules and module trees has anything to do with different versions of the same thing

Or with not installing things twice

Ginger

wdym?

Rydwolf Programs

Feb 22, 2023 17:51

Why do both modules and module groups exist

Seggan

@RydwolfPrograms how

Rydwolf Programs

Why isn't everything a module, and some modules just have dependencies

Ginger

suppose that we have two groups, A and B, which both depend on some module

Rydwolf Programs

@Seggan You'd have all of the libraries you install in a single place, and those would be symlinked in your project

Ginger

...wait

Rydwolf Programs

Feb 22, 2023 17:53

So to the interpreter/compiler it looks like you've just got a nice simple tree of dependencies, but it's actually already got dupes taken care of

Ginger

okay, there's actually another way to do this

mathcat

@mathcat wait, I still didn't understand how you got from (a+c)(a-c)(b+d)(b-d) to (n+1)^2-n^2.

Ginger

the whole module group thing was made to prevent situations where two projects depend on mutually incompatible versions of the same package, but I just realised that can be fixed by... installing two versions at once

I may be stupid

Rydwolf Programs

lol yeah that's what I was confused by

Ginger

HOWEVER!

Rydwolf Programs

Feb 22, 2023 17:55

uh oh

Ginger

I think that module groups, while not the only solution to the incompatibility issue, are a better solution than "globally install two versions"

Aiden Chow

@mathcat ummm i didnt make the contruction. m90 did

Rydwolf Programs

@Ginger But on the other side of the coin you also have to install dupes of the same version of a module if they do use the same one

mathcat

@m90 care to elaborate? :P

Ginger

why? because 1. you don't have to specify which version of a module you're importing, the correct one is always the one imported and 2. I like having all the dependencies organized in one place; it pleases me

Aiden Chow

Feb 22, 2023 17:57

@mathcat as i said before, they probably notice that (n+1)^2-n^2 = 2n+1 and its pretty easy to see the construction from there

Rydwolf Programs

1. You still would have to specify it _somewhere_, probably in that module config file you talked about
2. The dependencies are more scattered with module groups, not less

Ginger

@RydwolfPrograms that is definitely a consideration, but we live in an era where you can buy terabyte hard drives for 30 dollars

Aiden Chow

lmfao theres like two completely different convos going on rn

mathcat

@AidenChow I'm confused, I'll take a break :d

Rydwolf Programs

@Ginger You also save bandwidth and time tho

Ginger

Feb 22, 2023 17:58

@RydwolfPrograms wdym more scattered? all of a group's dependencies are in one place: the group

@RydwolfPrograms hmm, fair enough

Aiden Chow

@mathcat what are u confused about?

mousetail

@Ginger Yea this is what rust does, also NPM does this but not as well

m90

@mathcat Starting from (a²+b²)-(c²+d²), rearrange to (a²-c²)+(b²-d²). Then notice that setting a=c+1 can produce all odd values for a²-c², and find two suitable choices for b and d to cover everything.

Ginger

ooh, what if downloaded modules are cached somewhere and whenever you make a project the downloads are copied or *shudders* symlinked in

Rydwolf Programs

@Ginger No, because if it depends on another group, that group's somewhere else. And if it depends on exposed dependencies from that module group (if you're still doing that), they're a third place

@Ginger That's what I'm saying! :p

mousetail

Feb 22, 2023 18:00

@Ginger Many languages don't even need the pacakges to be in a specific folder but can read them directly from a central cache folder

mathcat

@m90 ooh, so your formula returns all values for consecutive a, c

nice

Ginger

well, the module group system works well with bundles because bundles are basically zipped module groups

but you have a good point abt storage and bandwidth

Rydwolf Programs

I still don't get how module groups work when there's libraries which are themselves module groups

Ginger

@RydwolfPrograms if I do your symlink idea this would become unnecessary tho

because there'd only be one copy

Rydwolf Programs

Like, are all of the module groups in one top level? Or do you have them inside the module group, like modules?

Where you'd have a nested structure?

mathcat

Feb 22, 2023 18:02

@mathcat I'm dumb lol that's clever

Rydwolf Programs

Which seems like it'd defeat a lot of the point

Ginger

there is no nested structure here

Aiden Chow

@mathcat lol ya i agree its pretty smart lmao

Rydwolf Programs

So what happens if my project, goat_recognizer, is a module group which depends on image_recognizer, a library which is a module group

Ginger

could you be more specific? lots of things happen all the time d:

Rydwolf Programs

Feb 22, 2023 18:04

The bundle would have to include both module groups, so a bundle can't be the same as a module group

Ginger

oh, that would be two bundles when packaged: goat_recognizer and image_recognizer

you'd have to come up with some other way of distributing them together, like the "application bundles" idea I had earlier

Rydwolf Programs

So the end user would install goat_recognizer as an application bundle, and image_recognizer would be (the interpreted equivalent of) a dynamically linked library?

Ginger

yup

Rydwolf Programs

Okay, that makes sense

Ginger

:D

Rydwolf Programs

Feb 22, 2023 18:05

That removes the issue of duplicate libraries for the end user too

Since presumably image_recognizer.bun would be stored in some central place where both goat_recognizer.abun and sheep_recognizer.abun could depend on the same file

Ginger

6 mins ago, by Rydwolf Programs

@Ginger That's what I'm saying! :p

@RydwolfPrograms yes, exactly!

Rydwolf Programs

Okay, this is sensible

Ginger

and if goat_recognizer and sheep_recognizer both depend on fancy_terminal, since that's not a big library each bundle would have its own copy (or if I did the cache-based solution the single download would be symlinked in during the install)

Rydwolf Programs

Well that wouldn't be for the end user right

m90

@Simd It can actually be done using bisect in version 3.10 or greater (for the key parameter), with some tricks: ato.pxeger.com/…

Rydwolf Programs

Feb 22, 2023 18:08

The end user would just have the bundle files

Ginger

correct, the installer handles everything else

Rydwolf Programs

IMO it still would make more sense to treat every dependency like a module tho

And then there'd be a way to declare which dependencies should be dynamic

Ginger

eh, this method works fine, and if I do the cache-based system it also fixes the duplicate issue (although I don't really like that solution)

wait, wdym?

to the code, everything is a module

Rydwolf Programs

No but like, I just still don't think module groups add much

Ginger

I think they do :p

Rydwolf Programs

Feb 22, 2023 18:11

They're basically just dependencies that are always statically linked

Ginger

correct (I think)

Rydwolf Programs

Why can't all modules/module-groups just be modules, and ones that would typically be the small dependencies in module groups would just be declared/imported/configured as static by the application

Ginger

I don't get it, could you explain in terms of goat_recognizer? :b

Rydwolf Programs

One program's tiny dependency is another's huge dependency. What if I'm super restricted resource-wise and have fifty programs that all depend on a bigint module?

I have no choice but to have fifty copies of that bigint module as the end user

DLosc

@RydwolfPrograms image_recognizer.bun??

Rydwolf Programs

Feb 22, 2023 18:14

I mean it's not much of a drawback, but I just still don't see what module groups add

It seems like removing them would have purely upsides

@DLosc Indeed :p

Ginger

@DLosc ~~call the language Rabbit~~

@RydwolfPrograms alright, so if I didn't have module groups what would goat_recognizer look like?

Rydwolf Programs

Okay so goat_recognizer uses image_recognizer and fancy_terminal, and image_recognizer uses neural_network and bigint, which are both small dependencies which would be modules in your module-module-group dichotomy

All of those would just be modules

goat_recognizer would mark image_recognizer as dynamic

Ginger

I don't get it

Rydwolf Programs

So image_recognizer would be bundled as its own thing, with neural_network and bigint statically linked, and goat_recognizer would have fancy_terminal statically linked and rely on image_recognizer as an external bundle

Basically instead of the library creator deciding it's either a module or a module group, they just write a module, which may or may not depend on other modules

Then the user of that library can decide to statically or dynamically link it

Ginger

and "statically linked" modules are included in bundles?

Rydwolf Programs

Feb 22, 2023 18:23

Yeah

This also gives you flexibility regarding libraries which themselves have dynamic dependencies. You could pass that decision on to the user of the library. E.g., if image_recognizer decides that neural_network is big enough to be worth dynamically linking in some situations, goat_recognizer would get the final say on whether or not neural_network gets statically linked into image_recognizer's bundle

Ginger

well, if goat_recognizer and sheep_recognizer depend on different versions of fancy_terminal, how does the interpreter know which one to import?

Rydwolf Programs

Wdym? They're two totally different projects

They'd have their own copies of fancy_terminal, and/or two different versions would be stored in the global cache

Depending on how you did it dev-side

Ginger

and how is that different from the module group system? my favorite feature of it was that all you had to do was give the interpreter a path to a module group and it would only use the modules in that group while running, thereby avoiding dependency issues

Rydwolf Programs

Every module would just list its dependencies, like any other programming language

Ginger

I feel like I'm not understanding something

Rydwolf Programs

Feb 22, 2023 18:27

@Ginger Is the last "it" referring to the module group or the thing importing it

Ginger

the modules in a group can only access 1. other modules in their group or 2. the public interfaces of other groups

which, now that I say it, seems redundant

Rydwolf Programs

@Ginger It would still be the case that modules can only access modules the module creator intended them to access

Ginger

huh?

Rydwolf Programs

If a module says it depends on thing_doer version 2.1, and you have thing_doer version 1.8 for something else, it's not like the module would suddenly be given thing_doer 1.8 and error out

You don't need to restrict what modules can access because you know what they access

Ginger

ohhhhhhh

I think I get it

yeah that makes sense

but now I'm not really sure this system is all that different from what already exists...

Rydwolf Programs

Feb 22, 2023 18:33

It doesn't need to be. Don't make shotgun changes, zoom in on what's not working and snipe it.

Ginger

I guess modules would have to include a metadata file along with their source, but that's not too big of an issue

Rydwolf Programs

I don't see why having a version in the source code is a bad thing

That should 100% be how it works IMO

Ginger

@RydwolfPrograms ~~it's hard to kill Godzilla with a sniper rifle~~

Rydwolf Programs

The version is just as critical as what the library is

import image_recognizer{2.2.*} or something

mousetail

@Ginger Don't change things for the purpose of changing things, change them if there is a concrete problem with the existing system that you have a solution for

I've often tried to re-invent stuff only to come up with the exact same thing it was before again

Rydwolf Programs

Feb 22, 2023 18:36

Although doing that is a valuable experience

Best way to understand how something works and why it's done that way is to reinvent it yourself

Ginger

@mousetail yeah, that's why I'm not trying to do this with python: I don't want to reinvent the wheel :p

The Nineteenth Byte

Participants