Sure. There's already a semi-automatic one, so you just need some logic to check which chunks can be replaced by dictionary words.

Hmm, 1. would it be possible to make it golfier :P 2. can I steal copy the compression from Jelly

There's definitely room for improvement. For starters, the words are sorted by length, then alphabetically. They should be sorted by frequency instead.

Hmm, okay so if I replace with a list sorted by frequency then the comression will be much better? :P

Pretty sure you can find one of those in a lot of places

@Dennis Anything else?

It's probably wise to split the words in common and uncommon, as they're currently split in short and long.

Also, there's a problem with spaces not being able to be encoded in certain positions. I don't recall where right now.

Hmm, okay. Could you give a basic overview of how the compression works? :P Right now AFAICT it uses one or more bits for a flag before each word, which doesn't seem right

Not actually bits. Before each new token, you have to choose between a character, a short word, and a long word.

Hmm so is it divmodded by 3?

← 12 messages moved from talk.tryitonline.net

Oops :/ sorry

No worries.

It seems like it could be possible to shorten the choosing possibly if some assumptions are made about token frequency?

@ASCII-only Yes. After a character, you put an integer from 0 to 94 (printable ASCII and newline). Before a word, there are two bits (capitalized or not), (leading space or not). Spaces have different defaults depending on where the word appears.

Hmm, how long are the short/long word dictionaries respectively?

20453 and 227845 words

thinking in progress

@ASCII-only Oh, I'm sure there's lots of room for improvement. I didn't really waste much thought on this. Afaik Jelly was the first golfing language with a custom string compressor.

Oh lol Charcoal has way too many different compression methods

But no dictionary compression yet

Such as? I'm curious.

Brotli, LZMA, charset + indices, permutations of symbols/lowercase/uppercase/newline or space/digits, base conversion

Not that any of them are any good lol

Oh also RLE

Well, most of these aren't exactly custom. I got the idea from several languages using shoco, and I thought I could do better for Jelly.

True :P but I'm not sure whether there are many custom compressors except for dictionary-based ones

Okay, not sure if my math is right but seems like a small dictionary size of size ~400 would be best, but not sure if it's worth it to have a separate compressor for large texts? (And I have no idea what are the optimal sizes for short phrases/sentences, and I'm not sure where I could find some examples)