@AndrovT I've refined the corpus a little to remove things like # main program comments and programs that are mostly compressed strings, and it seems to do a little better

input size: 32712 bits
output size: 25885 bits
ratio: 0.7912998288089997

that's the output from the test(pair_frequency2(16, 128)) test in the testing notebook

compared to the original:

input size: 32712 bits
output size: 29779 bits
ratio: 0.9103387136219124

github.com/Lyxal/Vycoder

Somehow, removing all instances of all strings makes the compression ratio worse

Puts it up to 82% on average

82% of the original length

So 18% shorter

Compared to the 21% currently in my fork

@lyxal I was going to do that but I didn't expect such a large effect

It looks like there could be a few more programs that could be removed, but not too many

There seems to be a certain limit

Like I removed all compressed strings and normal strings using regex and it went to ~26000 bits

I don't have that corpus pushed to github btw

There seems to be a problem with how you test it

Hey what do the alpha and beta parameters do in pair_frequencies_2?

@AndrovT is there a different way to do it than running the notebook cells?

It's important to validate how well the compression works on a different dataset from what you made the model from

Yeah the test data remained unchanged I think

What I did was take the first 1000 programs to train the prediction model (in TrainingData.csv) and validate it on the last ~300 (in TestData.csv)

You added all the ~1300 programs to training data so the result is not representative how the compression will preform on future programs

Ah

Does more training data give better results or does the principle of less is more apply?

more training data should be better

but it depends on how similar it is to what you're trying to compress

How about program length? Should training data be longer or shorter?

No idea

Well later tonight (read: 3 hours or so) I might try sorting by distance to median program length and take the first thousand of that for training

With comments and compressed string programs removed

See how that goes

Also

@lyxal What pair_frequency2 is sum different predictions based on the parameters alpha and beta

Is 16, 128 optimal?

Basically higher alpha gives more weight to uniform distribution, higher beta gives higher weight to a distribution based on frequency and the rest is a distribution based on the last symbol

16, 128 was close to optimal from my testing

:+1:

So I'll only be experimenting with the corpus then

I might add some week.golf, emkc and Codidact answers to the corpus for more data

I have a slightly better model than pair_frequency2 now.

It looks at more than just the last character and it looks like it save additional 1%

How does it work?

for each i in range (1, 9) I collect how many times does a symbol A appear i symbols after symbol B and then the weight of the next symbol being A given a list of symbols lst is roughly sum(0.5**distance(B, A)*(number of times B appeared that far away from A) for B in lst) + beta*(frequency(A) + alpha)

I wonder if making it recognise digraphs would make it any better

As in, call the lexer on input programs and add digraphs to the code page used

(code page would have to become a list for that obviously)

That's going to be one of the things I try along with corpus adjustments

And test data ordering

@Seggan it's still scored in fractional bytes :p

Well I don't know how valid the results are, but I might have a ~27% shorter program list

Programs sorted by length furthest away from average length

Training data = first 1k

test data = remaining

Test data is what is considered "average length" vyxal programs

training data is more of the "outliers"

input size: 30024 bits
output size: 22125 bits
ratio: 0.7369104716227018

modified corpus to remove some programs/program components that are: a) false positives from the SEDE query, b) comments, c) quine related and d) a single built-in

no extra data from weekgolf or other sources yet

input size: 33920 bits
output size: 25538 bits
ratio: 0.7528891509433963

with week.golf and emkc answers

input size: 88 bits (11 bytes), output size: 65 bits (8.125 bytes), ratio: 0.7386363636363636
11000010100101100101101110010010010011000101110010000001001100011

(fizzbuzz)

@AndrovT it really is better

I was running some tests against a fracbyte lang called myby and the new model gets more Ws than the pair_frequency model

it'd be nice if nibbles had a similar list of comparable tasks

@lyxal I wonder if there's a way to get fizzbuzz under 8 bytes

looks like vycoder shortens anywhere between 9% to 27%

at least, it does using the refined corpus

I'll copy the updated version of vycoder over once Androv has had a chance to have a look at what I've done to it

@lyxal I don't think this is the way to go. Test data should represent the programs you want to encode as closely as possible while being independent of the training data. Having only those of average length messes up the results because what you're measuring isn't how well it compresses vycal programs in general but only those of average length

For deploying it it's probably best to use all the data but if you want to estimate how well it works you should follow these rules

so if anything, the order of programs should be randomised?

It's potentially useful to filter the programs somehow. You just need to be careful whether better test result means an actual improvement or only the test got easier.

well I've filtered out some programs that are either a) not representative of actual code golf challenges (e.g. quines or restricted source) or b) contain series of characters that aren't actually commands chained together (like strings)

That sounds reasonable

and as mentioned, it does seem to do a little better

👍