Sandbox link, including some notes on why this scoring system was chosen. (I'll delete this after a while.)

"Your task is to write a program or function (or class, etc. -- see below) which will be given this file one character at a time" Does this mean we can have a function with an input-stream as input? Or do we have to include the code that 'opens' the file and create this input-stream?

@KevinCruijssen good question. What's important is that your code receives its input one character at a time, and produces one character of output before receiving the next character of input. If your language's I/O streams allow that kind of access then I guess it would be ok to use them, otherwise you'd have to do that externally, in the code that calculates your score.

Ah ok. So it's also fine to just have a character-array or -list as input, containing the characters of the file?

your guess is case sensitive right?

@KevinCruijssen hmm, I'm not sure I like that. It should be impossible, even in principle, for your program to look ahead at characters it hasn't processed yet. I'd apply that to I/O streams as well - if it's possible for your program to take more than one byte from the stream before returning a byte of output then you shouldn't use them.

@LiefdeWen yes, the guess has to exactly match the next byte from the file, otherwise it counts as incorrect.

... In short, all I/O format are fine, as long as you don't cheat.

@user202729 basically, yes. To be more precise, I'd say "all I/O formats are fine, as long as the program that calculates your score receives each byte of output before sending the next byte of input." (I'll edit that into the question.)

Regarding your 1st sandbox note: I won't include any link here, but some specialized packers can do much better than ~700K. For fairness sake, I guess we'd have to include the size of any 'exoctic' (un)packer (i.e. not included in the language by default) in our byte count. Is that correct?

@Arnauld let me do some research/thinking about that before giving a definite answer. I think what you suggest is probably fair, though.

@Arnauld I've included something along the lines of your suggestion into the question. I decided to draw the line at anything that includes a significant source of statistical data, so it's basically an extension of the rule on pre-trained neural network data. (That said, I tried a particularly well-renowned algorithm and it did about 25% better than bz2, which is significant but not Earth-shattering.)

@Arnauld I created a chat room in case we want to continue this discussion: chat.stackexchange.com/rooms/71390/compression-algorithms

+1 I don't see many great challenges on this site: This is one of them.

Let’s say I write an extremely inefficient code that randomly guesses a letter in the alphabet. Should I run through several times and use its average error score?

This isn't worth submitting as a solution, but is useful as a baseline. If we have a datafile a created with bzip2 -c -9 whale.txt > a and a program consisting of bzcat a, we get a score of 2*(381210+7) + 0 = 762434.

@Ray: Looks like I built something like it; only I actually made it eat its input. I'm in third place right now.

Can the program ingest a wordlist or dictionary file or does the contents of that file count against program length?

@DonielF your program should run deterministically, so it always gets the same score. (That should have been in the question, I'll add it in.)

@JimW since that's data used by your code, it would count against program length

I see that the challenge prohibits accessing the file whale.txt at that URL, but what about accessing another URL, one entirely unrelated to this challenge, which also happens to yield the full text of Moby Dick? Presumably that shouldn't be allowed either?

@DavidZ that's not allowed. (It's a standard loophole.)

@Nathaniel Ah, thanks, I forgot about that one. I take it that doesn't apply to the solutions which are just reading from a compressed version of the file, because in those cases the compressed data is included in the byte count.

@DavidZ that's correct.

@DavidZ basically no a priori data like lists of common English words (without statistical data other than the fact that all words in it are common) etc. are allowed right? The code has to learn entirely from scratch?

@DavidZ I wonder, are you seeing if anyone can beat methods from your own research?

@DonielF "random with fixed seed" is considred "deterministic".

@Andy Apparently some languages have built-in dictionary... Yes, it's unfair for some languages, but the advantage is not significant, in my opinion.

@Andy that's right. As user202729 says, you could use a dictionary if your language has it built in (since there's no reasonable way to ban that) -- but I suspect it will not make that much difference in the end.

@user202729 I generally quite like having that feature, but I'll remove it later if it doesn't turn out to be useful.

I will have the first answer that will actually score perfectly

“If for some reason your language or libraries include a feature that provides some or all of the text of Moby Dick, you may not use that feature.” Aha, I see we now have an anti-Mathematica provision.

Are we allowed to record the presence of a new line? Say that one line ends "about" and the next line begins "the". Will the sequence of characters fed to the guesser be "aboutthe" or "about" + newlinesignal + "the"?

@BrianRisk it would be "about" + newlinesignal + "the". Basically the input is treated as one giant string, including newlines.

Are the line breaks LF (\n), CR (\r) or CRLF (\r\n)?

@NeilSlater that would be fine - you can import libraries as long as you're not making use of statistical data included in them. I encourage the use of NN libraries!

Good choice of book. But! Too bad you didn't use Don Quixote (en.wikipedia.org/wiki/Pierre_Menard,_Author_of_the_Quixote)

@martinjakubik haha, if I'd thought of that I would have done! I love Borges.

It would be interesting to see a similar challenge where the algorithm tries to guess a whole word at a time rather than letter by letter.

@juniorRubyist sorry, I didn't see your post. They are LF (\n). (There are no other unprintables in the file, apart from \n, and the only other whitespace is the space character.)

How do you score a program that misses / adds a character? For example, if whitespace blocks are folded into 1 character?

@CSM your program should always return exactly one byte after receiving each byte of input. If the input was Hello, World! and the output was Helo, World! , then each guess after the first l would be counted as incorrect.

@Andy When submissions get word tails perfect, they will try to guess word by words. Not now.

@CSM In other words, "treat the file as a byte stream".

The scoring weight of 2*L seems misguided. There are trivial solutions packing a compressed version of the full text plus a decompressor in the source that would score better than the current leaders. I think the weight for L should have been more like on the order of 100-1000.

@R.. there has been quite some testing and discussion on that point already, both before and after the challenge was posted, and at this point I'm quite sure that the compression method is not able to beat the current top scores. However, if you're able to achieve a competitive score by that method (without any rule-breaking or loopholes etc.), then by all means post an answer.

@R.., if you weight L too heavily, the competition simply becomes a challenge to find ever-more-concise ways to output the most common character in the target text. At a weight of 1000, any program longer than 1215 bytes is non-competitive, because even a program that outputs Moby Dick perfectly will have a lower score than a zero-length program that gets every character wrong.

OK, 1000 was way too high. Probably something more like 10-50 would be reasonable.

@Nathaniel: Does xz count as standard library of shell script? :-) If so I can beat the top score by quite a bit.

@R.. yes it does. But remember that if your submission needs the compressed file you will have to include that in your byte count. Given that, I don't see how you can score better than 2*392960 = 785920 points. But if you can do so while following the rules, please go ahead and post it!

(p.s. I unintentionally deleted my previous comment before seeing yours - I'm not trying to obscure the discussion or anything. For posterity, its content was that I won't believe you can beat the top score with this method unless you post an answer that does so.)

Oh, yes, sorry, missed the 2x I was complaining about. So it does require some more clever compression but I think it can be done. Perhaps a lossy prepass to collapse similar words so that it compresses better.

@R.. it might be possible, and I'd be happy to see an answer that manages it, but it would require a huge amount of cleverness. The best-of-class algorithm cmix only gets whale.txt down to 290347 bytes, for a score of at least 580694 (but actually much more, because cmix contains a lot of statistical data that would need to be included in the byte count). You might improve on that slightly by using a different algorithm or by using whale2.txt, but not by a lot. If a lossless method can compete it will have to be very heavily specialised to the target text.

@Nathaniel I assume that Mathematica's WordFrequencyData is not allowed?

@KraZug yeah, that would come under statistical data.

If anyone is interested, I'm currently working on a Neural Network solution in C++. I'll post it here once I feel it's ready. Or should I post it already "reserving" my spot?

@BrainStone it's up to you really. If it beats the median score (now somewhat less than the 648412 points I mentioned in the question) then it will win the bounty, so if it does then I would post it now, but other than that it's your choice.

Does a 992,119 byte program that produces it accurately count?

@CSM it would count, but it would score 1984238 points, significantly worse than any answer so far.