The product should be used to create the score, but the individual probabilities should be higher.

If you have TEST as the input, then you want to score as:

oh i see, yeah they're super low, i think the highest is 2.71% - i'm using a log of them though to avoid underflow

p("T") *
p("E" | "T") *
p("S" | "E") *
p("T" | "S")

are those monogram's (probability of single characters)?

p(x) is the monogram, p(x|y) is the probability that the character x follows y

ah ok gotcha

And then in order to normalize for length you can take the length's root (which, as you said with using logs, just means divide the summed logs by the length)

So your log score is (log(p("T")) + log(p("E" | "T") ) + log(("S" | "E")) + log(p("T" | "S"))) / 4

Since you're dealing with emails, you might want to take the max of (score with numbers removed, score with numbers transliterated)

And then tune that threshold so that it grabs the addresses you want

(those are percentages, not logs fyi)

Out of curiosity, what are all the p(x|"E")s?

I would have pegged the combination "ES" to show up a lot more frequently

total (of the 26) is 0.12

Ah, that's your problem

The ...| "E") means ignore situations in which "E" isn't the previous character

so 12%

i'm reading p(x|"E") as probability of x given E

so if x was S, then probability of "ES"

and then i'm looking up the probability of ES in the digram frequency table (one sec i'll give you a link to the site i got them from)

In this case "given E" means that all of the p(x|"E") should sum to 1

i'm not quite following why that should be the case - here's what i did - i took the "Bigram Frequencies" from that site, which has a list of all 26*26 digrams and a count of how many times they appeared in the source text, i then normalized them to percentages such that the sum of the 26*26 percentages summed to 1

i.e. the probability that both "E" being one character and "S" being the next will happen

p(x | y) is defined as p(x∩y)/p(y)

ooooh ok, applied to markov chains, where you already know it's an E, you need to use a normalized to unity version of the 26 probabilities or the next character!

*of

Pretty much

ok so that changes the values of what i posted above - i can fix that, let me read through the rest of your steps now...

ok makes sense - out of curiousity, what do you expect to happen when scoring "test" versus "testtest", does having more text to analyze make it better at identifying, and so "testtest" would be expected to score higher?

I'd expect testtest to score slightly lower, because p(t|t) is not as likely of a combination

maybe this is where chain length comes in - having twice as many "decent" english letter combinations should probably outweight a single p(t|t) penalty

You can try dividing the logsum by a different function of length

len/2?

ah, interesting, will think about that

That's one of those things that you can play around with to tune exactly to what you want

gotcha - ok, will try this out now - thanks for the help, you've greatly improved my probability of success ;)

:D Glad I could help!