Primes and Squares

« first day (364 days earlier) ← previous day next day → last day (860 days later) »

Nathan Merrill

12:00 AM

@PhiNotPi aren't you dividing the number of phones you can use by 3 every round?

oh, I understand

you're trying to figure out how to best represent 20 votes with N phones?

isn't this like every other voting problem?

PhiNotPi

@NathanMerrill No not really, because there is an actual numerical score you can place on an algorithm, being the expected amount of money won. As opposed to a "voting problem" where thus goal is something fuzzy.

Nathan Merrill

right, but from an algorithmic perspective, all 3 answers are equivalent

PhiNotPi

Yes

Nathan Merrill

the only difference you see are the votes

it's like saying "You will win 20$ if X becomes president"

but I won't tell you who X is

PhiNotPi

Well I guess there's two steps to the process: (1) uses the votes to calculate the percentage chance each answer is correct (2) given those likelihoods, figure out the best way to distribute the limited number of phones.

Nathan Merrill

12:13 AM

I'm confused with step (1): Are you going to be identifying who is the best guesser?

or is it going to be a "66% voted for A, so there's 66% chance it's A"

PhiNotPi

Step 1 is a lot like a voting problem in that some people have stronger opinions / are more knowledgeable than others. Step 2 is more concrete and probably solvable once you pick a scoring function.

Trying to identify who is better/worse at guessing is probably very difficult since each question could have a totally random topic.

Nathan Merrill

I mean, if you're doing this repeatedly, I'd just make a neural net that predicts the ability of each player

PhiNotPi

But even if we assume that all people are equally-skilled, then I don't actually know if "66% voted for A, so there's 66% chance it's A" is the correct way to do it.

Nathan Merrill

I don't see any other reasonable solution. There's no reason to make a particular chance lower and another higher

I could understand that for the "distributing the votes" step

oh, I think I get it: If 66% of 2 million people that means its likely higher than 66% chance that that is true

you deviate to the extremes

In essence, lets say that each person has N% chance to know the answer.

PhiNotPi

Let's say that it's the very first round, 20 people and 20 phones. If 18 people say A is the answer, with 1 vote for B and 1 vote for C, then I think you should bet all 20 phones on A.

Nathan Merrill

12:20 AM

if they don't know, then they guess randomly

I need a more concrete number

lets say 50%

PhiNotPi

If you split the phones 18-1-1, then if A is correct then you just threw away 2 phones for nothing, but if it is incorrect then you have pretty much no chance to win anyways (to make it through many more rounds with only a single phone).

Nathan Merrill

I think that those numbers are proportional though

like, if the 18 are right, you have 18*X% chance to win, but if the 1 is right, then they have X% chance to win

anyways, lets say you have 18 phones, and each person has a 50% chance of knowing the answer

that means, on average, you're going to have a vote distribution of 12, 3, 3

because 9 of them will vote for the correct answer, 3 will randomly guess it right, and 6 others will randomly guess it wrong

so, if you got those vote totals, you should definitely go with A

because the chance of only 3 people getting it right are astronomically low

anyways, you would need to have a magic variable N that determines the likelyhood that a person is right

and that would determine the % chance that that is the correct choice

PhiNotPi

I think there should be a way to estimate N based on the vote results.

For example, if you have 12-3-3, then there's clear upper and lower limits on N.

Nathan Merrill

that feels wrong: You're reusing the data to analyze itself

PhiNotPi

(Not 100% clear because of random variability) but basically a value too high would make that distribution unlikely and a value too low will also make that distribution unlikely.

Nathan Merrill

12:32 AM

right, but then you're using that same number to say how likely a particular choice is. I don't think that works, but I don't know statistics well enough to say why

PhiNotPi

I think this is where we start using conjugate prior distributions.

Maybe we should simplify things further and look at questions with only 2 answer choices.

Actually, that simplification may or may not help. Using our model above, we predict ((1-N)/2)% votes for each of the wrong answers and the remainder for the correct answer.

There's 2 degrees of freedom in the result (since there's 3 categories) but only 1 degree of freedom in our model.

So it is a simplifying assumption.

When it comes to questions with 2 answer choices, our model is no longer a simplifying model.

Dirichlet-multinomial distribution

In probability theory and statistics, the Dirichlet-multinomial distribution is a family of discrete multivariate probability distributions on a finite support of non-negative integers. It is also called the Dirichlet compound multinomial distribution (DCM) or multivariate Pólya distribution (after George Pólya). It is a compound probability distribution, where a probability vector p is drawn from a Dirichlet distribution with parameter vector α {\displaystyle {\boldsymbol {\alpha }}} , and an observation drawn from a multinomial...

This looks like a promising article... I'll have to read it.

> It is a compound probability distribution, where a probability vector p is drawn from a Dirichlet distribution with parameter vector alpha, and an observation drawn from a multinomial distribution with probability vector p and number of trials n.

This is pretty similar to what our process is doing.

Any given question can be represented as a "probability vector" meaning the percentage of people who are expected to vote for each option.

These questions are drawn from some global distribution that is the space of all possible questions in the question bank.

And then, once we have a question, the X number of people are sampling from the distribution of that particular question.

But I think there's more to it... because each question is more than just a "probability vector"... it's a probability vector + the assignment of which answer is the correct one.

I'll work on this some more tomorrow when I have time.

22 hours later…