The specific example you mention, of asking a trusted individual from country X for advice in evaluating students from X, is I believe very common. I think what we're trying to do is essentially the same thing, though less subjective.

Exactly. The point was finding the "context" in which to evaluate the student. The same way the first (undergrad) study is trying to put the schools in context (i.e. you might get an invited admission if you're an excellent student from an excellent school, but not from an average school), the second one is trying to put the transcript in context (i.e. does the mediocre transcript by absolute value really mean a mediocre student or not). And of course, in your case of a large student pool, such a context has to be based on a large pool of "opinions" - evidence, and not on one individual.

@posdef Actually, it is very normal for GPA on one scale to not map linearly to GPA on another scale, so that job ad is perfectly plausible. See e.g. academia.stackexchange.com/questions/9058/… (and many similar questions on this site)

Well, actually, the 3/5 GPA in Croatia is more like 75% since the grade 1 is a failing grade... But again, not really, since all the passing grades are distributed as in lowest 15% of the students gets a 2, next 35% gets a 3, next 35% gets a 4, and the best 15% a 5. So, "yearly score" of 3/5 actually means that the student is "somewhere between the the best 75% and 50% of best students who got all passing grades"...or something similar like that. I'm just trying to demonstrate that to recalculate the GPA of ~87.5% to Croatian system is really not so trivial and obvious.

@posdef Ups, sorry, meant to @-reply with my last comment. But, also, I wanted to add what I edited in the question: I think interpreting the results from an internationally standard measure different across countries is not okay. But if a country has it's own country-specific grading system, studying and making criteria based on their grading scheme is better than just "wigging it" and hoping you interpreted it correctly. Also, I don't think the "job add" sounds very unfair -- I actually kept my GPA unnecessarily high during Uni, because I didn't think non-Croatians could interpret it well.

who might get rejected for no real reason - well, they would not be rejected for having a high GRE score! It just wouldn't help them as much; the other criteria would "count more" than the GRE, e.g. they might be accepted or rejected due to GPA. The alternative - if we consider GRE equally "important" for all candidates - is that we might reject someone based on GRE when we have no reason to think it will predict their success. For example, we'd reject half the good students from Country X in the example above for no reason if we count their GRE score.

I'd say, if it's permissive criteria (i.e. all the students above a medium good score in X will be examined further) instead of restrictive (i.e. all students below a fairly high score will be rejected), it is okay. But, on the other hand, if you have such permissive criteria and then additional indicators Y used to evaluate students, why not saying that everybody above this med-thresh will be further examined: surely the candidates with strong applications will confirm this even using indicator Y?

surely the candidates with strong applications will confirm this even using indicator Y? - I wish! that would make things easier if it was true :) An indicator that is "good" (has a meaningful relationship with potential success in our program) for one population is not necessarily "good" (i.e., has no meaningful relationship with success in our program) for another population. (Like GRE for Country X in the example in the question - successful candidates just as likely to have a low GRE score as a high one.)

@ff524 Well... I don't know what to say. I definitely support building a different "interpretation technique" for the different by-country systems. But, if something is internationally standardized, and you accept one candidate because only because of his scores on (one or more) standardized tests, you should accept any other candidate with the same standardized results. After all, he (possibly) did "everything in his power to have a profile (known to be) needed to get past the admissions". It's unfair, true, because the standardized tests are not a good measure, but... well... darn.

"internationally standardized" is really a misnomer, IMHO. It implies that the exam is given under the same circumstances and is equally meaningful for all candidates, which is simply not true...

@ff524 Can we move this to the chat? I won't be able to reply real-time, but at least we won't be spamming the comment section. And maybe somebody should flag the entire question and ask the mods to clean all the comments a bit.

I deleted my comments on your answer to clean up, since they're now preserved here

@ff524 Well.. I know. But the fact of the matter is, that's not your system that is broken and you can not fix it. Back to my high-school example: if the admission criterion was simply GPA-equivalent, well... there's schools where having straight-As is almost impossible, and there's small, rural schools which will give straight-As to their best students to "improve their chances against better schools". But the fact of the matter is, they all have a "certificate" issued by the same authority.

Yes. So the question is about what we can base admissions decisions on, given that the system is broken

@ff524 That authority is on a national level, and the university can not challenge those scores. If they're taking them in to account for one person, they are taking it in to account for everybody. Queue an admissions exam, the same for everybody, executed by the uni, which is their way to "amend" the system.

It's up to the university to decide how to use the scores, though

Yes, but if it's an certificate by a certain authority, even if you know the system is broken, you should value the certificate the same for everybody. Maybe, not give it much value at all, and base the decisions on more personalized / specialized tests, but, ...

If we had any single indicator that was good at predicting success for everyone, we would definitely use that

@ff524 It would be the same if you said that ppl who got their drivers licences in small rural places (with e.g. low trafic, low traffic ligts etc)... are not allowed to drive in the city, because you "know" the system is broken.

@ff524 just to clarify though, we are talking about some "internationally certified" scores which are good indicators for some and not for some other countries?

Suppose admissions decisions can be made on some combination of three major criteria: undergrad GPA (assume we have managed to scale it appropriately for the system the student came from), GRE, and TOEFL

well.. as I said, if the GPA is not calculated in the same way initially for all the applicants, it is okay IMO to make different criteria for each country. I'm not familiar with the GRE however, but if it's the same exam for everybody, yes, unfortunately, I think it is wrong to value it differently for different ppl.

Sorry, I stopped in mid-thought. Let me finish

:)

Suppose you come from a country where GPA is pretty standardized, so that I know that if you have a given GPA, it "means" something. So for you, I might count GPA 80%, GRE 10%, and TOEFL 10%

Whereas I come from a country where GPA is totally not standardized, so a GPA of 3.4 could just as easily mean a really excellent student or a really terrible student

For me, obviously it would be unfair to take into account my GPA as 80%

so suppose we use GRE 50%, TOEFL 50% for me

does that seem fair to you?

sorry, but... no :/ it just ... sits wrong. What about something like this: you say GRE of X is the minimal needed to be considered for admissions. (now I have a long taught...)

you say, everybody with GRE of 98% or higher gets directly admitted (or skip this part all together), and everybody with a GRE of 70% or higher will be further examined (sorry if the precentages don't make sense, I'm not familiar with the measure, but it's just an example)

or, maybe you count GRE and TOEFL scores together in some way to separate definetely-not, from maybe, from, definitely-yes

and, then, in the maybe pile, you examine the "non-objective" criteria. E.g., you examine the GPA based on country of origins, maybe add some value to recomendations and/or personal statements and any other supplementary material (e.g. publications, other professional experience, etc.)

but the problem is, not all criteria carry equal weight for all scenarios

and they shouldn't

well, that's fine, but I think that isn't the case in my suggestion. You're just using a relatively unreliable factor (GRE) to reject the definite rejects (I guess an extremely low GRE score would be bad even if good GRE scores weren't very indicative)

and then, once you have your maybe pile, you base the decision on a combination of different criteria

It sounds to me like your suggestion is to use GRE as a basic filter, and then use subjective criteria to make admissions decisions

that certainly sounds "fair" but we don't have any reason to believe those subjective criteria are indicators of success in our program

when I said subjective criteria, I meant "not global" , as in, examined case-by-case. so, wen doing that, you should know what to look for if you made your analysis

something like that, yes. Basically, I'm suggesting one or more criteria (e.g. GRE) as "weak classifiers" -- the technique from Machine learning that says in short: find a lot of classifiers that will be correct in just above random, maybe 50% of cases. (big taught)

then, reject the definete rejects form the first classifier, and give everything else to the second weak classifier. Repeat for as much weak classifiers as possible

and this, surprisingly, gives very good and accurate classification results

We did actually try that technique, and the results were nowhere near as accurate for our particular problem

in other words: When we apply the same criteria to everyone, we make admissions decisions that don't accurately reflect whether a student will succeed in the program

okay, but after initial rejections based on stuff that is considered more-or-less-objective for everybody, why not just say "you are now considered for admisions"

then , you take all the "under consideration" files, and go through every one in person. Sure, you have guidelines (e.g. GPA if the student is from X), but they're not hard rules any more

If we make those initial rejects based on criteria that apply to everybody, we end up only rejecting a very small number of applicants.

then we are left making completely subjective decisions about the rest, which is not a good way to run an admissions program

well... poppycock

The problem is that again, the objective criteria don't give us the same information gain on every population

on some populations, a given criteria reduces the entropy of the group a little bit; on another group, the same criteria doesn't reduce the entropy at all.

there isn't any single criteria that gives us meaningful information about every group, and reduces the entropy by a useful factor

I understand. But again, even if you know that statistically, city-dwellers are better drivers, you still can't forbid licenced people from rural areas to drive in the city.

We live in an imperfect system,... we do the best we can :(

I don't think the comparison to driving is valid

I'm not automatically rejecting a students from a certain country because statistically, they do worse in our program

I'm recognizing that the criteria that distinguish between an unsuccessful and a successful student from Country X are not the same as the criteria that distinguish between a successful and unsuccessful student from Country Y

Students from all countries still have a chance to be recognized as successful students

I know, I know. And I understand your dilemma, I do. But, if somebody has a internationally certified something, you can not just say "we will not consider that certificate in our decesion at all for a part of the population, because we found out it has no value for that population"

Really? Why not? If I've found (for the sake of argument) that people in Australia can somehow "buy" a GRE score, I think it would be valid to not consider GRE scores for people from Australia, even though some people there come by their GRE score legitimately

because I simply can't trust a GRE score with the population of students from Australia

And similarly, to let the GRE score from Australians stand would not be fair, because in other countries people can't "buy" a GRE score

Why not.. okay, well. If an international organization "A" is the authority for issuing some certificate, and you find out it is "bad" for a part of the population (e.g. they can buy a certificate), why would you trust that certificate, at all, or the organization A that issued it.

How would you be sure that, even if you don't have evidence, a similar thing is not happening somewhere else? If A allowed it in Australia, why wouln't they allow it somewhere else (which you don't know about)?

Well, it would be nice if we could discount the criteria that are "bad" for part of the population, but then we are left with no criteria left by which to evaluate our applicants

and I can't be sure that GRE cannot be bought in a country, but I might have evidence that says "GRE has a meaningful relationship with success in grad school for Papa New Guinea, but not for Australia"

I know, but... if something is issued by the same authority, the certificate "proves" that you are capable of certain things, etc, etc, it shouldn't matter where you got it. You either trust the quality of the certificate, or not

In a perfect world, yes

:)

Well.. go back to my high-school example... or even, I'll go a level lower and talk about highschool admisions in croatia that only uses the GPA (not a good system, but that's not the point here)

It is well known that rural schools give their students higher grades in average (to "level their chances" with the city kids), than in the city-elementary schools.

Yet, after they finish the last grade, all the children will have an identical diploma, signed by the Ministry of Education of Croatia, saying they finished their primary education with the grades such-and-such

If you were now considering admitting one of two straight-A students with only 1 open position left... And no other criteria than the GPA...

Would it be fair to accept one straight-A student over the other, even if you have statistical evidence that a GPA is more meaningful for some students than the others?

I don't even want to talk about choosing one student over another, because that's not really how M.S. admissions work; so let's change your example:

Well, it still comes down to the same thing, they don't need to chose 1 student out of two, but they have to draw a line somewhere, at a certain GPA.

Suppose I have found that for students from the city school, 80% of those with a GPA of 3.5/4.0 succeed in a given university program, and 20% fail,

while for the rural school, 20% of those with a GPA of 3.5/4.0 succeed in the program, and 80% fail.

If I get a student with a 3.5/4.0 GPA, should I admit him?

It's one of the following 3 options: a) yes, b) no, c) pending on the interview / standardized test you made for everybody

There's no c), you just told me I have no criteria other than GPA!

it's a) or b)

Keep in mind that the university costs money - so there are major problems with admitting someone "just to give them a chance" when you really have a strong reason to believe they will fail

what I ment to say was, you don't have other numerical criteria. You don't have other stuff they "bring with them" upon finishing their previous program.

You can't just say we should ignore numerical criteria if they don't apply equally to everyone

then admissions becomes entirely subjective and not evidence-based

our department doesn't do M.S. interviews (the volume of students is just too large)

I'm not saying that. I'm saying if the previous school does not provide enough information, you get it yourself.

and we couldn't add an extra exam

and I assumed you can't do interviews, that's why I didn't suggest it earlier on

I don't know what you mean by "get it yourself" - how?

I ment extra exam. But why not for an extra exam? I've seen this model function before

There are two major problems:

(1) Hardship to students

How would this exam be administered? It would have to be in-person at some testing center, otherwise we couldn't certify that the student submitted their own results

this involves some cost - usually borne by the applicant, our department certainly can't afford to subsidize it

not to mention hardship related to travel to a testing center, which is not accessible to everyone

for example, for student in Iran, they might have to travel to another country to take the exam because of the current sanctions against Iran

(2) Validity

I don't have any reason to believe we could design an exam that would accurately predict performance in our program

I also don't have any reason to believe that the results of this exam would be less biased by country than other "standardized" measures, such as GRE and TOEFL

Well, I simply think that if using standardized results, for anybody, you should use it the same for everybody.

While well aware that will probably cause some bad seeds to slip trough, as well as some good people to be rejected.

I don't think it's ethical to use standardized criteria different for different people. Even if I know the criteria is not a perfect indicator of what ever I'm trying to measure.

The problem isn't necessarily that the criteria isn't a perfect indicator

the problem is that the criteria is a better indicator for some than for others

e.g., for some it is just random noise

It seems unfair to apply a "standardized" criteria to a student when I know that it has no bearing on his success

I think the key issue here is that the so-called "standardized" exams are not really standardized

Oh, I understand the issue. And why it is frustrating and unfair.

There have been several comments along those lines on my question:

Ben Voigt: "It's well-known that GRE and other "standardized" tests are not administered in a standardized way worldwide. And why would it be up to a single department recruiting students to try to fix the irregularities in the GRE?"

JeffE: "If the GRE/GPA scores have been obtained under equal circumstances... — But they haven't. And you know that"

I don't necessarily think that GRE/TOEFL can really be considered "standardize" in that sense

and there is also the issue, that if you apply standardized criteria equally to everybody,

the part that is left is unequal

so suppose I apply 20% GRE and 20% TOEFL to everybody, and 60% GPA

that's not fair to those for whom GPA is not meaningful

you can't separate the "standard" and "nonstandard" part of the decision

You know, I understand you issue. I understand you trying to fix it. And I understand people saying tat if you use something internationally standardized, you should use it consistently. And honestly, finally, I don't know what kind of advice to give

Ok, on that note, I have to go :)

me too

Thanks for the chat :)

cheers