@EthanBolker Thanks for the suggestion, I'll move it there.

There are some advantages of placing it on stats, there are some advantages of placing it here too. I feel like the question requires more common sense than advanced knowledge. But it also begs for clarifications on many issues. First, when you say that you observe only some part of the objects every time, may we assume that the subsets are random and independent for different observations? Second, looking at your functional, one can notice that it is greater for 30,1,2,3,...29 (one badly misplaced object) than for 2,1,4,3,...,30,29 (15 short swaps). Is that really how you want it to be?

@fedja Taking your 2nd point first: this is not what I want, but I suspect it is what is driving the issue I point out (that sets with lower missing rates have a lower M). I suspect that solving this issue is key. To clarify, subsets are not random. The frequency which object 1 appears may be significantly different than object 2, but we can assume that any observation is independent from another observation (object 1 appearing in observation 1 is independent from whether it appears in observation 2, but it may have a higher chance of appearing in any observation than object 2 does).

The issue you point at may be solved by the trivial observation that the deviation in rank in a partial sample is roughly speaking the deviation in the full sample times $n^*_d/n$, so instead of $D$, you should really divide by $D^*=\sum_{d=1}^D (n_d^*/n)^2$ in your notation. Try it and see if the results get better. I'll think of the rest meanwhile.

@fedja Your first point is very intuitive, regarding the ratio of deviation in rank for a partial and full sample being proportional to the size of these groups. In truth, the 1/D does little for me, as all sets are observed equally, but this makes the measure more general. However, what I understand is that I should normalize ranks such that I compare similarly sized objects. Thus my interpretation of what you are saying is that my functional becomes $M=\frac{1}{D} \sum_d^D \sum_i^n (n/n^*_d)^2(rank_{id}-\bar {rank}_{id})^2$. This “scales up” the deviation for observations with high missings.

Looking at that, I also may also need to throw another 1/n into that so that I can compare, say, a set of size 25 to that of size 30.

What you suggest is very similar to what I suggest except you give equal weight to all observations and I give some preference to observations with more objects and you should leave $1/n_d^*$ in the summands where it was in both cases (you still estimate the average deviation, not the sum of deviations). There is no clear argument why one way is better than the other, so decide between the two options (they are both consistent) experimentally. $n$ is already accounted for by that extra $n_d^*$ in the denominator ($M=\frac 1D\sum_{d=1}^D\sum_{i=1}^{n}(n/n_d^*)^2(n_d^*)^{-1}(\Delta r_{id})^2$

I would also try to lower the power $2$ to $1$ (just average absolute values). That somewhat mitigates the effect of big misplacements of few objects. Of course, the scaling should be changed to $n/n_d^*$ then and $M$ replaced by roughly speaking $\sqrt M$.

Sure. I just wonder if the multiplication by $n/n_d^*$ improved the outcome before trying to say anything else. I still have a few questions though :-)

To update, I've tried constructing measures using equal weight for all observations (one using the square, and another using absolute value), but without success. I construct qqplots to compare the distribution of participation rates (missing rate) for those above and below the threshold (I tried a number of thresholds), but the plots showed them to be quite obviously of different distributions. I also ran the balancing test on key observables, and it was not balanced.

I can conclude that either there is some selection around which sets are ordered (I have no reason to believe that this sort of selection makes sense, but I am not sure how to test it either), or that the measure M is not invariant to class size. I can address the first by creating some simulated data and running this type of test for simulated data which I am sure does not suffer from a selection problem.

I may be able to address the second by pursuing the other type of measure you described, but I am not sure I fully understand your suggestion. Replacing D with D* means I am averaging over a different number than the days D, but over a number related to participation each day. I’m not sure I understand the intuition here.

I'm happy to try to answer further questions you have, and appreciate your thoughts thus far!

OK. Let's just do some sanity checks first to make sure that there is no misunderstanding. When you build your sums, do I understand right that the following goes on: you first discern the "suspected order" (say, 1,2,3,4,5,6,7,8) from all observations.

Then, if you have an observation order 7,3,5,1,8, you just compare it to the ordering of the same objects 1,3,5,7,8 and the rank differences become (4-1), (2-2), (3-3), (1-4), (5-5) (where (1-4), say, is the difference between the position of object #1 in the observed order and its position in the suspected order)?

Then, if I understand things right you just take the average of squares of those 5 differences (18/5 in this model case), multiply by (8/5)^2, and get the value from one observation. After that you average over observations and compare the result to $M$. Is this description correct?

Sorry, (1-4) is for object #7, not #1

the first part is correct, I'm now confirming the average of squares piece.

Yes, all of that looks correct

Where $M$ is some threshold

Great. Now with other averaging, you just don't multiply by $(n/n^*_d)^2$ at each step, but make the denominator $D^*$ the sum of $(n_d/n)^2$. It is like putting weights on the observations. My reason for it is that the larger $n_d$ is, the more meaningful the observation is (say, if $n_d=1$, you get no information whatsoever).

This is not the main thing though. You say that the distributions are different meaning, probably, that the passing rates are different. I wonder what happens with a fixed single set of observations if you just remove some part of objects at each observations at random and do the test for modified observations. Of course, the new value of M will be now slightly different but I'm curious how much different (on average: since we introduced some randomness, there will be some deviations).

What I'm trying to see is whether we have the correct adjustment for the size of the observed set (which I assume to be random in some sense). The intuitive factor is as I wrote, but, perhaps, there is some subtlety I overlooked. I'll try to play with some programming too in the meantime.

I think I follow – what you meant was not a direct substitution, but instead of averaging over D (or D*), reweight each observation as we sum them over the days. To make sure we’re on the same page, the other averaging would look something like $M=\sum_{d=1}^D (n_d^*/n) \sum_{i=1}^n (\Delta r_{id})^2$.

That sounds like an interesting test. I’ll introduce some noise to the sets (or maybe a subset of them) as you suggest, then compare the value of M in each before and after. I’ll post the results tomorrow.

$M=(1/D^*)\sum_{d=1}^D \sum_{i=1}^n (1/n_d^*)(\Delta r_{id})^2$ where

$D^*=\sum_{d=1}^D(n^*_d/n)^2$ and $\Delta r_{id}$ is understood as described in the sanity check part if object $i$ is observed and as $0$ if it isnt'.

One more question: how many observations do you have for a single set?