I think you may be reading their hypothesis test backwards. As I understand it, they are testing the alternative hypothesis "cases went up after the protests" against the null "they did not go up". The conclusion is that they fail to reject the null hypothesis. Indeed, if the raw numbers show a decrease, you certainly cannot reject the null.

That is a good point, I will revisit, but as you have pointed out - a CI containing zero renders any hypothesis inconclusive :S

@Nate Eldredge : Yes - so what it would be better then to report the results as would be "if there was a rise, it was too small to be 'heard' over the 'noise' of the much larger surge caused by reopening everything", not that there was no rise at all.

Couple of other points from the paper: It relies on the assumption of violence keeping people away (so it's not so much an endorsement of protests but violent activity), the reported effect is small (~2%) as to be negligible, it's reported the effect goes away after a week, and finally it's based on smartphone tracking which is unlikely to be an accurate representation of movements (if people EG leave it at home precisely to avoid being tracked when violating curfew etc).

The expression "almost certain" has a very specific meaning in mathematics. I wouldn't expect reviewers to have any issues with it nor would I call it "meaningless jargon" (although I would agree that it's not intuitive to a lay reader).

@Carl: Actually, the phrase in mathematics is "almost sure", and although "almost certain" may look like a synonym, mathematical writing doesn't use synonyms this way; technical terms are used verbatim and not substituted. (I am a mathematician who works in probability theory; I see "almost sure" in papers all the time, and I don't believe I've ever seen "almost certain".) But more to the point, if they really do mean "almost sure", i.e. "having probability 1", then they are claiming something that no amount of empirical evidence could possibly establish.

@The_Sympathizer: I think the writing is kind of inconsistent. In the abstract and on page 2 they write "We find no evidence that [cases increased]", which seems fine. It's where they say "we find that the protests had little effect" that I think is questionable, and they probably should have written "we did not find that the protests had a significant effect".

@NateEldredge I'm not a mathematician and you may be right about the gist of the article being inconsistent, but Wikipedia and Encyclopedia of Mathematics clearly state that "almost sure" and "almost certain" are synonyms. Not that it makes it true but it's certainly not unheard of. However I just didn't want the uninformed reader to think that terms like "almost sure" or "almost certain" mean that the paper is somehow unscientific or that the authors are just guessing.

The major problem with this paper is that it focuses on the immediate aftermath. What you would be actually worrying about as a medical professional is not a bunch of young people getting covid, as they would be largely asymptomatic or have mild symptoms. The real worry is if the protesters caused a secondary spread of the virus spread into riskier groups. Looked at this way, you may not fully see the effects of the protests until a month or more later.

Since when has the media ever refrained from reporting preliminary study results pending peer review and verification? Not that I disagree that they SHOULD wait, or they should be much more circumspect in how they describe the information, but they don't, as a matter of practice and policy. Is there any particular reason why this study, moreso than all the others that get reported on, should be subject to these standards? If the answer is "no," no more elaboration is needed, because I'm not saying that shoddy journalism (true for almost all reporting on medicine or science) is fine in general.

I'm not sure why the complaints about "almost certain." I just did a search in the linked paper, and the only place 'almost c' appears is "While it is almost certain that the protests caused a decrease in social distancing behavior among protest attendees" (p.25)--which seems like both a conventional colloquial usage and also a pretty obvious assumption.

@carl I am a mathematician. The specific technical meaning of "almost certain" is "except for probability zero events". Not "extremely unlikely", but "technically possible, infinitely unlikely". I'm talking "you flip a fair coin forever and it NEVER EVER comes up heads". Impossible? No. Going to happen? No. This language belong in a technical sense in any paper that deals with real-world data? No.

@obsucarans I'm familiar with the meaning and that's why I mentioned it in my original comment. However, as Tiercelet helpfully pointed out, none of the comments (including mine) are relevant in this context because the expression is clearly being used in the colloquial sense rather that the mathematical sense. I (and others) should have have read the paper in full before commenting. Lesson learned.

@Tiercelet you have a valid point. I chose that as my example, but it was a poor choice. The article is for sure presented with fair rigor. I was trying to sum up several instances of loose wording into one poor example, but I will modify my answer to note that overall, they have data to go with their arguments. I cannot comment on the quality of the data or data analysis

Plus one from a fellow Matter Modeler. The paper is not only not peer reviewed, it's also a "work in progress" meaning not even under review? To be under review it at least has to make it past the editor! It probably wouldn't make it past me.

-1 Highlighting and trying to paint "In most cases, the estimated longer-run effect (post-21 days) was negative, though not statistically distinguishable from zero" as something incorrect is a very bad sign for this analysis. The boldfaced conclusion is completely customary in scientific/statistical writing, and the meaning is perfectly clear. The sample rate decreased, so the population rate probably didn't increase.

@DanielR.Collins I disagree with your assertion. If you are testing a hypothesis for a rate, it doesn't matter whether you are testing if it is negative, positive, not negative, not positive, if the CI includes zero, you cannot conclude anything. You could drop the significance level, say down to 90% to get a tighter interval about your average, but then, there is the obvious downfall of really, not being that certain. 95% is customary,

@CharlieCrown: Your statement is wrong on multiple points. Hypothesis tests always have an alternative hypothesis of the form a > b or the like. Literally every time the associated CI for the difference contains 0, the hypothesis test will have a conclusion like "the data provide no evidence for a difference distinguishable from zero"...

... E.g. from Weiss, Introductory Statistics, Sec. 10.2, "The Relation Between Hypothesis Tests and Confidence Intervals": "the null hypothesis will be rejected if and only if the (1 - α)-level confidence interval... does not contain 0"... and, "If the null hypothesis is not rejected, we conclude that the data do not provide sufficient evidence to support the alternative hypothesis" (Sec. 9.1: "Key Fact 9.2"). That is exactly what's happened here.

@DanielR.Collins I don't think we are disagreeing so much as looking at the same object from different angles. In simple terms a confidence interval (CI) is used to determing the bounds of which we would expect to find the value, 95% of the time. (if we are using 95% CI). If in this case the CI includes 0, then we cannot say that 95% of the time we expect the rate to be negative. To which I am simply saying, there is nothing useful found. At 95% confidence, we expect the value to be negative, 0, or positive. What conclusion does that tell us?

Where I have made the assumption that by the CI including zero, it overlaps it sufficiently to include some positive values, however small... zero is not the limiting value from the left hand side)

You say, "we cannot say that 95% of the time we expect the rate to be negative". Do you think the quoted paper has said that? It does not. Moreover, that wasn't even the hypothesis being tested.