I did this problem in high school but no longer know how to

quite annoying.

@DanielSank Yeah, I know the questionnaire is somewhat complicated. It's limited by the capabilities of Google Docs. This is quick-and-dirty data collection, so I didn't feel the need to make a streamlined form. (Maybe that was a mistake?) However I didn't want to do meta posts because having the vote counts publicly viewable introduces a significant bias.

@DavidZ I see.

I'm a bit at a loss. I tried to fill out the form for the first two questions and in both cases kind of felt that the available axes on which to judge were not really part of why I thought the question were good or bad.

In fact, for several I'm not sure how to answer.

I wonder if we're artificially constraining the discussion with those axes.

"discussion" = "analysis", sorry.

I can see that experimental analysis of this homework issue is a rather difficult task!

@DanielSank Indeed, and there is no implication that they should be. I'm just looking to collect people's impressions of the questions on those axes, without tying it to a judgment on the quality of the question.

Right, so what should I do if I sort of have no opinion on any of the axes though?

Let's pick an example.

First question. Do I think the OP did enough research?

Well, yes, in a way. They worked through the problem carefully.

However, the real questions in the end include how to do an integral.

Is there research on that? No, none at all. Is that relevant? Not really. If someone's asking how to do an integral there's often little point in showing failed attempts.

But the real issue at hand is that the post asks too many separate questions, and all of the axes in the questionnaire can't capture that fact.

So, :\

This did really help me though! I think we might identify "focus" as something missing in a lot of closed-as-homework posts!

How to convey that through official channels relevant to the homework policy rework?

@DanielSank Too many separate questions are a case of being too broad.

@DanielSank I'm not quite sure, exactly. My best guess at a next step is that I'm going to put up a meta post showing the results of this round of data collection (which seem relatively inconclusive) and asking for community input on where to go from there. I guess that would be the place to make an official suggestion regarding focus.

If we're closing too broad posts as homework-like, then something is going wrong

@ACuriousMind Yes, but see there again the close reason fails us by not adequately explaining the problem

> There are either too many possible answers, or good answers would be too long for this format. Please add details to narrow the answer set or to isolate an issue that can be answered in a few paragraphs.

Oh, and yes, I agree with what @ACuriousMind said.

@DanielSank Hm.

This does not say "You're asking too many questions at once. Please separate into multiple more focused posts."

You're right

Which is something I've typed into a comment box one bajillion times.

I do think that's a bit of a distortion of "too broad" from what it was originally intended for, but it's close enough for my purposes.

No, it's not.

The close reason needs to tell the OP what to do.

If it doesn't, you get a confused, pissed off, less educated user.

@DavidZ Well, what reason are we supposed to choose for posts that ask several questions that could each be answered separately?

I use "too broad" and sometimes leave a comment

@ACuriousMind I write a comment every damned time.

Maybe we need to reword the too broad close reason to cover this!

Take that one to Meta Stack Exchange

@DavidZ Ah

@DanielSank Yeah, but this reason is common enough that I shouldn't have to choose the "other" off-topic reason to cover this case

@DavidZ ok, added to to my todo list, which is the only guarantee that I'll ever do anything :)

But I agree with you that the text of the too broad close reason doesn't actually match this case

@ACuriousMind Indeed not.

So let's not pretend it's a fix for the problems in the homework close reason.

Right?

@DanielSank Sure, but I think the "lack of focus" is another reason to close posts, but not one that should replace the homework policy. One can have perfectly focused posts that are still just boring and blatant requests to do the asker's work for them.

@ACuriousMind Sure.

The fact is, the instructions we have to give people to show them how to improve their questions to get a hold removed are far too varied to cover with even 5 custom close reasons (if we can get the full 5) plus the standard ones. The trick is to find custom reasons that do an acceptable job of covering as many cases as possible.

However, we're lumping lack of focus into the discussion about the homework policy and I think that's a mistake that's making this all more confusing than it should be.

@DavidZ I'm only suggesting adding a few words to the "too broad" one.

I'd even be ok to lump that into the homework close reason if we have to.

We can't add to that reason (unless you convince SE to change it)

@DavidZ Well, I'll raise the issue.

But my point is that asking multiple questions is qualitatively different from anything on the questionnaire, and from anything people are talking about w.r.t. homework.

Yet, it's a very common problem with homework-like questions.

Perhaps the new policy ought to call this out?

Sure, it could.

@DanielSank I think it's a secondary effect: People too lazy to ask a proper conceptual question also don't care about asking a focused question.

When I was making the form, I put on items for as many factors as I could think of that might be related to reasons people want to close questions. I didn't think of that one. But it's perfectly valid.

@ACuriousMind Perhaps. Then again, asking people to focus their post is a very clear way to put them on the right path.

That is one I would definitely put on the form if I were doing this again.

@DavidZ Sure sure. Just explaining why I'm having trouble filling it out, and why I'm uncertain as to how to interpret the results.

@DanielSank If I would offer one general suggestion, it'd be not to think about interpretations too much, and just take the items on the form at face value.

@DanielSank I agree - we should mention this somewhere. But perhaps this belongs more in a general post on writing good questions, and not in a close reason (unless we introduce a close reason specifically for asking too many questions)

@DavidZ Fine, but I'm still having trouble filling it out.

I really don't know how to judge research, for the reason explained above.

I can just click a bunch of numbers but I sincerely don't know what I mean by any of them.

It's your personal opinion. Do you think the question shows enough prior research to be on this site, or not? Or to put it in different terms, do you think the questioner would have to have done (and shown) more research to make the question acceptable for the site?

And of course you don't have to do any of this. All the questions are optional, and filling out the whole form is optional.

Honestly, I would say skip anything you find yourself thinking too hard about.

Well, I think the problem is that homework questions can't really show research - they're asking how to solve an equation or problem; if they had found the solution by research, they wouldn'T be asking.

I know it's optional but it's part of fixing what I think is the most important present "bottleneck" of the site!

@ACuriousMind There can be research shown that doesn't lead to a solution.

^ agreed

@DavidZ I think most questions that show such research usually tend to be more conceptual in nature anyway

Sure, that could be

DavidZ you just said something important. There's a difference between rating a question on a scale of "how much does this question do X" and "how much could improving X make this question better".

Those are very, very different.

The second one is considerably more answerable.

Not in the context I was talking about. I'm guessing what I meant was closer to what you mean by the second of those.

@DavidZ Hm, ok.

A typical thought process I had in mind is like this: you look at what is being asked, and presumably you have some idea of what you would expect someone asking that question to have checked before posting here (in your opinion). Does it look like the poster checked those things? If so, mark a high number. If it seems that they have not, mark a low number.

Oh, that's the first thing in my list of two ways of thinking.

Well actually, no it's not.

It's a third option.

Yeah, it's not quite the same. But I wouldn't say that's the way you have to go about it. That's what I had in mind, but if people have different ways of evaluating a question's level of prior research, that's cool too.

perhaps, but it completely changes the resulting numbers.

If I vote "could doing this more make the question better?", I might say zero, whereas if asked "does this do enough research?", that's a five (or whatever the scale is).

So clearly the former is not what we're asking.

I guess the other two options are more similar.

Yeah, I do think it's less like the former.

A 5 (top of the scale) is meant to indicate that it's clearly sufficient for the site, not necessarily that it's the best one could possibly do.

ok

I'm starting to think we might have to redo the data collection with much more precise instructions what the scales are supposed to mean

Maybe "interesting" is only so significant because everyone knew how to judge "interesting"

That would be a complicated process

Actually, when I redid the analysis, the correlation with "interesting" went down

@DavidZ It would be, yes

And I'm not sure it's worth it

@ACuriousMind Yes, and actually I think this reflects why people vote the way they do. The close reasons don't make sense and/or don't include what the voter wants to say, so they just click the gigantic catch-all called "homework".

That's an updated chart

What do the numbers mean, again?

The numbers are correlation coefficients.

@DanielSank Maybe here's a more interesting experiment, although I'm not sure how we would collect the results: For a week, everyone picks "custom" as the close reason and types out their reason for closing the question manually, even if they think one of the pre-made reasons would apply. Possible pitfalls: People not voting to close because it's too much of a hassle, or people just typing out "because it's unclear what you're asking" and "because it's too broad".

Although we would probably just end up with a caleidoscope of reasons with little rhyme or reason to them...

@ACuriousMind That certainly would be high quality data, even if very hard to analyze.

@ACuriousMind That depends on your ability to parse/search/categorize natural language, which is rather difficult.

...on a computer anyway.

Yeah, it's something that would in principle give interesting data, but I can't see a qay to actually extract meaningful information

Even getting all the comments is probably a bit difficult

It probably wouldn't be too hard to collect all comments that start with "I'm voting to close this question as off topic because"

Evaluating them probably amounts to people manually sorting them into categories

Yep

Well, we're closing what, like 200 questions a week? Less? more?

Probably less

My tools indicate about 30/day, roughly

Wow, that's a lot.

@DavidZ Heh, 200 was a good guess, then B)