Discussion on the closure of six betas

user2334

Apr 25, 2012 20:32

Astronomy, Economics, Literature, Firearms, Healthcare IT, and Theoretical Physics getting shut down

4

Gilles

Apr 25, 2012 20:44

@MarkTrapp and without warning that I could see

user2334

Well, the writing was on the wall for a long time

user2334

I have been saying SE needs to be more forthright with this stuff though

2

Shog9

Apr 25, 2012 21:00

@MarkTrapp Tough call... Posting, "Your site looks bad and we might have to shut it down" can be sorta... demotivating. Also covers like 20% of the sites, any one of which could turn it around. Mods have access to a ton of stats, and there's this Area51 wall where if there's a lot of "Needs work" you should at least... y'know... work on it.

Gilles

@Shog9 many sites have “needs work” in terms of traffic pretty much right up to the point they graduate

@Shog9 “your site looks bad and we're shutting it down now” is a lot more demotivating

A lot of good contributors won't be back for a second round

Shog9

@Gilles At that point, you kinda want folks to stop posting.

@Gilles Quite possibly. Which is why we're not shutting down 16 sites.

Gilles

“Your site looks bad and here's what you should do, if you don't we might have to shut it down” would be a lot better.

I would be angry if you shut down French Language and Usage without warning, after all the energy I've put into it. I understand that others would feel the same way about other sites.

Shog9

@Gilles Well, that's what I'm gonna be trying on a few of the other ones that were in the original list. Without the "or we'll shut you down" bit.

@Gilles Stats on FLU are decent at present. Biggest potential issue is the low number of questions per day, and even that isn't abysmal. Home page shows just over two weeks worth of activity. If we're into August and it still looks like that, we might be a bit concerned, but hopefully it will continue to grow.

user2334

Apr 25, 2012 22:20

@Shog9 The problem I have with the whole process is people get linked to the same 2 blog posts from ages ago whenever they ask for feedback up until the point they graduate or get shut down.

user2334

If SE could check in once a month or even once every two months with a meta post saying "this is where your site is at, here's what we'd like you see work on", I dunno: I feel like it'd be a lot less jarring

user2334

You guys did that once for Healthcare IT when you couldn't find any moderators and it lit a fire under them for a bit

Shog9

@MarkTrapp Well, that's what we're working on doing with quality evals, which should start showing up with more regularity as time progresses.

@MarkTrapp Yeah... I wouldn't exactly classify that as a success story

user2334

@Shog9 I still don't understand the point of them. Some of them have SE's thoughts on them, others don't.

user2334

I'm suggesting SE needs to be direct about what it thinks of the sites

user2334

Apr 25, 2012 22:29

You guys didn't wake up this morning and decide Literature needs to shut down, for example

Shog9

@MarkTrapp There's been a lot of experimentation. I'm a huge fan of including my own evaluations within them, certain other team members are... less enthused.

user2334

@Shog9 Fair enough. But even with the feedback, they're a check for quality: sites that have quality issues tend to be shut down in private beta. Public beta sites going back to Atheism and Gadgets get shut down for lack of content and traffic. But it's hard to tell what SE considers acceptable in those metrics

user2334

Like take BH: it's doing worse than half the sites that shut down today

user2334

Or Poker for that matter

Shog9

Apr 25, 2012 22:45

Poker's too young yet, but it doesn't look good.

BH is an interesting case - I have a meta post planned there too, as they were on the list at one point.

There are a lot of sites that are not doing particularly well. Our goal here was to shut down the ones we saw as clearly having no chance. As @Gilles notes, there's a huge risk of just driving people away when doing this - waiting until they've burned out trying to make it work and left in disgust doesn't improve upon that. Not a huge lot of SE 1.0 site owners active in SE 2.0...

@MarkTrapp We're looking very hard at abandoning, or at least making some fairly substantial changes to the current lifecycle. Part of this will absolutely involve some better metrics for "progress" - the Area51 stats are the best we have right now, but there's no clear division between a site that's doing poorly and a site that's all but dead.

For one thing, there are a bunch of stats that aren't exposed, or are exposed in a really hard-to-read fashion - just try getting per-question participation - a hugely important aspect - out of these sites anywhere.

user2334

Indeed. One thing I see even on launched sites like Programmers and Gaming (but also on sites like BH, Christianity, and the Workplace) is that people focus on traffic and question closures and a lot of metrics that don't seem to be at all relevant to a site's success

Shog9

Quality evals are the other part of this. One thing you'll see (if you look very closely) on these sites is that down-voting is... rare. If you're only getting one answer per question, and no one ever points out wrong answers, you get a site that looks (on A51 at least) a lot better than it really is.

user2334

And I think a large part of that is due to people guessing at what counts rather than being told, "hey, this is what we're looking at. Focus on this instead of those red herrings". BH not closing is indicative or something other than traffic saving it (question quality perhaps?)

Shog9

@MarkTrapp Short answer for BH is that the single most troubling metric there was time-to-answer, which given the subject matter I don't actually consider a bad thing. FGITWing something like that (or Linguistics, which fell into the same bucket) is probably more harmful than good.

Gilles

@MarkTrapp where “guessing what counts” usually (I've seen it on a lot of beta) means looking at the Area 51 stats and concentrating on making everything green. @Shog9 The Area 51 stats may be doing more harm than good if they're making people focus on the wrong things

Shog9

Apr 25, 2012 22:57

Which illustrates one of the problems with just making all of these stats public - it's tempting to optimize for the wrong thing. The A51 stat is "questions per day", but even that is troublesome, since what's really important there is that the folks who're showing up on the site to answer have something useful to do.

Gilles

@Shog9 interesting, this is the first time I see this metric mentioned

Shog9

@Gilles I'm pretty sure that's exposed somewhere, but I can't recall where

user2334

@Shog9 Right, and you see people suggesting the way you get more questions per day is to allow things like lists or fun questions or whatever

user2334

Or my favorite "don't close as much"

Gilles

@Shog9 yup, that's something that worries me on French Language and Usage sometimes. We have a bunch of highly motivated users who keep fixing typos, seemingly just to have something to do, but I'm afraid we lose the “middle people”

Are there publicly accessible or mod-accessible tools to evaluate user retention?

Shog9

Apr 25, 2012 22:59

@MarkTrapp Exactly. The root problem with most of these is simply that we launched sites that don't have enough of an overlap with the existing community on SE, and haven't been able to attract many outsiders. Everything else is just symptoms of that.

The Assembly

Participants