According to our Privileges section, you should only use comments to request clarification from the author or leave constructive criticism that guides the author in improving this post. Please review the When shouldn't I comment? section: I've cleaned up the pseudoanswers.

@ChrisW I believe it's true, and as me, many others. Am I not reasonable?

@Paul It's a notable claim and I ask for evidence. It's not unanswerable or philosophical: it's a real quantity that can be measured. There is no need to be skeptical about anything. The site is about scientific skepticism which is very different from being skeptical.

@Articuno the claim is not about a bayesian probability. It's reasonable to believe that many people (including me) think that the amount of money saved by Stack Overflow is in the billions. I want to see if the evidence supports it.

As a side note: there are many scientific studies regarding specificaly stack overflow, so it's not impossible that this value has been studied.

@Articuno the claim is that a billions estimate is a realistic estimate. "Probably" supports that. I am not quite sure what the point of contention is.

Oh please

your comments are completely out of place and are not helpful.

"Lemon juice probably cures cancer"

^ is it a claim?

"My left shoe is probably worth billions"

Which is what I did

I didn't mention a standard of evidence. And I am not interested if Carmack is justified in saying probably

I want to know if the statement he made is sustained by evidence

I don't think we are getting anywhere.. :-)

Then if you have a concrete suggestion I'm happy to hear it

if you don't and you think the question can't be saved, vote to close and move on :-)

The reason why I added the rest of the blurb is because I don't want someone making a calculation off a hankerchief like I did. To get it out of the way

Now I am quite sure I'll get poor answers by would be economists (which is why I started nuking pseudo answers in comments).

Laugh, Sklivvz. I have an actual PhD in Econ from Caltech, and have had a teaching career.

I am @Paul, the guy whose comments he nuked,

Yup.

That's fine. Your printing press, not mine.

@user33903 I explicitly linked to the policy where pseudo answers in comments are forbidden.

Was the policy not clear to you?

The good thing about back-of-the-envelope is that they can open discussion or serve as a draft for an elevator pitch. The bad thing is that the assumptions must be extreme in their simplifications. And, if allowed to proceed unchecked, can turn into something controversial like "contingent valuation" where saving a dolphin ends up being worth millions of dollars (would you pay $0.10 to save a dolphin?).

@Sklivvz Is this an inquisition, or a real question?

it's a honest question

It is clear.

I think that Carmack is also very influential, not only notable

like CodingHorror, which I used as a notability example before: also around 100k followers

also, what he said is not very controversial (among programmers). I disagree completely with the statement that "reasonable people don't believe it" as @ChrisW said

It is only correct by ignoring substitution effects and the law of diminishing returns.

@user33903 what is it that you are referring by "It"?

the claim or my calculation?

Both I'm afraid. SO is great, but if there were no stack overflow people would go to usenet, yahoo groups, quora, their coworker, etc.

yes and no

The classical way to prove the value of SO's service is over a billion dollars is to get someone to pay you a billion dollars for it.

Ah, but we don't sell our service

And that's where things begin to get controversial. Because when someone pays for something willingly, we know that their value for the thing was at least what they paid for it. Without that kind of data, claims about the value of a thing to a person are knowable only by less reliable means like surveys and guessing.

How do you know time was saved, and how much?

Can the programmer do something else when he is stuck?

Most workplaces involve multitasking....

If the next best alternative to going to SO is for a particular employee who makes $X/hour to spend time T hours, then the marginal productivity of SO could be said to be T*X. Notice how many "weasel words" are there, though.

But if the next best alternative is to read the next entry down in Google, for 3 minutes, then....

Well, by "particular employee" I mean least cost. For instance, if a person who makes 1/2 the salary can do it in 1.5x the time, the task could be assigned to them at a savings of 25%. An accurate valuation is typically "bottom up" from studying lots of cases, and it is case by case, so one either has to make extreme simplifying assumptions, or do a lot of case research, and it starts becoming a subject matter issue rather than economics.

let me read what you wrote there...

Yes, that is a marginal value for the case where there is an absence of alternatives. However, if 50% of the cleanup could be saved by adding a garbage can that costs $10/store, then you have an interesting alternative there that you have to work out.

For example, if the employees are paid $10/hr, and 5 spend 0.5 hours cleaning the store with 1 garbage can but only 2 are needed when there are 2 cans, and the wizard does it for free, then the marginal value added by the wizard is only $20/store not $25/store.

Note that in practice, with inexpensive wizardry, people may have never thought of just adding a garbage can.

Yes, that's an issue with the methodology. Similarly, if hiring a witch as a barista and making magical cleanup part of her duties would cost $5/store, the value is even less.

Remember Economics, as "dismal science", often explains why we are not all rich. Everyone likes to think they have a gold mine.

Yes, that can happen.

@Sklivvz "I disagree completely with the statement that "reasonable people don't believe it" -- The topic may be true (or not). But IMO the claim is so flippant that people are not led to or even meant to believe it just of that claim -- it's a joke or stab in the dark, a quip, a wild-ass guess.

... just because of ...

Well, when a question like that starts with 'people', the answer often seems self-evident. Yes, values change, and seem to be subjective rather than objective. Trade occurs because of subjective values that are different.

Value of tangible and intangible goods change all the time.

The stock market and eBay are good examples of this change in action.

The wizard, in his self interest, wouldn't reveal he was only worth $20 to the store with one garbage can.

@Articuno If he were accused under some Truth in Advertising law of making a reasoned claim with intent to be believed, and if I were on the jury, I'd acquit him. I'm not arguing the question should be closed as off-topic; I've downvoting it because IMHO it's not a worthwhile question.

Even though he is doing it for free, he appreciates the extra $5 worth of praise.,

@user33903 I just don't follow your reasoning: the SO community and body of knowledge provides great value to programmers: before SO it would take days before one would get an answer on forums (thus killing productivity) and also the value of an answer in a forum is much lower (because forums are terrible vessels for information). Surely other companies could have provided the same service, but it's irrelevant.

In other words the ways in which SO provides value are two: answers in minutes and not days, huge replay-ability (each question gets visitied on average 100x)

In practice it's a medium by which a person can help 100 others with the effort it takes to help one

Compared to previous systems, clearly

forums, usenet

that too ;-)

And for @ChrisW, I honestly think it's a reasonable claim. I think that the facts with demonstrate the huge value of SO. But -- of course facts win over hunches. :-)

That's why I made the two points above. There are no alternatives which are equal to it

A lot of the simpler questions have a few pages of results on google. What is the value of SO having the top spot? A great bulk of the questions are like that. Thats one end that is worth understanding. The other end is the questions for which SO has the only answer, and it is an important answer. That might be worth understanding too. My gut feeling, as an economist, is that the latter might matter more than the former.

As a hunch, I've only ever met a single developer who didn't know stack overflow (my guess: he was lying). Every single developer I know uses stack overflow as the preferred resource (even if not at the top spot in google) because of the quality of the information contained

you can measure how long it takes to get an answer on SO, in fact we do measure it routinely

you could compare it to other different media: forums, etc.

that would be what an economist does, right?

Thats not obvious. Often papers are written about phenomena that need to be seen before you know what questions you want to ask about them....

surely the value of internet sites is something that it's amply studied

scholar.google.com/…

@Sklivvz IMO you underestimate the time (finding the answer on the internet saves more than one hour of trying to figure it out unaided), and underestimate the cost of programmer time ($10/hour); but over-estimate the answers found on StackOverflow (you should divide total page views by the number of page views per visit, because it takes more than one view to find the right answer; and subtract the views of people who go there to answer questions, and of people who don't find their answer).

since we are not asking anybody to do this research, I think "as compared to what" is not very important

which one?

@ChrisW oh, yeah, my calculation is all over the place. I expect it to be right within a factor of ten at best

This preoccupation with measuring something is not really what I did as an experimental economist, which related more to explaining the workings of specialized markets, auctions, etc.

Hayek, I think, also had cautions about measurements and averages in "The Use Of Knowledge in Society"

@Articuno as compared to previous alternatives, i think those are the biggest factors or where we provide unique value

I think that's what the statement means. Cf. "Apple has provided millions with jobs by introducing the iPhone"

as compared to what is not fundamental in the sense that it's a parameter one can set in the answer

Is that a scientific statement, or rhetoric, advocacy, and marketing?

@user33903 again, it's irrelevant to a layman. it's relevant to an expert. it's up to the expert to qualify the answer, imo

I'd expect thousands to be employed working on iphone, and programs for iphone. How do you get to millions? A chauffer has an iphone, so we count him too?

@user33903 it's a statement i just made up

Ah, ok.

but one could say "as compared to what? those people would have had another job"

or they could have used another phone

right. however the fact is that the iPhone is what actually happen. SO is what actually happened.

there was some value created by the innovation. I can't really estimate it! but by my calculation it's not outright impossible that it's billions

e.g. if we had 1000 answers, each seen by 10 people, the claim would be clearly false

but we have 6 millions, each seen by 100+ people, so the claim could have legs

It is plausible enough to be an elevator pitch for additional VC funding.

Heh, we are not pitching at the moment. We are fine, thank you :-)

I don't know. I suppose VCs do calculate the values of these markets, I suppose there might be some reputable reference -- but maybe no one has thought of it.

The thing is, without SO, if I wanted good answers quickly, with a high chance of being correct, I'd either go to a reference site on an open source language (which costs my lookup time, not an hour), or I'd go to Oreilley Safari, which is a book search for Oreilley books, at like a fixed cost of $100+/year, but no per-search cost except my few minutes of time.

Google is often pretty good at scanning important reference sites for programmers, e.g. google "mdn CSS background color" would tend to grab from mozilla developer network, a reference site.

I get points by my name now. Wowwee. How do I make it say Paul instead of user33903?

Go here: chat.stackexchange.com/users/10578/user33903

change your parent account to e.g. skeptics

user33... is the name of your account on serverfault

says parent user Paul.... lets see what pops up....

Ah thats better

Consider the allegations that napster helped robbed the music industry of billions. A known flaw in some of the back-of-the-envelope style estimates involved assuming that all copies would have been purchased, or that each copy caused some amount of damage. This attempt to comment on the value of programmer time saved involves similar risks.

agreed, but that's a negative value.

one could certainly calculate the value of the internet music business

The modern internet music business is often $1 a song, or even less, like free or ad-supported.

So the industry changed in a way that deprived some people an easy living off the work of others.

a good artist can sell maybe millions of copies and there hundreds of artists. so the value is in the hundreds of millions

Intrinsic theories of value, like labor theory of value, are not a serious part of a modern mathematical economics curriculum. Generally, concepts like willingness to pay (buy) and willingness to accept (sell) are part of an underlying philosophy of rational choice in econ. Finance is a bit more interested in modelling values as though they were something in themselves, with random walks and such.

Anyway, the earlier point was that there are probably some usages of SO that occur because it is free and otherwise have no value in saving time, just like there were uses of napster because it was free and otherwise the user would not have bought a music CD. These cause issues in any simplified valuation. A person browsing SO for self-training may obtain value less than or greater than their wage rate. But we'll never know since they don't pay for it.

Yes.

agreed

Even then I'd worry.

This isn't measuring e or pi or the distance to the stars. The things we can clearly measure the value of, fluctuate over time. So valuing intangibles is still voodoo to me.

It doesn't matter if you can get two referees and an editor to publish it.

i think there are objective ways to measure it, and a bunch of quite objective experiments that can be set up

for example take a group of programmers and split it in half, one half being denied the use of stack overflow. give them the same problems and measure KPIs

if you measure the gain (or loss!) this way, you can probably project it over a larger set

SO could sponsor such work. I don't want to do it, but it could be done like this: provide some money, and/or access to the data, to a researcher, or sponsor a prize for the best paper (even as little as $500 or $1000) in a special edition of a journal. It takes some academia networking to pull this off.

we do help academia and give access to data already

and there are many studies, mostly around machine learning though

yes

scholar.google.com/…

yep. we actually have our own predictor

yes, we have a 76% success rate vs 67% of the article :-)

it's on a few sites, e.g. stack overflow

Yes you should be able to make it better

So whats the bounce rate from SO?

no idea

And can you distinguish between a bad bounce and a good bounce?

Google Analytics reports a bounce as a single page, short lived visit.

not reliably

right but even then... is it a bad search result or bad content?

we look at other factors, like voting, feedback etc.

Membership is a good sign, too....

I had a similar problem. I have a single page app in www.armdisarm.com that makes a calculation. It was getting a very high bounce rate (like 80+%) that dropped to around 40% when I added an analytics tracker to track people pushing the "begin calculation" button. Without tracking people pushing the button, I had no idea if there was relevance or not.

some more data here: quantcast.com/p-c1rF4kxgLUzNc

cool; google analytics is only beginning to do user demographics.

Of course, if you are a skeptic it is difficult to know whether to believe any of that stuff.

The hit counts, sure. Knowing a visitor makes $100k/year. not so much.

Well, I need to be on my way. Good luck with this.... SO is a valuable service and I think you can find plenty of facts to advocate and support that without inventing anything new.

Oh sure, good day to you (I would accept also answers to the contrary though!)