12:18 AM
Oh, I see, it has an answer already that points to an effective duplicate here (one that did indeed survive, likely because the underlying issue was stats)
The comments in the Note in the help on the function - while at face value only talking about the rank sum test - apply also to the signed rank test (i.e. echoing my earlier comment)
Different people - for reasons of simplicity of one thing or another, or to emphasize similarity to other statistics (typically for pedagogical reasons) - will choose one form of the statistic or another.
The unfortunate issue is that because people don't raise the existence of - and trivial equivalence of - the various alternative forms, people who encounter a different definition than they're used to become confused. Obviously the Wikipedia page would be a good place to mention the existence of alternative forms and their equivalence. (In fact it sort of does, in passing, in the History section)
@amoeba That reputation issue you mentioned, which I mentioned that I had also seen (and which I subsequently raised on meta.SE) now has an answer.
16

There seems to be a bug on some sites right now: e.g. http://stats.stackexchange.com/users?tab=Reputation&filter=month "month" tab says that it's the score from 2016-08-01 but some users show their rep from start of July onward. It's not like that for everyone. For example, my own months rep...

12:41 AM
@Glen_b Thank you. I remember going over the signed test with you a few months ago, and it rings a bell that the equivalence of different calculations was brought up. I'm starting to revisit some of these stats topics, which may be the best second way of learning, after the more organic situation - i.e. that of the practicing statistician, who uses these tests often enough to not forget.

I read something about p-values, swans and shades of gray. But, how do you build the distribution for the test? As I understand, your theory should give the theoretical bell curve for the swan colors for you to see if there are any too black to say that all swans are white. How do you build the curve?

1 hour later…
1:53 AM
If somebody ever felt their activity on this site was the best indicator of their potential as an employee, they could say "My CV is my CV."

2:03 AM
@Kodiologist Hi...
May I ask you a silly statistics & computer science question?

2:26 AM
@Antoni I forget stuff all the time. However with enough experience you may remember how to find it, which is almost the same thing as remembering.

@Glen_b Hi, can I ask you something?

Don't ask individuals if you can ask. Just ask a question whether anyone is here or not. Then people who are able to answer and want to do so will.
Post your question and let people respond
@LittleAlien I'm not 100% sure what you're asking. What's the particular null and alternative? Do you have test statistic?

@Glen_b I'm just worried if I post my question it'll get closed since SE community is so strict
Do you know a way to make random number generator is still reliable for conducting scientific experiments like Monte Carlo?
@Glen_b I'm looking procedure like Paul Wilmott's style
I meant a way to make random number generator in EXCEL still reliable
Sorry, I forgot to mention that @Glen_b

2:57 AM
I'm not sure what exactly you mean by "still reliable" and what you mean when you say "make random number generator in EXCEL still reliable"
Do you mean to write some random number generator function to call from Excel? Or do you mean something else?
The RNG in Excel isn't the dog it once was.
Are you asking how to test random number generators for particular kinds of nonrandomness?
I'm really not sure what you need.
Some questions related to what you might be asking may be on topic for us. Some might belong on scicomp, others on stackoverflow or superuser
It's just not quite clear what you're trying to ask
If you're trying to get normal random numbers, DON'T add 12 uniforms and subtract 6. I see a lot of excel users suggest this, and then try to use it for valuing options where accuracy in the tail matters. It's a really bad idea.
If you're asking about how to get normally distributed random numbers, you can use the inverse cdf to transform the uniform
Inverse transform sampling (also known as inversion sampling, the inverse probability integral transform, the inverse transformation method, Smirnov transform, golden rule) is a basic method for pseudo-random number sampling, i.e. for generating sample numbers at random from any probability distribution given its cumulative distribution function. Inverse transformation sampling takes uniform samples of a number u {\displaystyle u} between 0 and 1, interpreted as a probability, and then return the largest number x ...
In Excel that's slow but it will do what you need. If you want it fast don't use Excel.
If you want more I'll need more of a clue what you're really asking
You shouldn't post anywhere until you have a very clear question written. Then you can figure out the best place to post it.

3:14 AM
@Glen_b Sorry for the confusion, what I meant is we all know that The Excel random number generator does not fulfill the basic requirements for a random number generator to be used for scientific purposes, right? Now, I still want to use for doing so, so what's kind of procedure or algorithm that I should implement so that it's still reliable for conducting scientific experiment like Monte Carlo?
@Glen_b At least for certain degrees it can still pass the tests of randomness that had been conducted in these two papers: On the accuracy of statistical procedures in Microsoft Excel 2007 and On the accuracy of statistical procedures in Microsoft Excel 2010
Does kind of question still fit in CrossValidated to be asked so that it'd not be marked as Unclear, Too Broad, etc?
@Glen_b No need to be complicated, just a simple one yet it's still good enough for scientific purposes
I meant the procedure. If possible like Paul Wilmott's style

4:16 AM
I'm not sure we all know that the RNG in Excel is unsuitable; what's the evidence on the current one? As I said, the random number generator in Excel is not the dog used to be. (The one in visual basic is still a problem though)
If you're in the position of wanting to write a new random number generator, don't use Excel in the first place.
The RNGs even in older versions of Excel will probably be substantially better than anything you can do directly in Excel by hand.
The second of those links above explicitly calls the numbers you get via RAND() in Excel "possibly good". The first assessment they performed was not suitable as it now uses the Mersenne Twister algorithm, as Microsoft has said (and you can find this information in the help these days). They then performed a battery of tests bases on the Small Crush tests -- these are pretty strenuous tests. The only test it failed was the Periods in Strings test.

2 hours later…
6:40 AM
OK, thank you for your explanation @Glen_b. I really appreciate it... :)

1 hour later…
7:54 AM
@anast Unless you have some very particular applications that look at the kind of structure that particular test detects, this would likely be fine for almost anything.
I wouldn't want to try to write anything that would outperform Mersenne Twister, it's a highly regarded RNG. The tests they've done in that paper would seem to confirm that it hasn't been incorrectly implemented.

1 hour later…
9:20 AM
@Kodiologist: Do not overestimate things; it is only an indicator and a decent one only for very specialised tasks. (eg. if you work as a technical support engineer; I have worked as one and I can attest it would be a reasonable indicator.) It says little about how you work under pressure, how you work on stuff that you don't like and/or bores you, how you work within a team and how you work on a long term project that requires more than (at most) an hour or two of your attention.
Don't get me wrong, it definitely shows skills. Especially in communicating technical concepts as well as in identifying technical issues but these skills are a small part of a CV. In my job interviews I used my CV presence as a good extra-kicker with the interviewer; it played out nicely I think when the interviewer knew of SE's CV. (Some especially more senior statisticians/analysts has no idea about it)

2 hours later…
11:48 AM
I was pretty sure that \boxed{} environment does not work in our MathJax, but turns out it does now. Can be convenient sometimes.

2 hours later…
2:14 PM
@StasK Thanks for the Stata code
Should we maybe only reproduce 1 datapoint prediction, not 10?
Having 10 of them 4 times (r/python/matlab/stata) or more, is a bit excessive

2:43 PM
I am fine with one
Reproducing the results from first principles in Stata is freaking complicated, it is not a matrix language. But OTOH the syntax to get the job done is so straightforward!

@Glen_b The RNGs in older versions of Excel were the worst that ever saw the light of day! See the last third of the post at quantdec.com/arcview.htm. It takes only a half dozen lines of VB to improve greatly on them with a "modern" (i.e., 30-year-old) algorithm.

What do you mean by "from first principles", @StasK?
Performing PCA without calling built-in PCA function?

3:37 PM
How do I search for questions that I've answered that carry a specific tag?
I tried various combinations of "user:74500:[xgboost]" but the answers I get back have pretty much nothing to do with xgboost.
I tried removing some of the colons, adding spaces, etc.
Seems like I'm doing it wrong.

4:07 PM

@whuber Thanks.

Yah, that's perfect.

So space is the delimiter, and : means "this is an argument to this thing"

4:10 PM
@MatthewDrury Start a search--any will do, such as "user:me"--and click on the "Advanced Search Tips" link at the right. It will show the (idiosyncratic) syntax options.

I actually did do that, which is how I came up with the [xgboost] thing.
I found the instructions a bit confusing because they gave me a lot of options, but I didn't see any mention of how to combine them.
But, given your example, I think I understand now.

@MatthewDrury Combining them is meant to be as simple and mindless as any standard Web-search interface: just separate search elements by spaces.
I think quoting phrases will cause them to be searched as a unit.

Yah, for some reason "user:74500 [xgboost]" also gave me poor results, so I thought something else must be going on.

I'm not sure what punctuation does. I think it's largely ignored. I believe the search also does not necessarily focus on whole words: e.g., you can include the stem of a word to search for all variations.

Well, I guess that gives zero results.

4:12 PM
To some extent you can even search within TeX markup.

Maybe there is just nothing acutally tagged with xgboost.
Oh! You type the TeX markup?

@MatthewDrury xgboost gets seven hits.

Yah, playing around, I just overthought it.
Not unprecedented for me.

Try this: \int

Hahah, that's a lot of integral signs.
Looks like it finds them in the title.
as well.

4:15 PM
Although the search for TeX markup is imperfect, sometimes it has really helped me narrow a search when I know a particular formula appeared (or likely appeared) in a post.
Ciao!

Yah, much thanks.

5 hours later…
9:26 PM
@StasK I added outputs for all code excerpts. Curiously, Python gives me results that are different from all other languages in the third digits. No idea why!!

9:52 PM
Regarding my attempt to consolidate "how to reconstruct the original data from pca" type of questions, @ThomasBrowne wrote a critical comment: stats.stackexchange.com/questions/229092/#comment434417_229092 saying essentially that each question has its subtelties and it's better to answer them individually and not close as duplicates of some "canonical" thread.
We tend to close a lot of questions as duplicates of "canonical" threads and I think it's a good practice, but here is one disagreeing voice. Any comments on that?

10:03 PM
@Silverfish, what I wrote above is related to our today's conversation elsewhere.

@amoe

hi stas
nothing like a friday night in stats chat

yep
when the body of work for the week is not completed yet :-\
@amoeba @ThomasBrowne those are viable points of view, but the reality of the CV, as I see it, is that (1) the same questions are being asked, (2) plenty of questions go unanswered, and (3) I find myself repeating the same things over and over again.
Referring somebody to a generic answer will (1) at least, serve as a starting point for them to learn more about the topic, and (2) help them clarify and refine the question, as in, "I read this canonical answer, but here's what I am missing / how my problem is different"
May be this should be discussed on Meta rather than in the chat
But I am out of energy fighting that fight... after sucked my brains

There is currently 123 threads in that tag - are you planning to sort them all out of it?

yeah, but luckily the new ones do not appear too quickly
and hopefully additional tags with their wikis would help avoiding questions that are too generic
or too dumb
but I may be expecting too much of the hit-and-run users
May be the site should work by only letting post questions after a user has visited the site at least 10 times, and read at least 20 questions with 1+ answers scrolling all the way down.

10:13 PM
Plus passing a short maths and english syntax/grammar test

well you'd have to give some slack to the IP addresses outside of English-speaking countries ;)

@amoeba I think we need a Meta thread really
I have some sympathy with @ThomasBrowne - I don't disagree at all with the benefits of having a "main thread" for core questions which pop up again and again.
I suspect that in lots of cases, the main benefit of a "canonical thread" will not be that other threads form perfect duplicates of it, but that it serves as a good link to refer an OP to for "this is how the core mechanics of this generic issue work", and so answers to their question can then focus more on "and this is how it applies to your particular case" (the subtleties that ThomasBrowne is referring to)
(In case anyone is interested, this is the discussion in question)

@Silverfish I don't disagree, but I would still usually vote to close as a duplicate and if an OP edits to clarify what is still not clear to them, then re-opening. Or at least post a link in the comments and if there is no response, voting to close.

10:38 PM
@amoeba I can sympathise. I think a lot hangs on the nature of subtle differences. Often there are things which are "obviously equivalent" - particularly to someone well-versed in underlying theory - while the differences may be less clear to those at the level of understanding of the OP
At any rate, Meta is probably the right place to discuss the broader issues, since there are many users who might like to have a say but who do not use the chat

To be honest, if a question is good, I don't mind it staying open and getting its own answer even if it's pretty close to what we have already. What does annoy me sometimes, are poor questions similar to the existing ones. (This is not about Thomas's question which is very good)

One issue with CV is that there are many questions which are not especially good but which are not generally considered poor enough to be closed, and in some cases some of these questions have become "accidentally canonical", in the sense that many other threads have been closed as duplicates of them.
That's something where the idea of posing "deliberately canonical" questions can be really helpful.
I need to head off, have a good weekend all!

11:06 PM
@amoeba: The minor difference you see is because sklearn's iris dataset is not the same as R's or MATLAB's or Wikipedia's for that matter. For example if you check the 38th row on MATLAB you will get: 4.9 3.6 1.4 0.1 which is the same as R's. The same row in Python is: 4.9 3.1 1.5 0.1. Do not ask me why; I guess it's one of the cases sklearn is special in it's own way.
4

Wow
Thanks
Can you check if the number of samples is the same?

150, yeap.
It's relative easy to spot; even the colSums are not the same.

I am also wondering if there is only one inconsistency or more... Perhaps saving matlab's and python's version into a file and comparing with diff could
I mean, did they screw up one data point, or what is going on
We should file a bug report or something

I was a bit disappointed to be honest.. I hoped it was due to MATLAB using MKL and Python a reference BLAS (or OpenBLAS if you bothered a bit). But unfortunately it was something that mundane.
No, they are a few points here and there I think, not just this one.
I am sure they know already. There is not chance they didn't check their implementation against R's or MATLAB and they didn't notice. And I can tell you know that there is very little chance of fixing it either. The amount of testing code such a change would invalidate would be eye-watering...

I googled and could not find any mention of it
I think raising an issue on github.com/scikit-learn/scikit-learn/issues is a sensible thing to do. They can decide what to do with it.

11:15 PM
@amoeba Yep, seems worth reporting to me.

Test your own code: np.sum(X, axis=0) returns: Out[37]: array([ 876.5, 458.1, 563.8, 179.8]) and apply(X, 2, sum) returns 876.5 458.6 563.7 179.9. :D

@usεr11852 Wanna raise an issue on github? I guess you are in a best position
If you have a github login

I am checking the bug reports. I find it so unlikely not be reported beforehand.
I told you this must be a known issue.
They clearly know it, they just didn't bother about it. Somewhere down the pipeline this wrong dataset became common place in the Python community. If anything the github issue would be a feature request to change the reference dataset but as mentioned it would create such a shit-storm of failed unit-tests that it is very improbable they will even consider it...

@usεr11852 Hmm, so it only concerns two data vectors?
"This data differs from the data presented in Fishers article (identified by Steve Chadwick, spchadwick '@' espeedaz.net ). The 35th sample should be: 4.9,3.1,1.5,0.2,"Iris-setosa" where the error is in the fourth feature. The 38th sample: 4.9,3.6,1.4,0.1,"Iris-setosa" where the errors are in the second and third features."
I am not sure I understand this passage. The 28th sample as quoted here is from Matlab
Is it how it "should be" or how it is in Fisher's article, or where?

11:30 PM
@amoeba: Yes, it is like that in Fisher's article (like in the Wikipedia/R/MATLAB).
The passage is essentially an off-hand remark saying: "I know I am wrong and they would be the right values." :D

So sklearn people took their dataset from UCI website, and it happens to be wrong there for whatever reason
All other languages have it right, but sklearn developers decided to take the data from the only place where it is wrong?
Nice.

Yeap...

@Kodiologist Thanks for the edits! Highly appreciated. Apologies for not writing the alt text, by the way... I have seen the Meta discussion but did not think of it now

If it makes you feel better this is really minor. While MATLAB, numpy (and obviously C++) decided to initiate their random number Mersenne-Twister generator using unsigned integers (uint32) as the reference C implementation does, R decided to used signed integers, this means that the states of R random seed state are... incomprehensible to pretty much any other language. (Actually R uses 626 integers instead of the commonly used 625 or 624 but less don't dwell on that...)

Interesting. Does it mean that setting a seed to the same number in matlab and numpy would actually allow to generate the same random numbers in both?

11:44 PM
eg. rng(123); rand(1,3) ans = 0.6965 0.2861 0.2269 and np.random.seed(123); np.random.rand(3) Out[38]: array([ 0.69646919, 0.28613933, 0.22685145])`. Makes my simulation checks that much easier. :)