No problem! So, what's up with my earlier suggestions?
As I understand it, you want to ensure that your scale adequatly reflects the construct it purports to measure, taking into account multiple administrations of that very same scale using stimuli of different polarity. Is that right?
I understand your suggestions but since it is a procedure I have never done before I would need some more specific information on how to do it on spss.
I am no expert on SPSS but I do know how it used to work in the past ;-)
First idea: compute the correlation matrix on all items for the first run, then for the second run. Do they look close one to each other? This will indicate whether individuals keep answering in a coherent way to all items.
An alternative approach would be to carry out a principal component analysis, assuming individuals scores are stored as numerical values. Look at the scree plot (the distribution of eigenvalues) and the correlation circle (which summarizes the correlation between all variables/items). How do they compare?
Last, compute the total score for each participants, and plot individual scores for the first run against scores for the 2nd run. How does it look?
By the way, I took the liberty to clean up the comment threads by putting questions as an edits to your original post, and my answers as an edit to my reply. If you're happy with that, I suggest you delete all but the first and eventually last comment of yours.
I am here. So for the first idea I should simply present the two correlation matrices in order to show that the outputs are similar, but without presenting any statistical test to indicate this. Then why I would not simply print the factor loadings of the two EFA?
Your second idea looks like my idea about simply presenting the two EFA factor loadings, but then why would I use PCA instead than EFA?
As for your third suggestion, do you mean that I could simply 1) do the two EFA 2) calculate the global score of each individual for the found dimention at t1 and t2 3) calculate a correlation between global score at t1 and global score at t2 ?
Considering that at t1 and t2 the stimuli where of different polarity, the correlation will probably be negative
The proper way to do that would be to use a confirmatory factor analysis with two groups (here, defined by the time of administration) and check whether the loadings are identical. That's why I said that EFA alone is not the best option.
I was wondering if there was any way to compare the EFA factor loadings without going trhough the MCFA. The validation of the scale is not the main aim of the paper and since all of the other analyses in the paper (mixed MANOVA) are done on SPSS I thought it would have been more consistent to stick to this tool. If MCFA is the only way to compare factor loadings, I think that in this first version of the paper I'll stick with the double factor loadings, I'll keep the MCFA in mind in case.
Sure, at least you know what to respond to a reviewer would be picky about that. IN this case, using EFA alone I would simply summarize the difference between each loadings. If they don't vary too much (say, 0.1 on average), that's probably ok. Add to this the correlation between individual scores on run 1 and 2, with a confidence interval.
Is this a unidimensional scale (i.e., you only have one factor)?
All right! The fact is that EFA does not make any assumption regarding the distribution of the responses. So you can't use confidence interval, model fit indices and the like. But if the factior structure is close enough there's not much to worry about.
@Silvia The global score. In addition you can use the item/total (or rest) score like we do in item analysis.
