I don't think you can logically refute it, per se, but only use a cost-benefit analysis to justify implementing action based on your model. That is, compare the implementation_cost against the benefit*probability_your_model's_right. Or, more likely, some less naive cost-benefit calculation. (Of course, she also seems to have some career-based politics underlying her "dismissal", but dealing with that is a whole other story.)

John, you're precisely right, it was industrial politics- and it was not until years after her departure that the organization got "realigned" in a way which permitted problem-solving again. I would like to have had some refutation tools in hand at the time- including a cost model, as you point out- but I did not. it was a sorry mess. She left the company and went on to become a startup entrepreneur, well-respected and successful, and her funeral was standing-room only. I wasn't there.

Philosophy is not a tool to justify your preconceived notions of right and wrong.

@jishin noben, perhaps not, but it still gets used that way by clever people nonetheless, as my experience amply demonstrated.

There is no "logical way" to confute underdetemination; obviously our scientific theories and engineering practices may be wrong but they are still the best tools we have to understand the world around us and "act" on it. They are also the best way we have to understand why our bridges fall down (sometimes). Thus, undedetermination is not skepticism.

But, at the same time, scientific knowledge is not absolute certainty. It is a human activity subject to human needs and aims, and also to human errors and interests. We cannot avoid them in scientific topics, and we cannot avoid them in "business" topics eitehr.

We can't determine that all swans are white without examining every swan. Worse, we can't know that all future swans will be white. However, we can be sure that swans are almost always white. Science is usually happy to operate on 95% certainty in the absence of certainty. You could perhaps have given probabilities?

@Richard Without knowing all possible alternative causes, you can't describe a probability distribution over all possible causes.

@DanHicks yes. It wasn't clear from the question whether all possible causes were known.

@DanHicks And that is not true. No study ever done has known all the possible causes for what they studied. The value of statistics is that you can make reasonable inferences about the data you don't have by using the Law of Large numbers and the rules of probability. You don't get proof, but you do get odds.

@jobermark The probability distributions in any statistical model are always conditional on background modeling assumptions, eg, that unmodeled causal factors produce Gaussian residuals. You can construct higher-order distributions conditional on different sets of alternative assumptions if those alternatives are all specified. But you can't define a probability distribution conditional on unspecified alternatives. How would you determine what the distribution looks like?

@DanHicks I guess I should not have said it is not true. It is not relevant in practice. A study done using "normal theory" is looking the assumptions of its explicit model, plus the idea that there is enough data present to make the overall statistical framework apply. If the added assumption is false, you are going to get 'false' and you haven't got a result. If you know distributions, so that you can use any other kind of statistics, you will get much more powerful analysis. But that is not often the case.

In particular "that unmodeled causal factors produce Gaussian residuals" is not an assumption, it is a theorem in normal theory. So it is just equivalent to the assumption "I have enough data to use normal statistics." If you don't, you are not going to get a correct result. But the result you do get will be of the kind that has no implications, not the kind that misleads you.

The CLT only applies when you have a limit of a sum of iid variables. But it's often at best unclear whether what you have is a sum of iid variables: drive.google.com/open?id=0B6oYmzobonqoU05yU0NUaURVZjQ There's no getting away from background assumptions.

So sorry to hear your story, man. But don't forget that "do nothing" is a coherent strategy, too. I would not feel bad about the six months... they assigned the root cause to one of their senior workers, who developed a coherent proposal, which they were free to accept or reject. After your study was done, they were confident about both the proposal and the problem, and they made up their mind. If your work had been less sound, they would have kept studying the problem until they got a better answer. I suggest that you keep talking about what you learned for the benefit of the rest of us!

thank you for your kind comments. As I sort out my feelings, I may come back with more questions about this. There's a lot more to it than I have explained here, and underdetermination is only one facet of it.

@DanHicks Yeah, if all your data is internally redundant, it is not enough data. Again, normal theory is going to fail the added premise. That does not say anything about the other premises, but the background assumption is still that you have enough data, and not something more formal, or more questionable.

@danhicks, this event was the tip of the iceberg for the organization; a symptom of an underlying condition too big for one engineer to address from the bottom of the food chain. fixing it required the retirement of an entire cohort of managers who were all known for being very good at protecting their part of the organization, which meant pushing back when it began to look like a quality collapse could be traced to one of their processes. The underdetermination argument was only one of a long list of responsibility-evading techniques they deployed.

@elliotsvensson, I now know that the successful refutation of the "do nothing" response requires you to ask for the cost model, and compare the net present value of doing nothing in particular to the net present value of doing something specific (a tooling or metrology upgrade, for example) to prevent the recurrence of the problem.