reply on: When Data Is Missing, Scientists Guess. Then Guess Again \ stacker news ~science

97 sats \ 10 replies \ @south_korea_ln 2 Oct \ on: When Data Is Missing, Scientists Guess. Then Guess Again science

Rubin’s approach, called multiple imputation, takes that distribution into account. To use it, first make several copies of your data set. For a given missing value in one copy, randomly assign a guess from your distribution. By design, you’re more likely to pick one of the better guesses, but you’ll also have a small chance of picking one of the less plausible guesses. This process reflects the uncertainty in each guess. Repeat these steps for the missing value in each of the other copies of the data set.

Once you’ve filled in all the missing data, you can analyze each completed data set. You’ll end up with several different predictions for the effectiveness of your drug. Then you can use a recipe known as Rubin’s rules to pool your results and get an average prediction. By following these steps, you can also compute a better estimate of your final prediction’s uncertainty. For drug regulators like the U.S. Food and Drug Administration, being accurate about that uncertainty is crucial: It influences whether or not a drug will get approved.

It's confronting to read about a technique I never heard of. Granted, i don't need to use a lot of statistics in my work, but it's a reminder that in many fields where statistics is important (I'm looking at you, human science), it is likely that the researchers doing the work, are probably not aware of the state of the art knowledge on how to properly treat with uncertainty in their data.

37 sats \ 3 replies \ @SimpleStacker 2 Oct

I do think some assumptions need to be made about the missing data for the procedure to be valid though. Like, if the data is missing at random then I think the procedure would work great. If missing-ness is non-random, but only depends on observed variables, then the procedure could also work.

But if missing is non-random and also depends on unobserved correlates, especially if it depends on unobserved correlates of the outcome variable, then I think the procedure is likely to yield biased results.

10 sats \ 2 replies \ @Undisciplined 2 Oct

In the first scenarios, where it seems ok, are you introducing measurement error?

If so, you're going to have attenuation bias.

29 sats \ 1 reply \ @SimpleStacker 2 Oct

Hmm, good point. I’ll admit I haven’t thought carefully about imputation. Why wouldn’t any procedure that imputes data without the outcome variable lead to attenuation bias, and why wouldn’t any procedure that uses the outcome lead to endogeneity?

I’m assuming there’s a good answer if I read the literature. But it’s possible I’d be disappointed as well

13 sats \ 0 replies \ @Undisciplined 2 Oct

I hadn't thought about imputation leading to attenuation bias until just now, but it seems like it would (if I understand why measurement error has that effect).

I'm also sure this has been discussed at length in the literature. It surprises me a little that none of my advisors or econometrics professors mentioned it, though.

76 sats \ 5 replies \ @Undisciplined 2 Oct

We tend to use predicted values for missing variables. One of my advisors would recommend doing it a bunch of different ways and hoping they all tell the same story in the end.

34 sats \ 0 replies \ @kepford 2 Oct

I thought we could "trust the science"... :)

22 sats \ 3 replies \ @SimpleStacker 2 Oct

Ah yes, the magical "robustness check".

Possibly the most commonly asked for exercise by referees, but the least scientifically grounded one (at least, from the point of view of statistical rigor.)

For those not in the know, it basically means try your results under a variety of different assumptions and if the main result still holds, then the result is "robust".

I have to say, there's a certain logic to it---but it's weird that econometricians obsess with how to formally calculate the asymptotic variance of an estimator, and then on the other hand ask for these totally hand-wavy exercises like robustness checks.

15 sats \ 2 replies \ @Undisciplined 2 Oct

In my advisor's defense, she doesn't care that much about econometric nit-picking. Her preference is very much to find highly defensible natural experiments and then do a simple regression analysis.

I actually hadn't even connected this practice to robustness checks in my mind, because it comes up towards the beginning of the process. It always just seemed like her attitude was "Why don't you see if it's a problem before you spend a bunch of time worrying about it?"

33 sats \ 1 reply \ @SimpleStacker 2 Oct

Oh yeah I’m not really criticizing robustness checks.

If anything, the people who trust too much in the formal statistics deserve more criticism (imo). And I’m mainly referring to social science here as well.

19 sats \ 0 replies \ @Undisciplined 2 Oct

I remember my office-mate going down a crazy rabbit hole trying to figure out exactly what the right standard error calculations should be for his job market paper. He probably spent a month obsessing over it and never did come to a definitive conclusion.

Probably no surprise, but each of the half dozen different approaches told pretty much the same story.