This is the problem with the trolley problem.

If it were replaced with, say, being told to shoot one group or another by a sadistic guard, the possibility of refusing to choose would be more obvious in terms of what it means morally.