"Aren't you coming too?" "Yes...I mean, no? I mean, yes, I'm coming." When answering a normal, positive question, you're unlikely to fumble around like this. If the question is "are you coming too?" ...

Yes, it is correct that we cannot use a contracted negative form when we use inverted word order to express unreal or impossible condition in the past. Instead, we are obliged to use the full form of ...