Yes, I got it. The only thing that is still bothering me is the following
... you don't know who is going, not that you didn't know who was going ...
The first half makes sense; for the second, do you mean that ..."you didn't know who was going" is the truth about " ... if I knew who else were going"?
The idea is that "I didn't know who was going to it", despite the use of preterite (past tense) verb-forms, refers to the situation at the time of utterance, i.e. the present, cf:
(1) I would have gone to the party but I didn't know who was going to it. (past tense verb-form "was" and past time situation)
(2) I'd go to the party if I knew who else was going to it. (past tense verb-forms but present situation)
In (1) "was" is a straightforward past tense auxiliary verb indicating past time. You may now know who actually attended the party, but you didn't at the time. By contrast, the use of the modal preterites "knew" and "was" in (2) reflect modality, or counterfactuality, at the time of the utterance, not at some time in the past.