It is no coincidence that 'whom' ends in the letter 'm' like 'him' and 'them'. You see, in terms of grammar, 'who' is like 'he' and 'they', and 'whom' is like 'him' and 'them'.
You wouldn't say, "Of he?", but you you would say, "Of him?". Therefore, 'whom' is correct.