Please correct me if I'm wrong in the following explanation:
"If I were to be he, I would have done it rather smoothly."
I was tempted to write the above sentence as "If I were to be him..."; for a moment, I thought that the noun after the verb 'to be' should be in the subjective or nominative case since it's a subject complement.
For that reason, I decided to go with the subjective form of the pronoun instead of the objective case of it (i.e. 'he' instead of 'him').
First, you don't need to say "If I were to be he/him". Just say "If I were he/him".
The answer is that both are correct, though nominative "he" is very formal, while accusative "him" is considerably more common and natural.
Incidentally, your example is not subjunctive. This use of "were" is called the Irrealis mood. There is no past subjunctive in English, despite what the books may say!