You see, "boards" is the main word in the phrase (we call it the "head" word) so if you are referring to two or more of them, it must receive the plural "s".
Board of directors (one board)
Boards of directors (more than one board)
Btw, it's not actually a compound noun. Examples of compound nouns are "blackbird", "footpath", "girlfriend", city-dweller", "fox-hunting" and so on.