I will do a bit more work on the infinitive phrase page to clarify these points. But, everything you said is right and everything on the page is right too. We're all correct! Hooray.
He helped to build the roof. (noun)
Here the infinitive phrase replaces the role of a noun.
He helped Jane. (noun)
The officer returned to help the inspectors. (adverb)
Here the infinitive phrase replaces the role of an adverb.
The officer returned quickly. (adverb)
Let me show you the best way to paint the door.(adjective)
Here the infinitive phrase replaces the role of an adjective. (And, this is the hard one because it replaces a compound, pre-modifying adjective.)
Let me show you the best paint-the-door way.
I'll think of a better example than that, and I'll add the explanation above.
I also see your point about clauses and phrases, but they're called infinitive phrases. I think we're just going to have to live with that. (I don't make the rules.) If that's going to keep you awake at night (as it would me), then perhaps clauses (with a verb) can be phrases (less than a sentence), but phrases can't be clauses. Maybe, it doesn't go phrases, clauses, sentences. Maybe, it goes phrases and sentences with clauses being a special "verbed up" phrase.
I am waffling now. I hope some of this answer was useful!