No, it's not an adverb; it's a determinative functioning as a modifier in the phrases "the harder" and "the more". I know it seems like an adverb, but it keeps its status as a determinative irrespective of how it is functioning. PaulM