MMMM, this is a good question. I suppose it's an adverb. The only rule I can find is that if an adverbial phrase appears at the end of the sentence, then there is no need to use a comma. However, if it's at the front, you should:
At 4 pm, the bell rang.
The bell rang at 4 pm.
'at 4 pm' being the adverbial phrase.
Therefore, there is no need to use a comma, but that does not mean it is wrong to do so. Therefore, I judge this to be style preference. Let your desired flow of text govern whether you use the comma.