Both are fine, but they have slightly different connotations.
"Come to the dinner" speaks of a specific event which is likely to have a set date.
"Come to dinner" does not necessarily refer to a specific event. Without further context, I would expect the dinner in this case to be tonight. This is a more general statement; i.e. whenever the dinner is, he will not come.