Am I being too sensitive because he wouldn't give it to me?

In this case it has nothing to do with him remembering their birthdays. His attitude tells me that he is in bad with his parents and doesn't want to even see them nor to have you see them. Maybe he is hiding something from them like maybe they wouldn't approve of you or your nationality or religion. There is just something more going on here than you are seeing by the way he avoided the address and also said that they were just senile. No 'loving' child would say that about their parents so he sounds like he isn't in a good relationship with them at this time. I'd be cautious then about him too. How does he act about your parents. Watch his manners and eye contact to see what you really need to see but something is amiss in this for sure.