The actor, 41, tells PEOPLE that he didn't want anyone to make a fuss about the day ...