“Family affection was a kind of formal obligation, a duty only noticed by its neglect.”