Not only would he have felt abandoned, when his mother quickly remarried after his father's death, he could actually see the distant steeple where her mother had to relocate after her marriage - source of affection and emotional connect just tantalizingly out of reach.
That might explain his behaviour.