Not only would he have felt abandoned, when his mother quickly remarried after his father's death, he could actually see the distant steeple where her mother had to relocate after her marriage - source of affection and emotional connect just tantalizingly out of reach.
That might explain his behaviour.
A calculus fit to compute on, White light and a head to drop fruit on, A mind to absorb it, And soar into orbit — That's all that it takes to be Newton. — Gina Berkeley