It's not minor at all. For a ground-source heat pump to work at all, you need to sink the pipes deep enough that the soil temperature is pretty stable year-round, which means the heat it can exchange with the aboveground air is minor or insignificant. So the vast majority of heat flux into or out of that soil is due to the heat exchange fluid circulating in the pipes.
After a few years of pumping heat out of the ground below the frost line during the winter, they'll freeze the ground solid and stop working (and possibly destroy the foundation of the house in the process, since often the pipes are installed in trenches around the house).
The only exception is if they're one of these few borderline systems that drill so deep they really are bringing up fresh energy from the depths, like some of the systems mentioned in the article.