American here. I first heard the word conkers from Hitchhiker Guide to the Galaxy, but this was pre Google, so I just figured it was some scifi word.
Then I learned that it was a game from an episode of Kipper the Dog, in which his cheeky friend Tiger cheats at the game by painting something to look like a chestnut.