As it turns out, earlier in the special at the winter party, everyone seems to like Nestor and one donkey apologizes for making fun of him. It seems Nestor might have just came by to say hi, but then saw that Olaf was happy to see him. There is no evidence in the special that proves he stayed there for long.
This is an audio mistake. After the line, "Nestor's mother got him a bright pair of stockings...," the narrator lowers his voice, resulting in a mumble of words. If you listen carefully, you hear the rest of the sentence, "that Olaf's wife had thrown away." The words are also in the subtitles.
