Why does the speaker consider saying "elves" to his neighbor? What causes the speaker to change his mind?

I'm assuming this is the Mending Wall by Robert Frost.

The speaker considers saying "elves" because it refers back to the lines of the poem where the speaker explains "something there is that doesn't love a wall / and wants it down." The speaker wants to poke fun at his his everyday, ordinary neighbor and proposes that it is a magical creature that is pulling the stones and creating gaps.