Is the voltage difference potential or potential energy?

Electric potential energy is the energy it takes to put a charge in a given place, for example, near another charge. If the particle has to be held back (because it is attracted) then the potential energy is negative; that's because you can get energy from that particle as you move it in.

Electric potential of a charged particle is the potential energy divided by the electric charge of that particle.