As already stated, VM Swap only gets utilised when the phyiscal host runs out of real memory to service a guests needs. it is not like a windows swap file.
one quick way of reducing the swap file is to configure some memory reservation, this however has the side effect of reducing the efficiency of HA and DRS.