[FIX] server: Do not listen when --no-xmlrpc is set

This is a fix for f04f4099, which only
prevented the workers from being spawned in Prefork mode, while the
socket was still being bound to - this is a problem when starting a
worker-only server as it cannot coexist with the XMLRPC server on the
same machine.

Closes #1828