When I try to create a self-signed certificate, it says the WSUS server cannot issue a self-signed certificate.
Perchance this WSUS server is installed on Windows Server 2012 R2? If so, this is expected behavior. The capability for creating self-signed certificate was deprecated in WS2012, and disabled in WS2012R2.
This WSUS Product Team Blog article describes a registry value that can be created to disable this blocking behavior. Once that registry valeu is created, and the WSUSService restarted, then the Patch Manager server will be able to create the self-signed certificate.