|Do you have a analog line?|
VoIP doesn't go over an analog phone line but a digital one like DSL or broadband. You need to connect the rj11 to an analog line.
Doesn't matter if com1 or 3 as long as you don't use both.
Under control panel, other options is a modem test program in Vista [least my ultimate version has it]. Might start there. Also make sure in the bios the modem is enabled.