"...when i check the properties of the modem it says that the device is in use by .... "
What does it say there?
That may be very important.
"....when i rebooted it automatically detected the modem and installed the software. "
If there's something wrong with the software, or settings in the software, or if something in the software is corrupted, that may not be a good thing.
You might need to Un-install the drivers for the modem in Add/Remove Programs, then follow seawatch's directions in response 2, but of course you must have the drivers for the modem handy - if you don't, get them from the Dell web site for your model on another computer first, and copy them to something.
"... Is there anyway to check what coms are in use and by what?"
If you mean Com ports (serial ports), not that I know of - that can be difficult to determine because they use really old legacy rules.
In Device Manager you can tell whether a Com port is enabled and what resources it is using, and you can look at the dial-up modem's properties to see which Com port it is set to, but you can't directly tell what is using it at any particular time there.
The Com port the modem is using, and it's resources - IRQ and I/O port address - must not be being used by something else - usually nothing else is using the I/O address in any case, but it's possible something else is using or sharing the IRQ the modem wants to use.
If you have a physical Com port, you often must disable that in the bios Setup in order for an internal dial-up modem to be able to use that Com port's resources in Windows.
A Com port showing up in Windows doesn't necessarily indicate it is enabled in the bios Setup.
Also, it's possible the modem is damaged and that's why it won't connect, and the error message is a default one that really doesn't indicate the problem. Is the laptop connected to something that protects it and it's phone line from voltage surges and spikes? Is there any possibility it was exposed to a voltage surge or spike since it last worked properly, such as from a power outage? Any lightning in your area since it last worked properly?