|If you remove it without using the 'safely remove hardware' feature, you can corrupt files on the drive and even the drives file tables. When this happens, sometimes the only way to use the drive again is to format it and start over. This means that any file that you do not have another copy of, may be gone forever.|
This does not always happen, or at least not this severe, but you really do not want to take that chance.
If you do not get the option to 'safely remove hardware' from the tray, you have a better chance of not damaging files if you turn off the computer first, but before that, try right clicking on the drive in My Computer (Computer in Windows 7) and select 'Eject'. This will unmount the drive and you should be safe removing it.
You have to be a little bit crazy to keep you from going insane.