|The Library window replaced the old Downloads window in FF 20. The Library isn't the place on your computer where your download is located. The Library is the same window that helps organize your bookmarks and view your browsing history.|
Verify that your downloads aren't actually located on your E:\ drive like you specified. If they're not, download something small and make a note of the full name. Then use your computer's Search function for the download's full name, then report back where it downloaded. If they're indeed being directed to your hard drive rather than your USB drive, than it sounds like Firefox isn't recognizing that your USB drive is properly connected, so it's utilizing a default location instead. In the mean time, try using the "Always ask me where to save files" option located in Tools > Options > General tab.
Apologies if I don't respond to your reply immediately. I don't check this site daily, but you're welcome to PM me as a reminder.