Before you bought the keyboard, did you make sure that it was supposed to be Win98 compatible? Also, as indicated above by underdog, I believe that you may be missing some driver files. Although, normally that wouldn't prevent the keyboard from showing up at least as a standard keyboard, if nothing else.
Your main problem sounds to me, like a number of conflicts that you're having. In Windows, the yellow circle with an "!" mark always indicate a problem fault or a conflict.
Here's a solution for you. You said that you had it repaired and replaced stuff, to include the HD, on it. Well, can't you take it back to yuor computer guy and say: "Hey, Dude! How come I have these comaptibility issues showing up? After having paid you good money, isn't my computer supposed to be running perfectly?"
What's puzzling, is that you can't remove the device. I've never heard of that problem before. You should be able to remove it, and then immediately close the device manager, followed by restarting or shutting down the PC. THAT should always work. You have to shut down right azway though, and not 20 or 30 minutes later. Granted, with hardware conflicts, it can happen that windows attempts to reinstall the previously removed device after you restart - but that doesn't happen very commonly.
Is it possible that your new drive is attached to a PCI controller card, or that the replaced drive was attached to such a card? If so, then open up the case with the power off, and remove that thing once and for all (if you don't need it anymore).
Believe me, your new keyboard has nothing to do, with the IDE issue that shows up as a conflict.
Anytime that you bring a PC to a shop, you should get it back without any problems!
Hey Intel, AMD rulez!