|MAXMEM defaults to the maximum amount of installed RAM that Windows has detected and is intended only as a way of decreasing the amount of RAM that Windows uses (eg for test purposes).|
If the MAXMEM value is less than what it should be there's no point trying to increase the value beyond what Windows has detected.. The problem is with your hardware not Windows.
Maybe the two RAM modules just aren't compatible with each other, That's certainly happened to me in the past. I had to get two exact matching modules, same brand, same everything