"When determining what to get, I checked the freq. in the bios because I was told it could only take either 2100 or 2700. the freq read at 266 so i got them 2100. After I pulled out their 128 stick I saw it was labeled 2700 333. could this be part of the problem?"
No, that's not the problem.
"They're reported at a 512 max per slot but all the dell forums have people who say the limit is really a gig"
Why would you take the advice of someone in the Dell forum over the recommendation from Dell itself?
"is there a way to tell for sure?"
Check to see what Dell recommends. Oh yeah, you already did that & didn't follow their recommendation.
"can I change the max value somehow?"
No, you can't. The RAM limit is dictated by the hardware.
You're gonna have to face the fact that you bought the wrong RAM. There's nothing you can do to get the total amount properly recognized. Return it & buy the right stuff.