|"dual channel 400MHZ i think"|
If you want our help, you need to answer the follow up questions accurately. I'm still not sure if you have DDR2-400 or DDR2-800. Do you know the difference? DDR2-400 runs at 200MHz frequency but it's DDR rating is 400MHz. DDR2-800 runs at 400MHz frequency but it's DDR rating is 800MHz. Confusing the real (frequency) with the bogus (DDR rating) can cause all sorts of problems.
I'm going to give you some suggestions:
Start by temporarily lowering the CPU multiplier to 8x. Increase the CPU frequency from 266MHz (1066MHz FSB) to 400MHz (1600MHz FSB) & configure the RAM to run in sync at 400MHz (800MHz DDR). Lock the PCIe at 100MHz & disable any Spread Spectrum settings (if there are any).
If it works, the CPU will run at 3.2GHz @ 1600MHz FSB with a CPU/DRAM frequency ratio of 1:1. If you want to overclock even higher, try increasing the CPU multipler to 8.5x, 9x, 9.5x, etc. Go one step at a time & be prepared to increase the CPU voltage (if necessary) to stablize the overclock.