|First, you should have OC first with the RAM at stock to confirm stability, then increased the RAM and retested.|
Second If the RAM is intended to run this way, it should have a built in XMP setting, if your BIOS supports this, you should set it this way to get it to run as intended. If it does not support it or your MB BIOS does not support it, you need to understand that most RAM that runs this way uses 'looser' timings when run at the higher settings. Try increasing from 5-6-6-18 to 6-7-7-20 and seeing if it runs, if it does, try tightening up one at a time to see if it still runs. When you hit the one that does not run, try upping the volt by +0.1V to see if it will run. If not, then back off on the timing and the voltage and try any remaining timing to see how they run. The approach to Overclocking needs to be systematic and takes time to get the best settings for your system. Always monitor all temperatures and stress test to a reasonable degree to find the best for everyday use and understand that the highest settings are not always suitable for everyday use, especially if the system or component runs too hot. If the memory will not run at the higher setting, you can either try tighter settings at the stock speed to maximize the memory or return and purchase a different set of memory.
These are general recommendations based on the info you have supplied. For more specific advice, post all of your specs/components as riider has asked.
You have to be a little bit crazy to keep you from going insane.