I had solved the problem by re-installing the IIS 5.1 and followed the steps
1. run the aspnet_regiis.exe -I -enable command which will install the ASP.NET4.0 in IIS and also the mime type for the .svc files ( this file is at location c:\windows\microsoft.net\v4.0.30319)
2.create new wcf service project and host it in iis.
3.Finally browse the *.svc file under the iis.
Now it is working fine for me...