Well seems like you have defined a custom button for the standard layout. The visualforce page act really different and you cannot simply add a button without making any changes to the VF code.
You probably have to make changes in the page and add a button component <apex:commandButton>
Hi,
I have seen the same link in developer group. Please check the following link.
http://boards.developerforce.com/t5/Visualforce-Development/Can-not-add-Visual-Force-page-to-Custom-button/m-p/400783#M46798
If this post is helpful please throw Kudos.If this post solves your problem kindly mark it as solution.
Thanks,
Subhani,
Salesforce Certified Developer,
www.mydbsync.com.
You probably have to make changes in the page and add a button component <apex:commandButton>