Create a controller and VF page. From the VF page send the searching document name as parameter of ActionFunction and rerender the section on that you are going to show the result and in controller query the data using LIKE clause. If it returns any record then show it otherwise display some custom message that no such document exist.

Also tryout a workaround for document search :

                                                <apex:page showHeader="false" sidebar="false">

                                                <apex:pageBlock >

                                                <b>Enter search term: </b>

                                                <apex:form >

                                                <script>

                                                function doSearch()

                                                {

                                                                var searchStr = document.getElementById('txtSearch').value;

                                                                window.open('/_ui/common/search/client/ui/UnifiedSearchResults?sen=001&sen=015&str=' + encodeURIComponent(searchStr), "_parent" );

                                                }

                                                </script>

                                                <input/>

                                                <input value="Go!" id="btnSearch"/>

                                                </apex:form>

                                                (<strong>NOTE:</strong> Only searches Accounts and document)

                                                 </apex:pageBlock>

                                                </apex:page>