I've been trying to get a custom lookup page to work properly and keep running into issues.  I've looked thru various posts/blogs on the subject (like http://blog.jeffdouglas.com/2011/08/12/roll-your-own-salesforce-lookup-popup-window/) and can get a window that works like that, but that lookup page design still has some problems.  In particular, when you create a 'new' record, it doesn't close the lookup window and return to the main page, filling in the field which was being looked up.  

So I tried to address that.  First, I copied the ideas in (http://blog.jeffdouglas.com/2011/08/12/roll-your-own-salesforce-lookup-popup-window/) to create a the javascript and a test page to intercept the normal SF lookup window.  That page looks like:

<apex:page standardController="Member_Event__c" extensions="KeyPrefixes" id="customlookupTestPage">

  <script type="text/javascript"> 
  String.prototype.endsWith = function(suffix) {
    return this.indexOf(suffix, this.length - suffix.length) !== -1;
  };

  function openLookup(baseURL, width, modified, searchParam){
    var originalbaseURL = baseURL;
    var originalwidth = width;
    var originalmodified = modified;
    var originalsearchParam = searchParam;
    
    var lookupType = baseURL.substr(baseURL.length-3, 3);
    if (modified == '1') baseURL = baseURL + searchParam;
    
    var isCustomLookup = false;

    if(lookupType == "{!contactKeyPrefix}"){
      var urlArr = baseURL.split("&");
      var txtId = '';
      if(urlArr.length > 2) {
        urlArr = urlArr[1].split('=');
        txtId = urlArr[1];
      }

      // Following is the url of Custom Lookup page. You need to change that accordingly
      baseURL = "/apex/LookupPopup?txt=" + txtId;
      
      // Following is the id of apex:form control "myForm". You need to change that accordingly
      baseURL = baseURL + "&frm=" + escapeUTF("{!$Component.fm}");
      if (modified == '1') {
        baseURL = baseURL + searchParam;
      }

      // Following is the ID of inputField that is the lookup to be customized as custom lookup
      if(txtId.endsWith('patient')){
        isCustomLookup = true;
      }
    }
 
    
    if(isCustomLookup == true){
      baseURL += '&lkob=contact';
      baseURL += '&lkdisp=' + escapeUTF("Birthdate,HomePhone,MobilePhone,Phone,Medical_Record_Number__c")
      baseURL += '&lkmtch=' + escapeUTF("HomePhone,MobilePhone,Phone,Medical_Record_Number__c");
      baseURL += '&lkrt=Patient';
      openPopup(baseURL, "lookup", 350, 480, "width="+width+",height=480,toolbar=no,status=no,directories=no,menubar=no,resizable=yes,scrollable=no", true);
    }
    else {
      if (modified == '1') originalbaseURL = originalbaseURL + originalsearchParam;
      openPopup(originalbaseURL, "lookup", 350, 480, "width="+originalwidth+",height=480,toolbar=no,status=no,directories=no,menubar=no,resizable=yes,scrollable=no", true);
    } 
  }
</script>

 I think that part is working fine.  Next, I created a controller based on the ideas of (http://blog.jeffdouglas.com/2011/08/12/roll-your-own-salesforce-lookup-popup-window/) but added to it to pass in which fields to match and display for my lookup.  

public with sharing class LookupController {

  public String qString {get;set;} //  keyword
  public SObject lookupObject;
  public Contact lkupContact { get; set; }
  public List<SObject> results{get;set;} // search results
  public List<Contact> resultsContact { get { return (Contact [])results; } set; }
  public String formTag { get; set; }
  Public String textBoxTag { get; set; }

  String soqlQuery; 
  String objectName;
  String displayParms;
  String matchParms; 
  String recordTypeName;
  public LookupController() {
    system.debug('PH: LookupController: constructor');
    // PHTODO: page should run thru this displayParms list rather than have its own
    // PHTODO: save new doesn't close window & populate parent
    // PHTODO: hitting 'go' button twice for it to work
    
    // Get page parameters
    qString = System.currentPageReference().getParameters().get('lksrch');
    if(qString == null) return; // PHTODO not sure this is right
    //objectName = 'Contact';
    //displayParms = 'Birthdate,HomePhone,MobilePhone,Phone,Medical_Record_Number__c';
    //matchParms = 'HomePhone,MobilePhone,Phone,Medical_Record_Number__c';
    //recordTypeName = 'Patient';
    objectName = System.currentPageReference().getParameters().get('lkob');
    displayParms = System.currentPageReference().getParameters().get('lkdisp');
    matchParms = System.currentPageReference().getParameters().get('lkmtch');
    recordTypeName = System.currentPageReference().getParameters().get('lkrt');
    formTag = System.currentPageReference().getParameters().get('frm');
    textBoxTag = System.currentPageReference().getParameters().get('txt');
    
    lookupObject = newObject(objectName, recordTypeName);
    if(lookupObject == null) return; // PHTODO not sure this is right

    // get the current search string and create the SOQL query
    soqlQuery = dbQuery(displayParms, objectName, matchParms, qString, recordTypeName);

    setResults();
  }


  
  String dbQuery(String display, String obName, String match, String q, String recTypeName) {
    String queryStr = 'SELECT ID, Name';
    if(display != null && display != '') queryStr += ', ' + display;
    queryStr += ' FROM ' + obName + ' WHERE ' ;
    queryStr += ' (Name LIKE \'%' + q + '%\'' ;
    if(match != null) { 
      String [] matches = match.split(',');
      for (String s :matches) {
        queryStr += ' OR ' + s + ' LIKE \'%' + q + '%\'';
      }
    }
    queryStr += ') ';
    if(recTypeName != null) {
      queryStr += ' AND RecordType.Name = \'' + recTypeName + '\'';
    }
    // PHTODO: Should we add a limit? 
    // queryStr += ' LIMIT 100';
    system.debug('PH: LookupController: queryString:' + queryStr);
    return queryStr;
  }

  
  public void setResults() {
    results = database.query(soqlQuery);
    // PHTODO: Do we want to catch exceptions here?
    for (SObject o : results) {
    }

  }
  
  
  public PageReference updateResults() {
    system.debug('PH: LookupController: updating results qString:' + qString);
    soqlQuery = dbQuery(displayParms, objectName, matchParms, qString, recordTypeName);
    setResults();
    return null; // stay on same page
  }
  
  
  SObject newObject(String obName, String recType) {
    if(obName == 'Contact'){
      RecordType obrectype = [select ID from recordtype where name = :rectype];
      PH_System__c phSys = PH_System__c.getInstance();
      Account phAcct = [select ID from Account where name = :phSys.Patient_Account_Name__c];
      lkupContact = new Contact(recordTypeid=obrectype.id, accountid =phAcct.id);
      return lkupContact;
      // PHTODO: set record type, delete other
    }
    system.debug(LoggingLevel.ERROR, 'PH: LookupController: unknown objectName (' + obName + ')');
    return null; 
  }

  // save the new account record
  public PageReference saveObject() {
    system.debug('PH: LoookupController: saving object: ' + lookupObject);
    insert lookupObject;
    // reset the account
    lookupObject = newObject(objectName, recordTypeName);
    return null; // stay on page? 
  }
  
 
  // used by the visualforce page to send the link to the right dom element
  public string getFormTag() {
    return System.currentPageReference().getParameters().get('frm');
  }
 
  // used by the visualforce page to send the link to the right dom element for the text box
  public string getTextBox() {
    return System.currentPageReference().getParameters().get('txt');
  }
 
}

Again, I think that part is working fine (forgive the mix of using SObject and Contact  in some of the functions, I was in the process of generalizing the code to work on any object when I got stuck on this other issue).  

And then my lookup page (again liberally borrowing from http://blog.jeffdouglas.com/2011/08/12/roll-your-own-salesforce-lookup-popup-window/) looks like: 

<apex:page controller="LookupController" title="Search" showHeader="false" sideBar="false" id="LookupPopup">

  <apex:form id="actionForm">
    <apex:actionFunction action="{!saveObject2}" name="saveInJS" /> 
  </apex:form>


  <apex:outputPanel id="page" layout="block" style="margin:5px;padding:10px;padding-top:2px;">
    <apex:tabPanel switchType="client" selectedTab="name1" id="tabbedPanel">
 
      <!-- SEARCH TAB -->
      <apex:tab label="Search" name="tab1" id="tabOne">
      <apex:form id="lkupForm">
        <apex:actionRegion >  
          <apex:outputPanel id="top" layout="block" style="margin:5px;padding:10px;padding-top:2px;">
            <apex:outputLabel value="Search" style="font-weight:Bold;padding-right:10px;" for="txtSearch"/>
            <apex:inputText id="txtSearch" value="{!qString}">
              <!-- <apex:actionSupport event="onchange" action="{!updateResults}" />  PHTODO: this not firing -->
            </apex:inputText>
              <span style="padding-left:5px"><apex:commandButton id="btnGo" value="Go" action="{!updateResults}" rerender="searchResults"></apex:commandButton></span>
          </apex:outputPanel>
 
          <apex:outputPanel id="pnlSearchResults" style="margin:10px;height:350px;overflow-Y:auto;" layout="block">
            <apex:pageBlock id="searchResults"> 
              <apex:pageBlockTable value="{!resultsContact}" var="p" id="tblResults">
                <apex:column >
                  <apex:facet name="header">Name</apex:facet>
                    
                    <apex:outputLink value="javascript&colon;top.window.opener.lookupPick2('{!FormTag}','{!TextBox}_lkid','{!TextBox}','{!p.Id}','{!p.Name}', false)" rendered="{!NOT(ISNULL(p.Id))}">{!p.Name}</apex:outputLink>
                 
                </apex:column>
                <apex:column value="{!p.homephone}"/>
                <apex:column value="{!p.mobilephone}"/>
                <apex:column value="{!p.phone}"/>
                <apex:column value="{!p.birthdate}"/>
                <apex:column value="{!p.Medical_Record_Number__c}"/>
              </apex:pageBlockTable>
            </apex:pageBlock>
          </apex:outputPanel>
        </apex:actionRegion>
      </apex:form>
      </apex:tab>

  <script type="text/javascript">
    function saveFunction() {
      var x = saveInJS();
      console.log('in saveFunction after saveInJS: ' + JSON.stringify(x));
      // COMMENT OUT TO DEBUG      //top.window.opener.lookupPick2('{!FormTag}','{!TextBox}_lkid','{!TextBox}','{!lkupContact.Id}','{!lkupContact.Name}', false);
    }
  </script>   
 
      <apex:tab label="New Patient" name="tab2" id="tabTwo">
      <apex:form id="newform"  >
        <apex:pageBlock id="newPatient" title="New Patient" >
        
          <apex:pageBlockButtons >
            <apex:commandButton action="{!saveObject}" value="Save"/>
            <apex:commandButton onclick="saveFunction()" value="SaveInJS"/>
            <apex:commandButton action="javascript&colon;saveFunction();" value="SaveJSDirect"/>
            <button onclick="saveFunction()">MySave</button>
            
          </apex:pageBlockButtons>
          <apex:pageMessages />
 
          <apex:pageBlockSection columns="2" id="newPatientSection">
            <apex:repeat id="rptLoop" value="{!$ObjectType.Contact.FieldSets.New_Patient_Info}" var="f">
              <apex:inputField value="{!lkupContact[f]}" />
            </apex:repeat>
          </apex:pageBlockSection> 
        </apex:pageBlock>
      </apex:form>
      </apex:tab>
    
    </apex:tabPanel>
  </apex:outputPanel>
  
</apex:page>

 I've been trying different 'save' functions in attempt to make one that (a) saves the record, (b) updates the proper field on the main page with the 'new' record, and (c) closes the lookup window.  The idea being that the user is back in focus on the main page once they've created the desired record.  

I can get the record to save by just calling the controller saveObject function as an action parameter on a command button.  If, from that saveObject function, I return something like: 

  new PageReference('javascript:window.close()');

then I can both save the object and close the lookup window (so (a) and (c) above), but returning the page reference doesn't allow me to update the original field on the main page.  

So then I tried creating a javascript function (saveFunction) that calls the saveObject function via an actionFunction (saveInJS), but it doesn't look like the actionFunction is working (if I add debug code it says that saveInJS is undefined).  Howver, when I look at the source, there is a saveInJS function was defined on the page via the actionFunction statement:

<script>function saveInJS(){ SfdcApp.Visualforce.VSManager.vfPrepareForms(["LookupPopup:actionForm"]); if(typeof jsfcljs == 'function'){jsfcljs(document.forms['LookupPopup:actionForm'],'LookupPopup:actionForm:j_id0,LookupPopup:actionForm:j_id0','');}return false }</script><div id="LookupPopup:actionForm:j_id49"></div>

If the saveInJS function was getting called and saving the object, then I would have the desired behavior because the 

top.window.opener.lookupPick2

function seems to find and fill in the field correctly.  

I'm probably missing someting insanely obvious, but at this point my eyes are deceving me because every time I fix one aspect in an approach, a different aspect stops working.   Maybe its just my actionFunction is screwed up somehow, but I'm just not seeing it.  

Help?