I have a custom object that has a lookup field to the Contact object.  I did the related lookup filter so that it would only pull in results for Contacts on that related Account.  The issue is that in Lightning Experience, when I click on this Contact lookup field, it shows my Recently Viewed contacts, whether they were for the related account or not.  I'd like to either somehow filter that Recently Viewed list or remove the Recently Viewed when the user clicks on the lookup field.

Any ideas on how to do that?