Hi tfoley,

the import wizard does not look for duplicates using phone numbers generally.

For standard objects, like leads and contacts you can de-duplicate contacts by email. This is an option you have when you are  using the import wizard. 

For custom objects, you need to setup external IDs - from help:

An external ID is a custom field that has the “External ID” attribute, meaning that it contains unique record identifiers from a system outside of Salesforce. When you select this option, the import wizard will detect existing records in Salesforce that have the same external ID. Note that this operation is not case-sensitive - for example, “ABC” will be matched with “abc”. However, there is an exception: if the custom field has the separate “Unique” attribute and the case-sensitive option for that attribute is selected, uppercase and lowercase letters will not be considered identical. For more information, see Custom Field Attributes:

For each object that can have custom fields, you can set up to three custom text, number, or email fields as external IDs. An external ID field contains record identifiers from a system outside of Salesforce.

You can use an external ID field to update or upsert records using the API. When using the API or import wizards for custom objects and solutions, you can use this field to prevent duplicates by also marking the field as Unique.

When you import custom objects, if you choose to prevent duplicates,  the wizard will discover the external ID field and will prompt you to select it for matching purposes and what to do when a duplicate is found (you will need to map the column from your input file to the external ID correctly).  

External IDs can be used with the Apex Data loader too.