IN SOQL can't we compare Account.RecordType.Name to the contents of a set ?

In the code below if i add Account.RecordType.Name in :recordTypeSet  in the Where clause, the query returns 0 rows.

Is it a wrong way to check if the Account Recrd Type Name exists in a set?

Please advice? Thanks in advance.

Set<ID> recordTypeSet = new set<ID>();

List<RecordType> rtList = [Select Id, SobjectType, Name From RecordType  Where SobjectType in ('Lead', 'Account') and Name in ('Supplier Record Type', 'Temp Record Type', 'Supplier Registration', 'Temp Registration')];

system.debug('rtlist size ' +rtlist.size());

for(RecordType rt : rtList){
  recordTypeSet.add(rt.Id);
}

system.debug('record type set ' +recordTypeSet);

public Map<String, String> holdingMap = new Map<String, String>();

holdingMap.put('rova@acme.com','USA');
holdingMap.put('samp@axis.com','USA');


List<Contact> contactUser = [Select Id, Account.Id, Account.Name, Account.CreatedDate, Account.RecordType.Name,Name, Email From Contact WHERE Account.RecordType.Name in :recordTypeSet AND Email in :holdingMap.keySet()];

system.debug('SIZZZZZZZZ ' +contactUser.size());

for(Contact c : contactUser){
system.debug('Record Type ' +c.Account.RecordType.Name);
}