I'm building a list of users, and optionally, querying for an alternative list of users, and then removing the users from list #2, from list #1. I'm using Set<User>, so I should be able to use removeAll(setOfUsers) to accomplish this, but it doesn't seem to work. I know both lists contain at least 1 common user, but after this method executes nothing is changed. What is the criteria for matching? Can I influence it somehow? I would prefer not to have to loop through both sets of returns multiple times to find any matches.

Also, before anyone asks: No, I cannot simply include the optional filtering criteria in the initial query. I would love to do this, as it would be simpler, but the complexity of both queries precludes this option.

Here's the code:

        //iterate over the matches and load the users                
        foundSet = new Set<User>();
        for(sObject s : Database.query(qry))
                foundSet.add(new User(ID        = ((User_Info__c)s).User__c, 
                                        FirstName = ((User_Info__c)s).User__r.FirstName, 
                                        LastName  = ((User_Info__c)s).User__r.LastName,
                                        City      = ((User_Info__c)s).User__r.City,
                                        State     = ((User_Info__c)s).User__r.State));
        
        //exclude those users with existing events
        if(foundSet.size() > 0 && includeConflictingUsers.toLowerCase() == 'false')
            foundSet.removeAll(buildUsersWithConflictList(new List<User>(foundSet)));