We have written a de-duplication logic for contact records where we call a batch job from trigger (Yes, it sounds weird but the only thing that seems to work as we have variable criteria for each account). To overcome batch schedule limit of 5, we are using data loader with bulk API enabled and the natch size set to 1000 so that we can upload 5000 records successfully without hitting the 5 batch job limit. When I am testing with 3000 thousand contact records, let say they are named from Test0001 to Test3000 I am observing a strange behavior.

For 3000 records, 3 batch jobs start to run (as batch size is 1000). I am passing the newly inserted records in parameters to the stateful batch class. What I expect is that 1000 records will be passed for each of the 3 batch jobs and they will be compared to existing records for duplicates (which I query in start method of batch) but I only get Test0001 to Test0200 i.e. from batch of 1000 records inserted via data loader API only FIRST 200 records are passed in parameter to the batch class and rest 800 are not. This is something strance as it means only first 200 records are processes if I insert using a batch size of 1000 through data loader with Bulk API enabled.

Has anyone of you encountered this issue or have any ideas to share on how to deal with it? I can share code as well but I think the question is more conceptual. Any help is much appreciated.

Thanks