function readOnly(count){ }
Sign Up ›
Starting November 20, the site will be set to read-only. On December 4, 2023,
forum discussions will move to the Trailblazer Community.
Learn More

Browse by Topic
ShowAll Questionssorted byDate Posted
+ Start a Discussion
zabahzabah 

Batch Size Strategies?

Until now, I've been fine with a batch size of less than 2000 (the max as described by the API).  However, I have many more records than 2000 and am about to embark on several queries that will require me to exceed that max number.

I can run several queries in sequence, but it'll be a pain to keep track of which records I've included in the query and which one's I haven't.  Does anyone have any other ideas on how to work around the 2000 max number?
Rick.BanisterRick.Banister
Setting your batch size to 2000 is actually not a good idea, as the amount of data will consume quite a bit of virtual memory on your client machine and you will never know if you have your entire dataset. We find that the size should typically be between 100 and 200 if you are getting a lot of fields, depending on your VM size.

Use the query and queryMore methods to iterate through a large set of records. If all you want to do is get recent changes, use the getUpdated method to get a list of ID's that have changed, and then retrieve() to get the records themselves, 200 at a time. This does not work for some of the objects (notablly OpportunityHistory and Assets), so a generalized technique might be to use the SOQL query for all records.

If you want a productize solution to this, Relational Junction is a command line interface for Salesforce replication.There is zero coding required and the schema is automatically built and maintained for you.