+ Start a Discussion
aebenaeben 

Batch Apex limitation on records.

In Winter 10.

 

I have a batch apex that could potentially return at least  250000 records. 

 

Documentation says that batch jobs using a query locator can return 50 million records.

 

But, the batch job that I have does not work if the query returns even 20000 records.

 

Anyone with the same issue? 

Best Answer chosen by Admin (Salesforce Developers) 
Patrick BulaczPatrick Bulacz

Yes the case I logged has been resolved. I didn't get told exactly what the problem was or how it got fixed but somehow it's been resolved. As yet haven't tested all cases but the ones that were causing issues previously are now sorted.

 

Patrick

All Answers

TehNrdTehNrd
What is the error? I would guess that you are hitting a limit when processing a batch of 200.
aebenaeben

No. It was not calling the execute callback method. 

 

I think the issue is with a sub query. Let me confirm in a bit.

 

 

TehNrdTehNrd

I too am encountering a similar issue. I wonder if they are related.

 

http://community.salesforce.com/sforce/board/message?board.id=apex&thread.id=20851

 

I have an open case but no reponse yet.

aebenaeben

Looks mine was a sub query issue.

 

My old query that was not working

 

Select Id, Name, ChildObj__r.Parent__c from Account 

 

I changed the query to  

 

Select Id, Name from Account  

 

then it run successfully by executing chunks of 200, 132 times.

 

I am planning on querying the child relationship inside the execute method. Hopefully it will work. Will get back on that.

 

As for the your issue, yours is a straight forward query. I don't have a clue why its failing after 10000. 

TehNrdTehNrd

For mine, it's as if the 10,000 query rows governor limit is being used.

 

I'm curious if someone from salesforce could comment on the type of queries a batch job can take. I didn't see anything in the docs but after your testing there appears to be some restrictions.

aebenaeben

I have to agree with you.

 

BTW I made the change I had mentioned in my previous post. And it works at least for now.

 

I am still not sure about the 50 million limit.

 

Reason being even if i am still using the old query, there should have been only 52800 record. Because there is 26400 accounts and a sub query for each so 26400 times 2 is 52800. But the limit is 50 million. Either our interpretation of the doc is wrong or there is something wrong in the implementation.

 

 

 

 

EvanCEvanC

I found a similar issue where a simple batch apex class for deleting records works fine with 100,000 records, but fails for 300,000.  I am interested to hear if you get a response from support on batch apex limits.

 

Here is my code:

 

 

global public class BatchDelete implements Database.Batchable<SObject> { public String query { get; set; } global database.Querylocator start(Database.BatchableContext bc) { return Database.getQueryLocator(query); } global void execute(Database.BatchableContext bc, Sobject[] scope) { delete scope; } global void finish(Database.BatchableContext bc) { } public static testMethod void testBatch() { List <Account> accns = new List<Account>(); for(integer i = 0; i<10; i++) accns.add(new Account(name='pleaseDeleteThis'+i)); insert accns; Test.StartTest(); BatchDelete bd = new BatchDelete(); bd.query = 'select id from account where name like \'pleaseDeleteThis%\''; ID batchprocessid = Database.executeBatch(bd); Test.StopTest(); System.AssertEquals(0, database.countquery('SELECT count() FROM account where name like \'pleaseDeleteThis%\'')); } }

 

 

 

TehNrdTehNrd

This is the latest info I have, "The case is specific to your organization and it has been escalated."

 

Definitely some sort of bug.

pbulaczpbulacz

I have a similar issue. I have a batch apex job that has been deployed to live and has been tested with small batches.. and it works (largest test batch was only 195 records though).

 

However another batch I have 5000+ records that get queried.. for each of those 5000 I need to create a new record in another object and insert it.

 

I have triggers firing off of the object being inserted so I run into some script statement and heap size limits.

So I figured I'd be able to just do Batchable.Execute(Class,50);  To run my batch class in chunks of 50 as I know that 50 will work everytime.

 

My debug logs show "Operation Batch Apex" Success....  and nothing else!

 

And my Apex Jobs log shows"total batches = 0, batches processed = 0, failures = 0, status = completed"

 

I'm at a loss so any help would be great!

 

No apex errors no major failures nothing, just nothing happens at all basically. Strange as I know it has the capacity to work, just want to know where it's failing or if it's failing at all.

 

Debug Log Shows

 

SOQL locator query with 5705 rows finished in 935 ms

Cumulative resource usage:

Total email recipients queued to be sent : 0
Stack frame variables and sizes:

 

 

EvanCEvanC

I also had this same experience - a query that should return thousands of records comes back with 0 batches. In fact, when I add ' limit 1000' to the end of the query, I get 5 batches of 200 - but without that, I get nothing.

 

I have submitted a case and it is being escalated.

Patrick BulaczPatrick Bulacz

Thanks for the reply there EvanC. Let me know how your case escalation goes. I have also submitted a case.

 

limit 1000 works for you, you say? I might try that at least and see if I get some form batches running.   

 

Update:

 

Okay have tried with a limit put in place... It worked for a set of 327 records chunking at 50 records intervals.

 

However another batch with 3300+ records did not work.

 

This isn't making much sense! 

Message Edited by Patrick Bulacz on 10-15-2009 09:14 PM
tmatthiesentmatthiesen

There is a known issue with Batch which is causing requests of more than 10K records to be incorrectly blocked. Please note, this issue generates the explicit exception: SOQL locator query with 10001 rows

 

This issue is only affecting a subset of orgs - we are working on resolving this immediately with an ETC of Sunday night.  I will respond to this thread when this fix has been applied to all orgs.

 

The other issues: batch executes successfully with no items(zero items processed) is not a known issue and might be a problem with the way the querylocator is being constructed.  Please log a case if you believe your querylocator is returning records but no items are created/processed. 

EvanCEvanC
A release came out this week that appears to have fixed the issue of large queries.  I haven't had a chance to test all our use cases, but it looks promising.
Patrick BulaczPatrick Bulacz

Yes the case I logged has been resolved. I didn't get told exactly what the problem was or how it got fixed but somehow it's been resolved. As yet haven't tested all cases but the ones that were causing issues previously are now sorted.

 

Patrick

This was selected as the best answer
VarunCVarunC

Hi ..

 

I'm trying to develop a solution such that I fetch Accounts and its related records in another Object using a relationship query like this:

 

Select Id, Name,

   (Select Id, Name From Contacts)

From Account

 

 

Now when processed via Batch APEX it processes records in chunks of 200. Now my concern is, How does this chunk gets divided into batches?

 

Consider that I get over 500 ACCOUNTS, and For Each Account I got 300 Contacts, so will my batch break like 1 Account and its 200 Contacts processed in first batch, and in next batch 2 accounts with 100 contacts each and so on .... ?

OR it will create a batch such that 200 accounts with each Account holding its FULL share of 300 contacts from relationship query?

 

fgwarbfgwarb

I think that the following footnote means you'll get 2 accounts (2 of 200 top level queries) with 300 contacts each (600 of 600 sub-queries).  

 

Governors & Limits: (new window)
2 In a SOQL query with parent-child relationship sub-queries, each parent-child relationship counts as an additional query. These types of queries have a limit of three times the number for top-level queries. The row counts from these relationship queries contribute to the row counts of the overall script execution.

 

That's just how I interpret the footnote though and it doesn't really answer the question of "How does Apex handle a query that exceeds limits."  (If I'm right about the sub-queries, the example would be if I was querying an account with 601 contacts.

Message Edited by fgwarb on 11-20-2009 02:55 PM
VarunCVarunC

Thanks. I got my Batch APEX script created fine. But I'm receiving this Error now sometimes:

 

System.Exception: Attempted to schedule too many concurrent batch jobs in this org (limit is 5).

 

 

FYI, I've my Batch Script executed After/Before Insert/Update/Delete trigger. I thought the Batch Apexrun in asynchronous way such that it will execute when resources are available, but it seems now they are not queing Batch jobs and running whenever they are logged.

 

is there anyway I can make them run in queue ?

fgwarbfgwarb

http://www.salesforce.com/us/developer/docs/apexcode/Content/apex_batch.htm?SearchType=Stem&Highlight=batch|five

 

Warning
Use extreme care if you are planning to invoke a batch job from a trigger. You must be able to guarantee that the trigger will not add more batch jobs than the five that are allowed. In particular, consider API bulk updates, import wizards, mass record changes through the user interface, and all cases where more than one record can be updated at a time.