function readOnly(count){ }
Starting November 20, the site will be set to read-only. On December 4, 2023,
forum discussions will move to the Trailblazer Community.
+ Start a Discussion
Prachi SPrachi S 

Importing large CSV files via Batch Apex without hitting heap size governor limit?

Hi, 
I have implemented a visualforce page to upload a CSV file and batch apex that reads the CSV and loads data into multiple related custom objects. Now visualforce page allows me to load CSV upto 10 MB but when I try to parse this huge file in apex I run into heap size governor limits.

I have implemented the parsing of CSV file in batch apex exactly as per financialforce customization from below link:
https://developer.financialforce.com/customizations/importing-large-csv-files-via-batch-apex/

The post mentions:
You can create a batch process to read the file in chunks and define the number of lines to read for each chunk.
To do this, create a batch apex process where the scope size defines the number of records to read for each chunk. In the sample code that follows, lines from a CSV file are to be read rather than records. The start method returns an Iterable<String> that contains the lines to be processed in the execute method. Afterwards, the process reads the list of lines using the CSVReader in the same way as an online process.
 
global with sharing class ReadAndPopulateBatch implements Database.batchable<String>, Database.Stateful
{
   private String m_csvFile;
   private Integer m_startRow;
   private CSVParser m_parser;
   private static final Integer SCOPE_SIZE = 100;
   public ReadAndPopulateBatch(){....}
   public static ID run(){....}
   global Iterable<String> start(Database.batchableContext batchableContext)
   { 
       return new CSVIterator(m_csvFile, m_parser.crlf);
   }
   global void execute(Database.BatchableContext batchableContext, List<String> scope)  
   {
       //TODO: Create a map with the column name and the position.
       String csvFile = '';
       for(String row : scope)
       {
          csvFile += row + m_parser.crlf;
       }
       List<List<String>> csvLines = CSVReader.readCSVFile(csvFile,m_parser);
       //TODO: csvLines contains a List with the values of the CSV file.
       //These information will be used to create a custom object to
       //process it.
   }
   global void finish(Database.BatchableContext batchableContext){......}
}
Although this post recommends to read the file in chunks it doesnt explain how to do so. It defines a variable private static final Integer SCOPE_SIZE = 100; but doesnt really use it in the example provided.

The input my batch class constructor gets is a BLOB of size 10 MB. How do I read this file in chunks in my apex class so that the user doesnt have to split the file for the data load to work?

Any advice will be really helpful. Thanks!
 
SandhyaSandhya (Salesforce Developers) 
Hi,

I think you can specify the batch size like below when you are executing your batch class.
 
ExampleBatchClass b = new ExampleBatchClass(); 

//Parameters of ExecuteBatch(context,BatchSize)

database.executebatch(b,10);
In your above class, you can have this method.
 
global void execute(SchedulableContext context) {
        Database.executeBatch(this,SCOPE_SIZE );
    }



Please refer below link to get more information.

http://salesforcegurukula.blogspot.sg/2016/05/how-to-write-batch-class-in.html

Hope this helps you!

If this helps you, please mark it as solved so that it will make for others as a proper solution.

Thanks and Regards
Sandhya
Abu Zafar WahidAbu Zafar Wahid
Hi I've been working with same batch processing, can you please let us know how is it going?

Thanks,
Wahid
Iñigo PuigPeyIñigo PuigPey
Hello Prachi. I am looking for using this class of Financial Force, but they do not give the class "CSVParser". Could you help me so I can use it too? 

Thank you!
Mike Barbarelli 1Mike Barbarelli 1

 

Iñigo PuigPey: 

If you're still interested, you can find it here!

http://frombelvideres4thfloor.blogspot.com/2010/10/ietf-rfc-4180-compliant-csv-reader-for.html

Happy trails.