You need to sign in to do that
Don't have an account?
KrisMoyse
Data Loader Command Line Tool slow
I am trying to automate a data load using the Command Line tools and have everything working smoothly but it is taking 10x longer than if I do it through the Data Loader GUI.
Below is an exert from my process-conf.xml
> <bean id="csvUpsertOrderItem"
> class="com.salesforce.dataloader.process.ProcessRunner"
> singleton="false">
> <description>Upsert Transaction Headers into Orders standard object.</description>
> <property name="name" value="csvUpsertOrderItem"/>
> <property name="configOverrideMap">
> <map>
> <entry key="sfdc.debugMessages" value="false"/>
> <entry key="sfdc.endpoint" value="CUSTOM ENDPOINT"/>
> <entry key="sfdc.username" value="USERNAME"/>
> <entry key="sfdc.password" value="ENCRYPTED PASSWORD"/>
> <entry key="process.encryptionKeyFile" value="C:\Program Files (x86)\salesforce.com\Data Loader\bin\key.txt"/>
> <entry key="sfdc.timeoutSecs" value="540"/>
> <entry key="sfdc.loadBatchSize" value="2000"/>
> <entry key="sfdc.entity" value="OrderItem"/>
> <entry key="process.operation" value="upsert"/>
> <entry key="sfdc.useBulkApi" value="true"/>
> <entry key="sfdc.bulkApiSerialMode" value="true"/>
> <entry key="sfdc.externalIdField" value="SlId__c"/>
> <entry key="process.mappingFile" value="C:\Users\User\Google Drive\Automation\OrdersLine21Jan.sdl"/>
> <entry key="process.outputError" value="C:\Users\User\downloads\Logs\errorUpsertOrderItem.csv"/>
> <entry key="process.outputSuccess" value="C:\Users\User\downloads\Logs\successUpsertOrderItem.csv"/>
> <entry key="dataAccess.name" value="C:\Users\User\Google Drive\Automation\JAEG_TransactionDetails.csv" />
> <entry key="dataAccess.type" value="csvRead" />
> </map>
> </property> </bean>
From my research, it seems to be something to do with either the debug log (I think most likely) or batch size.
I have set the sfdc.debugMessages to 'false' so it is not writing the files but it does seem to write it to the command screen. I feel this could be causing the problem, is there a default log setting? Maybe a process command setting?
In the data loader document http://resources.docs.salesforce.com/200/6/en-us/sfdc/pdf/salesforce_data_loader.pdf it says the max sfdc.loadBatchSize is 200 but through the UI it sets it to 2000 when batch is true. If that does restrict it, that could explain it.
I just cant find anything recent about this problem, anyone had any luck doing this at full pace recently?
Below is an exert from my process-conf.xml
> <bean id="csvUpsertOrderItem"
> class="com.salesforce.dataloader.process.ProcessRunner"
> singleton="false">
> <description>Upsert Transaction Headers into Orders standard object.</description>
> <property name="name" value="csvUpsertOrderItem"/>
> <property name="configOverrideMap">
> <map>
> <entry key="sfdc.debugMessages" value="false"/>
> <entry key="sfdc.endpoint" value="CUSTOM ENDPOINT"/>
> <entry key="sfdc.username" value="USERNAME"/>
> <entry key="sfdc.password" value="ENCRYPTED PASSWORD"/>
> <entry key="process.encryptionKeyFile" value="C:\Program Files (x86)\salesforce.com\Data Loader\bin\key.txt"/>
> <entry key="sfdc.timeoutSecs" value="540"/>
> <entry key="sfdc.loadBatchSize" value="2000"/>
> <entry key="sfdc.entity" value="OrderItem"/>
> <entry key="process.operation" value="upsert"/>
> <entry key="sfdc.useBulkApi" value="true"/>
> <entry key="sfdc.bulkApiSerialMode" value="true"/>
> <entry key="sfdc.externalIdField" value="SlId__c"/>
> <entry key="process.mappingFile" value="C:\Users\User\Google Drive\Automation\OrdersLine21Jan.sdl"/>
> <entry key="process.outputError" value="C:\Users\User\downloads\Logs\errorUpsertOrderItem.csv"/>
> <entry key="process.outputSuccess" value="C:\Users\User\downloads\Logs\successUpsertOrderItem.csv"/>
> <entry key="dataAccess.name" value="C:\Users\User\Google Drive\Automation\JAEG_TransactionDetails.csv" />
> <entry key="dataAccess.type" value="csvRead" />
> </map>
> </property> </bean>
From my research, it seems to be something to do with either the debug log (I think most likely) or batch size.
I have set the sfdc.debugMessages to 'false' so it is not writing the files but it does seem to write it to the command screen. I feel this could be causing the problem, is there a default log setting? Maybe a process command setting?
In the data loader document http://resources.docs.salesforce.com/200/6/en-us/sfdc/pdf/salesforce_data_loader.pdf it says the max sfdc.loadBatchSize is 200 but through the UI it sets it to 2000 when batch is true. If that does restrict it, that could explain it.
I just cant find anything recent about this problem, anyone had any luck doing this at full pace recently?
I also see the sfdc.bulkApiSerialMode setting is true. At a guess, case that be set to false so that the bulk API is used in parallel to load the records?
<entry key="sfdc.loadBatchSize" value="10000"/>
<entry key="sfdc.entity" value="OrderItem"/>
<entry key="process.operation" value="upsert"/>
<entry key="sfdc.useBulkApi" value="true"/>
<entry key="sfdc.bulkApiSerialMode" value="false"/>
Still takes over 12 hours but through data loader it takes about 30 mins and that is with the batch size of 2,000.
Any other ideas?
I just noticed that you are running the bulkapi in serial mode.
<entry key="sfdc.bulkApiSerialMode" value="true"/>
Try running in parallel mode and it may do the trick for you.
Regards
Sumit
sumit@celebal.com
hi, I'm facing the same problem when uploading a large amount of records to SFDC using Dataloader CLI.
During execution of the load, it is generating the following warning messages:
visitor.BulkLoadVisitor writeSingleColumn (BulkLoadVisitor.java:222) - No value provided for field: Visiting_Housenumber_Suffix__c
This is slowing down the load enormously and eventually the load fails with error:
action.AbstractAction handleException (AbstractAction.java:204) - Exception occured during loading
com.salesforce.dataloader.exception.LoadException: ApiBatchItems Limit exceeded.
So I'm desperately seeking for a way to suppress these warning messages!
So here's what I found:
http://help.salesforce.com/HTViewSolution?id=000163407&language=en_US
https://www.interactiveties.com/blog/2009/bulk-api-nulls.php
Essentially, there's not a way to disable the warning message, even with sfdc.debugMessages set to false, because it's not a debug message. Bulk API ignores blank values (does not update) where as the default will push null values. I guess they wanted to make sure this was blatantly obvious. The #N/A work around appears to work based on the testing I did.
Need Improvements in Dataloader and Dataloader CLI - Ideas - Salesforce Trailblazer Community (https://trailblazer.salesforce.com/ideaView?id=0874V0000019ORwQAM)