+ Start a Discussion
Ashwin SidharthAshwin Sidharth 

Querying/Fetching Large Data

Hi ,

 

I am quite new to Saleforce. I did quite a bit of search in the community but could not find what i was looking for.

 

We have the following requirement - need to create a .net based application that will pull data records ( Contacts , Products , Feedback etc) / information from the salesforce env and export it to another system. The data volume might be quite large as we have a lot of instances or per country data. Expected volume for some of the countries will cross 300000. 

 

Initially we decided to use the Apex Api Soap service calls to retrieve the information. But we found that there is a limit in terms of the number of records that are retrieved per service request.i.e only 1000 records are returned per call, and we will need to make subsequent calls to get this remaning information. Worst part is that there is also a hard limit on the number of calls made (10000 per day). Can anyone suggest the best approach to get this done?  

 

Cheers,

Ashwin

 

 

DodiDodi

You can look to use an ETL tool if you are exporting replicating data. You can also script the data loader. Are you using any middleware inhouse? What is the daily volume of data daily?

 

You can also look to pull only updated records and not all records everytime.

Ashwin SidharthAshwin Sidharth

Hi,

 

Thanks a lot for your reply. Yes, we do have a middleware in use(Biztalk). But since there was already a tool to extract data in a similar manner from Siebel, the idea was to replace the data layer of the existing tool so that we could fetch the data from SalesForce.

Hence, we were looking at the Apex SOAP API solution to do the trick.But is this something that cannot be achieved using the SOPA API service?

The daily volume may vary , but we need to fetch data for the past 15days of activity. This might be around 100K to 300K records or even more.

When you say script the data loader, do you have any working samples around this?

 

Cheers,

Ashwin

 

DodiDodi

If you want to do it progrmatically (Apex, .Net of Java) you can look to use the Bulk API for large data sets. Personnally, unless you need realtime data, I would take a lower tech approach (depending on your budget : ) ) and script the data loader to export the records you need to CSV file and consume the files using Biztalk and process from there.


Here is a link to the details on how to do it. If you need any consulting help, check us out at cloudras.com.

 

http://wiki.developerforce.com/page/Using_Data_Loader_from_the_command_line

 

Thanks