function readOnly(count){ }
Starting November 20, the site will be set to read-only. On December 4, 2023,
forum discussions will move to the Trailblazer Community.
+ Start a Discussion
AlwaysConfusedAlwaysConfused 

API Performance seems extremely low when compared to the data loader

Hey Guys,

 

I'm curious as to why this code would be sooooo slow compared to the apex data loader:

it uses the recommended mechanisms / technologies :

1. Connection is held open

2. asynchronous calls are made

3. multithreading is used

 

From what I can see underlying all this .NET does the same as what the data loader source code does in that it takes the data stream and deserialises it in to an object (QueryResult) then that is handed to some other portion of code which can process the data in another thread.

 

The connection is kept and dropped only after the logout call is made (nowhere in this code).

 

In my class constructor i login to the salesforce service to get session details.

Then i attach some event handlers, then within the calling code i call Test() ...

 

service.queryCompleted += new queryCompletedEventHandler(service_queryCompleted);

service.queryMoreCompleted += new queryMoreCompletedEventHandler(AsyncFetch);

ResultsReceived += new ResultsReceivedHandler(SalesForce_ResultsReceived);

 

 public void Test()

{

try

{

string query = "select Id,AccountId,Salutation,FirstName,LastName,RecordTypeId,MailingStreet,MailingCity,MailingState," +

"MailingPostalCode,MailingCountry,Phone,MobilePhone,Email,Birthdate,CreatedDate,LastModifiedDate,LastModifiedById," +

"Gender__c,User_Login_ID__c,Date_Last_Cleaned_Checked__c,Age_Range__c from contact ";

QueryOptions qo = new QueryOptions();qo.batchSize =

1500;qo.batchSizeSpecified = true;

service.QueryOptionsValue = qo;

service.queryAsync(query, query);

}

catch (Exception ex)

{

throw ex;

}

}

void service_queryCompleted(object sender, queryCompletedEventArgs e)

{

QueryResult qr = e.Result; if (!qr.done)

{

service.queryMoreAsync(e.Result.queryLocator, e.UserState.ToString());

ResultsReceived.BeginInvoke(qr.records, qr.done, e.UserState.ToString(), null, null);

}

}

 

 void AsyncFetch(object sender, queryMoreCompletedEventArgs e)

{

QueryResult qr = e.Result; if(!qr.done)

service.queryMoreAsync(qr.queryLocator, e.UserState.ToString());

ResultsReceived.BeginInvoke(qr.records, qr.done, e.UserState.ToString(), null, null);

}

 

 void Salesforce_ResultsReceived(sObject[] Results, bool done, string Query)

{

if (Results.Length > 0)

{

// get components needed by table builder

 string[] fields = GetFieldsFromQuery(Query);

// build and return objects as datatable

DataReceived(proc.BuildTable(Results, fields), done);

}

}

AlwaysConfusedAlwaysConfused

Ok I've added this too ...

http://community.salesforce.com/sforce/board/message?board.id=NET_development&message.id=6971&jump=true

 

Seems slower now though, think i have a bug.

SuperfellSuperfell
Do you have some numbers, how much worse is your .NE code than the data loader.
AlwaysConfusedAlwaysConfused

Ok so i got really annoyed with it.

 

Re-thought about what I was doing, and the outcome is a class that now seems to perform better than the dataloader.

 

Initial test results:

 

Dataloader : 40 minutes per million records queried based on the query in my sample code above.

My code (with alterations) : 16 to 17 minutes per million.

 

Based on a total record set of roughly 2.67 million reocrds i can pull all 2.67 million in 45 minutes (ish).

 

I think i might have fixed it ...

 

The wierd thing is ... i don't even bother handling streams, i just let .NET do all that :)

This is based on me using about 4MB of internet connection bandwidth.

 

I stand corrected, the API isn't too bad once you figure it out.

VarunCVarunC
Could you please share your code with us? I'm also tryng to read data that big and speed is killing the application :(
AlwaysConfusedAlwaysConfused

Hmmm ... ok I can't post the full code listing here but if you email me with your request I will reply with the code :)

pward@ccr.co.uk