function readOnly(count){ }
Starting November 20, the site will be set to read-only. On December 4, 2023,
forum discussions will move to the Trailblazer Community.
+ Start a Discussion
VonveeVonvee 

WSDL .net communication speed

Hello all.  Hopefully a simple question; is there anywhere I can go to see what some expected benchmarks regarding data load and communication speed should be?  

 

For example; using our Enterprise WSDL, I just completed a test loop of Lead Inserts that worked around 100 Leads Per Minute.  To me that seems very slow.

 

What are your thoughts and suggestions?  Eventually the types of tasks (and scale)  I have to complete for my organization will be as follows below.   Prior to tackling those items, I'm looking to get an idea of best approaches.:

 

1.  alteration of our Lead database mess (probably 1.5 mill record update/delete)

2.  edit Lat & Long of leads (500,000+ at least)

3.  campaign Member refresh (500,000 at least).

 

(I was considering using some of the SQL Server CLR in conjunction with web WSDL messages, but I am rethinking this approach at the moment)

SuperfellSuperfell

There's wiki page somewhere that details the best practices for performance, 100 leads a minutes is absolutely terrible, make sure you're using batches of 200 records, use compression, make sure you're using persistent connections, reuse your sessionId (e.g. don't call login/create/login/create/login/create).

 

If you have a significant number of records to process, then you'll want to checkout the bulk api, which can process vast number of records efficently

VonveeVonvee

Simon 

 

So just to clarify, when you refer to batches it is as simple as the following code: 

 

sObject[] leads = new sObject[100];   <-- batch of 100

 

for (int xyx = 0; xyx < 100; xyx++)           

{              

  leads[xyx] = _newLead;

}

SaveResult[] sr = binding.create(leads);

 

I used this method and got immense improvement:   (note I loaded the same 100 batch 10 times for this experiment)

Starting 1000 lead load: 7/11/2011 8:19:10 AM

Ending 1000 lead load: 7/11/2011 8:20:40 AM

 

(I have not stumbled onto "compression" as you referred to yet, I will keep looking around). 

VonveeVonvee

While i'm at it, what are the hooks, catches, & perils regarding multithreading in the same context we're discussing?   I took the last experiment and racheted it up a notch by running three threads identically (using a single  private SforceService binding object, logged in prior to thread execution).  Overall the test was pretty successful, now with 3000 leads in approx 2 minutes.

 

Definately have made some significant progress since my OP; perhaps slightly refreshing it;  How Fast should the code be performing to consider it a  satisfactory/good  measure, in your opinion(s)?

 

thread test example

private void startThreeThreads()       

{

  Thread oThread1 = new Thread(new ThreadStart(loadLoopsOfData));           

  Thread oThread2 = new Thread(new ThreadStart(loadLoopsOfData));           

  Thread oThread3 = new Thread(new ThreadStart(loadLoopsOfData));

  oThread1.Start();           

  oThread2.Start();           

  oThread3.Start();

}

SuperfellSuperfell

There are issues with multithreading create/update/delete calls, depending on exactly which objects and their relationships, these operations can cause parent objects to be locked, these locks can then content between the different threads causing performance to actually be worse not better.

 

As i mentioned before, if you want the absolute best performace, use the builk API.

VonveeVonvee

Where is the best place to research the type of possible LOCK conditions you refer to?  My thought on multithread (because that's too broad a term I just realized), was to solely split like tasks into more than 1 thread...   i.e. 4 simultaneous LEAD-Load threads.... and NOT doing multiple types of object work( Lead/Opportunity/Account/Contact) threads at the same time.  

 

In reference to the bulk API; while I'm not ignoring the idea, it is not my first choice at the moment due to environment internals.  The original idea was to research easy SOAP implementation incorporated within the SQL Server CLR in our environment to create database-driven load/transformations to our cloud instance, as well as other business internals.  

 

I can't see the benefit at this time to write something that goes the entire "create job file" logic bulk requires when 99% of our usage will be of small scale where SOAP will work beautifully.

SuperfellSuperfell

I'm not sure where exactly its doc'd (other than the api docs recommend not multithreading.). Even if you mutlithread with just one object type this can still be problematic, e.g. thread 1 is inserting opportunities for account A and thread 2 is also inserting opportunties for account A, the 2nd request is going to be blocked waiting on the first to finish.

 

You should look at http compression, i saw 20-30% speed up after adding compression.