function readOnly(count){ }
Sign Up ›
Starting November 20, the site will be set to read-only. On December 4, 2023,
forum discussions will move to the Trailblazer Community.
Learn More

Browse by Topic
ShowAll Questionssorted byDate Posted
+ Start a Discussion
Evgeny KabantsevEvgeny Kabantsev 

SOAP responses are unacceptable slow on a certain endpoint. Could somebody help?

Hello there,

For our production and testing purposes we are using several Salesforce environments connecting them via related SOAP endpoints. 

In somewhere about 4 months ago our production service become affected by significantly decreased performance of Salesforce SOAP endpoint: all queries through SOAP endpoint are responded in average 10 seconds each.
At the same time the same requests to the other Salesforce environments (via different SOAP endpoints) are responded in less then a second (~0.5s average). It is the same performance, which was observed in communication between our production service and Salesforce via SOAP before degradation described above.
Also it is quite interesting, that Salesforce portal looks working just fine: there are no visible lags in operations if they are performed directly on Salesforce site.

Since then we have tried multiple times to solve this problem with Salesforce support tea, but there is still no positive dynamic: from their position we are using SOAP on base supscription and so then they cannot support us in case of degradation of performance of the Salesforce service behind SOAP.

Here are examples of simple queries we are performing via SOAP to Salesforce database and average(for 10 requests) timing for two different endpoints:

# SF Query:
 > SELECT Id, SolutionNumber, SolutionName, SolutionNote, Technical_Document__c FROM Solution IsPublished = true AND IsReviewed = true
# Average response time for https://www.salesforce.com/services/Soap/u/20.0 
 => 10.72s
# Average response time for https://cs43.salesforce.com/services/Soap/u/20.0
 => 0.57s

# SF Query:
 > SELECT Id, MasterLabel, ParentId, SortOrder, SortStyle FROM CategoryNode ORDER BY SortOrder
# Average response time for https://www.salesforce.com/services/Soap/u/20.0
 => 10.41s
# Average response time for https://cs43.salesforce.com/services/Soap/u/20.0
 => 0.49s

# SF Query:
> SELECT Id, SolutionNumber, SolutionName, SolutionNote, Technical_Document__c, 
  (SELECT Id, Name, Description, CreatedDate, CreatedBy.FirstName, CreatedBy.LastName FROM Attachments ORDER BY CreatedDate)
  FROM Solution WHERE Id = '50150000000T6sfAAC'
# Average response time for https://www.salesforce.com/services/Soap/u/20.0
 => 10.68s
# Average response time for https://cs43.salesforce.com/services/Soap/u/20.0
 => 0.52s
 
# SF Query:
> SELECT Body, BodyLength, ContentType FROM Document WHERE Id = '01550000001QnorAAC'
# Average response time for https://www.salesforce.com/services/Soap/u/20.0
 => 10.67s
# Average response time for https://cs43.salesforce.com/services/Soap/u/20.0
 => 0.54s


And so on... We have done a lot of measurements and we are confident that all requests via SOAP to SF are performed such slow with our production endpoint only.
Tables queried above has the same contents in production and test environments, so it is not related to queried table size.
We are querying SOAP endpoint quite rare even in production (about several times in hour with bunch of 3-5 queries for a single operation: list contracts, create ticket and so on), so it this issue is also not related to some kind of high load.

Our code related to SOAP integration was not changed for about 2 years, and our infrastructure was not updated in the DC where our service is running, so it is also not related to changes on our side.

The interesting thing is that our problems with SF are started to happening after Salesforce DC outage several months ago. Not sure, but maybe there is some kind of database index or even maybe hardware issue out there?

I hope all reading this post will agree with me that this timing is not acceptable for production database even during superhigh load.

Bad thing for us is that Salesforce support team seems not interested in solving our problem. And it is realy making our customers angry, that we could not solve it for 2 month already. Can somebody suggest how can we fix this issue?

With thanks in advance and looking forward hearing your advices,
Evgeny Kabantsev.
Best Answer chosen by Evgeny Kabantsev
Gordon EngelGordon Engel
The query analyzer says that these SOQL queries should finish in less than a second, and the number of rows retrieved is reasonably small. Since these queries are hitting different objects, I don't think there is a problem with the indexing or the database.

The next logical step would be to test the network to see if there is packet loss or a problem with the routing. Do you have some network tracing tools? 

Can you test these SOAP calls from a different source network, like a home computer or a remote office?

All Answers

Gordon EngelGordon Engel
It could be a stale index that needs to be coalsced.  Do you have a case number so I can review your org details?

You should be able to tell if the SOQL queries themselves are slow or if it is only related to the SOAP request.  Can you eliminate SOAP as a suspect?
Ksenia PletnevaKsenia Pletneva
ticket: 18725621
Gordon EngelGordon Engel
The query analyzer says that these SOQL queries should finish in less than a second, and the number of rows retrieved is reasonably small. Since these queries are hitting different objects, I don't think there is a problem with the indexing or the database.

The next logical step would be to test the network to see if there is packet loss or a problem with the routing. Do you have some network tracing tools? 

Can you test these SOAP calls from a different source network, like a home computer or a remote office?
This was selected as the best answer
Evgeny KabantsevEvgeny Kabantsev
Sorry for late response - since my last message additionally we experiensed problems with connection to SF SOAP API via this endpoint: https://www.salesforce.com/services/Soap/u/20.0 
Looks like this endpont become deprecated some time ago, and finally it stopped responding, so all our production connections to SF SOAP API started to become broken one by one.
If somebody is facing same problem, we fixed it by replacing endpoint URL by https://login.salesforce.com/services/Soap/u/20.0 (
according to this article (https://developer.salesforce.com/docs/atlas.en-us.api.meta/api/implementation_considerations.htm))

And as far as endpoint problem was resolved, today I was able to check response time for same queries from other location, and results are incredible: for every query listed above SOAP responded in average 0.8 seconds. :)

So looks like the problem is around networking in our datacenter. We will investigate and definitely resolve it. :)

Thank you so much for your help!

With best regards,
Evgeny Kabantsev.
Evgeny KabantsevEvgeny Kabantsev
Root cause of our problem was unconscious DNS server (one of many, but first in name resolving configuration): requests for dcl5-phx.na3-phx.salesforce.com were performed slow in phase of name resolution (up to 5 seconds): 
# ping dcl5-phx.na3-phx.salesforce.com -s 1372 -c 1
PING dcl5-phx.na3-phx.salesforce.com (136.147.42.23) 1372(1400) bytes of data.
1380 bytes from dcl5-phx.na3-phx.salesforce.com (136.147.42.23): icmp_seq=1 ttl=244 time=55.8 ms

--- dcl5-phx.na3-phx.salesforce.com ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 5070ms
rtt min/avg/max/mdev = 55.898/55.898/55.898/0.000 ms

Hotfix for us was to disable bad DNS server on all our application servers working with Salesforce SOAP API. After that, requests to SF backend become fast enough again. :)

Hope this will help somebody else to resolve their similar issue.

With best regards,
Evgeny Kabantsev.