function readOnly(count){ }
Sign Up ›
Starting November 20, the site will be set to read-only. On December 4, 2023,
forum discussions will move to the Trailblazer Community.
Learn More

Browse by Topic
ShowAll Questionssorted byDate Posted
+ Start a Discussion
jhartjhart 

MetaData API Bug - Code coverage of unrelated code prevents deploy of pages and static resources

Pages and static resources cannot be packaged yet.

So, as the final step in our app's install process (after the packageable stuff has been deployed), we use the metadata API to deploy pages and static resources.  We use the declarative metadata calls to deploy zip files, to be precise.

The expected behavior is that this would work fine - the zip file contains no code at all, so code coverage isn't a concern.  The packages should just upload & deploy.

However, we have run into a bug - Salesforce runs the tests for *all* unpackaged code - including things that aren't in the zipfile, and are totally unrelated to our application - and uses the results from those to prevent our deploy from succeeding.

As an example, we deploy a package of static resources.  These are just static files - no code, no tests, just files.

When we deploy those static resources into an org that has downloaded other unmanaged code (in our case, a piece of sample code from Salesforce labs), the deploy call fails because Salesforce runs the tests in that other, unrelated unmanaged code and then declares that our code coverage statistics aren't good enough.  Which doesn't really make sense because we're only deploying some gifs and stylesheets!

I've played with the various "deployOptions" settings but none of them help.  Apparently Salesforce says "hmm, you're deploying unmanaged code.  I guess I'll test all the *other* unmanaged code, and if it doesn't have good enough code coverage, I'll prevent *your* deploy from succeeding."

Here is the final error output in parse format.  I can post the XML too if anyone wants it:

Code:
deploy failed: {
  id => "04s600000004JHKAA2",
  messages => {
    changed  => "true",
    created  => "false",
    deleted  => "false",
    fileName => "resources/package.xml",
    fullName => "package.xml",
    success  => "true",
    },
  runTestResult => {
    codeCoverage     => {
      dmlInfo => { column => 9, line => 66, numExecutions => 1, "time" => "0.0" },
      id => "01p6000000001fPAAQ",
      locationsNotCovered => { column => 9, line => 70, numExecutions => -1, "time" => "-1.0" },
      methodInfo => { column => 37, line => 3, numExecutions => 2, "time" => "179.0" },
      name => "unsubscribe",
      namespace => undef,
      numLocations => 28,
      "numLocationsNotCovered" => 4,
      soqlInfo => { column => 20, line => 53, numExecutions => 1, "time" => "0.0" },
      type => "Class",
      },
    codeCoverageWarnings => {
      id => "01p700000000GCuAAM",
      message => "Average test coverage across all Apex Classes and Triggers is 15%,
 at least 75% test coverage is required",
      name => undef,
      namespace => undef,
      },
    numFailures      => 0,
    numTestsRun      => 2,
    successes      => {
      id => "01p6000000001fPAAQ",
      methodName => "testUnsubscribe",
      name => "unsubscribe",
      namespace => undef,
      "time" => "4.6454630156326666E18",
      },
    totalTime      => "955.0",
    },
  success => "false",
}

 


JonPJonP
The Visualforce and Metadata API teams are looking into this scenario.
Ron HessRon Hess
This sounds like a bug, can you open a case so that we can track the work related to addressing this,
thanks
jhartjhart
Ron,

OK, it's logged to case 01963697

thanks,
john
mtbclimbermtbclimber
jhart wrote:

However, we have run into a bug - Salesforce runs the tests for *all* unpackaged code - including things that aren't in the zipfile, and are totally unrelated to our application - and uses the results from those to prevent our deploy from succeeding.


Running tests for *all* unpackaged code is as designed.   Unpackaged/unmanaged code is *owned* by the customer that created/installed it. Your workaround is essentially acting as though you are the customer that owns this code so while this is unrelated to your application by virtue of using the metadata API as an installer your installation looks like the customer deploying their own code - not as an artifact of your application.  We have no way of differentiating these operations.

The other unmanaged code you referenced includes tests that cover it 85% so I'm not sure why this particular package would be blocking you.  If this is a production org into which you are trying to deploy then please update your case with the organization ID or a username within the customer account so we can look into that further. There should not be any cases where a customer has sub-75% coverage where this would block your deployment if there are no code changes.

I don't disagree that static resources should be exempt from test execution (not the same for pages or components, however). We are looking into that now. Just be aware that your deployment is going to be treated as though you are the customer doing the deployment so namespacing, use of global access modifiers, etc will come into play where you might otherwise not want/need them to do so.

Finally, rest assured that packaging of Visualforce is our #1 priority for the next release.

Regards,
jhartjhart
Andrew,

I think I understand what you're saying.  A few thoughts:

(a) If the one piece of non-managed code that we had in our org had 85% coverage, why was the static resource upload denied with a code coverage error?

My guess here is that the system ran only the unmanaged tests, but then compared that coverage number to the total # of lines in the org - including the managed code that wasn't tested.  (although I think that would result in a number much lower than 15%, as we have many lines of managed code and the sample labs code is pretty short if I recall).

The product org ID is 00D300000006HXS.  I'm posting it here b/c Tier 1 support has already closed the case with a "we don't support AppExchange apps" message.  I've already un-installed the sample code (in order to deploy the pages & static resources, which went fine after the sample code uninstall), but let me know if you'd like me to re-install to reproduce the issue.


(b) In any event, why should code-that-is-already-deployed prevent other code from uploading?

It seems to me that when you upload a package, the system should only test what's in that package plus items that depend on it.  By your inductive argument, if my org has already deployed a separate unmanaged package then that package must be OK coverage-wise, so why bother re-running those tests when uploading something unrelated?

Exempting static resources from testing would derive from this point.



Thanks for your notes re: namespace modifiers and upcoming priorities.  We really look forward to when these items are packageable, and we can remove steps from our post-install process.  We were really wishing that day were already here back when we coded up our current SSO+ANT way of doing it...

thanks,
john
jhartjhart
Andrew,

This is still happening in Winter09 - in fact it has effected two customers trying to install our product today.

Our product comes with an "automated page layout editor" so users can choose which page layouts they want customized with our application's custom fields and Visualforce "page snippets".

This page layout editor uses the MD api to download page layouts, edits them according to the user's preferences, and then uploads them again.

There is no code whatsoever being uploaded - just page layouts.

However, a number of our page layout uploads are being denied with code coverage errors.  Here's the most recent one:

Code:
deploy failed: {
  id => "04s40000000607HAAQ",
  messages => {
        changed  => "true",
        created  => "false",
        deleted  => "false",
        fileName => "unpackaged/package.xml",
        fullName => "package.xml",
        success  => "true",
      },
  runTestResult => {
        codeCoverage         => {
                                  dmlInfo => { column => 9, line => 66, numExecutions => 1, "time" => "0.0" },
                                  id => "01p400000004PHXAA2",
                                  locationsNotCovered => { column => 37, line => 69, numExecutions => -1, "time" => "-1.0" },
                                  methodInfo => { column => 37, line => 3, numExecutions => 2, "time" => "121.0" },
                                  name => "unsubscribe",
                                  namespace => undef,
                                  numLocations => 23,
                                  "numLocationsNotCovered" => 2,
                                  soqlInfo => { column => 20, line => 53, numExecutions => 1, "time" => "0.0" },
                                  type => "Class",
                                },
        codeCoverageWarnings => {
                                  id => "01p400000004QulAAE",
                                  message => "Average test coverage across all Apex Classes and Triggers is 15%, at least 75% test coverage
is required",
                                  name => undef,
                                  namespace => undef,
                                },
        numFailures          => 0,
        numTestsRun          => 2,
        successes            => {
                                  id => "01p400000004PHXAA2",
                                  methodName => "testUnsubscribe",
                                  name => "unsubscribe",
                                  namespace => undef,
                                  "time" => "4.6445306297723126E18",
                                },
        totalTime            => "3057.0",
      },
  success => "false",
}

 
When this happens, our users are forced to point-and-click their way through (possibly very large numbers of) page layout edits.  Any chance this will seen some developer attention?

I recall the argument for re-running tests was inductive, along the lines of "if the org has code, then that org must have uploaded it at some point, so it must have passed coverage tests then, so it should still pass coverage tests now".  This discounts the possibility of a poorly-written test succeeding at one point in time but then failing later (perhaps due to data dependencies), it also strikes me as an argument that could be used just as easily to justify *not* running tests on code that is installed (and thus has already passed coverage tests).

Another issue with running all tests when uploading something as non-programmatic as page layouts is what happens when all the existing code does have sufficient coverage, but there's a bug in the code coverage calculation itself (for example see this post).
mtbclimbermtbclimber
Can you have your customers log a case on this? We need to investigate how they've gotten code deployed that is not meeting our coverage requirements for deployment. I don't believe we have any plans to change the metadata API deployment requirements at this time.

Thanks,
jhartjhart
Andrew,

This is happening to another customer right now; this makes 3 in the last week.

I am happy to provide you with the org IDs of the customers effected, but asking them to log a bug doesn't make a lot of sense to me.

From their perspective:

(a) We provide an automated  page-layout-editor
(b) it doesn't work

If we then say "oh, this is a salesforce bug, please log a case with them" I believe their natural response would be "why do I have to go through all this extra work for something I know nothing about?"

If we convince them to do it, we would have to write the bug for them and they would just be cutting-and-pasting.  Then, once the case was opened, I suspect T1 support would immediately close the case with a comment "we don't support partner apps, please contact the partner".

Adding our customers as an extra layer of communication between us and salesforce would seem to only complicate matters.

Is there an internal bookkeeping reason why you need a case from the customer org?

thanks,
john
IC-TannerIC-Tanner
Hi everyone, I'm running to this same issue. My client has a managed package installed, "Absolute Automation (Version 2.15)". Initially when this package was installed, I was still able to deploy apex classes and triggers. But with the Winter '09 release, I am now receiving the same type of error in which I cannot deploy new apex classes and triggers because code coverage for unrelated classes is being required. The issue is that this code coverage is required for classes that are part of the managed packaged referenced above "Absolute Automation (Version 2.15)".  This is a big issue as I cannot deploy any classes/triggers at the moment and it seems my only alternative is having to uninstall the "Absolute Automation (Version 2.15)" package everytime I need to deploy a new apex class/trigger. Then re-install this managed package. Here are the screen shots (second screen shot shows that the class that coverage is being required on is part of a managed package):
 
jhartjhart
Wow, funny coincidence - I work for iHance, and that is our package.

The package in sum has >75% coverage (thus it is packageable and installable in the first place!).  Individual classes within the package range in their coverage percentage, or course.

Salesforce's code-coverage test seems to randomly pull in other classes, whether they be packaged or no, and then apply those statistics to the overall test.

See also this post for the exact same problem, but this time the "problem" package is salesforce's own License Management App.
IC-TannerIC-Tanner

I was hoping that this issue would no longer occur with the release of IDE 14.0, but I'm still having the same issues as before (I have installed the IDE 14.0 with Eclipse). I filed case number 02221826 today regarding this. I really hope this can be resolved soon as it is a major issue as I'm more or less locked out from doing anything for my client that requires the use of the IDE (Eclipse, Ant, etc.)
jhartjhart

Tanner,

We use the ANT tool and get the same bug - it's not a client/tool issue, it's a packaging issue on the server end.  The downside is that updates to Eclipse won't make any difference, but the upside is that when Salesforce does fix it you won't have to install a new Eclipse version.

-j
jhartjhart
Tanner,

One workaround I figured out to deploy unmanaged code was to add bogus methods that are very, very long, and then test them. For example, a single method called "workingAroundSalesforceCodeCoverageBug" which consists of "i++" repeated, say, ten thousand times, and then a testMethod which calls that method.

Keep adding lines until your coverage is high enough. Super ugly, but it's the only way to workaround this Salesforce bug.  I have no doubt that Salesforce would like to scold me and slap my wrist for even mentioning this, but we gotta do what we gotta do.
jhartjhart

Salesforce - please note that this issue is still ongoing.  A related issue was fixed a few months ago, but we're still seeing this issue at a number of customer locations.

 

All we're trying to do is use the metadata API to edit page layouts so our users don't have to edit layouts by hand.  It makes no sense that code coverage even comes up during a simple update to existing page layouts.

 

If salesforce provided their own way for managed packages to automatically update page layouts, or provided their own wizard for editing multiple page layouts at once, then this wouldn't be necessary.  Since neither of those things exist, we (the partner community) are left to invent this functionality on our own in order to prevent our install process from taking 20 minutes of page layout updating.  To then be stymied by utterly unrelated code coverage issues is quite frustrating.