+ Start a Discussion
Andrew WeinsteinAndrew Weinstein 

sfdx force:source:push fails with lots (thousands) of errors

we're trying to start using salesforce DX with our existing managed packages. i started with a small package - no custom objects, a few custom fields, one custom setting, a few apex classes, one VF page, and one VF component - and everything went pretty smoothly. i was able to convert the metadata, set up a scratch org, push/pull source, etc.

when i tried this with our main managed package, which is much larger, i ran into all kinds of errors trying to do force:source:push. the pretty-printed JSON output of the errors from the CLI command is almost 33,000 lines long. a few of the errors are known issues (e.g. having history tracking enabled), but the rest are things like "referenceTo value of 'Custom_Object__c' does not resolve to a valid sObject type," when "Custom_Object__c" is one of the custom objects included in the source i'm trying to push. there are a bunch of similar things like "Field Custom_Field__c does not exist," "In field: field - no CustomField named Custom_Object__c.Custom_Field__c found," "Invalid type: Custom_Object__c," etc.

i don't know how to determine the source of all of these errors. i've tried most of the things i've found on here already, like enabling external sharing, pushing with --ignorewarnings, etc., and none of it has made any difference.

FWIW, i've also noticed that the metadata i get from force:mdapi:retrieve is missing certain things, like view all permissions on Account in one of our custom profiles. this generates a few errors, but the vast majority remain even after fixing that. maybe that's an entirely separate issue, but it seems worth mentioning.

has anyone else encountered problems like this? any suggestions about where i should look to find the source?
Andrew WeinsteinAndrew Weinstein
i should also note that i've tried using force:mdapi:deploy to the scratch org with the pre-converted metadata, and while the list of errors from that is much shorter, it's the same kinds of things.
Josh Werkmeister 1Josh Werkmeister 1
We've had a very similar experince with DX over the last couple months.. curious if you were ever able to find a solution? We found more success using sandboxes rather than scratch orgs, but were never able to figure out exactly what was causing these misleading errors. 
Andrew WeinsteinAndrew Weinstein
Hi Josh - yes, kind of. I had a few helpful responses through the SF partner community, and I got the rest of the way through trial and error. I was able to convert the metadata and we've started using DX regularly. Here's a list of the things I encountered and what I did to fix them, along with a few other things I think are worth mentioning:
  • When you're preparing to convert the source, don't just use your existing repo or use ant retrieveCode. use sfdx force:mdapi:retrieve and specify a package name, like "sfdx force:mdapi:retrieve -r ./targetDir -p "Package Name" -u PackageOrgAlias. This will pull back the exact metadata that is currently in your package, so you can be confident you're starting with the right stuff.
    • NOTE: I had to make one manual modification to this. One of our profiles has read all permissions on Accounts and Contacts, and force:mdapi:retrieve did not correctly add that permission to the profile metadata, so I had to do it myself. That was one source of push errors.
  • Generally, the first error (or something near the top) is the root cause, and there's a subsequent cascade that can result in thousands of error messages, when really there's just one thing to fix. Unfortunately, the original error is sometimes pretty opaque.
  • History tracking: if you have history tracking enabled for custom fields on standard objects, force:mdapi:retrieve will not correctly generate metadata files to enable history tracking at the object level. It will only give you the field-level metadata files, but you need an object-level file with the <enableHistory>true</enableHistory> tag. I believe this is a SF bug, I have a support case open about it, and their R&D team is looking into it.
  • If you have the Salesforce to Salesforce feature enabled in your packaging org, you'll need to turn it on in your scratch orgs. Unfortunately, there is no way to access this feature through any API, or even through the Lightning UI; you'll need to open the scratch org, switch to classic, enable Salesforce to Salesforce, and then push your source. I don't even think we use this feature, but it was enabled for some reason and can't be disabled, so we're stuck. It's very annoying, and it's the only remaining scratch org setup step that I haven't been able to script.
  • Record types: when I started this process, there was a known SF issue (https://success.salesforce.com/issues_view?id=a1p3A000000ATmUQAW&title=error-deploying-compact-layouts-in-field-recordtypeid-no-customfield-named-account-recordtypeid-found) that meant you had to manually create at least one record type on the Account object before you could push source. SF fixed this in Spring 19, so it's no longer a problem.
  • There are a few other quirks of scratch orgs that required some code changes. For example, the method System.requestVersion() is supposed to work in namespaced orgs/package context, but it does not work in namespaced scratch orgs, and it throws an exception that cannot be rescued. There was one place in our code where we were using that, so we had to find a workaround, which was checking UserInfo.isCurrentUserLicensed('namespace') (which can be rescued) before even attempting the System.requestVersion() check.
So basically, there were a bunch of Salesforce bugs, and a lot of the error messages were hard to diagnose, but there wasn't really anything wrong with our source. It's been pretty smooth sailing since I worked through all of that stuff.

Hope some of that helps!
Josh Werkmeister 1Josh Werkmeister 1
Thanks for the reply, this is very helpful! Unfortunate that there are that many bugs with the tools, but glad to hear you were able to work through them and find a solution