function readOnly(count){ }
Starting November 20, the site will be set to read-only. On December 4, 2023,
forum discussions will move to the Trailblazer Community.
+ Start a Discussion
Tom Coffey 4Tom Coffey 4 

How to Add the Same Custom Sample Data Set - with Relationships - to Many Environments?

I'm working on an OEM solution.  I have created Dev and Test enviroments via Environment Hub.  I will probably create a couple more Dev and Test environments as I progress.

I've begun to build a custom - my solution has many custom objects - sample dataset that I'd like to be able to easily import into my various environments.

Just uploaded my first object from my custom data set and realized I don't know how I'm going to handle the relationships.

My concern is that I won't be able to keep record IDs consistent across environments and therefore I won't be able to programatically establish relationships.

Any suggestions on how to best address this?
Best Answer chosen by Tom Coffey 4
Alain CabonAlain Cabon
@Tom Coffey

To keep record IDs consistent across environments the common solution is to create new External Ids fields for all the objects that will contains the former Ids. Moreover, you will have to load the data according the order "parent first" ( ie; loading Account before Contact before Event ).

That is not the panacea because this trick doesn't work for the polymorphic fields like WhoId or WhatId for the events for example.
https://success.salesforce.com/ideaview?id=08730000000Br0QAAS

1) 7 Steps to Recovering Lost Data Using Salesforce Weekly Export: Step 4 (here)
https://pages.ownbackup.com/salesforce-weekly-export

2) Create parent child relationship using External Id and Upsert Operation in Salesforce Dataloader
https://www.youtube.com/watch?v=88xb62VwQaU

The copy of data between orgs based on csv exported files is very difficult (euphemism) if you have already data around the events, tasks, leads, opportunity and products that you want to reimport.

SALESFORCE: FAQs on Data Recovery service and cost:

1) What does this process cost?

Because of the manual intervention, there is a cost. The cost is relative to the amount of manual work and time needed to perform the recovery. The price for this service is a flat rate of $US 10,000 (Ten Thousand US Dollars) for the one Organization that's being recovered. The work involved actually costs us much more than that, but we pay for a portion of the service.

2) How long does it take?
The process must be performed manually and usually takes a minimum of 6 - 8 weeks to complete.

... Data Recovery between orgs is always a nightmare if you have a lot of used objects ... including for Salesforce (no tools, no automatical process, all is manual because each orgs has its own features).

For me, that is main weak point of the Salesforce governance (disastrous, almost fatal). There are the full copy (prod > sandbox) and the copy between sandboxes that are possible now but the import of data from csv files is almost impossible without days or weeks of preparation.
 

All Answers

Alain CabonAlain Cabon
@Tom Coffey

To keep record IDs consistent across environments the common solution is to create new External Ids fields for all the objects that will contains the former Ids. Moreover, you will have to load the data according the order "parent first" ( ie; loading Account before Contact before Event ).

That is not the panacea because this trick doesn't work for the polymorphic fields like WhoId or WhatId for the events for example.
https://success.salesforce.com/ideaview?id=08730000000Br0QAAS

1) 7 Steps to Recovering Lost Data Using Salesforce Weekly Export: Step 4 (here)
https://pages.ownbackup.com/salesforce-weekly-export

2) Create parent child relationship using External Id and Upsert Operation in Salesforce Dataloader
https://www.youtube.com/watch?v=88xb62VwQaU

The copy of data between orgs based on csv exported files is very difficult (euphemism) if you have already data around the events, tasks, leads, opportunity and products that you want to reimport.

SALESFORCE: FAQs on Data Recovery service and cost:

1) What does this process cost?

Because of the manual intervention, there is a cost. The cost is relative to the amount of manual work and time needed to perform the recovery. The price for this service is a flat rate of $US 10,000 (Ten Thousand US Dollars) for the one Organization that's being recovered. The work involved actually costs us much more than that, but we pay for a portion of the service.

2) How long does it take?
The process must be performed manually and usually takes a minimum of 6 - 8 weeks to complete.

... Data Recovery between orgs is always a nightmare if you have a lot of used objects ... including for Salesforce (no tools, no automatical process, all is manual because each orgs has its own features).

For me, that is main weak point of the Salesforce governance (disastrous, almost fatal). There are the full copy (prod > sandbox) and the copy between sandboxes that are possible now but the import of data from csv files is almost impossible without days or weeks of preparation.
 
This was selected as the best answer
Alain CabonAlain Cabon
To keep record IDs consistent across environments the common solution is to create new External Ids fields for all the objects that will contain the former Ids. Moreover, you will have to load the data according the order "parent first" ( ie; loading Account before Contact before Event ).

For me, that is the main weak point of the Salesforce governance.

As soon as you have a first "reference" sandbox after a laborious initialization, you can use it for the copy "sandbox to sandbox".

[sorry for my broken english]
Alain CabonAlain Cabon
There is an "embryo" of export/import commands of data with relationships by using SFDX but it is always based on queries.
  • sfdx force:data:tree:export
  • sfdx force:data:tree:import

sfdx force:data:tree:export -p -q "select id,name, (select name from contacts),(select subject from events) from account" -u DevHub
Wrote 12 records to Accounts.json
Wrote 20 records to Contacts.json
Wrote 1 records to Events.json
Wrote 0 records to Account-Contact-Event-plan.json

1) Accounts.json
{
    "records": [
        {
            "attributes": {
                "type": "Account",
                "referenceId": "AccountRef1"
            },
            "Name": "GenePoint"
        },
        {
            "attributes": {
                "type": "Account",
                "referenceId": "AccountRef2"
            },
            "Name": "United Oil & Gas, UK"
        },
...
}

2) Event.json
{
    "records": [
        {
            "attributes": {
                "type": "Event",
                "referenceId": "EventRef1"
            },
            "Subject": "Meeting",
            "WhatId": "@AccountRef2"
        }
    ]
}

3) Account-Contact-Event-plan.json : plan of import (well generated)
 
{
        "sobject": "Account",
        "saveRefs": true,
        "resolveRefs": false,
        "files": [
            "Accounts.json"
        ]
    },
    {
        "sobject": "Contact",
        "saveRefs": false,
        "resolveRefs": true,
        "files": [
            "Contacts.json"
        ]
    },
    {
        "sobject": "Event",
        "saveRefs": false,
        "resolveRefs": true,
        "files": [
            "Events.json"
        ]
    }
]

This sample is very simple but it is an "automatic" generation of exported data that can be reimported with the relationships

WhatId has never been written in the SOQL query for the events but it is well generated.

select id,name, (select name from contacts),(select subject from events) from account
Tom Coffey 4Tom Coffey 4
Alain, thanks for the input.  Fortunately I'm not dealing with events, tasks, leads, opportunity and products objects.  And no WhoID or WhatID issues.

I did run into the "parent" issue when trying to create Parent Account account relationships on the Account object.  But that was easily fixed by doing the Account object Upsert in two steps: once without this field mapped and then again with it mapped.

Also had to uncheck a restrict picklist values checkbox (even though I was only upserting values contained in the picklist).  But otherwise the External ID / Upsert technique is working perfect for me.  

Thanks again for your help!