Apex Code Development You need to sign in to do that
Don't have an account?
Vinnie B Generic data integration question - taking .csv data from an external source and importing into SFDC
I've been tasked with grabbing some .csv files from an external location and importing that data into an SFDC instance on a nightly basis. There will be some required manipulation of the data involved prior to importing.
I'm thinking the best approach is to develop an Apex class that runs as a daily scheduled job. The Apex class will do an HTTP request to get the files from the external location, process the data as needed in the Apex code, and populate the appropriate fields.
I've done a fair amount of Apex coding and think I can do this. My question is regarding the general strategy of this. Is there a better way to do this? Are there any gotchas that I need to worry about in doing it this way?
Thanks!!
I'm thinking the best approach is to develop an Apex class that runs as a daily scheduled job. The Apex class will do an HTTP request to get the files from the external location, process the data as needed in the Apex code, and populate the appropriate fields.
I've done a fair amount of Apex coding and think I can do this. My question is regarding the general strategy of this. Is there a better way to do this? Are there any gotchas that I need to worry about in doing it this way?
Thanks!!
Your approach is one of the possible options. One thing that comes to my mind is that you will need to build your own logging mechanism for the records that have failed/errors occured.
Not sure how much you're limited on the other side of the fence but other options would be:
- Data loader (depending on to what extent you need to manipulate the data before loading)
- Middleware ETL sitting between SF and your external source
- Email web service (if the other system can email that .csv), though you'll be limited with the max file size and you need to build your own logging mechanism
- Apex Rest web service (if there is capability on the other end to authenticate and to send the file to SF), again, the file size is a limit here as well as logging
There is no right or wrong solution, you just need to be aware of all the limitations and also make your solution robust and scalable.
- Dell Boomi (https://boomi.com/)
- Informatica (https://www.informatica.com/)
- Skyvia (https://skyvia.com/)
- etc.