function readOnly(count){ }
Sign Up ›
Starting November 20, the site will be set to read-only. On December 4, 2023,
forum discussions will move to the Trailblazer Community.
Learn More

Browse by Topic
ShowAll Questionssorted byDate Posted
+ Start a Discussion
buzzbuzz 

Data Loader automation issues

Hi all,

 

I'm working on a data loader automation project and I'm running into 2 issues, first how do I configure my bean to load the file based on a naming convention, rather than an actual file name, the files I'm loading are daily results from our survey trucks, or ARANs, in the field and the naming convention is Daily_ARAN##_YYYYMMDD.csv, the example in the sample file shows:

 

<entry key="dataAccess.name" value="c:\dataloader\samples\data\opportunityData.csv"/>

 

But there must be a way to get it to get files that meet a naming convention.

 

The second problem is that we only want to load files placed on the server yesterday, since the process is to run just after midnight each day and and should only insert files that are new from the day before.  I've looked for how to handle these problems online but haven't seen any solutions, if anyone can help it woud be deeply appreciated.  Thanks.

 

Buzz

buzzbuzz

Does anyone know if the * delimiter will work in this context?
vriavmvriavm

In Data loader xml file you cannot have dynamic file name but you can design the whole program to handle it in your code:

I have implemented this in UNIX scripting before and everything works great.

 

1. Create below folders in your working directory:

         Processing

         Archive

 

First step in your script (VB scripting, bat file or shell scripting,..)

      1. Download the file (fina name ex: Accounts_2012_02_02.csv) from the source FTP/system to processing folder

       2. Rename it to fixed file name which you specified in your XML bean file (ex: Accounts_2012_02_02.csv to Accounts.csv)

      3. Call you data loader program which will pick the file prom processing directory and process it

      4. Now after processing in the same script move file (Account.csv) from Processing directory to Archive dirrectory and rename it to (Accounts_<today date>.csv or retain the initial file name in a variable and rename it back to that (Accounts_2012_02_02.csv)).

 

Hope this solves your file processing issue and also you can keep archive of files locally to refer in future.