function readOnly(count){ }
Sign Up ›
Starting November 20, the site will be set to read-only. On December 4, 2023,
forum discussions will move to the Trailblazer Community.
Learn More

Browse by Topic
ShowAll Questionssorted byDate Posted
+ Start a Discussion
Sagar Hinsu 12Sagar Hinsu 12 

How to use Bulk Api in python

I know there are some docs where its explained that but.
I have a csv for an custom object with 10000 records.
I want to upload this data to my org using bulk api.
I know there are tools are dataloader.io and apex data loader. but i want it as custom tool in python.
Thanks.
Best Answer chosen by Sagar Hinsu 12
pconpcon
As I steated before you will need to create the dictionary yourself.  Below is the Python code that will create the dictionary
  
import csv

from salesforce_bulk import SalesforceBulk
from salesforce_bulk import CsvDictsAdapter

bulk = SalesforceBulk(username=username, password=password)

job = bulk.create_insert_job("user__c", contentType='CSV', concurrency='Parallel')

reader = csv.DictReader(open('user__c.csv'))
disbursals = []
for row in reader:
    disbursals.append(row)

csv_iter = CsvDictsAdapter(iter(disbursals))
batch = bulk.post_bulk_batch(job, csv_iter)
bulk.wait_for_batch(job, batch)
bulk.close_job(job)

print("Done. Data Uploaded.")

 

All Answers

pconpcon
Are you using any specific python library to connect to Salesforce or are you planning on rolling your own?
Sagar Hinsu 11Sagar Hinsu 11
yes i am using salesforce_bulk package. 
bulk = SalesforceBulk(username=username, password=password)
        
        job = bulk.create_insert_job("custom_object__c", contentType='CSV', concurrency='Parallel')
        
        disbursals = [dict(Name="custoom_object__c%d" % idx) for idx in xrange(5)]
        
        csv_iter = CsvDictsAdapter(iter(disbursals))    
        
        batch = bulk.post_bulk_batch(job, csv_iter)    
        
        bulk.wait_for_batch(job, batch)    
        
        bulk.close_job(job)
            
        print("Done. Data Uploaded.")

i want to upload through a csv. my file is custom_object__c.csv file.
pconpcon
You'll need to create the dictionary yourself by reading your CSV file from disk.
Sagar Hinsu 11Sagar Hinsu 11
I have an object name user__c in that i have 3 fields. [first_name__c,last_name__c,user_id__c]
I have a csv name user__c.csv.
user__c.csv has data like
first_name__c,last_name__c,user_id__c
a,b,1
c,d,2
e,f,3
i,j,4 etc etc(many rows like 20000)

now can you give small example for uploading using bulk api.
Thank You.
 
pconpcon
As I steated before you will need to create the dictionary yourself.  Below is the Python code that will create the dictionary
  
import csv

from salesforce_bulk import SalesforceBulk
from salesforce_bulk import CsvDictsAdapter

bulk = SalesforceBulk(username=username, password=password)

job = bulk.create_insert_job("user__c", contentType='CSV', concurrency='Parallel')

reader = csv.DictReader(open('user__c.csv'))
disbursals = []
for row in reader:
    disbursals.append(row)

csv_iter = CsvDictsAdapter(iter(disbursals))
batch = bulk.post_bulk_batch(job, csv_iter)
bulk.wait_for_batch(job, batch)
bulk.close_job(job)

print("Done. Data Uploaded.")

 
This was selected as the best answer
Sagar Hinsu 11Sagar Hinsu 11
Thank pcon... it worked... 
pconpcon
Great! If you could please choose a "best answer" so that this question can be removed from the unresolved queue and so others may easily find the answer.
Bharath NanduriBharath Nanduri
HI,

I am getting the following error when I am using this api:

RuntimeError: You must set SALESFORCE_CLIENT_ID, SALESFORCE_CLIENT_SECRET, SALESFORCE_REDIRECT_URI to use username/pass login

Any help on this is highly appreicated!
Deep Sukhwani 22Deep Sukhwani 22

u need to get the client id, client secret key and instance URL. 

https://developer.salesforce.com/forums/?id=906F0000000AfcgIAC
negannegan
Hi Sagar and pcon,

I also need to do a bulk update in our salesforce instance using python. Same scenario with you guys as stated above. 

I have a question though, where will i save the csv file that i'm going to use? what i can see above is only the csv filename. The path to the file was not mentioned. Please advise on this. This will greatly help me on the current project that i'm working. 

Hoping for your prompt response. Many thanks!
pconpcon
Negan,

The code above assumes that the csv file is in the same directory as the running code.  If you want to change that location then you will need to update the open call to have the path to your csv.  Or you can build the csv in memory and pass it to the DictReader.  Or you can build the dict in memory and use that directly.
Doga TuncyuzDoga Tuncyuz
I read on https://resources.docs.salesforce.com/sfdc/pdf/api_asynch.pdf that they added JSON format lately. I already have a JSON format list/dictionary and do you guys have a small example code for that? I did a good amount of research but couldn't find a JSON example. As far as I know, content-type changes to contentType='JSON' and I assumed in batch = bulk.post_bulk_batch(job, csv_iter), I'm just going to give my list/dictionary instead of csv_iter. Thanks in advance.
irfan azizirfan aziz
Hi
Does anyone have any idea how reliable is this salesforce_bulk library for production usage. Does it support data query to export 100mb data from SF. I have been using simple_salesforce which is easy to use however for large data something greater like 15mb the behavior is quite unpredictable. It may extract the data or it may fail.
Doga TuncyuzDoga Tuncyuz
I read a few different comments on salesforce_bulk about how unreliable it was. I used salesforce_bulkipy and worked perfectly. I don't know about data support for 100mb data though, what I'm doing is a data upload, not download.
irfan azizirfan aziz
Thank You Doga, I queried but it does not handly large size data efficiently. Its good for small datasets like simple_salesforce library.
Dhruv KadiaDhruv Kadia
How can i get a success and an error file after adding the new records in salesforce so that i can use the salesforce id of the new reocrds later in my code? 
Jass Singh 1Jass Singh 1
This is a very informative article and I'm sure it has helped many in the business. fore more update youcan visit on https://www.thetechlearn.com/
Stephen LoughinStephen Loughin

I am having a similar problem: 

bulk = SalesforceBulk(sessionId=self.access_token, host=urlparse(self.instance_url).hostname)
job = bulk.create_insert_job(object, contentType='CSV')
reader = csv.DictReader(open(csv_file))
items = []
for row in reader:
   items.append(row)
csv_iter = CsvDictsAdapter(iter(items))
batch = bulk.post_bulk_batch(job, csv_iter)
bulk.wait_for_batch(job, batch)
bulk.close_job(job)

The object I'm passing is 'Product2' and I can use the same SalesforceBulk object to successfully perform a bulk query on Product2 using bulk.create_query_job.  This code fails on create_insert_job with the following response:

[400] Bulk API HTTP Error: InvalidJob - No create access for object:Product2

Is this a permissions issue?  I am using an admin account.
Snehal Harshe 4Snehal Harshe 4
To access the Bulk API you need to authenticate a user into Salesforce. The easiest way to do this is just to supply username, password and security_token.
Click: Python Training in Pune (https://www.sevenmentor.com/best-python-classes-in-pune.php)