Business:
I encountered a problem - when operating with large datasets with Django ORM, canonical way is manipulate with every single element. But of course this way is very inefficient. So I decided to use raw SQL.
Substance:
I have a basic code which forms SQL query, which updates rows of table, and commiting it:
from myapp import Model
from django.db import connection, transaction
COUNT = Model.objects.count()
MYDATA = produce_some_differentiated_data() #Creating individual value for each row
cursor = connection.cursor()
str = []
for i in xrange(1, COUNT):
str.append("UPDATE database.table\n"
"SET field_to_modify={}\n"
"WHERE primary_key_field={};\n".format(MYDATA, i))
str = ''.join(str)
cursor.execute(str)
transaction.commit_unless_managed() #This cause exception
And on last statement I get this, even when SIZE
is small:
_mysql_exceptions.ProgrammingError: (2014, "Commands out of sync; you can't run this command now")
Maybe Django do not allow execute multiple SQL queries at once?
ps Closing cursor before commiting helps to avoid exception, but is this correct?
My expectations:
Im looking for every possible solid solution for bulk operations (preferably inside Django).
I dont care about will it be ORM or raw SQL, I would have stand with code I pasted above, if I could avoid error. In case of no solutions it will be good at least, just for curiosity, to know reason of this exception.
What I have learned besides answers:
In Django 1.4 was introduced bulk_create
, for efficient multiple INSERT
operations