Saturday 6 March 2010

Google Appengine upload_data to restore keys via bulkloader is broken (Using local SDK)

I've logged an issue. 2925

I've been trying to save and restore my dataset including keys using the bulkloader as docuemnted. There are a few undocumented things that are not mentioned and quite important. First is how to save your model keys. If we had the following model

class TestModel(BaseModel):
name = db.StringProperty(required=True)


Then the following is defined to export the model

class TestModelExporter(bulkloader.Exporter):
def __init__(self):
bulkloader.Exporter.__init__(self, 'TestModel',
[('__key__', str, None),
('name', str, None),
])

Notice the '__key__' Field. This saves the key in the csv for that model.
The following should load the model back in with the key, updating it if the model already exists.

class TestModelLoader(bulkloader.Loader):
def __init__(self):
bulkloader.Loader.__init__(self, 'TestModel',
[('key', lambda x: x.decode('utf-8')),
('name', lambda x: x.decode('utf-8')),
])

Notice the special field key. This is Used in the Model.__init__ constructor in file "google/appengine/ext/db/__init__.py" Line 655 to recreate the Key from the string. This is created in bulkloader.py line 1260.

This key is then used when _populate_entity "google/appengine/ext/db/__init__.py" is called from bulkload at line 1264

_populate_entity then fills out the 'id' field with using the id() call on the key. The Issue with this is that key.id() returns a long

The Entity is then Created but line 422 of file "google/appengine/api/datastore.py" checks to see if the id is an int but it's a long so the creation fails. Can the id be a long?

My solution is to cast the return from "self._key.id()" at "google/appengine/ext/db/__init__.py" line 823 to an int.

This then fixs stuff nicly.

No comments:

Post a Comment