Blog

Thomas Wanschik on January 12, 2010

Native Django on App Engine

Update 3: This post is outdated. Please read the djangoappengine documentation.

Update 2: Please read the djangoappengine documentation instead of this post for detailed installation instructions. The "common-apps" folder has been deprecated. Just put all dependencies directly into your project folder. The "djangotoolbox" repository folder structure has changed a little bit. The link to "djangotoolbox" should now point to the "djangotoolbox/djangotoolbox" subfolder instead of the repository itself. Also, the App Engine backend now supports IN and != queries and DecimalField.

Update: We've updated the installation instructions. You now also need djangotoolbox. Additionally, a simplified version of this post has been added to the documentation section of djangoappengine.

About a few months ago we started to port Django to support non-relational databases and to implement an App Engine database backend for this port. So far we ended up supporting basic Django functionality on App Engine. This post is intended to get you started using our port and to let you know what you can do and want you can't do with it at the moment. So let's start!

Installation

In order to use our port on App Engine you have to clone a few repositories first. These are the django-nonrel repository, djangoappengine and the django-testapp. So what are all these repos for? First in order to let you start a new Django project as fast as possible we created the django-testapp repo. It basically consists of a modified manage.py file to support the App Engine development server, and all corresponding settings in order to specify App Engine as the backend for Django. So the first step is to clone the django-testapp repo.

The djangoappengine repo contains all App Engine related backends for the non-relational Django port e.g. the email backend and the query backend for Django. So as the next step clone the djangoappengine package and link it into the django-testapp folder. Clone the djangotoolbox package and link the "djangotoolbox" subfolder within that repository into the django-testapp folder. The django-nonrel repo is our modified Django port. It contains changes in Django itself to support non-relational databases. Clone it and link the repository's "django" folder into the django-testapp folder. Your folder structure should now look similar to this:

  • .../django-nonrel/django
  • .../djangoappengine
  • .../djangotoolbox
  • .../django-testapp
  • .../django-testapp/djangoappengine -> ../djangoappengine
  • .../django-testapp/djangotoolbox -> ../djangotoolbox/djangotoolbox
  • .../django-testapp/django -> ../django-nonrel/django

Now you should be able to create a new app in the testapp folder and use native Django models on App Engine. Try it!

Supported and unsupported features for the App Engine backend

Field types

First we added support for most Django field types but not all. Here is the list of Django field types which currently are not supported (but there are chances that a few of them will get ported in the near future):

  • FileField
  • FilePathField
  • OneToOneField
  • ManyToManyField
  • ImageField

All other ones are supported fully. In addition we support all Django field options except for options which can't be used on App Engine. These are unique, unique_for_date, unique_for_month and unique_for_year.

Additionally, there are a few App Engine properties which haven't been ported to Django:

  • ByteStringProperty
  • blobstore.BlobReferenceProperty
  • UserProperty
  • GeoPtProperty
  • RatingProperty

For many-to-one relationships you can use a ForeignKey, but internally the ForeignKey does not store an App Engine db.Key value. The lack of a GAEKeyField in Django storing db.Keys makes it impossible to create entity groups with the current App Engine backend.

The following App Engine properties can be emulated by using a CharField in Django:

  • CategoryProperty
  • LinkProperty
  • EmailProperty
  • IMProperty
  • PhoneNumberProperty
  • PostalAddressProperty

QuerySet methods

We do not support any filters that are not supported by App Engine itself. This means that you can't use all field lookup types from Django. Nevertheless you can use:

  • __exact equal to (the default)
  • __in
  • __lt less than
  • __lte less than or equal to
  • __gt greater than
  • __gte greater than or equal to
  • __isnull

You can even use the exclude function in combination with the listed filters above. In addition you can use Q-objects to formulate more complex queries. Slicing is possible too. But in all cases you have to keep App Engine restrictions in mind. While you can perform a filter like

Model.objects.exclude(Q(integer_field__lt=5) | Q(integer_field__gte=9))

you can't do

Model.objects.filter(Q(integer_field__lt=5) | Q(integer_field__gte=9))

The reason for this is that the first filter can be translated into an AND-filter without using a logical OR but the second one cannot. Inequality filters are not supported for now. But many additional non return sets like count(), reverse(), ... do work. Maybe you are wondering how to set a keyname for your model instance in Django. Just set the primary key of your entity and it will be used as the keyname.

model_instance.pk = u'keyname'
model_instance.save()

but remember to set the primary_key field option of the CharField to True.

Many Django features are not supported at the moment. A few of them are joins, many-to-many relations, multi table inheritance and aggregates. On the other hand there are App Engine features not supported by the Django port mainly because of the lack of corresponding features in Django. Two of them are entity groups and batch put. But we plan to add some features for both sides while working on our own project. As the last point we do not support any Django transaction for the App Engine backend but we implemented a custom transaction decorator @commit_locked.

Differences in Django itself

While changing the Django code itself we had to hack some parts in. Django should behave as described in the Django documentation in all situations except for one: when deleting a model instance the related objects will not be deleted. This had to be done because such a deletion process can take too much time. In order to support deletion of related objects for non-relational Django in a clean manner Django should be modified to allow the backend to handle such a deletion process. For App Engine it can be done using background tasks.

Advantages using native Django

So maybe you are wondering why to use our current Django port instead of other projects providing you Django features on App Engine like our last project app-engine-patch. There are a few reasons for that. First of all and maybe the most important advantage of using native Django is that you will avoid the vendor lock-in problem. I am not saying that you should not use App Engine but you can find yourself in a situation where App Engine is not sufficient enough for your needs. Using native Django code will allow you to switch to another hosting/cloud provider with having minimal costs related to porting the existent code to the new database. Second using native Django will result in almost full reusability of existing Django apps and makes non-relational Django apps fully portable to any platform (including SQL). Maybe these apps have to be modified a little bit but changing these apps is a lot easier than to port an App Engine project to Django. In addition the model layer gives you many new features like select_related() which are not supported by other Django projects on App Engine. Another example is while using App Engine models we found it annoying to pass required fields to the model constructor. In Django you do not have to set required field types when creating a model instance. Only when saving the entity such fields have to be valid. So these are only a few reasons.

If you want to see more source code about all these things mentioned in this post take a look at our unit tests in djangoappengine. There you can find model definitions and queries such that you can get a better idea of how to use the App Engine backend for the non-relational Django port.

Help

A few days ago we cleaned up the non-relational django port itself. Now it should be easy to write a backend without modifying the django port. If you want to help improving our non-relational Django port, you can check out the wiki and the tasks list. We are happy about any other help too. You can help by testing the port, reaching out to other communities like MongoDB and SimpleDB, building a greater community around this project and also attracting developers who can contribute code.

While this post is mostly about how to use our port on App Engine it's possible to implement support for other non-relational databases using our non-relational Django port too. Again if you are planning to help, take a look at the djangoappengine repo to get an idea of how to implement your own non-relational database backend for Django and join the django non-relational group.

In order to help you write non-relational Django code we plan to write posts about how to write SQL-independent Django code and how to port existing app-engine-patch/webapp/Django apps to the new non-relational port.

So my first post got a little bit longer than i thought but i hope you enjoyed to read it. We'd love to hear your comments.