Uploads to Blobstore and GridFS with Django

We've finished our App Engine Blobstore and MongoDB GridFS storage backends for Django. With these backends you can directly use FileField and ModelForm to easily handle file uploads in a portable way. Please make sure you take a look at the source code of our demo.

The App Engine Blobstore backend is already integrated and pre-configured in djangoappengine. The GridFS backend is now part of django-storages. However, that's only the first half of the solution.

In addition to the storage backends we've also created a reusable Django app called django-filetransfers which provides a simple API for upload and download handling in your views. This API is an abstraction over the little details in the different file hosting services. For example, in a traditional Apache/Lighttpd/nginx setup you might want to efficiently serve files via your web server instead of Django by using the "X-Sendfile" extension. App Engine requires that you use a modified upload URL. Asynchronous uploads to Amazon S3 (i.e., the browser sends the file directly to S3 instead of piping it through your Django instance) require that you generate a custom upload URL and use additional POST data to authorize the upload. Each solution is different.

Instead of writing custom code for a specific hosting solution you can use django-filetransfers and specify a backend in your settings.py that handles the platform-specific details. This is particularly useful if you want to write a reusable Django app.

Note that we're also working on getting an API similar to django-filetransfers integrated into Django core. Once this is done we'll make a follow-up post with instructions on how to port your code.

How does it work?

Installation

You can install the package via setup.py install or by copying or linking the "filetransfers" folder to your project (App Engine developers have to use the copy/link method). Then, add "filetransfers" to your INSTALLED_APPS.

Note for App Engine users: In order to use the Blobstore on the App Engine production server you have to enable billing. Otherwise, the Blobstore API is disabled.

Model and form

In the following we'll use this model and form:

class UploadModel(models.Model):
    file = models.FileField(upload_to='uploads/%Y/%m/%d/%H/%M/%S/')

class UploadForm(forms.ModelForm):
    class Meta:
        model = UploadModel

The upload_to parameter for FileField defines the target folder for file uploads (here, we add the date).

Handling uploads

File uploads are handled with the prepare_upload() function which takes the request and the URL of the upload view and returns a tuple with a generated upload URL and extra POST data for the upload. This is an example upload view:

from filetransfers.api import prepare_upload

def upload_handler(request):
    view_url = reverse('upload.views.upload_handler')
    if request.method == 'POST':
        form = UploadForm(request.POST, request.FILES)
        form.save()
        return HttpResponseRedirect(view_url)

    upload_url, upload_data = prepare_upload(request, view_url)
    form = UploadForm()
    return direct_to_template(request, 'upload/upload.html',
        {'form': form, 'upload_url': upload_url, 'upload_data': upload_data})

Note that it's important that you send a redirect after an upload. Otherwise, some file hosting services won't work correctly.

Now, you have to use the generated upload URL and the upload's extra POST data in the template:

{% load filetransfers %}
<form action="{{ upload_url }}" method="POST" enctype="multipart/form-data">
  {% csrf_token %}
  {% render_upload_data upload_data %}
  <table>{{ form }}</table>
  <input type="submit" value="Upload" />
</form>

Here we use the {% render_upload_data %} tag which generates <input type="hidden" /> fields for the extra POST data.

Handling downloads

The serve_file() function primarily takes care of private file downloads for which you need to check permissions before the download starts, but in some configurations it might also have to take care of public downloads because the file hosting solution doesn't provide publicly accessible URLs (e.g., App Engine Blobstore). This means that you should also use that function as a fallback even if you only have public downloads. The function takes two required arguments: the request and the Django File object that should be served (e.g. from FileField):

from filetransfers.api import serve_file

def download_handler(request, pk):
    upload = get_object_or_404(UploadModel, pk=pk)
    return serve_file(request, upload.file)

The public_download_url function, which is also available as a template filter, returns a file's publicly accessible URL if that's supported by the backend. Otherwise it returns None.

Important: Use public_download_url only for files that should be publicly accessible. Otherwise you should only use serve_file(), so you can check permissions before approving the download.

A complete solution for public downloads which falls back to serve_file() would look like this in a template for an instance of UploadModel called upload:

{% load filetransfers %}
{% url upload.views.download_handler pk=upload.pk as fallback_url %}
<a href="{% firstof upload.file|public_download_url fallback_url %}">Download</a>

The second line stores the serve_file() fallback URL in a variable. In the third line we then use the public_download_url template filter in order to get the file's publicly accessible URL. If that returns None the {% firstof %} template tag returns the second argument which is our fallback URL. Otherwise the public download URL is used.

As you can see, it's a rather simple solution. You should read the documentation for more details about the available configuration options, permissions, and backends. Also, don't forget to check out the demo source code.

What do you think?

Please provide some feedback on the API. Do you know some file hosting service where it wouldn't work? Do you have improvement suggestions? Please comment, so we can take improvements into account when adding a file transfer API to Django core.

Also, if you have a favorite Django app which could benefit from django-filetransfers please let the app's author know or maybe even port it to django-filetransfers and send him a patch.