Flapping Head

For a while I’ve been using the CustomImageField as a way to specify an upload path for images. It’s a hack that lets you use ids or slugs from your models in the upload path, e.g.:

/path/to/media/photos/1234/flowers.jpg
or
/path/to/media/photos/scotland-trip/castle.jpg

CustomImageField no more

Since the FileStorageRefactor was merged in to trunk r8244, it’s no longer necessary to use the custom field. Other recent changes to trunk mean that it doesn’t work any more in its current state, so this is a good time to retire it.

Pass a callable in upload_to

It is now possible for the upload_to parameter of the FileField or ImageField to be a callable, instead of a string. The callable is passed the current model instance and uploaded file name and must return a path. That sounds ideal.

Here’s an example:

from django.db import models

def get_image_path(instance, filename):
    return 'photos/%s/%s' % (instance.id, filename)

class Photo(models.Model):
    image = models.ImageField(upload_to=get_image_path)

get_image_path is the callable (in this case a function). It simply gets the id from the instance of Photo and uses that in the upload path. Images will be uploaded to paths like:

photos/1/kitty.jpg

You can use whatever fields are in the instance (slugs, etc), or fields in related models. For example, if Photo models are associated with an Album model, the upload path for a Photo could include the Album slug.

Note that if you are using the id, you need to make sure the model instance was saved before you upload the file. Otherwise, the id hasn’t been set at that point and can’t be used.

For reference, here’s what the main part of the view might look like:

...
    if request.method == 'POST':
        form = PhotoForm(request.POST, request.FILES)
        if form.is_valid():
            photo = Photo.objects.create()
            image_file = request.FILES['image']
            photo.image.save(image_file.name, image_file)
...

This is much simpler than the hacks used in CustomImageField and provides a nice flexible way to specify file or image upload paths per-model instance.

Extra fields for Users

Most of the Django projects I’ve worked on need to store information about each user in addition to the standard name and email address held by the contrib.auth.models.User model.

The old way: User Profiles

The solution in the past was to create a “user profile” model which is associated 1-to-1 with the user. Something like:

the model

class UserProfile(models.Model):
    user = models.ForeignKey(User, unique=True, related_name='profile')
    timezone = models.CharField(max_length=50, default='Europe/London')

config in settings.py

AUTH_PROFILE_MODULE = 'accounts.UserProfile'

usage

profile = request.user.get_profile()
print profile.timezone

It works ok, but it’s an extra database query for each request that uses the profile (it’s cached during the request so each call to get_profile() is not a query). Also, the information about the user is stored in two separate models, so you need to display and update fields from both the User and the UserProfile models.

The new way: Model Inheritance

As part of the great work done on the queryset-refactor by Malcolm et al, Django now has model inheritance.

If you’re using trunk as of revision 7477 (26th April 2008), your model classes can inherit from an existing model class. Additional fields are stored in a separate table which is linked to the table of the base model. When you retrieve your model, the query uses a join to get the fields from it and the base model.

Inheriting from User

Instead of creating a User Profile class, why don’t we inherit from the normal User class and add some fields?

from django.contrib.auth.models import User, UserManager

class CustomUser(User):
    """User with app settings."""
    timezone = models.CharField(max_length=50, default='Europe/London')

    # Use UserManager to get the create_user method, etc.
    objects = UserManager()

Now each instance of CustomUser will have the normal User fields and methods, as well as our additional fields and methods. Pretty handy, no?

We add UserManager as the default manager so that the standard methods are available. In particular, to create a user, we really want to say:

user = CustomUser.objects.create(...)

If we just created the user from the User class, we wouldn’t get a row in the CustomUser table. Creation needs to be done in the derived class.

You can still get and update the underlying User model, no problem, but it won’t have the additional fields and methods found in our CustomUser class.

Getting the CustomUser class by default

So far, there’s one problem. When you get request.user, it’s an instance of User, not an instance of CustomUser, so you don’t get the extra fields and methods.

What we want is for Django to retrieve the CustomUser instance transparently. It turns out to be quite easy.

Users come from authentication backends

The default authentication backend gets the User model from the database, checks the password is correct then returns the User. You can write your own authentication backend, for example to check the username and password against some other data source or to use the email address instead of username.

In our case, we can use an authentication backend to return an instance of CustomUser instead of User.

the authentication backend in auth_backends.py

from django.conf import settings
from django.contrib.auth.backends import ModelBackend
from django.core.exceptions import ImproperlyConfigured
from django.db.models import get_model

class CustomUserModelBackend(ModelBackend):
    def authenticate(self, username=None, password=None):
        try:
            user = self.user_class.objects.get(username=username)
            if user.check_password(password):
                return user
        except self.user_class.DoesNotExist:
            return None

    def get_user(self, user_id):
        try:
            return self.user_class.objects.get(pk=user_id)
        except self.user_class.DoesNotExist:
            return None

    @property
    def user_class(self):
        if not hasattr(self, '_user_class'):
            self._user_class = get_model(*settings.CUSTOM_USER_MODEL.split('.', 2))
            if not self._user_class:
                raise ImproperlyConfigured('Could not get custom user model')
        return self._user_class

config in settings.py

AUTHENTICATION_BACKENDS = (
    'myproject.auth_backends.CustomUserModelBackend',
)
...

CUSTOM_USER_MODEL = 'accounts.CustomUser'

That’s it. Now when you get request.user, it’s an instance of the CustomUser class with whatever additional fields or methods you have added.

P.S. Looking for Django hosting? I’d recommend WebFaction for shared hosting and Slicehost for a VPS.

About a week ago Andrew and I launched a new Django-powered site called Hey! Wall. It’s a social site along the lines of “the wall” on social networks and gives groups of friends a place to leave messages, share photos, videos and links.

We wanted to gauge performance and try some server config and code changes to see what steps we could take to improve it. We tested using httperf and doubled performance by making some optimisations.

Server and Client

The server is a Xen VPS from Slicehost with 256MB RAM running Debian Etch. It is located in the US Midwest.

For testing, the client is a Xen VPS from Xtraordinary Hosting, located in the UK. Our normal Internet access is via ADSL which makes it difficult to make enough requests to the server. Using a well-connected VPS as the client means we can really hammer the server.

Server spec caveats

It’s hard to say exactly what the server specs are. The VPS has 256MB RAM and is hosted with similar VPSes, probably on a quad core server with 16GB RAM. That’s a maximum of 64 VPSes on the physical server, assuming it is full of 256MB slices. If the four processors are 2.4GHz, that’s 9.6GHz total, divided by 64 gives a minimum of 150MHz of CPU.

On a Xen VPS, you get a fixed allocation of memory and CPU without contention, but usually any available CPU on the machine can be used. If other VPSes on the same box are idle, your VPS can make use of more of the CPU. This probably means more CPU was used during testing and perhaps more for some tests than for others.

Measuring performance with httperf

There are various web performance testing tools around including ab (from Apache), Flood and httperf. We went with httperf for no particular reason.

An httperf command looks something like:

httperf --hog --server=example.com --uri=/ --timeout=10 --num-conns=200 --rate=5

In this example, we’re requesting http://example.com/ 200 times, with up to 5 requests per second.

Testing Plan

Some tools support sessions and try to emulate users performing tasks on your site. We went with a simple brute-force test to get an idea of how many requests per second the site could handle.

The basic approach is to make a number of requests and see how the server responds: a status 200 is good, a status 500 is bad. Increase the rate (the number of requests made per second) and try again. When it starts returning lots of 500s, you’ve reached a limit.

Monitoring server resources

The other side is knowing what the server is doing in terms of memory and CPU use. To track this, we run top and log the output to a file for later review. The top command is something like:

top -b -d 3 -U www-data > top.txt

In this example we’re logging information on processes running as user www-data every three seconds. If you want to be more specific, instead of -U username you can use -p 1, 2, 3 where 1, 2 and 3 are pids (process ids of processes you want to watch).

The web server is Lighttpd with Python 2.5 running as FastCGI processes. We didn’t log information on the database process (PostgreSQL), though that could be useful.

Another useful tool is vmstat, particularly the swap columns which show how much memory is being swapped. Swapping means you don’t have enough memory and is a performance killer. To repeatedly run vmstat, specify the number of seconds between checks. e.g.

vmstat 2

Authenticated requests with httperf

httperf makes simple GET requests to a URL and downloads the html (but not any of the media). Requesting public/anonymous pages is easy, but what if you want a page that requires login?

httperf can pass request headers. Django authentication (from django.contrib.auth) uses sessions which rely on a session id held in a cookie on the client. The client passes the cookie in a request header. You see where this is going.

Log in to the site and check your cookies. There should be one like sessionid=97d674a05b2614e98411553b28f909de. To pass this cookie using httperf, use the --add-header option. e.g.

httperf ... --add-header='Cookie: sessionid=97d674a05b2614e98411553b28f909de\n'

Note the \n after the header. If you miss it, you will probably get timeouts for every request.

Which pages to test

With this in mind we tested two pages on the site:

1. home: anonymous request to the home page
2. wall: authenticated request to a “wall” which contains content retrieved from the database

Practically static versus highly dynamic

The home page is essentially static for anonymous users and just renders a template without needing any data from the database.

The wall page is very dynamic, with the main data retrieved from the database. The template is rendered specifically for the user with dates set to the user’s timezone, “remove” links on certain items, etc. The particular wall we tested has about 50 items on it and before optimisation made about 80 database queries.

For the first test we had two FastCGI backends running, able to accept requests for Django.

Home: 175 req/s (i.e. requests per second).
Wall: 8 req/s.

Compressed content

The first config optimisation was to enable gzip compression of the output using GZipMiddleware. Performance improved slightly, but not a huge difference. Worth doing for the bandwidth savings in any case.

Home: 200 req/s.
Wall: 8 req/s.

More processes, shorter queues

Next we increased the number of FastCGI backends from two to five. This was an improvement with fewer 500 responses as more of the requests could be handled by the extra backends.

Home: 200 req/s.
Wall: 11 req/s.

Mo processes, mo problems

The increase from two to five was good, so we tried increasing FastCGI backends to ten. Performance decreased significantly.

Checking with vmstat on the server, I could see it was swapping. Too many processes, each using memory for Python, had caused the VPS to run out of memory and swap memory to and from disk.

Home: 150 req/s.
Wall: 7 req/s.

At this point we set the FastCGI backends back down to five for further tests.

Profiling – where does the time go

The wall page had disappointing performance, so we started to optimise. The first thing we did was profile the code to see where time was being spent.

Using some simple profiling middleware it was clear the time was being spent in database queries. The wall page had a lot of queries and they increased linearly with the number of items on the wall. On the test wall this caused around 80 queries. No wonder its performance was poor.

Optimise this

By optimising how media attached to items is handled we were able to drop one query per item straight away. This slightly reduced how long the request took and so increased the number of queries handled per second.

Wall: 12 req/s.

Another inefficiency was the way several filters were applied to the content of each item whenever the page was requested. We changed it so the html output from the filtered content was stored in the item, saving some processing each time the page was viewed. This gave another small increase.

Wall: 13 req/s.

Back to reducing database queries, we were able to eliminate one query per item by changing how user profiles were retrieved (used to show who posted the item to the wall). Another worthwhile increase came from this change.

Wall: 15 req/s.

The final optimisation for this round of testing was to further reduce the queries needed to retrieve media attached to items. Again, we shed some queries and slightly increased performance.

Wall: 17 req/s.

Next step: caching

Having reduced queries as much as we can, the next step would be to do some caching. Retrieving cached data is usually much quicker than hitting the database, so we’d expect a good increase in performance.

Caching the output of complete pages is not useful because each page is heavily personalised to the user requesting it. It would only be a cache hit if the user requested the same page twice with nothing changing on it in the meantime.

Caching data such as lists of walls, items and users is more useful. The cached data could be used for multiple requests from a single user and shared to some degree across walls and different users. It’s not necessarily a huge win because each wall is likely to have a very small number of users, so the data would need to stay in cache long enough to be retrieved by others.

Our simplistic httperf tests would be very misleading in this case. Each request is made as the same user so cache hits would be practically 100% and performance would be great! This does not reflect real-world use of the site, so we’d need some better tests.

We haven’t made use of caching yet as the site can easily handle its current level of activity, but if Hey! Wall becomes popular, it will be our next step.

How many users is 17 req/s?

Serving 17 req/s still seems fairly low, but it would be interesting to know how this translates to actual users of the site. Obviously, this figure doesn’t include serving any media such as images, CSS and JavaScript files. Media files are relatively large but should be served fast as they are handled directly by Lighttpd (not Django) and have Expires headers to allow the client to cache them. Still, it’s some work the server would be doing in addition to what we measured with our tests.

It’s too early to tell what the common usage pattern would be, so I can only speculate. Allow me to do that!

I’ll assume the average user has access to three walls and checks each of them in turn, pausing for 10 or 20 seconds on each to read new comments and perhaps view some photos or open links. The user does this three times per day.

Looking specifically at the wall page and ignoring media, that means our user is making 9 requests per day for wall pages. Each user only makes one request at a time, so 17 users can be doing that at any second in time. Within a minute the user only makes three requests so is only counted within the 17 concurrent users for 3 seconds out of 60 (or 1 in 20).

If the distribution of user requests over time was perfectly balanced (hint: it won’t be), that means 340 users (17 * 20) could be using the site each minute. To continue with this unrealistic example, we could say there are 1440 minutes in a day and each user is on the site for three minutes per day, so the site could handle about 163,000 users. That would be very good for a $20/month VPS!

To reign in those numbers a bit, lets say we handle 200 concurrent users in a minute for 6 hours per day, 100 concurrent users for another 6 hours and 10 concurrent users for the remaining 12 hours. That’s still around 115,000 users the site could handle in a day given the maximum load of 17 requests per second.

I’m sure these numbers are somewhere between unrealistic and absurd. I’d be interested in comments on better ways to estimate or any real-world figures.

What we learned

To summarise:

1. Testing the performance of your website may yield surprising results
2. Having many database queries is bad for performance (duh)
3. Caching works better for some types of site than others
4. An inexpensive VPS may handle a lot more users than you’d think

A while back I wrote about using edit inline with image and file fields. Specifically, I suggested adding an uneditable BooleanField as the core field of the related model. This means you don’t have to set the ImageField or FileField to be core (which would cause confusing behaviour).

Removing the related model

The downside to having an uneditable core field is that you can’t remove the related model instance using admin. At the time, I wasn’t trouble by this so I just left it. In a recent project I needed to associate photos with articles, use edit_inline for the photos and be able to remove them. So here’s an extended workaround.

As well as the uneditable BooleanField (”keep”) which keeps the ArticlePhoto from being deleted, we now have a “remove” BooleanField which the user can tick in admin to cause the ArticlePhoto to be deleted. The check for this is in the save() method.

class ArticlePhoto(models.Model):
    article = models.ForeignKey(Article, related_name='photos', edit_inline=models.TABULAR, min_num_in_admin=5)
    keep = models.BooleanField(core=True, default=True, editable=False)
    remove = models.BooleanField(default=False)
    image = CustomImageField()

    def save(self):
        if not self.id and not self.image:
            return
        if self.remove:
            self.delete()
        else:
            super(ArticlePhoto, self).save()

It’s a pretty easy way to work around the problem and gives a sensible looking “remove” checkbox in the admin interface. The database table will have a “remove” column that never gets used, but it’s a pretty small price to pay.

When the Django Gigs site first went live we noticed the ordering of developers by name was not right. Those starting with an uppercase letter were coming before those starting with a lowercase letter.

PostgreSQL and the locale

PostgreSQL has a locale setting which is configured when the cluster is created. Among other things, this affects the ordering of results when you use the SQL order by clause.

The local on my server was set to “C” which means it uses byte-level comparisons, rather than following more complex rules for a given culture. Although this is apparently good for performance, it means order by will be case sensitive - e.g. “Zebra” comes before “apple”.

Depending on how your system is set up, you may have locales such as en_GB. The locale can’t easily be changed in PostgreSQL because indexes and other data depends on it. To change locale, you need to start a new cluster and move databases to it.

Django and case-sensitivity

Django provides the order_by() function on QuerySets, but does not have an option for case insensitive ordering. Instead this is left to your database configuration.

When using SQL directly, you can sort case-insensitively using the PostgreSQL lower() function.

e.g.

select * from developer order by lower(name)

One way to do this in Django is to use extra to call the lower() function, creating a virtual column which you can then order by.

e.g.

Developer.objects.all().extra(
select={'lower_name': 'lower(name)'}).order_by('lower_name')

Using SQL functions could tie you to a particular database, though in this case the lower() function is standard and should work with most databases. Some other databases do case-insensitive comparisons so wouldn’t need it.

An informal survey of the Django community

This week, Andrew and I launched the Django Gigs website to help employers find Django developers. Andrew wrote about it and thanks to the Django Community feed aggregator we had quite a few visitors in the first couple of days.

It’s clear that Django is catching on and growing in popularity. The djangoproject.com site is getting close to 8 million hits each month. I thought it would be interesting to analyse my logs and see what I could tell about the Django community, or at least the section of it that read the blog and visited the Django Gigs website.

Visitors

1280 unique IP addresses

The number of IP addresses seems a pretty good indication of how many unique visitors we had in about two days.

Platforms

510	Windows
373	Mac OS X (including 4 iPhones)
312	Linux
85	Other (mostly bots, feed aggregator sites, a handful of BSD)

The platforms is a pretty even split among Windows, Mac and Linux. Which given the dominance of Windows on the desktop suggests Django is disproportionately popular with Mac OS X and Linux users. I suspect this is the case with Python in general, but I don’t have any stats to back that up.

Browsers

875	Firefox
148	Safari
40	IE
36	Camino
13	Konqueror
168	Other (mostly bots or feed readers like NetNewsWire)

No big surprise here: Firefox is the daddy.

One thing that surprised me was the number of different user agents. There were 408 unique user agent strings! Of course, most of them were from different versions of the same software. IE on Windows likes to report versions of the .NET framework and various browser extension installed on the machine.

Countries

423	United States
133	France
126	United Kingdom
67	Germany
61	Canada
45	Russian Federation
42	Brazil
34	Australia
33	Netherlands
23	Italy
16	Belgium
16	China
16	Spain
15	Poland
15	Sweden
14	India
13	Norway
13	Singapore
13	Switzerland
13	Austria
13	Japan
9	Ireland
8	Ukraine
8	New Zealand
7	Finland
7	Portugal
6	Czech Republic
5	Saudi Arabia
5	Iceland

Honourable mentions (1-5 visitors): Slovenia, Denmark, Romania, Greece, Republic of Korea, Serbia and Montenegro, Indonesia, Hong Kong, Philippines, Israel, Croatia, Estonia, Colombia, Peru, Slovakia, Thailand, Turkey, Malaysia, Chile, Puerto Rico, Latvia, Hungary, Belarus, Mexico, Kenya, Kuwait, Nigeria, Lithuania, Argentina, Bolivia, Europe, Iran, Islamic Republic of, Dominican Republic, Moldova, Republic of, Bulgaria, Jamaica, Egypt, United Arab Emirates, Kazakhstan.

I used the free version of GeoIP from MaxMind to look up countries from IP addresses. It’s not totally accurate, but good enough.

It’s very easy to use from Python, assuming you have the library installed:

import GeoIP
geo = GeoIP.new(GeoIP.GEOIP_MEMORY_CACHE)
print geo.country_name_by_addr('4.4.4.4')

It’s not surprising that North America and Western Europe are well represented, but Russia, Brazil and Australia seem to have a good Django following also.

We are the world

Obviously this is just a sample of the Django community and may not be representative, but it does given an indication that Django developers are spread across the world and across the major platforms. That can only be a good thing for the continued growth and success of the framework.

Django admin lets you edit related model objects “inline”. For example when editing a Recipe you can add/eding a group of Ingredient models.

Core fields for edit_inline

The related model being edited inline must specify one or more “core” fields using core=True. If the core fields are filled in, the related model is added. If the core fields are empty, the related model is removed.

This works great for normal objects with CharFields, etc, but not so well if you want to have images or files uploaded using inline editing. If the only core field is a FileField or ImageField, you’ll get strange behaviour like the file/image being removed when you edit an existing model in the admin.

Using inline editing with ImageField or FileField

In a recent project I wanted to have an item with title and description and zero or more photos. The Photo model just has an ImageField. To make it easy to edit, I wanted the photos set to edit_inline.

Here’s my first attempt:

class Item(models.Model):
    title = models.CharField(max_length=100)
    description = models.TextField()

    class Admin:
        pass

class Photo(models.Model):
    item = models.ForeignKey(Item, related_name='photos', edit_inline=models.STACKED)
    image = models.ImageField(blank=False, upload_to='items', core=True)

Notice that the ImageField in Photo has core=True to make it a core field.

This worked ok in the Django admin interface for adding a Photo to a new Item, but if I edited that Item, the Photo would be deleted.

This is a known issue (see Ticket #2534), but it’s marked as “pending design decision” and may be ignored for now since the Django admin is being rewritten to use newforms.

In the meantime I needed a workaround.

Workaround using a different core field

Instead of having the ImageField as a core field, we need something else. If you’ve got some other natural data, such as a caption, that would work fine.

In my case, I didn’t want to add any other fields to the interface, so I went with a BooleanField that is not editable.

Here’s the revised Photo model:

class Photo(models.Model):
    item = models.ForeignKey(Item, related_name='photos', edit_inline=models.STACKED)
    image = models.ImageField(blank=False, upload_to='items')
    keep = models.BooleanField(default=True, editable=False, core=True)

    def save(self):
        # Don't save if there is no image (since core field is always set).
        if not self.id and not self.image:
            return
        super(Photo, self).save()

The keep field has been added and set to be core instead of the image field. Since there is a core field and it’s not empty, the main Item model can be edited without the Photo models being deleted.

Don’t save the empties

The core field always has a value which means the Photo model is told to save even when its ImageField is empty. To prevent creating these empty objects, the Photo model overrides save() and checks if an image was uploaded to the ImageField. If not, it returns without saving.

The remaining issue is that you can’t delete a Photo using the Django admin interface. You can replace the image, but will need some other method for deleting. For me this isn’t a big problem, so the workaround solved the problem for now.

Note that this is just a workaround to the problem which hopefully will be fixed in Django at some point in the future. Ideally, the admin interface would properly handle having an ImageField or FileField as the only core field of the related model and optionally put a “remove” checkbox in the UI to allow removing the image/file.

Django makes it easy to upload images by adding an ImageField to your model. The images are uploaded to your media path in a subdirectory specified with the upload_to parameter which can contain a date/time pattern like %Y/%m/%d.

class Photo(models.Model):
    caption = models.CharField(blank=True, maxlength=100)
    image = models.ImageField(upload_to='photos/%Y/%m/%d')

In this example, images will be uploaded to a path like:

/path/to/media/photos/2007/07/31/flowers.jpg.

Sometimes you want to keep related images together, rather than spreading them over multiple date directories. But if you have a lot of images, you won’t want them all stored in a single directory.

It would be nice if there was a way to upload to a directory specific to the model, perhaps a path incorporating the model object’s id or a unique slug. Something like:

/path/to/media/photos/1234/flowers.jpg
or
/path/to/media/photos/scotland-trip/castle.jpg
/path/to/media/photos/scotland-trip/bonnie-purple-heather.jpg

Django doesn’t have a standard way to do this at the moment (it’s pending a design decision according to ticket #4113).

I needed to do this in a project recently and tried various different approaches. Here’s what I tried - skip to the working solution if you’re not interested in the failed attempts.

Attempt 1: Specify upload_to dynamically

Why not make upload_to include the id of the model?

class Photo(models.Model):
    caption = models.CharField(blank=True, maxlength=100)
    image = models.ImageField(upload_to='photos/%d' % self.id)

Because there is no self; that’s why. Django builds the model with the fields we specify, but at that point, there is no instance of the model, so self is meaningless. Whatever we put in upload_to here will apply to all instances of the model.

Attempt 2: Set upload_to on save

How about overriding the save method of the model and setting upload_to on the image field at that point. Something like:

class Photo(models.Model):
    caption = models.CharField(blank=True, maxlength=100)
    image = models.ImageField(upload_to='photos')

    def save(self):
        for field in self._meta.fields:
            if field.name == 'image':
                field.upload_to = 'photos/%d' % self.id
        super(Photo, self).save()

It’s a bit icky having to iterate through self._meta.fields to find the right one. But a bigger problem is the image file may well be written to the path before save is called.

Usually you would call photo.save_image_file(filename, content) to save the file content then photo.save() to save the model’s fields, including the path in the image field. By the time we set upload_to, it’s too late.

Attempt 3: Override ImageField and pass a callable for upload_to

Taking a different approach, how about making a new class that derives from ImageField and takes either a new parameter or a callable for the upload_to parameter. Something like:

class SpecialImageField(ImageField):

    def get_directory_name(self):
        if callable(self.upload_to):
            return self.upload_to()
        else:
            return super(SpecialImageField, self).get_directory_name()

We override the get_directory_name method which is actually defined in FileField (from which ImageField inherits).

The problem is, we’re still having to pass something (a callable) for the upload_to parameter. Again, at the time the ImageField is created, we are not in an instance of the model, so we can’t pass a it a model method. We could pass a module-level function, but that’s not enough information; we want to set the path using something in the model instance.

Attempt 4 - the one that worked: Override ImageField get model instance and ask it

After going round in circles and learning a few things on the way, I came across this page in the Django wiki (mental note: check wiki first in future).

It shows how a custom field can get hold of the model instance using dispatcher. I wrote CustomImageField to get the model instance when the model instance is initialised and ask it to supply a new upload_to path.

The field looks like this:

from django.db.models import ImageField, signals
from django.dispatch import dispatcher

class CustomImageField(ImageField):
    """Allows model instance to specify upload_to dynamically.

    Model class should have a method like:

        def get_upload_to(self, attname):
            return 'path/to/%d' % self.id

    Based on: http://code.djangoproject.com/wiki/CustomUploadAndFilters
    """
    def contribute_to_class(self, cls, name):
        """Hook up events so we can access the instance."""
        super(CustomImageField, self).contribute_to_class(cls, name)
        dispatcher.connect(self._post_init, signals.post_init, sender=cls)

    def _post_init(self, instance=None):
        """Get dynamic upload_to value from the model instance."""
        if hasattr(instance, 'get_upload_to'):
            self.upload_to = instance.get_upload_to(self.attname)

    def db_type(self):
        """Required by Django for ORM."""
        return 'varchar(100)'

Note the db_type method which replaces the get_internal_type method in Django trunk. It is used when you run manage.py syncdb to know what field type to create in the database.

The field is used in a model like this:

class Photo(models.Model):
    caption = models.CharField(blank=True, maxlength=100)
    image = CustomImageField(upload_to='photos')

    def get_upload_to(self, field_attname):
        """Get upload_to path specific to this photo."""
        return 'photos/%d' % self.id

get_upload_to is passed the attname of the field in that model (in this case “image”). This is so the model can distinguish between multiple custom image fields.

Ok, so the bit I’ve glossed over is that the model may not have an id at the time get_upload_to is called. If the model is new and hasn’t been saved you’ll need to save it or work something else out before you can return the dynamic upload_to path. But that was always the case, so I’m not taking the blame.

In my case, the Photo model was related to some other model (e.g. Room) which I called to get the path. It didn’t matter that Photo didn’t have an id because it was related to something that did.

So now I get to save images to paths like:

/path/to/media/photos/12345/front.jpg
/path/to/media/photos/12345/rooms/kitchen.jpg
etc

Update - 20 May 2008:

Here’s a small update to the CustomImageField class. The version above listens for the post_init signal and use it to get the dynamic upload path. This works fine when you use it like:

photo = Photo.objects.create(...)

Calling create saves the object and loads it so that post_init gets called. However, if you create the model object and upload a file before saving, it will not know about the dynamic upload path.

The version below listens for the pre_save signal instead and gets the dynamic upload path at that point. You can use it like:

photo = Photo()
photo.save_image_file(filename, content)

Note that you may still need to save the model before uploading if your dynamic path includes the model id (which is not set until it is saved).

Here is the new version of the field:

class CustomImageField(ImageField):
    """Allows model instance to specify upload_to dynamically.

    Model class should have a method like:

        def get_upload_to(self, attname):
            return 'path/to/%d' % self.id

    Based on: http://code.djangoproject.com/wiki/CustomUploadAndFilters
    """
    def __init__(self, *args, **kwargs):
        if not 'upload_to' in kwargs:
            kwargs['upload_to'] = 'dummy'
        self.prime_upload = kwargs.get('prime_upload', False)
        if 'prime_upload' in kwargs:
            del(kwargs['prime_upload'])
        super(CustomImageField, self).__init__(*args, **kwargs)

    def contribute_to_class(self, cls, name):
        """Hook up events so we can access the instance."""
        super(CustomImageField, self).contribute_to_class(cls, name)
        if self.prime_upload:
            dispatcher.connect(self._get_upload_to, signals.post_init, sender=cls)
        dispatcher.connect(self._get_upload_to, signals.pre_save, sender=cls)

    def _get_upload_to(self, instance=None):
        """Get dynamic upload_to value from the model instance."""
        if hasattr(instance, 'get_upload_to'):
            self.upload_to = instance.get_upload_to(self.attname)

    def db_type(self):
        """Required by Django for ORM."""
        return 'varchar(100)'

In some cases you will still want the path to be specified when the model is initialised rather than saved. If you are editing a model and want to be able to save a new image without saving the model first, it needs to get the dynamic upload path when the post_init signal is raised.

This new CustomImageField takes an optional prime_upload argument. If true, it will also listen for the post_init event and get the dynamic upload path. You can use it like:

class Photo(models.Model):
    image = CustomImageField(prime_upload=True)

photo = Photo.objects.get(pk=photo_id)
photo.save_image_file(filename, content)

This is all a bit fiddly still, but it does the job until Django has a native way to specify the upload path per-instance.

The MVC-style separation in Django makes it easy to write Python scripts to handle automated jobs. They can query the database using the Django database API and create and manipulate objects. Essentially everything you can do in a view, except you don’t have a request object since you aren’t in the context of an http request.

For my current project I have a couple of scripts that are executed by cron. One script emails users telling them what’s new. It gets this information by calling methods on various models. Another script uses the database API to gather some statistics on the state of the system (e.g. how many registered users and who were the last five) and sends them by email.

Here’s some code.

#! /usr/bin/env python

import sys
import os

def setup_environment():
    pathname = os.path.dirname(sys.argv[0])
    sys.path.append(os.path.abspath(pathname))
    sys.path.append(os.path.normpath(os.path.join(os.path.abspath(pathname), '../')))
    os.environ['DJANGO_SETTINGS_MODULE'] = 'settings'

# Must set up environment before imports.
setup_environment()

from django.core.mail import send_mail
from django.contrib.auth.models import User

def main(argv=None):
    if argv is None:
        argv = sys.argv

    # Do stuff here.  For example, send an email reporting number of users.

    user_count = User.objects.count()
    message = 'There are now %d registered users.' % user_count

    send_mail('System report',
              message,
              'report@example.com',
              ['user@example.com'])

if __name__ == '__main__':
    main()

The interesting bit is in setup_environment which adds paths to the Python path so when you import modules used in your project (e.g. import myapp.models) Python knows where to find them.

In my case, I put my scripts in to the Django project directory (i.e. next to settings.py). The path to the Django project needs to be in the Python path so that imports work. setup_environment uses sys.path.append to add the path of the script that is executing (e.g. /srv/www/mydjangoproject/status_report.py) and its parent directory. This is sufficient to make imports in the project work.

The other thing that needs to be set is the DJANGO_SETTINGS_MODULE environment variable. It’s just the name of the settings module, by default “settings”.

There may be a better way to do this setup, but it seems to work ok.

Now you can call the script directly or from cron. If you’re sending emails from your script, you can write straight text templates (rather than html) and use django.template.loader.render_to_string to render the template with some variables in to a string to send by email.

from django.template.loader import render_to_string

email_body = render_to_string('reports/email/score_report.txt',
                              {'user': user,
                               'top_score': top_score})

Update: Here’s a cleaner way from Jared. It uses the setup_environ function from django.core.management.

Update: James Bennett has an excellent write-up on standalone Django scripts.

I have various models in my current Django project that can be marked as “deleted”. They’re still in the database, but filtered out and as far as most of the code is concerned, they no longer exist. A simple way to do this is with a custom manager that filters the query set.

# Custom manager filters out deleted items.
class NotDeletedManager(models.Manager):
    def get_query_set(self):
        return super(NotDeletedManager, self).get_query_set().filter(deleted=False)

class Article(models.Model):
    author = models.ForeignKey(Author, related_name='articles')
    ...
    deleted = models.BooleanField(default=False)
    # Use custom manager as default manager for this model.
    objects = NotDeletedManager()

I can then refer to Article.objects.all() and get only those articles not marked deleted. Likewise, author.articles.all() gets articles for a given author that are not marked deleted.

Next I wanted to be able to access deleted items. Time for another manager:

class DeletedManager(models.Manager):
    def get_query_set(self):
        return super(DeletedManager, self).get_query_set().filter(deleted=True)

class Article(models.Model):
    ...
    objects = NotDeletedManager()
    deleted_articles = DeletedManager()

It seems a bit redundant, but two managers is not bad. Especially since they can be used with any model that has a “deleted” field.

Next I wanted to filter related objects based on whether they belong to a deleted object. For example, if an author is deleted, all articles by that author should not show up in articles.

class NotDeletedArticleOrAuthorManager(models.Manager):
    def get_query_set(self):
        return super(NotDeletedArticleOrAuthorManager, self).get_query_set().filter(deleted=False, author__deleted=False)

class Article(models.Model):
    ...
    objects = NotDeletedArticleOrAuthorManager()

This is getting a bit messy. I’d rather pass parameters to the manager telling it what to filter out. Something like…

class FilterManager(models.Manager):
    "Custom manager filters standard query set with given args."
    def __init__(*args, **kwargs):
        self.args = args
        self.kwargs = kwargs

    def get_query_set(self):
        return super(FilterManager, self).get_query_set().filter(*self.args, **self.kwargs)

class Article(models.Model):
    ...
    objects = FilterManager(deleted=False, author__deleted=False)
    deleted_articles = FilterManager(deleted=True)

Unfortunately, this doesn’t work. The filters are applied when I call the manager directly like Article.objects.all(), but no filters are applied if I go through a relationship like author.articles.all() (returns deleted articles as well).

The problem is the related managers (see the source django/db/models/fields/related.py e.g. ForeignRelatedObjectsDescriptor.__get__). They create a class that dynamically inherits from the models default manager (our custom manager) then return a new instance of it. The new instance doesn’t have any parameters passed to its __init__ method, so there are no filters.

My workaround is to use a function to return a custom class that has the filter parameters saved in the class, so they don’t get passed to an instance. When a related manager inherits from this class, it still has our filters in place.

def get_filter_manager(*args, **kwargs):
    class FilterManager(models.Manager):
        "Custom manager filters standard query set with given args."
        def get_query_set(self):
            return super(FilterManager, self).get_query_set().filter(*args, **kwargs)
    return FilterManager()

class Article(models.Model):
    ...
    objects = get_filter_manager(deleted=False, author__deleted=False)
    deleted_articles = get_filter_manager(deleted=True)

The class is declared in the function and gets args and kwargs from the function. They are therefore “baked in” to the class and don’t need to be passed as parameters to __init__.

This is only minimally tested, but seems to work. Perhaps there is a better way, but for now this is what I have.