Large read() calls may be problematic — for reasons besides memory usage — if they block for an uncomfortable period of time before returning, especially if your disk is highly contended for or slow (or over a network).

What I’m really trying to approximate here is asynchronous I/O where you are regularly given chances to empty a buffer that is filled as quickly as possible to allow more responsive streaming of bytes. A basic consumer/producer issue.

This problem is present for any time when read() may block for an extended period when you could be delivering chunks of the stream that you’ve already gotten.


class file(file):
def __iter__(self):
if 'b' in self.mode:
return iter(lambda: self.read(1024 * 16), '')
else:
return self


Or Django could incorporate similar logic, or just always serve open files as chunks. There’s no point seeking for lines when it’s not doing anything with them.

For big chunk sizes, I don’t think using alarms or more threads are going to help you. AFAIK, during a request, one thread is dedicated to servicing that request, right until it completes. Breaking out of a long write is not going to free that thread to service other requests unless you stop servicing the request entirely. The chunk size is probably only a concern if you’re planning on servicing a lot of concurrent downloads, when you might need to reduce memory consumption.

It is true, and I don’t ask it to be a *good* static file server, but the current performance penalty is atrocious…probably a factor of 50 to 100 in terms of CPU usage.

Also, client side interpretation is neither here nor there with regard to this issue, the issue that it’s extremely easy for a naive developer to type something like “open(‘file’)” and then send the file handle to Django to be sent. Django takes the very sensible action of making an iterator of the file object, and by default Python’s iterators over files seeks to return one line at a time. Or, more to the point, any series of bytes terminated by a newline byte.

When one wants to simply send something…such as open(‘movie.mkv’)…this seeking of newline bytes is a huge waste of time. Instead, something like the FileIterWrapper documented in the post will make the process much, much more efficient.

My only beef with this is that it hits someone who makes this honest, common mistake far too hard. It also may reflect badly on Django’s performance, whereas it’s not really Django’s fault.

I sympathise with your complaints about Django’s static file skiddishness. It’s obnoxious for developing and stuff, but I doubt the Django team wants to accept the responsibility for creating a robust and reliable static web server. Besides, you can enable static files w/o too much trouble.