Python, GIL and why it doesn’t scale well

By farcaller June 4th, 2008

You might already know about my other hobby project: L0st. It’s a file storage service (data is hosted locally) written in Django. Everything worked mostly well until I wanted to write some Jabber notification service. It was quite logical to use some XMPP library for python, logical until I remembered about GIL (aka Global Interpreter Lock). The problem is that you can’t have Django application waiting on fastcgi socket and jabber socket in the same time even if you have them locked in separate threads. I could use separate daemon and IPC but that would look like “open socket to daemon, tell him to do something, close socket”. No way for reverse communication (I could use Django ORM in the bot code, but that doesn’t look like a great solution).
I’ve ended with spawning sendxmpp in background. That works one way only, but it doesn’t lock Django application in response queue for long. Not the solution I’ve been really looking for, but it works for now.
C allows you much more flexibility. In fact it allows you complete flexibility. Now I’m thinking of porting L0st to upcoming FOW and I can think about requirements for two completely different projects. That would be a good improvement for FOW.

This entry was posted on Wednesday, June 4th, 2008 at 21:35 and is filed under Conceptual. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.

3 Responses to “Python, GIL and why it doesn’t scale well”

  1. rilian Says:

    just save message tasks on some log file and process them with outer daemon. easy

  2. farcaller Says:

    That’s not *instant*. Polling sucks any way

  3. QQ Says:

    Would Twisted help you out here? You’d get a deferred back that calls you instead of checking using polling.

Leave a Reply