Python, GIL and why it doesn’t scale well
You might already know about my other hobby project: L0st. It’s a file storage service (data is hosted locally) written in Django. Everything worked mostly well until I wanted to write some Jabber notification service. It was quite logical to use some XMPP library for python, logical until I remembered about GIL (aka Global Interpreter Lock). The problem is that you can’t have Django application waiting on fastcgi socket and jabber socket in the same time even if you have them locked in separate threads. I could use separate daemon and IPC but that would look like “open socket to daemon, tell him to do something, close socket”. No way for reverse communication (I could use Django ORM in the bot code, but that doesn’t look like a great solution).
I’ve ended with spawning sendxmpp in background. That works one way only, but it doesn’t lock Django application in response queue for long. Not the solution I’ve been really looking for, but it works for now.
C allows you much more flexibility. In fact it allows you complete flexibility. Now I’m thinking of porting L0st to upcoming FOW and I can think about requirements for two completely different projects. That would be a good improvement for FOW.
June 4th, 2008 at 22:16
just save message tasks on some log file and process them with outer daemon. easy
June 4th, 2008 at 22:18
That’s not *instant*. Polling sucks any way
October 2nd, 2008 at 08:22
Would Twisted help you out here? You’d get a deferred back that calls you instead of checking using polling.