I have done many primarily-RPC web applications using tomcat or PHP or twisted, but by far the best way is using Python and Django. However, I have always viewed the state of RPC in Django broken. While using Django for all sorts of other web application purposes is incredibly easy, getting started with RPC presents a lot more friction.

Introducing django-json-rpc, a very easy way to expose your web application to the world of integrated applications.

Installation is as simple as:

  easy_install django-json-rpc

Add a mount point to your urls.py file and import django-json-rpc.

  from jsonrpc import jsonrpc_site

  urlpatterns += patterns('',
    (r'^json/', jsonrpc_site.dispatch)
  )

The interface to django-json-rpc is exposed through only one function - the jsonrpc_method decorator. Functions that use this decorator may be placed anywhere in the source tree and need only import jsonrpc.jsonrpc_method.

  from jsonrpc import jsonrpc_method

  @jsonrpc_method('app.sayHello')
  def say_hello(request, name):
    return "Hello, " + name

To hook your views into the default jsonrpc site, they must simply be imported somewhere by Django. The best place for this is in your urls.py file, which should now look something like this:

  from jsonrpc import jsonrpc_site
  import myapp.views

  urlpatterns += patterns('',
    (r'^json/', jsonrpc_site.dispatch)
  )

Test your service with the provided Proxy:

  ./manage.py runserver 8080

  ./manage.py shell

  >>> from jsonrpc.proxy import ServiceProxy
  >>> s = ServiceProxy('http://localhost:8080/json/')
  >>> s.app.sayHello('Sam')
  {u'error': None, u'id': u'jsonrpc', u'result': u'Hello Sam'}

HTTP GET

Django-json-rpc supports JSON-RPC version 1.1 which includes support for HTTP GET, meaning, all your REST are belong to us as well. Add the HTTP GET mount point to your urls.py file:

  from jsonrpc import jsonrpc_site
  import myapp.views

  urlpatterns += patterns('',
    (r'^json/', jsonrpc_site.dispatch),
    (r'^json/(?P[a-zA-Z0-9.]+)$', jsonrpc_site.dispatch)
  )

It is required by django-json-rpc to mark each method safe for dispatch through HTTP GET. jsonrpc_method('sayHello') becomes:

  jsonrpc_method('app.sayHello', safe=True)

Now your method will be available at http://localhost:8080/json/app.sayHello?name=Sam. HTTP GET requests only support string and number typed arguments.

Authentication

The django-json-rpc package also supports authentication, by default using django.contrib.auth’s model backend - the User object we’re all so familiar with, however you may provide any method, including using any authentication middleware. If you aren’t using middleware, username and password arguments will automatically be added to the beginning of the argument list for your method.

  @jsonrpc_method('app.sayHello', authenticated=True)
  def say_hello(request):
    return "Hello, " + request.user.first_name

  >>> s.app.sayHello('samuraiblog', 'password')
  {u'error': None, u'id': u'jsonrpc', u'result': u'Hello Sam'}

Version Agnostic

django-json-rpc will continue to work regardless of which version JSON-RPC client you happen to be using. It supports all argument types supported in JSON-RPC versions 1.0, 1.1 and 2.0, and will respond if a client specifies the version in either the jsonrpc or version key.

Conclusion

I have used this package for the development of several commercial iPhone applications and it has greatly improved my productivity in developing a web service backend. Up next? A django-xmlrpc? Haha! Fuck that!

Say you have work unit WU which takes pickleable arguments (a, b, c). If your work unit is thread-safe (it modifies no global variables, and causes no side effects in your application) it can be preformed on in another process, or even on another machine, allowing you to return control to the user immediately.

For example, I want to add the ability for a user to import their contacts using X contacts service:

Fig 1: project/views.py

def import_contacts(request):
    from my.specialsauce.contacts import run_contacts_import
    contacts = run_contacts_import(request)
    return render_to_response('t/contacts-import.html', {'contacts': contacts})

Currently, the view retrieves and imports the users contacts in the view, making the user wait until the operation has completed to see the returned page. What if we could implement this in a way which would return control to the user immediately, firing off run_contacts_import in another thread, another process, or even another server!

To do this we will need Django, multiprocessing and Python 2.5+. Now, I will assume you are already as far as Fig 1. Say a user with 10,000 contacts (or even 100) ambles along and imports their contacts. Both users are going to be waiting much longer than the average gmail-generation-web-user can be expected to wait. They will hit refresh, restarting this process and crashing your server.

Since the Django request object is pickleable, we can pass it as an argument to another process using multiprocessing.

Fig 2: project/views.py

from multiprocessing import Pool

m = Pool(10)

def import_contacts(request):
    # run_contacts_import will have to update request.session['CONTACT_IMPORT_STATUS']
    has_status = 'CONTACT_IMPORT_STATUS' in request.session
    status = has_status and request.session['CONTACT_IMPORT_STATUS'] == True
    if status is True:
        return HttpResponseRedirect('/import-finished/')
    elif not status and has_status:
        return render_to_response('t/contacts-import-not-finished.html', {})
    elif not has_status:
        from my.specialsauce.contacts import run_contacts_import
        m.apply_async(run_contacts_import, args=(request,))
        return render_to_response('t/contacts-import.html', {'contacts': contacts})

This seems simple enough, and upon first inspection, seems to work rather decently. However, this adds a global variable to our Django process, which is a bad idea. Using this technique, it would be best to create a lock, spawn off a new thread, and acquire/release the lock while running apply_async (blegh). Unfortunately, this can eventually make your Django server unresponsive as many users come to run the expensive process (simply via system load). We need to move this process, and possibly a few others off our new server.

To do this, the Python multiprocessing package provides the high-level Manager class. The Manager class is used to pipe commands between managers in different instances of the python interpreter.

Create the Manager Class

First, we need to create the Manager class that will eventually be implemented as our server. Create a file called managers.py in project/managers.py.

Fig 3: project/managers.py

from multiprocessing.managers import BaseManager

from my.specialsauce.contacts import run_contacts_import

# This is the shared object. It simply implements a function which applies to the instance variable
# pool, which will be installed when the server is actually started (in a snippet below)
class ContactImporter(object):
    def __init__(self, pool=None):
        self.pool = pool

    def run_import(self, *args):
        self.pool.apply_async(run_contacts_import, args=args)

# instantiate the shared object
importer = ContactImporter()

# our manager
class CManager(BaseManager):
    pass

# you need to register a function which will return the shared object
# the shared object could be anything pickleable in python, in this case
# our new-style class descending from object
CManager.register('get_importer', callable=lambda:importer)

Create the Server

We are going to be making a daemon server that we can start and stop using your projects manage.py file. To do this create a file called lrpserver.py in project/management/commands/lrpserver.py.

Fig 4: project/management/commands/lrpserver.py

import os
import sys
import time
from optparse import make_option
from signal import SIGTERM

from multiprocessing import Pool

from django.core.management.base import BaseCommand
from django.conf import settings

from project.managers import CManager, importer

class Command(BaseCommand):
    option_list = BaseCommand.option_list + (
        make_option('--pidfile', dest='pidfile', default='', help="Specifies the PID file to use."),
        make_option('--start', dest='start', default='no', help='Starts the daemon'),
        make_option('--stop', dest='stop', default='no', help='Stops the daemon'),
        make_option('--restart', dest='restart', default='no', help='Restarts the daemon')
    )
    def handle(self, *args, **options):
        pidfile = options.get('pidfile')
        if not pidfile:
            print "--pidfile arg required"
            sys.exit(1)
        start = options.get('start')
        stop = options.get('stop')
        restart = options.get('restart')
        server = getattr(settings, 'LRP_SERVER_HOST', None)
        port = getattr(settings, 'LRP_SERVER_PORT', None)
        authkey = getattr(settings, 'LRP_SERVER_AUTHKEY', None)

        if server is None or port is None or authkey is None:
            print 'LRP_SERVER_HOST, LRP_SERVER_PORT and LRP_SERVER_AUTHKEY values required in settings file.'
            sys.exit(1)

        # This daemonizes our process and starts the server
        def _start():
            print 'Starting Long Running Process Server'
            from django.utils.daemonize import become_daemon
            become_daemon()
            fp = open(pidfile, "w")
            fp.write('%d\n' % os.getpid())
            fp.close()

            # set the pool that importer.run_import requires
            importer.pool = Pool(processes=10)
            # create an instance of CManager, get a server instance and start the server loop
            m = CManager(address=(server, port), authkey=authkey)
            s = m.get_server()
            s.serve_forever()

        def _stop():
            try:
                fp = open(pidfile, 'r')
                pid = int(fp.read().strip())
                fp.close()
            except IOError:
                pid = None
            if not pid:
                print 'Long Running Process Server Not Currently Running'
                return
            try:
                print 'Stopping Long Running Process Server'
                while 1:
                    os.kill(pid, SIGTERM)
                    time.sleep(0.1)
            except OSError, err:
                err = str(err)
                if err.lower().find('no such process') > 0:
                    if os.path.exists(pidfile):
                        os.remove(pidfile)
                else:
                    print err
                    sys.exit(1)

        def _restart():
            _stop()
            _start()

        if str(start) == 'yes':
            return _start()
        elif str(stop) == 'yes':
            return _stop()
        elif str(restart) == 'yes':
            return _restart()
        else:
            print 'Options: pidfile=%s start=%s stop=%s restart=%s' % (pidfile, start, stop, restart)
            print 'One of --(start|stop|restart)=yes is required.'

Add the Required Settings to Your settings.py File

The server and client require three settings, the host name of your server, the port on which the server is listening and the auth key of the required to access the server. In your settings.py file add the following with your values. For now, use the blank string ” as the setting for host name. This instructs the server to serve from localhost. (Hint: If you do not know this value use socket.getfqdn()).

Fig 5: settings.py

LRP_SERVER_HOST = ''
LRP_SERVER_PORT = 5667
LRP_SERVER_AUTHKEY = 'password'

Modify Your View Code

To connect to the server, you must spawn a thread which creates an instance of the manager, connects and sends the command.

Fig 6: project/views.py

from threading import Thread
from multiprocessing.managers import BaseManager

from django.conf import settings

class CManager(BaseManager):
    pass

CManager.register('get_importer')

# This function will become a thread which instantiates the Manager class,
# connects and starts the long running process
def _start_import_thread(request):
    gci_manager = CManager(address=(settings.GCI_SERVER_HOST, settings.GCI_SERVER_PORT),
                           authkey=settings.GCI_SERVER_AUTHKEY)
    gci_manager.connect()
    importer = gci_manager.get_importer()
    importer.run_import(request)

# This is not a view, it's a helper function which starts
# the thread above
def start_contacts_import(request):
    t = Thread(target=_start_import_thread, args=[request])
    t.setDaemon(True)
    t.start()

def import_contacts(request):
    # run_contacts_import will have to update request.session['CONTACT_IMPORT_STATUS']
    has_status = 'CONTACT_IMPORT_STATUS' in request.session
    status = has_status and request.session['CONTACT_IMPORT_STATUS'] == True
    if status is True:
        return HttpResponseRedirect('/import-finished/')
    elif not status and has_status:
        return render_to_response('t/contacts-import-not-finished.html', {})
    elif not has_status:
        start_contacts_import(request)
        return render_to_response('t/contacts-import.html', {'contacts': contacts})

Start the Server and Let it Fly!

And finally we may start the server.

python manage.py lrpserver.py --pidfile=/home/django/run/lrp.pid --start=yes

Now you can run your view.

Follow-Up

This process is applicable to more than just crawlers and contact importers. It can be used to minimize the load on your server for any number of tasks such as image and other file processing or running large database updates. The server can be put on another machine simply by duplicating your django installation on another machine and using that machine’s socket.getfqdn() (or ip address) value as the host name in your configs. You can also add a javascript status updater to notify the user of progress information related to their request.