SamuraiBlog.com

Have you ever wanted to put up a server made exclusively to handle the expensive actions in your web app? In this article I will be showing you how to create a server to that will be set up to receive commands from your web server and process them in the context of your django web app. This technique could be used many places such as a desktop GUI application, or a web service architecture.

Say you have work unit WU which takes pickleable arguments (a, b, c). If your work unit is thread-safe (it modifies no global variables, and causes no side effects in your application) it can be preformed on in another process, or even on another machine, allowing you to return control to the user immediately.

For example, I want to add the ability for a user to import their contacts using X contacts service:

Fig 1: project/views.py

 Python | 
 copy code |
?
 
1
def import_contacts(request):
2
    from my.specialsauce.contacts import run_contacts_import
3
    contacts = run_contacts_import(request)
4
    return render_to_response('t/contacts-import.html', {'contacts': contacts})

Currently, the view retrieves and imports the users contacts in the view, making the user wait until the operation has completed to see the returned page. What if we could implement this in a way which would return control to the user immediately, firing off run_contacts_import in another thread, another process, or even another server!

To do this we will need Django, multiprocessing and Python 2.5+. Now, I will assume you are already as far as Fig 1. Say a user with 10,000 contacts (or even 100) ambles along and imports their contacts. Both users are going to be waiting much longer than the average gmail-generation-web-user can be expected to wait. They will hit refresh, restarting this process and crashing your server.

Since the Django request object is pickleable, we can pass it as an argument to another process using multiprocessing.

Fig 2: project/views.py

 Python | 
 copy code |
?
 
01
from multiprocessing import Pool
02
 
03
m = Pool(10)
04
 
05
def import_contacts(request):
06
    # run_contacts_import will have to update request.session['CONTACT_IMPORT_STATUS']
07
    has_status = 'CONTACT_IMPORT_STATUS' in request.session
08
    status = has_status and request.session['CONTACT_IMPORT_STATUS'] == True
09
    if status is True:
10
        return HttpResponseRedirect('/import-finished/')
11
    elif not status and has_status:
12
        return render_to_response('t/contacts-import-not-finished.html', {})
13
    elif not has_status:
14
        from my.specialsauce.contacts import run_contacts_import
15
        m.apply_async(run_contacts_import, args=(request,))
16
        return render_to_response('t/contacts-import.html', {'contacts': contacts})

This seems simple enough, and upon first inspection, seems to work rather decently. However, this adds a global variable to our Django process, which is a bad idea. Using this technique, it would be best to create a lock, spawn off a new thread, and acquire/release the lock while running apply_async (blegh). Unfortunately, this can eventually make your Django server unresponsive as many users come to run the expensive process (simply via system load). We need to move this process, and possibly a few others off our new server.

To do this, the Python multiprocessing package provides the high-level Manager class. The Manager class is used to pipe commands between managers in different instances of the python interpreter.

Create the Manager Class

First, we need to create the Manager class that will eventually be implemented as our server. Create a file called managers.py in project/managers.py.

Fig 3: project/managers.py

 Python | 
 copy code |
?
 
01
from multiprocessing.managers import BaseManager
02
 
03
from my.specialsauce.contacts import run_contacts_import
04
 
05
# This is the shared object. It simply implements a function which applies to the instance variable
06
# pool, which will be installed when the server is actually started (in a snippet below)
07
class ContactImporter(object):
08
    def __init__(self, pool=None):
09
        self.pool = pool
10
 
11
    def run_import(self, *args):
12
        self.pool.apply_async(run_contacts_import, args=args)
13
 
14
# instantiate the shared object
15
importer = ContactImporter()
16
 
17
# our manager
18
class CManager(BaseManager):
19
    pass
20
 
21
# you need to register a function which will return the shared object
22
# the shared object could be anything pickleable in python, in this case
23
# our new-style class descending from object
24
CManager.register('get_importer', callable=lambda:importer)

Create the Server

We are going to be making a daemon server that we can start and stop using your projects manage.py file. To do this create a file called lrpserver.py in project/management/commands/lrpserver.py.

Fig 4: project/management/commands/lrpserver.py

 Python | 
 copy code |
?
 
01
import os
02
import sys
03
import time
04
from optparse import make_option
05
from signal import SIGTERM
06
 
07
from multiprocessing import Pool
08
 
09
from django.core.management.base import BaseCommand
10
from django.conf import settings
11
 
12
from project.managers import CManager, importer
13
 
14
 
15
class Command(BaseCommand):
16
    option_list = BaseCommand.option_list + (
17
        make_option('--pidfile', dest='pidfile', default='', help="Specifies the PID file to use."),
18
        make_option('--start', dest='start', default='no', help='Starts the daemon'),
19
        make_option('--stop', dest='stop', default='no', help='Stops the daemon'),
20
        make_option('--restart', dest='restart', default='no', help='Restarts the daemon')
21
    )
22
    def handle(self, *args, **options):
23
        pidfile = options.get('pidfile')
24
        if not pidfile:
25
            print "--pidfile arg required"
26
            sys.exit(1)
27
        start = options.get('start')
28
        stop = options.get('stop')
29
        restart = options.get('restart')
30
        server = getattr(settings, 'LRP_SERVER_HOST', None)
31
        port = getattr(settings, 'LRP_SERVER_PORT', None)
32
        authkey = getattr(settings, 'LRP_SERVER_AUTHKEY', None)
33
 
34
        if server is None or port is None or authkey is None:
35
            print 'LRP_SERVER_HOST, LRP_SERVER_PORT and LRP_SERVER_AUTHKEY values required in settings file.'
36
            sys.exit(1)
37
 
38
        # This daemonizes our process and starts the server
39
        def _start():
40
            print 'Starting Long Running Process Server'
41
            from django.utils.daemonize import become_daemon
42
            become_daemon()
43
            fp = open(pidfile, "w")
44
            fp.write('%d\n' % os.getpid())
45
            fp.close()
46
 
47
            # set the pool that importer.run_import requires
48
            importer.pool = Pool(processes=10)
49
            # create an instance of CManager, get a server instance and start the server loop
50
            m = CManager(address=(server, port), authkey=authkey)
51
            s = m.get_server()
52
            s.serve_forever()
53
 
54
        def _stop():
55
            try:
56
                fp = open(pidfile, 'r')
57
                pid = int(fp.read().strip())
58
                fp.close()
59
            except IOError:
60
                pid = None
61
            if not pid:
62
                print 'Long Running Process Server Not Currently Running'
63
                return
64
            try:
65
                print 'Stopping Long Running Process Server'
66
                while 1:
67
                    os.kill(pid, SIGTERM)
68
                    time.sleep(0.1)
69
            except OSError, err:
70
                err = str(err)
71
                if err.lower().find('no such process') > 0:
72
                    if os.path.exists(pidfile):
73
                        os.remove(pidfile)
74
                else:
75
                    print err
76
                    sys.exit(1)
77
 
78
        def _restart():
79
            _stop()
80
            _start()
81
 
82
        if str(start) == 'yes':
83
            return _start()
84
        elif str(stop) == 'yes':
85
            return _stop()
86
        elif str(restart) == 'yes':
87
            return _restart()
88
        else:
89
            print 'Options: pidfile=%s start=%s stop=%s restart=%s' % (pidfile, start, stop, restart)
90
            print 'One of --(start|stop|restart)=yes is required.'
91

Add the Required Settings to Your settings.py File

The server and client require three settings, the host name of your server, the port on which the server is listening and the auth key of the required to access the server. In your settings.py file add the following with your values. For now, use the blank string ” as the setting for host name. This instructs the server to serve from localhost. (Hint: If you do not know this value use socket.getfqdn()).

Fig 5: settings.py

 Python | 
 copy code |
?
 
1
LRP_SERVER_HOST = ''
2
LRP_SERVER_PORT = 5667
3
LRP_SERVER_AUTHKEY = 'password'

Modify Your View Code

To connect to the server, you must spawn a thread which creates an instance of the manager, connects and sends the command.

Fig 6: project/views.py

 Python | 
 copy code |
?
 
01
from threading import Thread
02
from multiprocessing.managers import BaseManager
03
 
04
from django.conf import settings
05
 
06
 
07
class CManager(BaseManager):
08
    pass
09
 
10
 
11
CManager.register('get_importer')
12
 
13
# This function will become a thread which instantiates the Manager class,
14
# connects and starts the long running process
15
def _start_import_thread(request):
16
    gci_manager = CManager(address=(settings.GCI_SERVER_HOST, settings.GCI_SERVER_PORT),
17
                           authkey=settings.GCI_SERVER_AUTHKEY)
18
    gci_manager.connect()
19
    importer = gci_manager.get_importer()
20
    importer.run_import(request)
21
 
22
# This is not a view, it's a helper function which starts
23
# the thread above
24
def start_contacts_import(request):
25
    t = Thread(target=_start_import_thread, args=[request])
26
    t.setDaemon(True)
27
    t.start()
28
 
29
def import_contacts(request):
30
    # run_contacts_import will have to update request.session['CONTACT_IMPORT_STATUS']
31
    has_status = 'CONTACT_IMPORT_STATUS' in request.session
32
    status = has_status and request.session['CONTACT_IMPORT_STATUS'] == True
33
    if status is True:
34
        return HttpResponseRedirect('/import-finished/')
35
    elif not status and has_status:
36
        return render_to_response('t/contacts-import-not-finished.html', {})
37
    elif not has_status:
38
        start_contacts_import(request)
39
        return render_to_response('t/contacts-import.html', {'contacts': contacts})

Start the Server and Let it Fly!

And finally we may start the server.

 Bash | 
 copy code |
?
 
1
python manage.py lrpserver.py --pidfile=/home/django/run/lrp.pid --start=yes

Now you can run your view.

Follow-Up

This process is applicable to more than just crawlers and contact importers. It can be used to minimize the load on your server for any number of tasks such as image and other file processing or running large database updates. The server can be put on another machine simply by duplicating your django installation on another machine and using that machine’s socket.getfqdn() (or ip address) value as the host name in your configs. You can also add a javascript status updater to notify the user of progress information related to their request.

Boneless creature of iron and oil,
Legless, headless and mindful of all.
Another series of rooms a-boil,
To chamber my froth, life, at your call.

This is my first creative writing post. It’s not much right now, as I’m still trying to figure out my first assignment. It seems I have some time today to write. No work until Thursday at 3P. Good. Poems are cool to write. :)

Genesis Log Page 43

June 9′th 1929; 1:30 AM

Twenty years ago today, I began building a machine. Now, nearly the size of a $2/hr room in San Francisco, I have finally been able to name this monster with a name signifying it’s being. Genesis. Why? Fuck you, that’s why. I call it Genesis.

Sure, that may be a piss-poor explanation, but this machine is beyond the realm of revolutionary. Fuck, it can build itself! Well, since last night that statement has been false. There’s a guy I met who owns a metal shop. Genesis, in big, brassy letters. On the door or awning or something–and it wouldn’t be so much for the letters, because obviously I could handle that myself, as to, finally validate our existence to the world–but not until this bitch is again reproducing.

June 9′th 1929; Around 3 AM

I’ve stopped working on the machine for a few minutes to log my progress and have a cigarette. Along the way I’ve discovered the master unit started binding on all metal materials. Explanation at this point is beyond me. Note to self: Discover a method to minimize vibrations in the cutting module near the back end of master unit without disturbing the spacing of lower modules (which would of course throw timing off too).

June 9′th 1929; 4:30 AM

Another smoke and log break. Two smokes, and now for the log. I should really get an eraser. I tried coating a fraction (every other twice) of the fingers (in the master unit, both main and secondary clusters) in a light polytetrafluoroethylene, but did too many. The machine dropped the first production unit and spilled molten glass all over the conveyance, which is luckily made of iron slats. The scrap pan below caught the rest. I cleaned the glass. And I still have no fucking clue why it was loading glass. Maybe it’s getting a bit late.

June 9′th 1929; 5AM

Another smoke. This is killing me. Not the smokes, they are too, it’s the nicotine. I guess it is the smokes

June 9′th 1929; Around 6AM

During my last log entry I had an idea regarding my smoking habit. I took four cigarette butts and coated them with the polytetrafluoroethylene and wedged them under plates that mount the cutting module. No more vibrations. Rather, few enough, and muted enough, that it can manage. Didn’t fix any other problems though. Also not really a production-ready design decision. Why is it still choosing glass? A logical explanation continues to allude me. Everything seems to work but the final product is still a heap of shit!

June 9′th 1929; 8AM

The cigarette idea was a bad one. Polytetrafluoroethylenes main property is that it’s slippery as fuck. I won’t bother with the coefficients, it’s just amazing stuff. Once it started producing again, before any new parts could get to the assembler, one butt shot clear across the room and landed in my hair. Note to self: don’t be an idiot, cigarettes should not be a part of any sound engineering decision. Nothing else in this shop can harness the vibrations of that tad of cotton wrapped in paper, however; so tomorrow during the day I’ll design something similar to a cigarette butt.

I’ve also realized these coatings reach a sweat spot after breaking in. Though the master unit still malfunctions like fuck, due to the slick shit, I can turn it up past ten which blurrs the movements of the finger clusters in my vision. This really helps with motivation to continue. My shoulders hurt. If I had a stroke right now I’d be very pissed.

June 9′th 1929; 9AM RECAP

They day is getting a bit long and I’m afraid all of this noise is going to start attracting some unwanted attention. Will continue tomorrow night. Both Genesis and I are very tired. Oh, the glass. I figured out why it was choosing glass. The entire time it was trying to install vacuum tubes everywhere. Perhaps it decided it needed more logic but wasn’t keeping track of what part of what module of what unit it was constructing.

The final machine we made today did, however, surprise me. When it came out, it turned itself on. Not sure if Genesis wired it that way, or if Genesis somehow evolutionarily developed the machine as I worked on her. We did generate four full iterations last night, not counting the most recent. It’s still a heap of shit, and just sits there. Still, I’ve yet to see how it’s actually built. I tried plugging Genesis’ material harness into it, but it had no valve control and spilled shit all over the floor. Another try, more spilt shit.

I get a different feeling from this machine though, and perhaps I won’t recycle it. It’s no where near as complete as the first successful repro, but seems to have more life. It sits, on. Powering itself much like it’s mother machine, but doing much, much less. I got the feeling when it powered on automatically, it may be that action that continues to prod at me. Or it could be that I’ve recycled four other machines tonight, is it my own guilt?

Most likely, it’s two fucking packs of Lucky Strikes. I need a regular god damn night job.

I was reading a blog, just now, not much unlike a lot of time that I spend reading blogs. Reddit.com is the death knell of productivity and it links to a shit load of blogs. The concept of a blog is great but I’m slowly realizing how stupid they really are. People get their feelings out, they write… and all is well in that regard. But blogs that are written to be read are fucking dumb. No one reads blogs until they have absolutely nothing else to do, eager to soak up more useless information seasoned with useless personal life information. As content, blogs suck.

People however do not read regular news, either. Newspapers are only read by a small microcosm of people… as a film maker I can not even count on people reading title cards. And yet, people read everything. Advertisements anyway, flashy shit that catches their attention in one word or less. I live in San Fran-fucking-cisco and it seems like the advertisement capitol of the world (though I realize that’s not true, many cities have LITTLE or NO regulation on the matter).

Practically every word one sees in this city is intended to sell the viewer something. There is signage everywhere, and it’s insane. I bargain that people’s senses, their ability to read or underlying, subconscious desire to read gets so watered down that when they actually _should_ be reading and _should_ be paying attention that they’re not. Fuckers. It’s subconscious though, and they can’t do anything about it. I just fucking ignore it.

And I certainly don’t expect people to realize what they should be reading.. and what they shouldn’t. No, they would have to be told to.

DON’T READ THIS SIGN

Perhaps it’s the problem with how little people are involved with art, and how it can help one become a better person. One learns an unmeasurable amount about themselves when creating art, no matter what it is, as long as it makes them artistic and they realize what that can do for them. Few people I’ve met other than artists even realize what self expression is, much less how to do it. It doesn’t involve buying new sneakers, or even buying old sneakers. It doesn’t involve buying anything, in fact, and it god damn well shouldn’t.

Pick up an art and start expression your emotions with it. Stick with it for long enough until you no longer are thinking about _the art_ and thinking about yourself. So many people think they can express themselves by how they dress… and mostly what that indicates is that person has absolutely no idea how to express themselves, and this is their closest interpretation. First time expressionists… we’ll call them expressionists, should also not look at blogging as a form of self expression. Choose something that is definitely not on a computer. Choose painting or acting (but for god sakes keep your head straight… remember to express!)

Most bloggers that I know, or not necessarily bloggers… people who write “blogs.” IE: friends, people from school, the regular joe schmoe… like me (?)… will pull up Firefox to write a blog, open tabs for My$psace, F@cebook, Gmail, etc, etc, read through those first, spend hours perhaps re-connecting with their friends they re-connected to last night. Their heads are wrapped up in an entirely wrong space. Social time is not expression time, for me anyway. I don’t believe people can really express themselves in social situations and that is increasingly what computers are becoming for young people.

Wow the thought process was weird on that entry…

If you are planning on blogging, just remember this: shut the fuck up. Just write exactly how you feel, as fast as you can. Force yourself to ignore punctuation, spelling, grammar… whether your writing’ll get you into trouble or not or whether your piece makes sense, those mechanics can get in the way of getting out what you really want to write, perhaps working through an inner mental problem (if you do it this way, writing will help you think).

In Half Life 2: Episode 2, the end arena is an amazing experience and something that (though I don’t claim to have a pulse, really, on the gaming industry) should stand out among all other games. SPOILER [: You are placed in a massive, valley-wide arena to fight a ridiculous onslaught of striders and hunters. Enemies that are used as bosses throughout the series are now trying to destroy the only chance your kind has for survival: and they're pummeling you with many times as many as you'd be expected to handle in the past. :]

The environment is a beautiful wooded mountain valley. There are many completely-destroyable buildings throughout the valley: and they destroy beautifully on a massive scale. A building destroyed by a strider completely breaks down into the many components of buildings-2 x 4s, and other building products that all crash and explode, each piece falling randomly-the animations are not scripted.

After your trouble with the strider/hunter battle, the cinematic experience really takes off. Even though it’s the end of the game, movie mode is put on overdrive. The characters all seem to pour into their performances. You’re knocked to the ground and your camera movement is restricted. Shit goes on around you that you can’t help. Some people may be annoyed by this-but it’s vital. The restricted camera movement, the low angle, you, Gordan Freeman, pinned to the ground and helpless, like a human on the very same level as the rest cast-is all really engaging.

Imagine a movie director: Terry Gilliam, and a movie: 12 Monkeys. By the time of 12 Monkeys, a re-make of an older French experimental film, La Jete, Terry Gilliam is already a brilliant and accomplished director. It is classic of his style to use camera angles to convey a feeling, perhaps that of the protagonist or subject of the scene. Though it’s just not camera angles that contribute to the feel of the Gilliam film-it’s the entire experience. Terry Gilliam creates cinematic experiences.

The game industry has always been one that emphasizes experiences. As games push the boundaries of realism, they will naturally draw upon their past experience as super-naturalistic experiences to become hyper-real: as in movies, they will exaggerate real life in real life terms in an effort to be more engaging and more realistic. Game companies know this, look at The Sims 2. Though they had the capability to create a life simulator, it’s much more fun in the cartoony manor they executed it. The Sims, however, can be analogous to a Judd Apatow (Superbad, 40 Yr Old Virgin) movie. They have some real elements but are largely a projection of real life, an obvious fabrication.

A movie like 12 Monkeys, however, is much more fitting as an analogy for the Half Life series. Sims, as with 40 Yr Old Virgin embodies what our psyche wishes it could be: fun, without care, essentially, as infants. Half Life 2 follows a much more serious thread: world destruction and the destruction of our race. It’s cinematic story line includes charismatic, likable characters, one of whom you get to be. The story line is as thick as it’s directing: you are but a mere human, your entire race is being used as slaves and thinned out, restricted from reproducing - your race backlashes and you can now reproduce, and who is this Alyx character, among the other Gordon admirers, her family and her familys collegues? It’s all rather impressive for a game. Which will, within a minute’s time take you frome near death to flirtatious conversation. Very compelling.

Where am I going with this? No where. I’ve come back to this post at least 10 times, wanted to post it and then haven’t because I have no ending.I’m just hoping games get more and more cinematic and compelling! I’m tired of Doom 3! Fuck Doom 3! What did Doom 3 innovate (or Quake 4 or COD 2,3,4 or Crysis or …)?

Some shots I took from my roof with a Canon XL2. This shouldn’t be looked upon as a full film or anything with a not insignificant amount of thought put into. Just some shots from a rooftop.

These are some pictures I took traveling back from Montana. The first one, a happy accident taken through a window with a UV protection screen at the top.

Out of all the places I’ve been in my life, next to the Bluegrass festival in bumfuck Hamilton, MT, San Francisco is the most invigorating. It has such a great vibe to it. You can’t seem to do anything in the city without running into a really dynamic selection of people. The people all seem to have energy. There are a lot of artists in the city, and this is a serious part of that energy.

You play guitar, don’t you? Everyone in San Francisco plays an instrument.

I took that picture out the window driving on the 101, still across the bay from San Francisco.

I’m back in it.

What I didn’t think was going to work out, ended out working with a lot of help from my parents. I’m moved into an apartment in San Francisco with two room mates in the inner richmond district. Two room mates: JUSTIN MARCUS and CHRIS MULKEY. They dig on movies and shit too.

As it turns out, I did not get more posts in last year than the year previous. Like I had hoped to. Which is fine because I wanted to only write when I felt like it.

Yesterday was the day I went in for a hiring event at the Apple store. Not so sure about the Apple store anymore. The place gives me the creeps. The people there… I’ve never met a group of people so extremely happy with the fact that their brand, in the past 10 or so years went from a 5% market share to… 7. I knew they existed… but I didn’t know they were real. Like, real people that you can touch, and not just weirdos on the internet. Imagine corner of the store filled with 35 digg-reading, macrumors-creeping mother fuckers biting at the chomp for a job. It’s competitive too. One guy had the gall to say he out-qualified everyone there, without a doubt.

Christ, I’d pop my head like it were a zit, working there. Though, you never know, I still might.

What I really want is a job at Revision3, which I’ve just submitted a cover letter and resume to, pointing them to my new SamuraiFilms website. If you’re reading this, and aren’t using a browser that sucks (you know who you are), then check it out. It’s not all the way finished, but I think it gets the point across. The point of the website is the video, so I designed, basically, a video player application. It’s designed for people with big monitors, or at least decent-enough resolution, higher than 1024×768. It’s also designed for people with new browsers (I don’t even know if IE7 works…), Firefox 2, Safari 2+ and a fast connection, 1.0mbps or more. I figure that’s most people who’ll be looking at the site. Also, blip.tv sucks. Their flash encodes are way jumpy and drop a ton of frames. Fuck blip.tv.Also, it seems that Safari’s javascript is quite a bit quicker than Firefox’s, because the animations and such are way smoother on Safari.

I think I’m going to make a film very soon. Perhaps another mob-themed sort of film, or maybe a horror. My apartment is cozy.

I flew back to Montana yesterday (I will be driving back down to SF with all my shit, to move into an apartment, very beginning of next year). It turns out my dad recently got a Cannon 400D which is a pretty sweet camera in my opinion. I took this photo with it, of my dog, Bella. You can click on it for a full sized version (actually, a very scaled down version, boo on flickr).

So for school I did this film. Turned out good. Shot silent on Super 8 in 3 hours - staring friends from San Francisco. It’s not a film for dumb people: you have to pay attention. I hear people like the pacing and mood.

Check it out: