SamuraiBlog.com

This morning I put the finishing touches on the web console and service browser for django-json-rpc. It’s a fully-bundled utility to aid in rapid development of your web service. It’s available on github now and will be a published to PyPI soon as I work out one more interface bug. A live demo is available. Check it out:

Handling JSON in Objective-C is already pretty easy with the excellent strict parser provided by json-framework. However, interacting with a web service in a graphical application can be hard to get right. Frequently unresponsive or unpredictable behavior is a big turnoff for users. Recently I have been developing a lot for the iPhone and more often than not, ideas that beget these applications revolve around central web applications. Many client-server interactions usually boil down to the familiar CRUD apps we’re all so familiar generating with Rails and Django but require much more effort.

Writing applications for a mobile device introduces several challenges: limited memory, slow network and a language and framework designed without an eye on modern agile web services. Recently while writing an iPhone application with heavy web interaction I got lazy doing all that work and wrote DeferredKit – a port of Twisted’s Deferred with many extensions for us Objective-C coders. DeferredKit makes asynchronous programming in Objective-C really easy with very little overhead and includes an asynchronous JSON-RPC client. It allows you to write code like this:

  - (void)doRegistration {
    [[[[[DKDeferred jsonService:SERVICE_URL]
       myApp] register:array_(self.username, self.password)] 
      addCallback:callbackTS(self, doRegistrationCompleted:)]
     addErrback:callbackTS(self, doRegistrationFailed:)];
  }
 
  - (id)doRegistrationCompleted:(id)result {
    // do something with result
    return result;
  }
 
  - (id)doRegistrationFailed:(NSError *)err {
    // do something with error
    return err;
  }

Jumping In

Getting started with DeferredKit is relatively easy. Download the or pull DeferredKit from github and drop everything from DeferredKit/CocoaDeferred/Source into your project (you can also compile DeferredKit as a direct dependency in Xcode, a how-to is in the README).

  id service = [DKDeferred jsonService:@"http://myservice.net/json/" name:@"myapp.sayHello"];

  DKDeferred *d = [service :args_array];

  [d addCallback:callbackTS(self, gotSayHelloResults:)];

  [d addErrback:callbackTS(self, handleSayHelloError:)];

  - (id)gotSayHelloResults:(id)results {
    self.helloLabel.text = [results objectForKey:@"result"];
    return nil;
  }

  - (id)handleSayHelloError:(NSError *)err {
    [self alertError:[[[error userDict] objectForKey:@"error"] objectForKey:@"message"]];
    return nil;
  }

I’ve only had the opportunity to touch on the basics of DeferredKit, but a post is coming soon with more details. It is a tested, lightweight framework for writing asynchronous code and can greatly improve your productivity when working with web services.

In application development, I’ve yet to be the bearer of a product who’s primary functionality has not been aided or entirely supported by a central web application. It is uncommon for developers in this situation to have the luxury of using a common programming language between server and client implementations. A lot of developers will end up rolling their own RPC which evolves over the lifetime of the product to meet its needs. This puts unneeded strain on the developer to fulfill the role of RPC expert as well as interface designer, and expert-everything-else.

I have done many primarily-RPC web applications using tomcat or PHP or twisted, but by far the best way is using Python and Django. However, I have always viewed the state of RPC in Django broken. While using Django for all sorts of other web application purposes is incredibly easy, getting started with RPC presents a lot more friction.

Introducing django-json-rpc, a very easy way to expose your web application to the world of integrated applications.

Installation is as simple as:

  easy_install django-json-rpc

Add a mount point to your urls.py file and import django-json-rpc.

  from jsonrpc import jsonrpc_site
 
  urlpatterns += patterns('', 
    (r'^json/', jsonrpc_site.dispatch)
  )

The interface to django-json-rpc is exposed through only one function – the jsonrpc_method decorator. Functions that use this decorator may be placed anywhere in the source tree and need only import jsonrpc.jsonrpc_method.

  from jsonrpc import jsonrpc_method
 
  @jsonrpc_method('app.sayHello')
  def say_hello(request, name):
    return "Hello, " + name

To hook your views into the default jsonrpc site, they must simply be imported somewhere by Django. The best place for this is in your urls.py file, which should now look something like this:

  from jsonrpc import jsonrpc_site
  import myapp.views
 
  urlpatterns += patterns('', 
    (r'^json/', jsonrpc_site.dispatch)
  )

Test your service with the provided Proxy:

  ./manage.py runserver 8080
 
  ./manage.py shell

  >>> from jsonrpc.proxy import ServiceProxy
  >>> s = ServiceProxy('http://localhost:8080/json/')
  >>> s.app.sayHello('Sam')
  {u'error': None, u'id': u'jsonrpc', u'result': u'Hello Sam'}

HTTP GET

Django-json-rpc supports JSON-RPC version 1.1 which includes support for HTTP GET, meaning, all your REST are belong to us as well. Add the HTTP GET mount point to your urls.py file:

  from jsonrpc import jsonrpc_site
  import myapp.views
 
  urlpatterns += patterns('', 
    (r'^json/', jsonrpc_site.dispatch),
    (r'^json/(?P<method>[a-zA-Z0-9.]+)$', jsonrpc_site.dispatch)
  )
</method>

It is required by django-json-rpc to mark each method safe for dispatch through HTTP GET. jsonrpc_method('sayHello') becomes:

  jsonrpc_method('app.sayHello', safe=True)

Now your method will be available at http://localhost:8080/json/app.sayHello?name=Sam. HTTP GET requests only support string and number typed arguments.

Authentication

The django-json-rpc package also supports authentication, by default using django.contrib.auth’s model backend – the User object we’re all so familiar with, however you may provide any method, including using any authentication middleware. If you aren’t using middleware, username and password arguments will automatically be added to the beginning of the argument list for your method.

  @jsonrpc_method('app.sayHello', authenticated=True)
  def say_hello(request):
    return "Hello, " + request.user.first_name

  >>> s.app.sayHello('samuraiblog', 'password')
  {u'error': None, u'id': u'jsonrpc', u'result': u'Hello Sam'}

Version Agnostic

django-json-rpc will continue to work regardless of which version JSON-RPC client you happen to be using. It supports all argument types supported in JSON-RPC versions 1.0, 1.1 and 2.0, and will respond if a client specifies the version in either the jsonrpc or version key.

Conclusion

I have used this package for the development of several commercial iPhone applications and it has greatly improved my productivity in developing a web service backend. Up next? A django-xmlrpc? Haha! Fuck that!

This entirely iPhone friendly tutorial highlights how easy it is to compress our familiar NSData objects using zlib – the most commonly used compression library on the planet. This is the same library that compresses the PNGs and PDFs, ZIPs and GZIPed files that make up your daily life.

The zlib c library, which is bundled with all Mac OS X and iPhone 2.0+ devices offers an extensive but non obtrusive API for compressing data. The most basic functions are the stream API which serve nicely when, for instance, dealing with a large files, or matching a specific compression requirement. The method presented here is one which is utility provided by the atop the streaming API. Readers familiar with python will recognize this if ever you’ve had to handle data with the zlib module.

First, Allocate a data structure of type const ByteF containing the bytes in our uncompressed NSData structure.

  const Bytef *inBytes = (const Bytef *)[uncompressedData bytes];

Allocate enough data to start compressing data. The zlib manual recommends allocating 1% more than the length of your uncompressed data, plus twelve bytes. This gives zlib enough room to start compressing data, resizing the output buffer as needed.

  uLongf outLength = ([uncompressedData length] * 1.1) + 12;
  Bytef *outBytes = (Bytef *)malloc(outLength);

Now simply call compress:

  int z_result = compress(outBytes, &outLength, inBytes, [uncompressedData length]);

compress will return one of three constants. To handle errors, check for the return value. Depending on your situation, you could return an NSError. In this case, I return an empty NSData and check for return value length.

  NSData *ret;
  if (z_result == Z_OK) {
    ret = [NSData dataWithBytesNoCopy:outBytes length:outLength freeWhenDone:YES];
  } else {
    ret = [NSData data];
    switch (z_result) {
      case Z_MEM_ERROR:
        printf("compressData got Z_MEM_ERROR out of memory. :(");
      case Z_BUF_ERROR:
        printf("compressData got Z_BUF_ERROR output buffer wasn't larege enough :(");
      default:
        break;
    }
  }
}

compressData function

NSData* compressData(NSData *uncompressedData) {
  if ([uncompressedData length] == 0) 
    return uncompressedData;
  const Bytef *inBytes = (const Bytef *)[uncompressedData bytes];
  uLongf outLength = ([uncompressedData length] * 1.1) + 12;
  Bytef *outBytes = (Bytef *)malloc(outLength);
  int z_result = compress(outBytes, &outLength, inBytes, [uncompressedData length]);
  NSData *ret;
  if (z_result == Z_OK) {
    ret = [NSData dataWithBytesNoCopy:outBytes length:outLength freeWhenDone:YES];
  } else {
    ret = [NSData data];
    switch (z_result) {
      case Z_MEM_ERROR:
        printf("compressData got Z_MEM_ERROR out of memory. :(");
      case Z_BUF_ERROR:
        printf("compressData got Z_BUF_ERROR output buffer wasn't larege enough :(");
      default:
        break;
    }
  }
  return ret;
}

compressData that returns an NSError.

#define ZLIB_COMPRESS_DOMAIN @"zlib_compress_domain"
 
NSError* compressData2(NSData *inData, NSData **outData) {
  if ([inData length] == 0) {
    *outData = inData;
    return nil;
  }
  const Bytef *inBytes = (const Bytef *)[inData bytes];
  uLongf outLength = ([inData length] * 1.1) + 12;
  Bytef *outBytes = (Bytef *)malloc(outLength);
  int z_result = compress(outBytes, &outLength, inBytes, [inData length]); 
  if (z_result == Z_OK) {
    *outData = [NSData dataWithBytesNoCopy:outBytes length:outLength freeWhenDone:YES];
    free(outBytes);
    return nil;
  }
  return [NSError errorWithDomain:ZLIB_COMPRESS_DOMAIN 
    code:z_result userInfo:[NSDictionary dictionary]];
}

NSData category

@interface NSData (CompressionAdditions)
 
- (NSData *)compress;
- (NSData *)compress:(NSError **)errorOut;
 
@end
 
 
@implementation NSData (CompressionAdditions)
 
- (NSData *)compress { 
  return compressData(self);
}
 
- (NSData *)compress:(NSError **)errorOut {
  NSData *ret;
  *errorOut = compressData2(self, &ret);
  return ret;
}
 
@end

Twisted is a python package for building asynchronous applications. It’s wide range of uses and sub-packages are daunting; I’ve developed with Twisted for several years now and still find it’s vast expanse of code intimidating. Twisted in 60 seconds is an excellent resource for anyone beginning with twisted and twisted.web. Here, we’re filling yet another gap in the twisted documentation.

Not surprisingly, handling file uploads in twisted.web is straight forward. Files are accessed through the args attribute of the request parameter to your render method. Since there can be multiple files for that given field, whether there are one or more, request.args[FIELD_NAME] will always be an array.

from twisted.internet import reactor
from twisted.web import resource, server
 
class FileUploadService(resource.Resource):  
  def render_GET(self, request):
    return '''<form enctype="multipart/form-data" method="post" action=".">
                 <input type="file" name="da_file"/>
                 <input type="submit"/>
              </form>'''
 
  def render_POST(self, request):
    return 'You submitted a file that was %i bytes' % len(request.args['da_file'][0])
 
root = resource.Resource()
root.putChild("upload", FileUploadService())
factory = server.Site(root)
reactor.listenTCP(8088, factory)
reactor.run()

According to the twisted.web documentation. This method of handling file uploads is not preferred. In fact, the preferred way is to use the more complex twisted.web2 package which supports streaming uploads. However, the twisted devs are working on porting most of twisted.web2’s features to twisted.web in an effort to consolidate the two. In production code, I have chosen to stick with twisted.web since it seems to be the package which will remain the most compatible with future versions of Twisted.

Have you ever wanted to put up a server made exclusively to handle the expensive actions in your web app? In this article I will be showing you how to create a server to that will be set up to receive commands from your web server and process them in the context of your django web app. This technique could be used many places such as a desktop GUI application, or a web service architecture.

Say you have work unit WU which takes pickleable arguments (a, b, c). If your work unit is thread-safe (it modifies no global variables, and causes no side effects in your application) it can be preformed on in another process, or even on another machine, allowing you to return control to the user immediately.

For example, I want to add the ability for a user to import their contacts using X contacts service:

Fig 1: project/views.py

1
2
3
4

def import_contacts(request):
    from my.specialsauce.contacts import run_contacts_import
    contacts = run_contacts_import(request)
    return render_to_response('t/contacts-import.html', {'contacts': contacts})

Currently, the view retrieves and imports the users contacts in the view, making the user wait until the operation has completed to see the returned page. What if we could implement this in a way which would return control to the user immediately, firing off run_contacts_import in another thread, another process, or even another server!

To do this we will need Django, multiprocessing and Python 2.5+. Now, I will assume you are already as far as Fig 1. Say a user with 10,000 contacts (or even 100) ambles along and imports their contacts. Both users are going to be waiting much longer than the average gmail-generation-web-user can be expected to wait. They will hit refresh, restarting this process and crashing your server.

Since the Django request object is pickleable, we can pass it as an argument to another process using multiprocessing.

Fig 2: project/views.py

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

from multiprocessing import Pool
 
m = Pool(10)
 
def import_contacts(request):
    # run_contacts_import will have to update request.session['CONTACT_IMPORT_STATUS']
    has_status = 'CONTACT_IMPORT_STATUS' in request.session
    status = has_status and request.session['CONTACT_IMPORT_STATUS'] == True
    if status is True:
        return HttpResponseRedirect('/import-finished/')
    elif not status and has_status:
        return render_to_response('t/contacts-import-not-finished.html', {})
    elif not has_status:
        from my.specialsauce.contacts import run_contacts_import
        m.apply_async(run_contacts_import, args=(request,))
        return render_to_response('t/contacts-import.html', {'contacts': contacts})

This seems simple enough, and upon first inspection, seems to work rather decently. However, this adds a global variable to our Django process, which is a bad idea. Using this technique, it would be best to create a lock, spawn off a new thread, and acquire/release the lock while running apply_async (blegh). Unfortunately, this can eventually make your Django server unresponsive as many users come to run the expensive process (simply via system load). We need to move this process, and possibly a few others off our new server.

To do this, the Python multiprocessing package provides the high-level Manager class. The Manager class is used to pipe commands between managers in different instances of the python interpreter.

Create the Manager Class

First, we need to create the Manager class that will eventually be implemented as our server. Create a file called managers.py in project/managers.py.

Fig 3: project/managers.py

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

from multiprocessing.managers import BaseManager
 
from my.specialsauce.contacts import run_contacts_import
 
# This is the shared object. It simply implements a function which applies to the instance variable
# pool, which will be installed when the server is actually started (in a snippet below)
class ContactImporter(object):
    def __init__(self, pool=None):
        self.pool = pool
 
    def run_import(self, *args):
        self.pool.apply_async(run_contacts_import, args=args)
 
# instantiate the shared object
importer = ContactImporter()
 
# our manager
class CManager(BaseManager):
    pass
 
# you need to register a function which will return the shared object
# the shared object could be anything pickleable in python, in this case
# our new-style class descending from object
CManager.register('get_importer', callable=lambda:importer)

Create the Server

We are going to be making a daemon server that we can start and stop using your projects manage.py file. To do this create a file called lrpserver.py in project/management/commands/lrpserver.py.

Fig 4: project/management/commands/lrpserver.py

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90

import os
import sys
import time
from optparse import make_option
from signal import SIGTERM
 
from multiprocessing import Pool
 
from django.core.management.base import BaseCommand
from django.conf import settings
 
from project.managers import CManager, importer
 
 
class Command(BaseCommand):
    option_list = BaseCommand.option_list + (
        make_option('--pidfile', dest='pidfile', default='', help="Specifies the PID file to use."),
        make_option('--start', dest='start', default='no', help='Starts the daemon'),
        make_option('--stop', dest='stop', default='no', help='Stops the daemon'),
        make_option('--restart', dest='restart', default='no', help='Restarts the daemon')
    )
    def handle(self, *args, **options):
        pidfile = options.get('pidfile')
        if not pidfile:
            print "--pidfile arg required"
            sys.exit(1)
        start = options.get('start')
        stop = options.get('stop')
        restart = options.get('restart')
        server = getattr(settings, 'LRP_SERVER_HOST', None)
        port = getattr(settings, 'LRP_SERVER_PORT', None)
        authkey = getattr(settings, 'LRP_SERVER_AUTHKEY', None)
 
        if server is None or port is None or authkey is None:
            print 'LRP_SERVER_HOST, LRP_SERVER_PORT and LRP_SERVER_AUTHKEY values required in settings file.'
            sys.exit(1)
 
        # This daemonizes our process and starts the server
        def _start():
            print 'Starting Long Running Process Server'
            from django.utils.daemonize import become_daemon
            become_daemon()
            fp = open(pidfile, "w")
            fp.write('%d\n' % os.getpid())
            fp.close()
 
            # set the pool that importer.run_import requires
            importer.pool = Pool(processes=10)
            # create an instance of CManager, get a server instance and start the server loop
            m = CManager(address=(server, port), authkey=authkey)
            s = m.get_server()
            s.serve_forever()
 
        def _stop():
            try:
                fp = open(pidfile, 'r')
                pid = int(fp.read().strip())
                fp.close()
            except IOError:
                pid = None
            if not pid:
                print 'Long Running Process Server Not Currently Running'
                return
            try:
                print 'Stopping Long Running Process Server'
                while 1:
                    os.kill(pid, SIGTERM)
                    time.sleep(0.1)
            except OSError, err:
                err = str(err)
                if err.lower().find('no such process') > 0:
                    if os.path.exists(pidfile):
                        os.remove(pidfile)
                else:
                    print err
                    sys.exit(1)
 
        def _restart():
            _stop()
            _start()
 
        if str(start) == 'yes':
            return _start()
        elif str(stop) == 'yes':
            return _stop()
        elif str(restart) == 'yes':
            return _restart()
        else:
            print 'Options: pidfile=%s start=%s stop=%s restart=%s' % (pidfile, start, stop, restart)
            print 'One of --(start|stop|restart)=yes is required.'

Add the Required Settings to Your settings.py File

The server and client require three settings, the host name of your server, the port on which the server is listening and the auth key of the required to access the server. In your settings.py file add the following with your values. For now, use the blank string ” as the setting for host name. This instructs the server to serve from localhost. (Hint: If you do not know this value use socket.getfqdn()).

Fig 5: settings.py

1
2
3

LRP_SERVER_HOST = ''
LRP_SERVER_PORT = 5667
LRP_SERVER_AUTHKEY = 'password'

Modify Your View Code

To connect to the server, you must spawn a thread which creates an instance of the manager, connects and sends the command.

Fig 6: project/views.py

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39

from threading import Thread
from multiprocessing.managers import BaseManager
 
from django.conf import settings
 
 
class CManager(BaseManager):
    pass
 
 
CManager.register('get_importer')
 
# This function will become a thread which instantiates the Manager class,
# connects and starts the long running process
def _start_import_thread(request):
    gci_manager = CManager(address=(settings.GCI_SERVER_HOST, settings.GCI_SERVER_PORT),
                           authkey=settings.GCI_SERVER_AUTHKEY)
    gci_manager.connect()
    importer = gci_manager.get_importer()
    importer.run_import(request)
 
# This is not a view, it's a helper function which starts
# the thread above
def start_contacts_import(request):
    t = Thread(target=_start_import_thread, args=[request])
    t.setDaemon(True)
    t.start()
 
def import_contacts(request):
    # run_contacts_import will have to update request.session['CONTACT_IMPORT_STATUS']
    has_status = 'CONTACT_IMPORT_STATUS' in request.session
    status = has_status and request.session['CONTACT_IMPORT_STATUS'] == True
    if status is True:
        return HttpResponseRedirect('/import-finished/')
    elif not status and has_status:
        return render_to_response('t/contacts-import-not-finished.html', {})
    elif not has_status:
        start_contacts_import(request)
        return render_to_response('t/contacts-import.html', {'contacts': contacts})

Start the Server and Let it Fly!

And finally we may start the server.

1

python manage.py lrpserver.py --pidfile=/home/django/run/lrp.pid --start=yes

Now you can run your view.

Follow-Up

This process is applicable to more than just crawlers and contact importers. It can be used to minimize the load on your server for any number of tasks such as image and other file processing or running large database updates. The server can be put on another machine simply by duplicating your django installation on another machine and using that machine’s socket.getfqdn() (or ip address) value as the host name in your configs. You can also add a javascript status updater to notify the user of progress information related to their request.

Boneless creature of iron and oil,
Legless, headless and mindful of all.
Another series of rooms a-boil,
To chamber my froth, life, at your call.

This is my first creative writing post. It’s not much right now, as I’m still trying to figure out my first assignment. It seems I have some time today to write. No work until Thursday at 3P. Good. Poems are cool to write. :)

Genesis Log Page 43

June 9′th 1929; 1:30 AM

Twenty years ago today, I began building a machine. Now, nearly the size of a $2/hr room in San Francisco, I have finally been able to name this monster with a name signifying it’s being. Genesis. Why? Fuck you, that’s why. I call it Genesis.

Sure, that may be a piss-poor explanation, but this machine is beyond the realm of revolutionary. Fuck, it can build itself! Well, since last night that statement has been false. There’s a guy I met who owns a metal shop. Genesis, in big, brassy letters. On the door or awning or something–and it wouldn’t be so much for the letters, because obviously I could handle that myself, as to, finally validate our existence to the world–but not until this bitch is again reproducing.

June 9′th 1929; Around 3 AM

I’ve stopped working on the machine for a few minutes to log my progress and have a cigarette. Along the way I’ve discovered the master unit started binding on all metal materials. Explanation at this point is beyond me. Note to self: Discover a method to minimize vibrations in the cutting module near the back end of master unit without disturbing the spacing of lower modules (which would of course throw timing off too).

June 9′th 1929; 4:30 AM

Another smoke and log break. Two smokes, and now for the log. I should really get an eraser. I tried coating a fraction (every other twice) of the fingers (in the master unit, both main and secondary clusters) in a light polytetrafluoroethylene, but did too many. The machine dropped the first production unit and spilled molten glass all over the conveyance, which is luckily made of iron slats. The scrap pan below caught the rest. I cleaned the glass. And I still have no fucking clue why it was loading glass. Maybe it’s getting a bit late.

June 9′th 1929; 5AM

Another smoke. This is killing me. Not the smokes, they are too, it’s the nicotine. I guess it is the smokes

June 9′th 1929; Around 6AM

During my last log entry I had an idea regarding my smoking habit. I took four cigarette butts and coated them with the polytetrafluoroethylene and wedged them under plates that mount the cutting module. No more vibrations. Rather, few enough, and muted enough, that it can manage. Didn’t fix any other problems though. Also not really a production-ready design decision. Why is it still choosing glass? A logical explanation continues to allude me. Everything seems to work but the final product is still a heap of shit!

June 9′th 1929; 8AM

The cigarette idea was a bad one. Polytetrafluoroethylenes main property is that it’s slippery as fuck. I won’t bother with the coefficients, it’s just amazing stuff. Once it started producing again, before any new parts could get to the assembler, one butt shot clear across the room and landed in my hair. Note to self: don’t be an idiot, cigarettes should not be a part of any sound engineering decision. Nothing else in this shop can harness the vibrations of that tad of cotton wrapped in paper, however; so tomorrow during the day I’ll design something similar to a cigarette butt.

I’ve also realized these coatings reach a sweat spot after breaking in. Though the master unit still malfunctions like fuck, due to the slick shit, I can turn it up past ten which blurrs the movements of the finger clusters in my vision. This really helps with motivation to continue. My shoulders hurt. If I had a stroke right now I’d be very pissed.

June 9′th 1929; 9AM RECAP

They day is getting a bit long and I’m afraid all of this noise is going to start attracting some unwanted attention. Will continue tomorrow night. Both Genesis and I are very tired. Oh, the glass. I figured out why it was choosing glass. The entire time it was trying to install vacuum tubes everywhere. Perhaps it decided it needed more logic but wasn’t keeping track of what part of what module of what unit it was constructing.

The final machine we made today did, however, surprise me. When it came out, it turned itself on. Not sure if Genesis wired it that way, or if Genesis somehow evolutionarily developed the machine as I worked on her. We did generate four full iterations last night, not counting the most recent. It’s still a heap of shit, and just sits there. Still, I’ve yet to see how it’s actually built. I tried plugging Genesis’ material harness into it, but it had no valve control and spilled shit all over the floor. Another try, more spilt shit.

I get a different feeling from this machine though, and perhaps I won’t recycle it. It’s no where near as complete as the first successful repro, but seems to have more life. It sits, on. Powering itself much like it’s mother machine, but doing much, much less. I got the feeling when it powered on automatically, it may be that action that continues to prod at me. Or it could be that I’ve recycled four other machines tonight, is it my own guilt?

Most likely, it’s two fucking packs of Lucky Strikes. I need a regular god damn night job.

I was reading a blog, just now, not much unlike a lot of time that I spend reading blogs. Reddit.com is the death knell of productivity and it links to a shit load of blogs. The concept of a blog is great but I’m slowly realizing how stupid they really are. People get their feelings out, they write… and all is well in that regard. But blogs that are written to be read are fucking dumb. No one reads blogs until they have absolutely nothing else to do, eager to soak up more useless information seasoned with useless personal life information. As content, blogs suck.

People however do not read regular news, either. Newspapers are only read by a small microcosm of people… as a film maker I can not even count on people reading title cards. And yet, people read everything. Advertisements anyway, flashy shit that catches their attention in one word or less. I live in San Fran-fucking-cisco and it seems like the advertisement capitol of the world (though I realize that’s not true, many cities have LITTLE or NO regulation on the matter).

Practically every word one sees in this city is intended to sell the viewer something. There is signage everywhere, and it’s insane. I bargain that people’s senses, their ability to read or underlying, subconscious desire to read gets so watered down that when they actually _should_ be reading and _should_ be paying attention that they’re not. Fuckers. It’s subconscious though, and they can’t do anything about it. I just fucking ignore it.

And I certainly don’t expect people to realize what they should be reading.. and what they shouldn’t. No, they would have to be told to.

DON’T READ THIS SIGN

Perhaps it’s the problem with how little people are involved with art, and how it can help one become a better person. One learns an unmeasurable amount about themselves when creating art, no matter what it is, as long as it makes them artistic and they realize what that can do for them. Few people I’ve met other than artists even realize what self expression is, much less how to do it. It doesn’t involve buying new sneakers, or even buying old sneakers. It doesn’t involve buying anything, in fact, and it god damn well shouldn’t.

Pick up an art and start expression your emotions with it. Stick with it for long enough until you no longer are thinking about _the art_ and thinking about yourself. So many people think they can express themselves by how they dress… and mostly what that indicates is that person has absolutely no idea how to express themselves, and this is their closest interpretation. First time expressionists… we’ll call them expressionists, should also not look at blogging as a form of self expression. Choose something that is definitely not on a computer. Choose painting or acting (but for god sakes keep your head straight… remember to express!)

Most bloggers that I know, or not necessarily bloggers… people who write “blogs.” IE: friends, people from school, the regular joe schmoe… like me (?)… will pull up Firefox to write a blog, open tabs for My$psace, F@cebook, Gmail, etc, etc, read through those first, spend hours perhaps re-connecting with their friends they re-connected to last night. Their heads are wrapped up in an entirely wrong space. Social time is not expression time, for me anyway. I don’t believe people can really express themselves in social situations and that is increasingly what computers are becoming for young people.

Wow the thought process was weird on that entry…

If you are planning on blogging, just remember this: shut the fuck up. Just write exactly how you feel, as fast as you can. Force yourself to ignore punctuation, spelling, grammar… whether your writing’ll get you into trouble or not or whether your piece makes sense, those mechanics can get in the way of getting out what you really want to write, perhaps working through an inner mental problem (if you do it this way, writing will help you think).

In Half Life 2: Episode 2, the end arena is an amazing experience and something that (though I don’t claim to have a pulse, really, on the gaming industry) should stand out among all other games. SPOILER [: You are placed in a massive, valley-wide arena to fight a ridiculous onslaught of striders and hunters. Enemies that are used as bosses throughout the series are now trying to destroy the only chance your kind has for survival: and they're pummeling you with many times as many as you'd be expected to handle in the past. :]

The environment is a beautiful wooded mountain valley. There are many completely-destroyable buildings throughout the valley: and they destroy beautifully on a massive scale. A building destroyed by a strider completely breaks down into the many components of buildings-2 x 4s, and other building products that all crash and explode, each piece falling randomly-the animations are not scripted.

After your trouble with the strider/hunter battle, the cinematic experience really takes off. Even though it’s the end of the game, movie mode is put on overdrive. The characters all seem to pour into their performances. You’re knocked to the ground and your camera movement is restricted. Shit goes on around you that you can’t help. Some people may be annoyed by this-but it’s vital. The restricted camera movement, the low angle, you, Gordan Freeman, pinned to the ground and helpless, like a human on the very same level as the rest cast-is all really engaging.

Imagine a movie director: Terry Gilliam, and a movie: 12 Monkeys. By the time of 12 Monkeys, a re-make of an older French experimental film, La Jete, Terry Gilliam is already a brilliant and accomplished director. It is classic of his style to use camera angles to convey a feeling, perhaps that of the protagonist or subject of the scene. Though it’s just not camera angles that contribute to the feel of the Gilliam film-it’s the entire experience. Terry Gilliam creates cinematic experiences.

The game industry has always been one that emphasizes experiences. As games push the boundaries of realism, they will naturally draw upon their past experience as super-naturalistic experiences to become hyper-real: as in movies, they will exaggerate real life in real life terms in an effort to be more engaging and more realistic. Game companies know this, look at The Sims 2. Though they had the capability to create a life simulator, it’s much more fun in the cartoony manor they executed it. The Sims, however, can be analogous to a Judd Apatow (Superbad, 40 Yr Old Virgin) movie. They have some real elements but are largely a projection of real life, an obvious fabrication.

A movie like 12 Monkeys, however, is much more fitting as an analogy for the Half Life series. Sims, as with 40 Yr Old Virgin embodies what our psyche wishes it could be: fun, without care, essentially, as infants. Half Life 2 follows a much more serious thread: world destruction and the destruction of our race. It’s cinematic story line includes charismatic, likable characters, one of whom you get to be. The story line is as thick as it’s directing: you are but a mere human, your entire race is being used as slaves and thinned out, restricted from reproducing – your race backlashes and you can now reproduce, and who is this Alyx character, among the other Gordon admirers, her family and her familys collegues? It’s all rather impressive for a game. Which will, within a minute’s time take you frome near death to flirtatious conversation. Very compelling.

Where am I going with this? No where. I’ve come back to this post at least 10 times, wanted to post it and then haven’t because I have no ending.I’m just hoping games get more and more cinematic and compelling! I’m tired of Doom 3! Fuck Doom 3! What did Doom 3 innovate (or Quake 4 or COD 2,3,4 or Crysis or …)?

Some shots I took from my roof with a Canon XL2. This shouldn’t be looked upon as a full film or anything with a not insignificant amount of thought put into. Just some shots from a rooftop.