SamuraiBlog.com

This morning I put the finishing touches on the web console and service browser for django-json-rpc. It’s a fully-bundled utility to aid in rapid development of your web service. It’s available on github now and will be a published to PyPI soon as I work out one more interface bug. A live demo is available. Check it out:

Handling JSON in Objective-C is already pretty easy with the excellent strict parser provided by json-framework. However, interacting with a web service in a graphical application can be hard to get right. Frequently unresponsive or unpredictable behavior is a big turnoff for users. Recently I have been developing a lot for the iPhone and more often than not, ideas that beget these applications revolve around central web applications. Many client-server interactions usually boil down to the familiar CRUD apps we’re all so familiar generating with Rails and Django but require much more effort.

Writing applications for a mobile device introduces several challenges: limited memory, slow network and a language and framework designed without an eye on modern agile web services. Recently while writing an iPhone application with heavy web interaction I got lazy doing all that work and wrote DeferredKit – a port of Twisted’s Deferred with many extensions for us Objective-C coders. DeferredKit makes asynchronous programming in Objective-C really easy with very little overhead and includes an asynchronous JSON-RPC client. It allows you to write code like this:

  - (void)doRegistration {
    [[[[[DKDeferred jsonService:SERVICE_URL]
       myApp] register:array_(self.username, self.password)] 
      addCallback:callbackTS(self, doRegistrationCompleted:)]
     addErrback:callbackTS(self, doRegistrationFailed:)];
  }
 
  - (id)doRegistrationCompleted:(id)result {
    // do something with result
    return result;
  }
 
  - (id)doRegistrationFailed:(NSError *)err {
    // do something with error
    return err;
  }

Jumping In

Getting started with DeferredKit is relatively easy. Download the or pull DeferredKit from github and drop everything from DeferredKit/CocoaDeferred/Source into your project (you can also compile DeferredKit as a direct dependency in Xcode, a how-to is in the README).

  id service = [DKDeferred jsonService:@"http://myservice.net/json/" name:@"myapp.sayHello"];

  DKDeferred *d = [service :args_array];

  [d addCallback:callbackTS(self, gotSayHelloResults:)];

  [d addErrback:callbackTS(self, handleSayHelloError:)];

  - (id)gotSayHelloResults:(id)results {
    self.helloLabel.text = [results objectForKey:@"result"];
    return nil;
  }

  - (id)handleSayHelloError:(NSError *)err {
    [self alertError:[[[error userDict] objectForKey:@"error"] objectForKey:@"message"]];
    return nil;
  }

I’ve only had the opportunity to touch on the basics of DeferredKit, but a post is coming soon with more details. It is a tested, lightweight framework for writing asynchronous code and can greatly improve your productivity when working with web services.

In application development, I’ve yet to be the bearer of a product who’s primary functionality has not been aided or entirely supported by a central web application. It is uncommon for developers in this situation to have the luxury of using a common programming language between server and client implementations. A lot of developers will end up rolling their own RPC which evolves over the lifetime of the product to meet its needs. This puts unneeded strain on the developer to fulfill the role of RPC expert as well as interface designer, and expert-everything-else.

I have done many primarily-RPC web applications using tomcat or PHP or twisted, but by far the best way is using Python and Django. However, I have always viewed the state of RPC in Django broken. While using Django for all sorts of other web application purposes is incredibly easy, getting started with RPC presents a lot more friction.

Introducing django-json-rpc, a very easy way to expose your web application to the world of integrated applications.

Installation is as simple as:

  easy_install django-json-rpc

Add a mount point to your urls.py file and import django-json-rpc.

  from jsonrpc import jsonrpc_site
 
  urlpatterns += patterns('', 
    (r'^json/', jsonrpc_site.dispatch)
  )

The interface to django-json-rpc is exposed through only one function – the jsonrpc_method decorator. Functions that use this decorator may be placed anywhere in the source tree and need only import jsonrpc.jsonrpc_method.

  from jsonrpc import jsonrpc_method
 
  @jsonrpc_method('app.sayHello')
  def say_hello(request, name):
    return "Hello, " + name

To hook your views into the default jsonrpc site, they must simply be imported somewhere by Django. The best place for this is in your urls.py file, which should now look something like this:

  from jsonrpc import jsonrpc_site
  import myapp.views
 
  urlpatterns += patterns('', 
    (r'^json/', jsonrpc_site.dispatch)
  )

Test your service with the provided Proxy:

  ./manage.py runserver 8080
 
  ./manage.py shell

  >>> from jsonrpc.proxy import ServiceProxy
  >>> s = ServiceProxy('http://localhost:8080/json/')
  >>> s.app.sayHello('Sam')
  {u'error': None, u'id': u'jsonrpc', u'result': u'Hello Sam'}

HTTP GET

Django-json-rpc supports JSON-RPC version 1.1 which includes support for HTTP GET, meaning, all your REST are belong to us as well. Add the HTTP GET mount point to your urls.py file:

  from jsonrpc import jsonrpc_site
  import myapp.views
 
  urlpatterns += patterns('', 
    (r'^json/', jsonrpc_site.dispatch),
    (r'^json/(?P<method>[a-zA-Z0-9.]+)$', jsonrpc_site.dispatch)
  )
</method>

It is required by django-json-rpc to mark each method safe for dispatch through HTTP GET. jsonrpc_method('sayHello') becomes:

  jsonrpc_method('app.sayHello', safe=True)

Now your method will be available at http://localhost:8080/json/app.sayHello?name=Sam. HTTP GET requests only support string and number typed arguments.

Authentication

The django-json-rpc package also supports authentication, by default using django.contrib.auth’s model backend – the User object we’re all so familiar with, however you may provide any method, including using any authentication middleware. If you aren’t using middleware, username and password arguments will automatically be added to the beginning of the argument list for your method.

  @jsonrpc_method('app.sayHello', authenticated=True)
  def say_hello(request):
    return "Hello, " + request.user.first_name

  >>> s.app.sayHello('samuraiblog', 'password')
  {u'error': None, u'id': u'jsonrpc', u'result': u'Hello Sam'}

Version Agnostic

django-json-rpc will continue to work regardless of which version JSON-RPC client you happen to be using. It supports all argument types supported in JSON-RPC versions 1.0, 1.1 and 2.0, and will respond if a client specifies the version in either the jsonrpc or version key.

Conclusion

I have used this package for the development of several commercial iPhone applications and it has greatly improved my productivity in developing a web service backend. Up next? A django-xmlrpc? Haha! Fuck that!

This entirely iPhone friendly tutorial highlights how easy it is to compress our familiar NSData objects using zlib – the most commonly used compression library on the planet. This is the same library that compresses the PNGs and PDFs, ZIPs and GZIPed files that make up your daily life.

The zlib c library, which is bundled with all Mac OS X and iPhone 2.0+ devices offers an extensive but non obtrusive API for compressing data. The most basic functions are the stream API which serve nicely when, for instance, dealing with a large files, or matching a specific compression requirement. The method presented here is one which is utility provided by the atop the streaming API. Readers familiar with python will recognize this if ever you’ve had to handle data with the zlib module.

First, Allocate a data structure of type const ByteF containing the bytes in our uncompressed NSData structure.

  const Bytef *inBytes = (const Bytef *)[uncompressedData bytes];

Allocate enough data to start compressing data. The zlib manual recommends allocating 1% more than the length of your uncompressed data, plus twelve bytes. This gives zlib enough room to start compressing data, resizing the output buffer as needed.

  uLongf outLength = ([uncompressedData length] * 1.1) + 12;
  Bytef *outBytes = (Bytef *)malloc(outLength);

Now simply call compress:

  int z_result = compress(outBytes, &outLength, inBytes, [uncompressedData length]);

compress will return one of three constants. To handle errors, check for the return value. Depending on your situation, you could return an NSError. In this case, I return an empty NSData and check for return value length.

  NSData *ret;
  if (z_result == Z_OK) {
    ret = [NSData dataWithBytesNoCopy:outBytes length:outLength freeWhenDone:YES];
  } else {
    ret = [NSData data];
    switch (z_result) {
      case Z_MEM_ERROR:
        printf("compressData got Z_MEM_ERROR out of memory. :(");
      case Z_BUF_ERROR:
        printf("compressData got Z_BUF_ERROR output buffer wasn't larege enough :(");
      default:
        break;
    }
  }
}

compressData function

NSData* compressData(NSData *uncompressedData) {
  if ([uncompressedData length] == 0) 
    return uncompressedData;
  const Bytef *inBytes = (const Bytef *)[uncompressedData bytes];
  uLongf outLength = ([uncompressedData length] * 1.1) + 12;
  Bytef *outBytes = (Bytef *)malloc(outLength);
  int z_result = compress(outBytes, &outLength, inBytes, [uncompressedData length]);
  NSData *ret;
  if (z_result == Z_OK) {
    ret = [NSData dataWithBytesNoCopy:outBytes length:outLength freeWhenDone:YES];
  } else {
    ret = [NSData data];
    switch (z_result) {
      case Z_MEM_ERROR:
        printf("compressData got Z_MEM_ERROR out of memory. :(");
      case Z_BUF_ERROR:
        printf("compressData got Z_BUF_ERROR output buffer wasn't larege enough :(");
      default:
        break;
    }
  }
  return ret;
}

compressData that returns an NSError.

#define ZLIB_COMPRESS_DOMAIN @"zlib_compress_domain"
 
NSError* compressData2(NSData *inData, NSData **outData) {
  if ([inData length] == 0) {
    *outData = inData;
    return nil;
  }
  const Bytef *inBytes = (const Bytef *)[inData bytes];
  uLongf outLength = ([inData length] * 1.1) + 12;
  Bytef *outBytes = (Bytef *)malloc(outLength);
  int z_result = compress(outBytes, &outLength, inBytes, [inData length]); 
  if (z_result == Z_OK) {
    *outData = [NSData dataWithBytesNoCopy:outBytes length:outLength freeWhenDone:YES];
    free(outBytes);
    return nil;
  }
  return [NSError errorWithDomain:ZLIB_COMPRESS_DOMAIN 
    code:z_result userInfo:[NSDictionary dictionary]];
}

NSData category

@interface NSData (CompressionAdditions)
 
- (NSData *)compress;
- (NSData *)compress:(NSError **)errorOut;
 
@end
 
 
@implementation NSData (CompressionAdditions)
 
- (NSData *)compress { 
  return compressData(self);
}
 
- (NSData *)compress:(NSError **)errorOut {
  NSData *ret;
  *errorOut = compressData2(self, &ret);
  return ret;
}
 
@end

Twisted is a python package for building asynchronous applications. It’s wide range of uses and sub-packages are daunting; I’ve developed with Twisted for several years now and still find it’s vast expanse of code intimidating. Twisted in 60 seconds is an excellent resource for anyone beginning with twisted and twisted.web. Here, we’re filling yet another gap in the twisted documentation.

Not surprisingly, handling file uploads in twisted.web is straight forward. Files are accessed through the args attribute of the request parameter to your render method. Since there can be multiple files for that given field, whether there are one or more, request.args[FIELD_NAME] will always be an array.

from twisted.internet import reactor
from twisted.web import resource, server
 
class FileUploadService(resource.Resource):  
  def render_GET(self, request):
    return '''<form enctype="multipart/form-data" method="post" action=".">
                 <input type="file" name="da_file"/>
                 <input type="submit"/>
              </form>'''
 
  def render_POST(self, request):
    return 'You submitted a file that was %i bytes' % len(request.args['da_file'][0])
 
root = resource.Resource()
root.putChild("upload", FileUploadService())
factory = server.Site(root)
reactor.listenTCP(8088, factory)
reactor.run()

According to the twisted.web documentation. This method of handling file uploads is not preferred. In fact, the preferred way is to use the more complex twisted.web2 package which supports streaming uploads. However, the twisted devs are working on porting most of twisted.web2’s features to twisted.web in an effort to consolidate the two. In production code, I have chosen to stick with twisted.web since it seems to be the package which will remain the most compatible with future versions of Twisted.

In Half Life 2: Episode 2, the end arena is an amazing experience and something that (though I don’t claim to have a pulse, really, on the gaming industry) should stand out among all other games. SPOILER [: You are placed in a massive, valley-wide arena to fight a ridiculous onslaught of striders and hunters. Enemies that are used as bosses throughout the series are now trying to destroy the only chance your kind has for survival: and they're pummeling you with many times as many as you'd be expected to handle in the past. :]

The environment is a beautiful wooded mountain valley. There are many completely-destroyable buildings throughout the valley: and they destroy beautifully on a massive scale. A building destroyed by a strider completely breaks down into the many components of buildings-2 x 4s, and other building products that all crash and explode, each piece falling randomly-the animations are not scripted.

After your trouble with the strider/hunter battle, the cinematic experience really takes off. Even though it’s the end of the game, movie mode is put on overdrive. The characters all seem to pour into their performances. You’re knocked to the ground and your camera movement is restricted. Shit goes on around you that you can’t help. Some people may be annoyed by this-but it’s vital. The restricted camera movement, the low angle, you, Gordan Freeman, pinned to the ground and helpless, like a human on the very same level as the rest cast-is all really engaging.

Imagine a movie director: Terry Gilliam, and a movie: 12 Monkeys. By the time of 12 Monkeys, a re-make of an older French experimental film, La Jete, Terry Gilliam is already a brilliant and accomplished director. It is classic of his style to use camera angles to convey a feeling, perhaps that of the protagonist or subject of the scene. Though it’s just not camera angles that contribute to the feel of the Gilliam film-it’s the entire experience. Terry Gilliam creates cinematic experiences.

The game industry has always been one that emphasizes experiences. As games push the boundaries of realism, they will naturally draw upon their past experience as super-naturalistic experiences to become hyper-real: as in movies, they will exaggerate real life in real life terms in an effort to be more engaging and more realistic. Game companies know this, look at The Sims 2. Though they had the capability to create a life simulator, it’s much more fun in the cartoony manor they executed it. The Sims, however, can be analogous to a Judd Apatow (Superbad, 40 Yr Old Virgin) movie. They have some real elements but are largely a projection of real life, an obvious fabrication.

A movie like 12 Monkeys, however, is much more fitting as an analogy for the Half Life series. Sims, as with 40 Yr Old Virgin embodies what our psyche wishes it could be: fun, without care, essentially, as infants. Half Life 2 follows a much more serious thread: world destruction and the destruction of our race. It’s cinematic story line includes charismatic, likable characters, one of whom you get to be. The story line is as thick as it’s directing: you are but a mere human, your entire race is being used as slaves and thinned out, restricted from reproducing – your race backlashes and you can now reproduce, and who is this Alyx character, among the other Gordon admirers, her family and her familys collegues? It’s all rather impressive for a game. Which will, within a minute’s time take you frome near death to flirtatious conversation. Very compelling.

Where am I going with this? No where. I’ve come back to this post at least 10 times, wanted to post it and then haven’t because I have no ending.I’m just hoping games get more and more cinematic and compelling! I’m tired of Doom 3! Fuck Doom 3! What did Doom 3 innovate (or Quake 4 or COD 2,3,4 or Crysis or …)?

You may have noticed I’ve been fleshing out this site recently with some of my works. When I got started doing this I thought I would scout out the best video sharing site for professionals or just people who don’t want their videos to look like shit.  So far I’ve tried YouTube, Revver, Veoh, Brightcove and Blip.tv (and that’s where I stopped).

You see, it seems video sharing websites are all out to get ‘cha. They get bumpers at the end of your video, overlays, special buttons and what seems to be the whole 9 yards when it comes to usage rights. The video quality is bad. Your uploads can not exceed 100 MB. They’re allowed to play your video in desecrate desktop applications and server advertising along-side.Whoa! No thanks.

What I’m going to do is just run down the list. Each site has its drawbacks but ultimately sucked, save one.

YouTube

YouTube is the traffic king. Yes, we know that and every other comparison out there puts them at an advantage for that. Well, fuck YouTube. The picture quality of the youtube flash transcodes are absolutely horrible because they use a very low bit rate for the re-encodes. They do this, it seems, so they do not have to drop frames as all other video sites I’ve tried seem to drop frames from the video and create a jerky feeling to smooth animations and forget about pans. YouTube videos also have relatively decent (compared to all but Blip.tv) audio encodes, but they still sound like no more than 64kbps, perhaps less, and still on a mono track.

The video I uploaded to YouTube turned out looking pretty shitty. The words in the titles blend together due to artifacts. The motion is nice as it seems full frame rate was preserved and audio was almost acceptable.

Veoh

Veoh – I don’t have much to say about Veoh. Their video upload process is not a nice single page interface like most other sites, in fact taking 3 pages. Like Revver, they tell you your video is ready to watch somewhat quickly but it usually still takes another 5-10 minutes. The picture quality is decent, and without scaling the videos look pretty clear. Even so, the frame rate is drastically lower than the source video, making the animations jerky. The audio quality was horrible, ditching 2 channels for 1 and using what sounded like like less than 36kbps.

The video I uploaded to Veoh looks decent. It’s way more jerky than the source footage. The titles and their edges are clear. If I had to choose between YouTube and Veoh I’d choose YouTube. Using Veoh would make me feel like a whore. They pretty much get to do whatever they want with your video, including whore it out to other video sites, like MySpace, YouTube and Google Video (primarily).

Brightcove.tv

Originally I picked up Brightcove because I perceived it as having the best video quality. I saw Garry Newman over at garry.tv go through this same process a few months ago. He eventually settled on Brightcove so I thought I’d give it a try. Their video player is really nice. The interface is clean and subtle. Not surprisingly, the video quality is half-way decent, relative to other flash video sites. I would put its video quality on the Veoh side of a comparison between Veoh and YouTube. Despite that-they drop a high number of frames to maintain that quality and the audio is terrible. The interface to their website also kind of sucks after you upload the video. The uploading process is nice, but the so-called “dashboard” is not.

The video I uploaded to Brightcove must be in some sort of limbo right now, possibly being reviewed for copyright infringement or something like Revver does.  Regardless, it turned out decent but with poor audio and low frame rate.

Blip.tv – the winner, surprisingly

After all this uploading and waiting, I finally choose Blip.tv. I did not expect to choose Blip when I started on this journey, well, because of it’s name. It sounds like another bullshit rip off video site (like Veoh, Revver or Metacafe). That said, they have the best interface, best upload process, quick processing, no ads, allow direct downloading, will transcode your video into multiple formats, and as far as I can tell there is no upload limit. Their flash video player is really clean and lightweight, and blends in well in white-background sites AND supports 16×9 videos.

Despite dropping frames, the videos look really good. The transcodes seem miles ahead of all the other video sites. The audio is damn near true to the source footage, with, get this: TWO channels, encoded at what sounds like 96kbps or maybe a little more.

The video I uploaded to Blip.tv seems a little washed out (more so than the source) and is a bit jerky with good audio. You already know the story, I won’t patronize you with the details. Instead I want to show off this feature which I think all the other sites try to do with “channels” and such-but executed horribly. Blip.tv calls them “shows” but none-the-less, its just a group of your videos-executed the right way. Check it out, my “show” on Blip.tv. It gives you a large video area with a small browser and information area-exactly how it should be. There seems to be built in support for serial episodes of whatever you like, and it automatically creates video a video podcast for you. Fucking sweet!