I gotta have my orange juice.

Jesu, Juva

Posts Tagged ‘Python’

My experience with Django and Rails

with 9 comments

I’ve had the opportunity to work on both Django and Rails frameworks recently as part of one project.  The core application for the church administrative tools that I am working on is written in Rails, while the church guest follow-up application that I am responsible for is written in Django.  Why two separate stacks?  GuestView began its life independently from Gospel Software, and I chose Django there because of my familiarity with Python.  Three developers are sharing responsibility for the core of Gospel Software, however, and we chose Rails as the most reasonable lingua franca.

What follows are my personal opinions and observations.  These are mostly aesthetic or other value judgments, and I offer them simply for your consideration.

Language

I’ve used the Python programming language for a number of years and like it a lot.  I particularly enjoy its functional aspects, although lately I’ve become more of a fan of using list comprehensions and generator expressions wherever possible compared to map() and filter() with lambdas.  Compared to Python, Ruby has much more powerful functional capabilities, although some things don’t feel natural to me (Ruby’s design choice to not require parentheses to denote function invocation means that you must use .call to call a lambda, which feels clunky).  There are also some cases in Ruby where choosing one of several alternative forms of an expression can have a significant impact on your performance.  Lambdas seem particularly costly in Ruby as of version 1.8.

Overall I think the languages are fairly on par.  Right now I prefer Python for aesthetic rather than technical reasons.  As I grow in familiarity with Ruby, and as it matures and its performance improves I think I may eventually grow to prefer it.

Object-Relational Mapping (ORM)

The Django ORM is very powerful and you can express complicated queries very efficiently using it.  Django queries are not executed until they are actually used, so you can construct your queries piecemeal, which helps in writing readable code.  Django also allows you some flexibility with adding custom SQL to your queries, but for anything complicated I’ve found that I have to break down and write my own SQL.

Rails 2.1 introduced the ActiveRecord named_scope functionality.  Prior to this Rails was significantly lacking compared to Django’s expressive power for query construction, but named_scope pretty much evens the playing field.  And for complicated queries, which you will surely face in any real-world project as you seek to tweak performance, ActiveRecord gives you a degree of control over your SQL that really puts Django to shame.

Both Django and Rails seem to have adequate support for PostgreSQL, my database of choice.

URLs

Django lets you express your URLs using regular expressions; Rails accomplishes this using routes.   I personally prefer Django’s method, but both work well enough.

Templates

While Rails’ Embedded Ruby allows you to include arbitrary code in your templates, Django’s template engine is much more spartan.  It provides ways of getting at variables passed to the template, including objects, dictionaries, lists, and even methods.  And it has some simle control structures, but not covering the full expressive power of Python.  I yearned for a more powerful template language in Django at first.  But I found over time that the discipline of  a simple template language was helpful to me, forcing me to move any complicated behaviors to the controller (or “view” as Django calls it) which was in most cases the right thing to do anyway.

There are still some areas where I think the Django template language is lacking.  However, there is an open-source alternative to the Django template engine that is similar but sufficiently more powerful to meet my needs: Jinja2.

For me it is a toss-up between Embedded Ruby and Jinja2.

Performance

I suspect it’s common knowledge that Rails has a little ways to go in performance.  For our own purposes, I didn’t find too much difference in time measurements between Django and Rails.  However, Rails clearly has a much larger memory footprint than Django.

I was surprised to learn that even with a FastCGI or WSGI model, Django still opens and closes a database connection for each request.  While there may be technical reasons that the Django architecture requires this, it was still a surprise to me.  Django performance still seems on par with Rails in spite of this.  Interestingly, having Django use pgpool to connect to PostgreSQL didn’t improve my performance at all, perhaps because my application and database are currently located on the same host.

Console

Both Django and Rails allow you to run a REPL session for your application.  The Rails script/console command beats out Django hands-down, because Rails’ internal magic automatically imports pretty much everything you need.  In Django you still need to import any models or framework modules before you can use them.

Debugging

The Rails built-in log is enabled out of the box and is very handy.  Django provides logging functionality but you have to do a little extra work to enable it.  Rails wins out on logging.  Django is better at in-browser rendering of exception tracebacks.  Overall the handiness its logging means a slight win for Rails here for me.

Admin Application

Django’s admin application is truly its crown jewel.  If you need a private admin interface to your web application, Django will give you a very attractive and powerful interface almost entirely for free.  I’m not aware of any equivalent for Rails that even comes close to this.

Deployment

I’ve deployed Django using FastCGI and Rails using Mongrel.  Right now I am using Nginx to proxy to Mongrel, and to connect directly to the Django FastCGI instance.  Neither Django nor Rails seems to have a unique advantage or disadvantage in deployment.

Summary

I’ve spent more cumulative time with Django than with Rails, so I feel subjectively more at home with Django.  If I were going to write a small toy project, I’d choose Django mainly for ease and efficiency.  In fact, I took this route for the meal and potluck scheduler application that I recently wrote for Google App Engine.  GAE has many similarities with Django, and even allows you to run much of the Django stack on it.

However, for larger projects my current framework of choice is Rails.  With the named_scope functionality in Rails 2.1, ActiveRecord is finally on par with Django’s ORM.   And for any complicated queries ActiveRecord is superior to Django’s ORM.  While Django’s admin application is handy, I don’t make much use of it.  And while Rails falls slightly behind in performance and storage characteristics, I believe that Ruby and Rails will both continue to improve in this regard.

Written by Scott Moonen

January 9, 2009 at 12:59 am

Posted in Python, Rails

Tagged with , , , , , , ,

SmugMug uploader

with 9 comments

[SmugMug]I’ve written a small Python script to upload pictures to a SmugMug gallery. I love SmugMug and use it extensively for family photos. I’m using this script for my personal use because it’s much simpler and much less of a resource hog than a browser-based uploader, and also because it was a fun exercise to try out the SmugMug API. You can run this script as follows to upload one or more files:

python upload.py gallery-name picture-file-name . . .

On Windows I’ve set up a desktop shortcut pointing to the script, and I can drag and drop a pile of picture files onto the icon and it will upload away. I’ve tested it using both Python 2.5 using simplejson, and also using Python 2.6 which has simplejson built in. Earlier versions of Python may require you to change the import of hashlib to md5, and change the hashlib.md5() invocation to a md5.new() invocation. You’ll also need to modify the script to contain your email address and SmugMug password, and obtain a SmugMug API key for your own development use, but this is a very painless process. Here is the script:

#!/usr/bin/python

##########
# Requirements: Python 2.6 or
#               simplejson from http://pypi.python.org/pypi/simplejson
##########

EMAIL='...'
PASSWORD='...'

##########
APIKEY='...'
API_VERSION='1.2.2'
API_URL='https://api.smugmug.com/services/api/json/1.2.2/'
UPLOAD_URL='http://upload.smugmug.com/photos/xmlrawadd.mg'

import sys, re, urllib, urllib2, urlparse, hashlib, traceback, os.path
try    : import json
except : import simplejson as json

if len(sys.argv) < 3 :
  print 'Usage:'
  print '  upload.py  album  picture1  [picture2  [...]]'
  print
  sys.exit(0)

album_name = sys.argv[1]
su_cookie  = None

def safe_geturl(request) :
  global su_cookie

  # Try up to three times
  for x in range(5) :
    try :
      response_obj = urllib2.urlopen(request)
      response = response_obj.read()
      result = json.loads(response)

      # Test for presence of _su cookie and consume it
      meta_info = response_obj.info()
      if meta_info.has_key('set-cookie') :
        match = re.search('(_su=\S+);', meta_info['set-cookie'])
        if match and match.group(1) != "_su=deleted" :
          su_cookie = match.group(1)
      if result['stat'] != 'ok' : raise Exception('Bad result code')
      return result
    except :
      if x < 4 :
        print "  ... failed, retrying"
      else :
        print "  ... failed, giving up"
        print "  Request was:"
        print "  " + request.get_full_url()
        try :
          print "  Response was:"
          print response
        except :
          pass
        traceback.print_exc()
        #sys.stdin.readline()
        #sys.exit(1)
        return result

def smugmug_request(method, params) :
  global su_cookie

  paramstrings = [urllib.quote(key)+'='+urllib.quote(params[key]) for key in params]
  paramstrings += ['method=' + method]
  url = urlparse.urljoin(API_URL, '?' + '&'.join(paramstrings))
  request = urllib2.Request(url)
  if su_cookie :
    request.add_header('Cookie', su_cookie)
  return safe_geturl(request)

result = smugmug_request('smugmug.login.withPassword',
                         {'APIKey'       : APIKEY,
                          'EmailAddress' : EMAIL,
                          'Password'     : PASSWORD})
session = result['Login']['Session']['id']

result = smugmug_request('smugmug.albums.get', {'SessionID' : session})
album_id = None
for album in result['Albums'] :
  if album['Title'] == album_name :
    album_id = album['id']
    break
if album_id is None :
  print 'That album does not exist'
  sys.exit(1)

for filename in sys.argv[2:] :
  data = open(filename, 'rb').read()
  print 'Uploading ' + filename
  upload_request = urllib2.Request(UPLOAD_URL,
                                   data,
                                   {'Content-Length'  : len(data),
                                    'Content-MD5'     : hashlib.md5(data).hexdigest(),
                                    'Content-Type'    : 'none',
                                    'X-Smug-SessionID': session,
                                    'X-Smug-Version'  : API_VERSION,
                                    'X-Smug-ResponseType' : 'JSON',
                                    'X-Smug-AlbumID'  : album_id,
                                    'X-Smug-FileName' : os.path.basename(filename) })
  result = safe_geturl(upload_request)
  if result['stat'] == 'ok' :
    print "  ... successful"

print 'Done'
# sys.stdin.readline()

I am donating this script to the public domain. You are welcome to use and modify it as you please without conditions. I’d appreciate hearing about your experience with this script or any changes and improvements you’ve made; please leave a comment. Thanks!

Update 2010-07-20

Since I first posted this, I’ve updated it as follows:

  1. Add a Content-Type header of ‘none’. This is to workaround a bug in the SmugMug API.
  2. Use basename() to send only the file’s basename for X-Smug-FileName.
  3. Rewrite safe_geturl() to loop up to five times if the upload attempt fails. I’ve found that uploading is surprisingly unreliable, and re-attempting the upload generally works fine.
  4. Add a commented call to readline() at the end of the script. In my case, I run my script by dragging files onto an icon on my Windows desktop, which causes it to run in a DOS window and vanish when done. If you uncomment this line, it will wait for you to press Enter when it is done uploading. You’ll be able to see any files that weren’t uploaded successfully.

Update 2010-11-28

SmugMug made a recent change to their API’s login behavior which broke this script. While the new login behavior is not documented in the API docs, the fix is apparently to use a session cookie along with the session ID. While it’s a bit of a kludge, I’ve updated the script above to save this cookie in a global variable and submit it on subsequent requests.

Update 2011-06-24

I’ve fixed a bug in the script causing it to wrongly report a failure for certain requests that don’t send back the session cookie. The fix involves testing whether a set-cookie header was returned before accessing the header.

Update 2013-10-01

Version 1.2.0 of the SmugMug API has stopped working, so I have updated the script to use version 1.2.2 of the API.

Written by Scott Moonen

December 1, 2008 at 3:33 pm

Posted in Programming, Python, Tools

Tagged with , , , , , , ,

Python generators – saving time and memory

leave a comment »

Python version 2.2 introduced support for generator functions. A generator is a function that returns something kind of like a list. You can think of a generator as a function that gets to return more than once; each time it returns, it produces a new item in the list. Here is a simple example of a generator:

def mygen() :
  print "calling generator"
  for x in [1, 2, 3, 4] :
    print "yielding next value"
    yield x

for item in mygen() :
  print item

This example prints the following:

calling generator
yielding next value
1
yielding next value
2
yielding next value
3
yielding next value
4

Instead of return, we used the yield statement inside our generator. yield returns an item from the generator, but marks that point in the code so that we continue processing when we go back to the generator to fetch the next item in the pseudo-list. Notice how the generator is only called once, but the yield points are interleaved with the print statements in the calling code. Each time the generator needs to produce a new value, it picks up from the previous yield point. When the generator reaches the end of the function, no more values are produced. You cannot use return within a generator.

Let’s look at some code where the use of generators might help us. The following code instantiates a list of objects and then creates HTML to display them. We’ll assume the existence of a database API to execute queries and retrieve results:

def get_objects() :
  result = []
  query = db_execute("...")
  row = query.fetchrow()
  while not row is None :
    result.append(MyObject(row))  # Build object from DB, append to result
    row = query.fetchrow()
  return result
. . .
for object in get_objects() :
  print object.getHTML()

This code creates the entire list of objects before it can print any of them. There are two problems with this — first, there is a huge delay to create all of the objects before any progress is made in printing; this means that the user’s browser has no partial results to display. Second, all of the objects must be held in memory at the same time; if there are many objects, this can cause significant overhead.

Generators allow us to attack both of these problems. Since a generator produces items one at a time, on demand, it avoids both of these problems. We don’t have to wait to construct all of the objects in the list before we use the first one. And once we are done using an object, the Python garbage collector is now free to immediately clean it up. Here’s how we might rewrite the above code to use generators:

def get_objects() :
  query = db_execute("...")
  row = query.fetchrow()
  while not row is None :
    yield MyObject(row)    # Build object from DB, yield to caller
    row = query.fetchrow()
. . .
for object in get_objects() :
  print object.getHTML()

With a small change we have now significantly improved our code’s memory footprint — all of the objects do not need to be held in memory at the same time. And we are now producing the output for each object as we create it, without needing to wait for all the objects to be instantiated first. This is a significant improvement!

We can also build a chain of generators. Let’s assume that we need to modify the code above to optionally display only those objects that have an “active” flag set. Ordinarily, if we use Python’s filter function to accomplish this, it needs to create the entire list all at once. But if we use a generator to perform the filtering, then we can still keep our optimizations since we are only ever creating and filtering one object at a time. Here’s an example:

my_objs = get_objects()         # This returns a generator object

if display_active_only :
  def active_filter(objects) :  # A filtering generator
    for object in objects :
      if object.active :
        yield object

  my_objs = active_filter(my_objs)

for object in my_objs :
  print object.getHTML()

A generator function doesn’t produce a real list; instead, it produces a generator object that behaves like something called an iterator. You can’t write either of the following statements for a generator or iterator:

print len(get_objects())
print get_objects()[3]

The for statement is smart enough to traverse a generator, and it will probably be sufficient for your needs. Perhaps you can get the count of objects by other means, such as executing an SQL COUNT request. If you absolutely need to access a generator as a list, you can coerce it to a list as follows:

objlist = list(get_objects())

But be aware that this removes all of the advantages that we’ve discussed here, since this causes all of the objects returned by the generator to be created at once and stored in the list. If you find yourself needing to do this, you should consider rewriting your code so that you don’t need to do so. Or perhaps generators aren’t the right solution for your particular problem.

Written by Scott Moonen

February 1, 2008 at 10:22 am

Posted in Python

Tagged with , , , , , , , , , , , , , , ,