I gotta have my orange juice.

Jesu, Juva

Archive for the ‘Python’ Category

My experience with Django and Rails

with 9 comments

I’ve had the opportunity to work on both Django and Rails frameworks recently as part of one project.  The core application for the church administrative tools that I am working on is written in Rails, while the church guest follow-up application that I am responsible for is written in Django.  Why two separate stacks?  GuestView began its life independently from Gospel Software, and I chose Django there because of my familiarity with Python.  Three developers are sharing responsibility for the core of Gospel Software, however, and we chose Rails as the most reasonable lingua franca.

What follows are my personal opinions and observations.  These are mostly aesthetic or other value judgments, and I offer them simply for your consideration.

Language

I’ve used the Python programming language for a number of years and like it a lot.  I particularly enjoy its functional aspects, although lately I’ve become more of a fan of using list comprehensions and generator expressions wherever possible compared to map() and filter() with lambdas.  Compared to Python, Ruby has much more powerful functional capabilities, although some things don’t feel natural to me (Ruby’s design choice to not require parentheses to denote function invocation means that you must use .call to call a lambda, which feels clunky).  There are also some cases in Ruby where choosing one of several alternative forms of an expression can have a significant impact on your performance.  Lambdas seem particularly costly in Ruby as of version 1.8.

Overall I think the languages are fairly on par.  Right now I prefer Python for aesthetic rather than technical reasons.  As I grow in familiarity with Ruby, and as it matures and its performance improves I think I may eventually grow to prefer it.

Object-Relational Mapping (ORM)

The Django ORM is very powerful and you can express complicated queries very efficiently using it.  Django queries are not executed until they are actually used, so you can construct your queries piecemeal, which helps in writing readable code.  Django also allows you some flexibility with adding custom SQL to your queries, but for anything complicated I’ve found that I have to break down and write my own SQL.

Rails 2.1 introduced the ActiveRecord named_scope functionality.  Prior to this Rails was significantly lacking compared to Django’s expressive power for query construction, but named_scope pretty much evens the playing field.  And for complicated queries, which you will surely face in any real-world project as you seek to tweak performance, ActiveRecord gives you a degree of control over your SQL that really puts Django to shame.

Both Django and Rails seem to have adequate support for PostgreSQL, my database of choice.

URLs

Django lets you express your URLs using regular expressions; Rails accomplishes this using routes.   I personally prefer Django’s method, but both work well enough.

Templates

While Rails’ Embedded Ruby allows you to include arbitrary code in your templates, Django’s template engine is much more spartan.  It provides ways of getting at variables passed to the template, including objects, dictionaries, lists, and even methods.  And it has some simle control structures, but not covering the full expressive power of Python.  I yearned for a more powerful template language in Django at first.  But I found over time that the discipline of  a simple template language was helpful to me, forcing me to move any complicated behaviors to the controller (or “view” as Django calls it) which was in most cases the right thing to do anyway.

There are still some areas where I think the Django template language is lacking.  However, there is an open-source alternative to the Django template engine that is similar but sufficiently more powerful to meet my needs: Jinja2.

For me it is a toss-up between Embedded Ruby and Jinja2.

Performance

I suspect it’s common knowledge that Rails has a little ways to go in performance.  For our own purposes, I didn’t find too much difference in time measurements between Django and Rails.  However, Rails clearly has a much larger memory footprint than Django.

I was surprised to learn that even with a FastCGI or WSGI model, Django still opens and closes a database connection for each request.  While there may be technical reasons that the Django architecture requires this, it was still a surprise to me.  Django performance still seems on par with Rails in spite of this.  Interestingly, having Django use pgpool to connect to PostgreSQL didn’t improve my performance at all, perhaps because my application and database are currently located on the same host.

Console

Both Django and Rails allow you to run a REPL session for your application.  The Rails script/console command beats out Django hands-down, because Rails’ internal magic automatically imports pretty much everything you need.  In Django you still need to import any models or framework modules before you can use them.

Debugging

The Rails built-in log is enabled out of the box and is very handy.  Django provides logging functionality but you have to do a little extra work to enable it.  Rails wins out on logging.  Django is better at in-browser rendering of exception tracebacks.  Overall the handiness its logging means a slight win for Rails here for me.

Admin Application

Django’s admin application is truly its crown jewel.  If you need a private admin interface to your web application, Django will give you a very attractive and powerful interface almost entirely for free.  I’m not aware of any equivalent for Rails that even comes close to this.

Deployment

I’ve deployed Django using FastCGI and Rails using Mongrel.  Right now I am using Nginx to proxy to Mongrel, and to connect directly to the Django FastCGI instance.  Neither Django nor Rails seems to have a unique advantage or disadvantage in deployment.

Summary

I’ve spent more cumulative time with Django than with Rails, so I feel subjectively more at home with Django.  If I were going to write a small toy project, I’d choose Django mainly for ease and efficiency.  In fact, I took this route for the meal and potluck scheduler application that I recently wrote for Google App Engine.  GAE has many similarities with Django, and even allows you to run much of the Django stack on it.

However, for larger projects my current framework of choice is Rails.  With the named_scope functionality in Rails 2.1, ActiveRecord is finally on par with Django’s ORM.   And for any complicated queries ActiveRecord is superior to Django’s ORM.  While Django’s admin application is handy, I don’t make much use of it.  And while Rails falls slightly behind in performance and storage characteristics, I believe that Ruby and Rails will both continue to improve in this regard.

Written by Scott Moonen

January 9, 2009 at 12:59 am

Posted in Python, Rails

Tagged with , , , , , , ,

SmugMug uploader

with 9 comments

[SmugMug]I’ve written a small Python script to upload pictures to a SmugMug gallery. I love SmugMug and use it extensively for family photos. I’m using this script for my personal use because it’s much simpler and much less of a resource hog than a browser-based uploader, and also because it was a fun exercise to try out the SmugMug API. You can run this script as follows to upload one or more files:

python upload.py gallery-name picture-file-name . . .

On Windows I’ve set up a desktop shortcut pointing to the script, and I can drag and drop a pile of picture files onto the icon and it will upload away. I’ve tested it using both Python 2.5 using simplejson, and also using Python 2.6 which has simplejson built in. Earlier versions of Python may require you to change the import of hashlib to md5, and change the hashlib.md5() invocation to a md5.new() invocation. You’ll also need to modify the script to contain your email address and SmugMug password, and obtain a SmugMug API key for your own development use, but this is a very painless process. Here is the script:

#!/usr/bin/python

##########
# Requirements: Python 2.6 or
#               simplejson from http://pypi.python.org/pypi/simplejson
##########

EMAIL='...'
PASSWORD='...'

##########
APIKEY='...'
API_VERSION='1.2.2'
API_URL='https://api.smugmug.com/services/api/json/1.2.2/'
UPLOAD_URL='http://upload.smugmug.com/photos/xmlrawadd.mg'

import sys, re, urllib, urllib2, urlparse, hashlib, traceback, os.path
try    : import json
except : import simplejson as json

if len(sys.argv) < 3 :
  print 'Usage:'
  print '  upload.py  album  picture1  [picture2  [...]]'
  print
  sys.exit(0)

album_name = sys.argv[1]
su_cookie  = None

def safe_geturl(request) :
  global su_cookie

  # Try up to three times
  for x in range(5) :
    try :
      response_obj = urllib2.urlopen(request)
      response = response_obj.read()
      result = json.loads(response)

      # Test for presence of _su cookie and consume it
      meta_info = response_obj.info()
      if meta_info.has_key('set-cookie') :
        match = re.search('(_su=\S+);', meta_info['set-cookie'])
        if match and match.group(1) != "_su=deleted" :
          su_cookie = match.group(1)
      if result['stat'] != 'ok' : raise Exception('Bad result code')
      return result
    except :
      if x < 4 :
        print "  ... failed, retrying"
      else :
        print "  ... failed, giving up"
        print "  Request was:"
        print "  " + request.get_full_url()
        try :
          print "  Response was:"
          print response
        except :
          pass
        traceback.print_exc()
        #sys.stdin.readline()
        #sys.exit(1)
        return result

def smugmug_request(method, params) :
  global su_cookie

  paramstrings = [urllib.quote(key)+'='+urllib.quote(params[key]) for key in params]
  paramstrings += ['method=' + method]
  url = urlparse.urljoin(API_URL, '?' + '&'.join(paramstrings))
  request = urllib2.Request(url)
  if su_cookie :
    request.add_header('Cookie', su_cookie)
  return safe_geturl(request)

result = smugmug_request('smugmug.login.withPassword',
                         {'APIKey'       : APIKEY,
                          'EmailAddress' : EMAIL,
                          'Password'     : PASSWORD})
session = result['Login']['Session']['id']

result = smugmug_request('smugmug.albums.get', {'SessionID' : session})
album_id = None
for album in result['Albums'] :
  if album['Title'] == album_name :
    album_id = album['id']
    break
if album_id is None :
  print 'That album does not exist'
  sys.exit(1)

for filename in sys.argv[2:] :
  data = open(filename, 'rb').read()
  print 'Uploading ' + filename
  upload_request = urllib2.Request(UPLOAD_URL,
                                   data,
                                   {'Content-Length'  : len(data),
                                    'Content-MD5'     : hashlib.md5(data).hexdigest(),
                                    'Content-Type'    : 'none',
                                    'X-Smug-SessionID': session,
                                    'X-Smug-Version'  : API_VERSION,
                                    'X-Smug-ResponseType' : 'JSON',
                                    'X-Smug-AlbumID'  : album_id,
                                    'X-Smug-FileName' : os.path.basename(filename) })
  result = safe_geturl(upload_request)
  if result['stat'] == 'ok' :
    print "  ... successful"

print 'Done'
# sys.stdin.readline()

I am donating this script to the public domain. You are welcome to use and modify it as you please without conditions. I’d appreciate hearing about your experience with this script or any changes and improvements you’ve made; please leave a comment. Thanks!

Update 2010-07-20

Since I first posted this, I’ve updated it as follows:

  1. Add a Content-Type header of ‘none’. This is to workaround a bug in the SmugMug API.
  2. Use basename() to send only the file’s basename for X-Smug-FileName.
  3. Rewrite safe_geturl() to loop up to five times if the upload attempt fails. I’ve found that uploading is surprisingly unreliable, and re-attempting the upload generally works fine.
  4. Add a commented call to readline() at the end of the script. In my case, I run my script by dragging files onto an icon on my Windows desktop, which causes it to run in a DOS window and vanish when done. If you uncomment this line, it will wait for you to press Enter when it is done uploading. You’ll be able to see any files that weren’t uploaded successfully.

Update 2010-11-28

SmugMug made a recent change to their API’s login behavior which broke this script. While the new login behavior is not documented in the API docs, the fix is apparently to use a session cookie along with the session ID. While it’s a bit of a kludge, I’ve updated the script above to save this cookie in a global variable and submit it on subsequent requests.

Update 2011-06-24

I’ve fixed a bug in the script causing it to wrongly report a failure for certain requests that don’t send back the session cookie. The fix involves testing whether a set-cookie header was returned before accessing the header.

Update 2013-10-01

Version 1.2.0 of the SmugMug API has stopped working, so I have updated the script to use version 1.2.2 of the API.

Written by Scott Moonen

December 1, 2008 at 3:33 pm

Posted in Programming, Python, Tools

Tagged with , , , , , , ,

Python generators – saving time and memory

leave a comment »

Python version 2.2 introduced support for generator functions. A generator is a function that returns something kind of like a list. You can think of a generator as a function that gets to return more than once; each time it returns, it produces a new item in the list. Here is a simple example of a generator:

def mygen() :
  print "calling generator"
  for x in [1, 2, 3, 4] :
    print "yielding next value"
    yield x

for item in mygen() :
  print item

This example prints the following:

calling generator
yielding next value
1
yielding next value
2
yielding next value
3
yielding next value
4

Instead of return, we used the yield statement inside our generator. yield returns an item from the generator, but marks that point in the code so that we continue processing when we go back to the generator to fetch the next item in the pseudo-list. Notice how the generator is only called once, but the yield points are interleaved with the print statements in the calling code. Each time the generator needs to produce a new value, it picks up from the previous yield point. When the generator reaches the end of the function, no more values are produced. You cannot use return within a generator.

Let’s look at some code where the use of generators might help us. The following code instantiates a list of objects and then creates HTML to display them. We’ll assume the existence of a database API to execute queries and retrieve results:

def get_objects() :
  result = []
  query = db_execute("...")
  row = query.fetchrow()
  while not row is None :
    result.append(MyObject(row))  # Build object from DB, append to result
    row = query.fetchrow()
  return result
. . .
for object in get_objects() :
  print object.getHTML()

This code creates the entire list of objects before it can print any of them. There are two problems with this — first, there is a huge delay to create all of the objects before any progress is made in printing; this means that the user’s browser has no partial results to display. Second, all of the objects must be held in memory at the same time; if there are many objects, this can cause significant overhead.

Generators allow us to attack both of these problems. Since a generator produces items one at a time, on demand, it avoids both of these problems. We don’t have to wait to construct all of the objects in the list before we use the first one. And once we are done using an object, the Python garbage collector is now free to immediately clean it up. Here’s how we might rewrite the above code to use generators:

def get_objects() :
  query = db_execute("...")
  row = query.fetchrow()
  while not row is None :
    yield MyObject(row)    # Build object from DB, yield to caller
    row = query.fetchrow()
. . .
for object in get_objects() :
  print object.getHTML()

With a small change we have now significantly improved our code’s memory footprint — all of the objects do not need to be held in memory at the same time. And we are now producing the output for each object as we create it, without needing to wait for all the objects to be instantiated first. This is a significant improvement!

We can also build a chain of generators. Let’s assume that we need to modify the code above to optionally display only those objects that have an “active” flag set. Ordinarily, if we use Python’s filter function to accomplish this, it needs to create the entire list all at once. But if we use a generator to perform the filtering, then we can still keep our optimizations since we are only ever creating and filtering one object at a time. Here’s an example:

my_objs = get_objects()         # This returns a generator object

if display_active_only :
  def active_filter(objects) :  # A filtering generator
    for object in objects :
      if object.active :
        yield object

  my_objs = active_filter(my_objs)

for object in my_objs :
  print object.getHTML()

A generator function doesn’t produce a real list; instead, it produces a generator object that behaves like something called an iterator. You can’t write either of the following statements for a generator or iterator:

print len(get_objects())
print get_objects()[3]

The for statement is smart enough to traverse a generator, and it will probably be sufficient for your needs. Perhaps you can get the count of objects by other means, such as executing an SQL COUNT request. If you absolutely need to access a generator as a list, you can coerce it to a list as follows:

objlist = list(get_objects())

But be aware that this removes all of the advantages that we’ve discussed here, since this causes all of the objects returned by the generator to be created at once and stored in the list. If you find yourself needing to do this, you should consider rewriting your code so that you don’t need to do so. Or perhaps generators aren’t the right solution for your particular problem.

Written by Scott Moonen

February 1, 2008 at 10:22 am

Posted in Python

Tagged with , , , , , , , , , , , , , , ,

Python combinations

leave a comment »

In September 2007, Brian Adkins posted a simple Logo program to print all possible combinations of lists of items, and asked for alternatives in other languages. He provided a Ruby alternative which would translate fairly easily into Python. For my own Python implementation I decided to try a completely different approach:

prod1   = lambda elem, vec : map(lambda x : elem+' '+x, vec)
xprod   = lambda vec1, vec2 : reduce(list.__add__, map(lambda elem : prod1(elem ,vec2), vec1))
choices = lambda x : '\n'.join(reduce(xprod, x))+'\n'

q = [['small', 'medium', 'large'],
     ['vanilla', 'ultra chocolate', 'lychee', 'rum raisin', 'ginger'],
     ['cone', 'cup']]

print choices(q),

Written by Scott Moonen

September 1, 2007 at 7:25 pm

Posted in Python

Tagged with , ,

Functional Python

with 3 comments

A friend of mine is learning Python and was curious about functional programming, so I have written this brief tutorial. Python isn’t a full-fledged functional language, but it supports some very useful functional idioms.

It’s best to approach this tutorial by programming along at the Python interactive prompt. Try typing everything in to see what the results are.

Introduction

Imagine you have a list of numbers and want to filter out all even numbers:

q = [1,2,5,6,8,12,15,17,20,23,24]

def is_even(x) :
  return x % 2 == 0

result = filter(is_even, q)

(Remember that the “%” operator is the modulus or remainder operator. If the remainder when a number is divided by two is zero, then the number must be even.)

If we only use is_even once, it’s kind of annoying that we have to define it. Wouldn’t it be nice if we could just define it inside of the call to filter? We want to do something like “q = filter(x % 2 == 0, q)“, but that won’t quite work, since the “x % 2 == 0” is evaluated before the call to filter. We want to pass a function to filter, not an expression.

We can do this using the lambda operator. lambda allows you to create an unnamed throw-away function. Try this:

q = [1,2,5,6,8,12,15,17,20,23,24]
result = filter(lambda x : x % 2 == 0, q)

lambda tells Python that a function follows. Before the colon (:) you list the parameters of the function (in this case, only one, x). After the colon you list what the function returns. This function doesn’t have a name, and Python disposes of it as soon as filter is done with it.

You can, however, assign the function to a variable to give it a name. The following two statements are identical:

def is_odd(x) :
  return x % 2 == 1
 
is_odd = lambda x : x % 2 == 1

map

Here’s another example. The map function applys a function to every item in a list, and returns the result. This example adds 1 to every element in the list:

result = map(lambda x : x + 1, [1,2,3,4,5])

(At this point, result holds [2,3,4,5,6].)

The map function can also process more than one list at a time, provided they are the same length. This allows you to do things like add all of the elements in a pair of lists. Note that our lambda here has two parameters:

result = map(lambda x,y : x+y, [1,2,3,4,5], [6,7,8,9,10])

(At this point, result holds [7, 9, 11, 13, 15].)

reduce

Python also provides the reduce function. This is a bit more complicated; you provide it a list and a function, and it reduces that list by applying the function to pairs of elements in the list. An example is the best way to understand this. Let’s say we want to find the sum of all the elements in a list:

sum = reduce(lambda x,y : x+y, [1,2,3,4,5])

reduce will first apply the function to 1,2, yielding 3. It will then apply the function to 3,3 (the first 3 is the result of adding 1+2), yielding 6. It will then apply the function to 6,4 (the 6 is the result of adding 3+3), yielding 10. Finally, it will apply the function to 10,5, yielding 15. The result stored in sum is 15.

Similarly, we can find the cumulative product of all items in a list. The following example stores 120 (=1*2*3*4*5) in product:

product = reduce(lambda x,y : x*y, [1,2,3,4,5])

Conditionals

Beginning in Python 2.5 you can even express conditions within your lambdas, using Python’s conditional expressions. So, for example:

reciprocals = map(lambda x : 1.0/x if x !=  0 else None, [0, 1, 2, 3, 4, 5])

At this point, reciprocals holds the list [None, 1.0, 0.5, 0.333..., 0.25, 0.2]. The expression “1/x if x != 0 else None” is the conditional expression, and it is used to prevent division by zero. If x is not 0, then the result is 1/x, but otherwise it is None.

Background

One of the advantages of lambda-functions is that they allow you to write very concise code. A result of this, however, is that the code is dense with meaning, and it can be hard to read if you are not accustomed to functional programming. In our first example, filter(is_even, q) was fairly easy to understand (fortunately, we chose a descriptive function name), while filter(lambda x : x % 2 == 0, q) takes a little longer to comprehend.

If you are a masochistic mathematical geek, you’ll probably enjoy this (I do). If not, you might still prefer to break things into smaller pieces by defining all of your functions first and then using them by name. That’s ok! Functional programming isn’t for everyone.

Another advantage of functional programming is that it corresponds very closely to the mathematical notion of functions. Note that our lambda-functions didn’t store any results into variables, or have any other sort of side effect. Functions without side effects cause fewer bugs, because their behavior is deterministic (this is related to the dictum that you aren’t supposed to use global variables). It is also much easier to mathematically prove that such functions do what you think they are doing. (Note that it’s still possible to write a lambda function that has side effects, if the lambda function calls another function that has side effects; for example lambda x : sys.stdout.write(str(x)) has the side effect of printing its parameter to the screen.)

Written by Scott Moonen

June 9, 2005 at 7:13 am

Posted in Python

Tagged with , , ,

Python odd word problem

leave a comment »

I crafted this compact solution to Dijkstra’s OddWordProblem:

from sys import stdin, stdout
def even(char) : stdout.write(char); return select(even, 0, "");
def odd(char)  : q = select(odd, 0, ""); stdout.write(char); return q
def select(fn, skipspace, prefix) :
  char = stdin.read(1)
  if char.isspace() and skipspace : return select(fn, 1, prefix)
  elif char.isspace()             : return 0
  elif char.isalpha()             : stdout.write(prefix); return fn(char)
  elif char == "."                : return 1
  else                            : raise "Invalid input"

if not select(even, 1, "") :
while not select(odd, 1, " ") and not select(even, 1, " ") : pass
stdout.write(".")

Written by Scott Moonen

November 10, 2004 at 5:36 am

Posted in Python

Tagged with ,

Python list partition

leave a comment »

I came up with these simple solutions to RonJeffries‘ CodingProblem. The first uses Python 2.2’s generators, the second uses the functional idiom, and the third uses list comprehensions. The third is my favorite:

def split1(list, num) :
  for index in range(num) :
    yield list[index * len(list) / num:(index + 1) * len(list) / num]

def split2(list, num) :
  return map(lambda x: list[x * len(list) / num:(x+1) * len(list) / num], range(num))

def split3(list, num) :
  return [list[x*len(list)/num : (x+1)*len(list)/num] for x in range(num)]

Written by Scott Moonen

November 9, 2004 at 2:10 pm

Posted in Python

Tagged with ,

Python Timestamp Increment

leave a comment »

I composed this concise solution to ncm‘s “coding challenge” in November 2004. I’m not entirely satisfied with its elegance, but I’m pleased with its brevity.

def dateinc(s) :
  def carry(list, modulus) :
    if len(list) == 1 : return [list[0] // modulus[0], list[0] % modulus[0]]
    else              : q = carry(list[1:], modulus[1:]); return [(list[0] + q[0]) // modulus[0], (list[0] + q[0]) % modulus[0]] + q[1:]
  s = reduce(lambda x,y:x+y, map(lambda x : (x >= '0' and x <= '9') * x, s))
  result = map(lambda x,y:x+y, map(int,(s[0:4],s[4:6],s[6:8],s[8:10],s[10:12],s[12:14])),(0,-1,-1,0,0,1))
  modulus = [10000,12,[31,28,31,30,31,30,31,31,30,31,30,31][result[1]],24,60,60]
  if result[0] % 4 == 0 and (result[0] % 100 != 0 or result[0] % 400 == 0) :
    modulus[2] += 1                               # Account for leap day.
  result = carry(result, modulus)[1:]             # Carry the extra second.
  result[1:3] = map(lambda x:x+1, result[1:3])    # Readjust to 1-base.
  return reduce(lambda x,y:x+y, map(lambda x : (x < 10) * '0' + str(x), result))

I further reduced this as follows, turning the carry function into a one-liner that computes the carry in-place (and also using builtins wherever possible). I’m much more satisfied with this solution.

def dateinc(s) :
  s = reduce(lambda x,y:x.isdigit()*x+y.isdigit()*y, s)
  ans = map(int.__add__, map(int,(s[0:4],s[4:6],s[6:8],s[8:10],s[10:12],s[12:14])),(0,-1,-1,0,0,1))
  mod = [10000,12,[31,28,31,30,31,30,31,31,30,31,30,31][ans[1]],24,60,60]
  if ans[0] % 4 == 0 and (ans[0] % 100 != 0 or ans[0] % 400 == 0) : mod[2] += 1
  ans = map(lambda x:(ans[x] + reduce(int.__mul__, map(lambda y:ans[y]>=mod[y]-1,range(x+1,len(ans))),x!=len(ans)-1)) % mod[x], range(len(ans)))
  ans[1:3] = map(int.__add__,ans[1:3],[1,1])      # Readjust to 1-base.
  return reduce(str.__add__, map(lambda x : (x < 10) * '0' + str(x), ans))

Written by Scott Moonen

November 9, 2004 at 12:52 pm

Posted in Python

Tagged with

Python persistence

leave a comment »

I’ve implemented a rudimentary persistent object store in Python. It is implemented as a Python extension module that implements a set of persistent types: strings, integers, floats, lists, and dictionaries. Each of these are backed by the persistent object store, which is implemented using a memory-mapped file.

In addition, using a specially crafted Python base class for Python objects, Python objects may be stored in the object store (as dictionaries) and instantiated out of the object store.

The result is an persistent object graph (the root of which is a persistent dictionary) whose objects and attributes may be manipulated in-place using native Python syntax. Rudimentary locking is provided so that multiple Python threads / processes may concurrently manipulate the object store.

Details

Some aspects of this system are:

  • It is a Python extension module written in C and C++.
  • It is tested on Linux. It will likely work on *BSD systems, though it is possible that the location of the mapped storage may need to be moved.
  • It is implemented in a hierarchical manner:
    • A page manager handles the allocation of 4kB pages within a memory-mapped file. It is multi-process safe. It is, in a sense, a glorified sbrk() for a memory-mapped file.
    • A heap manager abstracts the page manager’s services to manage the allocation and deallocation of arbitrary-sized storage segments within the memory-mapped file. It is essentially a malloc() and free() for a memory-mapped file. This is also multi-process safe.
    • An object manager manages five new base types (persistent int, float, string, list, and dictionary) backed by persistent storage, using the heap manager’s services. It also provides rudimentary locking facilities for concurrency-safeness.
    • The persist Python extension uses the object manager’s services to implement persistent types that mimic the equivalent Python types. Additionally, it has the ability to reinstantiate a Python object that was stored as a dictionary (using the appropriate Python base class). The object manager’s locking facilities are made available for application usage.
  • Only one file may be mapped at a time (because it is mapped to a fixed logical address).
  • It is available for use under the MIT license. Contact me if you are interested in using it.

Examples

Some examples of its use are a simple counter:

import persist

root = persist.root()
root.lockExcl()
try :    root['x'] += 1                # Increment counter
except : root['x'] = 1                 # First pass; initialize
print "Content-type: text/html\n"
print "<p>You are visitor " + str(root['x']) + " to visit this site!</p>"
root.unlock()

and rudimentary objects:

import persist
from pbase import pbase

class person (pbase) :
  def __init__(self, name = "", age = 0) :
    pbase.__init__(self)
    self.name = name
    self.age = age

  def printAge(self) :
    print "<p>" + self.name + " is " + str(self.age) + " years old</p>"

root = persist.root()
root.lockExcl()

if not root.has_key('Joe') :            # First time through
  root['Joe'] = person('Joe', 27)
if not root.has_key('John') :           # First time through
  root['John'] = person("John", 29)

# On subsequent passes we will retrieve the objects stored on the first pass.

print "Content-type: text/html\n"
root['Joe'].printAge()
root['John'].printAge()

root.unlock()

Written by Scott Moonen

October 1, 2004 at 11:07 am

Posted in Programming, Python

Tagged with , ,