I gotta have my orange juice.

Jesu, Juva

Archive for the ‘Programming’ Category

Contentment

with one comment

You should probably be using the tool that you hate the most. You hate it because you know the most about it.

Dan McKinley

Crossposted to full◦valence.

Written by Scott Moonen

July 3, 2019 at 8:10 am

Slow down to speed up

leave a comment »

What does James Michener’s novel Space have to do with software development?

I’ve posted some reflections on technical debt and organizational discipline on my other blog.

Written by Scott Moonen

February 6, 2016 at 9:49 pm

Posted in Programming

SmugMug uploader

with 9 comments

[SmugMug]I’ve written a small Python script to upload pictures to a SmugMug gallery. I love SmugMug and use it extensively for family photos. I’m using this script for my personal use because it’s much simpler and much less of a resource hog than a browser-based uploader, and also because it was a fun exercise to try out the SmugMug API. You can run this script as follows to upload one or more files:

python upload.py gallery-name picture-file-name . . .

On Windows I’ve set up a desktop shortcut pointing to the script, and I can drag and drop a pile of picture files onto the icon and it will upload away. I’ve tested it using both Python 2.5 using simplejson, and also using Python 2.6 which has simplejson built in. Earlier versions of Python may require you to change the import of hashlib to md5, and change the hashlib.md5() invocation to a md5.new() invocation. You’ll also need to modify the script to contain your email address and SmugMug password, and obtain a SmugMug API key for your own development use, but this is a very painless process. Here is the script:

#!/usr/bin/python

##########
# Requirements: Python 2.6 or
#               simplejson from http://pypi.python.org/pypi/simplejson
##########

EMAIL='...'
PASSWORD='...'

##########
APIKEY='...'
API_VERSION='1.2.2'
API_URL='https://api.smugmug.com/services/api/json/1.2.2/'
UPLOAD_URL='http://upload.smugmug.com/photos/xmlrawadd.mg'

import sys, re, urllib, urllib2, urlparse, hashlib, traceback, os.path
try    : import json
except : import simplejson as json

if len(sys.argv) < 3 :
  print 'Usage:'
  print '  upload.py  album  picture1  [picture2  [...]]'
  print
  sys.exit(0)

album_name = sys.argv[1]
su_cookie  = None

def safe_geturl(request) :
  global su_cookie

  # Try up to three times
  for x in range(5) :
    try :
      response_obj = urllib2.urlopen(request)
      response = response_obj.read()
      result = json.loads(response)

      # Test for presence of _su cookie and consume it
      meta_info = response_obj.info()
      if meta_info.has_key('set-cookie') :
        match = re.search('(_su=\S+);', meta_info['set-cookie'])
        if match and match.group(1) != "_su=deleted" :
          su_cookie = match.group(1)
      if result['stat'] != 'ok' : raise Exception('Bad result code')
      return result
    except :
      if x < 4 :
        print "  ... failed, retrying"
      else :
        print "  ... failed, giving up"
        print "  Request was:"
        print "  " + request.get_full_url()
        try :
          print "  Response was:"
          print response
        except :
          pass
        traceback.print_exc()
        #sys.stdin.readline()
        #sys.exit(1)
        return result

def smugmug_request(method, params) :
  global su_cookie

  paramstrings = [urllib.quote(key)+'='+urllib.quote(params[key]) for key in params]
  paramstrings += ['method=' + method]
  url = urlparse.urljoin(API_URL, '?' + '&'.join(paramstrings))
  request = urllib2.Request(url)
  if su_cookie :
    request.add_header('Cookie', su_cookie)
  return safe_geturl(request)

result = smugmug_request('smugmug.login.withPassword',
                         {'APIKey'       : APIKEY,
                          'EmailAddress' : EMAIL,
                          'Password'     : PASSWORD})
session = result['Login']['Session']['id']

result = smugmug_request('smugmug.albums.get', {'SessionID' : session})
album_id = None
for album in result['Albums'] :
  if album['Title'] == album_name :
    album_id = album['id']
    break
if album_id is None :
  print 'That album does not exist'
  sys.exit(1)

for filename in sys.argv[2:] :
  data = open(filename, 'rb').read()
  print 'Uploading ' + filename
  upload_request = urllib2.Request(UPLOAD_URL,
                                   data,
                                   {'Content-Length'  : len(data),
                                    'Content-MD5'     : hashlib.md5(data).hexdigest(),
                                    'Content-Type'    : 'none',
                                    'X-Smug-SessionID': session,
                                    'X-Smug-Version'  : API_VERSION,
                                    'X-Smug-ResponseType' : 'JSON',
                                    'X-Smug-AlbumID'  : album_id,
                                    'X-Smug-FileName' : os.path.basename(filename) })
  result = safe_geturl(upload_request)
  if result['stat'] == 'ok' :
    print "  ... successful"

print 'Done'
# sys.stdin.readline()

I am donating this script to the public domain. You are welcome to use and modify it as you please without conditions. I’d appreciate hearing about your experience with this script or any changes and improvements you’ve made; please leave a comment. Thanks!

Update 2010-07-20

Since I first posted this, I’ve updated it as follows:

  1. Add a Content-Type header of ‘none’. This is to workaround a bug in the SmugMug API.
  2. Use basename() to send only the file’s basename for X-Smug-FileName.
  3. Rewrite safe_geturl() to loop up to five times if the upload attempt fails. I’ve found that uploading is surprisingly unreliable, and re-attempting the upload generally works fine.
  4. Add a commented call to readline() at the end of the script. In my case, I run my script by dragging files onto an icon on my Windows desktop, which causes it to run in a DOS window and vanish when done. If you uncomment this line, it will wait for you to press Enter when it is done uploading. You’ll be able to see any files that weren’t uploaded successfully.

Update 2010-11-28

SmugMug made a recent change to their API’s login behavior which broke this script. While the new login behavior is not documented in the API docs, the fix is apparently to use a session cookie along with the session ID. While it’s a bit of a kludge, I’ve updated the script above to save this cookie in a global variable and submit it on subsequent requests.

Update 2011-06-24

I’ve fixed a bug in the script causing it to wrongly report a failure for certain requests that don’t send back the session cookie. The fix involves testing whether a set-cookie header was returned before accessing the header.

Update 2013-10-01

Version 1.2.0 of the SmugMug API has stopped working, so I have updated the script to use version 1.2.2 of the API.

Written by Scott Moonen

December 1, 2008 at 3:33 pm

Editor Color Scheme

leave a comment »

Recently Slashdot featured a discussion of the best color scheme for programming. From that discussion I’ve discovered the zenburn color scheme, and have switched to it. I like the fact that the background is not stark black; the reduced contrast feels easier on my eyes.

Here are some resources I found from that discussion for color schemes:

What color scheme do you prefer for programming? There are several others at the vim color scheme test, above, that I’m also interested in trying out.

See also: Programming Fonts

Written by Scott Moonen

July 7, 2008 at 12:16 pm

GCC binary conditional

with 3 comments

I recently ran into a nifty GCC extension to the C/C++ language, the binary conditional:

z = x ?: y;

At first glance this looks like the C++ ternary operator:

z = x ? y : w;

But notice that this new operator above is binary — it has only two parameters.  Unlike the dash and greater-than symbols in the C arrow operator (pointer -> member), GCC does not require that the binary conditional’s question mark and colon be adjacent, but you should probably write them adjacently to better distinguish them from the ternary operator.

But what does the binary conditional do?  In short, it is analagous to the Perl || operator, the Python or operator, and the Ruby || operator.  It evaluates to the value on the left unless the value on the left evaluates to false, in which case it evaluates to the value on the right.

x y x ?: y
0 0 0
0 80 80
NULL 0x16E212B4 0x16E212B4
15 0 15
15 20 15
0x16E212BC 0x16E212E8 0x16E212BC

You may wonder why the C || operator can’t be used for this same purpose. The reason for this is that C’s || operator performs a pure logical or operation: it always collapses the result value to 0 or 1. For example, the expression 80 || 0 evaluates to 1, not to 80. However, the expression 80 ?: 0 evaluates to 80.

That’s pretty nifty, although ?: is certainly a bit unfortunate; it’s not obvious from looking at the operator what it should do. Worse, it appears that the binary conditional is unique to GCC. I’ve tried this with several other C/C++ compilers without success.

There is, however, a more portable way to accomplish the same thing. Instead of writing x ?: y, you can write the equivalent x ? x : y. This is a little less concise, but it has the advantage that any skilled C programmer can immediately understand what it does. And it is more portable.

Written by Scott Moonen

May 27, 2008 at 8:14 am

Richard Scarry and hexadecimal

with 2 comments

When you read a hexadecimal number out loud, how do you pronounce the letters?

At my workplace, I’ve grown used to our custom of pronouncing the letters using the Joint Army/Navy Phonetic Alphabet standardized in 1941. The letter digits are pronounced Able, Baker, Charlie, Dog, Easy and Fox. Under this scheme, the hexadecimal number 0x7F8D3BC0 would be pronounced “Seven Fox Eight Dog Three Baker Charlie Zero.” This was disorienting to me at first, but after eight years this is now so natural that this is how I pronounce the digits in my mind even if I’m not speaking them.

We’ve started collecting Richard Scarry’s children’s books. Richard Scarry writes with a degree of detail and whimsy that holds an adult’s interest — much like old-school Sesame Street. (How far it has fallen — modern-day Sesame Street is much too postmodern, pluralistic, saccharine and juvenile for my taste. I console myself by searching for old Sesame Street clips on Youtube.) Recently I was amused and pleased to discover that one of Richard Scarry’s characters is named Able Baker Charlie! What a strange juxtaposition of worlds for me — programming and children’s books.

Able Baker Charlie is a mouse. He is a baker, and assists Baker Humperdink, a pig. Despite his small size, Able Baker Charlie is capable assisting with any step of the baking process, from stoking the oven, to mixing the dough, to putting loaves in the oven, and even delivering bread around Busytown. Below you may see a picture of Able Baker Charlie ably distributing French baguettes to Louie’s Restaurant.

Richard Scarry served in the U. S. Army during World War II. No doubt this is the source of the Able Baker Charlie aptonym. It still gives me a chuckle every time we read it.

Written by Scott Moonen

March 1, 2008 at 7:46 pm

Programming Fonts

with 4 comments

I’ve long been dissatisfied with Courier New as a programming font; I’ve found the characters to be bulky and the serifs distracting.  For the past year or so I’ve settled on the beautiful Lucida Console as my favorite font to code in.  Lucida Console is bundled with Windows XP and Windows Vista, and unfortunately it doesn’t seem to be available for free redistribution.  However, you can find some other aesthetically pleasing fixed-width programming fonts, some of which are available for free download, at Hamstu’s Typography of Code.

HT: Woot

Written by Scott Moonen

February 4, 2008 at 7:36 pm

Gödel, Escher, Bach: an Eternal Golden Braid

with 2 comments

Hofstadter, Douglas. Gödel, Escher, Bach: an Eternal Golden Braid. New York: Basic Books, 1999.

This book is an excellent and fun, if lengthy, romp through art (visual, literary, and musical), mathematics, logic, provability and computability, linguistics, cognitive science and artificial intelligence, and more. Hofstadter cleverly explores a myriad of amazing connections between all these fields. And while he ends up drawing no substantive conclusions about his final hypothesis of emergent intelligence, the journey is no less exciting.

I disagree with Hofstadter’s opinion of the nature of intelligence. But I thoroughly enjoyed this book for a number of reasons. First, despite its imposing size, it is accessible; Hofstadter presents mathematical proofs in an easily understood way, using examples, analogies, and much explanatory prose. Second, despite its imposing size, it is delightfully fun; this book is brimming with humor, wit, cleverness, and exciting coincidences. Lastly, it is an excellent introduction to a broad variety of topics.

I first read this book in eighth grade and deeply enjoyed it. In part this book served as my introduction to the exciting fields of logic, mathematics, and computer science.

Written by Scott Moonen

March 25, 2005 at 11:01 am

Python persistence

leave a comment »

I’ve implemented a rudimentary persistent object store in Python. It is implemented as a Python extension module that implements a set of persistent types: strings, integers, floats, lists, and dictionaries. Each of these are backed by the persistent object store, which is implemented using a memory-mapped file.

In addition, using a specially crafted Python base class for Python objects, Python objects may be stored in the object store (as dictionaries) and instantiated out of the object store.

The result is an persistent object graph (the root of which is a persistent dictionary) whose objects and attributes may be manipulated in-place using native Python syntax. Rudimentary locking is provided so that multiple Python threads / processes may concurrently manipulate the object store.

Details

Some aspects of this system are:

  • It is a Python extension module written in C and C++.
  • It is tested on Linux. It will likely work on *BSD systems, though it is possible that the location of the mapped storage may need to be moved.
  • It is implemented in a hierarchical manner:
    • A page manager handles the allocation of 4kB pages within a memory-mapped file. It is multi-process safe. It is, in a sense, a glorified sbrk() for a memory-mapped file.
    • A heap manager abstracts the page manager’s services to manage the allocation and deallocation of arbitrary-sized storage segments within the memory-mapped file. It is essentially a malloc() and free() for a memory-mapped file. This is also multi-process safe.
    • An object manager manages five new base types (persistent int, float, string, list, and dictionary) backed by persistent storage, using the heap manager’s services. It also provides rudimentary locking facilities for concurrency-safeness.
    • The persist Python extension uses the object manager’s services to implement persistent types that mimic the equivalent Python types. Additionally, it has the ability to reinstantiate a Python object that was stored as a dictionary (using the appropriate Python base class). The object manager’s locking facilities are made available for application usage.
  • Only one file may be mapped at a time (because it is mapped to a fixed logical address).
  • It is available for use under the MIT license. Contact me if you are interested in using it.

Examples

Some examples of its use are a simple counter:

import persist

root = persist.root()
root.lockExcl()
try :    root['x'] += 1                # Increment counter
except : root['x'] = 1                 # First pass; initialize
print "Content-type: text/html\n"
print "<p>You are visitor " + str(root['x']) + " to visit this site!</p>"
root.unlock()

and rudimentary objects:

import persist
from pbase import pbase

class person (pbase) :
  def __init__(self, name = "", age = 0) :
    pbase.__init__(self)
    self.name = name
    self.age = age

  def printAge(self) :
    print "<p>" + self.name + " is " + str(self.age) + " years old</p>"

root = persist.root()
root.lockExcl()

if not root.has_key('Joe') :            # First time through
  root['Joe'] = person('Joe', 27)
if not root.has_key('John') :           # First time through
  root['John'] = person("John", 29)

# On subsequent passes we will retrieve the objects stored on the first pass.

print "Content-type: text/html\n"
root['Joe'].printAge()
root['John'].printAge()

root.unlock()

Written by Scott Moonen

October 1, 2004 at 11:07 am