Nathan Ostgard

In ActionScript 3, you can extend a class from the Proxy class to override the behavior of many language level operations, making it a valuable tool for advanced development. However, they are much slower than sealed classes, so be careful with their use when high performance is necessary.

Proxy allows you to override:

// function calls
foo.bar(1, 2)
=>
foo.callProperty('bar', 1, 2)

// property get
foo.bar
foo['bar']
=>
foo.getProperty('bar')

// property set
foo.bar = 123
foo['bar'] = 123
=>
foo.setProperty('bar', 123)

// property delete
delete foo.bar
delete foo['bar']
=>
foo.deleteProperty('bar')

// in operator
'bar' in foo
=>
foo.hasProperty('bar')

// descendant operator
foo..bar
=>
foo.getDescendants('bar')

Proxy also allows you to override the handling of for..in and for each..in loops, using the nextNameIndex, nextName, and nextValue methods. This for..in loop:

for (var key:String in foo) {
  trace(key);
}

on a proxy, is equivalent to:

var i:int = 0;
while (true) {
  var index:int = foo.nextNameIndex(i);
  if (index == 0)
    break;
  var key:String = foo.nextName(index);
  trace(key);
  i++;
}

Similarly, this for each..in loop:

for each (var value:* in foo) {
  trace(value);
}

on a proxy, is equivalent to:

var i:int = 0;
while (true) {
  var index:int = foo.nextNameIndex(i);
  if (index == 0)
    break;
  var value:* = foo.nextValue(index);
  trace(value);
  i++;
}

Here is an example stub class implementing these methods:

package {
  import flash.utils.Proxy;
  import flash.utils.flash_proxy;

  public class ExampleProxy extends Proxy {
    private var items:Array;

    public function ExampleProxy() {
      items = [];
    }

    override flash_proxy function deleteProperty(name:*):Boolean {
      return delete items[name];
    }

    override flash_proxy function getProperty(name:*):* {
      return items[name];
    }

    override flash_proxy function hasProperty(name:*):Boolean {
      return name in items;
    }

    override flash_proxy function nextNameIndex(index:int):int {
      if (index > items.length)
        return 0;
      return index + 1;
    }

    override flash_proxy function nextName(index:int):String {
      return String(index - 1);
    }

    override flash_proxy function nextValue(index:int):* {
      return items[index - 1];
    }
  }
}

Game Objects is a collection of 2D and 3D maths classes, and algorithms for helping in the creation of games with Python.

Will McGugan’s Game Objects library has been invaluable for my dabblings in Python game development. It provides optimized objects for common graphics math (3D vector class, matrix class, etc.). The objects are flexible and framework independent, so you can use them with the platform of your choice, be it pygame, PyOpenGL, or DirectPython.

It’s a somewhat new project, and the documentation a bit sparse, but the library is pretty straightforward and he’s been updating it regularly.

Relevant links:

• Project Page
• Documentation
• Read more about Game Objects on Will’s blog.

Decided to move some of my vertex array code over to ARB_vertex_buffer_object, to see if I could get a bit of a speed boost out of PyOpenGL. Much to my dismay, these functions don’t have the sexy extension wrappers I talked about before, so it was a bit of a chore to get it all working.

Here’s a little wrapper class I wrote to make things easier:

from OpenGL.GL import *
from OpenGL.raw import GL
from OpenGL.arrays import ArrayDatatype as ADT

class VertexBuffer(object):

  def __init__(self, data, usage):
    self.buffer = GL.GLuint(0)
    glGenBuffers(1, self.buffer)
    self.buffer = self.buffer.value
    glBindBuffer(GL_ARRAY_BUFFER_ARB, self.buffer)
    glBufferData(GL_ARRAY_BUFFER_ARB, ADT.arrayByteCount(data), \
                 ADT.voidDataPointer(data), usage)

  def __del__(self):
    glDeleteBuffers(1, GL.GLuint(self.buffer))

  def bind(self):
    glBindBuffer(GL_ARRAY_BUFFER_ARB, self.buffer)

  def bind_vertexes(self, size, type, stride=0):
    self.bind()
    glVertexPointer(size, type, stride, None)

  ... snipped for length ...

Download buffers.py.

So, how to use it? Let’s say you have a Python list with a bunch of vertexes with float x, y, and z components:

verts = [[x, y, z], [x, y, z], [x, y, z], ...]

You’ll have to use NumPy or an equivalent to convert it to a PyOpenGL compatible array:

import numpy
numpy_verts = numpy.array(verts, dtype=numpy.float32)

Create the VertexBuffer object with it:

buffer = VertexBuffer(numpy_verts, GL_STATIC_DRAW)

Use it in your day to day rendering:

glEnableClientState(GL_VERTEX_ARRAY)
buffer.bind_vertexes(3, GL_FLOAT)
glDrawElementsui(GL_TRIANGLES, indexes)

They’re not shown in the snippet, but I’ve also defined bind_colors, bind_edgeflags, bind_indexes, bind_normals, and bind_texcoords, as shortcuts for the rest of the GL array functions.

When I have Django URL patterns like:

urlpatterns = patterns('',
  (r'^(?P<foo_id>\w+)/$', 'myproject.myapp.views.show'),
  (r'^(?P<foo_id>\w+)/edit/$', 'myproject.myapp.views.edit'),
  (r'^(?P<foo_id>\w+)/delete/$', 'myproject.myapp.views.delete'),
)

I always end up with views like:

def show(self, foo_id):
  foo = get_object_or_404(Foo, id=foo_id)
  ...

def edit(self, foo_id):
  foo = get_object_or_404(Foo, id=foo_id)
  ...

def delete(self, foo_id):
  foo = get_object_or_404(Foo, id=foo_id)
  ...

Having the same few lines at the top of every function in makes me feel dirty. You can clean this up with a Middleware class, replacing foo_id with the actual object before calling the view.

myproject/myapp/middleware.py looks like this:

from django.shortcuts import get_object_or_404
from someapp.models import Foo

class FindObjects:
  def process_view(self, request, view_func, view_args, view_kwargs):
    if 'foo_id' in view_kwargs:
      view_kwargs['foo'] = get_object_or_404(Foo, id=view_kwargs['foo_id'])
      del view_kwargs['foo_id']

I include it in settings.py:

MIDDLEWARE_CLASSES = (
  ...
  'myproject.myapp.middleware.FindObjects',
)

And now my views look like this:

def show(request, foo):
  ... yay! do stuff with foo! ...

Very useful for views with many parameters, or views with multiple optional parameters. In one project, I have views which process data for three different situations: data in a metropolitan area, data in a city, or data in a city within a metro. My middleware for that looks like this:

from django.shortcuts import get_object_or_404
from phuce.metros.models import City, Metro

class FindObjects:
  def process_view(self, request, view_func, view_args, view_kwargs):
    city = metro = None
    if 'metro_link' in view_kwargs:
      metro = get_object_or_404(Metro, link=view_kwargs['metro_link'])
      del view_kwargs['metro_link']
      view_kwargs['metro'] = metro
    if 'city_link' in view_kwargs:
      if metro:
        city = get_object_or_404(City, metro=metro, \
                                 metro_link=view_kwargs['city_link'])
      else:
        city = get_object_or_404(City, link=view_kwargs['city_link'])
      del view_kwargs['city_link']
      view_kwargs['city'] = city

It’s much tidier without that monster in my views.

Normally if you wanted to get access to an OpenGL extension, you’d have to do something like:

from ctypes import *    
from OpenGL import platform
gl = platform.OpenGL
glGetShaderiv = gl.glGetShaderiv
glGetShaderiv.argtypes = [c_int, c_int, POINTER(c_int)]

Gross. Luckily, PyOpenGL version 3.0.0a5 and later will automatically map ARB standard functions for you when they are available. Now all you have to do is:

from OpenGL.GL import *

And everything will be magically available to you, wrapped so you don’t have to use ctypes to create c_char_p or c_int variables to pass them.

Watch out, though! The functions won’t be mapped correctly if your display context hasn’t been created yet. Always make sure to import OpenGL after the pygame display has been initialized.

# first initialize pygame
import pygame
from pygame.locals import *
pygame.init()
pygame.display.set_mode((800, 600), OPENGL | DOUBLEBUF)

# now you can import it
from OpenGL.GL import *

I would love to know how to manually tell PyOpenGL to re-map everything after initializing pygame, though. I don’t like having to reposition the import like that. Nathan Gray has a deep reload replacement for the reload builtin, which might work, but I’m not sure if that’s the best solution to this problem…

There’s no way in Django (that I’ve found) to render a field’s name in a template. This means you end up with <th>Field Name</th> all over your templates. Why hello there, DRY violation!

The fields are stored in model._meta.fields, but templates don’t allow you to access variables which start with an underscore. I’ve got two little utility functions I wrote for myself to generate a dict of labels I can use in my templates:

def get_labels_for(model, cap=True, esc=True):
  from django.template.defaultfilters import capfirst
  from django.utils.html import escape
  labels = {}
  for field in model._meta.fields:
    label = field.verbose_name
    if cap:
      label = capfirst(label)
    if esc:
      label = escape(label)
    labels[field.name] = label
  return labels

def with_labels(context, cap=True, esc=True):
  from django.db.models import Model
  result = context.copy()
  for k, v in context.iteritems():
    if isinstance(v, Model):
      result[k + '_labels'] = get_labels_for(v, cap, esc)
    elif hasattr(v, '__getitem__') and len(v) > 0:
      if isinstance(v[0], Model):
        result[k + '_labels'] = get_labels_for(v[0], cap, esc)
  return result

The parameters:

• model can be a model class or a model instance.
• If cap is True, the first letter of each label will be capitalized.
• If esc is True, the labels will be escaped for HTML.

So, in your view:

def some_view(request, foo_id):
  foo = get_object_or_404(Foo, id=foo_id)
  context = {'foo': foo, 'foo_labels': get_labels_for(foo)}
  return render_to_response('foo.html', context)

And in your template:

{{ foo_labels.bar }}: {{ foo.bar }}

with_labels works the same way, except you can just surround your context with it:

def some_view(request, foo_id):
  foo = get_object_or_404(Foo, id=foo_id)
  bars = Bars.objects.all()
  context = {'foo': foo, 'bars': bars}
  return render_to_response('foo.html', with_labels(context))

It will detect the models and lists of models in the context and add foo_labels and bars_labels to the context.

In Python:

import os.path, random, string
def random_filename(chars=string.hexchars, length=16, prefix='', suffix='', \
                    verify=True, attempts=10):
  for attempt in range(attempts):
    filename = ''.join([random.choice(chars) for i in range(length)])
    filename = prefix + filename + suffix
    if not verify or not os.path.exists(filename):
      return filename

>>> random_filename()
'DC53e02B08eF47e9'
>>> random_filename(chars='hi', length=32)
'hhhihihhhiiihhhihhiiiiiihhhhiihh'
>>> random_filename(prefix='username', suffix='.txt')
'username7dbd29aBdD25BeB9.txt'

# returns None if it can't find a filename
>>> open('xxx', 'w').close()
>>> str(random_filename(chars='x', length=3))
'None'

In Ruby:

def random_filename(opts={})
  opts = {:chars => ('0'..'9').to_a + ('A'..'F').to_a + ('a'..'f').to_a,
          :length => 16, :prefix => '', :suffix => '',
          :verify => true, :attempts => 10}.merge(opts)
  opts[:attempts].times do
    filename = ''
    opts[:length].times { filename << opts[:chars][rand(opts[:chars].size)] }
    filename = opts[:prefix] + filename + opts[:suffix]
    return filename unless opts[:verify] && File.exists?(filename)
  end
  nil
end

When most pages in a site require authentication, decorating all the views with @login_required can be annoying. You can reverse the default behavior by creating a custom middleware class:

import urllib
from django.conf import settings
from django.contrib.auth import REDIRECT_FIELD_NAME
from django.http import HttpResponseRedirect

def allow_anonymous(view_func):
  view_func.allow_anonymous = True
  return view_func

class RequireLogin:
  def process_view(self, request, view_func, view_args, view_kwargs):
    if request.path != settings.LOGIN_URL and \
      not request.user.is_authenticated() and \
      not getattr(view_func, 'allow_anonymous', False):
      url = '%s?%s=%s' % (settings.LOGIN_URL, REDIRECT_FIELD_NAME, \
                          urllib.quote(request.get_full_path()))
      return HttpResponseRedirect(url)

That’s an ugly block of code but it’s not too complex. allow_anonymous is a function decorator, like login_required. It just tags the function to tell the middleware that authentication isn’t required. The RequireLogin class verifies that the user is logged in. If not, and if the function is not decorated with allow_anonymous, it redirects to settings.LOGIN_URL.

So put that code in a file in your project – let’s say, yourproject/yourapp/middleware.py. Then open settings.py and add "yourproject.yourapp.middleware.RequireLogin" to MIDDLEWARE_CLASSES. This tells Django about your new middleware class, and RequireLogin.process_view will be called any time a view is about to be rendered.

Now you can use it in your view:

from yourproject.yourapp.middleware import allow_anonymous

def some_private_view(request):
  # won't be accessible unless user is logged in
  return HttpResponse('Hello, user!')

@allow_anonymous
def some_public_view(request):
  return HttpResponse('Hello, world!')

If all is working correctly, some_private_view should ask for a login, but some_public_view will allow viewing without it.

Update: I’ve updated the code to fix a bug when using django.contrib.auth.views.login for your login view. As you can’t mark this view function with @allow_anonymous, it would infinitely redirect back to it. Oops! Thanks for pointing it out, Phil.

Simon Willison linked to an article which argues:

Warnings cause us to lose our work, to mistrust our computers, and to blame ourselves. A simple but foolproof design methodology solves the problem: “Never use a warning when you mean undo.” And when a user is deleting their work, you always mean undo.

The post spawned a discussion on undo techniques for Django. I decided to implement one method and post the results here. It only offers undo for deleting, and not for editing. Other than that, I like it.

How it works

It’s a pretty simple concept: add a trashed_at field to your model, with the default value of None. When delete() is called on an object, if trashed_at is None, set it to the current time but don’t delete it. If it’s not None, actually delete it from the database.

Continue reading...

Sometimes users want to bring text from an editor like Word into your web forms. You will often find nasty little characters hiding in the text, like ’\u2022’ (a.k.a. the notorious bullet). These characters will normally throw errors if you try to convert them to ASCII:

>>> u'\u2022'.encode('ascii')
Traceback (most recent call last):
  File "<console>", line 1, in ?
UnicodeEncodeError: 'ascii' codec can't encode ... (yadda yadda)

To sanitize these strings and make them XML/HTML safe:

>>> u'\u2022'.encode('ascii', 'xmlcharrefreplace')
'•'

It translates the invalid characters into their XML equivalents. Woo! You can also use 'ignore' or 'replace' (replaces with ?):

>>> u'\u2022'.encode('ascii', 'ignore')
''
>>> u'\u2022'.encode('ascii', 'replace')
'?'

If you’re getting nasty Unicode errors from your templates in Django now that they’ve merged the Unicode branch, this might help as a quick fix.

Foo.objects.get(keywords__contains='bar') will make your database cry when you run it against a 30 million row table. At times it becomes necessary to segment a large table into many smaller tables.

Say you have these models:

class City(models.Model):
  city = models.CharField(maxlength=255)
  state = models.USStateField()

class Listings(models.Model):
  name = models.CharField(maxlength=100)
  street = models.CharField(maxlength=100)
  city = models.ForeignKey(City)
  zip = models.CharField(maxlength=10)
  keywords = models.CharField(maxlength=255)

The listings table has gotten huge. Users only search within a single city, so you want to break the listings table into a table for each city. Instead of yourapp_listing, you now have yourapp_listing_1, yourapp_listing_2, etc. All the listings are in the matching table for their city.

I couldn’t find a documented way to make db_table dynamic in Django. So, how do you get Django to use the right table when you’re querying for listings? Here’s how I did it:

class ListingManager(models.Manager):
  def get_table_for(self, city):
    '''
    someapp_listing if city == None
    someapp_listing_012 if city.id == 12
    '''
    table = '_'.join((self.model._meta.app_label,
                      self.model._meta.module_name))
    if city:
      table += '_%03d' % city.id
    return table

  def in_city(self, city):
    self.city = city
    self.model._meta.db_table = self.get_table_for(city)
    return self

class Listing(models.Model):
  city = models.ForeignKey(City)
  objects = ListingManager()

  def delete(self):
    Listing.objects.in_city(self.city)
    super(Listing, self).delete()

  def save(self):
    Listing.objects.in_city(self.city)
    super(Listing, self).save()

It’s simple to use:

>>> city = City.objects.get(id=2)
>>> Listings.objects.in_city(city).all()

Edit 2007/08: Simplified the code. When I first wrote this I replaced db_table with a subclass of str that called ListingManager.get_table_for. It was overkill, and didn’t always work as expected (e.g., 'foo' + Listing._meta.db_table didn’t work).

This post on 2D Boy states:

“As love and effort increase, the probability of self destruction approaches 1.” - Kyle’s Theorem of Destruction #2a

Or, the more you care about what you’re working on, the less likely you’ll actually turn it into something totally awesome.

The more you work on something, the more personally invested you become. The more personally invested you are, the less likely you are to look at it objectively and trash or rethink problem areas. The product suffers. This applies to any type of development, but it’s especially true for game development.

How to fix it? 2D Boy suggests occasional distraction with side projects. I have some more thoughts for large teams…

First: organize your staff into small, independent, Scrum-ish) groups. Each group must be self-sufficient, with an artist, programmer, etc. When assigned a task, it must be able to create a playable product without outside interaction with other departments.

Next: identify the areas of development for the project. For example, in Battlefield 2142, these might be: infantry, vehicles, conquest gameplay, titan gameplay, statistics and awards, and user interface.

Finally: each group works on one specific area for one iteration of your Scrum development cycle. In the next iteration, rotate the group to a new area of development.

This results in:

• Fresh eyes looking over the important parts of your game each iteration of the development cycle.
• Less competition between departments.
• The game won’t suffer because one group doesn’t want to make changes to the precious content they’ve been slaving over for the past 9 months.

Did a bit of running around today to get Django sending email via Gmail. It’s simple once you figure it out.

If you’re running 0.96, upgrade to the latest development version or apply the patch from ticket #2897. 0.96 does not support TLS, which Gmail requires. Then add the appropriate values to settings.py:

EMAIL_USE_TLS = True
EMAIL_HOST = 'smtp.gmail.com'
EMAIL_HOST_USER = 'youremail@gmail.com'
EMAIL_HOST_PASSWORD = 'yourpassword'
EMAIL_PORT = 587

You can use the shell to test it:

>>> from django.core.mail import send_mail
>>> send_mail('Test', 'This is a test', to = ['youremail@somewhere.com'])

Edit: Bryan commented that send_mail is deprecated. Use EmailMessage instead:

>>> from django.core.mail import EmailMessage
>>> email = EmailMessage('Hello', 'World', to = ['youremail@somewhere.com'])
>>> email.send()

To install Python, pygame, and PyOpenGL on Windows:

1. Download Python here and install it.
2. Download pygame here and install it. The “Windows” section has an executable installer (e.g. pygame-1.7.1release.win32-py2.5.exe).
3. You should also download and install Numeric from the pygame page.
4. Download ez_setup.py and save it to a folder outside your Python installation (e.g. C:\).
5. Open a command prompt, change to the folder you saved ez_setup.py to, and run it:
   c:\python\python c:\ez_setup.py
   It will download the easy_install executables and put them in your Python\Scripts folder.
6. Now install PyOpenGL using easy_install:
   c:\python\scripts\easy_install PyOpenGL

All done!

As a programmer, I have the tendency to break things down from an engineering perspective. It’s a compulsion. Hearing a concept compels me to turn it into a specification in my head.

Specifications can be fundamentally flawed. Theoretical design can be useful, but the sooner you can start building an application the sooner you will find how it actually works in the wild. By creating the specification first, you may find yourself feeling obligated to adjust your frontend to work with the backend you’ve already designed. This is a bad idea. Things never work exactly as you expect. You want to quickly find the errors in your design.

An example from a project of my own: there was an area in our application where the user was creating start-to-finish timed sessions, with timed “breaks” during the session that the users could record. From the eye-in-the-sky perspective, it made sense to store these breaks as a start time and end time.

But by starting with the interface, I realized that entering breaks was a cumbersome process. It caused us to reevaluate it and realize that a simple number of minutes on break during the session was all that was really necessary, not groups of start and stop times. By catching it early on, we avoided the backend work that would have been scrapped in the end.

Interaction with the product is the most important to a user. If you can keep this in mind during your development your product will benefit tremendously from it. The user doesn’t care how your backend is designed, as long as it works for them. Your product should be designed around the frontend. It’s the responsibility of the development team to make sure the backend performs as needed to support the interface.

With the popularity of platforms like Ruby on Rails, people are building things faster, but applications often fall apart under heavy load.

You can help improve performance by caching dynamic content, reducing the number of database hits, etc. When you get hit really hard by Digg or Slashdot, measures like caching or reducing the number of database hits will only go so far.

For every page view, the browser requests not only the page itself, but every static asset you have in your page—stylesheets, javascript, images, etc. These additional queries can quickly add up until your server is saturated with requests and can’t keep up. Memory runs out, CPU is at 100%, money and traffic is lost.

An easy and effective optimization is to move static content to a secondary HTTP server. Images and stylesheets don’t need the flexibility or overhead of Apache. You just want to serve them fast.

But do it now, before you’re under heavy load. When you’re losing traffic, the last thing you want to be doing is scrambling around editing your templates and HTML to point all your static content to new URLs.

I’ve used both lighttpd and thttpd for serving static content. thttpd is lightning fast, but lighttpd is also very fast and is a bit easier to administer. Both did an excellent job of freeing Apache to deal with the heavy worker request.

Always keep the question in mind: are you getting the most out of your hardware? Inefficiencies can exist in any solution, regardless of how much it’s touted or how comfortable you are with a setup.

Developers wear too many hats. It’s unavoidable in small teams. And three months down the road, when you have an emergency with the database you haven’t looked at since you created it? You won’t remember how to get it running. You might get 9/10 steps right, but you’ll miss one and spend hours screaming trying to figure out what’s wrong.

Write yourself detailed notes—even better, write utility scripts so you don’t have to relearn it every time. The 30 minutes it takes will save you hours down the road.

Some tips when writing rarely used but mission critical utility scripts:

1. Don’t directly modify the database.
   Big no-no if your script has anything to do with database structure or administration. Odds are, database tables will change in the months between running the script. Output SQL statements to text files instead. Then you can check them over and execute them manually.
    
2. Keep framework usage at a bare minimum.
   Keep your script as framework independent as possible. This is especially true if using an in house framework or a framework that is still under heavy development. Framework dependent scripts won’t be kept up to date properly when they’re not being used.
    
3. Write command line help.
   Treat yourself like any other user. Take a few more minutes to add a -h parameter and print usage help. It’s too tempting to rewrite old code when you have to sift through it to remember how to use it.
    
4. Make it verbose.
   When you run the script and find yourself staring at a blank screen for 5 minutes, you’re going to get annoyed. You’ll want to know what’s going on when you run a script for the first time in months. Make your script tell you what it’s doing.

It’s pretty straightforward, really: always assume you’ll forget everything you know, and do everything you can to make your job easier when you do. Force yourself to do it and even worst case scenarios won’t keep you up at night.