Python Object System - Part II

14 Feb 2016

Part I already alluded one important concept: an object is an instance of a class, and a class (itself is also an object) is also an instance of a special class (a metaclass) type.

It actually makes sense if you think about it: a class is a type, so a type of a class should also be a class! In Python it is called metaclass (type of type)!

How are they related to each other? How does a metaclass create a class, and how does a class create an instance? Let’s dive in:

Attributes Of Objects

In Python, you can access attributes to an object easily similar to Javascript. However, this post will not elaborate more on attribute access (read, set, delete) in Python, instead, I will only mention necessary special attributes in order to understand Python’s object model (Part III will talk about attribute references).

Creating a class object (by executing the class statement) implicitly sets some class attributes, one of them is __dict__, which is the dictionary object the class uses to hold most of its class attributes.

Similar thing can be said for creating an instance of a class: creating an instance implicitly sets __dict__ which is used to most (except than __class__ and __dict__) instance attributes.

As you might have guessed already, attribute __dict__ is an essential and fundamental building block of Python’s object system. In fact, __dict__ implements scopes in Python; that means all the attribute (including methods, functions, variables, etc., except a few special ones) lookups of an instance, a class or even an module (also an object!) have to go through __dict__, since that is where they were stored. The take away: Python’s object system layered heavily on top of dictionaries (__dict__ attributes).

Special Methods

After understanding __dict__, let’s talk about special methods. Special methods are methods defined or inherited in a class whose names begin and end with double quotes. Each special method relates to a specific operation. Python implicitly invokes a special method whenever a related operation is performed on an instance object. The special methods that are relevant to our subject here are: __new__ and __init__. Be aware that these methods are class level attributes, not instance attributes, even though they are implicitly invoked for an instance object.

__new__ is a static method defined or inherited in each class. It is responsible for creating an uninitialized instance of a class. __init__ on the other hand, is a normal method defined or inherited in a class, and is used to perform any instance-specific initialization. __init__ method usually contains statements that bind instance attributes (attributes that will end up in instance __dict__). When creating an instance, __new__ is invoked first before __init__; when __init__ is executed, an instance is already created by __new__ and passed to __init__. Let’s see this concept materialized:

# creating a instance of class C:
class C(object):
    def __init__(self, num):
        self.__num__ = num
x = C(23)

# is equivalent to:
x = C.__new__(C, 23)
if isinstance(x, C): type(x).__init__(x, 23)

Metaclasses

We now finally come to the last point, metaclasses. An object’s behavior is mostly determined by the type of the objct; same for classes: a class’s behavior is mostly determined by the class’s metaclass; and type is the metaclass of all Python built-in types, including itself. type creates new ‘type’ objects, a.k.a classes (class objects). The devil is in the detail:

A class statement -

class classname(base-classes):
    statement(s)

contains several components:

  1. classname
  2. base-classes
  3. statement(s) - class body: including attributes, functions, etc.

When a class statement is executed, all the above three components will be used and a class will be created in the type system. Let’s go through the process:

  1. base classes are collected into a tuple t (empty if no base classes)
  2. class body body is isolated
  3. class dictionary d is created (in Python 3 it is created by the __prepare__ method)
  4. the class body is executed in the dictionary d. Afterwards, d is populated with class attributes:
exec(body, globals(), d)
  1. metaclass M of the class is determined (according to some internal algorithm, defaulted to type)
  2. metaclass M is called with three arguments: class name, tuple of base classes and the class dictionary:
  3. The call returns the class object C, which Python then binds to the class name, completing the execution of the class statement.
C = M(classname, t, d)

This is in fact an instantiation of type M, which means the above is also equivalent to the following:

C = M.__new__(M, classname, t, d)
if isinstance(C, M): M.__init__(C, classname, t, d)

How to use a metaclass? In Python 2 (with new-style classes), you do the following:

class Spam(object):
    __metaclass__ = type
    def __init__(self): pass
    def bar(self): print 'bar'

In Python 3, an even better syntax:

class Spam(metaclass=type):
    def __init__(self): pass
    def bar(self): print 'bar'

To create your own metaclass(es), subclass it from type:

class mytype(type):
    def __new__(cls, name, bases, clsdict):
        clsobj = super().__new__(cls, name, bases, clsdict)
        clsobj = modify_old_clsobj_and_return_new_one(clsobj)
        return clsobj

class Spam(metaclass=mytype):
    ...

Observation: metaclasses get information about class definitions at the time of definition, and that is powerful because:

A group of classes who were created with the same metaclass will share the same mutated behavior, this capability can be leveraged to do metaprogramming.

In conclusion, most attributes (including __init__ and __new__) of class and instance objects are stored in __dict__. Attributes __new__ and __init__ are used to create and initialize objects, respectively. A metaclass also uses __new__ and __init__ to create class objects when a class statement is executed.

P.S. This post focuses the discussion only on Python 2.5+ object model