1.10.3 Thin Ice
Keywords: ext

##
1.10.3 Thin Ice

There are a few situations where seemingly harmless use of a borrowed reference can lead to problems. These all have to do with implicit invocations of the interpreter, which can cause the owner of a reference to dispose of it.

The first and most important case to know about is using `Py_DECREF()` on an unrelated object while borrowing a reference to a list item. For instance:

      bug(PyObject *list) {
          PyObject *item = PyList_GetItem(list, 0);

          PyList_SetItem(list, 1, PyInt_FromLong(0L));
          PyObject_Print(item, stdout, 0); /* BUG! */
      }

This function first borrows a reference to `list[0]`, then replaces `list[1]` with the value `0`, and finally prints the borrowed reference. Looks harmless, right? But it's not!

Let's follow the control flow into `PyList_SetItem()`. The list owns references to all its items, so when item 1 is replaced, it has to dispose of the original item 1. Now let's suppose the original item 1 was an instance of a user-defined class, and let's further suppose that the class defined a `__del__()` method. If this class instance has a reference count of 1, disposing of it will call its `__del__()` method.

Since it is written in Python, the `__del__()` method can execute arbitrary Python code. Could it perhaps do something to invalidate the reference to `item` in `bug()`? You bet! Assuming that the list passed into `bug()` is accessible to the `__del__()` method, it could execute a statement to the effect of "`del list[0]`", and assuming this was the last reference to that object, it would free the memory associated with it, thereby invalidating `item`.

The solution, once you know the source of the problem, is easy: temporarily increment the reference count. The correct version of the function reads:

      no_bug(PyObject *list) {
          PyObject *item = PyList_GetItem(list, 0);

          Py_INCREF(item);
          PyList_SetItem(list, 1, PyInt_FromLong(0L));
          PyObject_Print(item, stdout, 0);
          Py_DECREF(item);
      }

This is a true story. An older version of Python contained variants of this bug and someone spent a considerable amount of time in a C debugger to figure out why his `__del__()` methods would fail...

The second case of problems with a borrowed reference is a variant involving threads. Normally, multiple threads in the Python interpreter can't get in each other's way, because there is a global lock protecting Python's entire object space. However, it is possible to temporarily release this lock using the macro `Py_BEGIN_ALLOW_THREADS`, and to re-acquire it using `Py_END_ALLOW_THREADS`. This is common around blocking I/O calls, to let other threads use the CPU while waiting for the I/O to complete. Obviously, the following function has the same problem as the previous one:

      bug(PyObject *list) {
          PyObject *item = PyList_GetItem(list, 0);
          Py_BEGIN_ALLOW_THREADS
          ...some blocking I/O call...
          Py_END_ALLOW_THREADS
          PyObject_Print(item, stdout, 0); /* BUG! */
      }