Python embedding (Part 2.5)

Last time I wrote about calling python functions from C/C++. I must admit I kept it very short, so today a bit more about this topic.

1. Wrapping PyObject* into a smart pointer

First I will show you how to use boost::intrusive_ptr to manage python objects. This frees us from doing ugly calls to Py_INCREF or Py_XDECREF all over our code and more important reduce the likelihood of introducing memory leaks by accident (python reference counting is really a mess in my opinion).

To use boost::intrusive_ptr you need to include the following file:

#include <boost/intrusive_ptr.hpp>

Next we need to add two additional functions to the boost namespace:

namespace boost
  inline void intrusive_ptr_add_ref(PyObject* p)

  inline void intrusive_ptr_release(PyObject* p)

They are used by the intrusive_ptr to do the reference counting for us.

Basically at this point we have everything we need, but I like to do one more thing to make my life a bit easier, a typedef for the intrusive_ptr:

typedef boost::intrusive_ptr<PyObject> nxPyObjectPtr_t;

This makes the code a lot cleaner.

2. Calling a python function by name

Now we have all tools needed to call a python function by name in a pretty safe way. This means keeping the probability of memory leaks to a minimum. Here is the method declaration which goes into the nx_PythonManager class:

nxPyObjectPtr_t callFunction(const char *name,
                             const nxPyObjectPtr_t& args = nxPyObjectPtr_t());

The first argument is the name of the python function you want to call. One thing to note here: The function must reside into the main python namespace (__main__ module). You can’t call functions from other python modules this way.

As you can see we are using our intrusive_ptr<PyObject> to pass arguments in and the return value out of the method. You’ll see in a second how exactly this works. If we wouldn’t have done it this way and returned raw pointers, the user of the engine would have needed to use Py_DECREF by themselves… not a good idea.

Now to the implementation:

nxPyObjectPtr_t nx_PythonManager::callFunction(const char *name,
                                               const nxPyObjectPtr_t& args)
  PyObject *main = PyImport_AddModule("__main__");
  PyObject *func = PyObject_GetAttrString(main, name);

  if (!func || !PyCallable_Check(func))
    return nxPyObjectPtr_t();

This code should be familiar if you’ve read parts 1+2 of these article.  We just get the __main__ python module (in which our function needs to reside). Than we use PyObject_GetAttrString to get our function object. The next 4 lines are for error checking.

  PyObject *result = PyObject_CallFunctionObjArgs(func, args.get(), NULL);

Here is the actual call to the python function. The 2nd value passed is the argument list for the python function. Because we wrapped the PyObject* into intrusive_ptr we need to use “.get()” to get the raw pointer. The argument list can be anything you want: nothing, one python object or a complete python dictionary. The called python function is responsible for interpreting the list.

Last but not least we wrap the return value into an intrusive_ptr and return it:

  if (result == Py_None)
    return nxPyObjectPtr_t();
    return nxPyObjectPtr_t(result, false);

Py_None is basically the same as “return void”, so we return an empty pointer.

One very important thing is the construction of the “nxPyObjectPtr_t”. You may have noticed the 2nd argument “false” this is extremely important. The intrusive_ptr contructor by default increases the reference count of the object: “intrusive_ptr(T *p, bool addref = true)”. Because we don’t want that, we need to explicity call it with “false”. If you’ll forget this you will leak memory. Because of the sneakiness of this behaviour I have created a macro which helps to “remember” the false:

#define NX_PY_OBJECT_PTR(pyObject) nxPyObjectPtr_t(pyObject, false)

Here is an example how to use it:

nxPyObjectPtr_t obj = nxPyObjectPtr_t(PyLong_FromInt(1000), false));
nxPyObjectPtr_t obj = NX_PY_OBJECT_PTR(PyLong_FromInt(1000));

The 1st line is without macro and the 2nd line with macro.

That’s all about it, wasn’t so bad after all.

3. Further reading

As usual some useful links about todays topics:


Smart pointers to boost your code (Tutorial)

Python reference counting

Python embedding (Part 2)

Last time I covered the basics of embedding python into the engine. Today I will show you how to expose C-functions or more exactly static C++ class methods to the python interpreter. Furthermore I’ll give you some hints how to call python functions from C/C++ code.

1. Exposing C/C++ functions to python

First we need to add a new static member variable to the nx_PythonManager class declaration:

  static PyMethodDef m_methodTable[];

All functions which should be callable from python must be added to this table. Here is the definition with some functions stripped out:

PyMethodDef nx_PythonManager::m_methodTable[] = {
  { "printVersion", nx_PythonManager::printVersion, METH_NOARGS, "Prints the python interpreter version." },
  { "executeFile", nx_PythonManager::executeFile, METH_O, "Executes a python script file." },
  { "registerEventListener", nx_PythonManager::registerEventListener, METH_VARARGS, "Registers a script event listener." },
  { NULL, NULL, 0, NULL }

Each line defines exactly 1 function which should be python callable. The 4 arguments in each line are from left to right:

  1. Python function name.
  2. C/C++ function which should be exposed. In our case we use static class methods which sit inside the nx_PythonManager class.
  3. Number and type of function arguments. METH_NOARGS = no argument at all, METH_O = exactly one python object as argument, METH_VARARGS = variable number of arguments (0, 1, 2, …)
  4. Description of the function.

The last line “{ NULL, NULL, 0, NULL }” shows python the end of the table.

Every python callable function must follow the same scheme:

PyObject* functionName(PyObject *self, PyObject *args);

The self argument is only needed by python for object orientet stuff and is therefore uninteresting for us at the moment. To prevent compiler warnings I tend to just comment “self” out like this:

PyObject* functionName(PyObject */*self*/, PyObject *args);

Ok enough of the basics an actual example of such a function:

// declaration from the nx_PythonManager class
static PyObject* printVersion(PyObject *self, PyObject *args);

// implementation
PyObject* nx_PythonManager::printVersion(PyObject */*self*/, PyObject */*args*/)
  std::cerr << Py_GetVersion() << std::endl;

Easy enough, all this function does is to print a python version string to std::cerr. This function was defined in the method table with METH_NOARGS, so no arguments are passed. Py_RETURN_NONE is a handy python macro for returning Py_None while keeping the reference count of the object right. Py_None is the python equivalent to C’s void.

Now there is only one thing left before we can call our function from python: Making a buildin module from our method table and add it to the main python module. This is done in the nx_PythonManager::init method. I gave you basic insight to this method in the first part of this article. These are the lines we need to add:

// this code goes after the 'PyObject *sysModule = PyImport_AddModule("sys");' line
PyObject *nxModule = Py_InitModule3("nx", m_methodTable, "NX-Engine python module.");

if (PyModule_AddObject(mainModule, "nx", nxModule) != 0)
  return false;

The last 4 lines should be familiar. They are basically the same as last time for adding the sys-module. Only difference is that this time we add our custom build module ‘nx’ which we get from the Py_InitModule3 function. This function takes the module name “nx”, our method table and a document string describing the module.

Now it is possible to run our little code function from python like this: nx.printVersion().

2. Calling python functions from C/C++

I actually do a call to a python function in the NX-Engine exactly at 1 point at the moment: Sending events to script listeners. So this will be pretty quick, here is the code needed for calling the python function:

PyObject *onEvent = PyObject_GetAttrString(m_callback, "onEvent");

Here the onEvent function is extracted from a script event listener. I haven’t talked about listener or event types at all, so just ignore the exact meaning of this line for the moment. All you need to know is that you need a way to get a python function object to your code. For example this could happen with a C-function which is exposed to python and takes a function object.

if (!PyCallable_Check(onEvent))
  return false;

This is actually just a safety check if the python object is really callable.

PyObject *result = PyObject_CallFunction(onEvent, (char*)"O", event.getPythonEvent())
bool r = (result == Py_True ? true : false)

return r;

Ok, here we call the python function with PyObject_CallFunction. In this case it takes exactly one argument of type object which is indicated with (char*)”O”. The last argument is the object passed, here of a custom python object type (I’ll eventually write about how to create custom types in the future). The result of the function call is converted to a C-type and returned. Furthermore notice the calls to Py_DECREF. This is necessary to free unused resources otherwise we would leak memory.

3. Further reading

I intentionally left out a few things for the moment like parsing METH_VARARG arguments or building an argument list with Py_BuildValue. I eventually will catch up to this in the future, meanwhile some links to the excellent python documentation on these topics:

Parsing arguments and building values

Buidling values

Extracting parameters