Python embedding (Part 2.5)

Last time I wrote about calling python functions from C/C++. I must admit I kept it very short, so today a bit more about this topic.

1. Wrapping PyObject* into a smart pointer

First I will show you how to use boost::intrusive_ptr to manage python objects. This frees us from doing ugly calls to Py_INCREF or Py_XDECREF all over our code and more important reduce the likelihood of introducing memory leaks by accident (python reference counting is really a mess in my opinion).

To use boost::intrusive_ptr you need to include the following file:

#include <boost/intrusive_ptr.hpp>

Next we need to add two additional functions to the boost namespace:

namespace boost
  inline void intrusive_ptr_add_ref(PyObject* p)

  inline void intrusive_ptr_release(PyObject* p)

They are used by the intrusive_ptr to do the reference counting for us.

Basically at this point we have everything we need, but I like to do one more thing to make my life a bit easier, a typedef for the intrusive_ptr:

typedef boost::intrusive_ptr<PyObject> nxPyObjectPtr_t;

This makes the code a lot cleaner.

2. Calling a python function by name

Now we have all tools needed to call a python function by name in a pretty safe way. This means keeping the probability of memory leaks to a minimum. Here is the method declaration which goes into the nx_PythonManager class:

nxPyObjectPtr_t callFunction(const char *name,
                             const nxPyObjectPtr_t& args = nxPyObjectPtr_t());

The first argument is the name of the python function you want to call. One thing to note here: The function must reside into the main python namespace (__main__ module). You can’t call functions from other python modules this way.

As you can see we are using our intrusive_ptr<PyObject> to pass arguments in and the return value out of the method. You’ll see in a second how exactly this works. If we wouldn’t have done it this way and returned raw pointers, the user of the engine would have needed to use Py_DECREF by themselves… not a good idea.

Now to the implementation:

nxPyObjectPtr_t nx_PythonManager::callFunction(const char *name,
                                               const nxPyObjectPtr_t& args)
  PyObject *main = PyImport_AddModule("__main__");
  PyObject *func = PyObject_GetAttrString(main, name);

  if (!func || !PyCallable_Check(func))
    return nxPyObjectPtr_t();

This code should be familiar if you’ve read parts 1+2 of these article.  We just get the __main__ python module (in which our function needs to reside). Than we use PyObject_GetAttrString to get our function object. The next 4 lines are for error checking.

  PyObject *result = PyObject_CallFunctionObjArgs(func, args.get(), NULL);

Here is the actual call to the python function. The 2nd value passed is the argument list for the python function. Because we wrapped the PyObject* into intrusive_ptr we need to use “.get()” to get the raw pointer. The argument list can be anything you want: nothing, one python object or a complete python dictionary. The called python function is responsible for interpreting the list.

Last but not least we wrap the return value into an intrusive_ptr and return it:

  if (result == Py_None)
    return nxPyObjectPtr_t();
    return nxPyObjectPtr_t(result, false);

Py_None is basically the same as “return void”, so we return an empty pointer.

One very important thing is the construction of the “nxPyObjectPtr_t”. You may have noticed the 2nd argument “false” this is extremely important. The intrusive_ptr contructor by default increases the reference count of the object: “intrusive_ptr(T *p, bool addref = true)”. Because we don’t want that, we need to explicity call it with “false”. If you’ll forget this you will leak memory. Because of the sneakiness of this behaviour I have created a macro which helps to “remember” the false:

#define NX_PY_OBJECT_PTR(pyObject) nxPyObjectPtr_t(pyObject, false)

Here is an example how to use it:

nxPyObjectPtr_t obj = nxPyObjectPtr_t(PyLong_FromInt(1000), false));
nxPyObjectPtr_t obj = NX_PY_OBJECT_PTR(PyLong_FromInt(1000));

The 1st line is without macro and the 2nd line with macro.

That’s all about it, wasn’t so bad after all.

3. Further reading

As usual some useful links about todays topics:


Smart pointers to boost your code (Tutorial)

Python reference counting

Python embedding (Part 2)

Last time I covered the basics of embedding python into the engine. Today I will show you how to expose C-functions or more exactly static C++ class methods to the python interpreter. Furthermore I’ll give you some hints how to call python functions from C/C++ code.

1. Exposing C/C++ functions to python

First we need to add a new static member variable to the nx_PythonManager class declaration:

  static PyMethodDef m_methodTable[];

All functions which should be callable from python must be added to this table. Here is the definition with some functions stripped out:

PyMethodDef nx_PythonManager::m_methodTable[] = {
  { "printVersion", nx_PythonManager::printVersion, METH_NOARGS, "Prints the python interpreter version." },
  { "executeFile", nx_PythonManager::executeFile, METH_O, "Executes a python script file." },
  { "registerEventListener", nx_PythonManager::registerEventListener, METH_VARARGS, "Registers a script event listener." },
  { NULL, NULL, 0, NULL }

Each line defines exactly 1 function which should be python callable. The 4 arguments in each line are from left to right:

  1. Python function name.
  2. C/C++ function which should be exposed. In our case we use static class methods which sit inside the nx_PythonManager class.
  3. Number and type of function arguments. METH_NOARGS = no argument at all, METH_O = exactly one python object as argument, METH_VARARGS = variable number of arguments (0, 1, 2, …)
  4. Description of the function.

The last line “{ NULL, NULL, 0, NULL }” shows python the end of the table.

Every python callable function must follow the same scheme:

PyObject* functionName(PyObject *self, PyObject *args);

The self argument is only needed by python for object orientet stuff and is therefore uninteresting for us at the moment. To prevent compiler warnings I tend to just comment “self” out like this:

PyObject* functionName(PyObject */*self*/, PyObject *args);

Ok enough of the basics an actual example of such a function:

// declaration from the nx_PythonManager class
static PyObject* printVersion(PyObject *self, PyObject *args);

// implementation
PyObject* nx_PythonManager::printVersion(PyObject */*self*/, PyObject */*args*/)
  std::cerr << Py_GetVersion() << std::endl;

Easy enough, all this function does is to print a python version string to std::cerr. This function was defined in the method table with METH_NOARGS, so no arguments are passed. Py_RETURN_NONE is a handy python macro for returning Py_None while keeping the reference count of the object right. Py_None is the python equivalent to C’s void.

Now there is only one thing left before we can call our function from python: Making a buildin module from our method table and add it to the main python module. This is done in the nx_PythonManager::init method. I gave you basic insight to this method in the first part of this article. These are the lines we need to add:

// this code goes after the 'PyObject *sysModule = PyImport_AddModule("sys");' line
PyObject *nxModule = Py_InitModule3("nx", m_methodTable, "NX-Engine python module.");

if (PyModule_AddObject(mainModule, "nx", nxModule) != 0)
  return false;

The last 4 lines should be familiar. They are basically the same as last time for adding the sys-module. Only difference is that this time we add our custom build module ‘nx’ which we get from the Py_InitModule3 function. This function takes the module name “nx”, our method table and a document string describing the module.

Now it is possible to run our little code function from python like this: nx.printVersion().

2. Calling python functions from C/C++

I actually do a call to a python function in the NX-Engine exactly at 1 point at the moment: Sending events to script listeners. So this will be pretty quick, here is the code needed for calling the python function:

PyObject *onEvent = PyObject_GetAttrString(m_callback, "onEvent");

Here the onEvent function is extracted from a script event listener. I haven’t talked about listener or event types at all, so just ignore the exact meaning of this line for the moment. All you need to know is that you need a way to get a python function object to your code. For example this could happen with a C-function which is exposed to python and takes a function object.

if (!PyCallable_Check(onEvent))
  return false;

This is actually just a safety check if the python object is really callable.

PyObject *result = PyObject_CallFunction(onEvent, (char*)"O", event.getPythonEvent())
bool r = (result == Py_True ? true : false)

return r;

Ok, here we call the python function with PyObject_CallFunction. In this case it takes exactly one argument of type object which is indicated with (char*)”O”. The last argument is the object passed, here of a custom python object type (I’ll eventually write about how to create custom types in the future). The result of the function call is converted to a C-type and returned. Furthermore notice the calls to Py_DECREF. This is necessary to free unused resources otherwise we would leak memory.

3. Further reading

I intentionally left out a few things for the moment like parsing METH_VARARG arguments or building an argument list with Py_BuildValue. I eventually will catch up to this in the future, meanwhile some links to the excellent python documentation on these topics:

Parsing arguments and building values

Buidling values

Extracting parameters

Python embedding (Part 1)

This will be the first post about embedding Python into the NX-Engine. I will try to write about the basics today and dig deeper into the embedding process in future posts.

1. Header files and project settings

To use python in your program there is only 1 header file to include. The only catch is that this must be the first file included, before any other header. If you happen to include it after a standard-c header there is a chance your program won’t compile.

#include "Python.h"

Furthermore you need to set some project paths or the compiler won’t be able to find python headers and libraries. You have to add the path to the python headers to your projects compiler search path. I’m using Linux with Codeblocks as IDE, so I added the following to my projects build options under the “Search directories -> Compiler” tab:


I haven’t worked on windows with other IDEs yet, but it should be something similar (google is your friend). While it is possible to change the #include… statement to something like: #include “python2.7/Python.h” this is not recomended due to compatiblity between different python versions and operating systems.

Finally to make the linker happy we add the following to the linker project settings:


If you aren’t using the 2.7 version of python or want to switch to a more recent version, just change the version numbers in the project settings and you’re done, no need for editing any source file.

2. Python manager class

Because the NX-Engine is heavily modular and python is a pretty big component I decided to put it in it’s own manager class. Here is a very basic class declaration with quite a few things stripped out for the moment. We will get to this parts in a future post.

class nx_PythonManager : public nx_Singleton<nx_PythonManager>
    bool init(const char *initScriptFilename = NULL);
    bool restart(const char *initScriptFilename = NULL);
    bool executeCommand(const char *command);
    bool executeFile(const char *filename);
    void shutdown();

As you can see the class is a singleton like any manager in the engine. The constructor is empty expect for some log writing. The destructor just calls the private shutdown() method. The actual initialization is done in the init method. The reason for not doing this in the constructor is that python startup could fail and we can’t return a error flag from a constructor.

Here is the implementation of the init method:

bool nx_PythonManager::init(const char *initScriptFilename)

This initializes the python interpreter. We used this function instead of Py_Initialize() to keep the memory footprint as small as possible and prevent some ugly reference leaks. Py_InitializeEx(0) doesn’t register signal handlers which aren’t useful in embedded python anyway. At least I haven’t missed them yet.

    PyObject *mainModule = PyImport_AddModule("__main__");
    PyObject *sysModule = PyImport_AddModule("sys");

Here we get pointers to the global namespace (__main__ module) and sys module. The references are borrowed, that means we don’t need to call Py_DECREF if we don’t need them anymore.

    if (PyModule_AddObject(mainModule, "sys", sysModule) != 0)
        return false;

Now we add the sys module to the global python namespace. Because PyModule_AddObject steals a reference to our sys module pointer we need to Py_INCREF it first or the program will horribly die. What we did with the last 4 lines of code is the same as invoking the following python command: “import sys”. So we don’t need to do this in any of our scripts and can immediately do something like: “print sys.version”

    if (initScriptFilename)

    return true;

Finally we run our init script and return from the method.

Next up the private shutdown method which is a single liner for now:

void nx_PythonManager::shutdown()

All it does is to shutdown the python interpreter and clean up as much data as it can. From own testing this is far from perfect and some reference and therefore memory is definetly leaked (~64 reference on my system). You should keep this in mind and don’t restart the interpreter in a release build like crazy, even small leaks will add up^^

This leads us to our restart method:

bool nx_PythonManager::restart(const char *initScriptFilename)
    return true;

This is exactly what you think it is: stop everything, than crank it up again. All of this with the methods written some lines earlier. And yes this method leaks about 64 references everytime called on my system. I double checked everything and this is python internal and can’t be helped.

At this point we have python up and running, could restart and kill it again. To do something useful I will introduce you to two more methods for today:

bool nx_PythonManager::executeCommand(const char *command)
    if (!command) return false;
    return true;

Just runs a python command. You could do something like: pythonManager->executeCommand(“print ‘Hello World!'”).

bool nx_PythonManager::executeFile(const char *filename)
    FILE *fp = fopen(filename, "r");
    if (!fp) return false;
    PyRun_SimpleFile(fp, filename);
    fp = NULL;
    return true;

Similar to executeCommand with the difference that the python code comes from a file. The file is opened and passed to the fitting python function. You should be able to execute virtually any python script this way.

3. Further reading

At this point you should have a very basic embedded python interpreter. I’ll post more about how to build own modules and data types for the interpreter and howto link python with an event system in the future. Meanwhile you can get some more information about python embedding here:

To learn more about the python C-API this comes in handy:

Reworked events + python integration

I recently embedded a fully functional python interpreter into the NX-Engine and linked it into the reworked event system. The result is a surprisingly powerful system. The engine now supports sending events from code to script and vice versa. Furthermore it is possible to create completely new event types only in script. It is even possible for the engine to restrict script access to some event types like keydown or keyup. Python can still listen for these events, but not create them.

Another great benefit of python is that parts of the engine can be prototyped and tested quickly in script and later ported to code. This hopefully speeds up development, which makes the programmer (me) happy. For this reason I hooked up the python interpeter with a nice ingame console. That way the program don’t event need a restart to make some changes in script or reload the interpreter. Just type in some commands, restart the whole interpreter or load/reload some scripts.

I’m still tracking down some python issues, mainly memory/reference leaks. I’m pretty sure they are python internal but couldn’t confirm it 100% yet. I’ll probably write some lines about the python embedding process in the future, meanwhile a screenshot of the new ingame console in action:


Hello and welcome to my new blog,

I created it because the maintenance of my old page got a bit ugly (only have html there, so editing every file by hand…). I will write here about the development progress of my game engine or other programming/computer related things.

For a start I have created some pages with informations about my running and ceased programming projects. If you are interested in any of them feel free to leave a comment.

Furthermore I have a Youtube-Channel with some videos of my planet renderer demo: Youtube

That’s it for now, happy reading


« Previous Page