Python embedding (Part 1)

This will be the first post about embedding Python into the NX-Engine. I will try to write about the basics today and dig deeper into the embedding process in future posts.

1. Header files and project settings

To use python in your program there is only 1 header file to include. The only catch is that this must be the first file included, before any other header. If you happen to include it after a standard-c header there is a chance your program won’t compile.

#include "Python.h"

Furthermore you need to set some project paths or the compiler won’t be able to find python headers and libraries. You have to add the path to the python headers to your projects compiler search path. I’m using Linux with Codeblocks as IDE, so I added the following to my projects build options under the “Search directories -> Compiler” tab:

/usr/include/python2.7/

I haven’t worked on windows with other IDEs yet, but it should be something similar (google is your friend). While it is possible to change the #include… statement to something like: #include “python2.7/Python.h” this is not recomended due to compatiblity between different python versions and operating systems.

Finally to make the linker happy we add the following to the linker project settings:

python2.7

If you aren’t using the 2.7 version of python or want to switch to a more recent version, just change the version numbers in the project settings and you’re done, no need for editing any source file.

2. Python manager class

Because the NX-Engine is heavily modular and python is a pretty big component I decided to put it in it’s own manager class. Here is a very basic class declaration with quite a few things stripped out for the moment. We will get to this parts in a future post.

class nx_PythonManager : public nx_Singleton<nx_PythonManager>
{
public:
    nx_PythonManager();
    ~nx_PythonManager()
    bool init(const char *initScriptFilename = NULL);
    bool restart(const char *initScriptFilename = NULL);
    bool executeCommand(const char *command);
    bool executeFile(const char *filename);
private:
    void shutdown();
};

As you can see the class is a singleton like any manager in the engine. The constructor is empty expect for some log writing. The destructor just calls the private shutdown() method. The actual initialization is done in the init method. The reason for not doing this in the constructor is that python startup could fail and we can’t return a error flag from a constructor.

Here is the implementation of the init method:

bool nx_PythonManager::init(const char *initScriptFilename)
{
    Py_InitializeEx(0);

This initializes the python interpreter. We used this function instead of Py_Initialize() to keep the memory footprint as small as possible and prevent some ugly reference leaks. Py_InitializeEx(0) doesn’t register signal handlers which aren’t useful in embedded python anyway. At least I haven’t missed them yet.

    PyObject *mainModule = PyImport_AddModule("__main__");
    PyObject *sysModule = PyImport_AddModule("sys");

Here we get pointers to the global namespace (__main__ module) and sys module. The references are borrowed, that means we don’t need to call Py_DECREF if we don’t need them anymore.

    Py_INCREF(sysModule);
    if (PyModule_AddObject(mainModule, "sys", sysModule) != 0)
    {
        return false;
    }

Now we add the sys module to the global python namespace. Because PyModule_AddObject steals a reference to our sys module pointer we need to Py_INCREF it first or the program will horribly die. What we did with the last 4 lines of code is the same as invoking the following python command: “import sys”. So we don’t need to do this in any of our scripts and can immediately do something like: “print sys.version”

    if (initScriptFilename)
    {
        executeFile(initScriptFilename)
    }

    return true;
}

Finally we run our init script and return from the method.

Next up the private shutdown method which is a single liner for now:

void nx_PythonManager::shutdown()
{
    Py_Finalize();
}

All it does is to shutdown the python interpreter and clean up as much data as it can. From own testing this is far from perfect and some reference and therefore memory is definetly leaked (~64 reference on my system). You should keep this in mind and don’t restart the interpreter in a release build like crazy, even small leaks will add up^^

This leads us to our restart method:

bool nx_PythonManager::restart(const char *initScriptFilename)
{
    shutdown();
    init(initScriptFilename);
    return true;
}

This is exactly what you think it is: stop everything, than crank it up again. All of this with the methods written some lines earlier. And yes this method leaks about 64 references everytime called on my system. I double checked everything and this is python internal and can’t be helped.

At this point we have python up and running, could restart and kill it again. To do something useful I will introduce you to two more methods for today:

bool nx_PythonManager::executeCommand(const char *command)
{
    if (!command) return false;
    PyRun_SimpleString(command);
    return true;
}

Just runs a python command. You could do something like: pythonManager->executeCommand(“print ‘Hello World!'”).

bool nx_PythonManager::executeFile(const char *filename)
{
    FILE *fp = fopen(filename, "r");
    if (!fp) return false;
    PyRun_SimpleFile(fp, filename);
    fclose(fp);
    fp = NULL;
    return true;
}

Similar to executeCommand with the difference that the python code comes from a file. The file is opened and passed to the fitting python function. You should be able to execute virtually any python script this way.

3. Further reading

At this point you should have a very basic embedded python interpreter. I’ll post more about how to build own modules and data types for the interpreter and howto link python with an event system in the future. Meanwhile you can get some more information about python embedding here:

http://docs.python.org/extending/index.html

To learn more about the python C-API this comes in handy:

http://docs.python.org/c-api/index.html

Reworked events + python integration

I recently embedded a fully functional python interpreter into the NX-Engine and linked it into the reworked event system. The result is a surprisingly powerful system. The engine now supports sending events from code to script and vice versa. Furthermore it is possible to create completely new event types only in script. It is even possible for the engine to restrict script access to some event types like keydown or keyup. Python can still listen for these events, but not create them.

Another great benefit of python is that parts of the engine can be prototyped and tested quickly in script and later ported to code. This hopefully speeds up development, which makes the programmer (me) happy. For this reason I hooked up the python interpeter with a nice ingame console. That way the program don’t event need a restart to make some changes in script or reload the interpreter. Just type in some commands, restart the whole interpreter or load/reload some scripts.

I’m still tracking down some python issues, mainly memory/reference leaks. I’m pretty sure they are python internal but couldn’t confirm it 100% yet. I’ll probably write some lines about the python embedding process in the future, meanwhile a screenshot of the new ingame console in action: