Cython Journey Part 1: The Basics

I’ve recently begun writing up some algorithms is C++ for the speed benefits, as they require an insane amount of trials and overall simulation time, and there is a significant speedup in moving code from Matlab or Python over to C++.

BUT

It’s a large project, and I’d like to

  1. Be able to easily plot the results in Python, using matplotlib
  2. Be able to “plug and play” C++ parts with Python parts for speed comparison and error checking

So I’ve been looking into Cython, which, after some digging, seems a very viable candidate for building up this interface. I’ve been following the examples and whatnot they have on the Cython page (http://docs.cython.org/src/userguide/wrapping_CPlusPlus.html#a-simple-tutorial), but it tends not to be abundantly clear what the finished version of the files are supposed to look like. So I’ve been writing up my own as I go along based on what their examples and trial and error.

I’ve called the project test. I’ve created two files to hold the C++ code, cpp_test.h and cpp_test.cpp, which look like this:
cpp_test.h

#ifndef TEST_H
#define TEST_H

class Test { 
public: 
    int test1; 
    Test();
    Test(int test1); 
    ~Test(); 
    int returnFour(); 
    int returnFive();
    Test operator+(const Test& other); 
    Test operator-(const Test& other);
};
#endif

and cpp_test.cpp

#include "cpp_test.h"

Test::Test() { 
    test1 = 0;
}

Test::Test(int test1) { 
    this->test1 = test1; 
}

Test::~Test() { }

int Test::returnFour() { return 4; }

int Test::returnFive() { return returnFour() + 1; }

Test Test::operator+(const Test& other) { 
    return Test(test1 += other.test1);
}

Test Test::operator-(const Test& other) { 
    return Test(test1 -= other.test1);
}

To start, we need a wrapper .pyx file that will specify how we actually interface between Python and the above C++ code. So, we make a new file, and call it “test.pyx”.

  • Important! Do not name the .pyc file the same as your .cpp file! When the cythonize function is called later, a new .cpp file with the same name as your .pyx file is generated. If you name both your original .cpp and your .pyx file the same, your .cpp file will be overwritten! Which is bad.

The first thing we need to do in our test.pyx file (which is written in Cython) is redefine all of the variables and functions that we want to use from our C++ code, including the constructors. In this example we’re going to expose the operator functions and returnFive(), which calls returnFour(), but not returnFour() itself. All of our C++ interaction is defined in this .pyx file.
Here is the first part of test.pyx:

cdef extern from "cpp_test.h": 
    cdef cppclass Test:
        Test()
        Test(int test1)
        int test1
        int returnFive()
        Test add "operator+"(Test other) 
        Test sub "operator-"(Test other)

Here we have a redeclaration of everything that we want to use from our C++ code, which is everything except returnFour(). All of the class variables and functions are defined inside cdef cppclass Test. One of the first things that jumps out from looking at this code is that we can rename C++ commands with Python names here, by declaring a name, and then the mapped C++ code inside quotation marks. The above code gives Cython a handle into the C++ code, but we are not done! We need to make this functionality available to regular ol’ Python.

So! Still inside test.pyx, we need to write some more code. I’ll show the code and then walk through it.

cdef class pyTest: 
    cdef Test* thisptr # hold a C++ instance
    def __cinit__(self, int test1):
        self.thisptr = new Test(test1)
    def __dealloc__(self):
        del self.thisptr

    def __add__(pyTest left, pyTest other):
        cdef Test t = left.thisptr.add(other.thisptr[0])
        cdef pyTest tt = pyTest(t.test1)
        return tt
    def __sub__(pyTest left, pyTest other):
        cdef Test t = left.thisptr.sub(other.thisptr[0])
        cdef pyTest tt = pyTest(t.test1)
        return tt

    def __repr__(self): 
        return "pyTest[%s]" % (self.thisptr.test1)

    def returnFive(self):
        return self.thisptr.returnFive()

    def printMe(self):
        return "hello world"

Let’s look at what’s going on. We define a new class, and it’s called Test. The first thing we do is then define a pointer to the class defined above in test.pyc, c_Test. This will be how we access the variables and functions of our C++ code. The first method we define is __cinit__, which is the very first thing called when we make an instance of Test, called even before __init__. It’s important to do this here, because of some issues that come up with __init__ itself not being called soon enough for instantiating our pointer, thisptr. To create pointers to the class we simply call new Test(int test1). The next method we define is __dealloc__, which just lets us delete our pointer, which calls the deconstructor of our C++ class and frees memory appropriately, through del self.thisptr.

After these, we create a method to call returnFive(), which we appropriately also title returnFive(). All this does is call upon thisptr to access it’s own returnFive() function, which links to the C++ returnFive() function. Note that the C++ code calls upon returnFour(), which we did not define in c_Test. This just shows that not everything has to be redefined in our .pyx file, only what we want exposed for access from Python.

The next two methods are the Cython equivalent of our operator+ and operator- C++ functions. These functions are a little trickier, because the c_Test functions return pointers to new c_Test instances, because the operator functions in our C++ code in turn returns new instances of the C++ Test class. So what we have to do is create a handle to the c_Test instance returned by the self.thisptr.add and self.thisptr.sub calls, and then create a new instance of the Test class from our .pyx code, and return that.

The __repr__ function is simply an aesthetic feature, such that when the user calls, from Python, for the Test class to print we get a nice printout.

And finally, just to show that this is a Python class we’re defining, we make another method that is strictly superfluous, and unrelated to the C++ code, just to emphasize that additional functionality specific to our Python implementation can be added here.

Now we have 3 files, cpp_test.h, cpp_test.cpp, and test.pyx. We need one more, setup.py. This is file that will compile the C++ and .pyx files together and generate a usable python module. The setup file can be written several different ways, here is template I’ve been using:
setup.py

from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext

setup(
  name = 'Demos',
  ext_modules=[ 
    Extension("test", 
              sources=["test.pyx", "cpp_test.cpp"], # Note, you can link against a c++ library instead of including the source
              language="c++"),
    ],
  cmdclass = {'build_ext': build_ext},

)

The important thing here is to set the sources parameter to include the relevant .pyx and .cpp files that we just wrote.
And that’s it! Now open up IPython and punch in:

run setup.py build_ext -i

and now we can call up our test module! i.e.

from test Import pyTest
k = pyTest(10)
k.returnFive()
k + k
k - k 
k.printMe()

And that’s it! We’ve successfully wrapped a basic C++ class and made it accessible from Python! This is a good first step, we’ll continue the Cython journey in later posts.

One last thing though, if you use IPython, it’s should be noted that if you change any files and recompile with the

run setup.py build_ext -i

you need to exit and then re-enter IPython before the updated code will be recognized. If you don’t know this you can get very frustrated trying to figure out why nothing is changing!

Tagged ,

3 thoughts on “Cython Journey Part 1: The Basics

  1. fcs pk says:

    Hi there,

    Wonderful example! I had the same problem with the tutorial on the Cython page not being very precise in as to what had to be put in the various files. So your blog posts really helped me out. Thanks a bunch!

  2. […] The next part was writing the Cython pyRobot.pyx file (I describe the steps involved in this in more detail in this post): […]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: