Cython Journey Part 2: Including Eigen

As I mentioned in my last Cython Journey post, the reason I’m using Cython is to take my speedy C++ code that does a bunch of matrix operations using the Eigen library, and to make them accessible from Python. Now, I have no desire to make an interface for the Eigen library, all I need is to be able to access the MatrixXf objects from the Eigen library, and convert them into numpy.ndarray objects for manipulation in Python. The logical first step towards this end would then seem to be to successfully compile C++ code that links to the Eigen library.

HOWEVER. After messing around with a bunch of linking problems and the like, I came across code by charanpald that provides an interface for Eigen sparse matrices and sparse matrix operations to Python (https://github.com/charanpald/SpPy). In this code, he links to the Eigen library through a buffer class that extends the Eigen::SparseMatrix class and adds some more functionality. There is very possibly a better way to do this, but I also needed to define some extra functions operating on the Eigen matrices, so defining an extension to the MatrixXf class worked out well. For this example, though, I’ll just use a barebones class extension to keep the size manageable. Here is my class:
cpp_matrixxfpy.h

#ifndef MATRIXXFPY_H
#define MATRIXXFPY_H
#include <Eigen/Dense>
using namespace Eigen;

class MatrixXfPy : public MatrixXf { 
    public: 
        MatrixXfPy() : MatrixXf() { }
        MatrixXfPy(int rows,int cols) : MatrixXf(rows,cols) { }
        MatrixXfPy(const MatrixXf other) : MatrixXf(other) { } 
};
#endif

Like, I said, barebones.
With this now we can go to our .pyx file and create a Cython handle into the MatrixXf object.
matrixxfpy.pyx

cdef extern from "cpp_matrixxfpy.h":
    cdef cppclass MatrixXfPy:
        MatrixXfPy()
        MatrixXfPy(int d1, int d2)
        MatrixXfPy(MatrixXfPy other)
        int rows()
        int cols()
        float coeff(int, int)

Same as for the cpp_test code that we wanted to interface with Python; first thing we do in the .pyx file is redefine the class and variables / functions that we want access to in Python, along with the constructors. In this simple example we’re going to only access three functions from MatrixXf, rows(), cols(), and coeff(int, int). These functions return the number of rows, the number of columns, and the matrix coefficient value at a given index.

So now that we have a Cython handle on the Eigen code, let’s make this functionality available to Python. HOWEVER. Let’s keep in mind the goal, we’re not here to provide a Python wrapper for MatrixXf objects. What we really want is just to have Eigen and our C++ code over there, somewhere else, doing all of the calculations etc and then for us to be over in Python and just be able to get a numpy.ndarray for Python to play with and plot. Let us set about that now.

Let’s go back to the Test class we defined in Cython Journey Part 1. First thing is first, we need to edit the C++ code to create and return a MatrixXf object. So let’s just throw in a simple function:
cpp_test.h

#ifndef TEST_H
#define TEST_H

#include "cpp_matrixxfpy.h"
using namespace Eigen; 

class Test { 
public: 
    int test1; 
    Test();
    Test(int test1); 
    ~Test(); 
    int returnFour(); 
    int returnFive();
    Test operator+(const Test& other); 
    Test operator-(const Test& other);
    MatrixXfPy getMatrixXf(int d1, int d2);    
};
#endif

and define it in cpp_test.cpp as

#include "cpp_test.h"

Test::Test() { 
    test1 = 0;
}

Test::Test(int test1) { 
    this->test1 = test1; 
}

Test::~Test() { }

int Test::returnFour() { return 4; }

int Test::returnFive() { return returnFour() + 1; }

Test Test::operator+(const Test& other) { 
    return Test(test1 += other.test1);
}

Test Test::operator-(const Test& other) { 
    return Test(test1 -= other.test1);
}

MatrixXfPy Test::getMatrixXf(int d1, int d2) { 
    MatrixXfPy matrix = (MatrixXf)MatrixXfPy::Ones(d1,d2);
    matrix(0,0) = -10.0101003; // some manipulation, to show it carries over
    return matrix;
}

I put up the whole class again so it’s all in one place.

Alright, that’s done. Now we can look at our .pyx file for Test. We’re going to need to make the MatrixXfPy class defined just above available to our Test class. To do this is easy, all we need to throw in is:
test.pyx

 
include "cpp_matrixxfpy.h"

at the top! And while we’re at it, we’re going to be adding some more functionality to test that requires numpy, so also throw in:

 
import numpy

This additional functionality is going to access the getMatrix(int,int), create an numpy.ndarray object, and fill it out with the values from the matrix. Let’s get about doing that. We need to expose the new C++ function to Cython, so add this line to the cdef cppclass Test declarations:

        MatrixXfPy getMatrixXf(int d1, int d2)

And now we can add the code that will be accessible from Python. I’ll post the code and then describe it below (although it’s pretty straightforward):

    def getNDArray(self, int d1, int d2): 
        cdef MatrixXfPy me = self.thisptr.returnMatrixXf(d1,d2) # get MatrixXfPy object
        
        result = numpy.zeros((me.rows(),me.cols())) # create nd array 
        # Fill out the nd array with MatrixXf elements 
        for row in range(me.rows()): 
            for col in range(me.cols()): 
                result[row, col] = me.coeff(row, col)   
        return result 

I just flat out stripped the form for this from the SpPy project and simplified it, so it’s not going to be optimized for speed in any way, but it gets the job done and it’s very easy to read. First we just set up a numpy.zeros matrix to index into, and then we go through each element in the array and fill it in with the appropriate value by calling to the coeff(int row, int col) function we defined above. Easy!

Now we can use the same setup file from the last Cython post to build our shared library:

from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext

setup(
  name = 'Test',
  ext_modules=[ 
    Extension("test", 
              sources=["test.pyx", "cpp_test.cpp"], # Note, you can link against a c++ library instead of including the source
              language="c++"),
    ],
  cmdclass = {'build_ext': build_ext},

)

Again reproduced here for hassle-freeness. Now we can compile and test it! At the IPython terminal:

run setup.py build_ext -i
import test
t = test.pyTest(10)
m = t.getNDArray(4,4)

And that’s it! We can see that that last command returns to us a (4x4) matrix with the first element set to -10.0101003, which is exactly what we did over in our C++ files. Now we’re all set to go ahead and generate matrices in C++, and then ship ’em out over to Python for plotting. Excellent.

Tagged , ,

8 thoughts on “Cython Journey Part 2: Including Eigen

  1. Robert Upton says:

    Hi Travis,

    Thankyou for going to the trouble to post this ‘how to’ on getting python to use the c++ classes and methods that are available in Eigen, it is very useful and informative.

    I have 1 question/problem: I wrote the code down as per your instructions, but when ask cython to do the compilation in the ipython environment (run setup.py build_ext -i) the compiler does not like the ‘using namespace Eigen;’ statement in the cpp_matrixxfpy.h file, which I have included below, along with the compiler error. I have ensured all the c++ code and library paths to Eigen are correct by running a simple program in a main() function within c++. I don’t know why the cython compiler is having a problem with the statement, especially that the code is correct. Please can you make a suggestion as to what may be wrong?

    Many Thanks,

    Robert

    cpp_matrixxfpy.h
    ———————-
    #ifndef MATRIXXFPY_H
    #define MATRIXXFPY_H
    #include

    using namespace Eigen;

    class MatrixXfPy:public MatrixXf{
    public:
    MatrixXfPy():MatrixXf(){}
    MatrixXfPy(int rows, int cols):MatrixXf(rows,cols){}
    MatrixXfPy(const MatrixXf other):MatrixXf(other){}
    };
    #endif

    and the error message issued by python.
    —————————————————-

    run setup.py build_ext -i
    running build_ext
    cythoning test.pyx to test.cpp

    Error compiling Cython file:
    ————————————————————

    #ifndef MATRIXXFPY_H
    #define MATRIXXFPY_H
    #include

    using namespace Eigen;
    ^
    ————————————————————

    cpp_matrixxfpy.h:5:6: Syntax error in simple statement list
    building ‘test’ extension

    • travisdewolf says:

      Hmm, just from a quick look it seems like the #include line is missing the ” following, although that might be just be something that was lost in the copy/paste, can you confirm that the line reads ‘#include ‘?

  2. Robert Upton says:

    Hi Travis,

    The include line contains #include . I will paste the code in properly this time:

    cpp_matrixxfpy.h
    _____________

    #ifndef MATRIXXFPY_H
    #define MATRIXXFPY_H
    #include

    using namespace Eigen;

    class MatrixXfPy:public MatrixXf{
    public:
    MatrixXfPy():MatrixXf(){}
    MatrixXfPy(int rows, int cols):MatrixXf(rows,cols){}
    MatrixXfPy(const MatrixXf other):MatrixXf(other){}
    };
    #endif

    In order to check that the g++ compiler (I am using a mac) included with Darwin is doing the correct things, I included a main() function in cpp_test.cpp, as follows. As I said in my last message/post the code works in c++, but when I attempt to do it with cython (both in ipython and python) there is an issue with the ‘using namespace Eigen’ statement. I included the error message at the bottom of the post.

    cpp_test.cpp
    __________
    #include
    #include “cpp_test.h”

    using namespace std;

    Test::Test()
    {
    test1 = 0;
    }

    Test::Test(int test1)
    {
    this -> test1 = test1;
    }

    Test::~Test(){}

    int Test::returnFour(){return 4;}

    int Test::returnFive(){return returnFour() + 1;}

    MatrixXfPy Test::getMatrixXf(int d1, int d2)
    {
    MatrixXfPy matrix = (MatrixXf)MatrixXfPy::Ones(d1,d2);
    matrix(0,0) = -1.0;
    return matrix;
    }

    int main()
    {
    Test obj;
    MatrixXf mat = obj.getMatrixXf(4,4);

    cout << mat << endl;

    return 0;
    }

    cython output
    ___________

    cythoning test.pyx to test.cpp

    Error compiling Cython file:
    ————————————————————

    #ifndef MATRIXXFPY_H
    #define MATRIXXFPY_H
    #include

    using namespace Eigen;
    ^
    ————————————————————

    cpp_matrixxfpy.h:5:6: Syntax error in simple statement list

  3. Robert Upton says:

    Hi Travis,

    Sorry for the confusion. I now know that you are referring to the test.pyx code and not the cpp_matrixxfpy.h code. I included the #include “matrixxfpy.h” at the top of test.pyx, and cython is complaining that the class we defined in cpp_matrixxfpy.h, MatrixXfPy is not a type and throws the exception:

    Error compiling Cython file:
    ————————————————————

    cdef cppclass Test:
    Test()
    Test(int test1)
    int test1
    int returnFive()
    MatrixXfPy getMatrixXf(int d1, int d2)
    ^
    ————————————————————

    test.pyx:10:8: ‘MatrixXfPy’ is not a type identifier

    As before, I checked the code within c++, and it works. H’mm … a mystery to investigate. Thanks for your help. I will welcome any other suggestions.

    Regards,

    Robert

    • Robert Upton says:

      Hi Travis,

      Now that I have unfortunately populated your page with my stream of consciousness, I have found the problem. I needed to include the #include at the top of the pyx code, and include the cython code from matrixxfpy.pyx. I all works beautifully. Thankyou and great job.

      Regards,

      Robert

  4. Moritz Maus says:

    Hi Travis,

    great tutorial, thank you very much!

    However, as a beginner, I had several problems getting your code to work. The main issues are:

    * following the idea of Robert, I put the content of matrixxfpy.pyx into test.pyx. (Is matrixxfpx.pyx required at all?)
    * there must not be an #include “cpp_matrixxfpy.h” statement in the test.pyx file, but in “cpp_test.h”
    * In my setup file, I had to include the “include_dirs = [‘/usr/include’, ‘/usr/include/eigen3’]” statement as named argument to the Extension(…)
    * In your code, I did not realize at the first reading that the getNDarray function is actually a method of the pyTest class! Further, you at some point you wrote returnMatrix instead of getMatrix

    hope that helps further readers!

    Thanks!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: