Python profiling trials

Python comes with some default profiling capabilities. However, as also noted here, this profiling can slow down your program to a crawl making the gathering of useful statistics very tedious. Although it is very easy to get the proposed alternative "plop" running the output it generates is very limited and I did not find a description to get a more traditional output from this profiler. Although statprof does generate traditional output without slowing down running your program very much, it does not really seem to deal correctly with wrapper functions which severely limits its applicability to my typical programs. Thus, unfortunately, I seem to be stuck with using CProfile for a while.

A nice introduction to CProfile is given on slippen's blog.

To turn on profiling of Cython code add the line

#cython: profile=True

to the top of your .pyx file (including the hash).

using SWIG to connect python and a c++ dll

Although I am a great fan of Cython when you already have a class defined in C++ code you do not want to rewrite it for Cython. Already a long time ago I once used SWIG to create a link between C++ classes and python. This is fairly easy as long as you have clear interface functions which you can use to pass data to and from your class. Nevertheless, it took me some effort to get things running for my new class.

Following the SWIG tutorial I started with something like this myExample.i

%module myExample
 %{
 /* Includes the header in the wrapper code */
 #include "myModule.h"
 #include "myModule2.h"
 %}
 
 /* Parse the header file to generate wrappers */
 #include "myModule.h"
 #include "myModule2.h"

After compiling this I got a module that I could import in Python. However, the module did not contain the classes I had defined in my C++ code. Because there are not so many clear examples of SWIG and C++ I could not really find an explanation for this. Only after recreating the exact example of the tutorial (using copy/paste from the web) I found that I made a simple typing error: the #include in the second part of the myExample.i should be %include!

%module myExample
 %{
 /* Includes the header in the wrapper code */
 #include "myModule.h"
 #include "myModule2.h"
 %}
 
 /* Parse the header file to generate wrappers */
 %include "myModule.h"
 %include "myModule2.h"

Now everything works nicely. Compiling is not very difficult. I have setup a Makefile for this:

CPP=g++

CFLAGS=  -Wunused  -O3

PYTH_INCL = /c/Python27/include
PYTH_LIB = /c/Python27/Libs

PYLIB = -lpython27

OBJS= myModule.o
modOBJS= myModule2.o
SRC_DIR = ..
all: myExample_mod

myExample_mod: $(modOBJS) myExample.i
 swig -I.. -python -c++ -shadow myExample.i
 $(CPP) -I$(PYTH_INCL) $(INCLUDE) -c myExample_wrap.cxx
 $(CPP) -shared -L$(PYTH_LIB) myExample_wrap.o $(OBJS) $(PYLIB) $(LIBS) -o _myExample.pyd

%.o: $(SRC_DIR)/%.cpp $(SRC_DIR)/%.h
 $(CPP) -c $(CFLAGS) $(INCLUDE)  $< -o $@

With this Makefile you can use import myExample in your python code to import the module. Thus far I have only used simple interface methods in my classed that work with integers and floats as input and output variables. This makes communication between the module and python very simple (no special effort seems to be required). I assume that if you want to pass arrays, strings, etc. things will become more difficult.

cython -mno-cygwin problems

The command I use to compile a cython extension on windows:

python setup.py build_ext --inplace --compiler=mingw32

(see also these two posts) has problems with newer mingw/cygwin installations. Already for some to the compiler option -mno-cygwin is no longer used and the mingw compiler should be called directly. When you run the setup.py script with the compiler flag as shown, however, python automatically adds the -mno-cygwin flag. This then stops the compiler. The best solution for this I have found so far is to change the

c:\Python27\Lib\distutils\cygwinccompiler.py

file. Simply remove the -mno-cygwin option everywhere in the definition of the Mingw32CCompiler class (just search for no-cygwin in this file and you will find it).

Looks like I also could have googled this instead of finding out again how to use grep to search in subdirectories which is not as simple as adding the -r option ( find ./* -type f -exec grep -l "no-cygwin" {} \;).