tag:blogger.com,1999:blog-29706615708385605482024-03-05T09:55:06.105+01:00Programming logA blog that serves as a log of mainly programming related issues that might also be useful for others.Korbininhttp://www.blogger.com/profile/00742811285841484666noreply@blogger.comBlogger45125tag:blogger.com,1999:blog-2970661570838560548.post-20474740142848843832015-09-23T20:56:00.000+02:002015-09-23T20:56:21.039+02:00PyQt5 on WinPython with mingw<div dir="ltr" style="text-align: left;" trbidi="on">Getting <a href="https://riverbankcomputing.com/software/pyqt/download5">PyQt5</a> running on <a href="http://winpython.github.io/">WinPython</a> required a lot of tweaking. WinPython (32 bit) 2.7.10.2 comes with mingw. This is installed in E:\WinPython-32bit-2.7.10.2\python-2.7.10\share\mingwpy. Following the instructions on the PyQt5 download page I first downloaded SIP from <a href="https://www.riverbankcomputing.com/software/sip/download">this page</a>. I downloaded the zip file for windows. Following <a href="http://pyqt.sourceforge.net/Docs/sip4/installation.html#configuring">these instructions</a> I configured sip for mingw by running<br />
<br />
python configure.py --platform win32-g++<br />
<br />
The next step then is to run make. However if you start a WinPython Command Prompt you do not have make. I fixed this by copying an old msys installation, from an older (not connected to WinPython) mingw installation, to the mingw folder in the WinPython folder (as noted above). Then by changing the fstab file in msys/1.0/etc to this e:/WinPython-32bit-2.7.10.2/python-2.7.10/share/mingwpy/ /mingw I have a msys system setup for working with the mingw from the WinPython distribution. From a msys shell I can then run make and make install for sip.<br />
<br />
Next step is to do the same for PyQt5 download. However, there the configure.py script does not recognize the --platform switch. Use the --spec=win32-g++ switch is also not good enough. The problem is that it somehow still puts "windows" commands in the Makefiles it generates for rename, deleting files folders etc. I could only fix this by changing some code in the configure.py script. In the function run_qmake I added the lines <br />
<br />
args.append('-spec') <br />
args.append('win32-g++')<br />
<br />
(do this before the "if recursive:" statement). While you are busy editing the configure.py file you can also follow the advice in <a href="https://forum.qt.io/topic/56095/error-this-version-of-pyqt5-and-the-commercial-version-of-qt-have-incompatible-licenses/12">this forum post</a> to handle any issues with mismatching licenses. Somehow configure.py was detecting a different Qt version. This version was actually located in the site-packages/PyQt4 folder (in WinPython-32bit-2.7.10.2/python-2.7.10). By temporarily renaming this PyQt4 folder you can fix this issue.<br />
<br />
After this. If you run<br />
<br />
python configure.py --spec=win32-g++<br />
<br />
from a msys shell should create the right Makefile (note that this means you have to put WinPython in your system path (environment variables in control panel from advanced system settings in the control panel of windows) because otherwise you have no python in your msys shell. Then you can run make and make install from the msys shell.<br />
<br />
Then you are almost done. If at this stage you run a PyQt5 python script it will probably complain that it 'Failed to load platform plugin "windows"'. This was a tricky one to solve. For c++ qt applications it is usually enough to copy the plugins folder to the location of your executable (+ any other needed qt dll's), but what is the location of your executable in this case? I only managed to get it running by adding a <a href="http://stackoverflow.com/questions/20495620/qt-5-1-1-application-failed-to-start-because-platform-plugin-windows-is-missi">new environment variable</a>:<br />
<br />
QT_QPA_PLATFORM_PLUGIN_PATH='E:\Qt\5.5\mingw492_32\plugins'<br />
</div>Korbininhttp://www.blogger.com/profile/00742811285841484666noreply@blogger.com3tag:blogger.com,1999:blog-2970661570838560548.post-53104887540165448852015-02-15T13:02:00.000+01:002015-02-15T13:02:34.531+01:00UI not updating in Qt Creator<div dir="ltr" style="text-align: left;" trbidi="on">After not having opened a project for while in QtCreator it often happens that changes made to the user interface in QtCreator are not coming through in the application after recompiling. This is then caused because I am now using a new version of QtCreator or I used to work on the project under Linux instead of Windows (or something like that). The problem is that QtCreator keeps changing where it puts the output of compiling the .ui file. Sometimes it is in the main project folder, sometimes where the .ui file is located and now it was changed into the folder with the impossible name yourProject-build-desktop-Qt_4_8_1_for_Desktop_-_MinGW__Qt_SDK__Release/Release (where it also puts the object files). And because previously it was stored in the project folder it will first find that old version. I guess removing the old version could work but I liked the solution I found <a href="http://qt-project.org/forums/viewthread/292">here</a> better: in the project file you can specify where the ui compilation output is stored:<br />
<br />
UI_DIR = c:\YourLocation<br />
</div>Korbininhttp://www.blogger.com/profile/00742811285841484666noreply@blogger.com1tag:blogger.com,1999:blog-2970661570838560548.post-40043526782270190802015-01-26T19:19:00.002+01:002015-01-26T19:19:23.036+01:00cx_freeze issues with scipy<div dir="ltr" style="text-align: left;" trbidi="on">When using <a href="http://cx-freeze.readthedocs.org/en/latest/index.html">Cx_freeze</a> with a script that uses an interpolation function from scipy I was getting import errors. More specifically the error was "ImportError: No module named _ufuncs_cxx"<br />
<br />
This can be solved by explicitly adding two scipy modules:<br />
cxfreeze --include-modules=scipy.special._ufuncs_cxx,scipy.sparse.csgraph._validation your_script.py --target-dir your_target_dir<br />
<br />
This solution is based on information from <a href="https://bitbucket.org/anthony_tuininga/cx_freeze/issue/43/import-errors-when-using-cx_freeze-with">this website</a>.<br />
</div>Korbininhttp://www.blogger.com/profile/00742811285841484666noreply@blogger.com1tag:blogger.com,1999:blog-2970661570838560548.post-1155832019637572052014-12-26T10:50:00.003+01:002014-12-26T10:51:43.742+01:00Minimize button in a QDialog in PyQt<div dir="ltr" style="text-align: left;" trbidi="on">Often it is easy to write a small application in PyQt as a QDialog (instead of using a QMainWindow). If it is intended as an application that has to be open for a longer time (e.g. while the user is using other programs) it is nice if the program can minimize (e.g. when the user clicks the show desktop button in windows (or the nice windows-key+d key shortcut). However, by default the QDialog window only has a close button. Luckily adding minimize and maximize buttons is easy:<br />
<br />
<pre class="brush: py">from PyQt4.QtCore import *
from PyQt4.QtGui import *
class MainForm(QDialog):
def __init__(self, fn=None,parent=None):
super(MainForm, self).__init__(parent,\
flags=Qt.WindowMinimizeButtonHint|Qt.WindowMaximizeButtonHint)
</pre><br />
Other flags can be found <a href="http://pyqt.sourceforge.net/Docs/PyQt4/qt.html#WindowType-enum">here</a><br />
<br />
</div>Korbininhttp://www.blogger.com/profile/00742811285841484666noreply@blogger.com0tag:blogger.com,1999:blog-2970661570838560548.post-17714158479359489212014-10-07T20:18:00.000+02:002014-10-07T20:19:18.072+02:00SQLITE DATETIME troubles<div dir="ltr" style="text-align: left;" trbidi="on">
Recently, I have started to play with <a href="http://www.sqlite.org/">SQLITE</a>. For the database creation and queries I actually use <a href="http://pyqt.sourceforge.net/Docs/PyQt4/qtsql.html">PyQt</a> but that is not really the issue. I was unaware of the complications with SQL databases and DATETIME formats. I made the wrong assumption that if I provided input that did not give any errors and looked good in the nice <a href="http://sqlitebrowser.org/">SQLite Database Browser</a> everything would be okay. What happened was that I filled a datetime column with strings like '09-11-2014 12:43:00'. This actually is one of the few <b>not</b> <a href="http://www.sqlite.org/cvstrac/wiki?p=DateAndTimeFunctions">supported datetime formats</a>. When running a query where the datetime has to be before or after another (similarly formatted) datetime this actually seemed to produce sensible results. Only when I tried to do more fancy datetime stuff like datetime('09-11-2014 12:43:00','-1 month') did things suddenly stop working. I think sqlite is treating the wrong input for a datetime column as a string and is actually doing a string comparison when comparing two wrongly formatted datetime values (which can sometimes give sensible output). The solution is of course simple: only use one of the supported datetime formats when inserting data into a table.</div>
Korbininhttp://www.blogger.com/profile/00742811285841484666noreply@blogger.com0tag:blogger.com,1999:blog-2970661570838560548.post-15399109835227815022014-10-06T20:14:00.001+02:002015-03-15T18:15:51.218+01:00Cython modules not automatically rebuild through setup.py on changes of pxd files<div dir="ltr" style="text-align: left;" trbidi="on">After some frustrating hours I discovered that if you use a <a href="http://korbinin.blogspot.nl/2011/07/using-cython.html">distutils setup.py script to create a cython module</a> the module is not updated if you only change a <a href="http://docs.cython.org/src/tutorial/pxd_files.html">pxd header file</a>. Especially in a bigger project with many cython modules this can be easily forgotten, so in case of unexpected behaviour just try removing all pyd files so everything will be build fresh.<br />
<br />
It is perhaps even saver to touch all .pyx files because if you remove the pyd file the .c file is not necessarily recreated!<br />
</div>Korbininhttp://www.blogger.com/profile/00742811285841484666noreply@blogger.com0tag:blogger.com,1999:blog-2970661570838560548.post-37091438545598677142014-10-06T20:07:00.000+02:002014-10-06T20:07:08.855+02:00interp1d x-axis order must be in ascending order<div dir="ltr" style="text-align: left;" trbidi="on">Today I discovered that the nice scipy 1d interpolation function <a href="http://docs.scipy.org/doc/scipy-0.14.0/reference/generated/scipy.interpolate.interp1d.html">interp1d</a> requires that the values in x-array passed as the first argument are in ascending order. Perhaps this makes sense but it took me some time to figure out that this was the reason I was getting out-of-bounds errors (my x-array was in descending order).<br />
</div>Korbininhttp://www.blogger.com/profile/00742811285841484666noreply@blogger.com0tag:blogger.com,1999:blog-2970661570838560548.post-44406014577539069102014-09-13T17:18:00.000+02:002014-09-13T17:18:01.025+02:00Simple txt2csv python script<div dir="ltr" style="text-align: left;" trbidi="on">Often I have a text files with columns that are separated by spaces or tabs or a mixture of them. These files can be read by a typical spreadsheet program but this usually requires some extra mouse clicks to tell the spreadsheet program how to interpret the text file. Csv files (comma separated files) are often much better recognized by a spreadsheet program. Therefore I have written a simple python function/script to change a text file to csv file where spaces and tabs are replaced by a separator (default set to semicolon). Multiple spaces are combined into a single separator (which is what I typically want). Leading and trailing spaces on a row are removed. It is amazing how little python code is needed for this (thanks to python's nice string handling). The magic python line csvline=s.join(line.split()) in the script below is based on a very useful <a href="http://stackoverflow.com/questions/8270092/python-remove-all-whitespace-in-a-string">item on Stack Overflow</a>.<br />
<br />
<br />
<pre class="brush: py">import sys
def txt2csv(fnIn,fnOut,separator=';'):
fin=open(fnIn,'r')
if(not fin):
print "Error opening %s"%(fnIn)
sys.exit(-1)
fout=open(fnOut,'w')
for line in fin.readlines():
s=separator
csvline=s.join(line.split())
fout.write(csvline+'\n')
fin.close()
fout.close()
if __name__== "__main__":
try:
fnIn=sys.argv[1]
fnOut=sys.argv[2]
except:
print "Usage: python %s fnIn fnOut"%(sys.argv[0])
txt2csv(fnIn,fnOut)
</pre><br />
</div>Korbininhttp://www.blogger.com/profile/00742811285841484666noreply@blogger.com0tag:blogger.com,1999:blog-2970661570838560548.post-27530076898237161842014-04-07T15:11:00.001+02:002014-04-28T19:39:23.154+02:00guiqwt string based function plotter<div dir="ltr" style="text-align: left;" trbidi="on">For almost all plotting from python I use <a href="http://matplotlib.org/">matplotlib</a>, but recently I noticed an interesting alternative called <a href="https://code.google.com/p/guiqwt/">guiqwt</a>. Guiqwt seems to be faster and more tuned towards using plots in a UI environment. In the youtube movie <a href="https://www.youtube.com/watch?v=_P9HqHVPeik">Introduction to Wolfram Language</a> a nice demo is shown where a gui is assigned automatically to a function with sliders for the function parameters so the user can easily explore the function behavior. If you often work with empirical functions to reproduce certain experimentally observed trends it can be very nice to have such a quick visualization of the influence of the chosen function parameters. This gave me the idea to try to make a generic function plotter with python/guiqwt where the user can supply the function by typing a string from which the parameters are then automatically recognized and assigned a slider so the user can explore the function behavior. This turned out to be a little more involved than anticipated but the result is acceptable:<br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg831f9rCL75ZEkuUoFCsqeZdeen8dLh0l9VgVO0yHtS0I_uFkaWS6RIht0AlvHyBnjv5-LyUT8EJwrfyO8BKpmhr-p_QWtQcetwspKhbZmYDRtTj4b5saulUB4X0nvZG8ggGnPRLG0QZs/s1600/Clipboard01.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg831f9rCL75ZEkuUoFCsqeZdeen8dLh0l9VgVO0yHtS0I_uFkaWS6RIht0AlvHyBnjv5-LyUT8EJwrfyO8BKpmhr-p_QWtQcetwspKhbZmYDRtTj4b5saulUB4X0nvZG8ggGnPRLG0QZs/s320/Clipboard01.jpg" /></a></div>To make life a little easier the user has to write function parameters (and independent variables) starting with an underscore. This saves the effort of trying to recognize all possible mathematical functions from the string (e.g. no searching from tan, sin, exp, sqrt etc. required) which would have been nicer of course. All in all the code is about 400 lines (see below), a bit more than expected. But it was a nice exercise to get familiar with <a href="http://www.riverbankcomputing.co.uk/software/pyqt/intro">PyQt</a> and guiqwt.<br />
<br />
<pre class="brush: py">from guidata.qt.QtGui import QWidget, QVBoxLayout, QHBoxLayout,QPalette
from guidata.qt.QtGui import QPushButton,QSlider,QGridLayout,QLabel,QFont
from guidata.qt.QtGui import QCheckBox,QLineEdit,QComboBox,QSizePolicy
from guidata.qt.QtCore import SIGNAL
from guidata.qt.QtGui import QApplication
from guidata.qt.QtCore import Qt
from guiqwt.plot import CurveWidget
from guiqwt.builder import make
from guidata.configtools import get_icon
import numpy
from numpy import *
from functools import partial
class TFunctionParameter:
"""
This class defines the properties of a function parameter.
The properties are:
-Name
-Value
-Minimum allowed allowed parameter value
-Maximum allowed allowed parameter value
"""
def __init__(self,name,value,minValue=None,maxValue=None,scientific=False):
"""
Constructor for the TFunctionParameter class
Arguments:
name: name of the parameter (this is the string you use in the function definition!)
value: (initial) value of the parameter
minValue: if provided the minimum allowed value of this parameter. Otherwise default 0.1*value
maxValue: if provided the maximum allowed value of this parameter. Otherwise default 10.0*value
"""
self._name=name
self._v=value
self._scientific=scientific
if(minValue):
self._minV=minValue
else:
self._minV=0.1*self._v
if(maxValue):
self._maxV=maxValue
else:
self._maxV=10.0*self._v
def Scientific(self):
return(self._scientific)
def v(self):
return(self._v)
def setValue(self,v):
if(v<self._minV):
self._v=self._minV
elif(v>self._maxV):
self._v=self._maxV
else:
self._v=v
def getValue(self):
return(self._v)
def minValue(self):
return(self._minV)
def maxValue(self):
return(self._maxV)
def name(self):
return(self._name)
def checkV(self):
"""
Checks if parameter value still between min-max. If not
value is adjusted to fall in allowed range.
"""
if(self._v>self._minV):
self._v=self._maxV
elif(self._v<self._minV):
self._v=self._minV
def setMinValue(self,minV):
"""
Adjust the minimum allowed value for this parameter. The parameter
value is adjusted to fall in the new range. This function returns
the current (possibly adjusted) parameter value.
The new min value is not allowed to be bigger than the current
maximum allowed value.
"""
if(minV>self._maxV):
self._minV=self._maxV
else:
self._minV=minV
self.checkV()
return(self._v)
def setMaxValue(self,maxV):
"""
Adjust the maximum allowed value for this parameter. The parameter
value is adjusted to fall in the new range. This function returns
the current (possibly adjusted) parameter value.
The new max value is not allowed to be smaller than the current
minium allowed value
"""
if(maxV<self._minV):
self._maxV=self._minV
else:
self._maxV=maxV
self.checkV()
return(self._v)
def setMinAndMAxValues(self,minV,maxV):
"""
Adjust the minimum and maximum allowed values simulataneously.
Use this function to specify a completely new range for this
parameter
This function returns
the current (possibly adjusted) parameter value.
"""
self._minV=minV
self._maxV=maxV
self.checkV()
return(self._v)
class TFunctionUI(QWidget):
"""
The idea of this class is to provide an automatic user interface
to adjust all function parameter values and see the effect on the
function behaviour.
"""
def __init__(self,parent):
"""Constructor of the TFunctionUI class"""
self._parameterDict={}
QWidget.__init__(self, parent)
self.setMinimumSize(520, 400)
self.x = [0,1]
self.xvec=[0,1]
self.y = [0,1]
self.func = self.calc
self.title="f"
self.autoScale=True
#---guiqwt curve item attribute:
self.curve_item = None
#---
def calc(self):
"""
Based on function string calculate function
"""
xpar=self._parameterDict[self._xName]
xvec=numpy.r_[xpar.minValue():xpar.maxValue():-250j]
evalString=self.funcString[:]
for par in self.paramLst:
if(par==self._xName):
evalString=evalString.replace(par,"xvec")
else:
evalString=evalString.replace(par,"self._parameterDict['%s'].v()"%(par))
self.xvec=xvec
f=eval(evalString)
return(f)
def addParameter(self,p):
"""
Add a parameter.
Arguments:
p - The parameter to add. Must be an instance of TFunctionParameter.
"""
self._parameterDict[p.name()]=p
def defineX(self,x):
"""
Define the value on the x-axis.
Arguments:
x - the parameter to plot on x-axis. Must be an a string and part of the parameter list
"""
self._xName=x
def parValue(self,p):
"""
Returns the current value for parameter p
"""
return(self._parameterDict[p.name()].v())
def getParametersAndFunction(self,fs):
"""
Extract the parameters from the function string
fs: function string
"""
math_symbols=['<','>','=','*','+','-','/','(',')']
#Process string
#find first occurence of _ to indicate a parameter
NParams=fs.count('_')
self.paramLst=[] #Stores parameter names in the order as they appear in the equation
self.funcString=fs[:]
cntParams=0
#~ print self.funcString
for np in range(NParams):
index=fs.index('_')
#Now look for first math symbol to signify end of parameter name
i=index+1
while(i<len(fs) and (fs[i] not in math_symbols )):
i+=1
#Now the parameter name is from index upto i-1
parName=fs[index:min(i,len(fs))] #include the underscore
fs=fs[i:]
if(parName in self.paramLst):
continue #Do not count same parameter double
p=TFunctionParameter(parName,1.0,-5.0,5.0,scientific=False)
self.paramLst.append(parName)
self.addParameter(p)
def slideChange(self,key):
"""
Process slide change. Key indicates to which parameter
the slide belongs to
"""
v=self.slideDict[key].value()
float_v=v/1000.0
p=self._parameterDict[key]
p_value=p.minValue()+(p.maxValue()-p.minValue())*float_v
p.setValue(p_value)
if(p.Scientific()):
self.valDict[key].setText("%5.4e"%(p_value))
else:
self.valDict[key].setText("%6.3f"%(p_value))
self.process_data()
def toggleAutoScale(self,v):
self.autoScale=self.autoScaleCheckBox.checkState()
self.process_data()
def parMinChanged(self,key):
"""
Process a change of a minimum parameter value
"""
p=self._parameterDict[key]
p.setMinValue(self.minLEDict[key].text().toFloat()[0])
self.process_data()
def parMaxChanged(self,key):
"""
Process a change of a minimum parameter value
"""
p=self._parameterDict[key]
p.setMaxValue(self.maxLEDict[key].text().toFloat()[0])
self.process_data()
def xAxisChange(self,key):
"""
The user has selected a different x-axis
"""
#Active slider of current x-axis
self.slideDict[self._xName].setEnabled(True)
self._xName=str(key)
self.slideDict[self._xName].setEnabled(False)
self.plot.set_axis_title(self.plot.X_BOTTOM,self._xName)
self.process_data()
def extend_widget(self):
func_string=str(self.eqLE.text())
self.getParametersAndFunction(func_string)
self.eqLE.setEnabled(False)
self.processButton.setEnabled(False)
self.title="f=%s"%(func_string)
#Check if _x in parameter list
if("_x" in self._parameterDict.keys()):
self.defineX("_x") #Make _x the x-axis parameter
else:
self.defineX(self._parameterDict.keys()[0]) #Make the first parameter the x-axis
#Loop over function parameters and setup sliders
cnt=0
glayout=QGridLayout()
self.slideDict={} #to store parameter value sliders
self.minLEDict={} #to store minimum parameter value LineEdits
self.maxLEDict={} #to store maximum parameter value LineEdits
self.valDict={} #to store labels containing parameter values
self.xCBDict={} #To store x-axis selection checkbox
palette = QPalette()
palette.setColor(QPalette.Foreground,Qt.blue)
key_lst=self._parameterDict.keys()
key_lst.sort()
self.xCombo=QComboBox()
sizePol=QSizePolicy(QSizePolicy.Preferred,QSizePolicy.Preferred)
for key in key_lst:
self.xCombo.addItem(key)
p=self._parameterDict[key]
nameLabel=QLabel(p.name()+': ')
glayout.addWidget(nameLabel,cnt,0)
if(p.Scientific()):
valLabel=QLabel("%5.4e"%(p.v()))
minLE=QLineEdit("%5.4e"%(p.minValue()))
#~ minLE.sizeHint(10)
maxLE=QLineEdit("%5.4e"%(p.maxValue()))
else:
valLabel=QLabel("%6.3f"%(p.v()))
minLE=QLineEdit("%6.3f"%(p.minValue()))
#~ minLE.sizeHint(10)
maxLE=QLineEdit("%6.3f"%(p.maxValue()))
self.connect(minLE,SIGNAL('editingFinished ()'),\
partial(self.parMinChanged,key))
self.connect(maxLE,SIGNAL('editingFinished ()'),\
partial(self.parMaxChanged,key))
minLE.setSizePolicy(sizePol)
maxLE.setSizePolicy(sizePol)
self.minLEDict[key]=minLE
self.maxLEDict[key]=maxLE
valLabel.setPalette(palette)
self.valDict[key]=valLabel
glayout.addWidget(valLabel,cnt,1)
glayout.addWidget(minLE,cnt,2)
glayout.addWidget(maxLE,cnt,4)
sld = QSlider(Qt.Horizontal)
sld.setRange(0,1000)
s_value=int(1000*(p.v()-p.minValue())/(p.maxValue()-p.minValue()))
sld.setValue(s_value)
if(p.name()==self._xName):
sld.setEnabled(False)
self.connect(sld, SIGNAL('valueChanged(int)'),\
partial(self.slideChange,key))
self.slideDict[key]=sld
#Connect this slider to the parameter
glayout.addWidget(sld,cnt,3)
cnt+=1
self.vlayout.addLayout(glayout)
hl=QHBoxLayout()
self.xCombo.setCurrentIndex(self.xCombo.findText(self._xName))
self.autoScaleCheckBox=QCheckBox("Auto scale y-axis")
self.autoScaleCheckBox.setCheckState(self.autoScale)
self.autoScaleCheckBox.setTristate(False)
self.connect(self.autoScaleCheckBox,SIGNAL('stateChanged(int)'),\
self.toggleAutoScale)
hl.addWidget(self.autoScaleCheckBox)
hl2=QHBoxLayout()
xlab=QLabel("X-axis: ")
hl2.addStretch()
hl2.addWidget(xlab)
hl2.addWidget(self.xCombo)
hl.addLayout(hl2)
self.connect(self.xCombo,SIGNAL('currentIndexChanged(QString)'),\
self.xAxisChange)
self.vlayout.addLayout(hl)
self.setLayout(self.vlayout)
self.plot.set_axis_title(self.plot.Y_LEFT,self.title)
self.plot.set_axis_title(self.plot.X_BOTTOM,self._xName)
self.process_data()
def setup_widget(self, title):
#---Create the plot widget:
self.curvewidget = CurveWidget(self)
self.curvewidget.register_all_curve_tools()
self.curve_item = make.curve([], [], color='b')
self.curvewidget.plot.add_item(self.curve_item)
self.curvewidget.plot.set_antialiasing(True)
self.plot = self.curvewidget.get_plot()
font = QFont()
font.setPointSize( 16 )
self.plot.set_axis_font("left", font)
self.plot.set_axis_font("bottom", font)
#---
self.eqLE=QLineEdit()
self.eqLE.setPlaceholderText(\
"e.g. sin(_x**2/_a+_y**2/_b), just start parameters with one underscore!")
self.eqLE.selectAll()
self.processButton=QPushButton("Process")
self.connect(self.processButton,SIGNAL('clicked()'),\
self.extend_widget)
hlayout=QHBoxLayout()
hlayout.addWidget(self.eqLE)
hlayout.addWidget(self.processButton)
self.vlayout = QVBoxLayout()
self.vlayout.addWidget(self.curvewidget)
self.vlayout.addLayout(hlayout)
self.setLayout(self.vlayout)
def process_data(self):
self.y = self.calc()
if(self.autoScale):
self.plot.set_axis_limits(self.plot.Y_LEFT,min(self.y),max(self.y))
self.update_curve()
def update_curve(self):
#---Update curve
self.curve_item.set_data(self.xvec, self.y)
self.curve_item.plot().replot()
class TestWindow(QWidget):
def __init__(self):
QWidget.__init__(self)
self.setWindowTitle("FunctionPlotter(guiqwt)")
self.setWindowIcon(get_icon('guiqwt.svg'))
hlayout = QHBoxLayout()
self.setLayout(hlayout)
def add_plot(self, title):
self.widget = TFunctionUI(self)
self.widget.setup_widget(title)
self.layout().addWidget(self.widget)
if __name__ == "__main__":
app = QApplication([])
win = TestWindow()
win.add_plot("")
win.show()
app.exec_()
</pre></div>Korbininhttp://www.blogger.com/profile/00742811285841484666noreply@blogger.com4tag:blogger.com,1999:blog-2970661570838560548.post-85160286941326668302014-02-24T17:28:00.001+01:002014-02-24T17:28:34.945+01:00Locating the site-packages folder from within python<div dir="ltr" style="text-align: left;" trbidi="on">Nice way to find where the site-packages folder is located for your linux distribution from within python itself (found <a href="http://www.tangowithdjango.com/book/chapters/requirements.html">here</a>)<br />
<br />
<pre class="brush: py">import site
print site.getsitepackages()
</pre></div><br />
Apparently not available in older python version.Korbininhttp://www.blogger.com/profile/00742811285841484666noreply@blogger.com0tag:blogger.com,1999:blog-2970661570838560548.post-76090300029150664592013-11-28T15:42:00.000+01:002013-11-28T15:42:15.129+01:00spreadsheet like line plot with filled areas in python<div dir="ltr" style="text-align: left;" trbidi="on">A nice way to show if a series of values fall within a certain, per sample in the series variable, range is to make a line plot with a shaded area indicating the range. <br />
<br />
Example:<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEivHB1HmXQPxCXlb4c09eFIrvBMECMVqM1iMLZw8KDdshnxDZb41s5LYDu3Hri4uoC9H8eiQ0bc74wRBb27r0BWM4S51vVN4Pjptg4dGqtJ4aD3DUi9qRP8NgGh6Psn4p5xpkyREQFHBAM/s1600/example.png" imageanchor="1" ><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEivHB1HmXQPxCXlb4c09eFIrvBMECMVqM1iMLZw8KDdshnxDZb41s5LYDu3Hri4uoC9H8eiQ0bc74wRBb27r0BWM4S51vVN4Pjptg4dGqtJ4aD3DUi9qRP8NgGh6Psn4p5xpkyREQFHBAM/s320/example.png" /></a><br />
<br />
<br />
A line plot is easily made in a typical spreadsheet program. Getting the correct region shaded in the plot by combining area and line plots was however much too cumbersome for me. Therefore I had a look if matplotlib also supports area plots and luckily it does: <a href="http://matplotlib.org/examples/pylab_examples/fill_between_demo.html">fill_between</a>. The only difficulty I encountered was that matplotlib is mainly intended for making scatter plots, i.e. a data series with meaningful x and y coordinates. A typical spreadsheet line plot however has text labels for all points on the x-axis (as shown in the example). The easiest solution I could come up with was to simply number the samples in the data series as 1, 2, 3 etc. and then change the ticks on the x-axis manually with the xticks function. If there are too many samples in the data series and the x-axis gets too full and labels start to overlap, most spreadsheet programs simply drop some labels. The code below does the same. The code seems a bit long but most lines are actually used for reading in the data from a csv file.<br />
<br />
<pre class="brush: py">import csv
#!/usr/bin/env python
import csv
from pylab import *
violet=(90.0/255.0,36.0/255.0,90.0/255.0)
red=(1.0,0.0,0.0)
green=(0.0,1.0,0.0)
def plotFancy(fn, label,figNum=None,ymin=0.0,ymax=500.0,maxticks=20):
"""
plots directly from a csv file (with a header row!!)
file layout:
column
1: strip name
2: Exp
3: Mod
4: min
5: max
use label to name y-axis
if figNum is supplied the grap will be plotted in the figure with
that number (and cleared first)
ymin and ymax determine the scale on the y-axis (i.e. ylim(ymin,ymax))
maxticks gives the maximum number of ticks (labels) allowed on the x axis
"""
f=open(fn,'rb')
reader=csv.reader(f,delimiter=';')
xlabel_lst=[]
y_min_lst=[]
y_max_lst=[]
y_mod_lst=[]
y_exp_lst=[]
line_cnt=0
for line_lst in reader:
line_cnt+=1
if(line_cnt==1):
continue
xlabel_lst.append(line_lst[0])
y_exp_lst.append(float(line_lst[1]))
y_mod_lst.append(float(line_lst[2]))
y_min_lst.append(float(line_lst[3]))
y_max_lst.append(float(line_lst[4]))
cnt_lst= [i for i in range(len(y_mod_lst))]
f.close()
if(figNum==None):
figure()
else:
figure(figNum)
clf()
fill_between(cnt_lst,y_min_lst,y_max_lst,facecolor=green,alpha=1.0)
plot(cnt_lst,y_mod_lst,'bo',color=violet,label="%s mod"%(label),ms=12)
plot(cnt_lst,y_exp_lst,'mv',color=red,label="%s exp"%(label),ms=12)
ylim(ymin,ymax)
ylabel("%s"%(label))
grid(b=True)
legend(loc=9)
show()
if(len(xlabel_lst)>maxticks):
delta=len(xlabel_lst)/float(maxticks-1.0)
tick_num_lst=[]
tick_text_lst=[]
index=0
for i in range(maxticks-1):
index=int(i*delta)
tick_num_lst.append(index)
tick_text_lst.append(xlabel_lst[index])
tick_num_lst.append(len(xlabel_lst)-1)
tick_text_lst.append(xlabel_lst[len(xlabel_lst)-1])
xticks(tick_num_lst,tick_text_lst,rotation=90)
else:
xticks(arange(len(xlabel_lst)),xlabel_lst,rotation=90)
xlim(0,len(xlabel_lst))
</pre></div>Korbininhttp://www.blogger.com/profile/00742811285841484666noreply@blogger.com1tag:blogger.com,1999:blog-2970661570838560548.post-18798204876720732712013-09-07T15:19:00.001+02:002013-09-07T15:19:27.673+02:00Reading/writing a python dictionary to file<div dir="ltr" style="text-align: left;" trbidi="on">To save time building a large dictionary every time I run my program I googled "saving a python dictionary to file". Of the suggested solutions I liked the option to <a href="http://stackoverflow.com/questions/4893689/save-a-dictionary-to-a-file-alternative-to-pickle-in-python">write to a csv</a> file best. However, the posted code did not work for me because the value in the dictionary was a very big nested list of lists and not a simple string. This was easy to fix by calling eval on the value obtained from the csv reader. Of course I was <a href="http://stackoverflow.com/questions/11026959/python-writing-dict-to-txt-file-and-reading-dict-from-txt-file">not the first one</a> to realize this.<br />
<br />
Below for completeness my code:<br />
<br />
<pre class="brush: py">import csv
def saveDict(fn,dict_rap):
f=open(fn, "wb")
w = csv.writer(f)
for key, val in dict_rap.items():
w.writerow([key, val])
f.close()
def readDict(fn):
f=open(fn,'rb')
dict_rap={}
for key, val in csv.reader(f):
dict_rap[key]=eval(val)
f.close()
return(dict_rap)
</pre></div>Korbininhttp://www.blogger.com/profile/00742811285841484666noreply@blogger.com0tag:blogger.com,1999:blog-2970661570838560548.post-6571920587354671312013-09-02T20:59:00.000+02:002013-09-02T20:59:10.342+02:00Creating a Cython extension type for use with multiProcessing for function fitting<div dir="ltr" style="text-align: left;" trbidi="on">If you have to fit a complex function to a very big data set it would be nice to be able to use all the cores your cpu has. Because the data set is very big it should be efficient to simply split the data set over a number of cores and calculate the total error sum (sum squared error) in parts in parallel. This sounds simple but it took me some effort to do this in python/Cython on both linux and windows. After googling for a while, I decided that using the multiProcessing module should work best for my specific situation (which contains a lot of python code which makes it difficult to turn the GIL temporarily off). On linux I had things running relatively fast, but on windows I could not get it to function. The difference is caused by the lack of real processes on windows (or at least they work differently). On a fork() in linux everything is copied but this does not happen on windows and you have to take care that all data is correctly passed to the child process (read "Explicitly pass resources to child processes" in the multiprocessing <a href="http://docs.python.org/2/library/multiprocessing.html">documentation</a>). <br />
<br />
To try things out I started with a simple example:<br />
<br />
<pre class="brush: py">#!/usr/bin/env python
from multiprocessing import Process,Queue
import sys,numpy,pylab
class TFitFunc:
def __init__(self,X0,x,y,pid=1):
self.a=X0[0]
self.b=X0[1]
self.c=X0[2]
self.x=x[:]
self.y=y[:]
self.pid=pid
def __call__(self,X):
self.a=X[0]
self.b=X[1]
self.c=X[2]
errsum=0
for i in range(100):
ymod=self.a*self.x**2+self.b*self.x+self.c
errsum+=numpy.sum((ymod[:]-self.y[:])**2)
return(errsum)
class TFitFuncComplex:
def __init__(self,X0,x,y,pid=1):
self.a=X0[0]
self.b=X0[1]
self.c=X0[2]
self.x=x[:]
self.y=y[:]
self.pid=pid
def __call__(self,X):
self.a=X[0]
self.b=X[1]
self.c=X[2]
errsum=0
for i in range(100):
ymod=self.a*self.x**2+self.b*self.x+self.c+\
numpy.sin(numpy.sqrt(self.x))*numpy.cos(self.x+0.5)\
/self.a*numpy.sqrt(self.b)
errsum+=numpy.sum((ymod[:]-self.y[:])**2)
return(errsum)
def f(fitfunc,X,Q=None):
errsum=fitfunc(X)
print "%d: errsum:%e"%(fitfunc.pid,errsum)
if(Q!=None):
Q.put(errsum)
return(errsum)
def main():
a=1.0
b=2.0
c=3.0
x1=numpy.r_[0:10:-10000000j]
x2=numpy.r_[0:10:-10000000j]
y1=a*x1**2+b*x1+c
y1+=y1*0.35*(numpy.random.random(len(x1))-0.5)
y2=a*x2**2+b*x2+c
y2+=y2*0.35*(numpy.random.random(len(x2))-0.5)
X0=[0.5,2.5,1.5]
f1=TFitFunc(X0,x1,y1,1)
f2=TFitFunc(X0,x2,y2,2)
ps=[]
for i in range(2):
if(i==0):
p=Process(target=f,args=(f1,X0))
else:
p=Process(target=f,args=(f2,X0))
p.start()
ps.append(p)
for p in ps:
p.join()
def main_complex():
a=1.0
b=2.0
c=3.0
x1=numpy.r_[0:10:-1000000j]
x2=numpy.r_[0:10:-1000000j]
y1=a*x1**2+b*x1+c
y1+=y1*0.35*(numpy.random.random(len(x1))-0.5)
y2=a*x2**2+b*x2+c
y2+=y2*0.35*(numpy.random.random(len(x2))-0.5)
X0=[0.5,2.5,1.5]
f1=TFitFuncComplex(X0,x1,y1,1)
f2=TFitFuncComplex(X0,x2,y2,2)
ps=[]
Qs=[]
errSum=0.0
for i in range(2):
Qs.append(Queue())
if(i==0):
p=Process(target=f,args=(f1,X0,Qs[i]))
else:
p=Process(target=f,args=(f2,X0,Qs[i]))
p.start()
ps.append(p)
for i in range(2):
errSum+=Qs[i].get()
ps[i].join()
print "Total errsum: %e"%(errSum)
def main_single_complex():
a=1.0
b=2.0
c=3.0
x1=numpy.r_[0:10:-1000000j]
x2=numpy.r_[0:10:-1000000j]
y1=a*x1**2+b*x1+c
y1+=y1*0.35*(numpy.random.random(len(x1))-0.5)
y2=a*x2**2+b*x2+c
y2+=y2*0.35*(numpy.random.random(len(x2))-0.5)
X0=[0.5,2.5,1.5]
f1=TFitFuncComplex(X0,x1,y1,1)
f2=TFitFuncComplex(X0,x2,y2,2)
errSum=f(f1,X0)
errSum+=f(f2,X0)
print "Total errsum: %e"%(errSum)
def main_single():
a=1.0
b=2.0
c=3.0
x1=numpy.r_[0:10:-10000000j]
x2=numpy.r_[0:10:-10000000j]
y1=a*x1**2+b*x1+c
y1+=y1*0.35*(numpy.random.random(len(x1))-0.5)
y2=a*x2**2+b*x2+c
y2+=y2*0.35*(numpy.random.random(len(x2))-0.5)
X0=[0.5,2.5,1.5]
f1=TFitFunc(X0,x1,y1,1)
f2=TFitFunc(X0,x2,y2,2)
f(f1,X0)
f(f2,X0)
if __name__=='__main__':
main_single()
#~ main()
#~ main_complex()
#~ main_single_complex()
</pre><br />
This works fine on linux and windows. However, this is pure python. Normally, I use a lot of Cython code in extension types (aka cython classes). And this I could not get to work without some <a href="https://groups.google.com/forum/#!topic/cython-users/vzG58m0Yr2Y">more research</a>. Now first my solution. The first part shows the .pyx file with two classes, one normal python class with some cython code in it and one real extension type. The second part shows the script using these classes.<br />
<br />
TFitFunctions.pyx<br />
<pre class="brush: py">#!/usr/bin/env python
import numpy as np
cimport numpy as np
class TFitFunc:
"""
Simple demonstration class to be used as fit function.
By definition of the __call__ member function an object of this
class is callable (functor). All additional data required
to calculate the error sum should be passed to the constructor.
The function here is simply
y=a*x**2+b*x+c
"""
def __init__(self,X0,x,y,pid=1):
"""
Constructor.
Arguments:
X0: list of initial values for the three model parameters [a, b, c]
x: array of x values
y: array of y values (typically experimentally determined data points)
pid: optional "process id"
"""
self.a=X0[0]
self.b=X0[1]
self.c=X0[2]
self.x=x[:]
self.y=y[:]
self._pid=pid
def pid(self):
return(self._pid)
def __call__(self,X):
"""
Make objects of this class callable. The argument is a list/array
of model parameter values [a,b,c]
The function returns the sum squared error
"""
cdef double errsum
cdef int i
self.a=X[0] #could also have used X directly in the calculation below
self.b=X[1]
self.c=X[2]
errsum=0
for i in range(100): #do this a hundred times to waste some CPU time
ymod=self.a*self.x**2+self.b*self.x+self.c #calculate model values
errsum+=np.sum((ymod[:]-self.y[:])**2) # calculate summed square error
errsum/=100.0
return(errsum)
cdef class TFitFuncComplex:
"""
Simple demonstration class to be used as fit function.
Very similar to TFitFunc but with a more complex (and time consuming)
function. Another big difference is that now the class is defined
as an extension type.
By definition of the __call__ member function an object of this
class is callable (functor). All additional data required
to calculate the error sum should be passed to the constructor.
The function here is simply
y=a*x**2+b*x+c+sin(sqrt(x))*cos(x+0.5)/(a*b*b)
"""
cdef double a,b,c #in an extension type class member variables must be defined here
cdef int _pid
cdef np.ndarray x,y
def __init__(self,X0,np.ndarray[double, ndim=1]x,np.ndarray[double, ndim=1]y,int pid=1):
"""
Constructor.
Arguments:
X0: list/array of initial values for the three model parameters [a, b, c]
x: array of x values
y: array of y values (typically experimentally determined data points)
pid: optional "process id"
"""
self.a=X0[0]
self.b=X0[1]
self.c=X0[2]
self.x=x[:]
self.y=y[:]
self._pid=pid
def pid(self):
return(self._pid)
def __call__(self,X):
"""
Make objects of this class callable. The argument is a list/array
of model parameter values [a,b,c]
The function returns the sum squared error
"""
cdef double errsum
cdef int i
self.a=X[0] #could also have used X directly in the calculation below
self.b=X[1]
self.c=X[2]
errsum=0
for i in xrange(100):#do this a hundred times to waste some CPU time
ymod=self.a*self.x**2+self.b*self.x+self.c+\
np.sin(np.sqrt(self.x))*np.cos(self.x+0.5)\
/self.a*np.sqrt(self.b) #calculate model values
errsum+=np.sum((ymod[:]-self.y[:])**2) # calculate summed square error
errsum/=100.0
return(errsum)
def __reduce__(self):
"""
Without this function the code will not run with multiProcessing
on Windows.
It has something to do with making an extension type
pickable. For a normal python class this is not required
(see TFitFunc)
"""
return TFitFuncComplex, ([self.a,self.b,self.c],self.x,self.y,self._pid)
</pre><br />
calling script:<br />
<br />
<pre class="brush: py">#!/usr/bin/env python
from multiprocessing import Process,Queue
import sys,numpy
from TFitFunctions import TFitFunc,TFitFuncComplex
"""
Demonstration of use Process with a callable Python/Cython classes
Can be easily extended into a real multiProcessing fitting
setup for use with (e.g.) fmin
The basic idea is that you have a huge amount of data points
that must be evaluated for the calculation of the summed squared error.
These data points are independent by nature and thus ideal for
parallel processing.
"""
def f(fitfunc,X,Q=None):
"""
Function to be passed to Process as target. The function
will call fitfunc with X as argument and put the result in the
Queue Q is one is passed as argument.
"""
errsum=fitfunc(X)
print "%d: errsum:%e"%(fitfunc.pid(),errsum)
if(Q!=None):
Q.put(errsum)
return(errsum)
def main():
"""
Demonstration of use of TFitFunc
"""
a=1.0 #model parameters
b=2.0
c=3.0
x1=numpy.r_[0:10:-10000000j] #data points for first process (x)
x2=numpy.r_[0:10:-10000000j] #measurement points for second process (x)
y1=a*x1**2+b*x1+c #"measured" data for process 1
y1+=y1*0.35*(numpy.random.random(len(x1))-0.5) #add some noise
y2=a*x2**2+b*x2+c #"measured" data for process 2
y2+=y2*0.35*(numpy.random.random(len(x2))-0.5) #add some noise
X0=[0.5,2.5,1.5] #initial guess for model parameters
f1=TFitFunc(X0,x1,y1,1) #Create fit function for process 1
f2=TFitFunc(X0,x2,y2,2) #Create fit function for process 2
ps=[] #to contain process
Qs=[]
errSum=0.0
for i in range(2):
Qs.append(Queue())
if(i==0):
p=Process(target=f,args=(f1,X0,Qs[i])) #create process 1
else:
p=Process(target=f,args=(f2,X0,Qs[i])) #create process 2
p.start() #start process
ps.append(p) #add process "handle"
for i in range(2):
errSum+=Qs[i].get() #collect error sums from processes
ps[i].join() #wait for process to finish
print "Total errsum: %e"%(errSum)
def main_complex():
"""
Same as main but now with TFitFuncComplex
"""
a=1.0
b=2.0
c=3.0
x1=numpy.r_[0:10:-1000000j]
x2=numpy.r_[0:10:-1000000j]
y1=a*x1**2+b*x1+c
y1+=y1*0.35*(numpy.random.random(len(x1))-0.5)
y2=a*x2**2+b*x2+c
y2+=y2*0.35*(numpy.random.random(len(x2))-0.5)
X0=[0.5,2.5,1.5]
f1=TFitFuncComplex(X0,x1,y1,1)
f2=TFitFuncComplex(X0,x2,y2,2)
ps=[]
Qs=[]
errSum=0.0
for i in range(2):
Qs.append(Queue())
if(i==0):
p=Process(target=f,args=(f1,X0,Qs[i]))
else:
p=Process(target=f,args=(f2,X0,Qs[i]))
p.start()
ps.append(p)
for i in range(2):
errSum+=Qs[i].get()
ps[i].join()
print "Total errsum: %e"%(errSum)
def calcErrorSum(X,fitfuncs):
"""
Calculate error sum for given model parameters (X)
by using the functions in fitfuncs in (parallel) processes
"""
ps=[]
Qs=[]
errSum=0.0
for i in range(len(fitfuncs)):
Qs.append(Queue())
p=Process(target=f,args=(fitfuncs[i],X,Qs[i]))
p.start()
ps.append(p)
for i in range(2):
errSum+=Qs[i].get()
ps[i].join()
print "Total errsum: %e"%(errSum)
return(errSum)
def main_complex2():
"""
Same as main_complex but now with data creation part seperated
from function evaluation part. PLease note that
the function calcErrorSum can be used as a (multiProcessing)
argument to (e.g.) fmin
"""
a=1.0
b=2.0
c=3.0
x1=numpy.r_[0:10:-1000000j]
x2=numpy.r_[0:10:-1000000j]
y1=a*x1**2+b*x1+c
y1+=y1*0.35*(numpy.random.random(len(x1))-0.5)
y2=a*x2**2+b*x2+c
y2+=y2*0.35*(numpy.random.random(len(x2))-0.5)
X0=[0.5,2.5,1.5]
f1=TFitFuncComplex(X0,x1,y1,1)
f2=TFitFuncComplex(X0,x2,y2,2)
fitfuncs=[f1,f2]
calcErrorSum(X0,fitfuncs)
def main_single_complex():
"""
Same a main_complex but now serial evaluation (for time comparison purposes)
"""
a=1.0
b=2.0
c=3.0
x1=numpy.r_[0:10:-1000000j]
x2=numpy.r_[0:10:-1000000j]
y1=a*x1**2+b*x1+c
y1+=y1*0.35*(numpy.random.random(len(x1))-0.5)
y2=a*x2**2+b*x2+c
y2+=y2*0.35*(numpy.random.random(len(x2))-0.5)
X0=[0.5,2.5,1.5]
f1=TFitFuncComplex(X0,x1,y1,1)
f2=TFitFuncComplex(X0,x2,y2,2)
errSum=f(f1,X0)
errSum+=f(f2,X0)
print "Total errsum: %e"%(errSum)
def main_single():
"""
Same a main but now serial evaluation (for time comparison purposes)
"""
a=1.0
b=2.0
c=3.0
x1=numpy.r_[0:10:-10000000j]
x2=numpy.r_[0:10:-10000000j]
y1=a*x1**2+b*x1+c
y1+=y1*0.35*(numpy.random.random(len(x1))-0.5)
y2=a*x2**2+b*x2+c
y2+=y2*0.35*(numpy.random.random(len(x2))-0.5)
X0=[0.5,2.5,1.5]
f1=TFitFunc(X0,x1,y1,1)
f2=TFitFunc(X0,x2,y2,2)
f(f1,X0)
f(f2,X0)
if __name__=='__main__':
#~ main_single()
#~ main()
#~ main_complex2()
main_complex()
#~ main_single_complex()
</pre><br />
The code in TFitFunctions.pyx shows that the trick for an extension type is to add the __reduce__ method to make it pickable. This is only needed on windows.<br />
</div>Korbininhttp://www.blogger.com/profile/00742811285841484666noreply@blogger.com4tag:blogger.com,1999:blog-2970661570838560548.post-25324360349626409512013-05-26T12:12:00.000+02:002013-05-26T12:12:32.562+02:00cython wraparound problems<div dir="ltr" style="text-align: left;" trbidi="on">For performance reasons I was including the line<br />
<br />
#cython: wraparound=False<br />
<br />
in most of my Cython files. After an update to a newer versions of Cython (0.19) my Cython scripts started to crash. It took me some time to find out that combined with the wraparound=False line you cannot use [-1] as the index for the last element anymore (this is also written somewhere in the Cython documentation).<br />
</div>Korbininhttp://www.blogger.com/profile/00742811285841484666noreply@blogger.com0tag:blogger.com,1999:blog-2970661570838560548.post-55251210896533046202013-05-18T15:11:00.002+02:002013-05-18T15:11:55.749+02:00Allowing only a single instance of a program to open using Qt<div dir="ltr" style="text-align: left;" trbidi="on">Many text editors open a new tab and not a new editor window when a second (etc. etc.) document is opened. For my program that displays curves of simulation results I wanted to have similar behaviour. This program is written in Qt using Qwt (also see <a href="http://korbinin.blogspot.nl/2010_09_01_archive.html">this</a> earlier post). This program already supports drag and drop and file association (i.e. double clicking on a file to open it with this program). To prevent a second program from opening when a second file associated with this program is double clicked I only had to follow <a href="http://www.qtcentre.org/wiki/index.php?title=SingleApplication">this example</a>.<br />
</div>Korbininhttp://www.blogger.com/profile/00742811285841484666noreply@blogger.com0tag:blogger.com,1999:blog-2970661570838560548.post-27431327854917939332013-03-25T21:16:00.002+01:002013-03-25T21:16:31.236+01:00Python profiling trials<div dir="ltr" style="text-align: left;" trbidi="on">Python comes with some default profiling capabilities. However, as also <a href="https://tech.dropbox.com/2012/07/plop-low-overhead-profiling-for-python/">noted here</a>, this profiling can slow down your program to a crawl making the gathering of useful statistics very tedious. Although it is very easy to get the proposed alternative "<a href="https://tech.dropbox.com/2012/07/plop-low-overhead-profiling-for-python/">plop</a>" running the output it generates is very limited and I did not find a description to get a more traditional output from this profiler. Although <a href="https://pypi.python.org/pypi/statprof/">statprof</a> does generate traditional output without slowing down running your program very much, it does not really seem to deal correctly with wrapper functions which severely limits its applicability to my typical programs. Thus, unfortunately, I seem to be stuck with using CProfile for a while. <br />
<br />
A nice introduction to CProfile is given on <a href="http://stefaanlippens.net/python_profiling_with_pstats_interactive_mode">slippen's blog</a>.<br />
<br />
To turn on profiling of Cython code add the line<br />
<br />
#cython: profile=True<br />
<br />
to the top of your .pyx file (including the hash).<br />
</div>Korbininhttp://www.blogger.com/profile/00742811285841484666noreply@blogger.com0tag:blogger.com,1999:blog-2970661570838560548.post-68267093456082117562013-03-10T16:18:00.001+01:002014-01-13T12:15:54.836+01:00using SWIG to connect python and a c++ dll<div dir="ltr" style="text-align: left;" trbidi="on">Although I am a <a href="http://korbinin.blogspot.nl/2011/07/using-cython.html">great fan of Cython</a> when you already have a class defined in C++ code you do not want to rewrite it for Cython. Already a long time ago I once used <a href="http://www.swig.org/">SWIG</a> to create a link between C++ classes and python. This is fairly easy as long as you have clear interface functions which you can use to pass data to and from your class. Nevertheless, it took me some effort to get things running for my new class. <br />
<br />
Following the <a href="http://www.swig.org/tutorial.html">SWIG tutorial</a> I started with something like this myExample.i<br />
<pre class="brush: c">%module myExample
%{
/* Includes the header in the wrapper code */
#include "myModule.h"
#include "myModule2.h"
%}
/* Parse the header file to generate wrappers */
#include "myModule.h"
#include "myModule2.h"
</pre><br />
After compiling this I got a module that I could import in Python. However, the module did not contain the classes I had defined in my C++ code. Because there are not so many clear examples of SWIG and C++ I could not really find an explanation for this. Only after recreating the exact example of the tutorial (using copy/paste from the web) I found that I made a simple typing error: the #include in the second part of the myExample.i should be %include!<br />
<br />
<pre class="brush: c">%module myExample
%{
/* Includes the header in the wrapper code */
#include "myModule.h"
#include "myModule2.h"
%}
/* Parse the header file to generate wrappers */
%include "myModule.h"
%include "myModule2.h"
</pre><br />
Now everything works nicely. Compiling is not very difficult. I have setup a Makefile for this:<br />
<br />
<pre class="brush: bash">CPP=g++
CFLAGS= -Wunused -O3
PYTH_INCL = /c/Python27/include
PYTH_LIB = /c/Python27/Libs
PYLIB = -lpython27
OBJS= myModule.o
modOBJS= myModule2.o
SRC_DIR = ..
all: myExample_mod
myExample_mod: $(modOBJS) myExample.i
swig -I.. -python -c++ -shadow myExample.i
$(CPP) -I$(PYTH_INCL) $(INCLUDE) -c myExample_wrap.cxx
$(CPP) -shared -L$(PYTH_LIB) myExample_wrap.o $(OBJS) $(PYLIB) $(LIBS) -o _myExample.pyd
%.o: $(SRC_DIR)/%.cpp $(SRC_DIR)/%.h
$(CPP) -c $(CFLAGS) $(INCLUDE) $< -o $@
</pre>
With this Makefile you can use
import myExample
in your python code to import the module. Thus far I have only used simple interface methods in my classed that work with integers and floats as input and output variables. This makes communication between the module and python very simple (no special effort seems to be required). I assume that if you want to pass arrays, strings, etc. things will become more difficult.
</div>Korbininhttp://www.blogger.com/profile/00742811285841484666noreply@blogger.com0tag:blogger.com,1999:blog-2970661570838560548.post-27050028262040239292013-03-06T21:40:00.000+01:002013-03-06T21:43:46.743+01:00cython -mno-cygwin problems<div dir="ltr" style="text-align: left;" trbidi="on">The command I use to compile a cython extension on windows:<br />
<br />
python setup.py build_ext --inplace --compiler=mingw32<br />
<br />
(see also <a href="http://korbinin.blogspot.nl/2011/07/first-steps-with-cython-and-pythonxy.html">these</a> two <a href="http://korbinin.blogspot.nl/2011/07/using-cython.html">posts</a>) has problems with newer mingw/cygwin installations. Already for some to the compiler option -mno-cygwin is no longer used and the mingw compiler should be called directly. When you run the setup.py script with the compiler flag as shown, however, python automatically adds the -mno-cygwin flag. This then stops the compiler. The best solution for this I have found so far is to change the <br />
<br />
c:\Python27\Lib\distutils\cygwinccompiler.py<br />
<br />
file. Simply remove the -mno-cygwin option everywhere in the definition of the Mingw32CCompiler class (just search for no-cygwin in this file and you will find it).<br />
<br />
Looks like I also could have <a href="http://stackoverflow.com/questions/6034390/compiling-with-cython-and-mingw-produces-gcc-error-unrecognized-command-line-o">googled</a> this instead of finding out again how to use grep to search in subdirectories which is not as simple as adding the -r option ( find ./* -type f -exec grep -l "no-cygwin" {} \;).<br />
</div>Korbininhttp://www.blogger.com/profile/00742811285841484666noreply@blogger.com1tag:blogger.com,1999:blog-2970661570838560548.post-61902911345164473262013-02-22T18:27:00.001+01:002013-02-22T18:27:42.343+01:00Location drive specification MinGWPortable<div dir="ltr" style="text-align: left;" trbidi="on">A long time ago I found a portable version of MinGW somewhere on the web. It is sometimes very convenient to have a gcc compiler ready to run on a computer that is not your own. However, I used it mainly on one computer and when I finally did try to run it on a different machine it didn't work. The problem was that it was looking for the MinGW and msys stuff on the wrong drive. Although not very portable it is easy to change this as long as you remember that msys emulated a unix environment! Just change the drive in <br />
<br />
/etc/fstab<br />
<br />
It took me some time to find this. But instead of searching myself I could have looked <a href="http://kemovitra.blogspot.nl/2009/07/fixing-mounts-in-mingw-portable.html">here</a>.<br />
<br />
Somehow I am no longer able to find my original MinWGPortable on the web anymore. That is a pity because the msys terminal is extremely nice: you can resize it in any direction! Perhaps I should have another look <a href="http://portableapps.com/node/18601">here</a>.<br />
</div>Korbininhttp://www.blogger.com/profile/00742811285841484666noreply@blogger.com0tag:blogger.com,1999:blog-2970661570838560548.post-65660128188605250672012-12-09T13:16:00.000+01:002012-12-09T13:16:11.229+01:00Windows application icon in Qt Creator<div dir="ltr" style="text-align: left;" trbidi="on">In an earlier <a href="http://korbinin.blogspot.nl/2012/03/adding-icons-to-application-with-qt.html">post</a> I described how to add icons to a Qt application using Qt Creator. With the procedure described there, the application shows the icon in the window title bar and on the taskbar. However, when you browse to the folder containing the application in windows, the executable will not be shown with the icon. The same is true when you create a shortcut to the executable (windows will also say that the executable does not contain an icon).<br />
<br />
As described <a href="http://doc.qt.digia.com/qt/appicon.html">here</a>, there is a simple windows specific method to add an application icon that will change the appearance of the executable when browsing under windows (and when creating a shortcut to the executable). You must first create a windows specific resource file which has the extension .rc e.g. myapp.rc. This simple text file then needs to contain the (single) line:<br />
<br />
IDI_ICON1 ICON DISCARDABLE "images/myapp.ico"<br />
<br />
The final step is then to add this resource file in the Qt project (.pro) file:<br />
<br />
RC_FILE = myapp.rc<br />
<br />
This works for me when I put the myapp.rc in the same folder as where the Qt project file is located (and that contains the images folder). I did not try any other file locations.<br />
</div>Korbininhttp://www.blogger.com/profile/00742811285841484666noreply@blogger.com2tag:blogger.com,1999:blog-2970661570838560548.post-33218713566275065052012-10-28T14:26:00.001+01:002012-10-28T14:26:52.450+01:00pythonxy cython mingw problems<div dir="ltr" style="text-align: left;" trbidi="on">Again I was having issues using a cython compiled package with pythonxy. For some reason when running from a ipython terminal started from pythonxy, I would get a DLL load error. If I would run the python/cython program directly (i.e. from a dos box: python my_program.py) then it would run fine. By default if you compile a program with mingw you need a dll with the std c or c++ library (typically this dll has a difficult name as libgcc_s_dw2-1.dll). As I have mingw installed in more than one location (32bit, 64bit, QtSDK, Python(x,y)) I think it cannot find the right dll (although I could not detect which dll it was using). The problem can be most easily solved if you link the std library statically. This you can do by simply adding "stdc++" to your libraries field in setup.py.<br />
<br />
So my new setup.py looks now something like<br />
<br />
<pre class="brush: py">from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
import numpy
ext_modules = [Extension("BvL_cython",["model_cython.pyx"],\
libraries=["m","stdc++"],include_dirs=[numpy.get_include()])]
setup(
name= 'Model class',
cmdclass = {'build_ext': build_ext},
ext_modules = ext_modules
)
</pre><br />
(This also shows how to automatically detect the location of the numpy include directories)<br />
</div>Korbininhttp://www.blogger.com/profile/00742811285841484666noreply@blogger.com0tag:blogger.com,1999:blog-2970661570838560548.post-29077203269730092082012-10-13T17:18:00.001+02:002012-10-28T14:15:20.193+01:00Color text output from python<div dir="ltr" style="text-align: left;" trbidi="on">At the moment I run a minimization routine build in Cython/Python that outputs a lot of text to the terminal. Not all text is equally interesting and I was wondering if it is difficult to output some of the more important messages in a different color. It turns out that this is very easy. On stackoverflow Dietrich Epp posted a <a href="http://stackoverflow.com/questions/2330245/python-change-text-color-in-shell">nice example</a> which I have changed a little to select the color with a single character string:<br />
<br />
<pre class="brush: py">def hilite(string, status, bold):
attr = []
if(sys.stdout.isatty()):
if status=='g':
# green
attr.append('32')
elif status=='r':
# red
attr.append('31')
elif status=='y':
# yellow
attr.append('33')
elif status=='b':
# blue
attr.append('34')
elif status=='m':
# magenta
attr.append('35')
if bold:
attr.append('1')
return '\x1b[%sm%s\x1b[0m' % (';'.join(attr), string)
else:
return(string)
</pre><br />
More info on ANSI escape codes is given on <a href="http://en.wikipedia.org/wiki/ANSI_escape_code">Wikipedia</a>. You can also do things like underline. So far I have only tested this on Linux so I do not know if it also works on Windows.<br />
<br />
Update:<br />
This does not work on Windows (windows terminals do not work with ANSI codes). Luckily there is a python package you can use to make it work on windows after all: <a href="http://code.google.com/p/colorama/">colorama</a>. Simply call colorama.init()<br />
(after import colorama) and you can use the above code again. However, I noticed that if you use ipython this garbles your prompt (and other things). To work around this I call colorama.init() in the functions that use the ANSI codes in the output. Then when the script is finished you have to call colorama.deinit() and ipython will behave normally. <br />
<br />
</div>Korbininhttp://www.blogger.com/profile/00742811285841484666noreply@blogger.com0tag:blogger.com,1999:blog-2970661570838560548.post-61601866797770835982012-06-10T14:41:00.000+02:002012-06-10T14:41:45.921+02:00header file location for numpy arrays (cython)<div dir="ltr" style="text-align: left;" trbidi="on">Today was the first time I tried working with numpy arrays in Cython. I got a compilation error saying that the header file for numpy could not be found. <a href="https://bugs.archlinux.org/task/22326">Here</a> I found an easy fix for this problem. The setup.py script can be changed to automatically detect and add the numpy include path:<br />
<br />
<br />
<pre class="brush: py">from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
import numpy
ext_modules = [Extension("xx_cython",["xx_cython.pyx"],\
libraries=["m"],include_dirs=[numpy.get_include()])]
setup(
name= 'xx model class',
cmdclass = {'build_ext': build_ext},
ext_modules = ext_modules
)
</pre><br />
</div>Korbininhttp://www.blogger.com/profile/00742811285841484666noreply@blogger.com0tag:blogger.com,1999:blog-2970661570838560548.post-28304750405387401302012-03-02T16:26:00.000+01:002013-02-28T19:34:04.556+01:00adding icons to an application with Qt CreatorThe book "C++ GUI Programming with Qt 4" gives a short description of how to use a resource file to include images into the executable which can then (for example) be used as icons. The example given is clear by itself but when using the latest QtCreator version (2.4.1) things turn out to work just a little bit different. Furthermore a resource file as given in the book does not load correctly in QtCreator.<br />
<br />
To add image resources I now do this:<br />
1. Create a new resource file by clicking File -> New File or Project and then Choose Qt resource file.<br />
2. Add a prefix for example /icons<br />
3. Add your images files for example: images/icon.png (with a images a subdirectory in your project folder<br />
4. Refer to these resources in your code as ":/icons/images/icon.png"<br />
<br />
The changed point compared to the resource file example in the book is the prefix. You now have to add a prefix which (probably) did not exist in the past. In the example application (called "application") that comes with QtCreator the prefix that is used is simply "/".<br />
<br />
Adding the icon to a widget (e.g. your main window) is now simple:<br />
<br />
<pre class="brush: cpp">setWindowIcon(QIcon(":/icons/images/icon.png"));
</pre>And as requested some screen shots.<br />
<br />
Step 1, creating a new resource file:<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgTSq7s8-1BPo0vcTEWWysgCFdO4LOwJcvAwfkfllZiuUnUdfdSqpE9D30T4oEt4v-LoPzjZGnQ0lCCid3pK4UGbYyxwjk90mVk7wAX0IZvtgh6k-vxtCfS4gAR0v5XtCM7jKQsXOmqqIE/s1600/newResourceFile.jpg" imageanchor="1" ><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgTSq7s8-1BPo0vcTEWWysgCFdO4LOwJcvAwfkfllZiuUnUdfdSqpE9D30T4oEt4v-LoPzjZGnQ0lCCid3pK4UGbYyxwjk90mVk7wAX0IZvtgh6k-vxtCfS4gAR0v5XtCM7jKQsXOmqqIE/s320/newResourceFile.jpg" /></a><br />
<br />
Step 2, add a prefix:<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjbDexxPAAq36Hmi8_IEc2WfaDmpqgaUZHUKPap86vJaLSkIYOTouoVsnA1K63eSQGfQNXzOj05hyphenhyphenu2I6guYk6bXu26wLVffPfd1YHK_MzajiNNq_tSO5U_-v_zSbt1-sHAXtgzI7A2_7g/s1600/addingPrefix.jpg" imageanchor="1" ><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjbDexxPAAq36Hmi8_IEc2WfaDmpqgaUZHUKPap86vJaLSkIYOTouoVsnA1K63eSQGfQNXzOj05hyphenhyphenu2I6guYk6bXu26wLVffPfd1YHK_MzajiNNq_tSO5U_-v_zSbt1-sHAXtgzI7A2_7g/s320/addingPrefix.jpg" /></a><br />
<br />
Simply type over the default prefix label generator by QtCreator. Your prefix will appear in the main window:<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhvZKzRY8HZHxeP2Lvm3Z6a4qQVHlEImJiqramIiYozxeN3jmko9rKeRi0-APfnVsDXT7yU3oWYNATVR0dcIjaCO6N4M5Wxg0rFGAAYo22ERoIFUrc_rNcusxj3oqduWQpPiPxcvSBHNgg/s1600/addingPrefix_part2.jpg" imageanchor="1" ><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhvZKzRY8HZHxeP2Lvm3Z6a4qQVHlEImJiqramIiYozxeN3jmko9rKeRi0-APfnVsDXT7yU3oWYNATVR0dcIjaCO6N4M5Wxg0rFGAAYo22ERoIFUrc_rNcusxj3oqduWQpPiPxcvSBHNgg/s320/addingPrefix_part2.jpg" /></a><br />
<br />
<br />
Step 3, add image files (after selecting the prefix in the main window):<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjbb3JtpRbjSKhwkHp_N4S6IyJjwMHdkH9JxmwHvp8INV_OwmrF02WlKM-RB8tw5yHloRfJQ7EkSoLcGDPRI05Z70Dq-YjtyPWjGQ_Ibqsqaof4o9YJdeY0FEuEuT9pRSO45T4qvtFZXG8/s1600/addingFiles.jpg" imageanchor="1" ><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjbb3JtpRbjSKhwkHp_N4S6IyJjwMHdkH9JxmwHvp8INV_OwmrF02WlKM-RB8tw5yHloRfJQ7EkSoLcGDPRI05Z70Dq-YjtyPWjGQ_Ibqsqaof4o9YJdeY0FEuEuT9pRSO45T4qvtFZXG8/s320/addingFiles.jpg" /></a><br />
<br />
The final result then looks something like this:<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjTZvGhkJeY9LIWy40TIG4gsBTUv-sUXlB1IqBypG9VXZdpZg7tPA-8BEuTyeBNsKSPr2S4Q54feRNY5x8fqtmv6S_xsio2qcPPXzbVNj-6E96bBfvpCEE1yB32P3G8Dez6-wEk7BCHeVI/s1600/finalResult.jpg" imageanchor="1" ><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjTZvGhkJeY9LIWy40TIG4gsBTUv-sUXlB1IqBypG9VXZdpZg7tPA-8BEuTyeBNsKSPr2S4Q54feRNY5x8fqtmv6S_xsio2qcPPXzbVNj-6E96bBfvpCEE1yB32P3G8Dez6-wEk7BCHeVI/s320/finalResult.jpg" /></a><br />
Korbininhttp://www.blogger.com/profile/00742811285841484666noreply@blogger.com7tag:blogger.com,1999:blog-2970661570838560548.post-17655275121523289812012-02-05T12:23:00.001+01:002012-02-05T12:23:53.685+01:00SPE versionsIf you use SPE version 0.8.4 instead of version 0.8.3 none of the earlier mentioned problems occur. It is a bit strange that if you go to the sourceforge download site for SPE you get 0.8.3 as the recommended version. If you download from the <a href="http://developer.berlios.de/project/showfiles.php?group_id=4161">archive link</a> on the <a href="http://pythonide.blogspot.com/">SPE blog</a> you can download version 0.8.4.Korbininhttp://www.blogger.com/profile/00742811285841484666noreply@blogger.com0