My GitHub profile provides an overview of much of my recent software engineering activities. My primary working language is Python with occasional help from C/C++ to do some of the computational heavy lifting.
Most of my current programming work is done as part of the esi-neuroscience organization on GitHub. One of the larger projects I am involved in is SyNCoPy, a framework for large-scale electrophysiology data-analysis in Python. I was one of its original authors and I am still a regular contributor and the main developer of ACME, a stand-alone concurrent processing wrapper which also serves as SyNCoPy’s parallelization engine.
During my postdoc I have developed research codes for applications in computational neuroscience, stochastic dynamical systems, and general graph theory which are located in the Analytic Tools repository. The full documentation can be found here.
A collection of codes that I have written during my doctoral studies is available in my Math Imaging repo. These routines perform various image processing tasks, such as de-noising or segmentation.
Computational Working Environment
I have profound knowledge and multiple years of practical experience working with the following languages and tools:
data storage models for processing and archiving large complex datasets, such as HDF5
HPC system administration and maintenance of a variety of Linux distributions (specifically Red Hat Enterprise Linux) and integration in existing active directory setups (single-sign-on via LDAP)
setup and administration of shared parallel cluster file-systems (specifically IBM Spectrum Scale GPFS) for HPC workloads
custom tooling and administration of a tape archive system (IBM Spectrum Archive LTFS)
kernel-level virtualization technologies (such as Docker) for sandboxing applications within containers to ensure software portability
system administration and maintenance of macOS clients (specifically setup, customization and administration of munki, a managed software center for macOS)
In addition, I routinely use LaTeX as well as all common Microsoft Office components.
Building NumPy and SciPy
The following instructions explain how to set up and build NumPy and Scipy on a Unix-like system. This is of course not the only tutorial explaining this procedure. However, I found the official notes a little sparse in some (in my opinion) crucial places. This is especially the case for UMFPACK, SciPy’s default solver for sparse linear systems. UMFPACK is part of Tim Davis’ SuiteSparse and widely used in many software packages (e.g. MATLAB’s “backslash” operator relies heavily on it). However, SciPy does not require UMFPACK to be installed on your system (it will fall back to SuperLU instead, which is built auto-magically together with SciPy). If you do not care/want/need to set up UMFPACK for the use in SciPy, jump directly to Step 2. Otherwise let’s start with Step 1 (note that it is assumed that you already have BLAS and LAPACK installed on your system)
In contrast to many other packages SciPy requires to be linked against a shared library
libumfpack.so. However, SuiteSparse does only build static libraries by default. Unfortunately tweaking SuiteSparse to build shared libraries
libamd.so(both needed by SciPy) turned out to be painstakingly cumbersome. Thus it may be easiest to just replace the original
Makefileswith my tuned versions and then run the build process as explained in the following.
Download SuiteSparse from here. Extract the archive and move the generated directory to a convenient location. Let’s call this folder
$SuiteSparseDIRin the following.
UMFPACK can use METIS for fast matrix reordering. If you do not care/want/need to set up METIS just skip this step. Otherwise download METIS-4.0.1 from here. Unpack the archive and move the
Makefile.inand place it in
$SuiteSparseDIR/metis/. Fire up a terminal,
make. If all goes well proceed to the next step.
SuiteSparse_config.mkand edit the lines where the variables
INSTALL_INCLUDEare defined to your liking. Then put the file in
make libraryfollowed by
INSTALL_INCLUDEare not in your home-folder). Note that these steps assume that you did not change the directory structure of SuiteSparse - if you moved any of
CHOLMODetc. additional modifications are necessary (more details here 1).
Download NumPy and SciPy and extract both archives. Edit the file
numpy-x.y.z/site.cfgas follows. Make sure that the install location of your SuiteSparse, i.e.
INSTALL_INCLUDE, appears in the
[DEFAULT]section (if you do not care/want/need to use UMFPACK ignore this). Thus if, for instance, SuiteSparse resides in
/usr/local/includeit should be something like
[DEFAULT] library_dirs = /usr/lib:/usr/lib64:/usr/local/lib include_dirs = /usr/include:/usr/local/include
Further, make sure to specify your BLAS/LAPACK/ATLAS installations
[blas_opt] libraries = ptf77blas, ptcblas, atlas [lapack_opt] libraries = lapack, ptf77blas, ptcblas, atlas
Finally make sure that your
site.cfghas a section like
[amd] amd_libs = amd [umfpack] umfpack_libs = umfpack
Since NumPy makes heavy use of BLAS it is crucial that the same FORTRAN compiler that built BLAS is used to build NumPy (see this note in the official documentation). If your shared BLAS is
ldd /usr/lib64/libblas.soin a terminal. If
libgfortran.soappears somewhere in the output then your BLAS was most probably built with
gfortran. Thus to build NumPy
python setup.py build --fcompiler=gnu95
Conversely, if you find
ldd‘s output then
g77has been used to build your BLAS. Hence type
python setup.py build --fcompiler=gnu
Finally the command
python setup.py install --prefix=/path/to/somewhere
installs NumPy in
sudowill be necessary again if
/path/to/somewhereis not in your home directory).
If you have installed NumPy in a non-standard location you have to tell Python where to find it before you can use it. Assume you have installed NumPy in the location
/path/to/somewhereusing Python 2.7 on a 64 bit system. Then type in a bash shell
to update your Python path (csh users might want to use
setenvinstead). Then fire up Python and type
import numpy numpy.__file__
This should result in the output
Thus the just installed NumPy version has been imported. Now you may want to check the output of
numpy.show_config()to make sure that all libraries have been detected correctly. Finally
numpy.test()(which needs nose) tests the installation.
After the successful installation of NumPy building SciPy is quite straight forward. Copy
scipy-x.y.z. Make sure that your
$PYTHONPATHpoints to the just installed NumPy (explained above). Depending on your findings before build SciPy using
python setup.py build --fcompiler=XXX
and install it with
python setup.py install --prefix=/path/to/somewhere
Test the installation with
import scipy scipy.__file__
which should similarly give
scipy.show_config()to make sure that SciPy uses the right libraries. Due to some
ImportError- silencing in SciPy (compare e.g. this post) it is not too easy to see if UMFPACK is actually used by SciPy after all. Even if
scipy.show_config()proudly announces that it found
linalg.spsolvemay produce surprising results. Thus you might want to run
testumfpack.pyto check UMFPACK’s functionality. If you encounter troubles try to
import _umfpack. This should work without raising an
ImportError. Finally you can test SciPy’s functionality by typing
scipy.test()(which also needs nose)
If you are tired of updating your
$PYTHONPATHin order to be able to use your homebrewed NumPy/SciPY each time you start a new shell you may want to include the line
.bashrc-file (for bash users).
If you have any comments/questions/criticism please do not hesitate to contact me.
CHOLMODetc. are found in their respective standard locations. If you have moved these static libraries, you should let the
GNUmakefilesknow by explicitly telling the
.sobuild command where to find them. For instance, if
CHOLMODis installed in the non-standard location
/where/ever, the build command should be changed from
-L /where/ever -lcholmod. Non-standard installations of
metiscan be handled analogously. Thanks to Stefan Becuwe from the HPC core facility CalcUA at the University of Antwerp for pointing this out to me!