Category Archives: Python

The Python programming language.

pickle, hickle and HDF5

Danny Price recently left a comment to let me know about a new Python package he’s developing called hickle. The goal of “hickle” is to create a module that works like Python’s pickle module but stores its data in the HDF5 binary file format. This is a promising approach, because I advocate storing binary data in HDF5 files whenever possible instead of creating yet another one-off binary file format that nobody will be able to read in ten years. The immediate advantage of using HDF5 to store picked Python objects is that HDF5 files are portable across many platforms, while “pickled” objects may not be readable on a different platform. Continue reading

Building NumPy and SciPy with Intel Composer 2013 and the MKL

Since Python is widely used as a high-productivity language for scientific computing, Intel has created a page showing how to build NumPy with Intel compilers and the Math Kernel Library (MKL). I would like to clarify a few items regarding building NumPy on a 64-bit Red Hat Enterprise Linux 5.4 system. Since this is a production system, I don’t want to replace the Python 2.4 binary -2.7.3-intel-composer-2013that ships with RHEL 5.4. Instead, I created a directory called

/apps/python/python/-2.7.3-intel-composer-2013

and set

PYTHONPATH=/apps/python/python-2.7.3-intel-composer-2013
LD_LIBRARY_PATH=/apps/python/python-2.7.3-intel-composer-2013/lib:$LD_LIBRARY_PATH

Continue reading

Tricks for Writing XML with Python 3

I’ve added a Python 3 XML example to my Shocksolution_Examples repo on GitHub.  The new example shows how to generate an XML file which functions as a template for building a GUI with wxGlade.  However, this example should be helpful for anyone who needs to create XML files with Python.  The full example is on GitHub, so I’m just going to highlight a few interesting snippets.
Continue reading

Building SciPy with Intel compilers and MKL on 64bit RHEL 5

This is a follow-up to my earlier post about building NumPy with Intel compilers and the Intel MKL on CentOS 5. I will now explain how to build SciPy (which requires NumPy).

First, download and unpack the SciPy source tarball. The following command can be used to build SciPy:

LDFLAGS="" FFLAGS="-fPIC -openmp" python2.7 setup.py build --fcompiler=intelem &> build.out

I set the LDFLAGS variable to an empty string to avoid using any LDFLAGS which are defined in my .bashrc. The -fPIC tells the compiler to create position-independent code, although this flag may not be required for newer compilers (ours are quote old: ifort (IFORT) 10.1 20070913 and icc (ICC) 10.1 20070913). I used the -openmp flag to be consistent with the flags I used for building NumPy. The option –fcompiler=intelem forces the Intel Fortran compiler to be used instead of g77. I redirected all output to a file called build.out, which is helpful for finding errors and warnings in the build output.

Once you have built SciPy successfully, install it with the following command:

python2.7 setup.py install --prefix=/apps/Python/Python-2.7.3/

Because I work on a Linux cluster that is shared by many people, I install all Python packages into the /apps directory. Only official RedHat RPMS get installed into standard system paths, maximizing the stability of the cluster. Anyone who wants to use newer versions of tools (like Python 2.7) has to specifically add the correct location to their paths.

Building NumPy on a 64-bit Red Hat Cluster with Intel MKL

In a previous post I described how to build an optimized version of NumPy using the Intel compilers and Math Kernel Library (MKL).  This post will update those instructions, since it has been a few years and I am now using NumPy 1.6.1, MKL 10.0.1.014 and Red Hat 4.1.1 (kernel 2.6.18-8.el5).

First, copy the file site.cfg.example to site.cfg and open it in an editor like vim. The commented lines in the file give some explanation of how it works. You only need to add the following lines:

[mkl]
mkl_libs =  mkl_intel_lp64, mkl_intel_thread, mkl_core, guide
lapack_libs = mkl_lapack
include_dirs = /opt/intel/mkl/10.0.1.014/include
library_dirs = /opt/intel/mkl/10.0.1.014/lib/em64t

Notes:

Continue reading

Python string format examples

The format method for Python strings (introduced in 2.6) is very flexible and powerful.  It’s also easy to use, but the documentation is not very clear.  It all makes sense with a few examples.  I’ll start with one and add more as I have time:

Formatting a floating-point number

"{0:.4f}".format(0.1234567890)
"{0:.4f}".format(10.1234567890)

The result is the following string:

'0.1235'
'10.1235'

Braces { } are used to enclose the “replacement field”
0 indicates the first argument to method format
: indicates the start of the format specifier
.4 indicates four decimal places
f indicates a floating-point number

Align floating-point numbers at the decimal point

There is no format specifier that aligns numbers at the decimal point. Decimal-point alignment is accomplished by fixing the field width, which is set by placing a number before the decimal point in the format string. Compare the previous example to the output shown below:

"{0:8.4f}".format(0.1234567890)
"{0:8.4f}".format(10.987654321)

Result:

'  0.9877'
' 10.9877'

Scientific Notation

"{0:.4e}".format(0.1234567890)

Output:

'1.2346e-01'

Multiple Arguments

In Python 2.6 you can include multiple arguments like this:

"sin({0:.4f}) = {1:.4e}".format(0.1234567890, sin(0.123456789))
'sin(0.1235) = 1.2314e-01'

In Python 2.7 and later, you may omit the first integer from each replacement field, and the arguments to format will be taken in order:

"sin({:.4f}) = {:.4e}".format(0.1234567890, sin(0.123456789))
'sin(0.1235) = 1.2314e-01'

Leave space for a minus sign

"{: .4e}".format(0.098765)
"{: .4e}".format(-0.1234567)

Output:

 9.8765e-02
-1.2346e-01

Note that there is a space between the colon and the dot in the format specifier. The decimal points for the two numbers are aligned in the output, which is handy for printing tabular data.

How to build ScipPy with Python 2.7.2 on CentOS5

EDIT: added –enable-shared to configure script for building Python, and added /home/yourname/lib to shared library path.  This is necessary for building some packages such as pycairo (which you may need to build pygtk and matplotlib).

EDIT 2: you should use the –prefix=/home/yourusername instead of –user.  The prefix option places packages in the standard location: /home/yourusername/lib/python2.7/site-packages.  The –user option places the packages in /home/yourusername/.local/lib/python2.7/site-packages which I think is screwed up!

 

I use CentoOS5 because I want enterprise-class stability, as well as binary compatibility with a commercial application that is built for RHEL5.  I need to use some “bleeding edge” packages, such as the latest version of SciPy, but I don’t want to affect the base stability of the system.  Here is how I did it.  First, with superuser privileges, use yum to install the following packages.  You may need to set up epel as an alternate repository:

Continue reading

Python: lists to tuples and tuples to lists

Get the code for this example

The zip function in Python accepts a number of sequences (such as lists or arrays) and returns a list of tuples.  The first element of each tuple comes from the first sequence, the second from the second sequence, and so on.  For example:

a = range(0, 5)
b = range(5, 10)
c = range(10, 15)
sequence_of_tuples = zip(a, b, c)
print(sequence_of_tuples)
[(0, 5, 10), (1, 6, 11), (2, 7, 12), (3, 8, 13), (4, 9, 14)]

I often use zip when I have a list of x coordinates and a list of y coordinates, and I need to pass a list of (x,y) pairs to a function.
The question is: what is the opposite of zip? How do you convert a sequence of tuples to multiple sequences? It turns out that zip is its own inverse, due to an obscure feature of the Python language called arbitrary argument lists.
Continue reading

A self-contained Fortran linear equation solver

I’ve just released a self-contained Fortran module that solves a system of linear equations using the LU decomposition.

Download the Fortran linear solver from github

This module is based on code that was implemented and released on the Web by Jean-Pierre Moreau.  His implementation was based on one of the Numerical Recipes books.  I updated his code to a more strict Fortran 90 standard and added the necessary comments so that it can be built as a Python module using f2py.  I replaced Jean-Pierre’s Fortran test program with a simpler, self-contained program.  I also included a Python script that implements the same test case.

I created this module because sometimes a self-contained routine is more appropriate than a full library.  I am compiling a library that implements a custom boundary condition for a proprietary computational fluid dynamics solver (CFD-ACE+).  The library has to be written in Fortran, and it has to be built using a proprietary set of build scripts.  I could either try to reverse-engineer the build process and to modify it to link to a shared library like LAPACK, or I could implement a self-contained solver.  Since Pierre had already implemented the solver, I was able to slightly modify his code and get it working relatively quickly.