Comments for shocksolution.com: scientific computing, modeling, and simulation

Comment on Profiling memory usage of Python code by Storing large Numpy arrays on disk: Python Pickle vs. HDF5 | shocksolution.com: scientific computing, modeling, and simulation

Sun, 10 Jan 2010 23:01:08 +0000

[...] Pickle module is fast and convenient for storing all sorts of data on disk. More recently, I showed how to profile the memory usage of Python code. In recent weeks, I’ve uncovered a serious limitation in the Pickle module when storing [...]

Comment on Python Pickle: Painless binary storage for Python objects by Storing large Numpy arrays on disk: Python Pickle vs. HDF5adsf | shocksolution.com: scientific computing, modeling, and simulation

Sun, 10 Jan 2010 23:00:23 +0000

[...] In a previous post, I described how Python’s Pickle module is fast and convenient for storing all sorts of data on disk. More recently, I showed how to profile the memory usage of Python code. In recent weeks, I’ve uncovered a serious limitation in the Pickle module when storing large amounts of data: Pickle requires a large amount of memory to save a data structure to disk. Fortunately, there is an open standard called HDF, which defines a binary file format that is designed to efficiently store large scientific data sets. I will demonstrate both approaches, and profile them to see how much memory is required. I am writing the HDF file using the PyTables interface. Here’s the little test program I’ve been using: [...]

Comment on LDI Report::Part 1::Eye Candy by LDI 2009 Report | shocksolution.com: scientific computing, modeling, and simulation

Mon, 23 Nov 2009 03:15:57 +0000

[...] LDI (Live Design International) last Friday in Orlando, FL. This was my second time at the show (read my report from 2007). Overall, I’d say this year’s show was scaled back from 2007, which is not [...]

Comment on Reading Labview binary files with Python by Primera lista de direcciones » Un atoq en el aula

Primera lista de direcciones » Un atoq en el aula — Sat, 21 Nov 2009 21:03:37 +0000

[...] 12-Post sobre lectura de archivos binarios Labview con Python. Página web: http://www.shocksolution.com/2008/06/25/reading-labview-binary-files-with-python/ [...]

Comment on Reading Labview binary files with Python by craig

craig — Tue, 04 Aug 2009 19:37:16 +0000

Hey Travis–thanks for reading, and thanks for Numpy! I will have to look into your suggestions and write an updated post, since this page seems to be quite popular. We are no longer using Labview binaries for our data storage, since we have figured out how to save the data in HDF format. The HDF files tend to be somewhat larger, but disk space is cheap and the self-documenting organization of the HDF files is very useful.

Comment on 3D Plotting Software for Python::Part 1::PyX by craig

craig — Tue, 04 Aug 2009 19:31:15 +0000

Thanks, Anton! I’m glad to hear that you liked it.

Francois–I was actually planning to do a “part 2″ of this review covering Asymptote when this site went down due to a server crash. I have hacked together a process to run Asymptote from Python so that I don’t have to manually go through the process of generating a data file and then running Asymptote. I will write a new post on this as soon as I get some time.

Comment on Profiling memory usage of Python code by Jonas Wallin

Jonas Wallin — Mon, 03 Aug 2009 13:24:42 +0000

I am working on a multigrid course using python.
Where memory profiling is really important.
And I had exactly the same problem when I tried to use heapy.
This post helped me a lot.
Thanks

Comment on Reading Labview binary files with Python by Travis Oliphant

Travis Oliphant — Thu, 16 Jul 2009 02:59:54 +0000

Thanks for sharing your experience. If you are dealing with LabView data you should really be using NumPy which is the standard N-dimensional array module for Python.

With NumPy you can use it’s excellent facilities for reading binary data into efficient memory structures that can be easily accessed using Python syntax:

Something like

data = numpy.fromfile(”Measurement_4.bin”, dtype=”>d”)

will give you the data in a NumPy array. This array can be plotted or visualized using any number of tools (Chaco, matplotlib, etc.).

Try out a distribution of Python like the Enthought Python Distribution to get the complete assortment of Python tools that let you interact easily with binary data no matter what the source.

You may also be interested in using Memory mapped files to access portions of a file very quickly. See the corresponding public webinar on this page for details.

http://www.enthought.com/trainin /SCPwebinar.php

Comment on 3D Plotting Software for Python::Part 1::PyX by Francois

Francois — Sun, 12 Jul 2009 13:37:01 +0000

You can look at a much better solution IMHO: Asymptote.

I may confess that it’s not in Python but an amazing solution for 3D with animation and 3D pdf generation have a look at Asymptote:

http://asymptote.sourceforge.net/gallery/

Comment on The Python configparser: a way to read simple data files by Anton

Anton — Fri, 17 Apr 2009 09:40:16 +0000

nice work!