shocksolution.com

Reading Labview binary files with Python

My research group uses Labview 7.1 to write custom data acquisition (DAQ) software. I code everything else in Python, so I need to get data from Labview into Python for processing. Our DAQ program produces Labview binary files, so I had to find a way to read them with Python. Binary files are nice because they are a compact way to store numerical data as compared to ASCI or (heaven forbid) XML, but they are much harder to read. The binary format used by Labview is documented only indirectly, so I had to hack a little.

The first thing to realize is that the Labview binary file is a direct dump of the data that was stored in RAM. How Labview stores data in memory is documented here. Indirectly, this documents how binary files are stored on disk. Our DAQ program writes a rather complex “cluster” (Labview’s version of a C structure) to disk. The elements of the cluster are stored contiguously as a sequence of bytes, and there’s no way to know which byte goes with which element, unless you know the size of each element and the order in which they are stored in the cluster. So, the first step is to document the cluster that’s being written to disk. You can use the context help in Labview to view the data type of the wire that leads to the VI that writes the file. With this in hand, you are ready to write Python code.

First, make sure you open the file in binary mode:

binaryFile = open("Measurement_4.bin", mode='rb')

Now you can use the file.read(n) command to read n bytes of data from the file. This data is read as a string, so you will need the struct module to interpret the string as packed binary data. This is what it looks like:

 (data.offset,) = struct.unpack('>d', binaryFile.read(8))

The first argument to struct is a format string. The ‘>’ tells struct to interpret the data as big-endian (the default for Labview) and the ‘d’ tells it to produce a data type of double. The data type has to match the number of bytes read in by file.read(). Remember that we’re reading the file sequentially, and there are no delimiters, so if you read in the wrong number of bytes for any data, all the subsequent data will be junk.

One final note about arrays: arrays are represented by a 32-bit dimension, followed by the data. If the array contains no elements, it is stored as 32 zero bits with no other data. It would probably be a good idea to use the Python array module if you need to read in large arrays efficiently.

Tags: binary, data, file, labview, python

Server move completed and general update

The site is back online after a server move.  Actually I can’t blame the server move for the downtime, because I went out of town and then had a lot of catching up to do, and as a result I didn’t switch the domain to point to the new server.

I haven’t been blogging much because I haven’t done much software development, Linux admin, or lighting design lately.  I have been busy in the lab at work and I’ve been supervising a trainee lighting operator instead of running my own shows.  We haven’t been moving forward with plans to purchase a new lighting control console, so no update on that, either.  Hopefully, I’ll have something interesting to post soon.

Presentation at BarCamp Orlando 2008

I gave a presentation at BarCamp Orlando on 5 April 2008. Here is a link to my presentation, Data Analysis with Python.

Connecting to a Windows file server from a Linux box

I was surprised how hard it was to find out how to mount a Windows file server as a drive on a Linux box. Fortunately, once I found out how, it was really pretty easy. First of all, you DO NOT need Samba. Most of the information I found on the web involved installing Samba and using your Linux box as a file server for Windows machines. If you want the Linux machine to be a client to a Windows server, Samba is not required. You need to enable support for the cifs filesystem in your kernel–in fact, it’s probably already compiled in (cifs is just a “rebranding” of smb). Then install the mount-cifs utility and you’re ready to go. Here is a command line you can use to mount a shared partition on a Windows server:
mount -t cifs //xx.xx.xx.xx/sharename /mnt/server -v -o user=myusername,prefixpath="myhomedirectory",noperm

It will prompt you for your password.  You might need to tweak the options–for example, the noperm option prevents Linux from trying to map Windows permissions to Linux permissions. This is necessary on my network to enable Linux to copy nested directories to the Windows share. Once you get it working, it can be added to fstab like any other file system.
//xx.xx.xx.xx/sharename /mnt/server cifs user,credentials=/etc/cifs_credentials,noperm 0 0

The cifs_credentials file can have any file name, in any location you choose, with the format described in the man page for mount.cifs.   Since this file will contain your network login and password, make sure it can be read only by root or the user who owns those credentials.

For reference: mount-cifs thread on the Ubuntu forums

Found a bunch of templates

I use XFCE as my window manager, with Thunar as my file manager.  Thunar has always had this “Create Document -> from template” entry in the File menu, but it didn’t come with any templates.  I finally went looking for some templates, and found a good collection at stalefries.  Just unpack the archive in a directory called “Templates” in your home directory.

BTW, isn’t it about time I created a “linux” category instead of trying to stuff my Linux entries under the “software development” category?

Cheap LED fixtures

Normally, we stay far away from anything marketed to DJ’s, but we’ve been experimenting with cheap LED fixtures as an inexpensive way to put more colors on stage.  LED fixtures have a few advantages: they efficiently produce light without much heat, they don’t require a dimmer, and they can produce multiple colors without gel changes.  They also have some disadvantages.  Color mixing is produced by combining LED’s of different colors (RGB) at different intensities, so the color is only “mixed” at the surface it is illuminating.  It can be difficult to produce white light that matches light from other fixtures.  LED’s cannot be dimmed by lowering the voltage, like a conventional light.  Instead the LED is strobed (pulsed) at a rate too fast for your eye to discern.  Unfortunately, the Chauvet ColorPalette fixtures shown below use a pulse rate that is slow enough that it can be picked up on a video camera.  When the fixtures illuminate a set piece like this one, which is in the background of many IMAG shots, the flicker is very distracting if the intensity is run at anything less than 100%

Practically speaking, when a Color Palette is visible on camera,we can only get seven full-intensity colors: red, blue, green, white, purple, blue-green, and yellow.   We’ve since moved them out into the auditorium where they are used to illuminate the walls.  We did find one really cool application for these lights.  We have a number of deep-purple, “blacklight” Color Palettes mounted backstage to enable people to move in the dark.  They stay on all the time, draw very little power, and are totally invisible to the audience, even during a dead blackout.

This is a Chauvet COLORado 1, which is meant to replace a small par can. It’s mounted on a pipe above and behind the musicians and lights them from behind.  This is the view you get from the front row in the audience.  See the red, blue, and green LEDs?  That’s producing white light, but you wouldn’t know it from this angle.  That’s what happens when the colors don’t mix at the light.  We don’t have the strobing problem on video with the COLORado’s, but I don’t know if that’s due to their positioning or their design.

For more information, see the LED lighting reviews by onstagelighting.co.uk.  There are also pro-grade LED fixtures that are much better…but I haven’t used them.

Cool set design: Part II

Here’s another cool set design.  What’s interesting about these two pictures is that the set is illuminated using the same lights at two different intensity levels, but the color seems different.  That’s a neat feature of incandescent lights–at low intensity you can get deep, saturated colors.

Here is a look in another color.

I didn’t think to take close-up pictures of this set.  It turns out that our set designer likes to have these pictures, so now I’m trying to do a more thorough job.   This one is constructed of a wooden frame with a light colored wrinkly fabric stapled to it.  Like I said, I don’t design or build these things, but if you want to know the details, I’m sure I can find out for you.

Tags: "discovery church" stage set lighting design

Cool set design: part 1

After a longer-than-expected holiday break, I’m back with a slightly different topic: set design and construction. Someone else designs and builds our sets, but I get to light them, and I’m usually tasked with documenting them. Our lighting console evaluations have been on hold for the holidays, but I plan to get back to it later this month.

The set I’m writing about today is one of the more exotic ones we’ve tried. Here’s how it looked when lit

Pieces were cut from white Styrofoam and stitched together to make a wall of crazy curves:

The set was lit from below with cheap LED panels. The black boxes in front of the fixtures are cinder blocks that are basically sitting there to keep people from stepping on or kicking the fixtures as they enter and exit the stage.

These LED fixtures are handy, but they have their drawbacks. That will be the subject of another post.

ChamSys MagicQ console review::part I

I don’t know how to pronounce it, but I like it!  This review is based on my impressions gathered from a few minutes of hands-on experience at LDI 2007, and the MagicQ PC software that I downloaded from ChamSys. I plan to review a lot of consoles by downloading the offline editors, and hopefully we will get demo units of our favorites.

Installation:  Download from the link above.  I’m thrilled that it runs natively on Linux.  I made a directory /opt/magicq/  and uncompressed the tarball.  I had to install libftdi as indicated in the instructions (version 0.7 works fine).  In order to run the software as a non-root user, I had to give write permission on the usb device, even though I’m not using any wings.  That was the only hitch.

Programming: The MagicQ has a programmer and a playback section–the playback section can be playing cues from lists, while the user is manually setting values in the programmer.  This is similar to the Hog, but different from the ETC Congo.  Like the Hog, there are multiple playback faders, each with a cue list, and the console mixes the outputs using HTP or LTP (highest- or latest-takes-precedence).

The MagicQ assigns HTP or LTP per DMX channel.  Conventional lights are usually run HTP, while movers are run LTP.  This overcomes one of the worst features of the Hog–you can only choose HTP or LTP for an entire cue list, meaning that it’s hard to mix conventional lights and moving lights in the same cue list.

Neat feature: the manual is available on the console by pressing the Help button.

Next time, I will try patching some fixtures and running the demo show.

LDI Report::Part 3::Midrange Lighting Control Consoles

My real mission at LDI was to review lighting consoles–management is considering replacing our Hog 1000 in the next year or so.  The Hog is serving our needs well, but we’ve already used up both DMX outputs, and Hog programming is not the easiest thing to teach to new lighting operators.  So, we’re looking at consoles that match or exceed the capabilities of the Hog 1000, with 4 DMX outputs and hopefully with a more user-friendly design.  We’re keeping the price below $30,000–hopefully well below!  So, without further ado, here are the consoles I got to see in person. Please keep in mind that these are first impressions, and I really need to download the offline emulators before I say anything more.

The Avolites Pearl Expert and Pearl 2008.  The main thing that caught my eye with this console is the fader paging system.  There’s basically a big drum embedded in the board above the playback faders.  You use console tape on the drum to label the faders.  To get to a new bank of faders, roll the drum one click and bring up a new set of labels.  It’s intuitively obvious when you play with it.

The MagicQ is a module console from a new company called ChamSys.  It’s being distributed in the US by PRG.  Some of the programmers previously worked on the Hog 2 and 3, so this board runs much like a Hog 2.  It might be described as “what the Hog 3 should have been.”  Very programmable–and Linux-based!

These are Vector consoles from Compulite.  They seem to be reasonably priced and extremely programmable, but retain some of the simplicity of a theater-style console.  I got to talk to a software developer on the project, and although the consoles run a Windows XP operating system, they have taken great pains to maintain realtime hardware response.

The Congo and Congo Jr. were originally made by Avab, and somehow acquired by ETC.  Compared to other brands, I was most impressed with the user interface of the Congo.  It’s designed like an ETC theater console, but integrates moving light support seamlessly.  There is no separate programmer–you are always working within a cue.  It’s probably too early to say this, but I think this would be the easiest one to teach to a beginner.  Unfortunately, it seems to be one of the more expensive consoles in this lineup.

Here is the LeapFrog from zero88.   I didn’t get much chance to use it, but with only 2 built-in DMX outputs, I’m not sure this is going to have the horsepower we need.

The Frog 2 seems to be very much like the Hog 1000 or Hog 2 in terms of its programming style.  I don’t think it adds much value when compared to the Hog 1000.

Overall, I was impressed with the consoles I saw at LDI.  My impression is that there are many powerful consoles to choose from, but many of them are too programmable for their own good.  I love programming features, but that’s because I’m a professional programmer who happened to learn lighting.  What about the theater lighting person who needs to run moving lights?  Is it really a good thing that bump button #7 can be programmed to shut off the console?  On another note, a lot of people were very helpful when showing me their consoles.  However, they would have been a lot more effective if they had just shut up and listened to my needs for a minute before going off and describing all kinds of features I don’t care about.  Stay tuned for more in-depth reviews!