TEM vision software

Prototype using Distributed Ruby for vision-based closed-loop control

Moving the tip using "drag-and-drop" without vision feedback. The circle marks the initial position of the mouse-cursor

Here vision-based closed-loop control is used to control the position of the tip. The cross-and-circle marks the last known position of the nano-indenter. The cross marks the current nominal position

As part of the Nanorobotics project a TEM vision software was developed. The software makes use of a JEOL 3010 transmission electron microscope with a TVIPS FastScan-F114 camera which is an IIDC/DCAM-compatible firewire camera. The nano-indenter is controlled by a Nanomagnetics SPM controller (the old version of the controller can be accessed with a PCI-DIO24 card).

The software runs under GNU/Linux and it makes use of Damien Douxchamps' libdc1394 to access the camera and Warren Jasper's PCI-DIO24 driver to access the PCI-card which interfaces with the SPM controller.

The software was implemented in Ruby using Qt4-QtRuby, HornetsEye, libJIT, and a custom Ruby-extension to access the SPM controller via the PCI-DIO24 card. Distributed Ruby and multiple processes were used to work around the problem that Ruby-1.8 does not offer native threads.

The vision algorithms are configured using a separate program and the configuration is saved in a file using Ruby marshalling. A plugin-based architecture, which accepts plugins for recognition and tracking, was implemented which allows one to select and configure Normalised Cross-Correlation, Lucas-Kanade tracking, or Connected Component Analysis.

Demonstration

Videos

Demonstration of TEM vision software including telemanipulation as well as closed-loop control using machine-vision feedback (also available as DivX3 videos configuration.avi (64 MByte), closed-loop.avi (44 MByte), and interaction.avi (19 MByte))

Setup procedure

1. The vision algorithms are configured

2. The SPM axes are calibrated against the camera image. A fourth step compensates for linear drift occurring during calibration

3. Using closed-loop control the nano-indenter is moved along a linear path

Download

The software can be downloaded here: visiongui-0.2.tar.bz2

The software is implemented in Ruby. It uses HornetsEye (current development of version 0.32) and Qt4-QtRuby.

Future Work

Possible future work is

• port to Ruby-1.9 which has native threads
• integrate serial-port interface of JEOL TEM
• access USB-controls for shutter and gain of the TVIPS camera
• feature-based recognition and tracking (less sensitive to brightness changes)
• offset- and gain-compensation for camera image

See Also

• Nanorobotics
• HornetsEye
• Microscope Control Software
• Microscope Vision Software

External Links

• Hardware
  • JEOL 3010
  • TVIPS FastScan-F114 IIDC/DCAM-compatible firewire camera
  • Nanomagnetics SPM controller
  • PCI-DIO24 card
• Software
  • Ruby
  • Warren Jasper's PCI-DIO24 driver
  • Qt4-QtRuby
• Related publications
  • Jung-Me Park, C. G. Looney, Hui-Chuan Chen: Fast connected component labeling algorithm using a divide and conquer technique, 15th International Conference on Computers and their Applications, March, 2000, pp. 373-6
  • J. P. Lewis: Fast Normalized Cross-Correlation, Industrial Light & Magic
  • S. Baker, I. Matthews: Lucas-Kanade 20 Years On: A Unifying Framework, International Journal of Computer Vision, Vol. 56, No. 3, March, 2004, pp. 221-255.