ABSTRACT

 

Author:  Edward Zhu

Partner: Hitesh Mehta

Title:  Automated Image Acquisition and Processing

Supervising Professor:  Dr. Dean Neikirk

 

This project attempts to solve the problem of automatically and accurately extract red-green-blue (RGB) values from pre-defined regions in a computer image.  However, a series of steps – such as acquire the image from the hardware, process the image to compensate for background noise, and isolate the appropriate regions inside the image – must occur before the extraction of RGB values from the image.  Thus, this project involves both hardware and software aspects: selecting the appropriate hardware and creating a set of software that will both interact with the hardware and the user to produce the final RGB values.  The hardware setup must be relatively inexpensive while able to produce consistent image outputs.  The software package, on the other hand, must be user friendly, accurate, versatile, and adaptable by future programmers.  In summary, this project involves the selection and adaptation of image acquisition hardware and the construction of a set of software tools that would allow for fast acquisition of images, easy input of user-defined processing parameters, and accurate extraction of RGB values from different images at real-time or close to real-time performance.

Introduction

The University of Texas Electrical and Chemical Engineering Departments are jointly developing a sensor (nicknamed the Electronic Tongue) that can detect the presence of targeted chemicals.  Early experiments have demonstrated the proof-of-concept for this project; however, the technique employed to produce data from the sensor is crude and time-consuming.  The goal of this project is to automate and simplify the process for collecting and analyzing data.  To achieve this end, this project will employ both hardware and software.  Inexpensive commercial imaging hardware will capture and collect the image data; custom written software will interface with the hardware and perform the necessary calculation on the captured images.

 

BACKGROUND

This project is an extension of a collaborative venture by several University of Texas professors to develop electronic chemical sensors.  The electronic chemical-sensing device employs special micro-machined chemical sensing beads that have the special property of producing strong and reversible changes in color when exposed to different types of chemicals.  Thus, by recording the color pattern of a set of beads in the presence of a known chemical, that chemical could be identified later by matching the color pattern of the beads with the pre-recorded pattern.  Currently, the chemical sensing beads are placed in a silicon substrate that contains “micro-test-tubes” arranged in a grid configuration.  The silicon substrate, along with state-of-the-art micro-machine technology, allows a large array of chemical sensing beads to be placed in an area no more than the tip of a finger.  Thus, various chemicals could be detected simultaneously by recording and then carefully comparing the color of the chemical sensing beads with a database of pre-recorded patterns, although this process could quickly become tedious and tiresome if performed by a human.

 

Currently, the apparatus is set up such that an optical microscope observes the chemical reactions in the tiny micro-machined sensor.  The microscope has a video camera attached to it that records the chemical reactions and streams uncompressed video data into a computer at approximately 30 frames a second.  Using a commercial software package, a researcher would record the reaction for several seconds and save the recorded data in video for Windows (AVI) format.  The researcher would then open the recorded video data and select the best frame to perform the RGB-value extraction operation.  With the appropriate frame selected, the researcher would manually isolate regions in the image for RGB extraction.  The RGB values extracted would then be processed and displayed by some other commercial software package such as Microsoft Excel, and the researcher would try to match the displayed data with some pre-recorded data in order to determine the chemical or biological agent detected by the sensor.  Clearly, this process is inefficient, time-consuming, and prone to errors in human judgement.  The goal of this project is to produce a computer solution that can automatically initiate image transfer operations from the video source to the computer, isolate cell regions in the captured images, and finally extract RGB values from the cell regions.  Future projects would create software packages that can record and compare color patterns generated by the software for this project, thus completely automate the decision making process and creating a true “electronic tongue.”

 

Hardware

An integral part of the computer solution for the electronic chemical-sensing device is an inexpensive hardware setup that can accurately capture the reaction inside the chemical sensor.  As figure 1 shows, the hardware setup for the entire project is relatively simply, consisting of four components: the sensor array, the video source, the image grabber, and the target system.  As mentioned earlier, the sensor array is a silicon substrate with embedded chemical sensing beads, and is still under development.  The other three components for this project are all available commercially.  Indeed, there exist a considerable variety of products to choose from for each of the three components.  This section will make recommendations for each component.

 

 

 

 

 

 

 

 

 

 

 


Figure 1: The hardware setup for the electronic chemical sensor project.

 

Video Source

For this project, a color charge-coupled-device (CCD) camera will provide the best video source.  CCD cameras produce relatively high-resolution video signals and are capable of refreshing much quicker than the image grabber hardware.  Moreover, a CCD is a physically small device, which allows it to be placed very near the sensor array.

 

Image Grabber

An image grabber is a device that converts an analog video signal to a digital format.  Many such devices are available today for the personal computer.  Image grabbers vary in capability but can be compared according to common feature parameters such as frames per second (FPS), pixel resolution, and color depth.  Another common feature for the image grabber is the ability to control it through an industry standard software interface such as Multimedia Control Interface (MCI), Video for Window (VFW), or Windows Driver Model (WDM).  The software for this project will use the MCI interface to manipulate the image grabber and capture images.

 

For this project, we chose the Hauppauge Win/TV tuner card as the image-capture device.  The Win/TV card is TV tuner device for the personal computer with frame-capture capabilities.  The card’s capture rate is slower than most digital image-capture cards, but is more than enough for this project.  The card is also inexpensive and widely available.  The major features of this card include free support for the Windows MCI control interface and frame capture at 640x480 pixel resolution with 24-bit color depth.  A complete specification of this product is located at http://www.hauppauge.com/html/wc_data.htm.

 

Hauppauge periodically releases new drivers for the Win/TV product.  This project uses version 2.0c of the VFW drivers for the Win/TV card, and does not support the WDM drivers.  A list of the latest drivers can be found at http://www.hauppauge.com/html/sw_wcst.htm.  Driver installation notes for Window 95 is located at http://www.hauppauge.com/html/osr2.htm. 

 

Target System

The target system is responsible for hosting the Win/TV capture card and the custom software that controls the card.  The recommended operating system for this project is Microsoft’s Windows 95 OEM Service Release 2 (OSR2).  Microsoft has announced diminishing support for the MCI interface, in lieu of WDM, thus some operating systems recently released by Microsoft do not contain MCI device drivers.  This project uses MCI device drivers to interface with the image-capture hardware; thus Windows 98 (and later operating systems) lacks features necessary for this project.  We recommend a Pentium P166, 32MB of memory, 2GB hard drive, and Windows 95 OSR2 operating system.  The Hauppauge Win/TV card also requires a video card that supports overlay.  Hauppauge maintains a list of compatible video chipsets at http://www.hauppauge.com/html/techfaq.htm#VGACOMPAT.  

 

Software

The software for this project is concerned with the first part of the automated chemical detection process, namely image acquisition and RGB-value extraction.  The software indirectly communicates with the image capturing hardware through the Windows MCI device drivers, and requests the hardware to transfer video frames into the hard drive in Windows-bitmap image format.  Once the image capturing operation is complete, the software then loads the bitmap image into the main memory, and performs RGB-value extraction functions.  To increase performance, the software uses a pre-fabricated image mask for RGB-value extraction.  An image mask is a list of cells, with each cell containing a list of pixel locations representing the image region associated with that cell.  The image mask is also static, remaining constant throughout the RGB-value extraction process.  Therefore, it is essential to create the most accurate image mask possible, as many images will use the same image mask for RGB-value extraction.  The extracted RGB values can be saved as text files, or automatically opened and graphed in Microsoft Excel.

 

The original intention of the project was to combine the image acquisition stage with the RGB-value extraction stage.  Instead of transfer video frames into the hard drive, the image acquisition hardware would directly transfer images into the main memory.  Once the images are in the main memory, the computer could apply the mask operation and extract RGB values into the hard drive.  This approach will drastically increase the performance of the software, as the two unnecessary image transfer operations (both to and from the hard drive) will be eliminated.  However, this approach requires manufacturer-supplied hardware drivers that were unavailable from Hauppauge.

 

Java and Classes

This project uses Java as the main development language in order to decrease software complexity and increase software manageability.  Java is an object-oriented computer programming language, which means basic building blocks of the software are classes.  In essence, a class is an entity that accepts some data, manipulates the data with internal methods, and then outputs the data in some fashion.  In this project, some classes also contain methods that allow the software to graphically communicate with the user.  When combined together, these classes make up the graphical user interface (GUI), which greatly enhances the software’s usability.  Figure 2 shows all of the classes in the software, and illustrates the interactions between the software and the hardware components.

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 


Figure 2: Flow chart of the interactions between software and hardware components.

Child Windows and the Root Window

The classes that make up the GUI present themselves as windows to the user.  A window is a graphical unit that displays some information, accepts user inputs, and performs specific actions based on user inputs.  There are two types of windows in the software for this project: child window and root window.  When the program starts, it creates a root window that acts as the parent to all child windows.  The root window constructs a specific child window based on user inputs, and destroys a child-window when prompted by the user.  Although not very interesting by itself, the root window is essential for the proper operation of all child-windows.  The root window provides the vital link between the separate child windows, which otherwise would not know about the existence of each other.  Figure 3 shows the root window controlling a child window.  Currently, there are five different types of child windows that allow direct user control: project file window, browse window, control window, image window, and mask window.

Figure 3: The root window and a child window.

The Project File Window and the Browse Window

The project file window is the first child window that interacts with the user.  Through the project file window, the user can create a project and specify some information about the project.  A project is a session that contains mundane but essential information such as the data output directory, the desktop configuration, and the current setting of the image-capture hardware.  A project also allows the user to easily manage the multitude of output files generated by the software.  As figure 2 shows, the project file window is the only child window that has access to the computer’s storage device.  Thus, it is responsible to save and retrieve any data generated by the program that require permanent storage.  Furthermore, the project window can save and retrieve project sessions to and from the hard drive, allowing the user to work on the same session at different times.  The project file window also controls the browse window, which allows the user to browse local hard drives and specify the output directory.  Figure 4 shows a project file widow and a browse window waiting for user inputs.

 

Figure 4: A project file window (left), and a browse window (right).

The Control Window

The control window is the main information conduit for all child windows.  As figure 2 shows, it has direct access to three other child windows, and the three child windows communicate with each other through the control window.  As figure 5 illustrates, the control window presents three different internal panels to the user, and each panel contains an array of control fields that the user can access and modify.

Figure 5: The three panels of the control window.

 

The first panel, called the device panel, is responsible for displaying and manipulating the current hardware settings.  Through the device panel, the user can alter the hardware settings, choosing from three different input sources and adjusting the parameters associated with the image capturing hardware.  The device panel does not perform any real work itself.  Instead, it accepts user inputs and passes the information to the device-interface class.  The device-interface class calls methods within the device driver that perform the actual work of selecting the input source and altering the hardware parameters.  The second panel within the command window is the mode panel, which contains essential information about the rate, the duration, and the type of image-capture that the user wishes to perform.  The mode panel saves this information internally, and later passes the information to the third panel when the user initiates the automated image capture and analysis process.  The third panel is the control panel, containing important features such as managing the automated image capture and analysis process, retrieving and saving image masks, and calling the appropriate child window to display captured images.  The control panel also establishes the communication channel between the project file window, the image window, and the device interface.

 

The Image Window

The image window displays images captured from the hardware, and allows the user to generate image masks used in the RGB extraction process.  The image window also contains some primitive image editing and selection tools.  The image selection tools also allow the user to create a custom image mask by adding or deleting image regions to or from the image mask.  Figure 6 shows an image window displaying a digitally captured image of the sensor array.

Figure 6: An image window displaying a digitally captured image of the sensor array.

Like the control window, the image window presents itself to the user with three separate internal panels: the image panel, the mask panel, and the grid panel.  The image panel is responsible for decoding and displaying the image file it receives from the command window.  Currently, the software can decode three types of image format: Joint Photographic Experts Group (JPEG), Graphics Interchange Format (GIF), and Device Independent Bitmap (DIB) image formats.  Around the displayed image are tools that allow the user to edit and selected image regions.  Both mask and grid panels contain tools that allow the user to create and edit image masks.  However, the image window does not perform any image or mask editing and selection operations itself.  Instead, the image window delegates image editing and selection operations to the drawing-tools class, and delegates mask generation and editing operations to the master-mask class.  Image editing and selection operations are relatively simple to implement, since most of these operations are inherent in the Windows operating system.  Image mask operations, on the other hand, are comparatively more complex, since the automated mask-generation routines must be able to correctly identify and isolate cell regions inside a captured image.  Indeed, much of the programming efforts for this project involved experimenting with different algorithms that automatically generate image masks.

 

The Mask Window

In order to help improve the accuracy of the automatically generated image mask, the user can enter information about the background noise level and the configuration of the sensor array through the mask window.  The mask window also allows the user to adjust some of the parameters associated with the mask generation routines.  By fine-tuning these parameters, the computer can create an image mask that accurately reflects the positions of the sensor cells inside the sensor array.  Figure 7 shows a mask window, which, like the control window and the image window, has three internal panels.  The user does not have to enter any information in the mask window, as the computer will automatically calculate the optimal values for each field.  However, the hints provided by the user will often greatly increase the accuracy of the computer generated image masks.

  

Figure 7: The three internal panels of the mask window.

 

Device Interface and Extraction Interface

Once the computer has an image mask, it can efficiently extract RGB values from captured images.  Two classes operate in the background as the agents responsible for capture images and extract RGB values.  As figure 2 shows, the device-interface class communicates with the hardware device driver to initiate image transfers from the image-capture card to the hard drive.  Once the image arrives at the hard drive, the extraction-interface class loads the image into the system memory, and performs RGB-value extraction operations.  The user can specify the final destination of the extracted RGB values.

 

As stated earlier, the current process – transferring images from the image-capture hardware to the hard drive and then from the hard drive to the system memory – is cumbersome but necessary, due to the lack of manufacturer-supplied hardware device drivers.  However, the modular construction of the software allows modification to one class without necessarily changing the other classes.  Therefore, once the device drivers become available in the future, the existing software can easily incorporate them with minimal changes to the computer code.  This will greatly increase the software’s performance, and eliminate current restrictions on the maximum image capture rate.

 

FUTURE IMPROVEMENTS

As stated in the abstract, this project is only concerned with extracting RGB values from pre-defined regions in an image.  The image must also be relatively free of background noise, and the pre-defined regions contrast sharply with the background.  The future needs of the electronic chemical-sensor project might very well exceed the limitations stated above.  The modular software construction and the inexpensive hardware setup address the future needs by allowing easy upgrade to every component of this project.  The software also has extensive online documentations, located at http://www.ece.utexas.edu/~ezhu/AutoImage.htm.  The online documentations contain detailed explanation on every aspect of the software.  Coupled with comments inside the computer code, a programmer proficient in the Java language can quickly understand the entire software construction.

 

The software’s default data output format is comma-delimited text, and if the user desires, the software can export the data to Microsoft’s Excel spreadsheet program.  This arrangement aims at satisfying the current need to graph the data in a commercial software package and the future need to analyze the data with custom pattern-recognition software.

Conclusion

The final software and hardware package delivered by this project has achieved all of the goals listed in the original project proposal.  The software provided by this project will greatly increase the efficiency in determining the type of chemical detected by the micro-machined chemical sensor.  The hardware recommended by this project provides an economical solution to the problem of accurately observe the reaction inside the electronic chemical sensor.  Combined together, this project is the first step in achieving a fast and completely automated solution to determine chemical and biological agents.

Reference

 

  1. “WDM Video Capture Overview,” http://www.windows.com/hwdev/desinit/vidcap.htm.
  2. J. J. Lavigne, S. Savoy, M. B. Clevenger, J. E. Ritchie, B. McDoniel, S.-J. Yoo, E. V. Anslyn, o. T. McDevitt, J. B. Shear, and D. Neikirk, "Solution-Based Analysis of Multiple Analytes by a Sensor Array: Toward the Development of an "Electronic Tongue"," JACS, vol. 120, pp. 6429-6430, July 1, 1998.
  3. “AutoImage Online Help,” http://www.ece.utexas.edu/~ezhu/AutoImage.htm.