Jason P. Fritz
Thomas P. Way
Kenneth E. Barner
Applied Science and Engineering Laboratories
A.I. duPont Institute/University of Delaware
1600 Rockland Road Wilmington, DE 19899
Voice: (302) 651-6830 FAX: (302) 651-6895
Internet: fritz@asel.udel.edu,
way@asel.udel.edu,
barner@asel.udel.edu
The term "tactile" refers to the sense of touch, while the broader "haptic" encompasses touch as well as kinesthetic information, or a sense of position, motion and force. "Tactile imaging" is the process of turning a visual item, such as a picture, into a touchable raised version of the image, so that this tactile rendition faithfully represents the original information. This provides access for blind and visually impaired persons to normally inaccessible information. Typically, there are textual and pictorial components to such information.
For textual data alone, screen reading speech synthesizers operating with scanners and optical character recognition (OCR) software, Braille cell displays and Braille embossing printers are the media of choice for blind computer users. Some users prefer the vibrotactile pin array of the Optacon, although its use appears to be on the decline. Persons with low vision typically rely on one or more of the these methods, combined with screen magnification software or hardware. Generally, any information obtained as text is accessible to a blind or visually impaired person. [2]
For pictorial scientific information, such as graphs, maps, drawings, and photographs, this is not the case. Static and dynamic means exist for output of many of these two-dimensional graphics. Braille embossing printers (in graphics mode), microcapsule paper and custom hand-made labor-intensive representations of paper, paste and string are used for static rendering of graphical information [4]. For active representation, the only dynamic display in everyday use is the Optacon. Active research in this area promises technologies such as nickel-titanium shape-memory alloy actuated pin arrays, an Optacon-based mouse, electrorheological fluid pads and full-page Braille cell displays [5,12]. These methods work well with simple two-dimensional graphs, maps and line drawings, but cannot adequately represent more complex photographic information. This shortcoming is the motivation for TACTICS, our tactile imaging system for translation of complex visual information into simplified tactually comprehensible form.
While tactile imaging is relegated to two dimensions, "haptic graphing" in Virtual Environments (VEs) allows modeling and exploration of scientific data in three or more dimensions. A haptic interface is a force feedback mechanism that is used to feel computer generated objects or forces that a remote robot is sensing. In the case of computer generated objects, the forces are calculated at a point called the interface point (IP). The IP location is determined in the coordinate system of the haptic mechanism via sensors, and then transformed to the coordinate system of the VE. Once in the VE, the force is determined based on the position of virtual objects with respect to the this point. For example, consider a "virtual wall" placed in the VE workspace. When the user moves the IP through the wall, the motors generate forces to oppose the motion. Given that no motor can have an instantaneous reaction time, the amount of the force is determined as a function of the distance into the virtual wall (e.g., Hooke's Law F=kx where x is the distance into the wall, and k is a proportionality constant). Thus, the haptic interface determines the coordinates of the IP, sends them to the VE, and then receives the value of the force to display to the user. Haptic display research is limited, but has been used to increase the immersive effect of virtual reality and scientific exploration such as molecule docking, nanoscale manipulation, and virtual surgery [1,3,10].
The hardware components of TACTICS are a Reprotronics Tactile Image Enhancer (C), a supply of microcapsule paper, and a typical office laser printer and photocopier machine. Operation of the system involves loading an image, applying processing algorithms in a predetermined sequence, printing the result, photocopying this result onto microcapsule paper, and finally developing or "puffing" the image using the enhancer.
A pilot study employing the TACTICS system was revealing. We prepared a number of 2.5 inch square images by applying K-means segmentation and Sobel edge detection algorithms individually, together in both orders, or not at all. The images included a number of faces, a hot air balloon and an illustration of a human heart, all oval in shape. The image set also included photographs of a house, laptop computer and space shuttle launch. Blindfolded subjects performed a closed-set matching task, feeling one of the tactilized images and then attempting to find an identical one from among a randomly ordered collection of all similarly processed images. This task was repeated for each of the images and results were tabulated for each of the five possible processing sequences. When no processing was performed on the images, tactile images tended to be virtually indistinguishable from one another, with results only slightly better than chance (12 percent). For images processed using K-means segmentation or Sobel edge detection alone, results were dramatically improved, with recognition rates of 60-80 percent. Mismatches usually resulted from similar shapes among some of the images. When K-means was applied first, followed by Sobel, recognition was better than 70 percent. Best results were achieved by applying edge detection first, then segmentation, yielding a match 97 percent of the time.
These results indicate that simplification of an image can greatly enhance its recognition potential. While this does not address understanding of content, it is an important first step towards that end. To successfully accomplish tactile image comprehension, we feel that it will be necessary to associate textual description with simplified tactile images, much as a photograph in a newspaper usually comes with some brief caption. This is due to the inherent ambiguities involved with representing three-dimensional objects in a two-dimensional medium combined with the lower resolution of the fingert ip. We are hopeful that the utility of this system will increase with user experience as well.
A two dimensional plot can easily be represented with static methods mentioned earlier, but dynamic methods are often desired when exploring scientific phenomenon. In a 3D VE, a 2D plot can be rendered on a plane anywhere in the virtual workspace. For example, plots of position, velocity, and acceleration can be generated while performing virtual physics experiments. Similar to graphical representations, 2D curves are represented as piecewise linear line segments connecting the data points.
In order to find and feel very thin lines with a single point (the IP) a technique known as virtual fixturing is used [11]. The purpose of these fixtures are to generate forces to guide the user in performing dexterous tasks, such as teleoperation of a robot. That concept can be applied to feel lines by generating forces to pull the user's fingertip to the line, just as a magnet attracts iron. These forces are only perpendicular to the line, though, which allows the user to easily slide along the line, and thus, follow the data curve.
A three dimensional plot is felt using a different method. A 3D data set can be a 2D matrix of values (the values being the Z coordinates on an XY grid) or an array of vertex coordinates in space (the same way 3D graphics objects are defined). The vertices of either representation are connected with triangular polygons (facets), making the object surface piecewise linear. The methods for haptic rendering of 3D objects in [8,9] only consider objects as simple volumes (e.g., spheres and cubes). The method in [13] can render polyhedral objects, but has several limitations such as the number of polygons that can be used. The general haptics problem is finding the appropriate direction of the force vector. If the IP could not penetrate the surface, the problem would be simple because the direction would be the normal vector of the surface at that point. However, since the IP penetrates the surface, the force direction is calculated by determining which facets the IP would lie on if it could not cross through the surface. This point is called the Shadow Point. For a simple plane, the Shadow Point is the point on the plane that is closest to the IP. The Shadow Point algorithm also considers intersections of multiple planes that are concave (e.g., the inside edges/corners of a box) or convex (e.g., the outside edges/ corners of a box).
Feeling discrete data points is accomplished in the same manner used for feeling a sphere with a haptic interface. To feel a sphere, the distance between the IP and the center of the sphere is calculated, and then compared to the sphere radius. If the distance is less than the radius, a force is generated to push the user out of the sphere, with a magnitude proportional to the distance of penetration. Therefore, in order to feel multiple points, a sphere is placed around the IP (the IP is the sphere center), and the distance between the IP and the data points are calculated. After all of the nonzero force vectors are determined, they are added together, and then sent to the haptic interface. In this manner, a user can feel a set of discrete points in space.
Even from a graphical representation of data, it is difficult to understand the nature of the data without numbered axes and tick marks. In a haptic environment, axes are represented with planes with periodic "bumps" for the tick marks. A haptic grid is also necessary to determine the approximate location on a plot without needing to return to the axes. To accomplish this, a grid is created by using thin walls that generate just noticeable forces when the IP passes through them. This allows the user to feel the grid without interfering with the data itself. As an additional aid for navigation, the user can request (at the touch of a button) for the current IP coordinates to be converted to speech via a text-to-speech converter. The spoken coordinates are relative to the origin of the plot currently being examined.
The data plots that have been represented with our haptic graphing system show great potential for opening a new door for humans to explore information without using vision. This is especially true in light of the current methods commercially available. Beyond the application for blind students, even sighted students can gain insight into scientific data through an additional medium. It is quite fascinating to be able to feel information that has previously been represented only on paper or a computer monitor.
Expected development of the TACTICS system includes improvements to the prototype and subsequent development of a stand-alone application, and ultimately an affordable end-user system. The theories and applicability demonstrated by the TACTICS system should prove invaluable in the prodution of an active dynamic tactile display system. A display is only as good as the information it displays, and a TACTICS based system should provide a viable front-end for such emerging technology. More research is necessary to determine the usability of the Haptic Graphing system. For one, human subject studies must be done to determine that a person can understand the contents of a haptic graph without the usual visual aids. More work is also needed to test and extend the limitations such as the complexity of the curve or surface, and the number of curves or surfaces on one plot. In the graphics community, interpolation schemes such as Phong shading are used to make a surface appear smoother without adding polygons. A similar interpolation of the facet normals in the haptic environment should allow a surface to feel smoother without adding facets.
[2] Burger, D., "Improved access to computers for the visually handicapped:
New prospects and principles". IEEE Transactions on Rehabilitation
Engineering, vol. 2, no. 3, pp. 111-118, 1994.
[3] Durlach, N., Mavor, A. ed., "Virtual Reality: Scientific and
Technological Challenges", National Academy Press, 1995.
[4] Edman, P.K., "Tactile graphics". American Foundation for the Blind, 1992.
[5] Fricke, J. and Baehring, H., "Design of a tactile graphic I/O tablet
and its integration into a personal computer system for blind users".
Electronic proceedings of the 1994 EASI High Resolution Tactile Graphics
Conference (http://www.rit.edu/~easi/easisem/easisemtactile.html).
[6] Lindley, C.A., "Practical image processing in C", John Wiley & Sons,
New York, 1991.
[7] Loomis, J.M. and Lederman, Susan J., "Tactual perception. Handbook
of perception and human performance", ch. 31, 1986.
[8] Massie, T. and Salisbury, K., "The PHANToM haptic interface: a
device for probing virtual objects". ASME Winter Annual Meeting, Symposium
on Haptic Interfaces for Virtual Environment and Teleoperator System, 19
94.
[9] Massie, T., Salisbury, K., Brock, D., Zilles, C., "Haptic rendering:
programming touch interaction with virtual objects". ACM Symposium on
Interactive 3D Graphics, ACM, 1995.
[10] Minsky, M., Ouh-Young, M., Steele, O., Brooks, F., Behensky, M.,
"Feeling and seeing: issues in force display". Proceedings of the Symposium
on 3D Real-Time Interactive Graphics, ACM, 1990.
[11] Sayers, C., Paul, R., "Synthetic Fixturing". Advances in Robotics,
Mechatronics, and Haptic Interfaces, vol 49, ASME, 1993.
[12] Vanderheiden, G.C., "Dynamic and static strategies for nonvisual
presentation of graphical information". Electronic proceedings of the 19
94 EASI High Resolution Tactile Graphics Conference
(http://www.rit.edu/~easi/easisem/easisemtactile.html).
[13] Zilles, C., Salisbury, K., "A constraint-based god-object method
for haptic display". IROS Proceedings, 1995.