GloveGRASP User's Guide

1 Introduction

NOTE: The GloveGRASP software and documentation is currently in Beta form. Any bugs or errors in the software or documentation should be reported to General Reality, email support@genreality.com. The GloveGRASP 1.0 final release will be December 1996.

2 Installation

GloveGRASP is distributed as tarred and compressed files on one floppy disk. These files include source code, make files, binary executables and object files in separate directories. The file structure within the tar file is as follows:

GloveGRASP
GloveGRASP/README
GloveGRASP/demo
GloveGRASP/demo/DDriver.c
GloveGRASP/demo/GRivHand.cxx
GloveGRASP/demo/GRivHand.h
GloveGRASP/demo/GestureClient.c
GloveGRASP/demo/GestureRecog.cxx
GloveGRASP/demo/GestureServer.c
GloveGRASP/demo/GestureTrain.cxx
GloveGRASP/demo/IvDemo
GloveGRASP/demo/IvDemo.cxx
GloveGRASP/demo/IvDemo32
GloveGRASP/demo/IvDemo64
GloveGRASP/demo/Makefile
GloveGRASP/demo/README
GloveGRASP/demo/test.txt
GloveGRASP/demo/GRASPmodel
GloveGRASP/demo/GRASPmodel/GRASPObjectList.cxx
GloveGRASP/demo/GRASPmodel/GRASPdefines.h
GloveGRASP/demo/GRASPmodel/GRASPdummy.cxx
GloveGRASP/demo/GRASPmodel/GRASPdummy.h
GloveGRASP/demo/GRASPmodel/GRASPfastrak.cxx
GloveGRASP/demo/GRASPmodel/GRASPlist.cxx
GloveGRASP/demo/GRASPmodel/GRASPmain.cxx
GloveGRASP/demo/GRASPmodel/GRASPmain.h
GloveGRASP/demo/GRASPmodel/Makefile
GloveGRASP/demo/GRASPmodel/README
GloveGRASP/demo/GRASPmodel/aux.h
GloveGRASP/demo/GRASPmodel/demo.txt
GloveGRASP/demo/GRASPmodel/rasters.h
GloveGRASP/demo/GRASPmodel/seriallib.c
GloveGRASP/demo/GRASPmodel/seriallib.h
GloveGRASP/demo/GRASPmodel/trakit.c
GloveGRASP/demo/GRASPmodel/trakit.h
GloveGRASP/doc
GloveGRASP/doc/README
GloveGRASP/doc/manual.ps
GloveGRASP/doc/manual.doc
GloveGRASP/include
GloveGRASP/include/GR.h
GloveGRASP/include/GR5glove.h
GloveGRASP/include/GRCandidate.h
GloveGRASP/include/GRContext.h
GloveGRASP/include/GRDeviceData.h
GloveGRASP/include/GRFrame.h
GloveGRASP/include/GRGrasp.h
GloveGRASP/include/GRMath5D.h
GloveGRASP/include/GRNet.h
GloveGRASP/include/GRTrainRecog.h
GloveGRASP/lib
GloveGRASP/lib/README
GloveGRASP/lib/libgrasp.a
GloveGRASP/lib/libgrasp32.a
GloveGRASP/lib/libgrasp64.a

The include directory contains the necessary header files with the function definitions of all the GloveGRASP functions. These header files should be included in the path searched by the -I flag of your C/C++ compiler. The functions themselves are contained in the binary file libgrasp.a. The directory this file is contained in should be included in the path searched by the -L flag on your C/C++ compiler. The makefile in the demo directory shows how to compile the GloveGRASP libraries with your applications. The *.c and *.cxx files in the demo directory are sample applications showing how the GloveGRASP libraries can be used in practice.

In order to install the software the following steps should be followed:

1/ Change to the directory where you want the GloveGRASP files to be located. All the files will go into a GloveGRASP directory structure below this current directory.

2/ Copy the GloveGRASP compressed file, GRASP_t.z into the current directory.

3/ Copy the file GRASP_t.z to GRASP_t.Z
mv GRASP_t.z GRASP_t.Z

4/ Uncompress the file:
uncompress GRASP_t.Z

5/ Extract the GloveGRASP files:
tar xvf GRASP_t

These instructions can also be found in the file README on the GloveGRASP disk. The tar command will produce a complete listing of files as it is extracting them from the tar file. Sample programs ready for compilation can be found in the demo directory and all directories have README files explaining what files are in that directory and how to use them.

2.1 Quick Start

The executable IvDemo in the demo directory is an Inventor application that can be run directly without compilation. This is a simple example of the type of applications that can be written using the GloveGRASP libraries. Typing IvDemo in the demo directory will start the application. There is also a 64 bit version of the application which can be run by typing IvDemo64. Users must then enter the name of the serial port the 5th Glove is connected to, and go through a simple calibration routine.

grof@kodiak/demo> IvDemo

IvDemo - 5DT Glove Inventor Demo
Input the port that the glove is connected to : /dev/ttyd2
Ready to initialise min values (y/n)y
Ready to initialise max values(y/n)y

When the software asks if they are ready to initialise the minimum glove values the user should rest their hand flat on the table and enter 'y', this measures the minimum bend of each finger. Similarly when asked for the maximum glove values the user should make a tight fist and enter 'y', this measures the maximum bend of each finger.

After the calibration is complete an Inventor Examiner window will open with a simple virtual hand model, as shown below. Moving the fingers on the glove will cause the virtual fingers to move in the same way. Note that this application is written for a right handed glove, so user's with a left handed glove will see the opposite fingers moving. The hand object can be manipulated using the standard Examiner window widgets.

3 Device Driver Functions

GloveGRASP has five device driver functions which allows access to all the 5th Glove input and output modes as described in Annexure B of the 5th Glove documentation.

These functions are:
FILE *GRopenPort(char *Port_name)
int GRwritePort(char *str,FILE *open_port)
int GRreadPort(FILE *open_port, int num_bytes, char *buf)
int GRreadHandData(FILE *file_port,int *hand_data)
int GRportClose(FILE *port)

and their prototypes can be found in the header file GR.h.

In order for the user to access raw glove data the serial port the glove is connected to must first be opened. This is accomplished with the GRopenPort(char *Port_name) function. This returns either a pointer to the port file name or NULL if the port couldn't be opened.

Once the port has been opened the user can use GRwritePort and GRreadPort to write and read data to and from the glove. The glove begins in command mode. The valid strings that can be sent to the glove in this mode are summarized below:

Command String	Returns	Description
'A'	55HEX	Reset the glove into command mode
'Bxx'	'xx'	Used to test the serial i/o
'C'	<glove data>	Set report data mode
'D'	<glove data>	Set continuous data mode
'G'	<info>	Request the glove info
'H' <data>	55HEX	Upload gestures
'I'	<data>	Download Gestures

Using these command strings the glove can either be set into report or continuous mode for reading raw data. In Report mode raw data is returned whenever the glove is polled by sending it any character except 'A', while in Continuous mode the glove streams raw data continuously to the serial port. In both case the string 'A' returns the glove back to command mode.

During Report mode:

Command String	Returns	Description
'A'	55HEX	Return to command mode
else	<glove data>	Return the glove data

During Continuous Mode:

Command String	Returns	Description
'A'	55HEX	Return to command mode
else	nothing	ignored

The raw data string returned from the glove consists of 9 bytes in the following format:
sy f1 f2 f3 f4 f5 tp tr cs

where:
sy is the leading header value and should always be 80 hex.

f1-f5 are the 8 bit (0-255) raw flex values for each of the five fingers.
For the right hand f1 corresponds to the thumb, f2 the index finger ...f5 the pinky finger. For the left hand the ordering is reversed with f5 the thumb, down to f1 the pinky.

tp is the 8 bit raw pitch value of the hand. The pitch sensor can measure pitch from -60 to +60 degrees and returns a value of 128 when it's centered at pitch = 0.

tr is the raw roll value of the hand. The roll sensor can measure roll from -60 to +60 degrees and returns a value of 128 when it's centered at roll = 0.

cs is the checksum value used to check the correctness of the other values. It should be equal to the xored result of f1 to f5 and tp and tr.

Sometimes the serial port may already have data in it's buffer when the glove is initialized and so the first few readings could be in error. This can be detected by using the checksum value. If the checksum value isn't equal to the xored result of f1 to f5 and tp and tr then the data bytes are not being read from the correct starting value. If this is the case more characters can be read until the checksum value is correct. The function GRreadHandData uses this approach to always ensure that the data values it returns back are correct. GRreadHandData returns back an array of nine integers, one for each of the raw data bytes. It is slightly slower than the GRreadPort function, but it always returns back the correct data string.

Once the user has finished accessing the 5th Glove the serial port can be closed by using GRportClose.

3.1 Device Driver Sample Code

The following partial listing taken from DDRiver.c shows how the device driver functions can be used in a simple application. DDriver.c simply opens the serial port, reads raw data from the glove and prints it to the screen. Complete code and a makefile is in the demo directory.

The serial port first needs to be opened using GRopenPort, and then the glove initialized:

  #include "GR.h"
  FILE *file_port;
  char Port_name[20],buff[20];
 
  int iterations;

  /* query the port name */
  printf("Input the port that the glove is connected to : ");
  scanf("%s",&Port_name);
  printf(" Input how many raw data values wanted : ");
  scanf("%d",&iterations);

  /* open the serial port */
  file_port = GRopenPort(Port_name);

  /* initialize the glove */
  GRwritePort("A",file_port);

  if(GRreadPort(file_port,1,buff)!=GR_ERROR)
	printf("\n 5DT glove opened on port %s\n",Port_name);
  else {
	printf("\n 5DT glove couldn't by opened on port %s - Quitting ..\n",Port_name);
 
  return(GR_ERROR);
 }

Data can be read in three ways; first by setting the glove into report mode and polling:

  int i;
  char buff[9],c;

  printf("\n First I'm going to let you Poll the glove for data \n");
  printf("  - hit 'q' to quit, (return) for data \n\n");
  c = getchar();

  /* setting it into report mode */
  while(c!='q'){
	GRwritePort("C",file_port);
	GRreadPort(file_port,9,buff);
	printf(" read : ");
	
	for(j=0;j<9;j++)
		printf(" %d ",(int)buff[j]);
		
	c = getchar();
  }

Note that in Report mode a character needs to be written to the port each time raw data is polled for. GRreadPort returns a array of characters which may need to be converted into integers for later use.

Data can also be read by setting the glove into continuous mode: int i,j;

  char buff[9];

  /* now set the glove into continuous mode*/
  printf("\n\n Now I'm going to read %d values in Continuous Mode \n",iterations);
  printf("  - hit (return) for data \n");
  c = getchar();
 
  /* setting it into continuous mode */
  GRwritePort("D",file_port);

  /* read values explicitly */
  for(i=0;i<iterations;i++){
	GRreadPort(file_port,9,buff);
	printf("\n read : ");

  	for(j=0;j<8;j++)
		printf(" %d ",(int)buff[j]);
  }

Raw data is streamed back continuously from the glove until the 'A' character is sent to set the glove into Command mode.

Finally, GRreadHandData can be used in either continuous or report mode:

  int i,j,data[9];
  
  /* read data values */
  for(i=0;i<iterations;i++){
	/* going into report mode */
	GRwritePort("C",file_port);
	GRreadHandData(file_port,data);
  }
  
  printf("\n read : ");
  for(j=0;j<9;j++)
	printf(" %d ",data[j]);
  }

Note that GRreadHandData returns an array of integers, in contrast to GRreadPort.

When the application is finished the port needs to be closed:

GRclosePort(file_port);

When the DDriver application is running it produces the following output:

kodiak/demo> DDriver

GloveGRASP 5DT Device Driver Demo Code

Input the port that the glove is connected to : /dev/ttyd2
Input how many raw data values wanted : 10

5DT glove opened on port /dev/ttyd2

First I'm going to let you Poll the glove for data - hit 'q' to quit, (return) for data

read :  128    0    0    0    0    0  128  128    0
read :  128    0    0    0    0    0  128  128    0
read :  128   46   59   43   39   66  115  136  160
  ....
read :  128  178  255  255  247  238   95  140  120
read :  128  176  255  255  247  238   95  140  122q

read :  128  1  0  0  0  6  -7.03   0.00  246
read :  128  1  0  0  0  7  -7.03   0.00  247
 ....
read :  128  1  0  0  0  6  -7.03   6.56  248
read :  128  2  0  0  0  6  -6.56   6.56  248

read :  128  1  0  0  0  6  114  142  251
read :  128  2  0  0  0  6  114  142  248
 ....
read :  128  2  0  0  0  6  113  142  251
read :  128  2  0  0  0  6  113  142  251

5DT glove closed

4 C++ Device Driver Class

The GloveGRASP C device driver code is encapsulated with some additional functions in the C++ class, GR5glove. This class is defined in the header file GR5glove.h and has the following methods:

Low Level Operations:
char * GR5glove::DefaultDevice()
int GR5glove::GRopenPort(char *Port_name)
int GR5glove::GRwritePort(char *test_str)
int GR5glove::GRreadPort(int num_bytes, char *buf )
int GR5glove::GRreadHandData(int *hand_data)
int GR5glove::GRclosePort( )

High Level Operations:
int GR5glove::Reset( )
int GR5glove::Read(GRDeviceData& fval)
int GR5glove::ReadCheck(GRDeviceData& fval)

The low level operations are equivalent to the C functions described in the previous section, except GR5glove::DefaultDevice() which returns a pointer to the string "/dev/ttyd2", the default port.

The high level methods GR5glove::ReadCheck, and GR5glove::Read, set the glove into Report mode and poll for values with and without using the check sum value respectively. GR5glove::Reset can be used to reset the glove into command mode.

GR5glove::DefaultDevice returns "/dev/ttyd2", while the other methods return GR_OK if they complete successfully or GR_ERROR if not.

4.1 Open Inventor Sample Code

An example of how to use the GR5glove class is shown in the IvDemo.cxx and GRivHand.cxx files. IvDemo.cxx is a simple Inventor application that displays a virtual hand model which responds to glove input. It is linked with the GRivHand.cxx code and GRivHand.h which defines the GRInventorHand class.

In developing a gesture based application there are several basic steps that must be followed, as shown in IvDemo.cxx. After initializing the graphics window, the glove serial port should be opened, the glove reset and calibrated.

  //initialize inventor - and make a scene containing a hand
  Widget myWindow = SoXt::init(argv[0]);
  if(myWindow==NULL) exit(1);
  SoSeparator *root = new SoSeparator;
  
  //connect the glove, reset and calibrate
  hand->ConnectGlove();
  hand->InitGlove();
  hand->CallibrateGlove();

Glove calibration involves finding the maximum and minimum joint bend values of each finger. The minimum bend values are measured by getting the user to make a flat hand gesture and averaging raw data over several readings. Similarly the maximum values can be found by getting the user to make a fist and averaging over several values. Care must be taken to calibrate the thumb which can assume a wide range of different positions with similar bend values. The GRivHand.cxx file contains simple calibration code for the four fingers only:

  // calibrate the hand
  int GRInventorHand::CalibrateGlove(){
  	int i,j;
	char c;

 	//check to make sure the glove is connected
 	if(glove_connected){
 		//read the minimum bend values 
 		// - user should have flat hand
 		cout<<"Ready to initialize min values (y/n)";
 		cin>>c;
 		
 		for(i=0;i<10;i++){
 			ReadData(); 
 			for(j=0;j<NUM_FINGERS;j++)
 				min_data[j] = min_data[j]+rawdata[j+1];
 		}
		cout<<endl;
		
		//read the maximum bend values 
		// - user should have a fist
		cout<<"ready to initialize max values(y/n)";
		cin>>c;
		
		for(i=0;i<10;i++){
			ReadData(); 
			for(j=0;j<NUM_FINGERS;j++)
				max_data[j] = max_data[j]+rawdata[j+1];
		}

		//average the data
		for(i=0;i<NUM_FINGERS;i++){
			in_data[i]=min_data[i]/10;
			max_data[i]=max_data[i]/10;
		}
		return(GR_OK);
	}
	else{
		//the glove mustn't be connected
		cout<<"The glove isn't connected.."<<endl;
		return(GR_ERROR);
	}
  }

Once the glove has been calibrated, an Inventor timer sensor is set to ensure the glove data is read 30 times a second. This timer sensor uses the callback function readDataCallBack to read the current glove data and update the graphical model.

  //reading the data and update the model
  static void readDataCallBack(void *data, SoSensor *){
  	((GRInventorHand *)data)->ReadData();
  	((GRInventorHand *)data)->UpDateModel();
  }
  
  //start the data sampling
  SoRotation *myRotation = new SoRotation;
  SoTimerSensor *DataCollectionSensor = new SoTimerSensor(readDataCallBack,(void *)hand);
  DataCollectionSensor->setInterval(TICK_RATE);
  DataCollectionSensor->schedule();

The 5th Glove only returns a single bend value for each finger, so an approximation must be made to update the joint rotations for each of the three joints. In this case we assume that each joint bends a maximum of 90 degrees. The actual rotation value can be found by using the maximum and minimum finger bend values to scale the glove raw data to fall in the range 0-90 degrees.

This is done in the UpDateModel function from GRivHand.cxx:

  int GRInventorHand::UpDateModel(){
	static float angle[NUM_FINGERS];
	
	//Look for rotation types
	cout<<" Rotating : ";
	
	for(int i=0;i<NUM_FINGERS;i++){
		angle[i] = -M_PI*(rawdata[i+1]-min_data[i])/(2*(max_data[i]-min_data[i]));

		//rotate object
		myRotation[i]->rotation.setValue(xaxis,angle[i]);
		cout<<" "<<angle[i];
	}
	cout<<endl;
	
	return(GR_OK);
  }

myRotation[NUM_FINGERS] is a global array containing the rotation about the x axis of each finger. We assume all the joints in a finger are rotated the same, so the myRotation value is applied at each joint.

The final step is to add a hand model to the scene, open the examiner window and begin the simulation loop:

  //add the hand model to the scene
  root->ref();
  root->addChild(hand->CreateHand());
  
  //Set up viewer
  SoXtExaminerViewer *myViewer = new SoXtExaminerViewer(myWindow);
  myViewer->setSceneGraph(root);
  myViewer->setTitle("Hand View");
  myViewer->show();
  SoXt::show(myWindow);
  SoXt::mainLoop();

The CreateHand method from GRivHand.cxx is used to create the geometry for a simple virtual hand from Inventor box and cylinder primitives. An improvement would to allow the user to specify more complicated geometry files that could be read in by the application.

IvDemo is designed to work with a right-handed glove; for a left-handed glove the rawdata indices will need to be changed since rawdata[1] is now the bend value of the left pinky, not the thumb.

When IvDemo is run the user will be prompted for connection and calibration information, when this is entered an Inventor examiner window containing a virtual hand model will appear. This is shown in figure 1.0 below. Moving the fingers of the glove glove will produce the corresponding motion in the virtual hand.

grof@kodiak/demo> IvDemo

IvDemo - 5DT Glove Inventor Demo

Input the port that the glove is connected to : /dev/ttyd2
Ready to initialise min values (y/n)y
Ready to initialise max values(y/n)y

5 Networking Functions

GloveGRASP provides TCP/IP networking functions that can be used to build client/server applications. This allows users to use one machine for gesture recognition and run their application on another.

The functions are defined in GRNet.h and are:

GRsocket *GRnetOpen(char *hostname, int port,int buffersize,char *mode, char *buffer);
char *GRnetRead(GRsocket *g);
void GRnetWrite(GRsocket *g, char *buffer);
void GRnetClose(GRsocket *g);
void GRsendNetData(GRsocket *s,char *message);
char *GRreadNetData(GRsocket *s);

In order for the user to run a network application the TCP/IP socket must be opened first by using the GRnetOpen function. Sockets can be opened in either read or write mode according to whether the user wants to retrieve data from them or send data to them. Once a socket is opened data can be written to it using GRnetWrite or GRsendNetData and read from it using GRnetRead or GRreadNetData. The connection can be closed by using GRnetClose. The maximum length of a string that can be read from or written to a socket is defined by the variable GR_SOC_SIZE, set in GRNet.h.

5.1 Networking Example

An example of how these functions can be used is in the files GestureServer.c and GestureClient.c, both in the demo directory. GestureServer.c illustrates how to write a simple gesture server while Client.c contains sample client side code. GestureServer reads raw data from the glove and sends it across the network to GestureClient.

5.1.1 Server Code

In order to write gesture server code there are only four steps that need to be taken:

First, the network connection should be opened in write mode:

  GRsocket *soc;
  char readbuf[2],address[256],Port_name[20],buff[20];
  int port;
  FILE *file_port;
  
  /* read the network address and open the connection */
  printf(" Remote Client Network Address-> ");
  scanf("%s",address);
  printf(" Remote Client Port Number ->");
  scanf("%d",&port);
  soc=GRnetOpen(address,port,GR_SOC_SIZE,"w",NULL);<
  if(!soc){
  	printf("Couldn't open the network connection\n");
  	return(GR_ERROR);
  }

Second, connect to the glove serial port:

  /* now we need to open the glove serial port */
  /* query the port name */
  printf("\n Input the port that the glove is connected to -> ");
  scanf("%s",&Port_name);
  
  /* open the serial port */
  file_port = GRopenPort(Port_name);
  if(!file_port){
  	printf("Couldn't open the serial port \n");
  	return(GR_ERROR);
  }
  
  /* initialize the glove and check if port is opened OK */
  GRwritePort("A",file_port);
  if(GRreadPort(file_port,1,buff) == GR_OK)
  	printf("\n 5DT glove opened on port %s\n",Port_name);
  else {
  	printf(" The port %s couldn't be opened - quitting \n",Port_name);
  	return(GR_ERROR);
  }

Then send data to the socket connection, using the user defined readandsendData function:

  int readandsendData(FILE *file_port,GRsocket *soc){<
	int data[9];
	char sendbuff[GR_SOC_SIZE];
	
	/* going into report mode */
	GRwritePort("C",file_port);
	GRreadHandData(file_port,data);
	
	sprintf(sendbuff,"h:%d f0:%d f1:%d f2:%d f3:%d f4:%d tp:%d tr:%d cd:%d \n",
		data[0],data[1],data[2],data[3],data[4],data[5],data[6],data[7],data[8]);
	printf("   Data Sent : %s",sendbuff);


	/* Send the data */
	if(!GRsendNetData(soc, sendbuff)){
		printf("Error sending the data string \n");
		return(GR_ERROR);
	}
	
	return(GR_OK);
  }

readandsendData is called from the main module:

  /* main loop - loop until sent a 'q'*/
  while (strcmp(readbuf, "q")!=0){
  	printf("\n Data to send (q=end) (c=continuous)(ret = data) -> ");
	gets(readbuf); /* read input string */
	
	/* going into report mode and sending the data */
	readandsendData(file_port,soc);
	if(strcmp(readbuf, "c")==0){/* send 50 values continuously */
 		for(i=0;i<50;i++)
 			readandsendData(file_port,soc);<
 	}
  }

Finally, close the network connection:

  /* send quit message to client */
  GRsendNetData(soc, "quit");
  /* close the network connection */
  GRnetClose(soc);
  return(GR_OK);

5.1.2 Client Code

On the client side there are three steps that need to be made:

First, the client connection should be opened in read mode:

  int port;
  char address[32];
  GRsocket *soc;

  printf("\n GloveGRASP Client Demo Code \n\n");
  printf(" Client Network Address (localhost)-> ");
  scanf("%s",&address);
  printf(" Client Port Number (1027) ->");
  scanf("%d",&port);
  
  soc = GRnetOpen(address,port,SIZE,"r",NULL);
  if(!soc){
  	printf("ERROR: couldn't open socket for reading \n");
	exit(1);
  }

Next, data should be read from the port until a "quit" message is received:

  char oldtmp[GR_SOC_SIZE] = "nil";
  char tmp[GR_SOC_SIZE] = "nil";

  void read_network(GRsocket *soc){
  	/* read data from the socket */
  	strcpy(tmp,GRreadNetData(soc));
  	/* check to see if it is new - print it out if it is*/
	if (strcmp(tmp,oldtmp)!=0){ 
		printf(" Received message -> %s \n", tmp);
	}
	strcpy(oldtmp, tmp);
  } /** read_network **/ 

  while((strcmp(tmp,"quit")!=0)){
	read_network(soc);
  }

Finally, closing the socket connect when the application is finished:

GRnetClose(soc);

When GestureServer and GestureClient are run, the server remote machine address and port numbers must be the same as the client local machine address and port number. To run GestureServer simply type GestureServer and enter the remote SGI machine name and the port you want to connect to:

kodiak/GloveGRASP/demo>GestureServer

GloveGRASP Server Demo Code

Remote Client Network Address -> Nanook
Remote Client Port Number -> 4500

Input the port that the glove is connected to -> /dev/ttyd2
5DT glove opened on port /dev/ttyd2

Data to send (q=end) (c=continuous) (ret= data) ->
Data Sent : h:128 f0:72 f1:0 f2:0 f3:0 f4:0 tp:149 tr:240 cd:44
Data to send (q=end) (c=continuous) (ret= data) ->
Data Sent : h:128 f0:73 f1:0 f2:0 f3:0 f4:0 tp:145 tr:242 cd:42
Data to send (q=end) (c=continuous) (ret= data) ->
Data Sent : h:128 f0:75 f1:0 f2:0 f3:0 f4:0 tp:130 tr:242 cd:59
Data to send (q=end) (c=continuous) (ret= data) ->q

At the same time GestureClient should be run on the client SGI machine:

nanook/GloveGRASP/demo>GestureServer

GloveGRASP Client Demo Code

Client Network Address -> nanook
Client Port Number -> 4500

Received message ->
Received message ->h:128 f0:72 f1:0 f2:0 f3:0 f4:0 tp:149 tr:240 cd:44
Received message ->h:128 f0:73 f1:0 f2:0 f3:0 f4:0 tp:145 tr:242 cd:42
Received message ->h:128 f0:75 f1:0 f2:0 f3:0 f4:0 tp:130 tr:242 cd:59
Received message ->quit
Client Quiting..

6 Gesture Input

The GloveGRASP library contains functions for accurate gesture recognition based on template matching. Gestural input involves two phases; creating a user-specific template file containing sample gestures, and using this template files for real-time gesture recognition within an application.

6.1 Gesture Training

Gesture training is supported by three functions, defined in the GRTrainRecog.h file:
void FingerCalibration(flex_slope, flex_min,glove,num_of_test, finger_threshold);
GRMultiContext * GestureTraining(GRMultiContext* m_context_ptr, GRInt5& flex_min,GRFloat5& flex_slope,bool which_hand,int num_to_train)
void SaveTrainingData(GRMultiContext * m_context_ptr)

GRMultiContext is used to create the template structure from sample gestural input, while GRSaveTrainingData saves the training data to an ASCII file for later use by the recognition functions. FingerCalibration calculates the scaling parameter for scaling the raw glove data, the minimum bend for each finger, and the gesture recognition threshold level.

GloveGRASP creates the template files from sample gestures input by the user. The user forms the desired gesture a number of times and the GestureTraining function records statistical information about each of the finger bend values. The number of training samples for each gesture is determined by the parameter num_to_train, flex_min is an array of minimum bend values, flex_slope a scaling parameter, and which_hand a boolean specifying which hand is being trained for (RIGHT_HAND or LEFT_HAND). The trained template is stored as several linked lists of values and gesture names, the head of which is pointed to by m_context_ptr.

The values of flex_min and flex_slope are found by calling the FingerCalibration function. This function measures the minimum and maximum joint bends values from the glove as the user bends and relaxes their hand a number of times. The number of hand measurements taken during calibration is set by num_of_test while finger_threshold determines a minimum bend threshold that the joint values should exceed. If this threshold isn't exceeded then default values are used.

GloveGRASP is unique in that it supports context dependent gesture recognition. This means that a given gesture can be interpreted differently according to the current context. For example, when the user makes a fist with their virtual hand in space it may be interpreted as a 'Show Menu' gesture, but when that same gesture is made with the virtual hand inside an object it could be interpreted as a 'Pick' gesture, simply because the interaction contexts are different. The use of context dependent recognition means that a large number of gestural interpretations can be applied to a small number of gestures. Using a small number of well defined gestures improves the recognition rate and makes it easier for the user to remember the gesture set.

The GestureTraining function will prompt the user for both the current context name and the name of each gesture as it is to be added to the context. Gesture names may be any single word of up to 20 characters in length, while conext names may be up to eight characters in length.

6.1.1 Training Example

The sample program GestureTrain.cxx shows how the gesture training functions can be used in practice. First the glove serial port needs to be opened and the glove reset:

  // Open the port for 5glove
  printf("Opening port:  %s\n", filename);
  if ( (glove.GRopenPort(filename)) == GR_ERROR) {
  	cerr << "Couldn't open glove device, exit\n";
	exit (EXIT_FAILURE);    // couldn't open file
	}
  else if ( (glove.Reset()) == GR_ERROR ){
	cerr << "Couldn't reset glove device, exit\n";
	exit (EXIT_FAILURE);<
  }

Then the FingerCalibration function is called to calculate the minimum bend values and raw data scaling parameters:

  char c;

  cout <<"Hit return when for Calibration: "
  cin >>c;
  cout << endl;
  
  // Now calibrate the finger flex values
  FingerCalibration(flex_slope,flex_min,glove,num_of_test,finger_threshold);
  

The gesture training function, GestureTraining, can now be called:

  //Construct a class for the training data
  GRMultiContext * m_context_ptr = new GRMultiContext();
  
  //Call gesture training function
  m_context_ptr = GestureTraining(m_context_ptr,glove,flex_min, flex_slope, which_hand,num_to_train);

Finally, the trained data set can be written out to a file and the serial port closed:

  // Save the gesture training data into a file
  SaveTrainingData(m_context_ptr);
   
  // Printout the training results
  cout << m_context_ptr << "\n";

  // Clean up
  delete m_context_ptr;
  glove.GRclosePort ();

The GestureTrain executable uses command line arguments to set the user definable parameters; allowable parameters are:

GestureTrain [ device_name ] [ -l ] [ -t num ] [ -n num ] [ -f num ] [ -h ]

where:
device_name = The glove serial port device name, the default is /dev/ttyd2

-l = Left hand training, the default is Right hand training

-t num = num is the number of initial calibration samples, the default is 300

-n num = num is the number of training samples for each gesture; default 5

-f num = num is the minimum noticeable finger bending, the default is 35

-h = Help for command-line definitions

For example if we wanted to train some left handed gestures using a glove connected on the serial port /dev/ttyd3 and only taking two training samples for each gesture we would type the following: > xinyu@kodiak/GloveGRASP> train /dev/ttyd3 -l -n 2

The output produced would be:

53 grof@kodiak/demo> GestureTrain

GestureTrain - gesture training example

Opening port: /dev/ttyd2

Hit return when ready for calibration ...

Glove calibration starts, strech and relax your hand .....

Summary of the glove calibration

    Max Flex are:    255 255 255 255 255
    Min Flex are:      0 0 0 0 0
    =================================================

During glove calibration the user needs to repeatedly make alternating flat hand and fist gestures so the maximum and minimum bend for each flex sensor can be measured. Following calibration the user can begin training:

  Start Hand Shape Training.......
  Train each gesture for 2 times .....
  Enter the next context name:  collided
  Enter the next gesture name (e.g., one):one
  Form the gesture, hit any key after ready:
  Form the gesture, hit any key after ready:
  
  ------------------  SUMMARY --------------------
  MEAN:      (0)251  (1)203  (2)255  (3)102 (4)104
  TOLERANCE: (0)4  (1)3  (2)0  (3)10  (4)9
  MAXIMUM:   (0)255  (1)205  (2)255  (3)112 (4)113
  MINIMUM:   (0)249  (1)200  (2)255  (3)95  (4)99
  
  Add gesture to current context (default YES)? (y/n):y
  Enter the next gesture name (e.g., one): two
  
  Form the gesture, hit any key after ready:
  Form the gesture, hit any key after ready:

  -------------------  SUMMARY --------------------
     MEAN:      (0)239  (1)206  (2)98  (3)63  (4)103
     TOLERANCE: (0)5  (1)6  (2)13  (3)5  (4)9
     MAXIMUM:   (0)244  (1)212  (2)111  (3)63 (4)112
     MINIMUM:   (0)234  (1)200  (2)88  (3)58  (4)97
     
     Add gesture to current context (default YES)? (y/n):n

At the start of training users will be prompted to enter the new context name and the name of the first gesture they are training for. Users are prompted for the name of each gesture before the training samples for that gesture are taken. After all the training samples have been taken a summary is printed out, showing the mean, tolerance, maximum and minimum raw finger values measured. The numbers in the brackets show which finger the data is for. For example, the 'one' gesture above is a single finger point so fingers (3) and (4), the index and thumb, have low bend values while the other three fingers are high. The mean value shows the average finger bend while the tolerance is the maximum deviation of samples.

If the users want to add an additional context or quit the training program they just need to type 'n' at the prompt "Add gesture to current context (default YES)? ".

   Add gesture to current context (default YES)? (y/n):n
  Add new context (default YES)? (y/n):y
  Enter the next context name:  free
      Enter the next gesture name (e.g., one): fist
    
    Form the gesture, hit any key after ready:
    Form the gesture, hit any key after ready:

    -------------------  SUMMARY --------------------
        MEAN:      (0)221  (1)218  (2)255  (3)255 (4)124
        TOLERANCE: (0)5  (1)3  (2)0  (3)0  (4)8
        MAXIMUM:   (0)226  (1)221  (2)255  (3)255 (4)132
        MINIMUM:   (0)217  (1)216  (2)255  (3)255 (4)120
        
    Add gesture to current context (default YES)? (y/n):y
    Enter the next gesture name (e.g., one): flat
    
    Form the gesture, hit any key after ready:
    Form the gesture, hit any key after ready:

  -------------------  SUMMARY --------------------
    MEAN:      (0)1  (1)29  (2)135  (3)65 (4)33
    TOLERANCE: (0)2  (1)1  (2)1  (3)1  (4)8
    MAXIMUM:   (0)3  (1)30  (2)136  (3)66 (4)41
    MINIMUM:   (0)0  (1)28  (2)134  (3)64 (4)29
    
    Add gesture to current context (default YES)? (y/n):n
    Add new context (default YES)? (y/n):n
    Save the training data (default YES)? (y/n):y
    Enter the file name:  test.txt

After the training data has been saved out to a file, the contents of the file are printed to the screen. There is currently no way to go back and retrain gesture in a template file, or reload a template file to add additional gestures, but the files can be edited by hand if needed. The template files are simply a list of the contexts and gestures with their corresponding mean and tolerance values. Each context ends with a semi-colon except for the last which has two semi-colons.

Collide

      fist  1     148   3     255   0     255   0     243   1     226   0
     point  1     238   1       0   0     255   0     255   0     213   1
      flat  1     201   7       0   0       0   0       0   0       0   0 ;

Free

      grab  1     181  24     255   0     255   0     251   4     241   1
       fly  1     200   6       2   7     254   1     255   0     207   4
 twofinger  1     185   4       0   0       0   0     230  25     219  12 ;;

The file test.txt in the demo directory is a sample template file.

6.2 Gesture Recognition

Once a template file has been created it can be used by the GloveGRASP gesture recognition functions for gesture recognition. These recognition functions are defined in GRTrainRecog.h and include:

bool SingleGestureRecog(char * matched_gesture, bool which_hand, int finger_threshold, int recog_threshold, GRInt5& in_flex_min, GRFloat5& in_flex_slope, GRDeviceData & fval,
GRCandidate * candi_ptr, GROneContext *context_ptr );
GROneContext * ContextSwitch (char * in_buffer, GRMultiContext * m_context_ptr, GROneContext* context_ptr )
GRMultiContext * LoadTrainingData(GRMultiContext* m_context_ptr);

SingleGestureRecog is the function called to actually perform the gesture recognition. It returns TRUE if the recognition was successful, FALSE otherwise. After a successful recognition the pointer candi_ptr will point to the name of the gesture closest to the input raw data. In order to prevent incorrect recognition the values finger_threshold and recog_threshold can be set to the minimum change in raw data that must be exceeded before the gesture recognition will be attempted. The template matching uses the gestures defined in the currently active context. This current context is pointed at by context_ptr and can be changed by calling the ContextSwitch function. In this function context_ptr is the pointer to the current context, in_buffer the pointer to the new context name and m_context_ptr the pointer to the entire gesture template structure. LoadTrainingData is used to load in a gesture template structure.

There are two threshold levels set to minimize the chance of an incorrect recognition, or a recognition when no gesture was made at all. The first, finger_threshold is used to detect when individual fingers have changed significantly since the last reading. This means that even subtle changes in gesture involving only a single finger can be detected. The second threshold recog_threshold is used to detect changes in the entire gesture. The lower these threshold values are the more likely the gesture recognition will incorrectly identify gestures. However high threshold values will make the system less likely to recognize valid gestures when they do occur.

6.2.1 Gesture Recognition Sample Code

The sample file GestureRecog.cxx shows how these functions can be used in a simple gesture recognition application.

First, the glove port needs to be opened, the hand calibrated and a previously trained gesture template loaded in:

  if ( (glove.GRopenPort(filename)) == GR_ERROR) {
     cerr << "Couldn't open glove device, exit\n";
     exit (EXIT_FAILURE);    // couldn't open file<
  } 
  else if ( (glove.Reset()) == GR_ERROR ) {
     cerr << "Couldn't reset glove device, exit\n";
     exit (EXIT_FAILURE);
  }
  
  cout <<"Ready for Calibration(y/n): "
  cin >>c;
  cout << endl;
  FingerCalibration(flex_slope,flex_min, glove,num_of_test, finger_threshold);
 
  // Construct a class for the training data
  GRMultiContext * m_context_ptr = new GRMultiContext();

  // Read in the training data file and print it out
  m_context_ptr = LoadTrainingData(m_context_ptr);
  cout << "Training data file: " << m_context_ptr << '\n';
  

Once the training file has been read the gesture recognition function can be either be called continuously or at the user's request.

  // Repeating gesture recognition
  cout << "\n==========================================\n";
  cout << "Gesture recognition using default context:\n";
  assert ( (context_ptr = m_context_ptr->GetHead()));

  forever {// Loop to detect each gesture input
  	if(!is_continue) {
  	    cout << "--------------------------------\n";
  	    cout << "Form gesture, hit return after ready ....";
	    cin.getline(answer, answer_len);
	}
	
	// Call the gesture recognition function<
	glove.ReadCheck(fval);
	SingleGestureRecog( matched_gesture, which_hand, finger_threshold, recog_threshold,flex_min,flex_slope,fval,candi_ptr,context_ptr);
	printf("RECOGNIZED:   %s\n",matched_gesture);
  }

The user can also change the recognition context between gestures:

  if(!is_continue) { 
    cout << "Change context (default No)? (y/n): ";
    cin.getline(answer, answer_len);
    
    if ((strcmp(answer,"y")==0)||(strcmp(answer,"Y")==0)) {
              cout << "New context name:  ";
              cin >> new_context_name;
              cin.get(); 
              context_ptr = ContextSwitch(new_context_name,m_context_ptr,context_ptr);
           
              //Print out the new context
              GRprintConext(context_ptr);
     } // if(getchar())
  } // if(!is_continue) 

Command line options are used to set the parameters in the GestureRecog executable. These options are:

GestureRecog[ device_name ][ -l ][ -d ][ -f num ][ -r num ][-h ]

where:
device_name = The glove serial port device name, the default is /dev/ttyd2

-l = Left handed recognition; the default is Right handed recognition

-d = Discrete or Continuous recognition; the default is continuous.

-f num = num is the minimum finger threshold level; the default is 35

-r num = num is the minimum recognition threshold; the default is 300

-h = Help for command-line definitions

If the program is set to run with continuous recognition then it performs real-time recognition, continuously streaming the currently recognized gesture. Unfortunately with this setting it is impossible for the user to enter a change of context. However with discrete recognition the user can form a gesture and then hit a key to do the recognition or type in a new context to change the recognition context.

If we wanted to use the program to do right handed gesture recognition with a glove connected to serial port /dev/ttyd3 and setting the threshold for gesture recognition at 400, we would enter the following command line:

kodiak/demo> GestureRecog /dev/ttyd3 -d-r 400

The output produced would be:

GestureRecog - sample gesture recognition program

Opening port: /dev/ttyd3

Hit return when ready to calibrate the glove..
Glove calibration starts, stretch and relax your hand .....

Summary of the glove calibration
   Max Flex are:    233 255 255 255 255
   Min Flex are:      8 0 0 0 0

Enter the context file name:  test.txt
Training data file:   

Once the template file name has been entered in the gestures contained in that file will be printed to screen:

Collide

      fist  1     148   3     255   0     255   0     243   1     226   0
     point  1     238   1       0   0     255   0     255   0     213   1
      flat  1     201   7       0   0       0   0       0   0       0   0 ;

Free

      grab  1     181  24     255   0     255   0     251   4     241   1
       fly  1     200   6       2   7     254   1     255   0     207   4
 twofinger  1     185   4       0   0       0   0     230  25     219  12 ;;

Now the user can attempt discrete gesture recognition by forming the test gesture and hitting a key to initiate the recognition. The recognized gesture will be printed on screen.

===================================================
Gesture recognition using default context:
----------------------------------------------
Form gesture, hit return after ready ....
RECOGNIZED:  Non
Change context (default No)? (y/n): n
----------------------------------------------
Form gesture, hit return after ready ....
RECOGNIZED:   fist
Change context (default No)? (y/n): y

Using discrete recognition they can also change the recognition context, causing the new context and gesture set to be displayed.

New context name:  Collide
New context:
      grab  1   181  24   255   0   255  0   251   4   241   1
      fly  1   200   6     2   7   254   1   255   0   207   4
      finger  1   185   4     0   0     0  0   230  25   219  12 ;
----------------------------------------------
Form gesture, hit return after ready ....
RECOGNIZED:   grab
Change context (default No)? (y/n): n
----------------------------------------------
Form gesture, hit return after ready ....
RECOGNIZED:   fly
Change context (default No)? (y/n): 

91 grof@kodiak/demo>

6.3 Choosing Your Gesture Set

The set of gestures chosen largely determines the accuracy of the gesture recognition. Several factors should be considered when choosing gestures for your application, including the limitations of the hardware, the difficulty of forming certain gestures, and the most natural gestures for your particular application. For example in a modeling application a fist gesture would be a natural choice for picking objects up - it is easily detected by the hardware, a natural action and simple to remember.

For applications where gestures are going to be used as the primary command input, the developer should try and use context switching as much as possible to minimize the number of gestures that a user must remember. The average user will have difficulty in remembering more than six gestures, however these could be translated into dozens of commands by using different contexts.

It is also wise to use gestures that are markedly different from one another, for example gestures that bend different fingers such as a fist and pointing hand. Not only are these gestures easier to remember, but also almost impossible for the recognition system to misinterpret. Figure 3.0 shows some typical gestures that produce good recognition rates. Note how these getsures are as different as possible to ensure good recognition results.

Fist	Flat Hand	Horns
One Finger Point	Two Finger Point	Three Finger Point

7 GRASPmodel

7.1 Introduction

GRASPmodel is a more complex OpenGL-based modelling application which shows how gesture recognition can be used in practice. GRASPmodel allows the user to use two handed input to create simple 3D scenes from a set of four primitives (cone, box, sphere, cylinder). The right hand is used for gestural commands while the left is used for 3D cursor control and specification of command parameters. Two separate interaction contexts are defined so although there are almost a dozen available commands the user needs only remember six gestures.

7.2 Installation

All the relevant files for the GRASPmodel are found in the /demo/GRASPmodel directory, and should include:

GRASPmain.cxx GRASPdummy.cxx GRASPObjectList.cxx
GRASPfastrak.cxx GRASPlist.cxx trakit.c
seriallib.c demo.txt Makefile
README

and the header files:
aux.h GRASPdefines.h GRASPmain.h rasters.h
GRASPdummy.h seriallib.h trakit.h

GRASPmodel uses the OpenGL Programming Guide Auxilary Library for some of it's graphics functions. This library can be obtained for free from the sgigate.sgi.com ftp site. After ftping to the site type the following commands;

cd pub/opengl
binary
get opengl.tar.Z
bye

The file opengl.tar.Z is compressed, but the auxilary library can be extracted using the following commands:
uncompress opengl.tar
tar xf opengl.tar

Sample OpenGL programs and the auxilary library will then be created as subdirectories from the current directories. The makefile will need to be altered to point to the auxilary library directory before GRASPmodel can be compiled. The relevant line in the makefile is marked.

In addition to software, the GRASPmodel code requires a right handed 5DT glove and a Polhemus Fastrak with at least two sensors. If a Fastrak is not available then the tracking functions in GRASPmain.cxx will need to be modified to use another six degree of freedom tracker.

GRASPmodel is designed to be used with the fastrak tracker source mounted upside down, the axis aligned as shown below in figure 4.0. The first fastrak sensor should be mounted on the back of the right hand glove, and sensor two held in the left hand. It may be useful to mount the source beneath a flat surface such as a piece of wood to give the user a convenient working surface.

7.3 Training

GRASPmodel takes advantage of GloveGRASP's context sensitive gesture recognition. There are two contexts defined, "Collide" for when the users virtual hand is collided with an object in the scene, and "Free" for when it isn't. These contexts are dynamically changed according to where the user's hand is while the application is running.

There are a total of 11 gesture commands that can be issued across these contexts, from a set of six gestures. The table below shows the six gestures and their function in each of the contexts. Note how the same gesture is interpreted differently according to the interaction context.

Gesture	Context: Collide	Context: Free
Fist	Pick	Menu
One Finger Point	Copy	N/A
Two Finger Point	Scale Object	Scale World
Three Finger Point	Color Object	Color Object
Flat hand	Drop	Hide Menu
Horns	Delete Object	Delete Scene

These gestures are the same as shown in figure 3.0 previously. As can be seen they have been chosen to be as simple as possible and to minimize the recognition errors.

Before GRASPmodel can be run these 11 gestures should be trained using the Gesturetrain program. Other gestures can be substituted for those above, but the context and gesture names can only be different if the GRASPmain.cxx source code is modified.

The table below shows the gesture names for each of the gestures. The file demo.txt is an example of the type of training file that could be used with GRASPmodel.

Gesture	Context: Collide	Context: Free
Fist	Pick	Menu
One Finger Point	Copy	N/A
Two Finger Point	ScaleObj	ScaleWorld
Three Finger Point	Colour	Colour
Flat hand	Drop	Drop
Horns	Reset	Delete

7.4 Running the Application

After gesture training, the GRASPmodel executable can be compiled by simply typing make GRASPmodel:
kodiak/GloveGRASP/demo/GRASPmodel>make GRASPmodel

When the code has been compiled, type GRASPmodel to run the application. Enter the serial ports the glove and fastrak are connected to and the desired display size.

kodiak/GRASPmodel> GRASPmodel

GRASPmodel - gesture recognition demo

Input the desired display size (width,height): 800 600
Input the fastrak port : /dev/ttyd3

Reseting fastrak
10..9..8..7..6..5..4..3..2..1..0..DONE
tracker started
Are you ready to boresight the fastrak (y/n)?y

Input the glove serial port : /dev/ttyd2
Glove opened on port /dev/ttyd2

Boresighting is used to zero the angles in the tracker. The user should hold their gloved hand flat and aligned with the tracker axis before boresighting. Once the serial ports have been opened and data read to make sure the devices are working correctly, glove calibration takes place and the gesture template file loaded.

Hit return to begin finger calibration
Glove calibration starts, stretch and relax your hand .....
. . . . . . . . . . . . . . . . . . . . .
Hit return to set the initial hand position..
Summary of the glove calibration

    Max Flex are:    255 255 255 255 250
    Min Flex are:     11 10 0 0 0

The application will now be running - users should see a graphical hand on the screen which moves and responds to their real hand motions. Figure 5.0 shows a screen snapshot of the application running. The title bar at the top of the screen shows the currently active context and the last recognized gesture.

There are a number of interface elements that will appear in response to user gestures. The following sections explains each gestural command, and the corresponding element that should appear on-screen. In all cases the right (gloved) hand is used for inputing gestural commands while the left hand, holding the second tracker, is used for specifying the parameters for these commands. The virtual hand should follow the right hand motion at all times. For some interface elements a white 3D cursor will appear that will following the users left hand motion. However the 3D cursor movement is often constrained within a particular region or direction to make the interface more intuitive.

When the user's virtual hand collides with an object a wireframe bounding box the same color as the object will appear. The size of the box depends on the objects maximum dimension so it may often be several times larger than the object itself. It is collisions with other objects that changes the recognition context from "Free" to "Collide".

A very simple algorithm is used for collision detection so bounding boxes may appear even when the virtual hand is not in contact with an object. This is especially true for large objects.

7.5 GRASPmodel Interface Elements

Context: FreeSpace

When a fist is made in free space a menu of four object primitives will appear, as shown in the figure above. The 3D cursor also appears and is constrained to move along the x axis between the object primitives. Moving the cursor causes object selection; the object the cursor is currently colliding with is selected and a copy of it attacted to the user's virtual hand. The menu remains active until the Flat hand gesture is made.

Context: Collided

When the user collides with an object and makes a fist, the object is picked up and becomes attacted to the palm of the virtual hand. It with follow the user's hand motions until released it by making a Flat hand gesture.

Context: Collided or FreeSpace

The Flat hand gesture is used to drop objects that are attached to the hand, or remove interface elements such as the object primitive menu from the screen. If an object is dropped onto another existing object it becomes a child of that object and is attacted to it. Picking the parent will cause both objects to be attached to the virtual hand.

Context: Collided

When the hand is collided with an object a one fingered point makes a copy of the object and attaches it to the user's palm as shown in the figure above. If the object has children they will also be copied. This gesture has no effect when the hand is in free space.

Context: FreeSpace

When the user makes a two fingered point in free space a widget for scaling the entire scene will appear, as shown. The white 3D cursor follows the motions of the second fastrak sensor, and is constrained to move along the x axis below the two arrows. The cursor starts in space between the arrows. When it moves below the blue up arrow the scene will scale up, all the objects getting uniformly larger and the distances between them increasing proportionately. Moving the cusor below the down red arrow has the opposite effect. Scaling is stopped when the cursor is moved into the space between the arrows, or the widget deleted by making a flat hand gesture.

Context: Collided

This will cause scaling to occur on the currently selected object. A cube shaped scaling widget will appear as shown in the figure. The widget has three different colored regions, black, red and blue. The innermost black region is where the white 3D cursor starts. While the cursor is in this region there is no change of object scale, but when the user moves the cursor into the adjacent red region the currently selected object will continuously change scale along one or more axis. Scaling can be stoped by moving the 3D cursor back to the black region or removing the widget by making the flat hand gesture. In order to scale the object along one particular axis the user needs only move the 3D cursor along that axis. Objects can be scaled along multiple axes by moving the cursor diagonally through space. If the 3D cursor is moved into the blue region, then continuous shrinking will occur instead. The 3D cursor is constrained to move within the scale widget, regardless of the user's real hand motions.

Context: FreeSpace

A color cube will appear with the 3D cursor located at the origin (0,0,0). As the user moves the 3D cursor through the cube it will change color to that of it's current location. X,Y,Z values in the colour cube are mapped onto R,G,B color values, so the origin (0,0,0) is black while the far corner (1,1,1) is white. Only three sides of the cube are shown, but colors within the cube volume are shown on the 3D cursor. If the user's virtual hand collides with another object when the colour cube is active the object will change colour to that currently selected by the 3D cursor. The colour cube can only be dismissed by making the flat hand gesture.

Context: Collided

This causes the same color cube to appear as above. However since the user is currently collided with an object, both that object and the 3D cursor will change colour as the cursor is moved through the color cube. As before,the colour cube can only be dismissed by making the flat hand gesture. Figure 10.0 shows the appearance of the color cube.

Context: FreeSpace

All of the current scene will be deleted and the interface reset into its original state. This gesture is deliberately difficult to make to prevent accidental deletions.

Context: Collided

The currently selected object will be deleted.

7.6 Extending the Application

GRASPmodel is a very simple application that can be extended in numerous ways. There are several fundamental functions missing, namely the ability to save or read in files, to scale the children of objects when the parents are scaled, and to change the users viewpoint. Other improvements could include the use of right and left handed gloves together for gestural input, more intuitive interface widgets and better collision detection and object selection. Six gestures may be too many for novice users to remember, and by adding extra contexts the size of the gesture set could be reduced while retaining the same command set. Despite these limitations, GRASPmodel shows how intuitive gestural input can be for 3D interactive graphics applications.

Full source code for GRASPmodel is included so the programmer can extend the application in any way they desire. This code and any applications based on it can be freely distributed with no licensing fees. The GRASPmodel files contain the following code:

GRASPmain.cxx

Contains the main module, sets up the virtual world and controls all user interactions. Once the devices and graphics have be initialized GRASPmain spends most of it's time in the display function, display(void). This function loops endlessly, reading the input devices, checking for collisions and context changes, recognizing gestures and updating the graphics. GRASPmain also contains all the functions called in response to gestural commands, such as changing object size and color or adding and deleting objects.

GRASPObjectList.cxx

Contains the object linked list class and method definitions. All the objects in the scene are add or removed from a single linked list. GRASPObjectList contains functions for adding and deleting list elements, and searching the list. There is also a smaller linked list used for the objects on the object primitive menu.

GRASPdummy.cxx

Contains the GObject and GHand class definitions. GObject is the base class for all the objects in the virtual world and GHand a subclass with methods for drawing a virtual hand, and glove initialisation functions. The cone, box, cylinder and sphere objects are also all sub-classes of the GObject class.

GRASPlist.cxx

Contains the color and geometry definitions for all the interface widgets.

rasters.h

Defines the raster bitmap for the font used in the title bar.

GRASPfastrak.cxx

Contains fastrak functions for initializing and shutting down the fastrak. GRASPfastrak uses functions from trakit.c and seriallib.c for actually reading the fastrak data.

trakit.c

A set of fastrak device driver functions for reading raw fastrak data. This includes fucntions for reading from multiple sensors, boresighting sensors and setting the fastrak into different operating modes.

seriallib.c

A simple serial port library that is used by trakit.c to access the serial port the fastrak is connected to.

Appendix