SmartBody : On-Line Retargeting

On-line retargeting does not create a new motion instance. Instead, it works as a filter when playing back motions to convert the joint angles to suitable values for the new target character on-the-fly. This feature simplifies the overhead to process each motion for each new character, but at the same time incurs some performance penalty at run-time. This also allows a rapid transfer of the default behavior sets such as locomotion, gesture, jumping, and other motions in SmartBody to a new character.

Retarget Viewer

The easiest way to test out the SmartBody on-line retargeting capability on a new character is to use Retarget Viewer. It can be used to transfer the behavior sets in SmartBody to a custom character. It can be triggered automatically when drag-and-drop in a skeleton file ( .sk for SmartBody skeleton file , or .dae for OpenCollada file ). It contains two tab windows, the Joint Mapper and the Behavior Set.

When a character first being retargeted, our system will automatically guess a correct joint name mapping to our standard skeleton. Sometimes the guessing might not get all the joint names correct, or the user may want to fine tune the joint name mapping result. The Joint Mapper let user adjust the joint name mapping result interactively. For example, the above is the screenshots from our system. The left image shows the Joint Mapper Window, which is used to adjust the guessed joint name mapping for 'mascot.dae' ( with its mesh shown on the right image ). We can see that some joint names are already filled out automatically from joint name guessing. The user can then make further changes in the name mapping using the drop-down list for each joint.  Once the joint name correspondences between the custom skeleton and our standard skeleton are established, we can move on to retarget the desired behavior animation set to the new character.

The Behavior Set Window let the user select the desired animation set to be used for the new custom character. The selected behavior will be automatically retargeted and applied on the new character. The following images shows the results of retargeting the "Punch" motion blend..

API Function For On-Line Retargeting

The above functionality can also be achieved by using SmartBody Python API. The following shows the steps to build the on-line retargeting so a source motion for a source skeleton can be playback normally on a new target skeleton.

1. We first need to find the correct joint name mapping for both the source skeleton, source motions, and target skeleton. The joint name guessing can be done using Automatic Skeleton Mapping.
   
   

2. We also need to tell SmartBody which skeleton is the default skeleton for the source motion, or the motion blend that contains the motion. This way SmartBody can properly convert the source motion to use on a target skeleton based on its default skeleton. 
   It can be done by API call : 

   motion.setMotionSkeletonName(skelName)

   The equivalent API call for motion blend is :

   blend.setBlendSkeleton(skelname)

   
   

3. Once we have proper joint names and default skeleton setup, we can create a retarget instance between source and target skeleton. The API calls look like this :

   retarget = retargetManager.getRetarget(srcSkelName,tgtSkelName)
   retarget.initRetarget(endJoints,relativeJoints)

   This code segment will create a retarget instance from srcSkelName to tgtSkelName and initialize it. Here, endJoints and their children will be skipped during the whole retargeting, and relativeJoints will be skipped during skeleton alignment stage. In most cases, you want to set wrist/ankle joints to be the end joints so the fingers and foots are not adjusted during retargeting. Also, it is usually not recommended to re-align the spine joints. Therefore they should be put in relative joints.
   
   

4. After creating the retarget instance, we can create the new character using target skeleton, and playback the source motion normally on this character. The suitable retarget instance will be used to convert joint angles automatically for the new character.

Example

The following code snippets demonstrate an example of setting up on-line retargeting.

# want to retarget a motion that is created for ChrBrad.sk to common.sk
srcSkelName = "ChrBrad.sk"
tgtSkelName = "common.sk"
motionName = "ChrBrad@Guitar01"

# apply zebra2 joint map for ChrBrad
scene.run("zebra2-map.py")
zebra2Map = scene.getJointMapManager().getJointMap("zebra2")
bradSkeleton = scene.getSkeleton("ChrBrad.sk")
zebra2Map.applySkeleton(bradSkeleton)
zebra2Map.applyMotionRecurse("ChrBrad")

# set the correct corresponding skeleton for Guitar motion 
motion = scene.getAssetManager().getMotion(motionName)
motion.setMotionSkeletonName(srcSkelName)
 
# these joints and their children are not retargeted
endJoints = StringVec();
endJoints.append('l_forefoot')
endJoints.append('l_toe')
endJoints.append('l_wrist')
endJoints.append('r_forefoot')	
endJoints.append('r_toe')	
endJoints.append('r_wrist')
# these joints are skipped during skeleton alignment
relativeJoints = StringVec();
relativeJoints.append('spine1')
relativeJoints.append('spine2')
relativeJoints.append('spine3')
relativeJoints.append('spine4')
relativeJoints.append('spine5')
relativeJoints.append('r_sternoclavicular')
relativeJoints.append('l_sternoclavicular')
relativeJoints.append('r_acromioclavicular')
relativeJoints.append('l_acromioclavicular')	
 
# create the retarge instance
retargetManager = scene.getRetargetManager()
retarget = retargetManager.getRetarget(srcSkelName,tgtSkelName)
if retarget == None:
	retarget = 	retargetManager.createRetarget(srcSkelName,tgtSkelName)
	retarget.initRetarget(endJoints,relativeJoints)
# afterward, we can use the motion 'ChrBrad@Guitar01' created for 'ChrBrad.sk' and apply it to another character created by 'common.sk'

Here we have a skeleton 'ChrBrad.sk' and its corresponding motion 'ChrBrad@Guitar01', and we want to apply this motion for a different skeleton 'common.sk'. ( Line 1 – 4 )

In order to do this, we first need to remap the joint names for 'ChrBrad.sk' and its motions. ( Line 6 – 11 )

Then we also need to let SmartBody knows that the skeleton 'ChrBrad.sk' is assocated with the motion 'ChrBrad@Guitar01'. ( Line 14 – 15 )

There are some parameters that can be adjusted for the retargeting. Here, endJoints and their children will be skipped during the whole retargeting, and relativeJoints will be skipped during skeleton alignment stage. We found that by setting wrist/foot as end joints and spinal joints as relative joints give good results in most cases. (Line 17 – 35 )

Once we have everything set up, we can go ahead to create a retarget instance from 'ChrBrad.sk' to 'common.sk'. Once this retarget instance is created, we can start apply the motion 'ChrBrad@Guitar01' normally on any characters created from skeleton 'common.sk'. The on-line retargeting will be triggered automatically to convert the joint angles so the motion looks proper on the new characters. (Line 37 – 42 )

The full example script that demonstrates the above operations can be found in /data/examples/OnLineRetargettingDemo.py