SmartBody : Configuring Speech

Created by shapiro, last modified by yxu on Jan 30, 2014

Characters can synthesize speech and match that speech with behaviors from either prerecorded audio or from a text-to-speech (TTS) engine. The details of speech BML format and usage can be found at Speech. 

Prerecorded audio 

To make a SmartBody character speak using recorded voice (via a .wav file), it is necessary to create a schedule of phonemes (individual sounds that comprise a spokem word) and the timings of each phoneme that can then be processed by SmartBody, and in turn, lip synched accurately by the character.

 

SmartBody uses an XML format that contains a list of words, their timings, and their phonemes. To use such a file:

 

1) Make sure the character uses 'audiofile' for its voice:

mycharacter = scene.getCharacter("Brad")
mycharacter.setStringAttribute("voice", "audiofile")

 

2) Set the main path to find the sound files (relative paths will prepend the media path):

scene.addAssetPath("audio", "sounds")

 

3) Set the specific directory for your character. If all the sound files are in the same directory, a '.' can be used:

mycharacter.setStringAttribute("voiceCode", ".")

 

4) Place the .wav file, .bml file (containing the phoneme and word schedule) into the directory

 

5) Play the sound via BML. For example, if the .wav file is called 'introduction.wav':

bml.execBML("Brad", "<speech ref="introduction"/>

 

CSLU Phoneme Scheduler

To generate phoneme schedules through prerecorded audio, phoneme alignment code from CSLU Toolkit (http://www.cslu.ogi.edu/toolkit/) is leveraged. You would need to download separately due to their license. Also Cygwin is needed to run the scripts. Following are the detailed instructions:

  • Download CSLU Toolkit, copy /fa from CSLU_ROOT/Toolkit/2.0/apps/fa and /Tcl80 from CSLU_ROOT/Tcl80 to SB_ROOT/tools/CSLUPhonemeScheduler
  • Put all the .wav files and their corresponding transcription .txt files into /files
  • Run auto.sh, .bml files would be generated under /files. 

Note that:

  • .wav files has to be mono track, preferable 16KHZ.
  • file layout can be found at SB_ROOT/tools/CSLUPhonemeScheduler/readme.txt

 

FaceFX Phoneme Scheduler

You can alternatively use commercial software FaceFX to generate phoneme schedules. To find how to configure FaceFX, check FaceFX.