University of Southern CaliforniaUSC
USC ICT TwitterUSC ICT FacebookUSC ICT YouTube

Only Lipsyncing | General SmartBody Discussion | Forum

Avatar

Please consider registering
guest

sp_LogInOut Log In sp_Registration Register

Register | Lost password?
Advanced Search

— Forum Scope —




— Match —





— Forum Options —





Minimum search word length is 3 characters - maximum search word length is 84 characters

sp_Feed Topic RSS sp_TopicIcon
Only Lipsyncing
June 6, 2017
5:58 am
Avatar
Germany
Member
Members
Forum Posts: 24
Member Since:
June 22, 2015
sp_UserOfflineSmall Offline

Hello,

I want to use a smartbody agent just to lipsync a TTS message that is created and replayed on a different system.

The bml command I generate looks as follows:

<speech type="application/ssml+xml">
<lips viseme="_" articulation="1.0" start="0" end="0.03" />
<lips viseme="H" articulation="1.0" start="0.03" end="0.12" />
<lips viseme="Ih" articulation="1.0" start="0.12" end="0.15" />
<lips viseme="D" articulation="1.0" start="0.15" end="0.24" />
<lips viseme="Ow" articulation="1.0" start="0.24" end="0.53" />
<lips viseme="_" articulation="1.0" start="0.53" end="0.63" />
<lips viseme="_" articulation="1.0" start="0.63" end="0.73" />
<lips viseme="Ay" articulation="1.0" start="0.73" end="0.87" />
<lips viseme="Ih" articulation="1.0" start="0.87" end="0.9" />
<lips viseme="BMP" articulation="1.0" start="0.9" end="0.992" />
<lips viseme="R" articulation="1.0" start="0.992" end="1.06" />
<lips viseme="Eh" articulation="1.0" start="1.06" end="1.17" />
<lips viseme="Sh" articulation="1.0" start="1.17" end="1.34" />
<lips viseme="Ih" articulation="1.0" start="1.34" end="1.38" />
<lips viseme="D" articulation="1.0" start="1.38" end="1.59" />
<lips viseme="_" articulation="1.0" start="1.59" end="1.99" />
</speech>

I try to trigger the lipmovement by execBML("Brad", bml), where bml is the bml code above.

However, nothing happens, although everything seems to be setup correctly since I get lipsyncing using the internal TTSRelay.

What am I missing or is it impossible to only do lipsyncing without playing an audiofile along?

 

Thanks for any help or pointing into the right direction
Jonathan   

June 15, 2017
9:37 am
Avatar
Admin
Forum Posts: 983
Member Since:
December 1, 2011
sp_UserOfflineSmall Offline

The data that you see above is output from the TTS system that indicates the phonemes and timings. It is not, in itself, a BML command that would cause the character to produce lip sync to that utterance. The conversion from a set of phonemes and timings into the lip syncing is done internal to SmartBody.

SmartBody has a method through when you give it a speech BML command like this:

 

bml.execBML('*', '<speech type="text/plain">hello</speech>')

it will look for how it does speech (does it get the data from a file and existing sound, or does it use a TTS?). If it uses a TTS, then it will send a command to the TTS system, get back the data that you are showing, then play the speech and lip sync to it. SmartBody not only produces the lip animation, but also plays the sound file that was generated.

If you want to have SmartBody use a different TTS system, then I suggest writing your own TTSRelay and interfacing with it directly - that way SmartBody won't know the difference.

Does this answer your question?

Ari

June 23, 2017
1:41 am
Avatar
Germany
Member
Members
Forum Posts: 24
Member Since:
June 22, 2015
sp_UserOfflineSmall Offline

Hello Ari,

the thing is that I create the TTS on a different machine connected via network (that is also responsible for replaying the sound, but not the animation) The creation of the sound etc. is done in my own application and the above visemes are returned (that I tuned to look like the stuff Smartbody produces). 

So far I managed to feed them back into smartbody by using pCharacter->setVoice("audiofile") and then dumping the visemes into a .bml file and calling execBML(strCharName, "<speech ref="dump.bml"/>")

This seems to work also if no .wav file is present and I can sync my sound "sound" events. However, this seems a bit too hacky and is somehow not working under Linux. Any other thoughts how that could be realized?

 

Thank you
Jonathan

June 24, 2017
6:48 am
Avatar
Admin
Forum Posts: 983
Member Since:
December 1, 2011
sp_UserOfflineSmall Offline

The process you described is the way that SmartBody works. Not sure why it wouldn't work on linux; the communication is the same. 

When a sound is played there is an event that is sent. To synchronize, you could respond to that "sound" event.

 

Ari

June 29, 2017
2:26 am
Avatar
Germany
Member
Members
Forum Posts: 24
Member Since:
June 22, 2015
sp_UserOfflineSmall Offline

Hello Ari,

I do not want to play the sound on the same machine (cluster) that Smartbody is running on, so for the .bml there does not exist a .wav file. Windows seems to handle this quite well.

However, the main question was whether there exists a way to hand over this viseme data (and only that, no sound) to smartbody and then get some kind of event when it starts the viseme animation. Writing the viseme data into a file first seems a bit cumbersome (and given the cluster environment even failure-prone) to me.

But if this is the only way to do it without having to adapt Smartbody, that is probably what I will have to use.

 

Thanks
Jonathan

June 29, 2017
9:26 am
Avatar
Admin
Forum Posts: 983
Member Since:
December 1, 2011
sp_UserOfflineSmall Offline

You can have SmartBody send BML events by doing this:

 

scene.setBoolAttribute("bmlstatus", True)

Then create an event handler to look for events called 'bmlstatus' which look something like this:

bmlstatus blockstart ...

bmlstatus blockend ...

bmlstatus syncpointprogress ...

 

Your event handler can then react to the blockstart message.

 

Ari

Forum Timezone: America/Los_Angeles

Top Posters:

jwwalker: 80

jyambao: 52

rbaral: 47

adiaz: 30

WargnierP: 29

lucky7456969: 28

mbarros: 28

avida.matt: 26

JonathanW: 24

laguerre: 23

Member Stats:

Guest Posters: 67

Members: 99543

Moderators: 3

Admins: 4

Forum Stats:

Groups: 1

Forums: 5

Topics: 458

Posts: 2401