University of Southern CaliforniaUSC
USC ICT TwitterUSC ICT FacebookUSC ICT YouTube

Text To Speech is not functional | General SmartBody Discussion | Forum

Avatar

Please consider registering
guest

sp_LogInOut Log In sp_Registration Register

Register | Lost password?
Advanced Search

— Forum Scope —




— Match —





— Forum Options —





Minimum search word length is 3 characters - maximum search word length is 84 characters

sp_Feed Topic RSS sp_TopicIcon
Text To Speech is not functional
September 9, 2016
2:25 am
Avatar
Member
Members
Forum Posts: 6
Member Since:
September 9, 2016
sp_UserOfflineSmall Offline

Hi All, 

I am trying to implement Test to Speech functionality using Smartbody interface using my own C++ code. 

I have used to following steps to achieve the same: 

1) Obtain an instance of Scene using SBScene

2) Obtain an instance of charecters created using SBCharecter { I have provided face defination with vimes) 

3) Obtain instance of Active MQ using SBVHMsgManager. (server : localhost, port:61616, setEnable: True) 

4) setVoice: Remote , VoiceCode: Microsoft Anna 

5) BML Command:Hi I am Rachel

However, I am stuck at the problem that, 

The BML Commands are getting processed and accepted by smartbody as the nodding feature works correctly. 

However, I am unable to get any voice or lip movement. I have checked all through the code and there does not seems to be any errors. Just the voice and lip movement are not working. 

Please suggest where am I going wrong or if I am missing anything.

September 9, 2016
4:56 pm
Avatar
Admin
Forum Posts: 983
Member Since:
December 1, 2011
sp_UserOfflineSmall Offline

The lip syncing to speech will only work if you have a face definition that includes the speaking facial visemes and you need to load the initDiphone-default.py file which contains all the animation curves for each pair of phonemes.

Ari

September 26, 2016
4:20 am
Avatar
Member
Members
Forum Posts: 6
Member Since:
September 9, 2016
sp_UserOfflineSmall Offline

Thanks Ari for your suggestion. 

However, that might not be the case, as I call SBFaceDefination constructor "SBFaceDefinition(SBFaceDefinition* source)" for setting up the face. It sets up the viseme and AU for the face. 

Please let me know if that is all which i need to setup of iI am missing something in it. 

Regards, 

Atul

September 27, 2016
2:49 pm
Avatar
Admin
Forum Posts: 983
Member Since:
December 1, 2011
sp_UserOfflineSmall Offline

Look at the script in data/examples/AddCharacterDemoRachel.py which shows how to set up the face to reflect lip syncing to speech.

 

Ari

September 30, 2016
6:00 am
Avatar
Member
Members
Forum Posts: 6
Member Since:
September 9, 2016
sp_UserOfflineSmall Offline

we are using the SBFaceDefination to set up the AU and Viseme. Please refer below. 

  • To create a default rachel charecter. 
    • SmartBody::SBCharacter* pCharacter = NULL
    • pCharacter = pScene->createCharacter(strCharName,"");  -> to create the charecter 
    • SmartBody::SBSkeleton* pSkeleton = pScene->createSkeleton(skelName);
    • pCharacter->setFaceDefinition(m_pFemaleFaceDefinition);
    • pCharacter->createStandardControllers();
  • This further calls to the face defination function of smartbody (SBFaceDefination) with following commands. 

int numVisemes = source->getNumVisemes();
for (int v = 0; v < numVisemes; v++)
{
std::string visemeName = source->getVisemeName(v);
SkMotion* motion = source->getVisemeMotion(visemeName);
_visemeMap.insert(std::pair<std::string, std::pair<SkMotion*, float> >(visemeName, std::pair<SkMotion*, float>(motion, 1.0f)));
}

int numAUs = source->getNumAUs();
for (int a = 0; a < numAUs; a++)
{
int auNum = source->getAUNum(a);
ActionUnit* sourceAu = source->getAU(auNum);
ActionUnit* au = new ActionUnit(sourceAu);
_auMap.insert(std::pair<int, ActionUnit*>(auNum, au));
}

 

As I see in the python file, same steps are being used to create the face defination of the agent. 

Please tell me if any steps or process is missing into these. 

September 30, 2016
9:54 am
Avatar
Admin
Forum Posts: 983
Member Since:
December 1, 2011
sp_UserOfflineSmall Offline

Make sure that you specifiy the neutral face pose  as well. The python code shows the proper order:

 

bradFace = scene.createFaceDefinition('ChrBrad')
bradFace.setFaceNeutral('ChrBrad@face_neutral')
bradFace.setAU(1, "left", "ChrBrad@001_inner_brow_raiser_lf")
bradFace.setAU(1, "right", "ChrBrad@001_inner_brow_raiser_rt")
bradFace.setAU(2, "left", "ChrBrad@002_outer_brow_raiser_lf")
bradFace.setAU(2, "right", "ChrBrad@002_outer_brow_raiser_rt")
bradFace.setAU(4, "left", "ChrBrad@004_brow_lowerer_lf")
bradFace.setAU(4, "right", "ChrBrad@004_brow_lowerer_rt")
bradFace.setAU(5, "both", "ChrBrad@005_upper_lid_raiser")
bradFace.setAU(6, "both", "ChrBrad@006_cheek_raiser")
bradFace.setAU(7, "both", "ChrBrad@007_lid_tightener")
bradFace.setAU(10, "both", "ChrBrad@010_upper_lip_raiser")
bradFace.setAU(12, "left", "ChrBrad@012_lip_corner_puller_lf")
bradFace.setAU(12, "right", "ChrBrad@012_lip_corner_puller_rt")
bradFace.setAU(25, "both", "ChrBrad@025_lips_part")
bradFace.setAU(26, "both", "ChrBrad@026_jaw_drop")
bradFace.setAU(45, "left", "ChrBrad@045_blink_lf")
bradFace.setAU(45, "right", "ChrBrad@045_blink_rt")

bradFace.setViseme("open", "ChrBrad@open")
bradFace.setViseme("W", "ChrBrad@W")
bradFace.setViseme("ShCh", "ChrBrad@ShCh")
bradFace.setViseme("PBM", "ChrBrad@PBM")
bradFace.setViseme("FV", "ChrBrad@FV")
bradFace.setViseme("wide", "ChrBrad@wide")
bradFace.setViseme("tBack", "ChrBrad@tBack")
bradFace.setViseme("tRoof", "ChrBrad@tRoof")
bradFace.setViseme("tTeeth", "ChrBrad@tTeeth")

 

The first parameter indicates the action unit or viseme, the second is the facial pose that represents that action unit or viseme. If you leave the second parameter empty, you will set up the channel that needs to be handled by the renderer (say, if you are setting up a blendshape). 

 

If you look at the files marked SBPython.cpp you will see a relationship between the Python function calls and the C++ one. For example, the function 'setViseme' maps to the C++ function SBFaceDefinition::setViseme().

 

Ari

October 5, 2016
3:18 am
Avatar
Member
Members
Forum Posts: 6
Member Since:
September 9, 2016
sp_UserOfflineSmall Offline
Awaiting Moderation

Forum Timezone: America/Los_Angeles

Most Users Ever Online: 733

Currently Online: jiangringgaard7, groth23rowland, mcclain06kearney, bugge59mackinnon, williamson15mogensen, raskrask1, JosephRoper17, heavenlyliar6390
97 Guest(s)

Currently Browsing this Page:
1 Guest(s)

Top Posters:

jwwalker: 80

jyambao: 52

rbaral: 47

adiaz: 30

WargnierP: 29

lucky7456969: 28

mbarros: 28

avida.matt: 26

JonathanW: 24

laguerre: 23

Member Stats:

Guest Posters: 69

Members: 122211

Moderators: 3

Admins: 4

Forum Stats:

Groups: 1

Forums: 5

Topics: 531

Posts: 2495