University of Southern CaliforniaUSC
USC ICT TwitterUSC ICT FacebookUSC ICT YouTube

Replace FaceViewer | General SmartBody Discussion | Forum

Avatar

Please consider registering
guest

sp_LogInOut Log In sp_Registration Register

Register | Lost password?
Advanced Search

— Forum Scope —




— Match —





— Forum Options —





Minimum search word length is 3 characters - maximum search word length is 84 characters

sp_Feed Topic RSS sp_TopicIcon
Replace FaceViewer
November 21, 2014
7:02 am
Avatar
Member
Members
Forum Posts: 20
Member Since:
November 18, 2014
sp_UserOfflineSmall Offline

Hello, 

How can I modify FaceViewer.cpp so that, rather than getting the AU units from the FaceView window, the AU units are obtained from another source in real-time? 

In addition, how can I add features to be tracked on the head and torso that aren't built-in? For example, rather than tracking AU units on the face, I could track another set of points. 

Thank you for your help. 

November 22, 2014
12:34 am
Avatar
Admin
Forum Posts: 983
Member Since:
December 1, 2011
sp_UserOfflineSmall Offline

You can write a controller or a service which sets the appropriate state information on the joints/bones/whatever on the character.

Controllers are per-character control mechanisms, services can affect more than a single character

You can look at smartbody/src/sb/SBFaceShiftManager.h and .cpp for an example of this.

 

Ari

November 25, 2014
8:57 am
Avatar
Member
Members
Forum Posts: 20
Member Since:
November 18, 2014
sp_UserOfflineSmall Offline

I've downloaded SmartBody twice and can't find SBFaceShiftManager.h. Do you have several releases of the code?

I am on a Windows machine and I downloaded the Regular Toolkit, Third Party Installer and vhtoolkitUnity Supplement.

November 26, 2014
3:59 pm
Avatar
Member
Members
Forum Posts: 20
Member Since:
November 18, 2014
sp_UserOfflineSmall Offline

I want to have the avatar respond to input in another program in real-time. I have two questions:

1. Can I load a Python program in real-time in the sbgui from a command prompt or external piece of software? From my understanding, I can only load one Python program at a time by going File>Load from Script. 

2. If I wanted to modify or build-on the FaceView code so that, rather than waiting for input from a dial, the AU units would change based on the input from another script written outside of SmartBody, how could I do that?

I am having difficulty navigating your code and figuring out what I should add and where. Can you clarify what these scripts do and how they relate to one another:

-me_ct_face

-SBFaceDefinition

-FaceViewer

-FL_Value_Slider   

Are there other pieces of code or scripts that you would recommend I look at?

I am thought I could add the following at the bottom of SBFaceDefinition:

void SBFaceDefinition::addNewAU(int auNum, double auVal, const std::string& motionName) {

 motion = SmartBody::SBScene::getScene()->getAssetManager()->getMotion(motionName);

 au = new ActionUnit(auNum, motion);

 SBFaceDefinition::addAU(int auNum, ActionUnit* au);

 ∕∕another line indicating how much the AU unit is incremented or decremented which may involve a map based on what I have understood from your code 

}

I`m not sure whether this will work or how it fits in with the rest of your code. Can you provide some insight? Is this feasible or is there some flaw in my logic? I initially thought that this would be a quick modification since I would be building on pre-existing functionality (namely the FaceViewer functionality) but perhaps I am mistaken.  

November 26, 2014
6:50 pm
Avatar
Admin
Forum Posts: 983
Member Since:
December 1, 2011
sp_UserOfflineSmall Offline

dilbert said
I've downloaded SmartBody twice and can't find SBFaceShiftManager.h. Do you have several releases of the code?

I am on a Windows machine and I downloaded the Regular Toolkit, Third Party Installer and vhtoolkitUnity Supplement.

The FaceShiftViewer is in the lastest SVN release, but not in the SDK release yet.

 

You can look at it on the SourceForge repository here:

 

http://sourceforge.net/p/smart.....dy/src/sb/

 

Ari

November 26, 2014
7:16 pm
Avatar
Admin
Forum Posts: 983
Member Since:
December 1, 2011
sp_UserOfflineSmall Offline

dilbert said
I want to have the avatar respond to input in another program in real-time. I have two questions:

1. Can I load a Python program in real-time in the sbgui from a command prompt or external piece of software? From my understanding, I can only load one Python program at a time by going File>Load from Script. 

2. If I wanted to modify or build-on the FaceView code so that, rather than waiting for input from a dial, the AU units would change based on the input from another script written outside of SmartBody, how could I do that?

I am having difficulty navigating your code and figuring out what I should add and where. Can you clarify what these scripts do and how they relate to one another:

-me_ct_face

-SBFaceDefinition

-FaceViewer

-FL_Value_Slider   

Are there other pieces of code or scripts that you would recommend I look at?

I am thought I could add the following at the bottom of SBFaceDefinition:

void SBFaceDefinition::addNewAU(int auNum, double auVal, const std::string& motionName) {

 motion = SmartBody::SBScene::getScene()->getAssetManager()->getMotion(motionName);

 au = new ActionUnit(auNum, motion);

 SBFaceDefinition::addAU(int auNum, ActionUnit* au);

 ∕∕another line indicating how much the AU unit is incremented or decremented which may involve a map based on what I have understood from your code 

}

I`m not sure whether this will work or how it fits in with the rest of your code. Can you provide some insight? Is this feasible or is there some flaw in my logic? I initially thought that this would be a quick modification since I would be building on pre-existing functionality (namely the FaceViewer functionality) but perhaps I am mistaken.  

1. You can call:

 

scene.run("myfile.py")

to do that. In order for SmartBody to know where the file is, you'll first have to add:

scene.addAssetPath("script", "/path/to/my/file")

and then SmartBody will look for your .py file in that (and other) script directory.

 

2. The way the SmartBody architecture is set up, you can only modify the state of the character (like the AU unit values) in a controller that is part of a character's controller stack. The controllers are evaluated in turn, one by one. Some controllers add to input values (like the head controller), others override the values completely (like the gaze controller or the face controller).

You can write a controller in Python like this:

class MyController (PythonController):

   def init(self, pawn):
      print "Sample controller..."

   def evaluate(self):
      # every time step, get the value from one channel and then modify another
      val = foo() # where the function foo() is where you get the data from

      self.setChannelValue("open", val) # sets a single value to a channel (such as a facial activation unit)

      #self.setChannelPos("base", SrVec(0,1,0)) # sets the position of a joint as x/y/z translation values

      #self.setChannelQuat("l_wrist", SrQuat(1,0,0,0)) # sets the rotation of a joint as a quaternion

# instantiate this controller once for each character
myc = MyController()
# get the character
character = scene.getCharacter("char0")
# run this controller in position 15 (place it after the 15th controller in the stack)
character.addController(15, myc)

# to remove the controller, do the following
#character.removeController(myc)

 

So if you wanted to gather data from a sensor (say, a Kinect) you would first have to store that data where it is accessible to SmartBody. The example that I gave before is to write some C++code as an SBService (see the http://sourceforge.net/p/smart.....dy/src/sb/ link in my other response).

 

Ari

November 27, 2014
2:40 am
Avatar
Member
Members
Forum Posts: 20
Member Since:
November 18, 2014
sp_UserOfflineSmall Offline

Thank you for your help thus far. I have a couple more questions:

1. I tried running the code below with both

character = scene.getCharacter("char0")
#character = scene.getCharacter('ChrBrad')

and got the following error messages: 

'NoneType' object has no attribute 'addController' 

'SBCharacter' object has no attribute 'addController'

Can you suggest what the problem might be?

 

2. How do I specify a AU unit?  

#self.setChannelValue("open", val) # sets a single value to a channel (such as a facial activation unit)
self.setChannelValue("au_1_left", val)

 

3. Using this method, if the value in val changes, will the facial expression of the avatar (Brad in this case) change as well? Or will I have to reload the script every time the value in val changes? If I do need to reload the script using this method, can you suggest another method where I don't have to?

 

import SmartBody

class MyController (PythonController):

def init(self, pawn):
print "Sample controller..."

def evaluate(self):
val = 1

#self.setChannelValue("open", val) # sets a single value to a channel (such as a facial activation unit)
self.setChannelValue("au_1_left", val)
#self.setChannelPos("base", SrVec(0,1,0)) # sets the position of a joint as x/y/z translation values
#self.setChannelQuat("l_wrist", SrQuat(1,0,0,0)) # sets the rotation of a joint as a quaternion

def main():
#Add asset paths
assetManager = scene.getAssetManager()
assetManager.addAssetPath('motion', 'ChrBrad')
assetManager.addAssetPath('mesh', 'mesh')
assetManager.addAssetPath('script', 'scripts')

#Load assets based on asset paths
assetManager.loadAssets()

#Set scene scale and reset the camera
scene.setScale(1.0)
scene.getActiveCamera().reset()

#Run a Python script file
scene.run('zebra2-map.py')
zebra2Map = scene.getJointMapManager().getJointMap('zebra2')
bradSkeleton = scene.getSkeleton('ChrBrad.sk')
zebra2Map.applySkeleton(bradSkeleton)
zebra2Map.applyMotionRecurse('ChrBrad')

#Setup Brad
brad = scene.createCharacter('ChrBrad', '')
bradSkeleton = scene.createSkeleton('ChrBrad.sk')
brad.setSkeleton(bradSkeleton)

#Set standard controller
brad.createStandardControllers()

#Deformable mesh
brad.setDoubleAttribute('deformableMeshScale', 0.01)
brad.setStringAttribute('deformableMesh', 'ChrBrad.dae')

#Show the character
brad.setStringAttribute('displayType', 'GPUmesh')

#Create empty face definition for character
bradFace = scene.createFaceDefinition('ChrBrad')
bradFace.setFaceNeutral('ChrBrad@face_neutral')

#Set face poses for each action unit and viseme
bradFace.setAU(1, "left", "ChrBrad@001_inner_brow_raiser_lf")
bradFace.setAU(1, "right", "ChrBrad@001_inner_brow_raiser_rt")
bradFace.setAU(2, "left", "ChrBrad@002_outer_brow_raiser_lf")
bradFace.setAU(2, "right", "ChrBrad@002_outer_brow_raiser_rt")
bradFace.setAU(4, "left", "ChrBrad@004_brow_lowerer_lf")
bradFace.setAU(4, "right", "ChrBrad@004_brow_lowerer_rt")
bradFace.setAU(5, "both", "ChrBrad@005_upper_lid_raiser")
bradFace.setAU(6, "both", "ChrBrad@006_cheek_raiser")
bradFace.setAU(7, "both", "ChrBrad@007_lid_tightener")
bradFace.setAU(10, "both", "ChrBrad@010_upper_lip_raiser")
bradFace.setAU(12, "left", "ChrBrad@012_lip_corner_puller_lf")
bradFace.setAU(12, "right", "ChrBrad@012_lip_corner_puller_rt")
bradFace.setAU(25, "both", "ChrBrad@025_lips_part")
bradFace.setAU(26, "both", "ChrBrad@026_jaw_drop")
bradFace.setAU(45, "left", "ChrBrad@045_blink_lf")
bradFace.setAU(45, "right", "ChrBrad@045_blink_rt")

bradFace.setViseme("open", "ChrBrad@open")
bradFace.setViseme("W", "ChrBrad@W")
bradFace.setViseme("ShCh", "ChrBrad@ShCh")
bradFace.setViseme("PBM", "ChrBrad@PBM")
bradFace.setViseme("FV", "ChrBrad@FV")
bradFace.setViseme("wide", "ChrBrad@wide")
bradFace.setViseme("tBack", "ChrBrad@tBack")
bradFace.setViseme("tRoof", "ChrBrad@tRoof")
bradFace.setViseme("tTeeth", "ChrBrad@tTeeth")

brad.setFaceDefinition(bradFace)

# instantiate this controller once for each character

myc = MyController()

#brad.addController(15, myc)

# get the character
character = scene.getCharacter("char0")
#character = scene.getCharacter('ChrBrad')

# run this controller in position 15 (place it after the 15th controller in the stack)
character.addController(15, myc)

# to remove the controller, do the following
#character.removeController(myc)

November 28, 2014
1:50 am
Avatar
Member
Members
Forum Posts: 20
Member Since:
November 18, 2014
sp_UserOfflineSmall Offline

addController isn't in SBCharacter.h or SBCharacter.cpp in the versions of SmartBody that I have downloaded. Is it included in other versions? 

If it is, can you post addController (preferably in a version of SBCharacter.cpp and SBCharacter.h that works with the most recent version of SmartBody)? If the addController was removed from the code for a particular reason, could you specify what that reason is? 

November 29, 2014
12:51 am
Avatar
Admin
Forum Posts: 983
Member Since:
December 1, 2011
sp_UserOfflineSmall Offline

I suggest you download the latest version of SmartBody from the repository. I haven't updated the SDK version in awhile.

If you just want to add that function you can add this header to SBCharacter.h:

 

void addController(int index, SBController* controller);

then to SBCharacter.cpp

void SBCharacter::addController(int index, SBController* controller)
{
int numControllers = this->getNumControllers();
for (int c = 0; c < numControllers; c++)
{
if (this->getControllerByIndex(c) == controller)
{
LOG("Controller %s already exists in controller tree for character %s.", controller->getName().c_str(), this->getName().c_str());
return;
}
}
// make sure the controller doesn't already exist in the tree
controller->init(this);
ct_tree_p->add_controller(controller, index);
}

 

then around line 133 in SBPythonCharacter.cpp:

.def("addController", &SBCharacter::addController, "Adds a controller to the character.")

 

Ari

December 1, 2014
2:58 am
Avatar
Member
Members
Forum Posts: 20
Member Since:
November 18, 2014
sp_UserOfflineSmall Offline

I am using the version of SmartBody from the SourceForge repository and I have made the changes that you suggested in your most recent post. I am loading the python script into the sbgui. However, I am still getting an error:

character.addController(15, myc)

AttributeError

:

'SBCharacter' object has no attribute 'addController'

Do I need to re-build or re-compile some/all the SmartBody code? If so, how?

I have tried compiling the SBCharacter.cpp directly from the command prompt on Windows and am getting several errors including 'SBAPI' does not name a type. I suspect this is because the parts of smartbody cannot be compiled separately and/or I am not including all the necessary files for compilation.

Thank you for your help.

December 1, 2014
6:54 pm
Avatar
Admin
Forum Posts: 983
Member Since:
December 1, 2011
sp_UserOfflineSmall Offline

In SmartBody/src/sb/SBPythonCharacter.cpp add:

 

        .def("addController", &SBCharacter::addController, "Adds a controller to the character.")

around line 132. That creates a Python function that maps to the C++ function. You will then need to rebuild the SmartBody code.

 

Ari

December 2, 2014
6:46 am
Avatar
Member
Members
Forum Posts: 20
Member Since:
November 18, 2014
sp_UserOfflineSmall Offline

I downloaded Smartbody here as suggested: http://sourceforge.net/project.....smartbody/.

Can you clarify how I should rebuild the SmartBody code?

Thank you for your help.

December 2, 2014
8:58 pm
Avatar
Admin
Forum Posts: 983
Member Since:
December 1, 2011
sp_UserOfflineSmall Offline

If you are running on Windows, you need visual studio 2010 (or visual studio 2010 express). There is a .sln file in the build/ directory that you load, then build the sbgui project.

 

Ari

December 3, 2014
6:27 am
Avatar
Member
Members
Forum Posts: 20
Member Since:
November 18, 2014
sp_UserOfflineSmall Offline

I am getting many errors when I try to build the project. Here is what I am doing:
1. I downloaded the version of SmartBody from sourceforge and made the recommended changes. I then installed SmartBody again without the changes.
2. I installed MVS C++ 2010 express and MVS 2010 Professional. 
3. I tried to build and rebuild the sbgui project only for both SmartBody versions and on both MVS 2010 versions.
In all cases a get several errors:

1>c:\smartbody-2\src\smartbody\sb\sbcharacter.h(107): error C2143: syntax error : missing ';' before '.'
1>c:\smartbody-2\src\smartbody\sb\sbcharacter.h(107): error C4430: missing type specifier - int assumed. Note: C++ does not support default-int
1>c:\smartbody-2\src\smartbody\sb\sbcharacter.h(107): error C2238: unexpected token(s) preceding ';'
1>c:\smartbody-2\src\smartbody\sb\sbcharacter.h(108): error C2143: syntax error : missing ';' before '<<'
1>c:\smartbody-2\src\smartbody\sb\sbcharacter.h(108): error C4430: missing type specifier - int assumed. Note: C++ does not support default-int
1>c:\smartbody-2\src\smartbody\sb\sbcharacter.h(108): error C2238: unexpected token(s) preceding ';'
1>c:\smartbody-2\include\boost\type_traits\detail\has_binary_operator.hpp : warning C4819: The file contains a character that cannot be represented in the current code page (932). Save the file in Unicode format to prevent data loss

December 3, 2014
8:11 pm
Avatar
Admin
Forum Posts: 983
Member Since:
December 1, 2011
sp_UserOfflineSmall Offline

Did you download the SDK version, or the source code directly from sourceforge?

 

Can you also send the code snippet? Specifically, lines 107 and 108 of SBCharacter.h?

 

Ari

December 5, 2014
7:03 am
Avatar
Member
Members
Forum Posts: 20
Member Since:
November 18, 2014
sp_UserOfflineSmall Offline

#ifndef _SBCHARACTER_H_
#define _SBCHARACTER_H_

#include <sb/SBTypes.h>
#include <sb/sbm_character.hpp>
#include <sb/SBMotion.h>
#include <sb/SBCharacterFrameData.h>

#include <iostream>
#include <fstream>
using namespace std;

namespace SmartBody {

class SBSkeleton;
class SBBehavior;
class SBSteerAgent;
class SBController;
class SBDiphone;
class SBReach;
class SBMotionGraph;

class SBCharacter : public SbmCharacter
{
public:
SBAPI SBCharacter();
SBAPI SBCharacter(const std::string& name, const std::string& type = "");
SBAPI virtual ~SBCharacter();

SBAPI const std::string& getName();
SBAPI void setName(const std::string& name);

SBAPI void setType(const std::string& type);
SBAPI std::string getType();

SBAPI virtual int setup();

SBAPI int getNumControllers();

SBAPI void setAutomaticPruning(bool val);
SBAPI bool isAutomaticPruning();
SBAPI void pruneControllers();

SBAPI void setUseVisemeCurves(bool val);
SBAPI bool isUseVisemeCurves();

SBAPI float getVisemeTimeOffset();
SBAPI void setVisemeTimeOffset(float val);

SBAPI void setVoice(const std::string& type);
SBAPI void setVoiceCode(const std::string& param);
SBAPI const std::string getVoice();
SBAPI const std::string& getVoiceCode();

SBAPI void setVoiceBackup(const std::string& type);
SBAPI void setVoiceBackupCode(const std::string& param);
SBAPI const std::string& getVoiceBackup();
SBAPI const std::string& getVoiceBackupCode();

SBAPI SBController* getControllerByIndex(int i);
SBAPI SBController* getControllerByName(const std::string& name);
SBAPI std::vector<std::string> getControllerNames();
SBAPI void startMotionRecord(double frameRate);
SBAPI void stopMotionRecord(const std::string& motionName, const std::string& type);
SBAPI void writeMotionRecord(const std::string& motionName, const std::string& type);

SBAPI int getNumBehaviors();
SBAPI SBBehavior* getBehavior(int num);
SBAPI std::vector<SBBehavior*>& getBehaviors();

SBAPI double getLastScheduledSpeechBehavior();
SBAPI std::string hasSpeechBehavior();

SBAPI virtual SBFaceDefinition* getFaceDefinition();
SBAPI virtual void setFaceDefinition(SBFaceDefinition* face);
SBAPI virtual void updateDefaultFacePose();

SBAPI void interruptFace(double seconds);

SBAPI void notify(SBSubject* subject);
SBAPI std::string getPostureName(); // get the current posture of character
SBAPI void setDeformableMeshScale(double meshScale);
SBAPI void copy(SBCharacter* origChar);
SBAPI void setReach(SmartBody::SBReach* reach);
SBAPI SmartBody::SBReach* getReach();

SBAPI void setMotionGraph(const std::string& moGraphName);
SBAPI void startMotionGraph(const std::string& moNodeName);
SBAPI void startMotionGraphWithPath(const std::vector<SrVec>& pathList);
SBAPI void startMotionGraphRandomWalk();

SBAPI void setUseJointConstraint(bool bUseConstraint);
SBAPI bool getUseJointConstraint();
SBAPI void addJointTrajectoryConstraint(const std::string& jointName, const std::string& refJointName);
SBAPI TrajectoryRecord* getJointTrajectoryConstraint(const std::string& jointName);
SBAPI std::vector<std::string> getJointConstraintNames();
float getJointTrajBlendWeight() const { return jointTrajBlendWeight; }
void setJointTrajBlendWeight(float val) { jointTrajBlendWeight = val; }

SBAPI void createBlendShapeChannel(const std::string& channelName);

SBAPI void setDeformableMeshName(const std::string& meshName);

SBAPI const SBM_CharacterFrameDataMarshalFriendly & GetFrameDataMarshalFriendly();

//I added a controller to receive AU unit input
void addController(int index, SBController* controller);

protected:
void InitFrameDataMarshalFriendly();
void FreeFrameDataJointsMarshalFriendly();
void FreeFrameDataMarshalFriendly();

protected:
std::vector<SBBehavior*> _curBehaviors;
SmartBody::SBReach* _reach;
SmartBody::SBMotionGraph* _curMotionGraph;
std::map<std::string, TrajectoryRecord*> jointTrajMap;
float jointTrajBlendWeight;
bool useJointConstraint;
SBM_CharacterFrameDataMarshalFriendly * frameDataMarshalFriendly;

};

};

#endif

December 5, 2014
7:06 am
Avatar
Member
Members
Forum Posts: 20
Member Since:
November 18, 2014
sp_UserOfflineSmall Offline

Hi Ari,

Above is SBCharacter.h. I have downloaded SmartBody from http://sourceforge.net/project.....smartbody/. I suppose this is the SDK version. The svn version is read-only and the connection failed. Can you suggest another way of downloading, compiling and running SmartBody?

In addition, if you think it would be helpful and are comfortable with it, I can send you SBCharacter.cpp and SBPythonCharacter.cpp. 

December 5, 2014
7:28 pm
Avatar
Admin
Forum Posts: 983
Member Since:
December 1, 2011
sp_UserOfflineSmall Offline

You can add

 

I'm about to create a new SDK version. You can just download that one that will already have those changes in it.

I'm surprised that you weren't able to download SmartBody from SVN. If you download the Tortoise SVN tool for Windows, you can just right-click in a directory, select SVN Checkout... then enter

svn://svn.code.sf.net/p/smartbody/code/trunk

in the 'URL of the repository' field.

 

Regardless, I'll have the new SDK ready soon. Stay tuned.

 

Ari

December 8, 2014
9:12 am
Avatar
Member
Members
Forum Posts: 20
Member Since:
November 18, 2014
sp_UserOfflineSmall Offline

Hi Ari,

I think you are missing something after "You can add...". I tried checking out several times on Tortoise SVN and it didn't work (connection failed). When do you think you will have the new SDK ready? Will you post a reply to question thread when you do or will you indicate that you have somewhere else?

Thanks.

December 9, 2014
12:54 am
Avatar
Admin
Forum Posts: 983
Member Since:
December 1, 2011
sp_UserOfflineSmall Offline

I'll release the new SDK today or tomorrow.

 

Ari

December 11, 2014
7:33 am
Avatar
Admin
Forum Posts: 983
Member Since:
December 1, 2011
sp_UserOfflineSmall Offline

I just released a new version for Windows (r5919). You can go to the download page to retrieve it.

 

Ari

December 12, 2014
2:24 am
Avatar
Member
Members
Forum Posts: 20
Member Since:
November 18, 2014
sp_UserOfflineSmall Offline

Hi Ari,

Thanks for the notification. I wrote a python script that should cause AU 45 left to increase on the avatar based on the recommendations you made above. However, when I run the code on the new version of SmartBody, SmartBody crashes.

The code is below. "val" will be assigned a value based on the output of another function. Here, I have just assigned it to 1.

Can you suggest why the code is crashing?

import SmartBody

class MyController (PythonController):
   def init(self, pawn):
      self.addChannel("au_45_left", "XPos")
      self.addChannel("open", "XPos")

   def evaluate(self):
      val = 1

      self.setChannelValue("open", val)

def main():
    #Add asset paths
    assetManager = scene.getAssetManager()
    assetManager.addAssetPath('motion', 'ChrBrad')
    assetManager.addAssetPath('mesh', 'mesh')
    assetManager.addAssetPath('script', 'scripts')

    #Load assets based on asset paths
    assetManager.loadAssets()

    #Set scene scale and reset the camera
    scene.setScale(1.0)
    scene.getActiveCamera().reset()

    #Run a Python script file
    scene.run('zebra2-map.py')
    zebra2Map = scene.getJointMapManager().getJointMap('zebra2')
    bradSkeleton = scene.getSkeleton('ChrBrad.sk')
    zebra2Map.applySkeleton(bradSkeleton)
    zebra2Map.applyMotionRecurse('ChrBrad')

    #Setup Brad
    brad = scene.createCharacter('ChrBrad', '')
    bradSkeleton = scene.createSkeleton('ChrBrad.sk')
    brad.setSkeleton(bradSkeleton)

    #Set standard controller
    brad.createStandardControllers()

    #Deformable mesh
    brad.setDoubleAttribute('deformableMeshScale', 0.01)
    brad.setStringAttribute('deformableMesh', 'ChrBrad.dae')

    #Show the character
    brad.setStringAttribute('displayType', 'GPUmesh')

    #Create empty face definition for character
    bradFace = scene.createFaceDefinition('ChrBrad')
    bradFace.setFaceNeutral('ChrBrad@face_neutral')

    #Set face poses for each action unit and viseme
    bradFace.setAU(1, "left", "ChrBrad@001_inner_brow_raiser_lf")
    bradFace.setAU(1, "right", "ChrBrad@001_inner_brow_raiser_rt")
    bradFace.setAU(2, "left", "ChrBrad@002_outer_brow_raiser_lf")
    bradFace.setAU(2, "right", "ChrBrad@002_outer_brow_raiser_rt")
    bradFace.setAU(4, "left", "ChrBrad@004_brow_lowerer_lf")
    bradFace.setAU(4, "right", "ChrBrad@004_brow_lowerer_rt")
    bradFace.setAU(5, "both", "ChrBrad@005_upper_lid_raiser")
    bradFace.setAU(6, "both", "ChrBrad@006_cheek_raiser")
    bradFace.setAU(7, "both", "ChrBrad@007_lid_tightener")
    bradFace.setAU(10, "both", "ChrBrad@010_upper_lip_raiser")
    bradFace.setAU(12, "left", "ChrBrad@012_lip_corner_puller_lf")
    bradFace.setAU(12, "right", "ChrBrad@012_lip_corner_puller_rt")
    bradFace.setAU(25, "both", "ChrBrad@025_lips_part")
    bradFace.setAU(26, "both", "ChrBrad@026_jaw_drop")
    bradFace.setAU(45, "left", "ChrBrad@045_blink_lf")
    bradFace.setAU(45, "right", "ChrBrad@045_blink_rt")

    bradFace.setViseme("open", "ChrBrad@open")
    bradFace.setViseme("W", "ChrBrad@W")
    bradFace.setViseme("ShCh", "ChrBrad@ShCh")
    bradFace.setViseme("PBM", "ChrBrad@PBM")
    bradFace.setViseme("FV", "ChrBrad@FV")
    bradFace.setViseme("wide", "ChrBrad@wide")
    bradFace.setViseme("tBack", "ChrBrad@tBack")
    bradFace.setViseme("tRoof", "ChrBrad@tRoof")
    bradFace.setViseme("tTeeth", "ChrBrad@tTeeth")

    brad.setFaceDefinition(bradFace)

    # instantiate this controller once for each character
    myc = MyController()

    # get the character
    character = scene.getCharacter('ChrBrad')

    # run this controller in position 15 (place it after the 15th controller in the stack)
    character.addController(15, myc)

if __name__ == '__main__':
    main()

December 12, 2014
3:09 am
Avatar
Member
Members
Forum Posts: 20
Member Since:
November 18, 2014
sp_UserOfflineSmall Offline

Hi Ari,

I realized that I left an argument in class MyController() in the above code. If I take it out, the code compiles but it doesn't work - no left blinking. For example, if I do the following nothing happens:

class MyController ():
   def init(self, pawn):
      print ("test")

   def evaluate(self):
      val = 1

      self.setChannelValue("tBack", val)

OR

class MyController ():
   def init(self, pawn):
      self.addChannel("tBack", "XPos")
      print ("test")

   def evaluate(self):
      val = 1

      self.setChannelValue("tBack", val)

I want a motion to be displayed on my avatar in real-time every time a AU value is changed (as indicated by a variable passed from another function). Can you explain how I can set the value of an action unit in real-time?

From my understanding, setViseme sets a motion (like "ChrBrad@open") to a name (like "open"). If I refer to open, it should cause the motion. setAU sets a motion (like "..brow_raiser_lf") to a side ("left") and name ("1"). If I refer to 1, it should cause the motion.

Can you explain what a channel is? From my understanding, I need to create a channel for each of these motions (setAU and setViseme) if I want to initiate one of these motions. I pass the name of the channel (like "1" or "open") and the value I want it to initialize to. Are these values overwritten each time I call the channel?

December 12, 2014
6:59 pm
Avatar
Admin
Forum Posts: 983
Member Since:
December 1, 2011
sp_UserOfflineSmall Offline

Your controller class needs to be derived from PythonController. Try this:

class MyController(PythonController):
    def init(self, pawn):
        print "setting up custom controller..."
    def evaluate(self):
        self.setChannelValue("open", 1.0) # sets the 'open' viseme to 1
        self.setChannelValue("au_45_left", .5) # sets the left Action Unit 45 to .7

myc = MyController()
character = scene.getCharacter("ChrBrad")
character.addController(15, myc)  # places this controller in the controller stack at position 15
# remove the controller like this;
#character.removeController(myc)

 

A channel represents streaming data. A joint typically has 4 channels (x position, y position, z position, quaternion) and facial activations typically only have one channel (representing their activation amounts).

For facial control, the FaceDefinition associates motions with facial movements, and automatically creates a channel for them. For visemes, the channel name is the same as the name in the FaceDefinition (i.e. "open"). For Action Units, the channel name looks like:

au_45_left

for the left side of Action Unit 45 (which is an eye blink).

Is you want to control these values explicitly, then using a controller like the code above will work. You can also control these movements to some extent at the behavioral level through BML, which in turn, will send instructions to the face controller.

 

Does this make sense?

Ari

 

 
December 15, 2014
1:18 am
Avatar
Member
Members
Forum Posts: 20
Member Since:
November 18, 2014
sp_UserOfflineSmall Offline

Hi Ari,

I'm running the following and it is still crashing. There is no error message hinting at what the problem might be. sbgui just crashes. When I remove PythonController from the class argument, sbgui doesn't crash but controller doesn't work (i.e. nothing happens).

Any suggestions?

import SmartBody

class MyController (PythonController):
   def init(self, pawn):
      print ("setting up custom controller...")

   def evaluate(self):
      self.setChannelValue("open", 1.0)
      self.setChannelValue("au_45_left", 0.5)

def main():
    #Add asset paths
    assetManager = scene.getAssetManager()
    assetManager.addAssetPath('motion', 'ChrBrad')
    assetManager.addAssetPath('mesh', 'mesh')
    assetManager.addAssetPath('script', 'scripts')

    #Load assets based on asset paths
    assetManager.loadAssets()

    #Set scene scale and reset the camera
    scene.setScale(1.0)
    scene.getActiveCamera().reset()

    #Run a Python script file
    scene.run('zebra2-map.py')
    zebra2Map = scene.getJointMapManager().getJointMap('zebra2')
    bradSkeleton = scene.getSkeleton('ChrBrad.sk')
    zebra2Map.applySkeleton(bradSkeleton)
    zebra2Map.applyMotionRecurse('ChrBrad')

    #Setup Brad
    brad = scene.createCharacter('ChrBrad', '')
    bradSkeleton = scene.createSkeleton('ChrBrad.sk')
    brad.setSkeleton(bradSkeleton)

    #Set standard controller
    brad.createStandardControllers()

    #Deformable mesh
    brad.setDoubleAttribute('deformableMeshScale', 0.01)
    brad.setStringAttribute('deformableMesh', 'ChrBrad.dae')

    #Show the character
    brad.setStringAttribute('displayType', 'GPUmesh')

    #Create empty face definition for character
    bradFace = scene.createFaceDefinition('ChrBrad')
    bradFace.setFaceNeutral('ChrBrad@face_neutral')

    #Set face poses for each action unit and viseme
    bradFace.setAU(1, "left", "ChrBrad@001_inner_brow_raiser_lf")
    bradFace.setAU(1, "right", "ChrBrad@001_inner_brow_raiser_rt")
    bradFace.setAU(2, "left", "ChrBrad@002_outer_brow_raiser_lf")
    bradFace.setAU(2, "right", "ChrBrad@002_outer_brow_raiser_rt")
    bradFace.setAU(4, "left", "ChrBrad@004_brow_lowerer_lf")
    bradFace.setAU(4, "right", "ChrBrad@004_brow_lowerer_rt")
    bradFace.setAU(5, "both", "ChrBrad@005_upper_lid_raiser")
    bradFace.setAU(6, "both", "ChrBrad@006_cheek_raiser")
    bradFace.setAU(7, "both", "ChrBrad@007_lid_tightener")
    bradFace.setAU(10, "both", "ChrBrad@010_upper_lip_raiser")
    bradFace.setAU(12, "left", "ChrBrad@012_lip_corner_puller_lf")
    bradFace.setAU(12, "right", "ChrBrad@012_lip_corner_puller_rt")
    bradFace.setAU(25, "both", "ChrBrad@025_lips_part")
    bradFace.setAU(26, "both", "ChrBrad@026_jaw_drop")
    bradFace.setAU(45, "left", "ChrBrad@045_blink_lf")
    bradFace.setAU(45, "right", "ChrBrad@045_blink_rt")

    bradFace.setViseme("open", "ChrBrad@open")
    bradFace.setViseme("W", "ChrBrad@W")
    bradFace.setViseme("ShCh", "ChrBrad@ShCh")
    bradFace.setViseme("PBM", "ChrBrad@PBM")
    bradFace.setViseme("FV", "ChrBrad@FV")
    bradFace.setViseme("wide", "ChrBrad@wide")
    bradFace.setViseme("tBack", "ChrBrad@tBack")
    bradFace.setViseme("tRoof", "ChrBrad@tRoof")
    bradFace.setViseme("tTeeth", "ChrBrad@tTeeth")

    brad.setFaceDefinition(bradFace)

    # instantiate this controller once for each character
    myc = MyController()

    # get the character
    character = scene.getCharacter("ChrBrad")

    # run this controller in position 15 (place it after the 15th controller in the stack)
    character.addController(1, myc)

    #remove the controller
    #character.removeController(myc)

if __name__ == '__main__':
    main()

December 19, 2014
10:16 pm
Avatar
Admin
Forum Posts: 983
Member Since:
December 1, 2011
sp_UserOfflineSmall Offline

Two problems:

1) You are creating the controller within the main() function, which is then automatically destroyed when you leave the function. Instead, you need to create the controller in the global context (not in main()) to prevent the crash.

 

2) The controller needs to be added at postion 15, not 1 (which means that it will run after the 15th controller).

 

Here's code that should work:

 

class MyController (PythonController):
   def init(self, pawn):
      this.setName("mycontroller")
    

   def evaluate(self):
      self.setChannelValue("open", 1.0)
      self.setChannelValue("au_45_left", 0.5)
   print "hello"

#Add asset paths
assetManager = scene.getAssetManager()
assetManager.addAssetPath('motion', 'ChrBrad')
assetManager.addAssetPath('mesh', 'mesh')
assetManager.addAssetPath('script', 'scripts')

#Load assets based on asset paths
assetManager.loadAssets()

#Set scene scale and reset the camera
scene.setScale(1.0)
scene.getActiveCamera().reset()

#Run a Python script file
scene.run('zebra2-map.py')
zebra2Map = scene.getJointMapManager().getJointMap('zebra2')
bradSkeleton = scene.getSkeleton('ChrBrad.sk')
zebra2Map.applySkeleton(bradSkeleton)
zebra2Map.applyMotionRecurse('ChrBrad')

#Setup Brad
brad = scene.createCharacter('ChrBrad', '')
bradSkeleton = scene.createSkeleton('ChrBrad.sk')
brad.setSkeleton(bradSkeleton)

#Set standard controller
brad.createStandardControllers()

#Deformable mesh
brad.setDoubleAttribute('deformableMeshScale', 0.01)
#brad.setStringAttribute('deformableMesh', 'ChrBrad.dae')

#Show the character
brad.setStringAttribute('displayType', 'GPUmesh')

#Create empty face definition for character
bradFace = scene.createFaceDefinition('ChrBrad')
bradFace.setFaceNeutral('ChrBrad@face_neutral')

#Set face poses for each action unit and viseme
bradFace.setAU(1, "left", "ChrBrad@001_inner_brow_raiser_lf")
bradFace.setAU(1, "right", "ChrBrad@001_inner_brow_raiser_rt")
bradFace.setAU(2, "left", "ChrBrad@002_outer_brow_raiser_lf")
bradFace.setAU(2, "right", "ChrBrad@002_outer_brow_raiser_rt")
bradFace.setAU(4, "left", "ChrBrad@004_brow_lowerer_lf")
bradFace.setAU(4, "right", "ChrBrad@004_brow_lowerer_rt")
bradFace.setAU(5, "both", "ChrBrad@005_upper_lid_raiser")
bradFace.setAU(6, "both", "ChrBrad@006_cheek_raiser")
bradFace.setAU(7, "both", "ChrBrad@007_lid_tightener")
bradFace.setAU(10, "both", "ChrBrad@010_upper_lip_raiser")
bradFace.setAU(12, "left", "ChrBrad@012_lip_corner_puller_lf")
bradFace.setAU(12, "right", "ChrBrad@012_lip_corner_puller_rt")
bradFace.setAU(25, "both", "ChrBrad@025_lips_part")
bradFace.setAU(26, "both", "ChrBrad@026_jaw_drop")
bradFace.setAU(45, "left", "ChrBrad@045_blink_lf")
bradFace.setAU(45, "right", "ChrBrad@045_blink_rt")

bradFace.setViseme("open", "ChrBrad@open")
bradFace.setViseme("W", "ChrBrad@W")
bradFace.setViseme("ShCh", "ChrBrad@ShCh")
bradFace.setViseme("PBM", "ChrBrad@PBM")
bradFace.setViseme("FV", "ChrBrad@FV")
bradFace.setViseme("wide", "ChrBrad@wide")
bradFace.setViseme("tBack", "ChrBrad@tBack")
bradFace.setViseme("tRoof", "ChrBrad@tRoof")
bradFace.setViseme("tTeeth", "ChrBrad@tTeeth")

brad.setFaceDefinition(bradFace)

# instantiate this controller once for each character
myc = MyController()

# get the character
character = scene.getCharacter("ChrBrad")

# run this controller in position 15 (place it after the 15th controller in the stack)
character.addController(15, myc)

#remove the controller
#character.removeController(myc)

December 24, 2014
6:20 am
Avatar
Member
Members
Forum Posts: 20
Member Since:
November 18, 2014
sp_UserOfflineSmall Offline

Hi Ari,

Thanks for the help. I still have a couple questions.

In the MyController class, there is a line:

self.setChannelValue("au_45_left", 0.2)

Is there a list of strings that I can populate the first field with? I have tried controlling other AU units but the motions don't always occur as expected.

Thanks!

December 24, 2014
6:25 am
Avatar
Member
Members
Forum Posts: 20
Member Since:
November 18, 2014
sp_UserOfflineSmall Offline

Hi Ari,

I noticed that ChrBrad, ChrRachel and ChrMaarten all have their mouths open. When I try to close them using self.setChannelValue("au_26_both", 1.0), their mouths remain open. How do I close them?

In addition, ChrMaarten doesn't work in the above code. Why is that? Is it because this character doesn't have its own skeleton (it uses ChrBrad)?

Lastly, I would like to use a Japanese person-looking avatar. ChrBrad, ChrRachel and ChrMaarten do not fit this criteria. Do you have any avatars that do? If not, can ChrRachel or ChrBrad be changed quickly to look like a Japanese person? If I need to make a new avatar, how do I do that?

Thanks!

December 27, 2014
10:46 pm
Avatar
Admin
Forum Posts: 983
Member Since:
December 1, 2011
sp_UserOfflineSmall Offline

There are two sets of facial controls: Action Units (like #26, the jaw drop) and a set of visemes for use with lip syncing. There can be a conflict between the Action Units that affect the lower face and the ones used for lip syncing. In your case, it sounds like the 'open' viseme was activated (which is similar to Action Unit 26), which is why the mouths remain open. I would recommend using the Action Units for the upper face, and the lip syncing visemes (open, Wide, Bmp, etc.) for the lower face.

ChrBrad and ChrMaarten both use the ChrBrad.sk skeleton, but their facial data is different. Try loading one or the other and seeif you have that same problem. Up until recently, their facial pose files had the same names (I've changed that in the past couple of weeks).

You can use any character that you want for SmartBody. I have an example of using a Mixamo character here:

You just need to create a JointMapping from your character to the SmartBody standard skeleton, and then SmartBody can retarget motions to your character. To activate the face, you need to create a FaceDefinition that matches the face controls that exist in your character rig. To get the lip syncing working, you need to have facial poses that match the lip syncing visemes (open, wide, etc) or you can spend the time to create your own lip syncing data set for whatever visemes your character has.

 

Ari