University of Southern CaliforniaUSC
USC ICT TwitterUSC ICT FacebookUSC ICT YouTube

Motion Capture device animating characters | General SmartBody Discussion | Forum

Avatar

Please consider registering
guest

sp_LogInOut Log In sp_Registration Register

Register | Lost password?
Advanced Search

— Forum Scope —




— Match —





— Forum Options —





Minimum search word length is 3 characters - maximum search word length is 84 characters

sp_Feed Topic RSS sp_TopicIcon
Motion Capture device animating characters
December 7, 2015
10:42 am
Avatar
Member
Members
Forum Posts: 6
Member Since:
August 24, 2015
sp_UserOfflineSmall Offline

Hi all,

I have to use a real-time motion-capture device (Xsens) with a VHTK character; e.g. Bard who is readily available in the toolkit's vhAssetsTestScene.

I have got the mo-cap working on a Unity MecAnim character, through a plug-in, but now I need to do it on Brad, because I have to integrate verbal, lip sync, facial movements, sound, etc... capabilities as well.

So, my question is: Is it possible to hook up a mo-cap device to Brad and animate it based on the live motion data coming in, without losing any VH capabilities? I am asking on this forum, because I understand SmartBody animation system is what animates VHTK characters.

To be clear, I am not trying to disable SmartBody completely, but only take over the nonverbal behaviors of limbs (e.g. arm gestures and body movements) and not face or lips or utterances.

Has anyone on the forum done this? Where would I start? There are quite a few scripts that deal with animating Brad and his joints' positions and rotations, so I don't know where to begin...

In summary, please kindly provide me with detailed pointers that would get me started on animating Bard by live mo-cap data in real-time while having all the VH system capabilities; e.g. verbal, lip sync, facial movements, sound, etc...

Thank you,

December 7, 2015
6:16 pm
Avatar
Admin
Forum Posts: 980
Member Since:
December 1, 2011
sp_UserOfflineSmall Offline

The VHTK integration works with SmartBody by running a SmartBody instance, then copying the state onto the Unity character every frame. If you wanted Unity to control the character, and not SmartBody, you could copy over only the state that you are interested in (i.e. lips and face movements).

I don't know the integration code very well (perhaps you can ask that question on the VHTK forum), but if you were able to disable the copying of joints for all the body (except the face and head) then you would probably have the effect that you are looking for. Somewhere there is a loop that gets the name of the joint, finds the correspoding Unity joint, then copies the data to it. I would add a few lines checking to see if the joint is a lips/mount/face joint or not.

Likewise, you could have SmartBody function normally, then have your Xsens plugin copy over the state  ~after~ the SmartBody plugin copies over the state. In that way, the Xsens data might be able to override the SmartBody data. I'm not sure how the execution of such things happen in Unity; in SmartBody, there is a stack of controllers that get executed in order, so each later controller gets the opportunity to override the results of the previous one.

 

Ari

December 11, 2015
9:36 am
Avatar
Member
Members
Forum Posts: 6
Member Since:
August 24, 2015
sp_UserOfflineSmall Offline

Hi Ari,

Thanks for this.

What you are referring to takes place in UnitySmartbodyCharacter.cs and SmartbodyManager.cs of the VHTK.

Can I continue asking questions about these here, or would you rather have questions only to do with SmartBody and not Unity?

Thanks

December 11, 2015
11:48 pm
Avatar
Admin
Forum Posts: 980
Member Since:
December 1, 2011
sp_UserOfflineSmall Offline

You are welcome to ask questions here. I can give you all manner of detail about SmartBody (and how it should be used and its capabilities) but my knowledge of the Unity side is limited. 

 

Ari

December 15, 2015
2:52 pm
Avatar
Member
Members
Forum Posts: 6
Member Since:
August 24, 2015
sp_UserOfflineSmall Offline

Great, Thanks.

I understand it might be possible to disable the SmartBody controllers that one would not need.

Apparently, SmartBody provides a hierarchical set of controllers: one for gaze, one for face, one for lip sync, and one for body.

(I say apparently because my basic knowledge of SmartBody is limited to its integration in VHTK)

Do you know how I can enable/disable various controllers at various times?

I am thinking, instead of trying to ignore and then overwrite the joints that are animated via SmartBody by my MoCap device data, perhaps I could simply enable the face and lip sync controllers and so not receive competing data on bodily joints, and then I will worry about copying the motion data onto joints...

Cheers

December 15, 2015
6:25 pm
Avatar
Admin
Forum Posts: 980
Member Since:
December 1, 2011
sp_UserOfflineSmall Offline

A typical SmartBody character uses 20+ controllers to handle all of its functionality (breathing, reaching, talking, physics, etc.). Some exclusively control part of the body, others override or add to existing motion. Some controllers work in tandem; for example, a blend controller handles motion blending, but then the constraint controller fixes the IK to prevent foot sliding, so not all the controllers can be removed individually for the effect that you want.

You can get all the controllers like this (in Python):

character = scene.getCharacter("mycharacter")

mycontrollernames = character.getControllerNames()

for c in range(0, len(mycontrollernames)):

   controller = character.getControllerByName(mycontrollernames[c])

if you want to disable the controller you can do:

controller.setIgnore(True)

and reenable it:

controller.setIgnore(False)

If you only want gazing and face control, then try only enabling:

X_gazeSchedule, X_faceController, X_headSchedule

and disabling everything else (where 'X' is the name of your character).

 

Ari

Forum Timezone: America/Los_Angeles

Most Users Ever Online: 733

Currently Online: mondayneedle1, huberdavidsen3, casey29english, kinneyreynolds06, andersen19eaton, kristensen00leblanc, bager59mayo, eric506784, t9rrzpi307, g2pvwfg490
108 Guest(s)

Currently Browsing this Page:
1 Guest(s)

Top Posters:

jwwalker: 80

jyambao: 51

rbaral: 47

adiaz: 30

WargnierP: 29

lucky7456969: 28

mbarros: 28

avida.matt: 26

JonathanW: 24

laguerre: 23

Member Stats:

Guest Posters: 65

Members: 52505

Moderators: 3

Admins: 4

Forum Stats:

Groups: 1

Forums: 5

Topics: 427

Posts: 2343