University of Southern CaliforniaUSC
USC ICT TwitterUSC ICT FacebookUSC ICT YouTube

General Question about Smartbody in context of Android | General SmartBody Discussion | Forum

Avatar

Please consider registering
guest

sp_LogInOut Log In sp_Registration Register

Register | Lost password?
Advanced Search

— Forum Scope —




— Match —





— Forum Options —





Minimum search word length is 3 characters - maximum search word length is 84 characters

sp_Feed Topic RSS sp_TopicIcon
General Question about Smartbody in context of Android
February 23, 2014
10:17 pm
Avatar
Member
Members
Forum Posts: 8
Member Since:
February 23, 2014
sp_UserOfflineSmall Offline

Hello, I do have a general question concerning Smartbody.

I would like to use a Smartbody character animation within an Android application - Is it (for example) possible to create a character by use of the desktop version and use it in context of my application? Does anyone have experience with using Smartbody within Android applications?

Thank you so much for your help!

February 26, 2014
3:48 pm
Avatar
Admin
Forum Posts: 983
Member Since:
December 1, 2011
sp_UserOfflineSmall Offline

We have used the Android version for a number of lightweight applications and demos - there are a couple of examples in the source code distribution. 

The scripts that can be saved from the desktop version of SmartBody can indeed work on the Android versions - they are simply a bunch of Python commands that are interpreted the same on both platforms. SO in theory you could set up an application on the desktop, save it to a file, then run it on the Android application.

Ari

February 26, 2014
5:41 pm
Avatar
Member
Members
Forum Posts: 8
Member Since:
February 23, 2014
sp_UserOfflineSmall Offline

Thanks for your reply!

I still have a few questions (Sorry, but I couldn't find them within the board and I guess others are also interested in the answers :)

--Android specific--

1.) I followed the manual and built the sbjniapp example locally but the sample character isn't being displayed. Should the example work out of the box?

 

2.) Now, if I want to create my own hello world Android application (which e.g. just contains a static line) from scratch, which libraries do I have to include? Do you probably have something like step-by-step instructions? Would be great.

 

--General--

3.) Unfortunately, every time I want to load one of the tutorial scripts (4_CreatingCharacter), the smartbody standalone application crashes.

 

Again, thank you so much for support!!

February 26, 2014
7:00 pm
Avatar
Admin
Forum Posts: 983
Member Since:
December 1, 2011
sp_UserOfflineSmall Offline

1) The latest version needs some fixes - Andrew will commit some changes to get it working again.

 

2) You'll need to include the same libraries as are in the sbjniapp - that's a fairly minimal Android application

 

3) Which SmartBody version are you using/did you download? What platform are you running it on? There's a smartbody.log file in the bin/ directory that shows some output, and if you send me an image of the call stack during the crash., that would be helpful too.

 

Ari

 

February 26, 2014
8:42 pm
Avatar
Admin
Forum Posts: 983
Member Since:
December 1, 2011
sp_UserOfflineSmall Offline

Ok, Andred checked in a fix (version 5477 in the trunk) that fixes the Android build problem. Can you update and try again?

 

Ari

 

February 27, 2014
2:10 pm
Avatar
Member
Members
Forum Posts: 8
Member Since:
February 23, 2014
sp_UserOfflineSmall Offline

Hi, thanks for your efforts!

 

I have built the updated project and now, the application shows some parts of the character, but it doesn't work entirely:

Screenshot 1  

When I install the apk file which was delievered with the project, there is no such problem.

 

By the way: I also made the experience (I run the application on a Samsung Galaxy S / Android 2.3.6) that when I press the home button and return to the application again, it becomes grey:

Screenshot 2

 

Thanks for your help! :)

February 27, 2014
7:13 pm
Avatar
Admin
Forum Posts: 52
Member Since:
August 8, 2012
sp_UserOfflineSmall Offline

Hi,

I have made a new update to fix the first issue of mesh not showing correctly. Please let me know if this fixes the problem on your end.

As for the second issue, I think that is related to the program not restoring all texture resource correctly. I will look into fixing this problem as well.

Andrew

February 28, 2014
11:44 am
Avatar
Member
Members
Forum Posts: 8
Member Since:
February 23, 2014
sp_UserOfflineSmall Offline

Hello Andrew,

thanks for the effort.

Unfortunately, it still doesn't fully work (although the skeleton lines are gone), as can be seen in this screenshot:

Screenshot

Does it work on your Android device?

 

February 28, 2014
7:10 pm
Avatar
Admin
Forum Posts: 52
Member Since:
August 8, 2012
sp_UserOfflineSmall Offline

Hi,
Yes, I tested the latest program on a Nexus 7 2nd Gen with NDK r8e. From the screenshot it looks like the texture is not working correctly. Can you let me know which NDK revision and device you are using for the test ? I can try to use the same setup to verify the cause.

Thanks,

Andrew

February 28, 2014
7:29 pm
Avatar
Admin
Forum Posts: 52
Member Since:
August 8, 2012
sp_UserOfflineSmall Offline

Also, here is the apk I am using on my device. 

https://www.dropbox.com/s/3e0u12ve9na4ujm/sbmjni.apk

Can you check to see if this apk works on your device ? Thanks.

Andrew

March 1, 2014
4:17 pm
Avatar
Member
Members
Forum Posts: 8
Member Since:
February 23, 2014
sp_UserOfflineSmall Offline

Hello Andrew,

I am sorry - it was my mistake - I only updated the Android folder and therefore missed the new version of /trunk/core/smartbody/SmartBody/src/sr/sr_gl_render_funcs.cpp.

Now, after building, the app works! :-) Thank you so much with this issue. By the way, I have tested it with both NDK r9c and NDK r8e. Probably you still have time with the restoring problem (which appears after pausing an application).

_______________

I know that most of this sample application should be quite self-explanatory, but I am still struggling a bit with the inner structure.

What I want to do now, is to display a face on my smartphone. My plan is to prepare everything on the standalone application and to use the Python scripts for the mobile application. Is the sbjniapp.cpp meant as a default library for android applications integrating SmartBody? So, can I just use it or do I have to create a customized library (e.g. I am quite confused that the script brad.py is called within the library? Why is there a server IP defined in the library?)?

Thank you so much for all the help. Your project team really does an excellent job.

March 3, 2014
11:43 pm
Avatar
Admin
Forum Posts: 983
Member Since:
December 1, 2011
sp_UserOfflineSmall Offline

Thanks.

 

The .py file contains the instructions to SmartBody for setup and possibly interaction with the virtual character.

sbjniapp.cpp handles the rendering and connection between the Android app and the SmartBody library.

SmartBody has the capability to connect to an ActiveMQ server and transmit/receive commands. You can comment out these lines if you don't want to use this capability - comment out lines 179 to 207 in sbjniapp.cpp).

 

Ari

 

 

 

March 21, 2014
4:31 pm
Avatar
Member
Members
Forum Posts: 47
Member Since:
March 1, 2014
sp_UserOfflineSmall Offline

Hi,

 

I would like to add one question in this context. If SmartBody is just an engine then what type of renderer/viewer should be used for Android applications?Should we create our own viewer?Because it might not be possible to use bulky 3rd party softwares just for rendering the virtual characters in deployment.

 

Thank you.

March 21, 2014
5:20 pm
Avatar
Admin
Forum Posts: 983
Member Since:
December 1, 2011
sp_UserOfflineSmall Offline

The SmartBody code base does include a scene graph renderer that is part of the 'engine' library (you can see this if you look at the core/smartbody/SmartBody/src/sr/ directory.) So in the absense of a separate rendering engine, you can use SmartBody's own internal one. You can see this in the Android examples that we provide (android/sbjniapp/jni/sbjniapp.cpp in drawSB(), in particular the call to SrGlRenderFuncs::renderDeformableMesh()).

 

So effectively, SmartBody can be its own self-contained engine-and-renderer for such platforms.

If you wanted to create, say, a complex scene with lots of geometry, special lighting effects, particles, and so forth, I would recommend a rendering engine. But for simply rendering a character and some simple objects in the scene, the built-in renderer works fine.

 

Ari

 

 

March 22, 2014
1:45 am
Avatar
Member
Members
Forum Posts: 47
Member Since:
March 1, 2014
sp_UserOfflineSmall Offline

Hi Ari,

Thanks for your answer. I will definitely try that one.

Can we leverage the SmartBody in Android upto the level of interactive agent(dynamically changing states and behaviors) by using the built-in renderer? Or will it be limited to render only the preconfigured objects?

I am asking this because I couldnot find satisfying clue in the manual, though the demo videos show interactive agents too.

 

Thank you.

March 22, 2014
2:00 am
Avatar
Admin
Forum Posts: 983
Member Since:
December 1, 2011
sp_UserOfflineSmall Offline

The Android renderer is similar to the rendering in the sbgui application; you can render characters, pawns, geometry and so forth under their dynamic conditions (animation, speech, whatever).

You can also modify the OpenGL code to render whatever else you want as well.

Does this answer your question?

Ari

 

March 22, 2014
7:32 pm
Avatar
Member
Members
Forum Posts: 47
Member Since:
March 1, 2014
sp_UserOfflineSmall Offline

Dynamic in the sense that if it can interact with the user input via console(or other medium) and render the behavior on the renderer. So is the BML only the way to make the character interactive to user inputs?

I am interested to explore the possibilities of using SmartBody to make a user interactive application.

 

Thank you.

March 24, 2014
4:47 pm
Avatar
Admin
Forum Posts: 983
Member Since:
December 1, 2011
sp_UserOfflineSmall Offline

You can use BML or you can use the Python command interface. Character actions/behaviors are generally controlled via BML. Scene state changes, responding to events, and so forth can be done using the Python scripting.

 

Ari

 

May 21, 2014
6:27 pm
Avatar
Member
Members
Forum Posts: 47
Member Since:
March 1, 2014
sp_UserOfflineSmall Offline

Hi,

I was trying to build sbjniapp and run in Android device. I tried building the Android application as instructed in the user manual and when I ran in my Nexus 4, I was only able to see the text field with the Run button.I had the sbjniappdir folder with its contents into the sdcard as well.

 But when I installed the file (https://www.dropbox.com/s/3e0u12ve9na4ujm/sbmjni.apk) provided by Andrew in the previous posts, I was able to see Brad, just showing some gestures. I tried giving some BML commands(ex: for head NOD) in the text field but couldn't render it.

Also, when I close the application and re-open it, the graphics is destroyed, I don't know if that's the problem of my Android device or something else.

Is there any elaborated user manual for Android specific settings for SmartBody?

 

Thank you.

May 22, 2014
12:13 am
Avatar
Admin
Forum Posts: 52
Member Since:
August 8, 2012
sp_UserOfflineSmall Offline

Hi,

I just tested the latest sbjniapp code on the Nexus 4. However, I can not reproduce the error of character not rendering. Can you let me know the revision number of SmartBody you use to build the apk ?

As for the BML command not working for head nod, I found an error in the setup script and have fixed it. You can replace the brad.py in sbjniappdir folder with this new file, and the BML command should work :

https://www.dropbox.com/s/c8nvepd72139ri1/brad.py

However, I am not sure about the reason the Android application you built didn't work on the device. Can you send me the .apk you created for me to test the debug message ? Also, here is the latest apk I just built.

https://www.dropbox.com/s/hi1o7lgr6t6afss/sbjniapp.apk

Thanks,

Andrew

May 22, 2014
6:10 pm
Avatar
Member
Members
Forum Posts: 47
Member Since:
March 1, 2014
sp_UserOfflineSmall Offline

Hi Andrew,

Thanks for your response.

The revision I used for sbjniapp was:
Last Commit Version:5487, Wednesday March 5,2014.

You were correct, after adding the skeleton in brad.py (the code that you provided) made the character do head nod.

In my build, the character was lost somewhere, I don't know if that was due to scrolling or something else.

Here is my apk file

https://www.dropbox.com/s/ryz9583i6lu20dq/sbmjni_newbuild.apk . It is working now.

 

I have another concern here. I am interested to extend this application to make the character speak. So I added some folders as per required by SpeechDemo.py. I renamed SpeechDemo.py to brad.py because I hope the same build may be able to execute it.

I tried to adjust the settings for ChrRachel as per the file SpeechDemo.py. But it seems that I am missing something. I have following files/folders inside my sdcard folder:

sbjniappdir

-brad.py

-commons.sk

-default.py

-zebra2-map.py

-ChrBrad.sk

-ChrBrad@Idle01_BeatHighBt01.skm

-HandsAtSide_Arms_Sweep.skm

-mesh

--ChrBrad (folder contains files ChrBradFlash.dae,ChrBrad_Flash_Dif.png)

--ChrRachel(folder that contains files ChrRachel.sk and all contents of folder face and motion for Rachel)

-scripts

--default-viewer.py

--init-diphoneDefault.py

--ogre-sinbad-map.py

--zebra2-map.py

 

Is this feasible?

Thank you.

May 22, 2014
6:41 pm
Avatar
Admin
Forum Posts: 983
Member Since:
December 1, 2011
sp_UserOfflineSmall Offline

Most of those supporting libraries are intact and work in the Android version.

You can use prerecorded speech on the Android platform in the same way that the desktop platform uses it.

We have examples of SmartBody with Android working with commercial text-to-speech engines (such as Cereproc).

If you only want to show a character or other simple elements in the 3D viewer, using Unity is a bit overkill.

 

Ari

May 22, 2014
7:11 pm
Avatar
Member
Members
Forum Posts: 47
Member Since:
March 1, 2014
sp_UserOfflineSmall Offline

Hi Ari,

Thanks for the response.

I prefer SBGUI in Android. Is it possible to achieve the android version of SmartBody(all those features in desktop version-TTS,gesture,bml windows...) with the configuration changes in the sbjniapp? I tried for this as stated in my previous post but no luck.

 

Thank you.

May 23, 2014
8:18 am
Avatar
New Member
Members
Forum Posts: 1
Member Since:
May 22, 2014
sp_UserOfflineSmall Offline

thanks for your apk file........

May 23, 2014
7:38 pm
Avatar
Admin
Forum Posts: 983
Member Since:
December 1, 2011
sp_UserOfflineSmall Offline

rbaral said
Hi Ari,

Thanks for the response.

I prefer SBGUI in Android. Is it possible to achieve the android version of SmartBody(all those features in desktop version-TTS,gesture,bml windows...) with the configuration changes in the sbjniapp? I tried for this as stated in my previous post but no luck.

 

Thank you.

Which features are not working for you? The message system should be working; you should be able to send a message from a desktop computer to your sbjniapp and control the character that way assuming that you know the mobile's IP number.

TTS works on the mobile platform, although we don't package the commercial TTS systems with the Android version (we can get you details on how to do that, however).

 

Obviously, we don't have the windows and drop down menus from sbgui on Android.

 

Ari

May 24, 2014
1:38 am
Avatar
Member
Members
Forum Posts: 47
Member Since:
March 1, 2014
sp_UserOfflineSmall Offline

Hi Ari,

 

Mine folder structure in sdcard is like this

https://www.dropbox.com/s/0pf6bs8a9p0cnh7/sbjniapp.png

 

I have brad.py as followings:

print "|--------------------------------------------|"
print "| data/sbm-common/scripts/brad.py |"
print "|--------------------------------------------|"

### Load data/sbm-common assets
scene.setMediaPath('/sdcard/sbjniappdir/')
#scene.addAssetPath('motion', './')
scene.addAssetPath("script", "scripts")
scene.addAssetPath("mesh", "mesh")
scene.addAssetPath("motion", "ChrBrad")
scene.loadAssets()
scene.setScale(1.0)
print 'media path = ' + scene.getMediaPath()

# set the default camera
camera = scene.getActiveCamera()
camera.setEye(0, 2.0478, 4.69259)
camera.setCenter(0.012, 1.60887, 2.75628)
camera.setUpVector(SrVec(0, 1, 0))
camera.setScale(1)
camera.setFov(0.4)
camera.setFarPlane(100)
camera.setNearPlane(0.1)

# map to the SmartBody standard
scene.run("zebra2-map.py")
zebra2Map = scene.getJointMapManager().getJointMap("zebra2")

 

bradSkeleton = scene.getSkeleton('ChrBrad.sk')
zebra2Map.applySkeleton(bradSkeleton)
#bradMotion = scene.getMotion('ChrBrad@Idle01_BeatHighBt01')
#zebra2Map.applyMotion(bradMotion)
zebra2Map.applyMotionRecurse('ChrBrad')
brad = scene.createCharacter("ChrBrad", "")
bradSkeleton = scene.createSkeleton("ChrBrad.sk")
brad.setSkeleton(bradSkeleton)
bradHPR = SrVec(0, 0, 0)
brad.setHPR(bradHPR)
bradPos = SrVec(0.0, 0.0, 0.0)
brad.setPosition(bradPos)
brad.createStandardControllers()
# deformable mesh
brad.setStringAttribute("deformableMesh", "ChrBradFlash.dae")

# start the simulation
sim.start()
bml.execBML('ChrBrad', '')
bml.execBML('ChrBrad', '')
sim.resume()

bml.execBML('ChrBrad', '')
bml.execBML('ChrBrad', '')

 

In this case, the character is not rendered. I was just trying to organize the files.

I am interested to make the Android version standalone rather than controlling from PC.

Also can you please elaborate the usage of commercial TTS system with Android version of SmartBody? 

Thank you.

May 27, 2014
11:08 pm
Avatar
Member
Members
Forum Posts: 47
Member Since:
March 1, 2014
sp_UserOfflineSmall Offline

Hi,

I came to know that the .dae file used for android app is smaller than the one used for PC. I tried for Rachel and couldn't manage to display in the screen. I simply took ChrRachel.sk, ChrRachel.dae(and related textures),the idle morph(..@idle.skm) for Rachel and placed in the appropriate folders. I updated the brad.py to adjust the changes.

I can see there is some distorted display but cant make the full character to be visible. I was interested in getting some lip synch in android.

I don't see any error in the log dialog.

May 28, 2014
3:46 pm
Avatar
Member
Members
Forum Posts: 47
Member Since:
March 1, 2014
sp_UserOfflineSmall Offline

Hi,

I am able to render Rachel with its mesh. Now I am interested to see the lipsync in mobile.

 

Thank you.

June 19, 2014
1:14 am
Avatar
Admin
Forum Posts: 983
Member Since:
December 1, 2011
sp_UserOfflineSmall Offline

Were you able to get the lip sync working on mobile?

Ari

June 27, 2014
9:48 pm
Avatar
Member
Members
Forum Posts: 47
Member Since:
March 1, 2014
sp_UserOfflineSmall Offline

Hi Ari,

Till now, no luck with the lip sync. I expected having the python script(speech.py) and the wireframe of Rachel(that has all the morphs required for facial animation), should be enough to get the lip sync. But there seems to be more beyond that. If, I can get it done, then it would be really awesome.

Thank you.

July 3, 2014
5:13 pm
Avatar
Admin
Forum Posts: 983
Member Since:
December 1, 2011
sp_UserOfflineSmall Offline

Where are you with this? You'll need to add to your script:

 

scene.addAssetPath("script", "c:/users/shapiro/smartbody/trunk/data/scripts") # or wherever this path is locally

scene.run('init-diphoneDefault.py')

brad.setStringAttribute('lipSyncSetName', 'default')
brad.setBoolAttribute('usePhoneBigram', True)

 

Ari

July 7, 2014
7:32 pm
Avatar
Member
Members
Forum Posts: 47
Member Since:
March 1, 2014
sp_UserOfflineSmall Offline

Hi Ari,

Yes, I had that too.

I wanted to use the Google TTS available in Android. I don't know how the TTSRelay(in PC version) can be replicated for Android.

In PC version there was the Apache ActiveMQ in between TTSRelay,Speech Engine, SmartBody. So getting lipsync with Android TTS seems to be really challenging with SmartBody.

 

Thank you.

July 7, 2014
9:44 pm
Avatar
Admin
Forum Posts: 52
Member Since:
August 8, 2012
sp_UserOfflineSmall Offline

Hi,

Getting TTS to work can be done through local relay. The local speech relay can handle the text-to-speech on the SmartBody process directly, without using ActiveMQ to send messages back and forth to a remote TTS relay. You can check local_speech.cpp : CereprocSpeechRelayLocal for some reference implementations. Basically the class will call the require TTS API directly, and create the synthesized .wav file and its corresponding viseme schedule for lip-syncing. 

Thus if Google TTS can output both the voice and its phoneme timings, you can integrate the TTS engine with SmartBody by extending the SpeechRelayLocal class (following a similar way we integrate Cereproc TTS). This will enable the SmartBody Android app to use Google TTS.

Andrew

July 8, 2014
11:00 pm
Avatar
Admin
Forum Posts: 983
Member Since:
December 1, 2011
sp_UserOfflineSmall Offline

Andrew,

 

Do you have a code snippet that we could post?

 

Ari