Adding a face to an AI

Update 02/12/2014: I made a slightly better version using graphics that implement a little bit of the actual mouth movements (see here for code):

So for a face, I thought I would try making some sort of realtime voice animating system. I made a very very simple one in python. Here is an example of what it looks like and sounds like:

The source code is very simple. Get the whole code at my github page (the following excludes the saySomething function which is part of mouth_function. Basically the following code opens a process to say something and at the same time tries to animate it with an open/closing mouth. The timing between the mouth open and mouth closed comes from an average timing I got from recording the Google TTS and recording how much time it takes to say a word and the amount of time between words.

The other trick is to count the number of syllables in a word. Python has a fast way of doing this using nltk (coded below).

import pygame, sys, time, random
from pygame.locals import *
from time import *
import curses
from curses.ascii import isdigit
import nltk
from nltk.corpus import cmudict
import os
import thread
import threading

d = cmudict.dict()

def nsyl(word):
    return [len(list(y for y in x if isdigit(y[-1]))) for x in d[word.lower()]]

pygame.init()
windowSurface = pygame.display.set_mode((500, 400), 0, 32)
pygame.display.set_caption("Bounce")

BLACK = (0, 0, 0)
WHITE = (255, 255, 255)
RED = (255, 0, 0)
GREEN = (0, 255, 0)
BLUE = (0, 0, 255)
YELLOW = (255,255,0)

info = pygame.display.Info()
sw = info.current_w
sh = info.current_h
y = 0
phrase = "Hi there. How are you doing"
windowSurface.fill(WHITE)
pygame.draw.circle(windowSurface, YELLOW , (250,200), 80, 0)
pygame.draw.circle(windowSurface, BLACK,(280,170), 10, 0)
pygame.draw.circle(windowSurface, BLACK,(220,170), 10, 0)

pygame.draw.ellipse(windowSurface,BLACK,(225,230,50,10),0)
myfont = pygame.font.SysFont("ComicSans", 35)
pygame.display.update()
sleep(1)
paragraph =  str(sys.argv[1])
thread.start_new_thread( saySomething,(paragraph,"en",))
workingSentence = ""
sleep(0.26)
for phrase in paragraph.split("?"):
	for sentence in phrase.split("."):
		for word in sentence.split():
			windowSurface.fill(WHITE)
			pygame.draw.circle(windowSurface, YELLOW , (250,200), 80, 0)
			pygame.draw.circle(windowSurface, BLACK,(280,170), 10, 0)
			pygame.draw.circle(windowSurface, BLACK,(220,170), 10, 0)

			pygame.draw.ellipse(windowSurface,BLACK,(225,220,50,30),0)
			myfont = pygame.font.SysFont("ComicSans", 17)
			workingSentence += word + " "
			label = myfont.render(workingSentence, 1, BLACK)
			windowSurface.blit(label, (5, 5))
			pygame.display.update()
			syl = nsyl(word)
			syl = syl[0]
			sleep(0.185*float(syl))

			windowSurface.fill(WHITE)
			pygame.draw.circle(windowSurface, YELLOW , (250,200), 80, 0)
			pygame.draw.circle(windowSurface, BLACK,(280,170), 10, 0)
			pygame.draw.circle(windowSurface, BLACK,(220,170), 10, 0)

			pygame.draw.ellipse(windowSurface,BLACK,(225,230,50,10),0)
			myfont = pygame.font.SysFont("ComicSans", 17)
			label = myfont.render(workingSentence, 1, BLACK)
			windowSurface.blit(label, (5, 5))
			pygame.display.update()
			sleep(0.082)
		windowSurface.fill(WHITE)
		pygame.draw.circle(windowSurface, YELLOW , (250,200), 80, 0)
		pygame.draw.circle(windowSurface, BLACK,(280,170), 10, 0)
		pygame.draw.circle(windowSurface, BLACK,(220,170), 10, 0)

		pygame.draw.ellipse(windowSurface,BLACK,(225,230,50,10),0)
		myfont = pygame.font.SysFont("ComicSans", 17)
		label = myfont.render(workingSentence, 1, BLACK)
		windowSurface.blit(label, (5, 5))
		pygame.display.update()
		sleep(0.16)

sleep(1)

That’s it! Future implementations will try to make the timing a little better. Unfortunately it would probably work better with a worse TTS because the Google TTS has some special phrasing for common phrases that changes the time in between words. Also, I am going to work with a linguist to figure out the right kind of mouth openings.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: