Victor Hung, Miguel Elvir, Avelino Gonzalez, Ronald DeMara
Intelligent Systems Laboratory
University of Central Florida
Orlando, FL
USA

victor@isl.ucf.edu, miguel.elvir@isl.ucf.edu, gonzalez@ucf.edu, demara@mail.ucf.edu

"Towards a Method For Evaluating Naturalness in Conversational Dialog Systems"
IEEE International Conference on Systems, Man, and Cybernetics, San Antonio, Texas, October, 2009.

The evaluation of conversational dialog systems
has always remained a controversial topic, as it is unclear on how
to quantitatively describe how well a conversation agent
performs, or how much better one is over another. Furthermore,
one of the hurdles in this quandary remains the definition of
naturalness, as in how well a dialog system can maintain a
natural conversation flow. This paper surveys existing evaluation
practices and provides its own methodology to determine the
effectiveness and naturalness of a dialog system. A prototype
evaluation methodology for the LifeLike virtual avatar project is
presented.