Innovative Applications of Natural Language Processing and Digital Media in Theatre and Performing Arts
Keywords:
natural language processing, digital art, virtual characters, immersive theatreAbstract
The objective of our research is to investigate new digital techniques and tools, offering the audience innovative, attractive, enhanced and accessible experiences. The project focuses on performing arts, particularly theatre, aiming at designing, implementing, experimenting and evaluating technologies and tools that expand the semiotic code of a performance by offering new opportunities and aesthetic means in stage art and by introducing parallel accessible narrative flows. In our novel paradigm, modern technologies emphasize the stage elements providing a multilevel, intense and immersive theatrical experience. Moreover, lighting, video projections, audio clips and digital characters are incorporated, bringing unique aesthetic features. We also attempt to remove sensory and language barriers faced by some audiences. Accessibility features consist of subtitles, sign language and audio description. The project emphasises on natural language processing technologies, embedded communication and multimodal interaction to monitor automatically the time flow of a performance. Based on this, pre-designed and directed stage elements are being mapped to appropriate parts of the script and activated automatically by using the virtual "world" and appropriate sensors, while accessibility flows are dynamically synchronized with the stage action. The tools above are currently adapted within two experimental theatrical plays for validation purposes.
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
References
Ars Electronica. (2006), “Le sacre du printemps” (The rite of spring), available at: https://ars.electronica.art/futurelab/en/project/le-sacre-du-printemps/ (May 22, 2020)
Bird, T. (2006), “Sunday in the Park with George”, available at: http://www.projctn.com/sunday-park-george (22 April 2020)
Costea, G.-A. (2018), “Do you speak tech? Report from an interdisciplinary and cross-cultural partnership”, available at: https://www.europeantheatrelab.eu/speak-tech-report-interdisciplinary-cross-cultural-partnership/ (22 May 2020)
Fiveash, K. (2016), “The Tempest review: Real-time digital avatar brews storm in a teacup”, available at: https://arstechnica.com/gaming/2016/11/tempest-review-real-time-digital-avatar-performance-capture/ (22 April 2020)
Gao, J., Zhao, Q., Yan, Y. (2011), “Towards precise and robust automatic synchronization of live speech and its transcripts”, Speech Communication, Vol. 53, No. 4, pp. 508-523.
González-Carrasco, I., Puente, L., Ruiz-Mezcua, B., López-Cuadrado, J. L. (2019), "Sub-sync: automatic synchronization of subtitles in the broadcasting of true live programs in spanish," IEEE Access, vol. 7, pp. 60968-60983.
Heimbach, H., Klein, A., Lin, A., Lu, M. (2016), “Developing theatrical lighting control with Arduino”, available at: https://soe.rutgers.edu/sites/default/files/imce/pdfs/gset-2014/GSET2015-DevelopingTheatrical.pdf (22 May 2020)
Henry M. (2019), “Idiomatic”, available at: https://theatredeliege.be/evenement/idiomatic/ (22 April 2020)
Mathewson, K. W., Mirowski, P. (2017), “Improvised theatre alongside artificial intelligences”, The Thirteenth AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment AIIDE-17, 5-9 October, Association for the Advancement of Artificial Intelligence, Palo Alto, California, pp. 66-72.
Perkowski, M., Sasao, T., Kim, J. H., Lukac, M., Allen, J., Gebauer, S. (2005), “Hahoe KAIST robot theatre: Learning rules of interactive robot behavior as a multiple-valued logic synthesis problem”, 35th International Symposium on Multiple-Valued Logic ISMVL ‘05, 19-21 May, IEEE, Calgari, Canada, pp. 236-248.
Secară, A. (2018), “Surtitling and captioning for theatre and opera”, In Pérez-González, L. (Ed.), The Routledge Handbook of Audiovisual Translation, Routledge, London, pp. 130-144.
Shaw D. (2012), “Transforming digital drama: the technology”, available at: http://www.bbc.com/news/technology-17079364 (22 May 2020)
Siddiqui M. U. (2015), “Hakanai | Adrien M & Claire B”, available at: https://www.arch2o.com/adrien-m-claire-b-hakanai/ (22 April 2020)
Thonon, J. (2018), “The re-invention of surtitling. a new technical approach to the international market”, in Pfaud, T., Dusol, J., Wiley, H. (Eds.), Digital theatre - a casebook, European Theatre Convention, Berlin pp. 50-55.
