Data from surveys and conversation logs were anonymised in Anonymise Data.ipynb and converted into pickle files.
These pickle files represent demographic data (df_dem.pkl), chatbot data (df_chatbotdata.pkl) and a combination of all data from the study (df_merged.pkl).
An in depth analysis into user enagement can be found in FYP-Analysis.ipynb where pickle files were used. A description of the analysis (FYP-Analysis.ipynb) can be found below:
The FYP-Analysis.ipynb notebook provides an overview of this research product by describing the research questions, the data gathered, and the analysis conducted. All analysis is complete in this notebook.
- Which chatbot did the participant prefer? (show counts for each chatbot - bar chart, pie chart, ratio etc.)
- How does this align with the participant's personality? (correlation between extraversion and preference, agreebleness and preference, point biserial correlation)
- Is the preferred chatbot experience positively correlated with conversation engagement metrics such as conversation length?
- Is the preferred chatbot experience positively correlated with ratings provided by the user such as Quality of Conversation?
- Do participants reliably identify differences between chatbot personality?
- What languguage do participants used to describe each chatbot?
- Is this as intended?
- Do participants prefer one chatbot over the other because of its personality?
- Do self-described personality traits (beliefs) correlate with interaction metrics (behaviour)?
- Can we use NLP on participant utterances to extract personality traits?
- How do extracted personality traits correlate with self-described personality traits?
Data for this analysis was gathered via an online user study in which participants interacted with two chatbots designed with disctinct personalities. The implementation of the chatbots can be found in the following two repositories: https://github.com/sineadfarrell/Makoto-Bot & https://github.com/sineadfarrell/Nasoto-Bot
The personalities differ across two factors of the Big 5 Personality Trait model (https://en.wikipedia.org/wiki/Big_Five_personality_traits). These factors are Extroversion (outgoing/energetic vs. solitary/reserved) and Agreeableness (friendly/compassionate vs. challenging/detached). Participants filled out a demographic and personality survey before interacting with the chatbots. After each interaction the participants filled out a survey about how they perceived the chatbot they just used. At the end of the experiment participants filled out a final survey describing which chatbot they preferred.
We analyse survey responses (text) as well as conversation logs (json). Extracting features from the json data including number of conversation turns, average length of user utterances etc.