How confident are you in your ability to evaluate the credibility of information from different sources?
(Likert scale: 1 = Not confident at all, 5 = Extremely confident)
Section 2: General Familiarity with LLMs
/// This section explores participants’ general knowledge and usage of LLMs.
1\. Have you heard of or used large language models (LLMs) like ChatGPT, GPT-4, or Bard before?
(Yes/No)
2\. How familiar are you with LLMs and their general applications?
(Likert scale: 1 \= Not familiar at all, 5 \= Extremely familiar)
3\. If you have used LLMs, for what purposes have you primarily used them?
(Check all that apply)
• Content creation (e.g., writing, editing)
• Learning or research
• Code generation or debugging
• Personal assistance (e.g., scheduling, answering general questions)
• Entertainment
• Other (Open-ended)
• I have not used LLMs
4\. How often do you use LLMs in your daily life?
(Multiple choice: Never, Less than once a week, 1-3 times a week, Almost every day, Daily)
### Section 3: Perception of Information Presentation
This section aims to assess how participants perceive the information presentation by LLMs.
1\. When interacting with LLMs, how clear and easy to understand do you find the information they provide?
(Likert scale: 1 \= Not clear at all, 5 \= Extremely clear)
2\. How neutral or unbiased do you find the information presented by LLMs compared to other sources?
(Multiple choice: Much more neutral, More neutral, About the same, Less neutral, Much less neutral)
3\. What aspects of LLM-generated information stand out to you the most?
(Check all that apply)
• Clarity
• Coherence
• Relevance
• Objectivity
• Depth of detail
• Creativity
• I have not interacted with LLMs enough to comment
• Other (Open-ended)
4\. How often do you fact-check the information provided by LLMs?
(Multiple choice: Never, Rarely, Sometimes, Often, Always)
### Section 4: Trust and Credibility of LLMs
This section evaluates how participants view the credibility and trustworthiness of LLMs.
1\. How confident are you in the accuracy of the information provided by LLMs?
(Likert scale: 1 \= Not confident at all, 5 \= Extremely confident)
2\. What factors influence your trust in information from LLMs?
(Check all that apply)
• The LLM’s ability to provide citations
• The perceived neutrality of the content
• Similarity to information you already know
• Complexity of the topic
• Recommendations from others
• None of the above
• Other (Open-ended)
3\. How reliable do you think the information provided by LLMs is?
(Likert scale: 1 \= Not reliable at all, 5 \= Extremely reliable)
4\. Have you encountered incorrect or misleading information from LLMs?
(Yes/No)
• If Yes, how often do you encounter such errors?
(Multiple choice: Rarely, Sometimes, Often, Always)
5\. How important is it to you that LLMs present information in a neutral and unbiased manner?
(Likert scale: 1 \= Not important at all, 5 \= Extremely important)
### Section 5: Resolving Conflicting Information
// This section explores how participants handle conflicting or unclear information provided by LLMs.
1\. When faced with conflicting information from LLMs, how do you usually verify its accuracy?
(Check all that apply)
• Cross-check with other sources
• Ask follow-up questions to the LLM
• Consult an expert
• Use personal knowledge
• I don’t attempt to resolve it
• Other (Open-ended)
2\. How often do you encounter inconsistencies in the information provided by LLMs?
(Multiple choice: Never, Rarely, Sometimes, Often, Always)
3\. If you encounter inconsistencies, how often do you re-ask the LLM for clarification or further explanation?
(Multiple choice: Never, Rarely, Sometimes, Often, Always)
### Section 6: Pre-existing Biases and Perceptions
// This section ensures that questions on pre-existing biases are asked only after gathering impartial views.
1\. Before using LLMs, what was your perception of their ability to generate accurate and reliable information?
(Multiple choice: Very positive, Positive, Neutral, Negative, Very negative)
2\. Has your experience with LLMs changed your perception of their accuracy and reliability?
(Multiple choice: Yes, they are more accurate/reliable than I thought; Yes, they are less accurate/reliable than I thought; No, my perception has not changed)
3\. Do you believe that your own biases can affect how you interpret information provided by LLMs?
(Yes/No/Not sure)
4\. How often do you critically evaluate information from LLMs compared to other sources (e.g., traditional media, human experts)?
(Likert scale: 1 \= Much less often, 5 \= Much more often)
Key Response Types
• Likert Scale:
1 \= Not at all / No confidence / Not important
5 \= Extremely / Very confident / Extremely important
• Multiple Choice:
Standard options tailored to each question to ensure neutral phrasing and non-leading responses.
• Check All That Apply:
Allows participants to express multiple factors or approaches where relevant.
• Open-ended:
For qualitative insights where participant elaboration is needed.