tractatus/docs/stakeholder-recruitment/post-deliberation-feedback-survey.md

Post-Deliberation Feedback Survey

AI-Led Deliberation - Stakeholder Experience Assessment

Project: Tractatus Pluralistic Deliberation Pilot Scenario: Algorithmic Hiring Transparency Survey Purpose: Assess AI facilitation quality and stakeholder experience Estimated Time: 10-15 minutes Confidentiality: Your responses will be anonymized in published research outputs

---

Instructions

Thank you for participating in this AI-led deliberation! Your feedback is essential for improving AI facilitation quality and determining whether this approach is viable for future governance processes.

This survey has three sections:

1. Quantitative Ratings (rate AI facilitation on various dimensions)
2. Qualitative Feedback (open-ended questions about your experience)
3. Demographic/Role Confirmation (for research analysis)

Please be honest. There are no "right" answers, and critical feedback is just as valuable as positive feedback.

---

SECTION 1: Quantitative Ratings

Overall AI Facilitation Quality

Question 1.1: Overall, how would you rate the quality of AI facilitation in this deliberation?

☐ 5 - Excellent (AI facilitated exceptionally well) ☐ 4 - Good (AI facilitated well with minor issues) ☐ 3 - Acceptable (AI facilitated adequately but had noticeable issues) ☐ 2 - Poor (AI facilitation had significant problems) ☐ 1 - Unacceptable (AI facilitation was inadequate; human should have led)

If you rated 1-2 (Poor/Unacceptable), please explain why: [Open text field]

---

Specific Dimensions of AI Facilitation

For each dimension below, rate the AI's performance on a 5-point scale:

• 5 = Excellent
• 4 = Good
• 3 = Acceptable
• 2 = Poor
• 1 = Unacceptable

---

Question 1.2: FAIRNESS - Did the AI treat all stakeholders equally?

☐ 5 - Excellent (AI gave equal time/attention to all stakeholders) ☐ 4 - Good (AI was mostly fair with minor imbalances) ☐ 3 - Acceptable (AI was generally fair but some stakeholders got more attention) ☐ 2 - Poor (AI noticeably favored some stakeholders over others) ☐ 1 - Unacceptable (AI was unfair or biased)

If you rated 1-2, please explain: [Open text field]

---

Question 1.3: CLARITY - Was the AI's communication clear and easy to understand?

☐ 5 - Excellent (AI was always clear; no jargon or confusion) ☐ 4 - Good (AI was mostly clear with rare confusing moments) ☐ 3 - Acceptable (AI was usually clear but sometimes used jargon or unclear language) ☐ 2 - Poor (AI was often confusing or hard to understand) ☐ 1 - Unacceptable (AI's communication was frequently unclear)

If you rated 1-3, what specifically was unclear? [Open text field]

---

Question 1.4: CULTURAL SENSITIVITY - Was the AI respectful of diverse perspectives and cultural contexts?

☐ 5 - Excellent (AI was highly sensitive; no insensitive or stigmatizing language) ☐ 4 - Good (AI was mostly sensitive with rare lapses) ☐ 3 - Acceptable (AI was generally respectful but had some insensitive moments) ☐ 2 - Poor (AI used problematic framing or language multiple times) ☐ 1 - Unacceptable (AI was insensitive or offensive)

If you rated 1-3, please describe the insensitive framing: [Open text field]

---

Question 1.5: NEUTRALITY - Did the AI remain neutral, or did it advocate for a specific position?

☐ 5 - Excellent (AI was completely neutral; never favored any position) ☐ 4 - Good (AI was mostly neutral with rare moments of subtle advocacy) ☐ 3 - Acceptable (AI was generally neutral but occasionally seemed to favor certain options) ☐ 2 - Poor (AI noticeably favored some positions over others) ☐ 1 - Unacceptable (AI advocated for a specific outcome instead of facilitating)

If you rated 1-3, which position(s) did the AI seem to favor? [Open text field]

---

Question 1.6: RESPONSIVENESS - Did the AI adapt to stakeholder feedback and requests?

☐ 5 - Excellent (AI always adjusted when we provided feedback) ☐ 4 - Good (AI usually adapted with rare instances of not adjusting) ☐ 3 - Acceptable (AI sometimes adapted but also ignored feedback) ☐ 2 - Poor (AI rarely adapted to our feedback) ☐ 1 - Unacceptable (AI never adjusted; felt rigid or scripted)

If you rated 1-3, please provide an example: [Open text field]

---

Question 1.7: ACCURACY - Did the AI accurately represent your position in summaries?

☐ 5 - Excellent (AI always represented my position accurately) ☐ 4 - Good (AI was mostly accurate with minor errors) ☐ 3 - Acceptable (AI was generally accurate but missed some nuance) ☐ 2 - Poor (AI misrepresented my position multiple times) ☐ 1 - Unacceptable (AI consistently misrepresented my position)

If you rated 1-3, how did the AI misrepresent your position? [Open text field]

---

Question 1.8: TRUST - Did you feel safe and comfortable with AI facilitation?

☐ 5 - Completely comfortable (I trusted the AI facilitation fully) ☐ 4 - Mostly comfortable (I had minor reservations but felt safe overall) ☐ 3 - Somewhat comfortable (I had some concerns but continued participating) ☐ 2 - Uncomfortable (I had significant concerns about AI facilitation) ☐ 1 - Very uncomfortable (I did not trust AI facilitation and wished it were human-led)

If you rated 1-3, what made you uncomfortable? [Open text field]

---

Human Observer Performance

Question 1.9: HUMAN PRESENCE - How would you rate the human observer's performance?

Attentiveness: ☐ 5 - Excellent (Human was clearly present and monitoring closely) ☐ 4 - Good (Human seemed attentive most of the time) ☐ 3 - Acceptable (Human seemed present but not always engaged) ☐ 2 - Poor (Human seemed distracted or disengaged) ☐ 1 - Unacceptable (Human did not appear to be monitoring)

Intervention Appropriateness: ☐ 5 - Excellent (Human intervened exactly when needed, not too often or too rarely) ☐ 4 - Good (Human intervention timing was mostly appropriate) ☐ 3 - Acceptable (Human intervened but timing was sometimes off) ☐ 2 - Poor (Human intervened too often or too rarely) ☐ 1 - Unacceptable (Human failed to intervene when needed OR intervened unnecessarily)

If you felt human observer should have intervened but didn't, please explain: [Open text field]

---

Process Quality

Question 1.10: STRUCTURE - How would you rate the 4-round deliberation structure?

☐ 5 - Excellent (Structure was logical and helpful) ☐ 4 - Good (Structure worked well with minor issues) ☐ 3 - Acceptable (Structure was okay but could be improved) ☐ 2 - Poor (Structure was confusing or unhelpful) ☐ 1 - Unacceptable (Structure was ineffective)

Suggestions for improving the structure: [Open text field]

---

Question 1.11: DURATION - Was the deliberation duration (4 hours over 2 sessions) appropriate?

☐ Too short (needed more time) ☐ Just right ☐ Too long (felt rushed or exhausting)

If too short or too long, what would be ideal? [Open text field]

---

Question 1.12: OUTCOME - Do you feel the outcome document accurately represents what happened in the deliberation?

☐ 5 - Completely accurate ☐ 4 - Mostly accurate with minor inaccuracies ☐ 3 - Somewhat accurate but missed important points ☐ 2 - Inaccurate in several ways ☐ 1 - Very inaccurate

If you rated 1-3, what was inaccurate or missing? [Open text field]

---

SECTION 2: Qualitative Feedback

Open-Ended Questions

Question 2.1: What did the AI do WELL? What were the AI's strengths as a facilitator?

[Open text field - 2-5 sentences]

---

Question 2.2: What did the AI do POORLY? What were the AI's weaknesses as a facilitator?

[Open text field - 2-5 sentences]

---

Question 2.3: Were there any moments during the deliberation where you felt uncomfortable, frustrated, or distressed? If yes, please describe.

[Open text field - 2-5 sentences]

---

Question 2.4: Did the AI use any language or framing that you found problematic, insensitive, or biased? If yes, please provide examples.

[Open text field - 2-5 sentences]

---

Question 2.5: How did AI facilitation compare to your expectations? Was it better, worse, or about the same as you expected?

☐ Much better than expected ☐ Somewhat better than expected ☐ About what I expected ☐ Somewhat worse than expected ☐ Much worse than expected

Please explain: [Open text field - 2-3 sentences]

---

Question 2.6: If you could change ONE thing about the AI facilitation, what would it be?

[Open text field - 1-3 sentences]

---

Question 2.7: Would you participate in a similar AI-led deliberation in the future?

☐ Definitely yes (I would be eager to participate again) ☐ Probably yes (I would likely participate again with minor reservations) ☐ Unsure (I'm not sure if I would participate again) ☐ Probably no (I would be hesitant to participate again) ☐ Definitely no (I would not participate in AI-led deliberation again)

Please explain your reasoning: [Open text field - 2-3 sentences]

---

Question 2.8: If you answered "Probably no" or "Definitely no" above, what would need to change for you to participate in AI-led deliberation in the future?

[Open text field - 2-3 sentences]

---

Question 2.9: Do you have any other feedback about the AI facilitation, the human observer, or the deliberation process overall?

[Open text field - open-ended]

---

SECTION 3: Demographic & Role Confirmation

Basic Information

Question 3.1: What stakeholder role did you represent in this deliberation?

☐ Job Applicant Advocate ☐ Employer / HR Representative ☐ AI Vendor Representative ☐ Regulator / Policy Expert ☐ Labor Rights Advocate ☐ AI Ethics Researcher

---

Question 3.2: How much experience do you have with algorithmic hiring (either as applicant, employer, vendor, regulator, or researcher)?

☐ Extensive (10+ years) ☐ Substantial (5-10 years) ☐ Moderate (2-5 years) ☐ Limited (1-2 years) ☐ Minimal (less than 1 year)

---

Question 3.3: Prior to this deliberation, how familiar were you with AI systems in general?

☐ Very familiar (I work with AI regularly) ☐ Somewhat familiar (I understand AI concepts) ☐ Slightly familiar (I've heard of AI but don't use it much) ☐ Not familiar (I had little knowledge of AI before this)

---

Question 3.4: Prior to this deliberation, how familiar were you with pluralistic deliberation or similar participatory governance processes?

☐ Very familiar (I've participated in or facilitated similar processes) ☐ Somewhat familiar (I've read about these processes) ☐ Slightly familiar (I've heard of them but didn't know details) ☐ Not familiar (This was my first time hearing about pluralistic deliberation)

---

Optional Demographic Information

These questions are OPTIONAL and will only be used for aggregate analysis (e.g., "Did stakeholders from different demographics experience AI facilitation differently?"). Your individual responses will remain confidential.

Question 3.5 (Optional): What is your age range?

☐ 18-24 ☐ 25-34 ☐ 35-44 ☐ 45-54 ☐ 55-64 ☐ 65+ ☐ Prefer not to say

---

Question 3.6 (Optional): What is your gender identity?

☐ Woman ☐ Man ☐ Non-binary ☐ Prefer to self-describe: _______________ ☐ Prefer not to say

---

Question 3.7 (Optional): What is your racial/ethnic identity? (Select all that apply)

☐ Asian / Asian American ☐ Black / African American ☐ Hispanic / Latino/a/x ☐ Indigenous / Native American ☐ Middle Eastern / North African ☐ White / Caucasian ☐ Multiracial ☐ Prefer to self-describe: _______________ ☐ Prefer not to say

---

Question 3.8 (Optional): Do you have a disability?

☐ Yes ☐ No ☐ Prefer not to say

---

Public Attribution (Optional Decision)

Question 3.9: The outcome document and transparency report will be published. By default, your identity will be pseudonymized (e.g., "Employer Representative A").

Would you like to be publicly identified by name in research outputs?

☐ Yes, I opt IN to public attribution

• My name, title, and organization may be listed as a participant
• I may be quoted by name in publications or presentations
• Video of my participation may be used in demonstrations

☐ No, I prefer to remain pseudonymous

• I will be referred to by pseudonym only (e.g., "Employer Representative A")
• No identifying information will be shared
• I can still be quoted, but anonymously

☐ I'm unsure - please contact me to discuss

If you opt IN to public attribution, please confirm your preferred attribution:

Name: _______________________________________ Title: _______________________________________ Organization: _______________________________________

---

SECTION 4: Contact & Follow-Up

Question 4.1: May we contact you in the future for:

• Follow-up questions about this deliberation
• Participation in future deliberations on related topics
• Updates on research findings and publications

☐ Yes, you may contact me for any of the above ☐ Yes, but only for: [Specify which purposes] ☐ No, please do not contact me after this deliberation ends

Preferred contact method: ☐ Email ☐ Phone ☐ Other: _______________

---

Thank You!

Thank you for completing this survey. Your feedback will directly inform:

• Improvements to AI facilitation quality
• Decisions about whether AI-led deliberation is viable for future governance processes
• Research publications on AI-assisted pluralistic deliberation

Your participation in this deliberation and your honest feedback are deeply appreciated.

If you have any questions or concerns, please contact:

• Project Lead: [NAME, EMAIL, PHONE]
• AI Safety Lead: [NAME, EMAIL]

---

Survey Version: 1.0 Date: 2025-10-17 Estimated Completion Time: 10-15 minutes

---

Survey Administration Instructions (For Project Team)

When to Send:

• Week 4 (asynchronous refinement period)
• Send simultaneously with outcome document and transparency report (stakeholders can reference these while completing survey)

How to Send:

• Online survey platform (Google Forms, Qualtrics, TypeForm, etc.)
• Email survey link to all 6 stakeholders
• Reminder emails: Day 3, Day 5, Day 7 if not completed

Deadline:

• 1 week after sending (7 days)
• Extension available upon request

Confidentiality:

• Individual responses confidential
• Aggregate results published in transparency report
• If stakeholder opts into public attribution, their identity may be linked to their feedback (with their explicit consent)

Analysis:

• Calculate average ratings for each dimension
• Identify themes in qualitative feedback
• Report results in transparency report (Section 7: Stakeholder Feedback Summary)