Language Agents in Interaction

Key Info

• Edstem: https://edstem.org/us/join/zGrhX5 (only accessible to Berkeley accounts)
• Reading Schedule (link to the role assignment sheet is available through Edstem)
• Meetings: Tuesday and Thursday 3:30-5pm, Soda 310 (in person only, no recordings will be made)
• Office Hours: by appointment
• Contact: suhr@berkeley.edu. Please include "294-258" in the email subject.
• Intro Slides: see this link for a direct link to a PDF of the first meeting's slides. This goes over course policies/structure and a list of research questions we might cover in the class.

Overview

In this discussion-based seminar course, we'll study language agents as agents (either human or automated) that use language (i.e., generates or understands utterances in the context of a multi-agent interaction). We'll discuss topics including (but not limited to): language and knowledge, discourse and situated dialogue, language acquisition and change, computational semantics and pragmatics, human-agent interaction, dialogue systems, and multimodal automated language agents (including computer vision, vision and language, robotics, reinforcement learning, etc.). Language Agents in Interaction will be centered around (a) reading and discussing foundational and recent work in (psycho)linguistics, cognitive science, information science, and computer science; and (b) a course project.

Readings will be roughly orthogonal to recent advancements in NLP/AI (including LLMs, etc.). While we will certainly discuss these technologies, the course will not be limited to recent language technology only, and discussions will be held from the variety of perspectives the course is intended to introduce (e.g., from the perspective of child language acquisition). We will only minimally or inadvertently discuss technical details (e.g., architectures, prompting methods) of modern approaches. This course will focus on language as an intrinsically interactive dynamical system, rather than uses of language data as a prior for an individual agent's perception, action, or decision-making in isolation from other agents.

Grading

Grading will be based on participation and a course project. See below for a breakdown of this. This is subject to change. Numerical grade to letter grade conversion will be determined near the end of the semester.

• Course participation (50%)
• Presentations as role representative (25%) I will assign roles so that you serve as the representative for roughly every role through the semester. You will end up as role representative several times through the course (about every other week) and I will average over each of those for this part of the grade.
• After-meeting summaries (25%) These are due before the next class meeting (I will close the Edstem discussion for each paper then). This grade will comprise the proportion of these you complete.
• Course project (50%) For each of these, there will be a final date by which you need to have submitted the work. However, I am more than happy to provide feedback at any time before the final due dates. This includes looking at drafts of written work, slides, or watching talk rehearsals. See below for more details about the project.
• Project proposal (15%) In the form of a "grant proposal", including meeting with me to discuss the proposal with your group.
• Final project report (20%) In the form of a paper that could be submitted to a conference or workshop.
• Final presentation (15%) In the form of a conference-style talk, though for group projects, this should be split across group members.

Discussions

In most of the class meetings, we'll hold discussions focused on paper(s) using a modified version of the role-playing paper-reading seminar described by Alec Jacobson and Colin Raffel. The roles we will use in this seminar are described below. I (Alane) will always be one of the role representatives on any given day; I'll rotate between roles in the order below.

Before each discussion-based meeting, you will read the assigned paper(s) and put together a short summary in a slidedeck:

1. Check your role and group number in the assignment sheet. These will be randomly assigned and posted in the Google sheet at least a week in advance of each meeting. You'll have the same role for every paper in each module.
2. Reach out to the other classmates with your role for the module, especially for roles that require more preparation in advance (e.g., Archaeologist). I'll put together Edstem threads for each role and module so you can coordinate.
3. For each meeting, check if you are assigned as a role representative, and if you cannot make a particular class meeting, please let me know ASAP so I can assign a new representative.
4. Read the assigned paper(s) with your role in mind, and complete any additional role-specific requirements (see below).
5. Add slides to the day's slidedeck to summarize your thoughts on each paper (the empty slidedeck will be posted on Edstem at least a week in advance of each meeting).
6. Sometimes I will encourage specific lightweight, non-graded activities to complete before each meeting, so please complete these as well!

During each discussion-based meeting:

1. We'll start with 10 minute small discussions splitting by assigned roles. During this time, each role representative will lead a discussion with the other people who have the role, and will prepare to present the slides.
2. Then, role representatives will present the role's slides to and lead a discussion with the whole class. Each role will get roughly 10 minutes. We'll go in the order of the roles below.
3. In the last 10 minutes of class, or with whatever time remains, we'll hold small discussions split by group numbers (with the group number you were assigned in the sheet). Each of these groups will have roughly one member of each role.

After each discussion-based meeting:

1. In the meeting-specific thread on Edstem, post a short reflection on the paper(s) and the discussion. See below for a description on what these summaries should look like!

Role Representatives

Through the semester, you'll be assigned as a role representative for roughly every one of the roles below. I anticipate you'll be a representative about once every other week. Since 25% of your total grade comprises this activity (which will primarily be evaluated on the presentation of your role to the class), it's strongly encouraged that you be proactive about reading, making slides, and contacting the other role members if necessary.

If you can't serve as a role representative for a given meeting, please let me know ASAP so I can assign it to somebody else.

Role Descriptions

Special responsibilities are underlined and bolded below! For all roles except the Original Author, the presentations should be under 5 minutes, so we have about 5 minutes of discussion available for each role.

• Original Author: Present the paper(s) as if you are an original author at a conference. Talks should be ~8 minutes, to allow some time for Q&A. If you are the role representative, then before the meeting, you should prepare the slides that will support an 8 minute conference-style talk (this is more preparation than what other roles require).
• Time Traveler (only at the end of each module): You are a researcher who can travel through time! You use your power to get all of the authors we've read for the current module into a room together to discuss their work. Your job is to report back what this discussion might look like to the class. How would the authors think of each others' work? What insights might they bring back to their own research? What assumptions are they making the others did not? How has the terminology and methodology changed over time and/or field?
• Archaeologist: Your job is to characterize the context in which the work was conducted. You'll "dig up" papers that strongly influenced (and were influenced by) the work we're discussing, and you'll look into the backgrounds of the authors and their interests at the time of the work. Why did the authors choose this research direction? What existing work influenced their choices in experimental design, assumptions made, and conclusions drawn from their study? What existing theories was the work building upon? What are the main innovations in this work in the context of what was happening at the same time? For what kind of work did this serve as a foundational element? Before the meeting, you should look at and skim (or fully read) influential papers the work cites (and was cited by) -- I suggest using Semantic Scholar to do this -- as well as information about the primary authors of the works. I also suggest sharing these related works ahead of time with the other Archaeologists so the discussion can focus on a small number of related works.
• Scientific Peer Reviewer: Your job is to evaluate the paper as if it was a submission to a modern conference venue that you've published in before, or a workshop associated with the conference. I suggest using the NeurIPS review form. Many of the older papers we read will not be appropriate submissions to NeurIPS -- that's ok! In that case, treat the paper as a workshop submission to a workshop affiliated with NeurIPS. It's also completely expected that these papers will be more difficult to review!
• Sociologist: Your job is to evaluate the existence of this work in a broader social context. This could include intrinsic questions about the work -- what assumptions does it make about people? Are there social biases embedded in the design of the experiments, or ethical concerns about methodology? Whose voices are being included and excluded in the scope of the work? This can also include extrinsic questions about the work and its conclusions -- what social implications might the work suggest? If the work proposes a kind of technology, what are the impacts, both positive and negative, that technology might have on society? On specific groups in society? How can we address concerns about ethical impacts?
• Industry Practitioner: You are responsible for transforming this work into a technology that will be deployed to consumers, or incorporating some of the work's findings into an existing technology. You want to convince someone to invest resources into your tech. Demos, sketches, or examples of the proposed product will make it more convincing!
• Academic Researcher: You'll create and present a proposal for a new research project (or projects) to the class. I suggest focusing your reading, discussion, and presentation on the set of research questions that the paper inspired: what does this paper make you curious about? If it's an older paper, how would you apply its findings to modern language technologies like LLMs? If it's a newer paper, how would you like to build on it? What experiments will you run, and what hypotheses will they test? How will you set up your experiments -- what data and models will you use, will human subjects be involved, what ablations will you try?

After-Meeting Summaries

After each meeting, you should post a short reflection on the paper and in-class discussions that were held. This can be pretty open-ended: my suggestion is to post a Tweet-length (~280 characters) comment or question on something in the discussions or paper that was particularly inspiring or intriguing to you (I highly recommend posing this as an open-ended research question!). Post this on the Edstem thread for the paper.

Project

More info on the project (groups, deliverables, due dates, expectations, etc.) will be coming soon!

Policies

Enrollment

Enrollment in this course is currently full. In general, my expectation is that students have some existing background in NLP (equivalent to CS 288). Students should be familiar with modern AI/ML tools.

COVID-19

COVID is still a serious illness. Personally, I now have long-term health issues that were triggered by a COVID infection in August 2022. I strongly encourage students to wear masks in the classroom. If you would like one, I will try to have several extras with me during meetings. If you are feeling sick, please do not come to class. Feel free to email me when you are feeling better. Though students are expected to attend each class meeting, I don't expect you to attend when you are ill, and this will not count against your attendance/participation-related evaluations.

Laptop Policy

Please feel free to bring and use your laptops / other electronics to class if it helps in the discussion (e.g. to look at your paper notes) or you use it to take notes. However, I expect students to actively participate in discussion, so please don't use your laptop for non-class related things. E.g., if you are on a tight paper deadline, please let me know that you can't attend and work on the paper in an environment that would be more conducive to writing.

AI Tools

You can use automated tools like Grammarly, Copilot, ChatGPT, etc. However, just like I will ask for attribution in final group project reports, you need to report the use of these tools in any relevant artifacts you produce (code and writing).

Academic Integrity and Collaboration

In day-to-day readings and discussions, you should read papers independently, but discussions will be heavily collaborative. After-meeting summaries should be your own work. In the course project, I strongly encourage group work: however, please keep me updated throughout the semester with respect to who is in which groups, and in final project, you will need to report attribution for different members of the group (i.e., who did what). You may not copy other students' work. Academic integrity and ethical conduct are of utmost importance at UC Berkeley.

Inclusion

We are committed to creating a learning environment welcoming of all students that supports a diversity of thoughts, perspectives and experiences, and respects your identities and backgrounds (including race/ethnicity, nationality, gender identity, socioeconomic class, sexual orientation, language, religion, ability, etc.) To help accomplish this:

• If you have a name and/or set of pronouns that differ from those that appear in your official records, please let us know. The first-week intro form will allow you to share this with me.
• If you feel like your performance in the class is being impacted by your experiences outside of class (e.g., family matters, current events), please don’t hesitate to come and talk with us. We want to be resources for you.
• We (like many people) are still in the process of learning about diverse perspectives and identities. If something was said in class (by anyone) that made you feel uncomfortable, please talk to us about it. You may also contact the CS department’s Faculty Equity Advisor Prof. Armando Fox (fox@berkeley.edu).
• As a participant in this class, recognize that you can be proactive about making other students feel included and respected.

Accomodations

We honor and respect the different learning needs of our students, and are committed to ensuring you have the resources you need to succeed in our class. If you need religious or disability-related accommodations, if you have emergency medical information you wish to share with us, or if you need special arrangements in case the building must be evacuated, please share this information with us as soon as possible. You may speak with either instructor privately after class or during office hours.

The Disabled Student’s Program (DSP, 260 César Chávez Student Center #4250; 510-642-0518; http://dsp.berkeley.edu) serves students with disabilities of all kinds. Services are individually designed and based on the specific needs of each student as identified by DSP's Specialists.