Course Description

This course covers several important machine learning algorithms for natural language processing including decision tree, kNN, Naive Bayes, support vector machine, maximum entropy / multinomial logistic regression, conditional random fields, and neural networks. Students implement many of the algorithms and apply these algorithms to some NLP tasks.

Days Time Location
Tuesday and Thursday 1:00 - 2:20 PM ECE 045

Teaching Staff

Role Name Office Office Hours
Instructor Shane Steinert-Threlkeld Guggenheim 418-D (and Zoom) Tuesday, 2:30 - 4:30 PM
Teaching Assistant Yuanhe Tian Guggenheim 417 (the Treehouse) Wednesday, 3 - 4 PM
Friday 10 - 11 AM

Textbook

There is no required textbook. Instead, the course readings will be drawn from contemporary articles and tutorials available online. Some material will be pulled from the following books, which will be referenced by their parenthecized names:

Prerequisites

  • CSE 373 (Data Structures) or Equivalent
  • MATH/STAT 394 (Intro to Probability) or Equivalent
  • LING 570
  • Programming in one or more of Java, Python, C/C++, or Perl
  • Linux/Unix Commands

Course Resources

N.B.: All homework grading will take place on the patas cluster using Condor, so your code must run there. I strongly encourage you to ensure you have an account set up by the time of the first course meeting.

Policies

Unless explicitly mentioned below, the shared policies of the LING 57x course series apply to this course. Please read those policies for more information.

Grading

  • 100%: Homework Assignments
  • Up to 2% adjustment for significant in-class or discussion participation

Communication

As per the policy above, all communication outside of the classroom should take place on Canvas. You can expect responses from teaching staff within 48 hours, but only during normal business hours, and excluding weekends.

N.B.: while CLMS students have a private Slack channel, I strongly encourage questions concerning course content and assignments to be posted to the Canvas discussion board, for two reasons. (i) Teaching staff will not look at Canvas, so misinformation can spread. (ii) Not every student in the course is in the CLMS program, but they deserve to be included in course discussions and likely have many of the same questions.

Religious Accommodation

Washington state law requires that UW develop a policy for accommodation of student absences or significant hardship due to reasons of faith or conscience, or for organized religious activities. The UW’s policy, including more information about how to request an accommodation, is available at Religious Accommodations Policy (https://registrar.washington.edu/staffandfaculty/religious-accommodations-policy/). Accommodations must be requested within the first two weeks of this course using the Religious Accommodations Request form (https://registrar.washington.edu/students/religious-accommodations-request/).

Access and Accommodations

Your experience in this class is important to me. If you have already established accommodations with Disability Resources for Students (DRS), please communicate your approved accommodations to me at your earliest convenience so we can discuss your needs in this course.

If you have not yet established services through DRS, but have a temporary health condition or permanent disability that requires accommodations (conditions include but not limited to; mental health, attention-related, learning, vision, hearing, physical or health impacts), you are welcome to contact DRS at 206-543-8924 or uwdrs@uw.edu or disability.uw.edu. DRS offers resources and coordinates reasonable accommodations for students with disabilities and/or temporary health conditions. Reasonable accommodations are established through an interactive process between you, your instructor(s) and DRS. It is the policy and practice of the University of Washington to create inclusive and accessible learning environments consistent with federal and state law.

Safety

Call SafeCampus at 206-685-7233 anytime – no matter where you work or study – to anonymously discuss safety and well-being concerns for yourself or others. SafeCampus’s team of caring professionals will provide individualized support, while discussing short- and long-term solutions and connecting you with additional resources when requested.

Schedule


Date Topics Suggested Readings Assignment out Assignment due
Jan 7 Overview
Information theory


Background:
[classification] [probability] [MALLET]
MS 2.2
Cover and Thomas, ch 2
HW1 out
[pdf, tex]
Jan 9 Decision Trees ML ch 9
Jan 14 Naive Bayes IR 13.2-13.4
JM ch 4
McCallum and Nigam 1998
HW2 out
[pdf, tex, slides]
Jan 16 k-Nearest Neighbors (kNN) IR 14.3 -14.4 HW1 due
Jan 21 Feature Selection IR ch 13.5
Yang and Pedersen 1997
HW3 out
[pdf, tex, slides]
Reading #1 [pdf]
Jan 23 Chi-square
Unit 1 recap
IR ch 13.5 HW2 due
Jan 28 Optimization
Maximum entropy (MaxEnt) I: main concept
Berger et al 1996, "A maximum entropy approach to NLP" HW4 out
[pdf, tex]
Reading #1 in
Jan 30 MaxEnt II: modeling and decoding Ratnaparkhi 1997 HW3 in
Feb 4 MaxEnt II (cont) HW5 out
[pdf, tex, slides]
Feb 6 MaxEnt III: training
MaxEnt IV: beam search
Klein and Manning 2003
Ratnaparkhi 1996
HW4 in
Feb 11 Conditional Random Fields (CRF) Sutton and McCallum 2006 HW6 out
[pdf, tex, slides]
Reading #2
Feb 13 Support Vector Machines (SVM) I: linear
Hyperplanes
IR, Ch 15 HW5 in
Feb 18 SVM II: non-linear
libSVM
IR Ch 15 Reading #3 Reading #2 in
Feb 20 SVM III: tree kernel
SVM IV: transductive SVM
Collins and Duffy 2001
Joachims 1999 (ICML)
HW6 in
Feb 25 Neural Networks: Introduction DL book ch 6.1-6.4
3blue1brown NN videos
HW7 out
[pdf, tex]
Reading #3 in
Feb 27 Neural Networks: Computation + Gradient Descent DL book ch 6.1-6.4
3blue1brown NN videos
Mar 3 Backpropagation DL book ch 6.5
CS 231n notes 1
CS 231n notes 2 (vector/tensor derivatives)
Yes, you should understand backprop
HW8 out
[pdf, tex]
Mar 5 Recurrent Neural Networks DL book ch 10
The Unreasonable Effectiveness of Recurrent Neural Networks
Understanding LSTMs
Sequence to Sequence Learning with Neural Networks (original seq2seq paper)
Neural Machine Translation by Jointly Learning to Align and Translate (original seq2seq + attention paper)
HW7 in
Mar 10 Transformers
Pre-training / transfer learning
Attention is All You Need (original Transformer paper)
The Annotated Transformer
The Illustrated Transformer
NLP's ImageNet Moment Has Arrived
The Illustrated BERT, ELMo, and co. (How NLP Cracked Transfer Learning)
HW9 out
[pdf, tex]
[DUE March 19]
Mar 12 Summary HW8 in