CS224n, 2019W - Assignment3 Solution
HW3: Dependency parsing and neural network foundations
you can find material: code handout
Table of contents
- 1. Machine Learning & Neural Networks (8 points)
- 2. Neural Transition-Based Dependency Parsing (42 points)
1. Machine Learning & Neural Networks (8 points)
(a) Adam Optimizer
i.
Momentum is working as acceleration reflecting the previous gradients tendency.
Hight variance is well known to raising overfitting problem.
ii.
Parameters that previous gradients are small and not volatile get larger updates.
This helps model to handle with sparse gradients(merits of AdaGrad) and also non-stationary objectives(merits of RMSProp)
(b) Dropout
i.
\[\gamma = \frac{1}{1-P_{drop}}\]
ii.
Drop out is one of the regularizations, which restrict an overfitting. Randomly setting units zero is for make units stronger on abrupt absence of other units.
And evaluation is made from train, to model being improved. But if we use drop out also at evaluation, it does re-trained with same circumstances, so that can’t do well in test.
2. Neural Transition-Based Dependency Parsing (42 points)
(a)
| Stack | Buffer | New dependency | Transition |
|---|---|---|---|
| [ROOT] | [I, parsed, this, sentence, correctly] | Initial |
|
| [ROOT] | [I, parsed, this, sentence, correctly] | SHIFT |
|
| [ROOT, I] | [parsed, this, sentence, correctly] | SHIFT |
|
| [ROOT, I, parsed] | [this, sentence, correctly] | parsed$\rightarrow$ | LEFT-ARC |
| [ROOT, parsed] | [this, sentence, correctly] | SHIFT |
|
| [ROOT, parsed, this] | [sentence, correctly] | SHIFT |
|
| [ROOT, parsed, this, sentence] | [correctly] | sentence$\rightarrow$this | LEFT-ARC |
| [ROOT, parsed, sentence] | [correctly] | parsed$\rightarrow$sentence | RIGHT-ARC |
| [ROOT, parsed] | [correctly] | SHIFT |
|
| [ROOT, parsed, correctly] | [] | parsed$\rightarrow$correctly | RIGHT-ARC |
| [ROOT, parsed] | [] | ROOT$\rightarrow$parsed | RIGHT-ARC |
| [ROOT] | [] |
(b)
- 2n
- n steps for each of buffer and stack
Whole scripts are here.
(c), (d), (e)
Tips: problem 2-d needs shallow copy, not deep copy and TA advised not to use del function, here is good explaination remove vs del
(f)
i.
- Error type: Verb Phrase Attachment Error
- Incorrect dependency: wedding → fearing
- Correct dependency: heading → fearing
ii.
- Error type: Coordination Attachment Error
- Incorrect dependency: rescue → and
- Correct dependency: rescue → rush
iii.
- Error type: Prepositional Phrase Attachment Error
- Incorrect dependency: named → Midland
- Correct dependency: guy → Midland
iv.
- Error type: Modifier Attachment Error
- Incorrect dependency: element → most
- Correct dependency: crucial → most
