\documentclass[11pt]{article}
\usepackage{hyperref}
\usepackage[margin=0.75in]{geometry}
\usepackage{amsmath}

\newcommand{\bos}{\textless s\textgreater\:}
\newcommand{\eos}{\textless /s\textgreater\:}
\newcommand{\pad}{PAD\:}


\begin{document}

\title{Ling 575j hw7}
\date{\vspace{-0.2in}Due 11PM on May 18, 2023}
\maketitle


\noindent In this assignment, you will 
\begin{itemize}
  \item Develop understanding of recurrent neural networks, especially as used for language modeling
  \item Implement components of data processing 
  \item Implement masking of losses for an RNN language model
\end{itemize}
All files referenced herein may be found in \texttt{/dropbox/22-23/575j/hw7/} on patas.


\section{Recurrent Neural Network Decoders/Taggers [35 pts]}

\noindent {\bf Q1: Understanding Masking [15 pts]}  Suppose that we want to train a (word-level) language model on the following two sentences:
\begin{center}
  \bos the cat sits \eos

  \bos the model reads the sentence \eos
\end{center}
We saw in HW6 that padding is necessary to make these sentences have the same length so that they can be batched together, as:
\begin{center}
  \bos the cat sits \eos \pad \pad

  \bos the model reads the sentence \eos
\end{center}
Please answer the following questions about these sequences:
\begin{itemize}
  \item In a recurrent language model, what would the input batch be?  What would the target labels be? \hfill [4 pts]
  \item Recurrent language models use a \emph{mask} of ones and zeros to `eliminate' the loss for \pad tokens.  What would the mask be for this batch? \hfill [3 pts]
  \item Suppose that we have the following per-token losses:
    \[ \begin{bmatrix} 0.1 & 0.3 & 0.2 & 0.4 & 0.7 & 0.5 \\ 0.2 & 0.6 & 0.1 & 0.8 & 0.9 & 0.4 \end{bmatrix} \]
    What is the \emph{masked} loss matrix? \hfill [3 pts]
  \item Why is it important to mask losses in this way?  What might a model learn to do if the loss is not masked? \hfill [5 pts]
\end{itemize}

\vspace{2em}
\noindent {\bf Q2: Evaluating Language Models [20 pts]} Given a corpus $W = w_1 w_2 \dots w_N$ (so $N$ is the number of tokens in the corpus), a common (intrinsic) evaluation metric for language models is \emph{perplexity}, defined as
\[ PP(W) = P(w_1 \dots w_N)^{-\frac{1}{N}} \]
This can be thought of as the inverse probability that the model assigns to the corpus, normalized by the size of the corpus.
\begin{itemize}
  \item Is a lower or higher perplexity better? \hfill [2 pts]
  \item For a recurrent language model, write an expression for $P(w_1 \dots w_N)$ using the chain rule of probability.  How is this different from the expression for a feed-forward language model? \hfill [5 pts]
  \item Show that
  \[ PP(W) = e^{-\frac{1}{N} \sum_{i=1}^N \log P(w_i \mid w_{<i})} \]
  where $w_{<i} = w_1 w_2 \dots w_{i-1}$ and $\log$ is the natural (base $e$) logarithm. \hfill [5 pts]
  
  [Note: using base $e$ measures perplexity in a unit known as \emph{nats}.  Using base $2$ would measure it in bits.]

  \item What is another name for the exponent $-\frac{1}{N} \sum_{i=1}^N \log P(w_i \mid w_{<i})$ in the above expression? [Hint: it appears in training as well.] \hfill [3 pts]
  \item Suppose that the same text corpus were tokenized with two different vocabularies of different sizes (perhaps, e.g., one replaces infrequent tokens with an UNK token) and two language models were trained on the resulting tokenized text.  All else being equal, would you expect perplexity to be lower or higher for the model with a smaller vocabulary?  What consequences does this have for comparing different language models? [5 pts]
\end{itemize}


\section{Implementing an RNN Character Language Model [25 pts]}

In the coding portion of this assignment, you will implement (components of) an LSTM character-level language model for the Stanford Sentiment Treebank, using an RNN as tagger.

\vspace{2em}
\noindent {\bf Q1: Data processing} Recall from the lectures that we can view language modeling as a sequence tagging task.  That is, each element of an input sequence is tagged with a certain target.  This gets operationalized in the data processing pipeline; you will generate the inputs/targets for one line of text.

\noindent In \texttt{data.py}, please implement the \texttt{example\_from\_characters} method.  Please read the method signature and docstring carefully for details on the input and output. \hfill [10 pts]


\vspace{2em}
\noindent {\bf Q2: Masking the Loss} In the written portion above, you explained why masking the loss is important for a recurrent language model.  Now, you will implement said masking.  In \texttt{run.py}:
\begin{itemize}
  \item Implement \texttt{get\_mask}, which generates the mask to apply to the losses. \hfill [6 pts]
  \item Implement \texttt{mask\_loss}, which takes per-token losses, masks out the `bad' ones, and then returns a mean.  See the doc-string for details. \hfill [9 pts]
\end{itemize}


\section{Running the Language Model [15 pts]}

\texttt{run.py} contains a basic training loop for SST language modeling. It will record the training and dev loss (and perplexity) at each epoch, and save the best model according to dev loss.  Periodically (as specified by a command-line flag), it also outputs generated text from the best model.

\vspace{2em}
\noindent {\bf Q1: Default parameters} Execute \texttt{run.py} with its default arguments.  Paste below the texts that are generated every 4 epochs, as well as the epoch with the best dev loss and the dev perplexity from that epoch.  In 2-3 sentences, describe any trends that you see.  [Note that generated text will not necessarily be completely coherent: recall that this is a \emph{character-level} language model.] \hfill [5 pts]

\vspace{2em}
\noindent {\bf Q2: Modify hyper-parameter(s)} Re-run the training loop, modifying some combination of the following hyper-parameters, which are specified by command-line flags:
\begin{itemize}
  \item Hidden layer size
  \item Embedding size
  \item Learning rate
  \item Number of epochs [in particular: making it larger]
  \item Softmax temperature. 
  \item $L_2$ regularization coefficient.
  \item Dropout (probability with which neurons are dropped from the input and to the output during training)
\end{itemize}
Include your model's generated texts here.  In 2-3 sentences, state exactly what hyper-parameter change(s) you made, and what effects (if any) you see in terms of the dev set perplexity and text that the model generated. \hfill [5 pts]

\vspace{2em}
\noindent {\bf Q3: Comparison to feed-forward language model} In 2-3 sentences, please explain what differences you see in the text generated by this LSTM language model and the feed-forward language model that you trained in HW5. What do you think may be causing these effects (or lack thereof)? \hfill [5 pts]


\section{Testing your code}

In the dropbox folder for this assignment, you will find a file \texttt{test\_all.py} with a few very simple unit tests for the methods that you need to implement.  You can verify that your code passes the tests by running \texttt{pytest} from your code's directory, with the course's conda environment activated.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section*{Submission Instructions}

In your submission, include the following:
\begin{itemize}
  \item readme.(txt$\mid$pdf) that includes your answers to \S1 and \S3. 
  \item \texttt{hw7.tar.gz} containing:
  \begin{itemize}
    \item run\_hw7.sh.  This should contain the code for activating the conda environment and your run commands for \S3 above.  You can use run\_hw2.sh from the previous assignment as a template.
    \item data.py
    \item run.py
  \end{itemize}
\end{itemize}





\end{document}



