\documentclass[11pt]{article}
\usepackage{hyperref}
\usepackage[margin=0.75in]{geometry}
\usepackage{amsmath}
\usepackage{listings}

\lstdefinestyle{mystyle}{
    basicstyle=\ttfamily\footnotesize,
    breakatwhitespace=false,         
    breaklines=true,                 
    keepspaces=true,                 
    showspaces=false,                
    showstringspaces=false,
    showtabs=false,                  
    tabsize=2
}

\lstset{style=mystyle}

\begin{document}

\title{Ling 575j hw8}
\date{\vspace{-0.2in}Due 11pm on May 25, 2023}
\maketitle


\noindent In this assignment, you will 
\begin{itemize}
  \item Develop an intuition of the flow of recurrent Seq2Seq models
  \item Implement key pieces of the model architecture
  \item Better understand the formal process of attention
  \item Learn and practice writing useful Python docstrings
\end{itemize}
All files referenced herein may be found in \texttt{/dropbox/22-23/575j/hw8/} on patas.


\section{Understanding Attention [20 pts]}

\noindent {\bf Q1: Padding and Softmax} \hfill [3 pts]
\begin{itemize}
  \item What is $e^{-\infty}$? Or more accurately, what is $\lim_{x \to -\infty} e^x$?
  \item Give the equation for softmax over an input vector $x$ for the value $x_i$
  \item What would happen to the softmax output if every $x_i$ in $x$ were set to $-\infty$?
\end{itemize}


\vspace{2em}
\noindent {\bf Q2: Attention Calculation} Complete the computation of attention given the following values for Queries and Keys. Interpret the Queries and Keys as having a hidden dimension of 2, with the other dimension being sequence length. Following the case of RNN Seq2Seq, Keys and Values are the same. You can use a library like Numpy or Pytorch to do the computations, but make sure to do them step by step. When filling in the matrices, round to two decimal places, but do not round in your actual calculations. \hfill [10 pts]
\[
\begin{array}{c}
\begin{bmatrix}
0.2 & 0.4 \\
0.8 & 0.6 \\
1.0 & 1.2 \\
\end{bmatrix}\\
\text{Queries}
\end{array}
\times
\begin{array}{c}
\begin{bmatrix}
0.1 & 0.7 & 0.9 & 1.5\\
0.3 & 0.5 & 1.1 & 1.3\\
\end{bmatrix}\\
\text{Keys}^T
\end{array}
=
\begin{array}{c}
\begin{bmatrix}
- & - & - & - \\
- & - & - & - \\
- & - & - & - \\
\end{bmatrix}\\
\text{Compatibility}
\end{array}
\]

\[
+
\begin{array}{c}
\begin{bmatrix}
0 & 0 & 0 & 0 \\
0 & 0 & 0 & -\infty \\
0 & 0 & -\infty & -\infty \\
\end{bmatrix}\\
\text{Mask}
\end{array}
=
\begin{array}{c}
\begin{bmatrix}
- & - & - & - \\
- & - & - & - \\
- & - & - & - \\
\end{bmatrix}\\
\text{Masked Compatibility}
\end{array}
\xrightarrow{\text{softmax}}
\begin{array}{c}
\begin{bmatrix}
- & - & - & - \\
- & - & - & - \\
- & - & - & - \\
\end{bmatrix}\\
\text{Attention}
\end{array}
\]

\[
\times
\begin{array}{c}
\begin{bmatrix}
0.1 & 0.3 \\
0.7 & 0.5 \\
0.9 & 1.1 \\
1.5 & 1.3 \\
\end{bmatrix}\\
\text{Values}
\end{array}
=
\begin{array}{c}
\begin{bmatrix}
- & - \\
- & - \\
- & - \\
\end{bmatrix}\\
\text{Output}
\end{array}
\]

\newpage
\noindent {\bf Q3: Attention Questions} \hfill [7 pts]
\begin{itemize}
  \item Give the equation for attention (i.e., the Output of Q2) in matrix notation, using Q, K, and V to represent the Queries, Keys, and Values matrices (no need to show actual numbers).
  \item In the Compatibility matrix, what does each row represent?
  \item Why do Keys and Values have to be the same length?
  \item In a recurrent encoder-decoder model, where do the Queries, Keys, and Values come from respectively?
\end{itemize}

\section{Documenting Code [10 pts]}

Docstrings are an important part of Python development, and so are an important part of developing a Deep Learning system. Use these links as a reference for Google style docstrings, which we will use here \url{https://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_google.html}, \url{https://google.github.io/styleguide/pyguide.html#s3.8.1-comments-in-doc-strings}. You can also use the docstrings in \texttt{model.py} as examples. This format has the advantage of being able to be used to automatically generate html documentation from code for a project's website. Here, you will provide docstrings for the \texttt{Seq2SeqDataset} class in \texttt{data.py}, as well as its methods \texttt{batch\_as\_tensors} and \texttt{from\_files}. Please only include the main description, \texttt{Args}, and \texttt{Returns}, giving roughly the level of detail in \texttt{model.py} (as opposed to the link).

The following section of the document is an environment that allows you to type monospaced plain-text in Latex. If you are using Latex, please make sure to keep your docstrings in these environments. If you are submitting a .txt file, you can just use \texttt{"""} to demarcate the beginning and end of your docstrings. This environment will wrap your lines for you, but try practicing manually formatting your lines to be 80 characters. Most text editors will allow you to show a ruler line at a specific number of characters, so try drafting your docstrings there and pasting them here.

\begin{lstlisting}
class Seq2SeqDataset(Dataset):
"""
TODO: replace with your docstring here. Do NOT include the code.
"""
\end{lstlisting}

\begin{lstlisting}
def batch_as_tensors(self, start: int, end: int) -> dict[str, Any]:
"""
"""
\end{lstlisting}

\begin{lstlisting}
@classmethod
def from_files(
    cls, source_file: str, target_file: str, vocab: Vocabulary = None
):
"""
"""
\end{lstlisting}

\section{Implementing a Recurrent Seq2Seq Model [37 pts]}

In the remainder, you will implement key components of a character-level Seq2Seq translation model, based on LSTMs. The dataset we are using is called Europarl, and consists of proceedings of the EU parliament. The data files can be found at \texttt{/dropbox/22-23/575j/data/europarl-v7-en-es/}. We are translating English to Spanish. All of the methods that you will be implementing are in \texttt{model.py}. \textbf{Read all docstrings and comments closely for desired behavior}.

\vspace{2em}
\noindent {\bf Q1: Seq2SeqModel.forward} Implement the \texttt{forward} method for the \texttt{Seq2SeqModel} class. This method should take in batches of input and target sequences, embed them, encode the input sequence and then use its final states and attention to decode the target sequence. \textbf{This method will call Seq2SeqModel.encode and Seq2SeqModel.decode}. \hfill [9 pts]

\vspace{2em}
\noindent {\bf Q2: Seq2SeqModel.encode} Implement the \texttt{encode} method for the module. This method should take in the batch of input sequences and their lengths, and return the position-wise hidden states as well as the final hidden and cell states. \hfill [9 pts]

\vspace{2em}
\noindent {\bf Q3: Seq2SeqModel.decode} Implement the \texttt{decode} method for the module. This method should take in a batch of target sequences, plus the hidden and final states from the encoder, and use the decoder LSTM + attention to convert to logits over the vocabulary for the target sequence. \textbf{This method will call Seq2SeqModel.attention} \hfill [9 pts]

\vspace{2em}
\noindent {\bf Q4: Seq2SeqModel.attention} Implement the \texttt{attention} module. This method should take series of vectors acting as Queries, as well as those acting as Keys/Values, plus a padding mask that we will provide for you. \textbf{The padding mask is created for you in forward, and must be passed to this method}. \hfill [10 pts]

\section{Running the Translation Model [8 pts]}

\texttt{run.py} contains a training loop to use the Seq2Seq translation model, which will record the training loss and generate text every $N$ epochs (controlled by \texttt{--generate\_every}, set to 1 by default). \textbf{Warning: the run of this model will take several hours or more. It is important that you plan ahead, and only use the default parameters of the script (which are small), until you are sure your code runs end to end without errors and it is time to run your final script.}

\vspace{2em}
\noindent {\bf Q1: Run with Full Parameters}  Execute \texttt{run.py} with the following (full) arguments. Paste below the texts that are generated every epoch, as well as the training and validation loss.  In 2-3 sentences, describe any trends that you see. \hfill [4 pts]
\begin{itemize}
    \item \texttt{--train\_source /dropbox/22-23/575j/data/europarl-v7-es-en/train.en.txt}
    \item \texttt{--train\_target /dropbox/22-23/575j/data/europarl-v7-es-en/train.es.txt}
    \item \texttt{--output\_file test.en.txt.es}
    \item \texttt{--num\_epochs 8}
    \item \texttt{--embedding\_dim 16}
    \item \texttt{--hidden\_dim 64}
    \item \texttt{--num\_layers 2}
\end{itemize}

\vspace{2em}
\noindent {\bf Q2: Evaluate Translations} Finally, we will evaluate the translations using Character F-Score (\url{https://www.aclweb.org/anthology/W15-3049/}). Using the provided script \texttt{chrF++.py}, calculate the score of the output Spanish translations using the following command, and report the score for the line that says \texttt{c6+w0-F2}: \hfill [4 pts]
\begin{lstlisting}
    python chrF++.py -nw 0 \ 
        -R /dropbox/22-23/575j/data/europarl-v7-es-en/test.es.txt \
        -H test.en.txt.es > test.en.txt.es.score
\end{lstlisting}

\section{Testing your code}

In the dropbox folder for this assignment, you will find a file \texttt{test\_all.py} with a few very simple unit tests for the methods that you need to implement.  You can verify that your code passes the tests by running \texttt{pytest} from your code's directory, with the course's conda environment activated.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section*{Submission Instructions}

In your submission, include the following:
\begin{itemize}
  \item readme.(txt$\mid$pdf) that includes your answers to \S1, \S2, and \S4. 
  \item \texttt{hw8.tar.gz} containing:
  \begin{itemize}
    \item run\_hw8.sh.  This should contain the code for activating the conda environment and your run commands for \S4 above.  You can use run\_hwX.sh from any previous assignment as a template.
    \item model.py
    \item test.en.txt.es
    \end{itemize}
\end{itemize}





\end{document}



