COS4861 Assignment 2 (NLP) Due 2025

Okay, here's a polished and enhanced version of your document's description, suitable for an academic assignment: COS4861/0/2025 Assignment 2: Automata and NLP Preprocessing This document presents a comprehensive response to Assignment 2 for COS4861/0/2025, delving into both the theoretical underpinnings of automata theory and the practical application of Natural Language Processing (NLP) data preprocessing. Theoretical Foundations: Automata in NLP The theoretical section provides a rigorous exploration of Deterministic Finite State Automata (DFSA) and Non-Deterministic Finite State Automata (NDFSA). It meticulously defines their fundamental components (Q, Σ, δ, q 0 ​ , F), operations, and key distinctions using formal notation and illustrative examples. A detailed proof of the equivalence between NDFSA and DFSA is presented, employing the subset construction algorithm, complemented by a clear, step-by-step conversion example. Furthermore, the critical significance of these automata in core NLP tasks, such as tokenization and syntax parsing, is thoroughly analyzed with relevant practical applications. Practical Implementation: NLP Preprocessing Pipeline The practical component details the implementation of a robust NLP preprocessing pipeline. Using Python's NLTK library, the pipeline systematically applies essential preprocessing steps to a sample text dataset, including: Tokenization: Breaking down text into individual words or units. Stopwords Removal: Eliminating common, low-information words. Stemming: Reducing words to their root form heuristically. Lemmatization: Reducing words to their base dictionary form (lemma) using linguistic knowledge.