CS 7643 Quiz 4 Exam Questions And Answers Verified 100% Correct

CS 7643 Quiz 4 Exam Questions And Answers Verified 100% Correct Word2vec Overview - ANSWER -Word2vec - a framework for learning word vector Idea: - we have a large corpus of text - every word in. fixed vocabulary represented by a vector - Go through each position t in the text, which has a center word c and context words o - Use the similarity of the word vectors for c and o to calculate the probability of o given c (or vice versa) - Keep adjusting the word vectors to maximize this probability Word2vec Variants - ANSWER -Skip-Gram: Predict context words given center word Continuous Bag of Words: Predict center word from (bag of) context words Word2vec Objective Function - ANSWER -- product over all possible center words - product over all words in the context window - P( w(t+j) | w(t); theta ) - J(theta) = - 1 / T * log (L) Word2vec P( w(t+j) | w(t) ) - ANSWER -- Two sets of vectors for each word in vocabulary 1. u(w) for when w is the center word 2. v(o) for when w is a context word P( w(t+j) | w(t) ) = softmax( u(wt) * v(wt+j) ) Word2vec Expensive to Compute Solutions - ANSWER -1. Hierarchical Softmax 2. Negative Sampling Negative Sampling Intuition - ANSWER -- For each (w, c) pair, sample k negative pairs (w, c') - maximize probability real word appears and minimize the probability random word appears Evaluating Word Embeddings Intrinsic - ANSWER -- Evaluation on a specific/intermediate subtask - Fast to compute - Helps to understand the system - Not clear if really helpful unless correlation to real task is established Example: Evaluate word vectors by how well their cosine distance after addition captures intuitive semantic and syntactic analogy questions Evaluating Word Embeddings Extrinsic - ANSWER -- Evaluation on real task - Can take a long time to compute - Unclear if the subsystem is the problem or its interaction - if replacing exactly one subsystem with another improves accuracy - winning Why Graph Embeddings - ANSWER -- They are a form of unsupervised learning on graphs - Results in task-agnostic entity representations - Features are useful on downstream tasks without much data - Nearest Neighbors are semantically meaningful Graph Embeddings Loss Function - ANSWER -- Margin loss between the score of an edge f(e) and a negative sampled edge f(e') - Negative sampled edges are constructed by taking real edge and replacing either the source or destination vertex with a random node - the score of an edge f(e) is a similarity (dot product) between the source embedding and a transformed version of the destination embedding - f(e) = cos( theta(s) , theta(d) + theta(r) ) Graph Embedding is Slow: Reason and Solution - ANSWER -- Training time dominated by computing scores for "fake edges" - Corrupt a sub-batch of edges with the same set of random nodes Debiasing word2vec - ANSWER -- identify gender subspace with gendered words - project all words onto this subspace - subtract those projections from the original word Problem: Not that effective and bias pervades the word embedding space t-SNE things to remember - ANSWER -1. Run until it stabilizes 2. Set perplexity b/w 2 and N - perplexity loosely measures # neighbors - balances b/w local and global aspects of nodes 3. Re-run t-SNE multiple times to ensure we get the same shape t-SNE general concept - ANSWER -- Maps inputs from high dimensional space to lower dimensions for visualization - recursively moves similar points closer and distance points further - expands dense clusters and contracts sparse cluster Teacher Forcing - ANSWER -- next input to model is not predicted value, but the actual value from the training data - allows model to train effectively even if a mistake was made - if used instead of hidden-to-hidden recurrence nodes, can allow for parallelization, but model becomes less powerful - emerges from MLE - issues may arise if network is later going to be used in "closed-loop" mode where output is fed back as input Skip-Gram Model: Loss/Objective Function - ANSWER -Loss - for each position t, we try to predict the context words within a fixed window size given some context word - multiple these probabilities to get a likelihood - L(theta) = product(product(P(w_(t+j) | w_(t) ; theta)) - Objective function: J(theta) = - 1/T log(L(theta)) Skip-Gram Model: Calculate P(w_(t+j) | w_(t) ; theta) - ANSWER -- Two vectors for each word: 1. u_w when w is center word 2. v_o when o is a context word - uses inner product (u_w, v_o) to measure how likely it is that u_w appears with context word v_o