PRINCIPAL COMPONENT ANALYSIS (PCA) ACTUAL EXAM QUESTIONS AND ANSWERS

What is PCA? (5 key points) Principal Component Analysis is a statistical technique used for dimensionality reduction, crucial when dealing with high-dimensional data in machine learning. It works by transforming original variables into new ones, called principal components, which are linear combinations of the original variables. Key Points: 1. Principal Components: Principal components are the directions in the data that maximize variance. The first principal component captures the most variance, and each subsequent component (orthogonal to the previous ones) captures progressively less variance. 2. Reduces Dimensions: By selecting the top principal components, PCA reduces the number of variables, minimizing information loss and simplifying the dataset. 3. Visualization and Analysis: PCA aids in visualizing and understanding complex data by reducing it to two or three principal components. 4. Preprocessing Step: Often used before applying machine learning algorithms, PCA can enhance performance and efficiency, and reduce overfitting risks. 5. Eigenvalues and Eigenvectors: PCA involves computing the eigenvalues and eigenvectors of the data's covariance matrix. Eigenvectors define the direction of the new feature space, while eigenvalues define their magnitude. In essence, PCA is a vital tool for data simplification, pattern recognition, and improving machine learning algorithm efficiency. WHat are the 7 mathematical steps to calculating PCA? Here's a condensed explanation of the PCA process: 1. Standardize the Data: Adjust data to have a mean of zero and a standard deviation of one. This is crucial as PCA is sensitive to variances of the initial variables. 2. Compute the Covariance Matrix: Calculate the covariance matrix to understand the relationships between variables. The covariance matrix reflects how much variables change together. 3. Calculate Eigenvalues and Eigenvectors: From the covariance matrix, derive eigenvalues and eigenvectors. Eigenvectors represent the directions of the new feature space, while eigenvalues indicate the magnitude of these directions. 4. Sort Eigenvalues and Eigenvectors: Organize the eigenvalues and their corresponding eigenvectors in descending order of the eigenvalues. This order signifies the importance of each eigenvector in explaining the variance in the data. 5. Choose Principal Components: Select the top eigenvectors (now principal components) based on the largest eigenvalues. The number chosen depends on the desired amount of variance to capture from the original data. 6. Transform the Original Dataset: Multiply the original data matrix by the matrix of chosen eigenvectors to transform the data into a new dataset with reduced dimensions. 7. Interpretation: The transformed dataset, in terms of principal components, can now be analyzed. Each principal component is a combination of original variables, offering a simplified but informative representation of the original data. This process effectively reduces the dimensionality of data, retaining significant information while minimizing complexity. Describe the 6 steps of PCA as if you were looking at a 3D graph of data? Here's a distilled explanation of PCA visualized on a graph: 1. Original Data Scatter: Imagine a scatter plot with data points spread across X and Y axes. Each point represents a data observation with its respective values. 2. Standardization (If Applied): If the scales of X and Y are different, standardization adjusts them to be uniform. This prevents one feature from dominating the analysis due to scale differences. 3. First Principal Component (PC1): A line is drawn through the data, aligning with the direction of maximum variance. This line, PC1, captures the most variance in the data. 4. Second Principal Component (PC2): Another line, perpendicular to PC1, is drawn. This is PC2, capturing the most variance not already accounted for by PC1. 5. Projection of Data Points: Data points are projected onto these lines, forming a new scatter plot in the principal component space. 6. Reduced Dimensionality Representation: By keeping only PC1, the data points align along this line, reducing the dimensionality from two to one. In essence, PCA transforms the data to align with new axes (principal components) that better represent the variance. The original scatter plot changes, aligning the data along these new axes, simplifying the data and revealing underlying patterns. What are the 5 pros of PCA? Pros of PCA: 1. Reduces Dimensionality: PCA is excellent for reducing the number of features in a dataset, especially when many variables are correlated. This simplification can make subsequent analyses more efficient and easier to interpret. 2. Minimizes Information Loss: It focuses on preserving the maximum variance in the data, which often means retaining the most significant features while minimizing information loss. 3. Visualization: PCA can help visualize complex data (especially when reduced to two or three dimensions), making it easier to spot patterns, trends, and outliers. 4. Improves Algorithm Performance: Reducing the number of features can lead to faster processing times and, in some cases, better performance of machine learning algorithms, especially when dealing with the curse of dimensionality. 5. Data De-noising: PCA can help in filtering noise from the data by capturing the principal components and ignoring components with low variance, which often represent noise. What are the 6 cons of PCA? Cons: 1. Data Interpretation Challenges: The principal components are linear combinations of the original features and may not have a direct, interpretable meaning. This can make it difficult to interpret the results in the context of the original data. 2. Sensitivity to Scaling: PCA is sensitive to the relative scaling of the original features. Variables on larger scales can dominate the principal components unless the data is properly standardized. 3. Loss of Some Information: While PCA seeks to minimize information loss, some loss is inevitable when reducing dimensions. Important variables with lower variance may be discarded. 4. Assumption of Linearity: PCA assumes that the principal components are a linear combination of the original features. It may not work well with data structures that have complex, nonlinear relationships. 5. Outlier Sensitivity: PCA can be significantly affected by outliers in the data, which may skew the direction of the principal components. 6. Not Suitable for All Data Types: PCA is best suited for continuous data and may not perform well with categorical data without proper preprocessing. What are the 8 alternatives to PCA for dimensionality reduction? Here's a condensed overview of alternatives to PCA for dimensionality reduction and data analysis: 1. Factor Analysis: Focuses on uncovering latent factors from observed variables, commonly used in social sciences. 2. Independent Component Analysis (ICA): Separates multivariate signals into additive components, useful in signal processing and medical imaging. 3. t-Distributed Stochastic Neighbor Embedding (t-SNE): A non-linear technique for data visualization, especially effective for reducing high-dimensional data to two or three dimensions. 4. Uniform Manifold Approximation and Projection (UMAP): A recent method that excels in preserving both local and global data structures during dimensionality reduction. 5. Linear Discriminant Analysis (LDA): Combines dimensionality reduction with classification, focusing on maximizing class separability. 6. Non-negative Matrix Factorization (NMF): Useful for decomposing multivariate data, particularly in image and text analysis. 7. Autoencoders (Deep Learning): Neural network-based approaches for learning compressed data representations, used in denoising, dimensionality reduction, and feature learning. 8. Multidimensional Scaling (MDS): Analyzes similarity or dissimilarity data, placing objects in an N-dimensional space to preserve their distances. Each method has unique strengths and is suitable for specific data types and analysis goals. The choice depends on the data nature, the analysis objectives, and computational resources. What are the 8 situations where you should avoid using PCA? 1. Non-Linear Relationships: Avoid PCA for data with complex, non-linear relationships. It's better suited for linear data structures. 2. Categorical Data: PCA is ideal for continuous variables. For datasets rich in categorical features, other methods are more appropriate. 3. Small Datasets: With too few data points, PCA may lead to overfitting, especially in machine learning applications. 4. Need for Interpretability: If understanding the original variables in their original form is crucial, PCA’s transformed components, which are less interpretable, can be a drawback. 5. Extensive Missing Values: PCA requires a complete dataset. High levels of missing data can lead to unreliable results. 6. Inconsistent Feature Scales: If standardizing different scales of features isn't desirable, PCA's sensitivity to feature variances might skew the results. 7. Outlier Sensitivity: PCA can be heavily influenced by outliers. If your data has many outliers and they aren't managed, PCA might yield misleading insights. 8. Preserving Data Structure: If maintaining the original structure of data is important, PCA may not be the best choice as it focuses on maximizing variance, sometimes at the cost of losing important structures. In short, while PCA is useful for many scenarios, its limitations in handling non-linearities, categorical data, small datasets, interpretability issues, missing values, scaling discrepancies, sensitivity to outliers, and the need to preserve original data structures make it less suitable for certain types of analysis. What are 6 options you should use to look for non-linearity in your data when preprocessing for PCA? Which can be automated? Here's a condensed guide to quickly assessing non-linear relationships in your data before applying PCA: 1. Visualization: - Scatter Plots: Use automated tools to generate scatter plots for different feature combinations. Look for patterns that aren't linear. - Pairwise Plot: Tools like Seaborn in Python can create comprehensive visual overviews of all feature combinations. 2. Correlation Coefficients: Generate an automated correlation matrix. Linear correlation coefficients reveal linear relationships, but poor model performance despite high correlation may suggest non-linearities. 3. Statistical Tests: Apply tests like Spearman Rank Correlation to check for non-linear relationships. This requires some statistical interpretation. 4. Dimensionality Reduction Techniques: Use non-linear techniques like t-SNE or UMAP and compare with PCA results. This approach is more automated but needs computational resources. 5. Residual Analysis of Linear Models: Fit a linear model and analyze residuals for patterns. Semi-automated, but requires knowledge of regression analysis. 6. Machine Learning Model Comparison: Compare performances of linear vs. non-linear machine learning models. Superior performance of non-linear models suggests non-linear relationships. While tools and methods can help identify non-linear relationships, a completely effort-free automation is challenging. Typically, a combination of these methods, leveraging automation for initial analysis and expert judgment for interpretation, is the most effective approach. What are 6 options you should use to look for outliers in your data when preprocessing for PCA? Which can be automated? Here's a concise overview of six methods for detecting outliers when preprocessing for PCA, along with their automation potential: 1. Statistical Tests: - Z-Score: Fully automatable. Identifies outliers as points significantly far from the mean in terms of standard deviations. - IQR (Interquartile Range) Score: Also fully automatable. Flags data points lying beyond 1.5 times the IQR from the quartiles as outliers. 2. Visualization Tools: - Box Plots: Can be generated automatically; however, interpreting these plots to identify outliers requires manual effort. - Scatter Plots: Useful for spotting outliers in multidimensional data. Automation can produce these plots, but visual inspection is manual. 3. Dimensionality Reduction: - PCA Itself: Applying PCA and observing data point distribution can help highlight outliers. The process is somewhat automatable, but interpreting the results requires expertise. 4. Proximity-Based Methods: - DBSCAN or k-Means Clustering: Fully automatable. These algorithms can isolate outliers as points not belonging to any main cluster. 5. Machine Learning Models: - Isolation Forest or One-Class SVM: Advanced, automatable models d