CPCE 2022 - Appraisal & Assessment, questions and answers, rated A+

CPCE 2022 - Appraisal & Assessment, questions and answers, rated A+ Z score - -Most common standard score Mean=0 ; s.d.=1 A z-score allows one to compare people's scores on the basis of standard deviation units. Each z "point" is one standard deviation. T score - -Most common standard score Mean=50 ; s.d=10 Standard Score - -Descriptive statistics allows for the development of standard scores which are described and interpreted by using the appropriate mean and s.d. in relation to the normal curve. Any set of score in which a mean and standard deviation is known. Types: T-score, Z-score, Stanine, Sten Stanine - -9 divisions of the normal curve. Range=1-9; Mean= 5 ; s.d.= 2 The term stanine is a contraction of "standard nines." Stanines provide a single-digit scoring metric with a range from 1 to 9, a mean of 5, and a standard deviation of 2. Each stanine score represents a specific range of percentile scores in the normal curve. Stanines are useful when a researcher is interested in providing a "band" interpretation rather than a single score cutoff. Sten - -a 10 division of the normal curve Mean = 5.5 ; s.d = 2) Percentage score - -is the number of correct items divided by total number of items. (6/10 = 60%) Normal curve - -Linear line is the base line of curve. Line referred to as z-line. A z-line has mean of 0 and SD of 1. Note +-1 sd = 68% of curve (34+34), +-2 sd = 95% of curve (34+34+13.5+13.5) and +-3 sd = 99% of curve (95+2+2). The normal curve can be established for any set of data when the mean and sd are known. Variability - -How the scores spread away or disperse from the central tendency. Some ways to examine variability are range and standard deviation. Range - -done by subtracting the lowest number (score) from the highest number (score) and add one. Standard deviation - -s.d is square root of variance. A calculation in which each individual score is determined to be how many units away from arithmetic mean. Used when the mean is reported as the best indicator of the average. This is the most reliable measure of variability for a sample. Construct validity - -Correlation r. Is extent to which a test measures an abstract trait or psychological notion theoretical construct like intelligence, self-esteem, feelings, artistic talent. Used for personality theory & non-cognitive tests. A construct is any trait you cannot "directly" measure or observe. Criterion validity - -Evidence based on relations to other variables. Relating test scores with relevant criteria to which scores from a test can be used to predict performance on some criterion such as a test or future performance. 2 types: Predictive or Concurrent ***Item discrimination - -The degree to which an item differentiates correctly among test takes in the behavior that the test is designed to measure. Is to determine if the item can discriminate the learner from the non-learner. The item analysis results can be utilized to arrange the test in a spiral form. Evaluation - -Results in answers to qualitative questions. Formative Evaluation Summative Evaluation Assessment - -A FORMATIVE EVALUATION: It is conducted while a program is being formed. The purpose of a formative evaluation is to obtain the information needed to revise a program as it is being developed in order to ensure that the program's goals are met. A SUMMATIVE EVALUATION, which is the systematic evaluation of a program after it has been administered to determine if the program's objectives have been met. An ASSESSMENT is conducted prior to the implementation of a program or an intervention to obtain the information needed to formulate the goals and objectives of the proposed program. Formative & Summartive Evaluation - -Formative- Continuous information used to modify a program to improve effectiveness (4 exams during semester). Summative- Information gathered at the end of term. One time. Measurement - -Results in answers to quantitative questions. The test score (quantity) is used to identify a number answer (usually expressed as raw score or percentage correct). Quality is to infer how well they did in comparison to a group. Non-cognitive tests - -There are no right/wrong answers. Faking may be a concern. Error types: halo, criterion, leniency, acquiescence, social desirability, ect Acquiescence- when a client always agrees with something Deviation (in reference to testing)- when an individual purposely, or when in doubt, gives unusual responses. Social desirability- when someone tries to answer the questions in a socially acceptable manner. True Score - -is the average of all the scores for a person upon retesting a large number of times, hypothetical score that could be attained if the assessment were perfectly reliable. Chi-square nonparametric test - -What examines whether obtained frequencies different significantly from expected frequencies. Goodness-Of-Fit Test: A nonparametric inferential hypothesis test that examines how well an observed frequency distribution of a nominal (categorical) variable fits some expected pattern of frequencies. Observed frequencies: the frequencies observed in the sample Expected frequencies: frequencies expected in the sample based on some pattern of frequencies such as those in the population. Probability of throwing heads on a coin is 50% and is referred to as an expected frequency.The observed frequency would be throwing heads 71%. Percentile or Percentile Rank - -A percentile indicates the percentage of people in the reference group who performed at or below the examinee's score. This score type is easily confused and unfortunately is widely misused, despite its popularity. Percentiles are an ordinal or rank-order scale of measurement, rather than an equal-interval scale. That means one cannot subtract or average percentile scores in order to represent growth or change. Quasi Experimental - -Pre-established groups Any research that fails to use random assignment or lacks a control group. Useful in studying treatment effects on a single subject and behavioral research. In the quasi-experiment the researcher uses preexisting groups, and hence the IV (independent variable) cannot be altered. In a quasi-experiment you cannot state with any degree of statistical confidence that the independent variable (IV) caused the dependent variable (DV). Ex post facto or "after the fact" quasi-experiments connot a correlational study or research in which intact, preexisting groups are utilized. The independent variable (IV) was administered before the research began. Correlation - -An expression of a relationship between 2 independent variables. Viewed on a scattergram. Expressed as number between -1.00 and +1.00 Correlations are BIVARIATE (there are 2 variables being compared). Deductive Logic Inductive Logic - -Deductive- general to specific Inductive- specific to general (study of treating client's phobia using a paradoxical strategy, then writes research report that paradox is treatment of choice for phobics) Performance Test - -measure interests, attitudes, and other noncognitive attributes of personality. Examples are projective tests and personality inventories. Variance - -Variance- how far on average the scores deviated from the mean. The variance is the standard deviation squared. The standard deviation is the square root of variance. Within-subject design Between-subject design - -WITHIN-SUBJECTS DESIGN (pre-post design)- The same group of subjects participate in all of the treatments. Sometimes called a repeated-measures design. It looks for differences between treatment conditions within the same group of participants. Primary advantages are that it eliminates the confounding problems based on Individual Differences between groups; since each participant appears in every treatment condition, each individual serves as his own control or baseline which makes it possible to measure and remove the variance caused by Individual difference. It also requires fewer participants than Between-Subjects designs which reduces variance (more likely to detect treatment effect). Primary disadvantage are the Order Effects: The relationship between scores across treatments creates the potential for the scores in one treatment to be influenced by previous treatments, previous measurements, or previous experience. Other disadvantage is threats to internal validity- main two are environmental variables (different locations of the test) and time-related factors (history, maturation, instrumentation, testing effects, regression). Within-subject design Between-subject design - -BETWEEN SUBJECTS DESIGN- comparing scores from separate groups. Also known as an independent-measures experimental design. Requires a separate, independent group of individuals for each treatment condition. The goal is to determine whether differences exist between two or more treatment conditions. Requires a larger # of participants. Primary advantage is that each individual score is independent of the other scores because each participant is measured only once. There are no time-related factors or order effects. Primary disadvantage is Individual Differences because they can become confounding variables and produce high variability (making it difficult to determine whether the treatment has any effect). If you anticipate a treatment having a large or longstanding effect that may influence participants in future conditions, it's better to use Between-Subjects design. Within subjects designs - -TESTING EFFECTS (threat to internal validity) are directly related to experience obtained by participating in previous treatment conditions. For example, participants may learn new skills in one treatment that can influence future behavior. Or participant can become fatigued from one treatment and that fatigue affects their scores in later treatment. The testing effects are often called ORDER EFFECTS to emphasize that participants go through a series of treatments in order, and that performance in any treatment may be influenced by treatments that occurred earlier in the order. When an order effect is caused by a specific treatment, it is called a CARRYOVER EFFECT. Internal threats to Validity (design type error) - -HISTORY- events which occur during experiment MATURATION- change in subject (fatigue/hunger) TESTING- test content familiarity; taken test before REGRESSION- regression to mean for purpose of extremes MORTALITY- attrition External threats to Validity (design type error) - -Pretest treatment Multiple treatment- simultaneous application of multiple interventions Selection- not random (susceptible to change) Reactive- differences in experimental and real life situations (pretest sensitization) Experimenter bias Population Sample - -Population- this is the group to which the researcher wants the results generalized Sample- smaller group of the population selected for the research Experimental Design - -Manipulate the independent variable Strengthen study by making IV stronger Minimize error of variance by randomization and experimental control. There is often a control group. Random group assignment requirement and allows for control of internal validity Examples: posttest-only control group, pretest-posttest control, Solomon Four group You completed instruction and administers a test for a career learning unit. You derived a group mean of 32 and s.d of 4 from the individual test score. The range of the scores for +-3 standard deviations is? - -20-44 3 s.d equals 4x3 or +- 12 of 32 32-12 = 20 and 32+12 = 44 Descriptive Statistics Categories - -CENTRAL TENDENCY- Mean (interval) Median (ordinal, used for extreme scores) Mode (nominal, least stable) VARIABILITY- Range (interval, least stable) Standard Deviation (most stable) Quartile (interval, not affected by extreme scores) Inferential Statistics Categories - -PARAMETRIC TESTS (Normal) (Ratio or Interval): t-test z-test ANOVA 2-way ANOVA ANCOVA Factorial -Covariance NON-PARAMETRIC TESTS (Non-normal) (Ordinal or ratio): Chi-Square Median Test Sign Test t-test example 2 groups or 2 tests and Random - -Students (n=30) took the NCE six weeks ago with mean of 111. For six weeks the group studied NCE material once a week in a lecture format. The group retook the test six weeks later and derived mean of 119. The null hypothesis stated there would be no significant difference in the mean score. 2 scores / 1 Independent Variable Shortcut to determine whether or not something is positively or negatively skewed... - -Determine if the mean is to the left or right of the median. The median would be in the center. Positive = Mode > Mean > Median Negative = Mode < Mean < Median positively skewed - majority of scores are at the lower end of the range of scores negatively skewed - majority of score are at the higher end of the distribution scale hard test > positively skewed Coefficient of Determination (Variance accounted for) - -To find percentage of shared variance between 2 variables, simply square the correlation coefficient. Example: If the correlation coefficient is .90, then the common variance would equal 81%. Coefficient of Determination is 81% (variance accounted for ~ shared variance); 19% (variance unaccounted for). The Coefficient of Determination tells us how much of the variance in the scores of one variable can be understood or explained by the scores on a second variable. When 2 variables are related or correlated with each other, there is a certain amount of shared variance between them. The stronger the correlation, the greater the amount of shared variance, and the higher the coefficient of determination. Cognitive Tests - -APTITUDE TEST: a test used to predict future performance, "intended to predict success in some occupation or training course"; GRE ACHIEVEMENT TEST: measure what you know (NCE) INTELLIGENCE TEST: WAIS, Stanford-Binet, Kurtosid Mesokurtic Leptokurtic Platykurtic - -KURTOSIS- refers to the peakedness or flatness of a distribution MESOKURTIC- normal distribution/ normal curve LEPTOKURTIC- more scores in center and higher peaks PLATYKURTIC- more scores on outer slopes and fewer in the center Types of Validity - -1. Construct Validity a. Convergent b. Discriminant 2. Criterion Validity a. Predictive b. Concurrent 3. Content Validity 4. Face Validity ***Item Analysis AKA item easiness AKA Alpha coefficient Item Discrimination - -ITEM ANALYSIS - take the percentage correct score of the upper group and adding it to the percentage correct score of the lower group and dividing by 2. i.e. 43% + 15% = 58% divided by 2 would equal an item easiness index of 29% for this item. ITEM DISCRIMINATION- subtract the lower group percentage from the upper group percentage. i.e. 43% - 15% for an item discrimination index of 28%. Covariate - -A source of variation that affects the response variable, but was not controlled for during study design (often observational studies). Quasi-experimental Designs - -When you cannot random assign you can use quasi-experimental groups (ready made groups). An experimental research design that does not provide for full control of potential confounding variables primarily because it does not randomly assign participants to comparison groups Quasi-experimental and single-case designs do have manipulation of the independent variable (otherwise they would not be "experimental research" designs). A-B-A-B Design - -In an ABAB research design, baseline data on a behavior is taken (phase A), a treatment is applied and the behavior is measured again (phase B), the treatment is withdrawn and baseline data is taken again (the second phase A), and the treatment is re-applied and the behavior is measured again (the second phase B). The purpose of an ABAB design is to insure that changes in the behavior really have to do with the treatment and not an extraneous factor. However, if the behavior does not revert to baseline levels after the treatment is withdrawn, the design cannot be effective. Baseline — the behavior of the participant prior to the administration of a treatment condition Inferential vs Descriptive Stats - -Inferential statistics — the division of statistics focused on going beyond the immediate data and inferring the characteristics of population based on samples; statistics that go beyond the immediate data and infer the characteristics of populations based on samples Descriptive statistics — statistics that focus on describing, summarizing, or explaining data Percentile rank - -the percentage of area in a histogram that is to the left of the score referred. The is an area of the curve. A percentile rank tells you the percentage of scores in a reference group (the norm group that's used to determine percentile ranks) that fall below a particular raw score. For example, if your percentile rank is 93 then you know that 93 percent of the scores in the reference group fall below your score. Percentile - -is a point on the raw score scale. Special percentiles are referred to as quartiles. (Quartile 3= Q3 = the 75%; Q2 = 50%; Q1 = 25%) Significance Level Significance Testing - -Significance level — the cutoff the researcher uses to decide when to reject the null hypothesis; also called the alpha level Significance testing — a commonly used synonym for hypothesis testing; NOTE: in significance or hypothesis testing, the researcher sets the significance (alpha) level, analyzes the data to obtain the probability value, and then the researcher compares the empirical probability value with the preset significance level to determine whether the finding is statistically significant Testing VS Assessment - -Testing is the process of measuring variables by means of devices or procedures designed to obtain a sample of behavior and Assessment is the gathering and integration of data for the purpose of making an educational evaluation, accomplished through the use of tools such as tests, interviews, case studies, behavioral observation, and specially designed apparatus and measurement procedures. Predictive validity - -Refers to the correlation between test scores based on the relationship between test scores collected at one point in time and criterion scores obtained at a later time. Extent to which future level of performance can be predicted from knowledge of prior test performance. Also know as "empirical validity". Examples are GRE, SAT Traits and States - -A TRAIT is a relatively enduring (i.e., long lasting) characteristic on which people differ; A STATE is a less enduring or more transient characteristic on which people differ. Traits and states are actually social constructions, but they are real in the sense that they are useful for classifying and organizing the world, they can be used to understand and predict behavior, and they refer to something in the world that we can measure. 4 ways to measure Reliability - -1. Test-retest