{"id":4276,"date":"2018-09-18T18:46:10","date_gmt":"2018-09-18T22:46:10","guid":{"rendered":"https:\/\/www.amyork.ca\/academic\/zz\/?p=4276"},"modified":"2018-09-18T18:57:52","modified_gmt":"2018-09-18T22:57:52","slug":"reliability","status":"publish","type":"post","link":"https:\/\/www.amyork.ca\/academic\/zz\/psychological-measurement\/reliability\/","title":{"rendered":"Reliability"},"content":{"rendered":"<ul>\n<li>Is pertinent to the consistency of measurement<\/li>\n<li>There are various types of reliability, all of which are estimated quantitatively<\/li>\n<li>Most basic way to understand the concept is through repeated measurements<\/li>\n<li>Reliability and measurement error are essentially antonymous<\/li>\n<li>We calculate various reliability estimates to understand the nature and magnitude of the measurement error associated with scores obtained from instruments and tests (tools and tests don\u2019t have reliability, only scores)<\/li>\n<\/ul>\n<p><strong>Importance of reliability:<\/strong><\/p>\n<ul>\n<li>A very large percentage of psychology is relevant to the estimation associations between variables, broadly defined (regression o ANrVA).<\/li>\n<li>If the variables we use to test our hypotheses are associated with a larger amount of measurement error, then the whole exercise is pointless.<\/li>\n<\/ul>\n<p><strong>Classical test theory (CTT):<\/strong><\/p>\n<ul>\n<li>Is a measurement theory that defines the conceptual basis of reliability<\/li>\n<li>Also specifies procedures for estimating the reliability of scores derived from a psychological test or instrument<\/li>\n<li>A person\u2019s observed score on a test is a function of that person\u2019s true score, plus error:<\/li>\n<\/ul>\n<p>X<sub>r<\/sub> = X<sub>T<\/sub> + X<sub>E<\/sub><\/p>\n<p><strong>True scores:<\/strong><\/p>\n<ul>\n<li>A hypothetical score devoid of measurement error, never be able to actually measure them<\/li>\n<li>Conceived in the context of a particular test or instrument- not a construct<\/li>\n<li>Note that true scores are not \u2018construct scores\u2019, there is no such thing<\/li>\n<li>True scores may be perfect but this is only true in the context of measurement error associated with data derived from a particular test or instrument<\/li>\n<li>True scores can be perfect from error standpoint but absolutely terrible from a valid representation of a construct standpoint<\/li>\n<li>g. person steps on scales 10 times, each time the weight is the same. This score is perfectly reliable, a true score. However how do you now that the weight the scale says is actually the person\u2019s weight? Weight could be consistently wrong<\/li>\n<\/ul>\n<p><strong>Observed scores:<\/strong><\/p>\n<p>We obtain from tests or instruments, the actual measurement.<\/p>\n<p>&nbsp;<\/p>\n<p>We want our observed scores to be close to their corresponding true scores as much as possible<\/p>\n<ul>\n<li>The differences between the two scores is the reliability.<\/li>\n<li>Discrepancy between observed scores and true scores is considered to be due to measurement error<\/li>\n<\/ul>\n<p><strong>Observed, true and error scores:<\/strong><\/p>\n<ul>\n<li>All other things equal, you want there to be a large positive correlation between observed scores and true scores <strong>o <\/strong>This correlation exists only in theory<\/li>\n<li>By contrast, you want observed scores and error scores to be uncorrelated<\/li>\n<li>If the observed scores and the error scores are correlated highly, it means they are measuring the same process: error.<\/li>\n<\/ul>\n<p><strong>Error scores:<\/strong><\/p>\n<ul>\n<li>Should have a mean of zero o This is because there should e just as many people that should have an observed score that is too large as too small<\/li>\n<li>Should be a random process o As they are random, they should not correlate with anything (except possibly their corresponding observed scores)<\/li>\n<li>Error scores should be uncorrelated with true scores o Whether you are genuinely high or low on self-esteem, the \u2018extraneous\u2019 error related factors should effect people equally<\/li>\n<\/ul>\n<p><strong>R<\/strong><strong><sup>2<\/sup> between observed and true scores:<\/strong><\/p>\n<ul>\n<li>When you square the reliability index (r), you get a conceptual estimation of the reliability<\/li>\n<\/ul>\n<p>RXX = rot2<\/p>\n<ul>\n<li>The final term (e.g R<sub>XX<\/sub>= 0.48) concludes that 48% of the variance in observed scores is shared with true scores<\/li>\n<\/ul>\n<p><strong>Interpretation guidelines for variability:<\/strong><\/p>\n<ul>\n<li>60 \u2013 too low for any purpose<\/li>\n<li>70 \u2013 bare minimum acceptable for beginning stage research<\/li>\n<li>80 \u2013 good level for research purposes<\/li>\n<li>90+ &#8211; necessary in applied contexts where important decisions are made about individuals<\/li>\n<\/ul>\n<p><strong>Ratio of true score variance to observed variance:<\/strong><\/p>\n<p>This conceptualisation is similar to eta squared: the ratio of SS<sub>EFFECT<\/sub> to SS<sub>TrTAL<\/sub><\/p>\n<p>Conceptually, in the reliability case it is the ratio of SS<sub>TRUE <\/sub>to SS<sub>rBSERVED<\/sub><\/p>\n<ul>\n<li>Formulated as: R<sub>xx <\/sub>= s<sub>t<\/sub><sup>2<\/sup> \/ S<sub>o<\/sub><sup>2<\/sup><\/li>\n<\/ul>\n<p><strong>Lack of correlation between O and E:<\/strong><\/p>\n<ul>\n<li>If reliability is the correlation between true scores and observed scores, then it is necessarily the case that it is the relative absence of a correlation between observed and error scores<\/li>\n<li>Rxx = 1 -r2oe<\/li>\n<\/ul>\n<p><strong>Relative lack of error variance:<\/strong><\/p>\n<ul>\n<li>Instead of the ratio of true score variance to observed variance, in this case we speak of the ratio of error variance to observed variance<\/li>\n<li>We subtract this ratio by 1 to place in the same context of reliability (rather than error)<\/li>\n<li>Rxx = 1 \u2013 s2e \/ s20 <strong>Conceptualisations review:<\/strong><\/li>\n<\/ul>\n<p><strong>Parallel tests:<\/strong><\/p>\n<ul>\n<li>Two tests are considered parallel if they are identical to each other psychometrically, but differ in the actual items that make up each test<\/li>\n<li>All tau-equivalence assumptions:<\/li>\n<\/ul>\n<p>o Implies that the true scores associated with each test represent the same construct o Thus a person\u2019s true score on one test would be expected to be identical on the other test<\/p>\n<ul>\n<li>Assumes equal error variance between the two tests as well<\/li>\n<\/ul>\n<p><strong>Parallel tests and reliability:<\/strong><\/p>\n<p>According to CTT, the correlation between the composite scores on test 1 and the composite<\/p>\n<p>scores on test 2 represent the reliability associated with the scores<\/p>\n<p>The closer the correlation is to 1.0 the more reliable we consider the scores<\/p>\n<ul>\n<li>Note the correlation between tests can still be reliable, but invalid at representing the attribute or construct of interest<\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>Is pertinent to the consistency of measurement There are various types of reliability, all of which are estimated quantitatively Most basic way to understand the concept is through repeated&hellip; <a class=\"continue\" href=\"https:\/\/www.amyork.ca\/academic\/zz\/psychological-measurement\/reliability\/\">Continue Reading<span> Reliability<\/span><\/a><\/p>\n","protected":false},"author":4,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[107],"tags":[],"_links":{"self":[{"href":"https:\/\/www.amyork.ca\/academic\/zz\/wp-json\/wp\/v2\/posts\/4276"}],"collection":[{"href":"https:\/\/www.amyork.ca\/academic\/zz\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.amyork.ca\/academic\/zz\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.amyork.ca\/academic\/zz\/wp-json\/wp\/v2\/users\/4"}],"replies":[{"embeddable":true,"href":"https:\/\/www.amyork.ca\/academic\/zz\/wp-json\/wp\/v2\/comments?post=4276"}],"version-history":[{"count":0,"href":"https:\/\/www.amyork.ca\/academic\/zz\/wp-json\/wp\/v2\/posts\/4276\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.amyork.ca\/academic\/zz\/wp-json\/wp\/v2\/media?parent=4276"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.amyork.ca\/academic\/zz\/wp-json\/wp\/v2\/categories?post=4276"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.amyork.ca\/academic\/zz\/wp-json\/wp\/v2\/tags?post=4276"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}