{"id":4293,"date":"2018-09-18T18:54:41","date_gmt":"2018-09-18T22:54:41","guid":{"rendered":"https:\/\/www.amyork.ca\/academic\/zz\/?p=4293"},"modified":"2018-09-18T18:57:47","modified_gmt":"2018-09-18T22:57:47","slug":"item-response-theory","status":"publish","type":"post","link":"https:\/\/www.amyork.ca\/academic\/zz\/psychological-measurement\/item-response-theory\/","title":{"rendered":"Item response theory"},"content":{"rendered":"<p><strong>Classical test theory:<\/strong><\/p>\n<ul>\n<li>Classical test theory incorporates terms such as \u2018observed scores\u2019 and \u2018true scores\u2019<\/li>\n<li>There is a substantial emphasis on the description and estimation of the reliability of scores<\/li>\n<li>Additionally, observed scores are considered a function of the sum of true scores and error scores<\/li>\n<li>In CTT a person\u2019s observed score on a test is a function of that person\u2019s true score, plus error<\/li>\n<\/ul>\n<p><strong>Modern test theory:<\/strong><\/p>\n<ul>\n<li>An alternative to CTT, more commonly referred to as item response theory (IRT)<\/li>\n<li>More complicated than CCT, however its benefits are so great that it outweighs the drawbacks<\/li>\n<li>The application of computer adaptive testing (CAT) is the main difference and benefit between CCT and IRT, both actually produce relatively similar results<\/li>\n<\/ul>\n<p><strong>IRT:<\/strong><\/p>\n<ul>\n<li>rperates under the pretence that the response to any given item is influenced by two factors:<\/li>\n<\/ul>\n<p>o Qualities of the individual o Qualities of the item<\/p>\n<ul>\n<li>There are three well established models of IRT \uf0b7 In the most basic model:<\/li>\n<\/ul>\n<p>o rnly item characteristics are taken into consideration<\/p>\n<ul>\n<li>Most common characteristic is item difficulty<\/li>\n<li>That is, the probability a person may answer the question correctly<\/li>\n<\/ul>\n<ul>\n<li>g. on a five item test of mathematical ability o Likelihood that person will respond correctly to any item will be affected by their level of mathematical ability and the item difficulty<\/li>\n<\/ul>\n<p><strong>IRT and self-report questionnaires:<\/strong><\/p>\n<ul>\n<li>Although IRT is typically used in the context of intelligence or education testing, the basic IRT model can be extended into personality type questions<\/li>\n<li>Principles are effectively the same o How much of the trait does the person possess o How likely is that someone would endorse or agree with the item<\/li>\n<\/ul>\n<p><strong>Trait level:<\/strong><\/p>\n<ul>\n<li>Across all models of IRT, the notion of a person\u2019s trait level is fundamental<\/li>\n<li>People are considered to possess certain levels of a trait<\/li>\n<\/ul>\n<p>Effectively includes all psychological constructs (personality, intelligence, attitudes etc.)<\/p>\n<p><strong>Item difficulty:<\/strong><\/p>\n<ul>\n<li>An item\u2019s level of difficulty is a factor affecting an individual\u2019s probability of responding in a particular way o g. 2+2 or square root of 10 000\n<ul>\n<li>g. \u2018 I enjoy socially with groups of people (high probability) or \u2018I enjoy speaking before large audiences\u2019 (low probability)<\/li>\n<li>g. my job is ok (high probability of involvement) or I enjoy my job (moderate probability)<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p><strong>Trait level and item difficulty:<\/strong><\/p>\n<ul>\n<li>Trait level and item difficulty are intrinsically connected concepts in IRT<\/li>\n<li>A difficult item requires a relatively high trait level in order to be answered correctly<\/li>\n<\/ul>\n<p><strong>IRT metric:<\/strong><\/p>\n<ul>\n<li>Trait levels and item difficulties are usually scored on a standardised metric o Mean = 0, SD = 1<\/li>\n<li>Therefore a person with a trait level of 0 is average<\/li>\n<li>Similar, an item associated with a difficulty level of 0 is considered of average difficulty<\/li>\n<li>An item\u2019s difficulty is defined at the trait level required to have a .50 probability of answering the item correctly o If an item has a difficulty level of 1.5, then it will take a trait level of 1.5 to have a<\/li>\n<\/ul>\n<p>50% chance of responding correctly o If an item has a difficulty of -1.5, then a person with\u00a0 a trait level of 1.0 would have a greater than 50% chance of answering correctly<\/p>\n<p><strong>Item discrimination:<\/strong><\/p>\n<ul>\n<li>In addition to different levels of difficulty, items can also be differentiated with respect to discrimination- known as item discrimination<\/li>\n<li>It is similar to the item-total correlation that we came across in the context of classical test theory<\/li>\n<li>An items discrimination value indicates the relevance of the item to the train being measured by the test (similar to component loadings)<\/li>\n<li>Items that have large and positive discrimination values are good<\/li>\n<li>Items will have a bad item discrimination values when they are not particularly relevant to the train of interest<\/li>\n<\/ul>\n<p><strong>IRT models:<\/strong><\/p>\n<ul>\n<li>Variety of models developed from the IRT perspective<\/li>\n<li>The main way the models differ from each other is with respect to the nature and number of parameters they include<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<p>Three common models: o rne-parameter logistic model o two-parameter logistic model o three-parameter logistic model<\/p>\n<p><strong>One-parameter logistic model:<\/strong><\/p>\n<ul>\n<li>Also known as the Rasch model<\/li>\n<li>Additionally to an individual\u2019s trait level, it states that an individual\u2019s response to an item is determined by:<\/li>\n<\/ul>\n<p>o The difficulty of the item (this is the one parameter)<\/p>\n<p><strong>Two-parameter logistic model:<\/strong><\/p>\n<ul>\n<li>Additionally to an individual\u2019s trait level, it states that an individual\u2019s response to an item is determined by:<\/li>\n<\/ul>\n<p>o The difficulty of the item (parameter 1) o Discrimination of the item (parameter 2)<\/p>\n<ul>\n<li>Not surprisingly, the two-parameter model is much more useful than the one-parameter model<\/li>\n<li>Probably the most common used IRT model<\/li>\n<\/ul>\n<p><strong>Three-parameter logistic model:<\/strong><\/p>\n<ul>\n<li>Additionally to an individual\u2019s trait level, it states that an individual\u2019s response to an item is determined by:\n<ul>\n<li>The difficulty of the item (parameter 1) o Discrimination of the item (parameter 2)<\/li>\n<li>The probability which the question can be answered by guessing<\/li>\n<\/ul>\n<\/li>\n<li>Especially useful in multi-choice testing, but not commonly used (rarely much benefit to the third parameter)<\/li>\n<\/ul>\n<p><strong>IRT \u2013 dichotomous and polytomous items:<\/strong><\/p>\n<ul>\n<li>IRT is typically discussed in the context of dichotomously scored items (correct\/incorrect)<\/li>\n<li>However there are models that can accommodates items responded to on rating scales<\/li>\n<\/ul>\n<p><strong>Item characteristics curve (ICC):<\/strong><\/p>\n<ul>\n<li>People who work with IRT often evaluate the quality of an item using a graph known as an item characteristics curve<\/li>\n<li>Reflects the probability with which individuals across a range of trail levels are likely to answer each item correctly<\/li>\n<li>The logistic formula and the parameters included in the mode are used to predict the probabilities <strong>o <\/strong>Similar to using a regression equation<\/li>\n<li>Lowest line suggest that for an item answered three standard deviations about the mean, has a \u2018correct\u2019 chance of 80% (very hard question)<\/li>\n<li>Top line suggests for an item three standard deviations below the mean, has a correct chance of 30% (very easy question).<\/li>\n<\/ul>\n<p><strong>IRT and reliability:<\/strong><\/p>\n<ul>\n<li>You cannot calculate something like coefficient alpha in an IRT model<\/li>\n<li>In CTT, reliability represents the level of true score variance associated with a group of composite scores<\/li>\n<li>The reliability estimate is assumed to apply across all levels of composite scores (from low to high)<\/li>\n<li>In IRT this notion is abandoned, o For a given test different levels of measurement error are acknowledged across different levels of a particular trait.<\/li>\n<\/ul>\n<p><strong>Applications of IRT:<\/strong><\/p>\n<ul>\n<li>Test development and improvement<\/li>\n<li>Differential item functioning (occurs when people from different groups with the same trait have a different probability of giving a certain response)<\/li>\n<li>Person fit ( example where you look at characteristics of a person and a job, try to match the right person to the right job)<\/li>\n<li>Computer adaptive testing<\/li>\n<\/ul>\n<p><strong>CTT- testing example:<\/strong><\/p>\n<ul>\n<li>Generate 30 items, order them in terms of difficulty<\/li>\n<li>Administer the items to participants in order of item difficulty o Either administer all of the items<\/li>\n<\/ul>\n<p>o rr stop after the participant makes three errors in a row<\/p>\n<ul>\n<li>This is the conventional, static approach<\/li>\n<\/ul>\n<p><strong>Computer adaptive testing (CAT):<\/strong><\/p>\n<ul>\n<li>A dynamic approach to measuring a construct<\/li>\n<li>CAT involves an algorithm which selects items to present to respondents based on their responses to previous items<\/li>\n<\/ul>\n<p>In practice, no two people 9or very few) who take a CAT will be expected to respond to the same items<\/p>\n<p><strong>The approach to CAT:<\/strong><\/p>\n<ul>\n<li>A large pool of items (say 300) are tested on a large normative sample (say 500)<\/li>\n<li>Using the data from 500, each item can be estimated with respect to its item difficulty<\/li>\n<li>Items are then placed into categories of difficulty (up to 20 categories)<\/li>\n<li>For the first item administered, randomly selects an item with a difficulty level of 0<\/li>\n<li>If the person gets the first item correct, the computer will move them up a difficulty level and select a randomly assigned item<\/li>\n<li>The computer will keep increasing difficulties until the person makes a mistake<\/li>\n<li>As the computer continues to give the respondent items at this particular item difficulty level, the standard error of estimate associated with the latent trait estimate continues to narrow<\/li>\n<li>Bas on an arbitrary cut off point, the CAT will terminate once the standard error of estimate confidence interval has narrowed sufficiently<\/li>\n<\/ul>\n<p><strong>Advantages of CAT:<\/strong><\/p>\n<ul>\n<li>Takes as much as 50% less time for testing<\/li>\n<li>More accurate scores, because respondents answer more items in \u2018their area of difficulty\u2019 o In CTT testing, many respondents waste time answering really easy items o In CTT testing, typically there are only a handful of items that are in the respondents area of difficulty<\/li>\n<\/ul>\n<p><strong>Disadvantages of CAT:<\/strong><\/p>\n<ul>\n<li>More time\/money to develop<\/li>\n<li>Participants don\u2019t \u2018trust\u2019 it (how can you measure me in 10 minutes?)<\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>Classical test theory: Classical test theory incorporates terms such as \u2018observed scores\u2019 and \u2018true scores\u2019 There is a substantial emphasis on the description and estimation of the reliability of&hellip; <a class=\"continue\" href=\"https:\/\/www.amyork.ca\/academic\/zz\/psychological-measurement\/item-response-theory\/\">Continue Reading<span> Item response theory<\/span><\/a><\/p>\n","protected":false},"author":4,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[107],"tags":[],"_links":{"self":[{"href":"https:\/\/www.amyork.ca\/academic\/zz\/wp-json\/wp\/v2\/posts\/4293"}],"collection":[{"href":"https:\/\/www.amyork.ca\/academic\/zz\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.amyork.ca\/academic\/zz\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.amyork.ca\/academic\/zz\/wp-json\/wp\/v2\/users\/4"}],"replies":[{"embeddable":true,"href":"https:\/\/www.amyork.ca\/academic\/zz\/wp-json\/wp\/v2\/comments?post=4293"}],"version-history":[{"count":0,"href":"https:\/\/www.amyork.ca\/academic\/zz\/wp-json\/wp\/v2\/posts\/4293\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.amyork.ca\/academic\/zz\/wp-json\/wp\/v2\/media?parent=4293"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.amyork.ca\/academic\/zz\/wp-json\/wp\/v2\/categories?post=4293"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.amyork.ca\/academic\/zz\/wp-json\/wp\/v2\/tags?post=4293"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}