IAS-OECD Rio pilot data collection and analysis

(1)

IAS-OECD Rio pilot data collection and

analysis

ESP Experts Meeting on Skill Instruments

Marina Palace Hotel, Rio de Janeiro, 2-6 December 2013

Ricardo Primi (University of São Francisco, Brazil) & Daniel Santos (University

of São Paulo, Brazil)

(2)

Context and Goals

• Context

• Ayrton Senna Institute and OEDC ESP longitudinal study need to define

and measure non-cognitive skills

• Rio de Janeiro’s state and municipal government partnership with ASI to

monitor some innovative programs in community education

• Long list of instruments and constructs

• The proposal of Big 5 as a conceptual framework for organizing

non-cognitive skills (Kyllonen, Lipnevich, Burrus, & Roberts, 2008; Kyllonen,

Walters & Kaufman, 2011).

• Questions of measurement invariance / measurement validity /

• Questions

• Could the list be shortened without loosing information?

• Could they be mapped into B5 ?

• Can we measure the same construct in the same way in different

ages/countries ?

• Pilot Study

(3)

Phases

Phase 1. The review of literature of non-cognitive constructs and instruments

and assessment of their feasibility for large scale assessment (started

4/2012). Selection of a core battery of tests.

Phase 2. Translation, adaptation and sensible content review of core

instruments selected in phase 1.

(4)

Phase 1: Literature review of

non-cognitive constructs and instruments

• Review of definitions of non-cognitive constructs

• Assessment of instruments in two dimensions on a five point scale:

• Predictive validity (number of studies, extent of important outcomes

magnitude of the associations)

• Feasibility for large scale assessment (extension, availability of Portuguese

version, reliability and valid data with Brazilian samples)

• Also: Malleability and school effect

• Book with the revisions of approximately 75 instruments

• Selection of instruments to build a core battery for the next phases (also input

(5)

Selected instruments

• Big Five Inventory (BFI)

• Big Five for Children (BFC)

• Strengths and Difficulties Questionnaire (SDQ)

• Self-Efficacy Questionnaire for Children

• Grit Scale

• Core Self Evaluations

• Rosenberg Self-Esteem Scale

• Norwick-Strickland Locus of Control Scale

• HiPIC – Hierarquical Personality Inventory for Children

• SSIS – Social Skills Improvement System

• Possible Framework for Organizing the Scales: Big Five

(6)

(7)

BORSA, Juliane Callegaro; DAMASIO, Bruno Figueiredo e BANDEIRA, Denise Ruschel.

Cross-cultural

adaptation and validation of psychological instruments: some considerations.

Paidéia (Ribeirão Preto)

[online]. 2012, vol.22, n.53, pp. 423-432. ISSN 0103-863X.

1. Two to four members

of our team reviewed

and/translated core

battery instruments

3. Focus groups with

teachers from target

sample children

2. A consensus version

4. A qualitative study with

focus groups with target

children

5. Back translation

(8)

Phase 3 Pre-pilot study 2. A qualitative

study with focus groups of target children

• Test comprehension, size of admissible test and response process (likert

scale, average behavior and trait-situation problem)

• Confirm the size of the test discovered in the pre-pilot study 1 (see following

slides)

• Item priority: items classified as abstract and with difficult vocabulary (154

items)

• Qualitative study with 48 children from 5

th

and 9

th

grades in groups of 5

children from two schools

• Read aloud together with experimenter

• Children answer and then discuss with experimenter

• Results

• Positive motivation

(9)

Phase 3. Pre-pilot study 1: time needed to

answer the instruments

• Question

• How many items could our instrument have ?

• Constraints: a test session of 50 minutes (10 minutes for instruction and

socio-economic questionnaire)

• Objective

• To test the average time required for children to complete each test

• Sample

• N=228 students (low grades and schools that are below average in

achievement tests)

• Two 5

th

Grade classes: 60 students; two 6

th

Grade classes: 42 students;

four of 10

th

_{Grade classes: 126 students}

• Design

• Seven core batteries

• 50 minutes

• Start and end time for each instrument. Changing different color pens at

intervals of 10 minutes

(10)

(11)

(12)

Phase 5 Pre-pilot study 3: Psychometric

Properties of Core Battery

• Questions

• What are the psychometric properties of the selected instruments?

• What is the underlying dimensionality of these scales (Is

d

<16?)

• Which are the best items to measure those

d

dimensions?

(13)

Study Design

• Constraints

• At the time of this study it was not decided whether, in the pilot,

we would use the original tests or create a new test with

different combinations of items from original scales. We

therefore studied each test as it was in its original form

(combination and order of items ).

• We have 50 minutes. Therefore we could not’ apply all the tests

to the same children

(14)

Balanced Incomple Block Desing (BIB) and

Spiriling (Sailer, 2005)

[1,] "BFC" "BFI" "109"

[2,] "BFC" "SDQ" "90"

[3,] "BFC" "Locus" "86"

[4,] "BFC" "Self-ef" "89"

[5,] "BFC" "RosmCore" "83"

[6,] "BFC" "Grit" "75"

[7,] "BFI" "SDQ" "69"

[8,] "BFI" "Locus" "65"

[9,] "BFI" "Self-ef" "68"

[10,] "BFI" "RosmCore" "62"

[11,] "BFI" "Grit" "54"

[12,] "SDQ" "Locus" "46"

[13,] "SDQ" "Self-ef" "49"

[14,] "SDQ" "RosmCore" "43"

[15,] "SDQ" "Grit" "35"

[16,] "Locus" "Self-ef" "45"

[17,] "Locus" "RosmCore" "39"

[18,] "Locus" "Grit" "31"

[19,] "Self-ef" "RosmCore" "42"

[20,] "Self-ef" "Grit" "34"

[21,] "RosmCore" "Grit" "28"

[1,] "BFI" "Self-ef" "RosmCore" "86"

[2,] "BFC" "SDQ" "Self-ef" "114"

[3,] "BFI" "SDQ" "Locus" "90"

[4,] "BFC" "Locus" "RosmCore" "104"

[5,] "SDQ" "RosmCore" "Grit" "53"

[6,] "BFC" "BFI" "Grit" "119"

[7,] "Locus" "Self-ef" "Grit" "55" “

5th grade (balanced block design)

(15)

Sample and Power Considerations

• Usually item correlations are in the moderate range (.30). In

Cohen’s (1992) simple tutorial, an N=85 is required to detect

correlations in the moderate range with .05 statistical significance

and a power of .80.

• 20 booklets for 5

th

grades -> target sample of N=1700

• 7 booklets for 10

th

grades -> target sample of N=595

• Spiraled distribution of booklets within classes in order to have

(16)

Final Sample

• N=3023 (5

th

N=697, 6

th

N=710, 9

th

N=674, 10

th

N=488, 12

th

N=454)

• 16 Schools (86 classes) with achievement in the average range

• From two systems: Municipal N=2081; state N=942

• Booklets N ranged from 93 to 252

• N by tests:

• BFC N=920 , BFI N=927

• SDQ N=1055, Locus N=1026

• Auto N=1011

• Grit N=985

• Rosemberg N=602 (only 4-6

th

)

(17)

Statistical analysis

• Scale level Exploratory Factor Analyses

• Underlying dimension question at well known construct level

• Parallel analysis to investigate the number of factors

• Two runs were compared - one with all the samples and another with 5

th

-6

th

_grades

• Item level Exploratory Factor Analysis

• Underlying dimension question at item level. Which were the best items to

measure the intended constructs ?

• We looked at up to 10 factors and compared two runs (all samples and

with 5

th

_-6

th

_{grades) searching for factors similar to the ones defined in the}

(18)

(19)

(20)

Legend

Big Five Inventory (BFI)

:

Consc2, Open2, Extra2, Neuro2 (emotional stability), Agree2

Big Five for Children (BFC)

:

Consc1, Open1, Extra1, Neuro1 Agree1

Strengths and Difficulties Questionnaire (SDQ)

:

Emotional Symptoms (EmoSym), Conduct Problems (CndProb),

Hyperactivity (HypAc), Peer Problems (PeerProb), Prosocial Behav.

(ProSoc).

Self-Efficacy Questionnaire for Children

:

Academic (SlfAcd), Social (SlfSoc), Emotional (SlfEmo).

Grit Scale

: Grit

Core Self Evaluations and Rosemberg Self-Esteem Scale

: SE

(standardized in each grade group)

Norwick-Strickland Locus of Control Scale

: Locus

B5

(21)

(22)

(23)

(24)

Final Item Selection

• Four researchers independently reviewed item-factor analysis to

select items

• Constraints for selection:

• High loadings and variance

• Positive and negative poles representation (better

representation and possible use in acquiescence control, Soto,

John, Gosling, & Potter, 2008)

• Keep a maximum of 90 items (and 60 for younger children)

• Passed on qualitative and focus group study

• Selections were contrasted and a consensus version was final

decided upon

(25)

Final internal consistency coefficients

Scale

Alfa FA

Items

Alfa FB

Items

F1 C

.88

17 .83

10 F2 E

.79

14 .74

11 F3 N1

.82

14 .72

9 F4 N2

.77

15 .72

12 F5 A

.76

15 .70

12

(26)

See supplementary files:

IAS-Rio Pilot Dec. 2013 Apendix.xlsx

and

(27)

Phase 6 : Pilot study

Large scale assessment in Rio de Janeiro

State

(28)

Goals

• Assess non-cognitive skills and describe their distribution with respect to

important variables related to the school system

• Study the psychometric properties of the new instrument created in the

previous study

• Investigate the associations of non-cog with outcomes (achievement and

other )

• Investigate the relationship and of non-cog with school variables (cross level

interaction)

• Method

• Representative sample of students in RJ state

• 82-item and 62-item tests assessing the 6 factors

• Socioeconomic questionnaire

• Anchoring vignettes for controlling Person DIF on rating scale usage

• More about this later

(29)

Sample

• Sample from the State of Rio de Janeiro:

• 14 regions, 79 cities, 431 schools, 1,062 classes (N for each

class M=23, DP=7.6 range 1 to 50)

• N = 24.605 children and adolescents, 41.8% boys and 58.2% girls

• Grades

•

5 th

grade N=1.472 – 6% (problem due a teachers strike),

•

10 th

grade N=14.504 - 58.9%

•

12 th

grade N=8.629 - 35.1%.

• Ethnic origin

• 28.9% white, 40.4% ‘pardo’ (mixed-race), 21.1% black, 5.9%

Asian.

• Ages

(30)

(31)

Item Factor Analysis

• Question

• Do we recover the factor structure of six factors with the same indicators ?

• Statistical Analysis

• Parallel analysis to investigate the number of factors (using simulation of

matrices of polychoric correlations with the Psych R package -Revelle,

2010)

• Exploratory Structural Equation Modeling (ESEM, Marsh et all 2010) by the

WLSMV – weighted least square parameter estimates (Muthen & Muthen,

2012)

• Assumed categorical scale (accommodates non-normality).

• Oblimin rotation (Geomin)

• Extraction of 6 factors

(32)

(33)

(34)

(35)

Measurement invariance with respect to

grades

ESEM approach with MPLUS

• Analysis of the 62 item scale common to all grades

• Questions

• Configural Invariance

• Is the six-factor structure suitablefor representing the underlying

structure of the 62-item covariance matrix across grades? Does this

latent structure not vary across grades ?

• Metric Invariance

• Do six factors and item loadings not vary across grades ?

• Scalar Invariance

• Do six factors, item loadings, and item indicators means (controlled for

(36)

Results of Multiple Group ESEM

• This time: indicators were assumed as continuous variables (ESEM + metric + categorical variable is

not available in MPLUS)

• Only configural and scalar invariance is available for categorical indicators. The tell the same story.

• ESEM with MLR maximum likelihood parameter estimates with robust to non-normality standard errors

(37)

Summary

• Strong evidence of internal structure validity of the 92 item scale

• High Internal Consistency Coefficients

• Measurement invariance: evidence that the test is measuring the same

construct in the three groups studied

• Configural and scalar

(38)

IRT analysis

Goals

• To investigate the psychometric properties of the scales

• Category structure, scale information function

• To equate scores of the full 92-full item test with the 62-short-form (common item

equating)

• To analyze item maps (construct maps) to improve the understanding of the

underlying construct and provide additional information on scale interpretations

(item-referenced meaning, Embretson & Reise, 2000)

• Test Differential Item Functioning (DIF) with respect to grades and gender

Method

• An analysis was repeated six times, one for each factor

• A Rasch-Andrich Rating Scale Model (Wright, & Masters, 1982) was used

It is the IRT model applied to the polytomous item format. It estimates item

threshold parameters for each item score transition 1 to 2, 2 to 3, 3 to 4 and 4

to 5 (analogous to item difficulty )

• Item (from long and short form) and parameters were calibrated simultaneously.

The 62-item test was treated as different from the full test but with all common

items with the full version.

• Metric was identified fixing the entire sample mean of theta as M=0 and standard

(39)

Some Preliminary results

• Rating Scale Structure

• Item Map + DIF

(40)

F1 C

---

|CATEGORY STRUCTURE | SCORE-TO-MEASURE | 50% CUM.| COHERENCE |ESTIM| | LABEL MEASURE S.E. | AT CAT. ----ZONE----|PROBABLTY| M->C C->M RMSR |DISCR| |---+---+---+---+---|

| 1 NONE |( -2.97) -INF -2.21| | 76% 6% 1.6093| | 1 | 2 -1.47 .01 | -1.35 -2.21 -.69| -1.87 | 39% 32% .9380| .90| 2 | 3 -.95 .00 | -.05 -.69 .61| -.76 | 48% 64% .5464| 1.01| 3 | 4 .74 .00 | 1.33 .61 2.28| .64 | 43% 58% .6482| 1.08| 4 | 5 1.68 .01 |( 3.10) 2.28 +INF | 1.98 | 76% 23% 1.1334| .97| 5 ---

M->C = Does Measure imply Category? C->M = Does Category imply Measure?

CATEGORY PROBABILITIES: MODES - Structure measures at intersections P -+---+---+---+---+---+---+---+---+-

R 1.0 + + O | | B |111 55| A | 111 555 | B .8 + 11 55 + I | 11 55 | L | 1 5 | I | 1 55 | T .6 + 11 5 + Y | 1 5 | .5 + 1 5 + O | 1 33333333 5 | F .4 + 1 3 3344444**4 + | 2222*22** 443 5 444 | R | 22 13 222 44 335 44 | E | 222 3311 2 44 53 44 | S .2 + 22 3 1 4*2 55 33 444 + P | 222 33 114 22 5 33 444 | O |2222 333 44411 5**2 333 44| N | 33333 4444 5***1 2222 33333 |

S .0 +******************55555555 1111111*******************+ E -+---+---+---+---+---+---+---+---+-

(41)

F1 C

EXPECTED SCORE: MEAN (Rasch-score-point threshold, ":" indicates Rasch-half-point threshold) (ILLUSTRATED BY AN OBSERVED CATEGORY) -5 -4 -3 -2 -1 0 1 2 3 4 5

|---+---+---+---+---+---+---+---+---+---| NUM ITEM

1 1 : 2 : 3 : 4 : 5 5 4 i80.f1k1_T5_Hw can you study when there are other interesting things to do? | |

| |

1 1 : 2 : 3 : 4 : 5 5 37 i21.f1k1_T2_Does things efficiently 5 5 : 4 : 3 : 2 : 1 1 57 i41.f1k0_T2_Tends to be lazy

1 1 : 2 : 3 : 4 : 5 5 15 i91.f1k1_T5_Hw can you study a chapter for a test? | |

1 1 : 2 : 3 : 4 : 5 5 27 i11.f1k1_T2_Does a thorough job

1 1 : 2 : 3 : 4 : 5 5 1 i77.f1k1_T5_Hw do you succeed in finishing all your homework every day? 1 1 : 2 : 3 : 4 : 5 5 42 i26.f1k1_T1_I like to keep all my school things in great order

1 1 : 2 : 3 : 4 : 5 5 22 i06.f1k1_T8_I am diligent (hard working and careful). | |

5 5 : 4 : 3 : 2 : 1 1 52 i36.f1k0_T2_Is easily distracted

1 1 : 2 : 3 : 4 : 5 5 13 i89.f1k1_T5_Hw do you succeed in passing a test? 1 1 : 2 : 3 : 4 : 5 5 17 i01.f1k1_T8_I am a hard worker.

5 5 : 4 : 3 : 2 : 1 1 32 i16.f1k0_T8_I have difficulty maintaining (keeping) my focus on projects that take more than a few months to complete. | |

1 1 : 2 : 3 : 4 : 5 5 7 i83.f1k1_T5_Hw can you pay attention during every class?

5 5 : 4 : 3 : 2 : 1 1 62 i46.f1k0_T3_I am easily distracted, I find it difficult to concentrate 5 5 : 4 : 3 : 2 : 1 1 67 i51.f1k0_T2_Tends to be disorganized

1 1 : 2 : 3 : 4 : 5 5 10 i86.f1k1_T5_Hw do you succeed in satisfying your parents with your schoolwork? | |

| |

5 5 : 4 : 3 : 2 : 1 1 47 i31.f1k0_T2_Can be somewhat careless |---+---+---+---+---+---+---+---+---+---| NUM ITEM

-5 -4 -3 -2 -1 0 1 2 3 4 5

11111111111

1223679647877753115763212111

111253922790589823962079066408809411935 6 1 3 3

4 2 7165384436623031782810874237097898192535491522 42 1 8 PERSON

T S M S T

0 10 20 40 60 80 90 99 PERCENTILE

DIF analysis: is more difficult to endorse for kids of 5th grade

(42)

F2 E

---

| 1 NONE |( -3.35) -INF -2.38| | 64% 3% 1.8546| | 1 | 2 -1.25 .01 | -1.35 -2.38 -.66| -1.87 | 35% 24% 1.1780| .83| 2 | 3 -.66 .01 | -.06 -.66 .57| -.63 | 35% 47% .6834| .96| 3 | 4 .38 .01 | 1.32 .57 2.45| .48 | 40% 67% .5991| 1.06| 4 | 5 1.53 .01 |( 3.50) 2.45 +INF | 2.00 | 76% 21% 1.0705| 1.06| 5 ---

M->C = Does Measure imply Category? C->M = Does Category imply Measure?

CATEGORY PROBABILITIES: MODES - Structure measures at intersections P -+---+---+---+---+---+---+---+---+-

R 1.0 + + O | | B | | A |111 5| B .8 + 111 555 + I | 11 55 | L | 11 555 | I | 11 5 | T .6 + 1 55 + Y | 11 55 | .5 + 1 5 + O | 1 55 | F .4 + 11 44444*4 + | 2222*22 333333444 55 4444 | R | 2222 11***2 44333 5 444 | E | 222 331 2*4 **3 444 |

S .2 + 2222 333 1144 22 55 33 4444 + P |2222 333 4411 **2 333 44| O | 333 444 1*5 222 3333 |

N | 3333333 44444 55555 11111 22222 3333333 | S .0 +***************55555555 1111111****************+ E -+---+---+---+---+---+---+---+---+-

(43)

F2 E

EXPECTED SCORE: MEAN (Rasch-score-point threshold, ":" indicates Rasch-half-point threshold) (ILLUSTRATED BY AN OBSERVED CATEGORY) -7 -5 -3 -1 1 3 5

|---+---+---+---+---+---| NUM ITEM

1 1 : 2 : 3 : 4 : 5 5 11 i87.f2k1_T5_Hw can you have a chat with an unfamiliar person?

| | | |

5 5 : 4 : 3 : 2 : 1 1 83 i67.f2k0_T2_Is reserved | |

5 5 : 4 : 3 : 2 : 1 1 75 i59.f2k0_T2_Tends to be quiet | |

1 1 : 2 : 3 : 4 : 5 5 14 i90.f2k1_T5_Hw can you tell a friend that you don’t feel well? | |

5 5 : 4 : 3 : 2 : 1 1 89 i73.f2k0_T2_Is sometimes shy, inhibited 1 1 : 2 : 3 : 4 : 5 5 92 i76.f2k1_T2_Generates a lot of enthusiasm | |

1 1 : 2 : 3 : 4 : 5 5 16 i92.f2k1_T5_Hw can you tell other children that they are doing something that you don’t like? 1 1 : 2 : 3 : 4 : 5 5 5 i81.f2k1_T5_Hw can you tell a funny event to a group of children?

1 1 : 2 : 3 : 4 : 5 5 79 i63.f2k1_T2_Is full of energy | |

1 1 : 2 : 3 : 4 : 5 5 91 i75.f2k1_T1_I like to be with others

1 1 : 2 : 3 : 4 : 5 5 2 i78.f2k1_T5_Hw can you become friends with other children?

1 1 : 2 : 3 : 4 : 5 5 8 i84.f2k1_T5_Hw do you succeed in staying friends with other children? 1 1 : 2 : 3 : 4 : 5 5 87 i71.f2k1_T1_I like to talk with others

1 1 : 2 : 3 : 4 : 5 5 71 i55.f2k1_T1_I am happy and lively |---+---+---+---+---+---| NUM ITEM

-7 -5 -3 -1 1 3 5

1122212211

11235824612331167554322 1 1

111542658652779640302495898417 0 6 4 2

1 1 1 177945823122177259282434375462895071268 110 7 PERSON

T S M S T

0 10 30 50 70 90 99 PERCENTILE

(44)

F3 N intern.

---

| 1 NONE |( -2.84) -INF -1.99| | 79% 19% 1.1944| | 5

| 2 -1.23 .01 | -1.08 -1.99 -.48| -1.62 | 40% 57% .7259| 1.11| 4 | 3 -.35 .00 | .03 -.48 .53| -.42 | 36% 55% .6377| 1.04| 3 | 4 .50 .01 | 1.10 .53 1.95| .49 | 34% 34% 1.0005| .93| 2 | 5 1.08 .01 |( 2.75) 1.95 +INF | 1.55 | 69% 11% 1.5305| .88| 1 --- M->C = Does Measure imply Category? C->M = Does Category imply Measure? CATEGORY PROBABILITIES: MODES - Structure measures at intersections P -+---+---+---+---+---+---+---+- R 1.0 + + O | | B |1111 |

A | 111 |

B .8 + 111 555+

I | 11 55 |

L | 11 55 |

I | 11 5 |

T .6 + 1 55 +

Y | 11 5 |

.5 + 1 55 +

O | 11 5 |

F .4 + 2*222 5 +

| 2222 1 2222333333 444**4444 | R | 222 11 33322 4*3 5 444 |

E | 222 3* 2*4 *33 444 |

S .2 + 222 333 11 444 2255 33 444 +

P | 2222 33 4*1 5522 333 44|

O |2222 3333 444 1*5 22 333 |

N | 333333 44444 55555 11111 22222 333333 |

S .0 +********************5555555 1111111************+

E -+---+---+---+---+---+---+---+- -4 -3 -2 -1 0 1 2 3

(45)

F3 N Intern.

EXPECTED SCORE: MEAN (Rasch-score-point threshold, ":" indicates Rasch-half-point threshold) (ILLUSTRATED BY AN OBSERVED CATEGORY) -5 -4 -3 -2 -1 0 1 2 3 4 5

|---+---+---+---+---+---+---+---+---+---| NUM ITEM 1 1 : 2 : 3 : 4 : 5 5 43 i27.f3k1_T1_I am sad

1 1 : 2 : 3 : 4 : 5 5 28 i12.f3k1_T1_I am in a bad mood | |

1 1 : 2 : 3 : 4 : 5 5 33 i17.f3k1_T2_Gets nervous easily

1 1 : 2 : 3 : 4 : 5 5 48 i32.f3k1_T3_I get very angry and often lose my temper 1 1 : 2 : 3 : 4 : 5 5 58 i42.f3k1_T2_Can be tense

1 1 : 2 : 3 : 4 : 5 5 23 i07.f3k1_T1_I easily get angry

5 5 : 4 : 3 : 2 : 1 1 9 i85.f3k0_T5_Hw do you succeed in becoming calm again when you are very scared? 1 1 : 2 : 3 : 4 : 5 5 18 i02.f3k1_T1_I easily loose my calm

5 5 : 4 : 3 : 2 : 1 1 12 i88.f3k0_T5_Hw do you succeed in suppressing unpleasant thoughts? | |

5 5 : 4 : 3 : 2 : 1 1 6 i82.f3k0_T5_Hw can you control your feelings? 1 1 : 2 : 3 : 4 : 5 5 38 i22.f3k1_T1_I am impatient

| |

5 5 : 4 : 3 : 2 : 1 1 3 i79.f3k0_T5_Hw can you prevent to become nervous? | |

5 5 : 4 : 3 : 2 : 1 1 53 i37.f3k0_T2_Is relaxed, handles stress well. | |

| |

1 1 : 2 : 3 : 4 : 5 5 63 i47.f3k1_T2_Worries a lot |---+---+---+---+---+---+---+---+---+---| NUM ITEM

-5 -4 -3 -2 -1 0 1 2 3 4 5

1111111111

11 23449624677765829876523111

2 3 5 9 5836338726895095454715102460186 4 3 2 1

02 11 16211438141969273635572051453968298216182153 1 9 2 PERSON T S M S T

0 10 20 40 60 80 90 99 PERCENTILE

(46)

F4 N loc.

---

| 1 NONE |( -3.98) -INF -2.76| | 85% 32% .9774| | 1

| 2 -.87 .01 | -1.52 -2.76 -.71| -1.99 | 29% 65% .6038| 1.26| 2 | 3 -.90 .01 | -.01 -.71 .69| -.66 | 34% 44% .7443| 1.03| 3 | 4 .82 .01 | 1.51 .69 2.77| .63 | 26% 12% 1.4103| .82| 4 | 5 .94 .01 |( 4.01) 2.77 +INF | 2.02 | 64% 1% 2.2457| .57| 5 --- M->C = Does Measure imply Category? C->M = Does Category imply Measure? CATEGORY PROBABILITIES: MODES - Structure measures at intersections P -+---+---+---+---+---+---+- R 1.0 + + O | 55555|

B | 55555 |

A |1111 5555 |

B .8 + 11 55 +

I | 111 55 |

L | 1 55 |

I | 11 55 |

T .6 + 11 5 +

Y | 1 55 |

.5 + 1 5 +

O | 1 5 |

F .4 + 11 55 +

| 1 5 |

R | 222222*33333333**4444444 | E | 2222 333**2 44*5333 4444 |

S .2 + 2222 33 1**25 33 444 +

P | 2222 333 4441*522 333 4444 |

O |22 333 44 55 11 222 333 444444 |

N | 333333 4444445555 111122222 333333 444444|

S .0 +*************555555 11111************************+

E -+---+---+---+---+---+---+- -5 -3 -1 1 3 5 7

(47)

F4 Loc.

EXPECTED SCORE: MEAN (Rasch-score-point threshold, ":" indicates Rasch-half-point threshold) (ILLUSTRATED BY AN OBSERVED CATEGORY) -6 -4 -2 0 2 4 6 8

|---+---+---+---+---+---+---| NUM ITEM

1 1 : 2 : 3 : 4 : 5 5 24 i08.f4k1_T3_I am often accused of lying or cheating

| | | |

1 1 : 2 : 3 : 4 : 5 5 29 i13.f4k1_T4_Do you usually feel that it’s almost useless to try in school because most other children are just plain smarter than you are?

| |

1 1 : 2 : 3 : 4 : 5 5 19 i03.f4k1_T6_All in all, I am inclined to feel that I am a failure. | |

| |

1 1 : 2 : 3 : 4 : 5 5 68 i52.f4k1_T4_Do you feel that most of the time it doesn’t pay to try hard because things never turn out right anyway? 1 1 : 2 : 3 : 4 : 5 5 34 i18.f4k1_T4_Are you often blamed for things that just aren’t your fault?

| | | |

1 1 : 2 : 3 : 4 : 5 5 39 i23.f4k1_T4_Do you feel that when a kid your age decides to hit you, there’s little you can do to stop him or her 1 1 : 2 : 3 : 4 : 5 5 64 i48.f4k1_T3_Other children or young people pick on me or bully me

1 1 : 2 : 3 : 4 : 5 5 84 i68.f4k1_T4_Do you usually feel that you have little to say about what you get to eat at home? 5 5 : 4 : 3 : 2 : 1 1 44 i28.f4k0_T6_On the whole, I am satisfied with myself.

5 5 : 4 : 3 : 2 : 1 1 76 i60.f4k0_T4_Most of the time, do you feel that you can change what might happen tomorrow by what you do today? | |

1 1 : 2 : 3 : 4 : 5 5 80 i64.f4k1_T7_I am filled with doubts about my competence.

1 1 : 2 : 3 : 4 : 5 5 72 i56.f4k1_T4_Do you feel that when you do something wrong there’s very little you can do to make it right? | |

| |

1 1 : 2 : 3 : 4 : 5 5 49 i33.f4k1_T4_Do you feel that one of the best ways to handle most problems is just not to think about them? 5 5 : 4 : 3 : 2 : 1 1 54 i38.f4k0_T7_I am confident I get the success I deserve in life.

5 5 : 4 : 3 : 2 : 1 1 59 i43.f4k0_T4_Do you feel that most of the time parents listen to what their children have to say?

|---+---+---+---+---+---+---| NUM ITEM -6 -4 -2 0 2 4 6 8

121223211

1 2245660376507687321

1 1 4 5 4109196696062789947337512

5 71 13757792232781117090465030741006413111 1 PERSON

T S M S T

0 10 30 60 80 90 99 PERCENTILE

(48)

F5 A

---

| 1 NONE |( -3.89) -INF -2.78| | 71% 2% 2.0034| | 1

| 2 -1.58 .01 | -1.60 -2.78 -.77| -2.24 | 35% 19% 1.2069| .76| 2 | 3 -.86 .01 | -.05 -.77 .70| -.76 | 39% 48% .6745| .93| 3 | 4 .64 .01 | 1.57 .70 2.84| .65 | 40% 72% .5392| 1.14| 4 | 5 1.80 .01 |( 4.01) 2.84 +INF | 2.34 | 78% 18% 1.0667| 1.10| 5 --- M->C = Does Measure imply Category? C->M = Does Category imply Measure? CATEGORY PROBABILITIES: MODES - Structure measures at intersections P -+---+---+---+---+---+---+---+---+---+---+- R 1.0 + + O | | B |1 |

A | 1111 5555|

B .8 + 111 555 +

I | 11 55 |

L | 11 55 |

I | 1 55 |

T .6 + 11 5 +

Y | 1 55 |

.5 + 11 5 +

O | 1 55 |

F .4 + 1 4444*4 + | 2222**222 33333333*44 5 4444 | R | 222 133*22 44 33 55 444 |

E | 222 3311 2244 *33 444 |

S .2 + 2222 33 1 4422 55 33 444 +

P | 2222 333 4** 2** 333 4444 |

O |22 333 444 11*55 222 333 4|

N | 3333333 44444 55555 1111 22222 333333 |

S .0 +*****************55555555 11111111******************+

E -+---+---+---+---+---+---+---+---+---+---+- -5 -4 -3 -2 -1 0 1 2 3 4 5

(49)

F5 A

EXPECTED SCORE: MEAN (Rasch-score-point threshold, ":" indicates Rasch-half-point threshold) (ILLUSTRATED BY AN OBSERVED CATEGORY) -7 -5 -3 -1 1 3 5 7

|---+---+---+---+---+---+---| NUM ITEM

1 1 : 2 : 3 : 4 : 5 5 50 i34.f5k1_T1_I trust in others | |

1 1 : 2 : 3 : 4 : 5 5 40 i24.f5k1_T1_I let other people use my things | |

| |

1 1 : 2 : 3 : 4 : 5 5 81 i65.f5k1_T3_I usually do as I am told

| |

1 1 : 2 : 3 : 4 : 5 5 30 i14.f5k1_T2_Has a forgiving nature | |

| |

5 5 : 4 : 3 : 2 : 1 1 85 i69.f5k0_T3_I get on better with adults than with people my own age | |

1 1 : 2 : 3 : 4 : 5 5 45 i29.f5k1_T2_Likes to cooperate with others

1 1 : 2 : 3 : 4 : 5 5 20 i04.f5k1_T2_Is considerate and kind to almost everyone 1 1 : 2 : 3 : 4 : 5 5 65 i49.f5k1_T2_Is helpful and unselfish with others

1 1 : 2 : 3 : 4 : 5 5 73 i57.f5k1_T3_I am helpful if someone is hurt, upset or feeling ill 1 1 : 2 : 3 : 4 : 5 5 25 i09.f5k1_T1_I treat my peers with love and warmth

1 1 : 2 : 3 : 4 : 5 5 35 i19.f5k1_T1_I behave with others with great kindness

1 1 : 2 : 3 : 4 : 5 5 55 i39.f5k1_T1_If a classmate has some difficulty, I help him/her | |

5 5 : 4 : 3 : 2 : 1 1 60 i44.f5k0_T3_I fight a lot. I can make other people do what I want 1 1 : 2 : 3 : 4 : 5 5 77 i61.f5k1_T3_Other people my age generally like me

5 5 : 4 : 3 : 2 : 1 1 69 i53.f5k0_T2_Starts quarrels with others

|---+---+---+---+---+---+---| NUM ITEM -7 -5 -3 -1 1 3 5 7

122232111

1338933045538578421 1

3 362071538638237306279239 4 1

1 11 21233563768621354700943042453370741324 3 5 29 PERSON

T S M S T

0 10 30 60 80 90 99 PERCENTILE

DIF analysis: is more difficult to endorse for kids of 5th grade (dif to desagree in – items)

(50)

F6 O

---

| 1 NONE |( -4.41) -INF -3.11| | 79% 1% 1.8041| | 1

| 2 -1.65 .01 | -1.74 -3.11 -.81| -2.45 | 35% 27% 1.0427| .85| 2 | 3 -.85 .01 | -.01 -.81 .80| -.76 | 36% 58% .5820| .96| 3 | 4 .81 .01 | 1.73 .80 3.12| .75 | 36% 55% .7208| 1.08| 4 | 5 1.68 .01 |( 4.43) 3.12 +INF | 2.47 | 81% 12% 1.2780| 1.08| 5 --- M->C = Does Measure imply Category? C->M = Does Category imply Measure? CATEGORY PROBABILITIES: MODES - Structure measures at intersections P -+---+---+---+---+---+---+- R 1.0 + + O | | B |111 555|

A | 111 555 |

B .8 + 111 555 +

I | 11 55 |

L | 11 55 |

I | 11 5 |

T .6 + 1 55 +

Y | 11 5 |

.5 + 1 55 +

O | 1 5 |

F .4 + 11 5 +

| 2222*22 3333333 444**444 | R | 2222 13**2 44*335 4444 |

E | 222 3311 22*4 5533 444 |

S .2 + 222 33 1 44 22 5 33 4444 +

P | 2222 333 4*1 5*2 333 444 |

O |2222 333 444 *** 222 333 4444|

N | 333333 44444 5555 1111 22222 333333 |

S .0 +******************5555555 1111111******************+

E -+---+---+---+---+---+---+- -6 -4 -2 0 2 4 6

(51)

F6 O

EXPECTED SCORE: MEAN (Rasch-score-point threshold, ":" indicates Rasch-half-point threshold) (ILLUSTRATED BY AN OBSERVED CATEGORY) -7 -5 -3 -1 1 3 5 7

|---+---+---+---+---+---+---| NUM ITEM

1 1 : 2 : 3 : 4 : 5 5 51 i35.f6k1_T8_New ideas and projects sometimes distract me from previous ones. 1 1 : 2 : 3 : 4 : 5 5 86 i70.f6k1_T1_I am able to create new games and entertainments

1 1 : 2 : 3 : 4 : 5 5 78 i62.f6k1_T1_I like scientific TV shows | |

1 1 : 2 : 3 : 4 : 5 5 36 i20.f6k1_T2_Is inventive | |

1 1 : 2 : 3 : 4 : 5 5 41 i25.f6k1_T2_Is sophisticated in art, music, or literature 1 1 : 2 : 3 : 4 : 5 5 56 i40.f6k1_T2_Values artistic, aesthetic experiences | |

1 1 : 2 : 3 : 4 : 5 5 21 i05.f6k1_T2_Is original, comes up with new ideas

5 5 : 4 : 3 : 2 : 1 1 90 i74.f6k0_T4_Do you believe that most kids are just born good at sports?

5 5 : 4 : 3 : 2 : 1 1 31 i15.f6k0_T4_Do you feel that when someone doesn’t like you there’s little you can do about it? | |

1 1 : 2 : 3 : 4 : 5 5 46 i30.f6k1_T2_Likes to reflect, play with ideas

5 5 : 4 : 3 : 2 : 1 1 66 i50.f6k0_T4_Most of the time, do you feel that you have little to say about what your family decides to do? 1 1 : 2 : 3 : 4 : 5 5 74 i58.f6k1_T1_I have a great deal of fantasy

1 1 : 2 : 3 : 4 : 5 5 82 i66.f6k1_T2_Is curious about many different things

1 1 : 2 : 3 : 4 : 5 5 70 i54.f6k1_T2_Is ingenious, a deep thinker 1 1 : 2 : 3 : 4 : 5 5 26 i10.f6k1_T2_Has an active imagination | |

| |

11 : 2 : 3 : 4 : 5 5 88 i72.f6k1_T1_I would like very much to travel and to know the habits of other countries 11 : 2 : 3 : 4 : 5 5 61 i45.f6k1_T1_I like to know and to learn new things

|---+---+---+---+---+---+---| NUM ITEM -7 -5 -3 -1 1 3 5 7

112232111

12478778221863975211

1 32471218863980293397218794221

12 3212151253927019842746241325753117305479 5182 4 PERSON T S M S T

0 10 30 60 80 90 99 PERCENTILE

DIF analysis: is more difficult to endorse for kids of 5th grade (dif to desagree in – items)

(52)

(53)

Preliminary descriptive statistics

• Grade and gender by Factors

• Item descriptive satistics

(54)

(55)

(56)

(57)

(58)

Questions

1. What do experts have to say about the interpretations of the underlying

constructs that each six dimensions have represented?

2. What are the particularities of viewing personality traits like those six

dimensions as competencies? Can we approach these dimensions as

competencies? Do we expect the same patterns of cumulative development

as in cognitive skills? Do both poles indicate some kind of competency?

3. Is some important construct missing?

4. What would be the limitations of self-reporting for the measuring of these

competencies? What would be the alternatives? What about other

approaches (objective personality tests, situated judgment tests)

1. Validity : “a test is valid for measuring a theoretical attribute if and only if

variation in the attribute causes variation in the measurement outcomes

through the response process that the test elicits” (Borsboom, &

(59)

Questions

5. What would be the suggestions for recoding the answers using the

anchoring vignettes?

6. What would be the feasible main purpose of the use of the proposed

instrument and their adequacy? Monitoring interventions? Describing

student’s strengths and weakness? Predicting outcomes?

7. Are the current instrument forms suitable for monitoring purposes?

8. In a longitudinal design, is it technically justified to assume

uni-dimensionality for each of the six scales measured over time, but at younger

ages by means of observer-report (teachers/parents) and later by

(60)

Questions

9. What are the pros and cons of presenting reports based on these

instruments to students? To family and teachers? To school directors? What

would be the content for these reports?

10. What are the expert views of the causal mechanisms (direct, indirect,

mediation, moderation) underlying the constructs that are being measured in

the proposed instrument and the important outcomes (education, learning,

the job market, health, civic engagement, safety, family, wellbeing, salary).

What would be a justifiable intervention plan (for instance is it justifiable to

plan interventions to foster consciousness directly or to plan interventions to

capitalize on the mediating/moderating process)

11. Can we consider the reports given to students as a kind of intervention in

(61)

Thank You!

(62)