Technology Literacy Assessment Project (TLAP)

Proposal to Colorado Department of Education
Power Results Grant Program

Appendix D — Existing Consortial Technology Literacy Assessment Project

 

In order to comply with NCLB reporting requirements and to inform their technology, information, and content instructional designs, a consortium of school districts formed in September 2007 to create a limited-response (aka multiple choice) technology literacy assessment for administering in the districts’ schools in spring 2008. The consortium invested at least $120,000 in staff time, consultant expenses[1], and technology and other resources. The 36-item assessment addresses three of the six CDE-endorsed TL standards (Research & Information Fluency, Digital Citizenship, and Technology Operations & Concepts). It can be administered online or in paper-pencil format. The assessment instrument also contains a brief survey of student demographics and technology use patterns, which the districts use in analyzing the literacy test results.

 

Assessment directors and staff from participating districts participated actively in the design and development of the instrument. Significantly, these assessment experts guided validity and reliability studies of the questions, including evaluation of the language, context, grade-level appropriateness, readability, and prediction value. The evaluation included beta and pilot testing of the instrument with students in 7th and 9th grades in order to measure predictability before and after the target 8th grade delivery.

 

Some of the consortium members partnered with Edusoft for delivery and scoring/reporting of the assessment. One consortium member utilized Blackboard for online delivery and scoring. The consortium experimented with both online and paper-pencil delivery of the assessment. In the paper-pencil format, Scantron sheets were used, and assessment results were scanned to the Edusoft platform.

 

The consortium has made the assessment available at no charge to any district in the state that requests it. Approximately 15 non-consortium school districts have taken advantage of this offer. A sample of these districts is shown in Appendix A — Current District Efforts to Assess Technology Literacy.

 

The timeline for the consortium’s work in developing, validating, and implementing the assessment is shown in the table on the next page.


Timeline of Assessment Development — 2007-08

Stages of Process

Calendar

Details

Initial meeting

September 2007

·         Determine definition of Technology Literacy proficiency

·         Decide Purpose for assessment

·         Identified needs

·         Discuss Design, Delivery, and function of assessment

·         Identify what will actually be assessed?

·         What standards will be assessed?

·         What are the critical skills/concepts within these standards?

·         Begin to develop Test Bank of questions

o Look at what other districts have done

o Begin to cull questions from other resources

·         What will we do with results?

Development Work

October 2007

·         Develop framework for assessment

·         Re-write and tweak questions

·         Rank questions in order of difficulty

·         Format the layout of questions

·         Delivery Options

·         Look at language of questions

·         Get feedback from districts

·         Define a process for validation of assessment

·         Set timelines/deadlines

Planning Meeting

November 2007

 

·         Questions reviewed in category format

·         Questions/Answers reviewed for readability

·         Decisions made about number of questions and types to be included

Think-Aloud Review

November 2007

·         Questions organized by standard

·         Details provided on implementation process

·         Conducted “Think-Alouds” with individual 8th grade students

o One male

o One female

o ELL or disability

·         Modifications according to feedback

Planning Meeting

December 2007

·         Information and recommendations from Think Aloud reviewed and applied

·         Core Test designed

·         Planning for Beta and Pilot delivery and scoring — how to collect and pull information together.

Development Work

December 2007

·         Questions assembled, and order determined for CORE assessment

·         Questions to correlated standards, benchmarks, concepts

·         Excel spreadsheet template created for recording assessment data

Beta Assessment Instrument

 

January 2008

·         Beta assessment given to five 9th grade students

·         Beta purpose-

o Determine how long to take the assessment

o Determine level of difficulty

o Determine effectiveness of process

Planning Meeting

January 2008

·         Revise test based on Beta results

·         Build in accommodations

·         Plan for the sharing of the assessment with other districts

·         Initial draft of  Performance indicators of Tech Literacy: Advanced, Proficient, Partially Proficient, or unsatisfactory)

·         Reference CDE and Office of Learning documents with NETS Performance Indicators 

·         Develop process for socializing the assessment within participating districts- Peer sharing

·         Add survey questions to assessment

Pilot Assessment

February 2008

·         Distribution of Scantron sheets

·         Selection of demographically representative classrooms within each district to pilot assessment

·         Conduct pilot assessments; report data

Group Meeting

March 2008

 

·         Complete final assessment instrument

·         Determine proficiency cut points

·         Finalize performance descriptors

·         Plan for delivery of assessment

·         Finalize support documents

o Cover Letter

o Talking Points

o Directions for Administration

o Proficiency framework

·         Distribute assessment instrument to participating districts after March 10th

Full Administration

April-May 2008

Participating districts conduct assessments

Group Meeting

June 2008

 

·         Item Analysis of assessment results

·         Consideration of next steps

 

The sample of assessment items presented on the next page addresses only one of the three standards assessed. It is provided more to show the nature of the instrument and question/answer design than to demonstrate the range of the assessment’s coverage. A complete version of the assessment and arrangements for online or paper-pencil delivery may be obtained by contacting representatives of the districts that have participated in its creation.

 

1. Plagiarism is _____________ . A. the act of copying other people’s ideas or writing without giving credit to the author B. the act of linking to other people’s work as part of a blog C. the act of sending unwanted instant messages or e-mails to many people at the same time

 

2. Which of the following personal information is safe to share on a web site such as My Space? A. Age and home town B. Name and school C. Screen name only

 

3. A student locates an essay on the Internet, copies it, and submits it to the teacher as his or her own work. A. This is okay because the information was online and found while at school. B. This is not okay because it is using someone else’s work as your own. C. This is okay because everyone does it.

 

4. Which of the following is legal when you buy one license of a software program? A. You can burn copies of the software and give to all your friends B. You can load the software on any number of computers.

C. You can only load the software on one computer.

 

5. Domingo was setting up an account on MySpace and needed to establish a password. Of the choices, which password would be the most secure? A. An unrelated word such as…Drag0Nfl1 B. A short version of his name…mingo C. His name with birth year…domingo1991

 

6. You found a MySpace page that had pictures and very negative comments about you that was posted by a classmate at school. What should you do? A. Call the classmate and make threatening remarks. B. Create a page about the classmate that is just as bad. C. Inform an adult at home or school.

 

7. A student opened a teacher’s electronic gradebook and changed grades without the teacher knowing. This act is ____________________ A. against school rules, but is legal. B. illegal and may have consequences from the school and police. C. not illegal.

 



[1] The consortium’s primary consultant was Bernajean Porter, a nationally recognized expert in technology literacy and assessment design. See www.bjpconsulting.com/index.html