Technology Literacy Assessment Project (TLAP)

Proposal to Colorado Department of Education
Power Results Grant Program

Narrative

Section A: Project Design

Background and Context

Current federal legislation (NCLB) decrees that by 2013 educators will obtain “technology literacy” (TL) for all students by the time these students complete the 8th grade. The U.S. Department of Education has mandated that states must report annually, beginning in 2008, on the percentage of students who have achieved TL by the end of the 8th grade[1] — leaving interpretation and implementation of this requirement to the discretion of each state. To comply with the federal mandate, the Colorado Department of Education has ordered school districts to report by fall 2008 the percentage of their students who have achieved 8th grade TL, declaring a common definition and standards for TL but ceding to districts the authority (and responsibility) for determining the proficiency levels that indicate attainment of TL and for devising the mechanism for determining attainment of said literacy. (See Appendix C — CDE Statement of Technology Literacy Reporting Requirement.)

 

In response to these federal and state expectations, Colorado school districts have been scrambling to create/acquire and conduct TL assessments. The result is a potpourri of assessment methodologies, ranging from district-developed performance assessments to standards-based report cards.[2] Definitions of TL diverge, but most districts rely, directly or implicitly, on the 2007 ISTE-NETS technology literacy standards.[3] Among the various assessments in use is a limited-response (aka multiple choice) exam developed during 2007 by the districts involved in this proposal and offered without charge to other districts across the state. Significantly, this exam does not address the three most Information-Age-oriented of the six CDE-endorsed standards (Creativity & Innovation; Communication & Collaboration; Critical Thinking, Problem Solving, & Decision-Making).

 

Now, Colorado Power Results grant funds create the opportunity for the state’s educators to shift from muddled NCLB compliance to transformative support for 21st century learning. This proposal presents a multi-year project designed to achieve that transformation.

Proposed Project Outcomes

The project outlined in this proposal, collaboratively conducted by a consortium of school districts and BOCES[4], would accomplish the following:

·        Determine a common definition, standards, and proficiency criteria for TL;

·        Develop or adapt an assessment instrument that addresses the determined standards and proficiency criteria;

·        Develop common procedures for delivery of the assessment that will apply to all districts that use the assessment instrument (“participating districts”);

·        Establish assessment results reporting mechanisms for all participating districts;

·        Facilitate development of assessment analysis practices and application of assessment analyses to improved curricula and instructional design;

·        Work to incorporate assessment and acquisition of TL into Colorado content standards and instructional practice.

 

The project builds on the experience of consortial partners and other districts in devising and implementing a limited-response assessment that addresses three of the six CDE-endorsed TL standards (Research & Information Fluency, Digital Citizenship, and Technology Operations & Concepts). Investing at least $120,000 in staff time, assessment design consultants[5], technology, and other resources, these districts collaborated from September 2007 through May 2008 to create an assessment that would be easy to administer and would provide reliable results to inform instructional planning as well as fulfill NCLB and state reporting requirements. (See Appendix D — Existing Consortial Technology Literacy Assessment Project.) While that work provides a solid foundation for the proposed project, much more remains to be accomplished in creating and using a comprehensive TL assessment. Additional work includes:

·        Benchmarking the CDE-endorsed TL standards to grade levels;

·        Developing an assessment for the more complex and sophisticated elements of TL (Creativity & Innovation; Communication & Collaboration; Critical Thinking, Problem Solving, & Decision-Making)

·        Developing an assessment that emphasizes performance-based demonstrations of student proficiency;

·        Making the assessment instrument available in formats that maximize usability for all interested school districts, with a preference for online delivery;

·        Establishing a data reporting system and procedures;

·        Designing and delivering professional learning related to implementation of a TL assessment and the use of assessment data to improve instruction;

·        Incorporating TL assessment into instructional design and practice.

 

The additional work outlined above comprises the core of this proposal. Further explanation and elaboration of the intended project outcomes are provided in the paragraphs that follow.

 

Common definition, standards, and proficiency criteria — In order to assess (and report) TL, Colorado educators must first agree on what it is. Despite CDE’s endorsement of a single definition and set of standards, definitions and standards used by districts across the state do not necessarily conform to the CDE statement. This project would help to clarify such ambiguities.

 

The 2007 ISTE-NETS standards, which CDE modified and endorsed in August 2007, provide valuable common ground for linking technology and information literacies[6], re-orienting the conceptualization of these literacies as instrumental rather than terminal.[7] Significantly, these standards reflect the growing national agreement that technology is a means rather than an end, and, correlatively, that TL should be understood as a way to support learning and working rather than an isolated accumulation of skills and knowledge.[8] The six CDE-endorsed and ISTE-NETS categories (Creativity & Innovation; Communication & Collaboration; Research & Information Fluency; Critical Thinking, Problem Solving, & Decision-Making; Digital Citizenship; and Technology Operations & Concepts) reflect this growing agreement, and portend a shift toward Information Age learning.

 

Accordingly, guiding principles for a common understanding of TL include:

·        Emphasis on information and communication skills;

·        Emphasis on application of tools to learning and productivity outcomes;

·        Connection of TL to student achievement in content areas;

·        Emphasis on broad tool-use and problem-solving strategies rather than narrow mastery of particular technologies;

·        Focus on enduring skills, understandings, and dispositions that transcend short- and medium-term technology developments.

 

Significantly, the CDE-endorsed modification of the ISTE-NETS standards adds elements of technology literacy not addressed by the national standards — that of “design-build” technologies and processes (e.g., the creation of solutions, process, design, and products).[9] This addition adds an important element to TL that presents interesting performance-based assessment design challenges.

 

An assessment instrument — The project will develop or adapt a TL assessment that addresses all six state-endorsed TL standards and follows the guiding principles described above. The developmental process will include review and, potentially, adaptation of assessment instruments already developed (or in the process of development) by other organizations with proven assessment expertise. To the extent practicable, the assessment instrument will be performance-based. The assessment will be available free to all Colorado school districts. Of course, districts may use this instrument or create their own. In order to assure common ground and meaningful application of assessment results, districts that use the assessment will align their TL standards and proficiency criteria with those determined through the project.

 

Experience gained through the consortial work already conducted member districts of this project, in creating the existing 3-standard TL assessment, lays the groundwork for the extensive validity and reliability studies that must be conducted to create an instrument that produces usable results. Assessment staff within the districts guided such studies on the existing assessment; that work included analysis of questions for bias and age-appropriate language, placement and order of questions, misleading answers, non-indicative answers, and test-delivery of the assessment to representative populations (considering not only demographic, language, and ability diversity but also anticipated technology literacies of the populations to be assessed). Through this process the districts’ assessment staff acquired deeper understanding of the particular challenges presented by TL assessments, and their increasing expertise will guide the proposed project. In addition, grant funds make it possible to consult with other assessment design experts who have particular knowledge in this area. (See Appendix D — Existing Consortial Technology Literacy Assessment Project.)

 

The project will also explore the value and feasibility of assessing students in the 4th and 8th grades — the earlier assessment serving a formative function that helps districts design and adapt curricular and instructional plans.

 

Common procedures for delivery  — Although district flexibility in timing and manner of implementation will be a priority, core implementation procedures will be standardized in order to obtain reliable, usable results. Development of these procedures will seek to minimize the burden of implementation on Colorado educators.

 

The project-produced assessment will be available for online delivery (using a single platform, preferably one, like Moodle[10], that is free or inexpensive), in order to make reporting of results more efficient and uniform. Variations in the mode of delivery will accommodate districts' divergent technology capacities. Alternatives to online delivery (e.g., making some or all parts of the assessment available in non-digital environments, or facilitating administration by individual districts through their LANs, with uploaded reports following implementation) will be explored to ensure maximum feasible accommodation.

 

Reporting mechanisms — A primary value in conducting a common assessment is that it facilitates extensive analysis of the results — including, for example, demographic and item analysis of assessment results. Online delivery of the assessment will enable automated and uniform reporting of results, which is one reason online delivery will be the primary method of implementation. Whatever the assessment delivery modality, the project will develop common reporting procedures, tools, and platforms to ensure that all results reported from districts that use the project-developed assessment are consistent, and can be collectively and meaningfully analyzed. The project consortium will coordinate with CDE to establish common proficiency ratings, reporting procedures, and a state database that makes assessment results easily and promptly available to all education stakeholders. Individual student identities will be protected in the same manner as they are in CSAP administration.

 

Assessment analysis practices — The primary application of TL assessments should be to enable Colorado educators to improve learning and curriculum design — not just with respect to students’ acquisition of TL but in content-area learning as well. Professional learning related to the assessments will emphasize analysis of assessment results (e.g., use of data-driven dialogue) that guides teaching and learning practices.

 

Work to incorporate assessment and acquisition of TL into Colorado content standards and instructional practice — Ongoing dissemination efforts throughout the project will seek opportunities within individual districts and across the state to tie TL into content-area teaching and learning. These efforts will focus in particular on initiatives to revise the state’s content standards, improve students’ preparedness for postsecondary options, and adopt 21st century learning.

Statewide Availability

All products developed through this project will be available free to all school districts in Colorado, although, of course, use of the assessment by districts not participating directly in the project will be optional. Delivery methodologies will adapt to the range of districts’ TL development and technology infrastructures in order to meet the needs of underserved districts. In addition, professional development — via workshops, webinars, and other mechanisms — will be available statewide. The professional development plan calls for four to eight regional workshops in order to enhance accessibility for all districts.

 

The member districts of the project consortium have more than 150 high need schools, based on the CDE criteria. Full-scale use of the assessment, of course, will apply to all of these schools. In addition, pilot assessment efforts will focus on high need schools in order to ensure that the assessment serves not only their capacities in delivering the assessment but their needs in using the assessment to improve student learning.

Research Support for the Proposal.

The Technology Literacy consortium has conducted its own research, in the context of its existing work creating a TL assessment; and project work will apply these findings, while continuing to pursue additional research. In addition, research nationwide on TL and other related assessments, ranging across age cohorts and subject areas, have demonstrated (1) the viability and value of performance-based assessments for measuring the skills and knowledge described by the CDE-endorsed TL standards, and (2) the feasibility to deliver such assessments online (and obtain reports of the assessment results). Significant examples of these efforts include:

·        Student Tool for Technology Literacy — Florida Department of Education — assesses the six ISTE/NETS TL standards[11];

·        iSkills — Educational Testing Service — assess the Information and Communication Technology (ICT) literacy of students seeking admission to four-year colleges[12];

·        College Learning Assessment — Council for Aid to Education — assesses information, communication, and problem-solving skills of four-year college students[13]

·        TechYES — Generation Yes — assesses the six ISTE/NETS TL standards through student projects[14];

·        Learning.com — may assess the six ISTE/NETS TL standards (though it appears to emphasize technical operations), primarily through a limited response test but partly through simulations[15]

 

All of the assessments listed here have substantial research support, undertaken by the organizations creating the assessments. This research informs the proposed TL assessment project, especially since the organizations involved, along with others, will be sought as partners in the project. The project’s assessment development plan makes examination of  other instruments, especially Florida’s, a high initial priority. Adaptation of an instrument already developed and analyzed would move the project significantly forward.

 

In addition, the project is informed by the research already conducted by the work of the consortial-member districts that have created the existing 3-standard TL assessment. That research included extensive validity and reliability studies, conducted under the guidance of those districts’ assessment directors. (See Appendix D — Existing Consortial Technology Literacy Assessment Project.)


 

 



[1] For the sake of brevity and simplicity, further references to the expectation that students will attain technology literacy by the end of the 8th grade will be stated as “8th grade TL”. In fact, the imagined literacy could be attained by a student (and assessed) at any time prior to the completion of the 8th grade.

[2] A sample of district assessment strategies is shown in Table 1 at the end of this document.

[4] Consortium membership includes Centennial BOCES, Denver Public Schools, Jeffco Public Schools, St. Vrain Valley School District, and Widefield School District.

[5] The consortium’s primary consultant was Bernajean Porter, a nationally recognized expert in technology literacy and assessment design. See www.bjpconsulting.com/index.html

[6] The integration of technology and information literacies was initiated by CDE’s Educational Technology and Information Literacy (ET-IL) initiative (begun in 2002). Integration is also reflected in the more common national (and international) label of “ICT” (Information and Communications Technology) literacy. (See the ICT Digital Literacy webpage, http://www.ictliteracy.info/, and the Educational Testing Service’s iSkills test webpage, www.ets.org/portal/site/ets/menuitem.1488512ecfd5b8849a77b13bc3921509/?vgnextoid=1ebb0e3c27a85110VgnVCM10000022f95190RCRD&vgnextchannel=34b4a79898a85110VgnVCM10000022f95190RCRD.

[7] http://www.cde.state.co.us/edtech/StandardsForStudents.htm

[8] See “Tests of Tech Literacy Still Not Widespread Despite NCLB Goals,” in Education Week, 1-30-08, pp. 1, 12; www.edweek.org/ew/articles/2008/01/30/21techtests.h27.html (accessed 6-8-08).

[9] See Colorado’s Technology Literacy Standard IV-B: “Students use critical thinking skills to plan and conduct research, design and manage projects, solve problems, engineer solutions and make informed decisions using appropriate digital tools and resources. Students: … (B) plan, design and manage activities to develop a solution or complete a project (italics shows text added to original ISTE-NETS standard by CDE technology literacy standards committee).

[10] The Moodle platform is used here as an example only, not a recommendation.

[11] See www.flinnovates.org/sttl/default.htm (accessed 6-8-08) for a description of the Florida assessment project. Kate Kemker, the Florida Department of Education Bureau Chief for Interactive Media, reports (in e-mail correspondence with the applicants for this grant) that the state recently completed pilot studies of the assessment and plans statewide implementation in Fall 2008. According to Len Scrogan, Director of Educational Technology for Boulder Valley School District, the Florida assessment project was funded by a $2.5-million grant from the National Science Foundation.

[12] See www.ets.org/portal/site/ets/menuitem.1488512ecfd5b8849a77b13bc3921509/?vgnextoid=159f0e3c27a85110VgnVCM10000022f95190RCRD&vgnextchannel=e5b2a79898a85110VgnVCM10000022f95190RCRD

[13] According the Generation Yes promotional material, “TechYES encourages all students to complete technology projects that are creative and personally involving. The projects can also meet requirements for core curriculum classes or community service. These projects are the basis for the TechYES evaluation and certification.” See www.cae.org/content/pro_collegiate.htm.

[15] Publicity materials from Learning.com claim, “When taking the online test, students interact with assessment content in ways that allow them to demonstrate their proficiencies. Often, they must perform actions via simulations, rather than pick answers from among multiple choices. Thus, students must be able to format a paragraph, apply a spreadsheet formula, or conduct a database search. And they must demonstrate durable skills via generic menus and commands, not through brand-specific memorized shortcuts.” See Arizona Pioneers Statewide Measurement of Students’ Technology Literacy Skills, from Learning.Com, http://www.learning.com/casestudies/arizona.htm (accessed 6-7-08).