Technology Literacy Assessment Project (TLAP)

Proposal to Colorado Department of Education
Power Results Grant Program

Section C: Management Plan                                                                                                     .

Project Implementation and Activities

In keeping with the restrictions of the RFP, this proposal seeks funding for only one year, although a thorough TL project will ultimately require more than a year. Even with the experience obtained by the project consortium in developing a three-standard assessment during 2007-08 (see Appendix D), it is not possible to develop and implement a comprehensive TL assessment within a year, much less to meet the larger needs the assessment is intended to address — including analysis of assessment results, incorporation of TL into content learning, development and implementation of effective TL teaching and learning, and refinement (and updating) of the assessment instruments. (The multi-year nature of the project is further addressed in the section on Sustainability.) The project timeline presented in this section encompasses more than one year of activity in order to help grant readers and CDE staff frame this proposal in a larger context. But only the “Project Year One” section of the timeline applies to the specific funding request proposed in this application.


As the timeline indicates, a one-year project can reasonably expect to produce and conduct a pilot assessment. The experience and results obtained through the first year can inform full development and implementation of a comprehensive TL assessment program as well as use of TL assessment results (along with analyses of those results) to improved technology, information, and content instructional design.


The applicants plan to submit a Year Two proposal to CDE in spring 2009 to obtain funding that will enable this project to be more fully completed, including a comprehensive TL assessment instrument and procedure that has been tested through pilot implementation and can be made available to all Colorado school districts by April 2010.




Activity Dates


Prior  Work

Collaborative development of a limited-response assessment on three CDE-endorsed standards

September 2007 – May 2008

·         Form collaborative partnership among several districts (See Appendix D)

·         Devise assessment questions

·         Analyze questions for validity and reliability

·         Adapt assessment for diverse students

·         Conduct and analyze pilot assessment

·         Conduct and analyze complete assessment

·         Analyze assessment results





Activity Dates


Project Year One —- Only year for which funding sought in this proposal

Strategic and logistical arrangements

July-August 2008

·         Constitute project executive board

·         Select project manager

·         Select project support staff

·         Finalize project first-year plan and timeline

·         Finalize first-year project budget

·         Contract with consultant specialists for assessment design and analysis

·         Invite private school participation

·         Review and refine evaluation plan

Needs analysis

August-October 2008

·         Conduct preliminary examination of school districts’ TL assessment needs

·         Conduct preliminary examination of districts’ capacities for implementing TL assessment

Instrument development for pilot assessment

September 2008 - January 2009

·         Design/Develop instruments for TL assessment

·         Conduct validity and reliability studies on assessment elements

Procedure development  for pilot assessment

November 2008 - March 2009

·         Determine options and steps for assessment delivery

·         Develop or acquire platform(s) and methods for reporting assessment results

·         Draft manuals and guides

Window for conducting pilot assessment

April-May 2009

Only applies to project years. In subsequent years districts will conduct assessment annually but at times of their own choosing.

Professional learning; Dissemination

January-June 2009

·         Create webpage, webinars, video, and workshop presentations

·         Provide information regarding the pilot assessment

·         Provide learning in assessment analysis and incorporation in instructional design

Analysis of pilot assessment  results

May-June 2009

·         Analyze pilot assessment data

·         Analyze testing instrument and process (including validity and reliability confirmation)

Project completion

May-June 2009

·         Report analysis of pilot assessment

·         Report analyses of data produced by assessment

·         Finalize technical reports, administrative guide, and scoring guide

·         Finalize dissemination and professional learning plan

·         Final report and recommendations to CDE





Activity Dates


 (Tentative) Project Year Two — if funding available

Strategic and logistical arrangements

July-August 2009

·         Re-constitute project executive board

·         Re-constitute project staff and consultants

·         Finalize second-year project plan and timeline

·         Finalize second-year project budget

Needs analysis

August-October 2009

·         Conduct comprehensive examination of TL assessment needs of school districts

·         Conduct comprehensive examination of districts’ capacities for implementing TL assessment

·         Conduct preliminary examination of curricular and instructional implications from pilot assessment

·         Review evaluation of Year One project

·         Review findings from pilot assessment

Instrument development for assessment

November 2009 - March 2010

·         Design/Develop instruments for TL assessment

·         Pilot TL assessment at selected sites in representative districts

Procedure development  for assessment

February 2010

·         Determine options and steps for implementation

·         Develop or acquire platform(s) and methods for reporting assessment results

·         Develop any PD resources needed for implantation

Window for conducting assessment

April 2010

·         In subsequent years, districts will conduct assessment annually but at times of their own choosing.

Curriculum development

September 2009 - May 2010

·         Develop TL curriculum scope and sequence based on pilot assessment results

·         Develop strategies for integrating TL curriculum with content-area curriculum

Analysis of assessment  results

May-June 2010

·         Conduct professional learning in analysis and application of assessment results to support student achievement and development of 21st century learning.

Project completion

May-June 2010

·         Final report and recommendations to CDE

·         Make plans for project follow-up by participating districts

Professional Learning and Dissemination

Effective use of the TL assessment depends as much on professional learning connected to the assessment as research-based design of the assessment instrument. Professional learning conducted by the TL project will go far beyond how to administer the instrument and report the resulting data. The more important professional learning will examine how to make meaning of the assessment results (e.g., data dialogue and analysis workshops) and how to incorporate assessment results into improving curricula and instruction.


The project plans to work with the Colorado Consortium for Data-Driven Decision (C2D3) to design and implement four to eight regionally located assessment analysis workshops as well as online professional development opportunities (including webinars and courses). Using the specific assessment products and results created through the project, these workshops and online services will enable educators to learn how to examine assessment results and to apply assessment (and results) to instructional design. The professional development sessions will examine specific items and assessment results, applying the learning concretely and adapting it especially well to performance-based assessment. This professional learning builds directly on the four years of work already performed by C2D3, through which that organization has gained a national reputation for insight and high-quality professional learning.


The project’s webpage, in addition to hosting the online professional development opportunities, will also provide extensive information and other resources related to delivery and use of TL assessment in general (not just the project’s instrument).


Project participants — staff, leadership, technical committees, and consultants — will also engage in professional learning in order to deepen and extend their work. This professional learning will focus primarily on the design and development of the assessment instrument and on analyzing and applying assessment results. The learning design will target the specific work in which project participants are engaged, building knowledge from a contextual base. The learning experiences will emphasize onsite coaching rather than offsite workshops.


All professional learning opportunities, including the workshops and online presentations, will be available to educators across the state, regardless of whether the educators’ districts are members of the TL Consortium or use the TL instrument created through this project. In developing both the professional development opportunities and the webpage, the project plans to work with districts that have already created (or will create) their own assessments — with particular interest in those assessments that are performance-based. This collaboration will extend the reach and value of the professional learning, as well as help to coordinate assessment efforts statewide and enhance sustainability.

Technology Infrastructure

The project will evaluate methods and modes of online delivery of the developed assessment. These may include either consortially developed systems, or partnering with either commercial, or non-commercial providers. Priorities will be to provide a web based delivery, scoring, and reporting system that can be used by all schools and districts. For districts without the infrastructure to adequately support an online delivery, a paper and pencil version of the assessment will be developed.


Technology considerations will include bandwidth utilization, user friendly interface, security of data and communication, ease of integration and management, and the ability to meet diverse technical needs and specifications of districts. The online system will ideally be web based, with some server supported options included. Current programming standards will be incorporated into the development of the system, which will assist in optimal performance, user interaction and feedback, and data warehousing and reporting.


The consortium will build on its expertise and success in using online delivery systems piloted during 2007-08. Although these systems were commercial products, delivery and reporting system was highly successful, and much insight was gained in the use of this system. The consortium's evaluation data of piloted systems will be incorporated into the development of the new online system.  


Reported experience in the Florida assessment project is instructive. Among other lessons, Florida created significant elements of the instrument using Flash software to minimize connectivity and infrastructure demands on districts in an online delivery.

Project Participation and Leadership

TLC — The Technology Literacy Project Consortium — Initial consortium participants include Widefield School District, St. Vrain Valley School District, Jeffco Public Schools, Denver Public Schools, and Centennial BOCES (and member school districts). Additional partners include Northeast BOCES, Thompson School District, Lewis-Palmer School District, and Aurora School District. Centennial BOCES will serve as the consortium’s fiscal agent. Initial consortium districts include more than 150 high need schools (according to CDE criteria) and a diverse range of district characteristics (ranging from 15 rural districts to the keystone urban district of Denver). The additional partners extend that diversity. Other districts have expressed substantial interest in joining the project; the list of participating districts is expected to expand significantly before the end of summer 2008.


Private schools within the boundaries of the participating districts will be invited to have meaningful participation in the project — including use of the produced assessment instrument and access to all professional learning related to the use of the assessment, as well as participation in the consortium’s advisory council and technical committees — in the first month of the project, as part of the project’s initial strategic and logistical arrangements. The invitation will be communicated via e-mails to these schools and a video conference arranged and hosted by the project. All TLC districts have already committed, through Title II-D fiscal agreements with CDE, to provide opportunities for private schools within their boundaries to share in II-D funds and programs.


Executive Board — A seven-member board, meeting monthly, will make policy related to development and implementation of the assessments, and maintain oversight of all project activities. Individual board members, selected from participating LEAs in the project consortium, will also have specific policy, implementation, and oversight responsibilities related to various aspects of the project. Board membership will reflect diversity in the location and nature of participating consortium districts. Board members or their districts will be compensated by the project for their service.


Advisory Council — Consortium districts or BOCES that are not represented on the Executive Board will have representation on the Advisory Council, which will meet quarterly for general oversight of the project and to facilitate development and dissemination. Council participation will be provided as in-kind support by consortium members.


Technical Committees — The project’s work will be supported, in an advisory capacity, by staff members from various consortium districts who have expertise in particular areas related to the development and implementation of the assessment. Such committees have proven to be invaluable in the existing consortial TL assessment project (see Appendix D — Existing Consortial Technology Literacy Assessment Project). Technical committee participation will be provided as in-kind support by consortium members. Technical committees will include:

·        Assessment Development — dealing with issues related to the nature and quality of the assessment instrument (e.g., validity and reliability);

·        Platform Development —issues related to the nature and quality of the online platform and alternative delivery modalities (e.g., infrastructure requirements);

·        Assessment Analysis — examination and use of assessment results to improve curriculum and instruction;

·        Curriculum Development — curriculum that supports TL and integrates TL into content-area learning;

·        Professional Learning enhancing the ability of educators to use assessment results and implement TL-related curricula.


These committees will meet every two months; a member of the Executive Board will serve on each committee. Between committee meetings, individual members of committees will work with project staff and consultants.


Project Director (.5 FTE) — Selected by and reporting to the Executive Board, this person will provide daily policy direction, work with the Executive Board to refine the project vision, translate the project vision into strategic and action plans, supervise all project consultants and staff, set and monitor the project budget (with Executive Board supervision), and liaison the project with other organizations and initiatives. This person’s authority and policy direction will come through the Executive Board.


Project Manager (PM - .5 FTE) — Selected by the Project Director (with Executive Board approval), this person will coordinate all project activities (e.g., specific staff and consultant tasks; timelines; budget management; accounts management; resource management; procurement). The PM will be an FTE employee in one of the consortium partners, with the person’s FTE assignment to the project specified. To the extent of the specified FTE assignment, the PM will report to, and be supervised by, the Project Director.


Project Office and Fiscal Support (.5 FTE) — Through the Project Director, the project will employ a person who will be assigned specific project support roles (e.g., budget and accounting, accounts payable, clerical, and administrative support services). This staff person will be an FTE employees in one of the consortium partners, with a specified .5 FTE assignment to the project. To the extent of the specified FTE assignment, staff will report to, and be supervised by, the Project Director.


Consultant Specialists — The consortium will contract with consultants who specialize in various areas of the project’s work. These consultants will provide the primary expertise and development leadership related to project activities. Although some of the work for which consultants will be hired could be done by staff from one or more of the consortium members, the use of consultants is preferred so that they can focus on the project work and avoid the distraction of in-house business (or politics). Particular areas of consultant work include:

·        Assessment design and development

·        Assessment analysis and application

·        Research into promising practices, instruments, process and progress in various district or states, platforms, incorporation into broader standards, and

·        Technical developers (e.g., assessment delivery and reporting infrastructure);

·        Professional learning design and implementation

·        Curriculum and instructional design

·        Dissemination and communication

·        Project evaluation


The project consortium will actively seek to work with other organizations that have particular expertise in the areas of assessment, technology (and information) literacy, curriculum and instructional design, and/or professional development. Such organizations may include (without limitation):

·        The Center to Transforming Teaching and Learning (CTLT, formerly C2D3 — Colorado Consortium for Data-Driven Decisions) —

·        Council on 21st Century Learning —

·        Partnership for 21st Century Skills —

·        Denver Area School Superintendents Council (DASSC) — assessment task force

·        International Society for Technology in Education (ISTE) —

·        Technology Leadership Forum (TLF) —

·        Colorado Association of Leaders in Educational Technology (CALET) —

·        Colorado Association of School Executives (CASE) —

·        Colorado Association of School Boards (CASB) —

·        Colorado Association of School Libraries (CASL) —


Important potential partners include states or education-service organizations — e.g., the Florida Department of Education (FLDOE) and The Institute for Library and Information Literacy Education, at Kent State University[1] — that have already developed, or are in the process of developing, TL assessments. The work in Florida is especially important because the FLDOE, with substantial financial support from the federal government has created an online-delivered, performance-based, individualized assessment instrument that addresses all six of the ISTE/NETS standards.[2]


Also worth considering are vendors (e.g., InfoSource Learning and that want to adapt their assessments according to Colorado criteria and/or can offer a significant cost-savings in assessment development and continuation. (See Appendix B — Assessment Developments Elsewhere.) In addition, several information literacy or thinking skills assessments have been developed by postsecondary educational organizations (e.g., the Educational Testing Service[3] and the Council for Aid to Education[4]). These organizations, which generally indicate interest in aligning their assessments with (or adapting them to) K-12 education standards (and students), could help incorporate technology literacy work into content area standards and learning.


Consortium members have, for example, already conducted exploratory conversations with one potential partner, ALTEC, The Advanced Learning Technologies project at the University of Kansas Center for Research on Learning.[5] With federal funding ALTEC has created several assessment tools, including the state’s standards-based assessments of K-12 student progress, and delivers these online. Their experience demonstrates the feasibility of online delivery and reporting of assessments and could be invaluable to this project.


In addition, the consortium will actively seek to work with CDE, the several CDE- and legislatively-created committees, and the other state policymakers and education stakeholders. In particular, the consortium will work with these organizations to incorporate students’ acquisition of TL into content area teaching and learning.


The consortium partners have already demonstrated their commitment to sustaining this project in the work they have undertaken during 2007-08 to create a limited-response TL assessment (see Appendix D). In addition, these districts have committed substantial funds (through Title II-D and other sources) to professional learning dedicated to professional learning in use of technology to improve student achievement and build 21st century skills. Denver Public Schools, for example, provides year-round professional development in these areas, applying more than $300,000 in Title II-D and mill levy funds.


Although the project will require two years to develop and implement a statewide TL assessment, even this work will not be sufficient without a long-term state commitment. This commitment must include:

·        Continuing to provide a TL assessment without charge to all districts that want to use it;

·        Continuing to update and revise the TL assessment (e.g., writing new assessment items);

·        Continuing to maintain and update a platform and database for delivering the assessment and recording, storing, reporting, and distributing assessment results.

·        Ongoing professional learning in analysis and use of assessments;

·        Integration of TL learning with content-area standards and learning.

ET-IL Alignment

Since all school districts in the state are required to report technology literacy status of their students completing the 8th grade, TL assessment is a given in all districts’ ET-IL plans. More importantly, the current understanding of TL, encompassing not only the combination of technology with information literacy but the application of both in the service of broader learning, fits precisely with the vision that guides ET-IL planning. In addition, successful ET-IL programs require incorporation of ET-IL (or ICT) literacies into content-area curricula and instruction, which is a specific goal of this project.



[1] TRAILS — Tool for Real-Time Assessment of Information Literacy Skills,

[2] Based on his notes from at a NECC “Birds-of-a-Feather” session in June 2007, Len Scrogan, Director of Educational Technology for Boulder Valley School District, reports that an interesting feature of the Florida assessment strategy is that the state’s education agency reports to the U.S. Department of Education only the overall percentage of Floridian 8th graders who are technologically proficient — not providing the federal agency individual district statistics. Instead, those are reserved for the districts themselves to analyze and use. According to Education Week, “Florida has taken the customization idea to the nth degree, having its technological-literacy assessment built to suit at Florida State University’s Florida Center for Interactive Media, in Tallahassee. …’You can’t build something that’s one-size-fits-all,’ said Kate J. Kemker, the state’s bureau chief for instruction and innovation. (From “Tests of Tech Literacy Still Not Widespread Despite NCLB Goals,” in Education Week, 1-30-08, pp. 1, 12; (For more on assessment work in other states, see Appendix B — Assessment Developments Elsewhere.)

[3] At present the ETS markets the iSkills assessment primarily to postsecondary institutions. According to Education Week, only 5% of iSkills assessments were taken by precollegiate students, and ETS does not have plans to create an assessment for younger students. (See “Tests of Tech Literacy Still Not Widespread Despite NCLB Goals,” in Education Week, 1-30-08, pp. 1, 12;

[4] Collegiate Learning Assessment (CLA), In an e-mail to Stevan Kalmon, then Coordinator for Information Literacy and Technology for Denver Public Schools, the CLA program director, Richard Hersh, stated that his organization was interested in partnering with K-12 educators to adapt the CLA to pre-collegiate students.