Technology Literacy Assessment Project (TLAP)

Proposal to Colorado Department of Education
Power Results Grant Program

Section B: Project Evaluation

This section addresses the evaluation plan for the one year of the project proposed for the 2008-09 timeframe. The Public Good, Inc., a Colorado research firm with expertise in K-12 educational technology use, will lead the project evaluation, working with the project leadership, executive board, and advisory council to ensure that all necessary data are available in a timely manner to decisionmakers, participating districts, and the Colorado Department of Education.

Evaluation Methods

The evaluator will collaborate with the project management team to develop and refine a plan for formative and summative evaluation. The evaluator will provide quarterly reports and recommendations to the project team, and a final report documenting project outcomes. The evaluation design will seek to answer these guiding evaluation questions:

·        How is the project being implemented?

·        To what extent are stated goals and objectives being met?

·        What is the quality of the work (resources, services and products)?

·        What are the project outcomes?

·        How are the project outcomes improving learning?

 

The evaluator will draw on multiple methods and resources to answer these questions, including project documentation (administrative activities and records), documentation of project activities and artifacts, observation of educator professional development activities, surveys of participating districts and workshop participants, and data analysis. A further breakdown of the data collection plan is presented in the next part of this section.

 

The evaluation plan presents an opportunity not just to review the outcomes but to disseminate project work to educators and policymakers statewide. In addition, the evaluation plan assumes an active role by the evaluator in the project, rather than detached observation — providing more formative than summative assessment to inform the work throughout the project

Evaluation Data

Both quantitative and qualitative data will be collected relevant to the project goals and objectives. The proposed data collection plan is subject to revision with the approval of the project leadership.

 

Evaluation Question

Potential Data Sources

How is the project being implemented?

·         Administrative records and documents, including project oversight procedures, professional development plans, technical committee meeting records

·         Project correspondence

·         Observation of meetings, professional development opportunities, piloting

To what extent are stated goals and objectives being met?

·         Conduct needs analysis

·         Define standards and proficiency benchmarks

·         Develop assessment instrument and protocol

·         Conduct pilot study of assessment

·         Analyze results of pilot

·         Facilitate use of assessment instrument and resulting data

Documentation of

·         Needs assessment and analysis

·         Tools, techniques and resources associated with the assessment instrument and pilot activities

·         Professional development materials and participation

·         Pilot study results, adjustments of instrument, data analysis

·         Outreach and dissemination records

What is the quality of the work?

·       Resources

·       Services

·       Products

·         Surveys of participants and end users of assessment tools, resources, and technical assistance

·         Professional development observation, pre- and post-activity surveys

What are the project outcomes?

·         Number of district participants; extent of participation

·         Degree of consensus on standards and proficiency benchmarks;

·         Qualitative and quantitative data on pilot study participation;

·         Data analysis of pilot study results;

·         Evidence of outreach beyond participating districts; evidence of assessment data use and analysis resulting from project activities

·         Evidence of impact on educators’ and students’ learning

Project Development and Product Evaluation

The evaluator will develop data collection instruments suitable to the project needs and subject to approval by the project leadership. An early meeting of project participants will focus on defining the elements of a successful TL assessment instrument. This measure of quality, developed in collaboration with project participants, will guide the development and piloting process. Quarterly reports will include recommendations and provide the impetus for program reflection and discussion. The final report will include an assessment of success in reaching project objectives, along with recommendations for future efforts.