COLORADO OFFICE OF WORKFORCE DEVELOPMENT

The State of Colorado is committed to building a skilled, internationally competitive workforce for the twenty-first century.

Office of Workforce Development

Training Program Evaluation Measures

Section IV
Carl D. Perkins Vocational Education Act

The Carl D. Perkins Vocational and Applied Technology Education Act is Federal legislation that provides over a billion dollars nationally in funds for vocational-technical education programs at the secondary and post-secondary levels. Funds are directed to programs that provide academic and occupational skills to participants, including "special populations." The purpose of the Act is to equip a workforce with the academic and vocational skills needed to compete in a technologically advanced society.

Overview. Vocational-technical education programs are organized sequences of courses which prepare individuals for employment in current or emerging occupations requiring preparation other than a college degree. Vocational education programs emphasize the need to integrate classroom and on-the-job learning.

The Perkins Act mandates that programs provide students with strong experience in and understanding of all aspects of the industry they are preparing to enter. Vocational education programs take place in various industries, with a specific emphasis on "emerging" or high-demand fields such as mechanical engineering technology, automotive technology and health occupations.

The U.S. Department of Education’s Office of Vocational and Adult Education (OVAE) administers the Perkins Act. State Boards for Vocational Education apply for State Basic Grants by submitting a State plan to the OVAE. In Colorado, the sole State agency authorized to administer vocational education and distribute Federal funds is the State Board for Community Colleges and Occupational Education (SBCCOE). This is a nine-member Governor-appointed Board with two non-voting members representing students and faculty.

The State Board awards sub-grants to "eligible recipients" which include local education agencies (secondary schools) and post-secondary institutions. Eligible recipients must qualify for a minimum amount of Perkins dollars, which amounts to $15,000 for secondary institutions and $50,000 for post-secondary institutions. However, secondary institutions that don’t meet the minimum limits can enter into a consortium with other institutions to meet this requirement.

States allocate funds to post-secondary schools based upon the number of enrolled Pell grant recipients and the number of Bureau of Indian Affairs recipients (as compared to the total number of qualified recipients in the state). The funding formula for secondary schools is based upon the percentage of Title I funds directed to local education agencies, the number of students with handicaps who have individualized educational programs, and the number of students in schools and adults in programs in local education agencies.

Basic Grants. States receive the majority of funding in the form of Basic Grants that are based upon a formula that includes the states’ population in certain age groups as well as their respective per capita income. In FY 1999, Basic Grants totaled more than $1 billion dollars, of which Colorado received $13.4 million dollars for Basic Grants. Exhibit IV-1 demonstrates the distribution of funds.

The statute requires states to allocate 25 percent of the Basic Grant to the following activities:

• 10.5 percent for equity programs. The statute requires a distribution ratio of 7 percent for single parents programs and 3 percent for equity programs. Each state can allot the remaining .5 percent to either program.

Of the $1.4 million Colorado received in FY 1999 for equity programs, about $880,000 was distributed to single parents/displaced homemakers/pregnant single women programs. These grants provide for instruction, career guidance and counseling, and supportive services such as child care. About 1,200 persons are served by these grants in FY 1999.

The remaining $474,000 was distributed to gender equity programs. Gender equity grants fund programs, guidance and counseling, and other activities to eliminate gender bias and stereotyping in secondary and post-secondary occupational education.

• 8.5 percent for State leadership. This supports professional development, curriculum development, program assessment, vocational student organizations and partnerships between educational, business, and community-based organizations. In Colorado, 90 percent of leadership funds, slightly more than $1 million, was distributed internally within CCCOES for the purpose of supporting staff and meeting Perkins mandates. The remaining 10 percent, or approximately $113,000, was distributed to the field.
• 5 percent for administration. Colorado received almost $670,000 for administrative uses, which are used to develop and implement performance standards, perform State assessments, and conduct program evaluation.
• 1 percent for corrections. Colorado received about $134,000 for corrections programs, which it distributed to the Department of Corrections and the Division of Youth Services. Department of Corrections programs that target soon-to-be-released or job-ready individuals are given funding preference.

Local sub-grants. Of the total Basic Grant allotment, states must allocate 75 percent to the local level (local education agencies, area vocational-technical schools, and post-secondary institutions) for program improvement. States are required by the legislation to use the following criteria in distributing basic grants: programs must be of sufficient size, scope and quality; programs must demonstrate integration of academic and vocational education; and programs must serve special populations.

State Boards determine what percentage of these local funds will go to secondary and post-secondary schools. In Colorado, the SBCCOE authorized a 60 percent distribution to the post-secondary schools and 40 percent to secondary schools in FY 1999.

Colorado distributed $3.9 million in FY 1999 to secondary school districts and secondary consortiums. In total, there were 58 eligible recipients for secondary formula funds. Colorado distributed over $5.9 million to post-secondary institutions. Twenty institutions were funded through post-secondary formula funds. Funds are used at the secondary and post-secondary levels for many purposes, including program improvement, supplementary services to special populations, curriculum and equipment upgrades, staff development and in-service training, school-to-career resources, and career guidance and counseling.

Tech Prep Grants. A lesser amount of Perkins funds is distributed to states for special programs, which primarily include "Tech Prep" grants. Tech Prep is a planned sequence of study in a technical field which begins in ninth grade and continues through two years at the community college.

About $103 million dollars in Tech Prep grants were distributed to states in FY 1999, of which Colorado received $1.4 million. Of that amount, the State uses five percent for administration and additional support for statewide activities. The purpose of statewide Tech Prep projects is to support the curriculum expansion efforts for new and developing Tech Prep models.

Program authority. The Carl D. Perkins Vocational and Applied Technology Education Act of 1990 is a reauthorized, amended, and renamed version of the Carl D. Perkins Vocational Education Act of 1984.

Federal Perkins funds represent a relatively small proportion of total funding for career and technical programs at both the secondary and post-secondary levels. To better understand the extent to which local and State sources overshadow Federal funds, it is useful to compare the funding sources for secondary and post-secondary programs. The most recent year for which these numbers are available is FY 1995. Exhibit IV-2 illustrates the various funding sources for secondary vocational education.

The Federal contribution for secondary career and technical programs, at $5.6 million, was slightly less than ten percent of total State and local expenditures which totaled almost $60 million in FY 1995. The Colorado Vocational Act, funded by the Colorado General Assembly, provided over $15.1 million in FY 1995. Local support accounted for almost $40 million. Exhibit IV-3 illustrates the various funding sources for post-secondary vocational education.

The Federal Perkins contribution for post-secondary programs, at $8.4 million in FY 1995, was also about 10 percent of total funds for technical and career programs. Of the $85.2 million total funds, a little more than half came from State general funds, 32 percent from tuition, and 7 percent from local property taxes.

Beneficiaries of the program. Carl Perkins Funds provide supplementary funding to vocational programs whose core funding is provided from other sources. A primary goal of these funds is to increase access of "special populations" to vocational training programs. As such, the primary beneficiaries are as follows:

• "Special populations." Minorities, women, disabled persons, and other groups designated as "special populations" are beneficiaries to the extent that Carl Perkins Funds increase their access to, and quality of, vocational training.
• Other vocational training students. The bulk of Carl Perkins funding supplements programs that are funded via other sources. Assuming that the supplemental funding increases the overall quality or scope of training in those programs, all enrolled students will benefit.
• Hiring firms. To the extent that Carl Perkins Funds increase the number of students receiving training and/or the quality of training received, firms that hire trainees will benefit from the increased skills that their new employees acquire.

Program operations. The main operating features include application and approval for funding at both the State and local levels. Other operational features include program monitoring and evaluation.

Approval of local plans. Under the Perkins program, secondary and post-secondary institutions submit an application to the State Board for approval. In some states, including Colorado, the State Boards handle both secondary and post-secondary; in others, there are separate agencies.

All vocational programs in Colorado are required to submit a local plan in order to receive State funds. The criteria for receiving Basic Grant funding include, among other things, a demonstration that the program responds to labor market demand, serves special populations, integrates academic and vocational education, and meets standards for size, scope and quality of programs.

A more streamlined version of the local application for funds is provided for renewals. These continuations are submitted to the State on an annual basis. The State Board reviews the applications and determines whether they satisfy the above requirements, as well as other funding formula requirements as defined by the Act.

Approval of State plans. State Boards are required to submit State plans on an annual basis, although amendments are being accepted for years 1997 through 1999. These State plans are reviewed by the U.S. Department of Education, which determines whether the State plans meet the statutory requirements for funding.

Colorado submits an Annual Performance Report to the Department of Education to comply with legislative requirements. This report includes various program statistics for vocational education programs, including data on enrollment figures, enrollment by gender, number of completers and distribution by program area. In addition to these statistics, the report contains program descriptions which include program goals, significant achievements and impacts.

Program monitoring and evaluation. State Boards are required to annually evaluate the effectiveness of their programs based upon the measures and standards as defined in the Perkins Act. In Colorado, every program is monitored on an annual basis and evaluated on a five-year basis.

The monitoring responsibilities are divided among the Colorado Community College and Occupational Educational System and the eligible recipients. The eligible recipients are responsible for areas such as effectiveness and quality, outcomes, fiscal and program documentation, and program data for equity programs. The monitoring at the State level includes several tasks, such as reviewing and approving/disapproving local plans; conducting program reviews; conducting on-site reviews; and collecting and reviewing annual performance reports.

Current Program Statistics

The Act requires each state to develop and implement a statewide system of standards and performance measures for secondary and post-secondary vocational education programs. Among these are general output statistics which are intended to measure how program resources are being used. Evaluation measures, used to determine the impact of the program, are discussed later in the section.

Colorado, like other states, collects information on the number of total enrollees and a breakdown by sub-populations, number of program completers, number of "eligible recipients" funded, and the number of special populations served. Some of these program statistics are discussed below.

Secondary enrollment and completion rates. In FY 1996-97, a total of 68,929 individuals enrolled in vocational education programs. The male/female split was 48 percent and 52 percent, respectively. There were 17,102 completers in this period, which represents about a quarter of total enrollees. Approximately 19,000 individuals — or about 28 percent — were classified as disadvantaged, disabled or limited English proficiency.¹

Post-secondary enrollment and completion rates. In FY 1996-97, 48,089 individuals were enrolled in vocational education in a post-secondary institution. The male/female split was 52 percent and 48 percent, respectively. There were 6,148 completers — or 13 percent of the total enrollees. Approximately 27 percent of total enrollees were classified as disadvantaged, disabled or limited English proficient.² 1,367 individuals participated through a correctional program.

Follow-up results for secondary students. The largest percentage of secondary completers continue their education (35 percent). The next largest category, at 30 percent, finds employment related to their training. Just nine percent were employed in a job unrelated to their training.

Follow-up results for post-secondary students. The largest percentage of post-secondary student completers (67 percent) find employment related to their training. About 13 percent are employed in unrelated work and 10 percent continue their education.

Current Performance Measures

The State of Colorado has established nine performance measures to be used in its annual performance reports from funded organizations. These performance measures evaluate Perkins-funded programs, but do not evaluate the actual impact of Perkins funds in instances where those funds supplement other funding sources for the programs. Each organization receiving Perkins funds defines its own numeric performance standard based on either state guidelines or that organization’s own past performance. As of 1997, statewide figures are not presented in the annual Carl D. Perkins Federal Colorado Vocational Education Annual Performance Report prepared by the Colorado Community College and Occupational Education System.

• Achievement of Basic and Advanced Academic Skills. Measures academic skills, which include basic skills (e.g., reading, writing, and speaking), thinking skills (e.g., reasoning and problem solving), and personal qualities (e.g., responsibility and self-esteem).
• Attainment of General Occupational Skills. Measures workplace competencies (i.e., the individual demonstrates interpersonal skills, knowledge of resources and ability to use technology).
• Attainment of Specific Occupational Competencies. Examines specific occupational skills related to training area.
• Successful Program Persistence and Completion. Measures the extent to which individuals follow through with their training.
• High School Graduation or its Equivalent. This measure is only used at the secondary level.
• Continuation of Education or Job Placement. Measures extent to which program completers continue their education or find jobs.
• Client Satisfaction. Examines satisfaction among currently enrolled students, alumni and employers.
• Sex/Ethnicity Equity. Measures equity based upon gender and ethnicity by disaggregating data in the above standards and by looking at program enrollment data by sex and ethnicity.
• Equal Access for Special Populations. Measures access by disaggregating data in each of the above standards and by looking at enrollment data by special population groups.

Each program is required to provide evidence to the State on the methods and instruments it uses to measure some or all of these standards and measures. The eligible recipients are required to produce data on these performance measures during the five-year evaluations. However, the State is required by legislation to only collect — not evaluate — these data.

Evaluation measures used or recommended in analyses of other vocational education programs. A review of existing evaluations and studies on vocational education and Carl D. Perkins revealed that the majority of identified measures focused on skills attainment, wages and placement following participation in vocational education programs. Some of these include "before and after" comparisons. Other common performance measures include a variety of process or output data which describe, among other things, enrollment, participation and completion characteristics. These measures are summarized below.

• Pre- and post-tests of skills
  • Basic skill attainment, including comparisons of program participants and non-participants in grades, dropout rates and continuing education (Colorado Performance Measures)
  • Occupational skill attainment, including comparisons between program participants and non-participants in work readiness (Colorado Performance Measures)
• Post-program data
  • Placement into continuing education or the military (Colorado and U.S. Department of Labor)
  • Wage and earnings (Department of Labor)
  • Client satisfaction (Colorado)
• Process or output data
  • Program participation, retention and completion rates (Colorado, GAO)
  • Sex/Ethnicity Equity (Colorado)
  • Equal access for special populations (Colorado)
  • High school graduation or its equivalent (Colorado)

These measures may be somewhat useful in assessing performance of vocational education programs in general. However, each measure fails to evaluate the impact of the vocational training or, more specifically, Perkins funds compared with what would have occurred without these programs.

Because each state is allowed to define its own performance standards and evaluation measures, little direct comparison is available between states or to a national "average." While many states have common themes such as placement, academic gains, and development of occupational skills, the combination of measures that define each state’s performance standards differ. Also, where states have nominally similar elements of performance standards, the means of measuring those standards often differ, as illustrated by the following examples:

• Idaho includes equity and access as a performance measure. However, performance is measured in different ways than for the sex/ethnicity measure in Colorado. For example, sex equity/access in Colorado is measured in terms of the rates at which males and females meet the other listed performance standards, while equity in Idaho is measured in terms of student/instructor ratios, dissemination of education information, and the development of student career plans.
• Kansas and Colorado both state that gains in basic and advanced academic skills are performance goals. However, Kansas defines these skills as performance in math, reading, comprehension, and writing. Colorado defines these skills, as basic skills defined by the Secretary of Education, plus thinking skills and personal qualities.
• Wyoming’s performance standard of "competency attainment" is measured via performance against course competency checklists for each course. Colorado’s performance standard of "attainment of occupational competencies" is also measured against locally developed assessment standards, but it is not clear that Wyoming’s competency checklists are equivalent to Colorado’s assessment standards.
• Utah defines successful placement as employment, military service, or additional education for at least one month within the first nine months of program completion, whereas Colorado defines the same measure less specifically, as entry into follow-on education, training, employment or military service upon completion of the program.

Such differences in the selection and definition of performance standards make a comprehensive comparison between states difficult.

Potential impact of Perkins funds. One way to look at the outcomes attributable to Perkins funds is to ask what would have happened without these funds. Based on interviews with program administrators, the following were identified as potential positive effects directly attributable to Perkins funds and mandates.

• The Perkins requirements may have accelerated the emphasis on the development of local plans. These plans are required to demonstrate an awareness and responsiveness to local labor market conditions. Arguably, these requirements better ensure that the programs respond to local labor market conditions.
• Performance data requirements may have accelerated the collection of performance data by local schools and programs. Without these requirements, local education agencies and post-secondary institutions would not have been required to collect such data.
• Perkins’ requirements to serve special populations may have resulted in providing funds targeted toward special populations that might not otherwise be available.
• The legislation requires that funds be allocated to special programs such as single parents, displaced homemakers and single pregnant women. No other programs are required to serve such programs; thus, this requirement may provide funds for special programs that otherwise would not have been funded.
• The issue of gender equity was raised in the Perkins Act. The requirement that specific funds go toward equity programs may have increased awareness of equity as an issue which benefits other programs, and it may have redirected funds to such programs that otherwise would not have been allocated.
• The legislation requires that one percent of funds go toward corrections programs. No other vocational programs are required to provide services to this population; therefore, the Perkins dollars may be the only source for correctional funding.
• The Perkins Act possibly increases the total funding of Vocational Education over what the State and local school districts and post-secondary institutions would have done. This is not clear and probably varies by school district.
• Tech Prep probably advances the integration of secondary and post-secondary education programs and may have funded programming that would not have otherwise been available.

The above list presents a number of hypotheses and researchable questions, rather than conclusions.

Summary and Recommendations of Outcome Measures

The Perkins Act funds represent a relatively small portion of total vocational education funds. This blending of Federal funds with local and State sources presents challenges to effectively evaluating the impact of Perkins funds. As the State of Washington’s evaluation suggests (see Appendix B), the most common performance measures used to evaluate the effectiveness of Perkins funds are neither appropriate nor informative. It is difficult — if not impossible — to isolate the impact of Perkins funds when they only represent a small proportion of overall vocational education program funds.

Thus, evaluating the vocational education program as a whole (rather than isolating Perkins funds) may be a first step to understanding the effect of any additional funding provided through Perkins. Certain inferences can be made about the utility of Perkins funds once this program evaluation has been performed. In other words, a positive return on investment from vocational education as a whole implies that there is likely a positive return on investment from the additional funding provided through Perkins (if, in fact, Perkins increases the total amount of funds devoted to vocational education). However, the degree of return on investment may vary for each additional increment of funding, so a second analysis is necessary to determine the degree of payback afforded by the increase in program funding provided by Perkins Funds.

BBC recommends a three-step evaluation process. First, what value is provided by the programs that receive Perkins funding? Second, what is the impact of the Perkins funding on the operations and utility of those programs? In other words, beyond the additional funding, how has the Perkins Act affected the delivery of vocational education? Third, how do these impacts relate to the state’s investment in the program, in light of the fact that the funding source is Federal and not the State’s own coffers?

Step 1. Return on investment for vocational education as a whole. The State and local levels are currently collecting data on participation and completion rates and post-training earnings, employment and placement. These data do not capture what would have happened in the absence of such funds or programs. There is no comparison of outcomes for program participants and non-participants.

The most effective means of evaluating the program’s impact requires the use of a control group in which one group receives services and another group does not. This type of experimental study would inform administrators about the employment and earnings impacts that the program had on participants that they would not have otherwise experienced without taking the program. However, the logistical and administrative requirements can impose significant burdens on program evaluators and can also raise ethical questions with respect to the program’s withholding of services. In addition, this may not be practical with certain vocational education programs where individuals may merely enroll in classes with little or no "program" involvement. Even so, the benefits of such an experiment can provide valuable information on the long-term consequences of participating in vocational education versus not participating.

An alternate approach to conducting experiments is an econometric approach. Econometric studies statistically create a control group without random assignment of program applicants to a "treatment group" and a "control group." These studies require data on individuals who enter the training program and individuals who do not, ideally including pre-training data in addition to post-training employment outcomes.

An alternative econometric approach, which would only require data for individuals who entered vocational education programs, may be possible. Program administrators would obtain pre-training data for all entrants and then collect data on skills attainment and other performance indices for individuals participating in the program. (For example, if grades in these courses indicate how well individuals have absorbed the training, then grades would be compiled.) Individuals who exit the vocational education programs would be identified in these databases. Finally, employment outcomes would be tracked for all entrants to the vocational training programs whether or not they completed the programs. With these data, it may be possible to detect whether students who are well-trained in specific job skills upon program completion have better employment outcomes than students who are less well-trained or did not complete the training.

If experiments or econometric analyses are not feasible, post-program employment data coupled with detailed interviews of program participants may provide some indication of program outcomes. Using this evaluation approach, program participants would be interviewed at entry, exit and for a follow-up period. Customer satisfaction interviews may already be providing some of this information. Data on employment and earnings would also be collected. This approach has a number of limitations, most particularly the possible bias of interviewees, but it offers a less resource-intensive (if less rigorous) approach to achieving the same evaluation goals as the experimental and econometric approaches.

Questions to be answered in this step of the analysis include:

• What would the participant have done without the program (e.g., become employed or participated in another program)?
• How does the individual assess the value of his/her vocational education training?
• Is the individual currently employed?
• If so, how long did it take to find employment?
• What is the individual doing? Is it related to training?
• What is the individual earning? What type of benefits does the individual receive?
• What additional training did you receive? If any, describe the type of training and why the individual received it?

Particular measures that might be used in this evaluation are linked to the common concepts used by several states to evaluate vocational programs: academic progress, development of occupational skills, placement, retention, and other factors.

Step 2. Incremental effects of Perkins. The above analyses could lead to return on investment estimates for vocational education as a whole. Alone, they would not indicate the effect of Perkins funds. Remaining questions include:

• Does Perkins provide funds which enable local educational institutions to serve more students than they would have served without the funds or offer them different types of classes or more intense courses?
• Does Perkins simply substitute for local or State funds that would have been spent if the additional Perkins funding were not available?
• Have the requirements that come with Perkins dollars changed how the State or local education providers implement vocational education programs?

The most straightforward evaluation approach available to answer these questions is conducting counterfactual interviews with State and local officials. These interviews would probe each of the above questions.

The Perkins-specific issue of increasing access to special populations will need to be examined in parallel to the above analysis of the incremental effects of Perkins funds. Colorado’s existing measure of examining the participation of special populations should be sufficient for this purpose, but, as with other programs, there appears to be no baseline analysis to compare participation against, i.e., answering the question of whether participation would be lower without Perkins funding.

Step 3. Fiscal return on investment to the State of Colorado. Perkins provides an addition to State funding for vocational education, with little or no additional financial investment required by the State. If the Perkins funds provide any additional benefits that would have not occurred without the Perkins Act, then there will be positive returns to the State (as there was no State cost). To the extent that Perkins funds are linked to the provision of "matching" State funds or a "maintenance of effort" by the State, it is appropriate to assess whether the programs funded by the matching funds (not the Perkins funds) provide a positive return to the state in ultimate tax revenues or other measures.

¹National 1993-1994 figures for all vocational education programs indicate that slightly less than 30 percent of students are economically disadvantaged, 3 percent have limited English proficiency, and 6 percent are disabled. Some students may be included in more than one group, which overstates the sum of these figures.

²Same as Footnote 1.