EMPLOYMENT AND TRAINING ADMINISTRATION
ADVISORY SYSTEM
U.S. Department of Labor
Washington, D. C. 20210
CLASSIFICATION

WIA/Performance Reporting

CORRESPONDENCE SYMBOL

PRO

ISSUE DATE

December 10, 2003

RESCISSIONS

 

EXPIRATION DATE

Continuing

ADVISORY

:

TRAINING AND EMPLOYMENT GUIDANCE LETTER NO. 15-03

 

TO

:

ALL STATE WORKFORCE LIAISONS
ALL STATE WORKFORCE AGENCIES

 

FROM

:

EMILY STOVER DeROCCO
Assistant Secretary

 

SUBJECT

:

Common Measures Policy

 

  1. Purpose. To inform states and grantees of the Employment and Training Administration’s (ETA) policy on common performance measures for Federal job training and employment programs, including guidelines on program participation and exit. The common measures represent one component in ETA’s development of a single comprehensive system to collect, record, and report program performance for workforce programs.

  2. References. President’s Management Agenda, http://www.whitehouse.gov/omb/budintegration/pma_index.html; OMB Director’s Memorandum M- 02-06, http://www.whitehouse.gov/omb/budintegration/common.html; Training and Employment Notice No. 8-02, "Implementation of Common Performance Measures for Job Training and Employment Programs."

  3. Background. ETA’s statutory and regulatory authority to administer job training and employment programs includes provisions allowing for the requirement of performance reporting from states and grantees. ETA is in the process of developing and implementing a single comprehensive system to collect, record, and report program performance for workforce programs. This system will include a new individual record for participants, a new consolidated report of workforce activities, financial reports that relate costs to activities, common performance measures, data validation, and information on services provided to employers through the One-Stop system.

    In 2001, the President announced a Management Agenda to improve the management and performance of the Federal government. One of the five government-wide goals, budget and performance integration, emphasizes program effectiveness. As part of the President’s budget and performance integration initiative, the Office of Management and Budget (OMB) and other Federal agencies developed performance measures for certain programs with similar goals. OMB Director’s Memorandum M-02-06 and subsequent documents from OMB specify indicators of performance for job training and employment programs.

    The common measures are an integral part of ETA’s performance accountability system; however, these measures provide only part of the information necessary to effectively oversee the workforce investment system. ETA will continue to collect from states and grantees all the data on program activities, participants, and outcomes that is necessary for program management and to convey full and accurate information on the performance of workforce programs to policymakers and stakeholders. This data collection will continue to include information on individuals facing significant barriers to employment, such as individuals who have disabilities, have Limited English Proficiency, are age 55 or older, are homeless, and are migrant and seasonal farmworkers.

    Four common measures apply to programs serving adults and four common measures apply to youth programs:

    Adult Measures Youth Measures
  4. Entered Employment
  5. Retention
  6. Earnings Increase
  7. Efficiency
  8. Placement in Employment or Education
  9. Attainment of a Degree or Certificate
  10. Literacy and Numeracy Gains
  11. Efficiency
  12. Programs administered by six Federal agencies – the Departments of Labor, Education, Health and Human Services, Veterans Affairs, Interior, and Housing and Urban Development – are subject to the common measures. Attachment A contains a list of Federal programs outside of ETA impacted by the common measures.

    The implementation of common performance measures across Federal job training and employment programs will enhance the ability to assess the effectiveness and impact of the workforce investment system, including the performance of the system in serving individuals facing significant barriers to employment. Multiple sets of performance measures have burdened states and grantees as they are required to report performance outcomes based on varying definitions and methodologies. By minimizing the different reporting and performance requirements, common performance measures can facilitate the integration of service delivery and break down barriers to cooperation among programs.

    This guidance includes the following sections:

    Section 4: Program Participation and ExitPage 3
     Section 5: Employment and Training Administration
    Common Measures Policy
    Page 5
    Section 6: Data SourcesPage 15
     Attachment A: Federal Job Training and Employment
    Programs Impacted by Common Measures
    Page 18
    Attachment B: Common Measures At-A-Glance Page 19
    Attachment C: Definitions of Key Terms Page 20
    Attachment D: Educational Functioning Level Descriptors Page 23

  13. Program Participation and Exit.

    A. Point of Program Participation for Reporting on Common Measures

    When does program participation occur?

    Individuals who are determined eligible and receive any service funded by the program in a physical location (e.g., a One-Stop career center) are participants. The criteria that are used to determine whether an individual is eligible to participate will be based on the guidelines for the program.

    Note: Individuals who only receive a determination of eligibility, but do not receive an additional service, are not participants. Individuals who visit a physical location for reasons other than its intended purpose (e.g., use of restrooms or ask staff for directions) are not participants. States and grantees may choose, but are not required, to consider as participants those individuals who receive services that are available through the Internet and are not accessed through a physical location (e.g., a One-Stop career center).

    What is the date of participation?

    The date of participation is the date of the first service received after the individual is determined eligible.

    When do individuals count in the common performance measures?

    All participants will be taken into account when measuring performance using the common measures.

    There may be some circumstances in which certain participants counted in the common measures may not be included in other measures of performance due to statutory or other reasons. For example, under WIA Section 136(b)(2)(A) and WIA Regulations 666.140, the core indicators of performance apply to all individuals registered for the Adult and Dislocated Worker programs, except for those individuals who participate exclusively in self-service or informational activities. Any such circumstances will be addressed in future program guidance.

    B. Point of Exit for Reporting on Common Measures

    What is the purpose of exit?

    For performance accountability and reporting purposes, the term "exit" is used in most of the common measures (except the efficiency as well as the literacy and numeracy gains measures) to identify which participants are to be included in the computation of the measures. Comparability of performance information across states and other ETA grantees is possible only if a single point-in-time is used to begin measurement. Any participant who exits from a program becomes part of an exit cohort, a group of individuals who are determined to be "exiters" and used collectively for performance measurement purposes.

    When does exit from the program occur?

    Exit occurs when a participant does not receive a service funded by the program or funded by a partner program for 90 consecutive calendar days.

    The term "service" does not include activities that are open to non-participants, services and activities specifically provided as follow-up services, or regular contact with the participant or employer to only obtain information regarding his/her employment status, educational progress, or need for additional services.

    What is the date of exit?

    The date of exit is the date on which the last service funded by the program or a partner program is received by the participant.

    Are there any exceptions to the definition of exit?

    Participants who have a planned gap in service of greater than 90 days should not be considered as exited if the gap in service is due to one of the following:

    Service providers/grantees must document any gap in service that occurs and indicate the reason for the gap in service.

    How are youth who also receive adult services counted in the measures?

    In the case of a youth participant who becomes a participant receiving adult services (i.e., receiving services funded with both youth and adult funds), the individual will be counted in both the youth and adult common measures. On the other hand, neither set of exit-based measures will be applied until the individual has exited the adult program services (and any other partner services).

    C. Excluding Participants from the Common Measures

    What is the purpose of excluding certain participants from the measures?

    Occasionally, circumstances arise that are beyond the control of both the participant and the program. The intent here is to identify a common list of specific reasons as to why a participant can be excluded from common performance measure calculations. Service providers and grantees must document the reason that excludes the participant from performance calculations.

    What are the specific exclusions?

    Participants in the following categories, either at the time of exit or during the three-quarter measurement period following the exit quarter, will be excluded from the measures:

  14. Employment and Training Administration Common Measures Policy. All programs administered by ETA will be subject to the common measures policy, unless explicitly excluded. The ETA programs required to implement the adult and/or youth common measures include the following:

    Program Common Measures
    H-1B Technical Skills TrainingAdult
    Job CorpsYouth
    Labor Exchange1 Adult
    Migrant and Seasonal FarmworkersAdult
    Native American Employment and TrainingAdult and Youth
    Responsible Reintegration of Youthful Offenders2 Youth
    Senior Community Service Employment ProgramAdult
    Trade Adjustment AssistanceAdult
    WIA AdultAdult
    WIA Dislocated Workers
    (including National Emergency Grants)
    Adult
    WIA YouthYouth

    1 ETA and the Veterans’ Employment and Training Service (VETS) share a data collection and reporting system. While the VETS-funded grant programs are not included in the above list of ETA programs, decisions impacting the application of the common measures to the labor exchange will extend to those activities reported via the Labor Exchange Reporting System. ETA and VETS will provide additional guidance as necessary.

    2 Future grantees of the Responsible Reintegration of Youthful Offenders program will be subject to the common measures. Grantees receiving funding under this program through the Department of Justice will not be subject to this Department of Labor policy document.

    Designation that a program is subject to the adult and/or youth measures in no way impacts the existing age eligibility requirements of the program.

    ETA’s common measures policy will be implemented during Calendar Year (CY) 2004. Guidelines on specific implementation timelines in CY 2004 for each program and any accommodations needed to implement the common measures (including those that might be required to effectively assess performance in serving people with disabilities) will be issued separately for each program. A notice on new reporting instructions, which will include information on the common measures, will be placed in the Federal Register for public comment in early 2004. Final reporting guidelines will be issued in spring 2004.

    The remainder of this section describes the methodologies of the measures. Operational parameters and definitions of key terms for each measure are also provided in this section to further clarify details needed to implement the measures. The adult common measures are discussed in Part B of this section; the youth measures are addressed in Part C; and the efficiency measure, which applies to both adult and youth programs, is discussed in Part D.

    A. Introduction to Common Measure Methodologies

    The methodologies of the common measures are written as an equation, clearly identifying who is in the numerator and who is in the denominator. In cases where there are conditions that apply to both the numerator and denominator, the condition is represented in italics at the beginning of the measure. For example, the adult entered employment rate is defined as:

    Of those who are not employed at the date of participation:
    The number of adults who are employed in the first quarter after exit divided by the number of adults who exit during the quarter.

    The condition "of those not employed at the date of participation" applies to both the numerator and denominator as follows: the number of adults who are not employed at the date of participation and are employed in the first quarter after exit divided by the number of adults who are not employed at the date of participation and exit during the quarter. In addition, the phrase "who exit during the quarter" is used frequently in the denominators of the measures. This phrase refers to the exit quarter.

    B. Adult Measures

    Entered Employment

    Methodology
    Of those who are not employed at the date of participation:
    The number of participants who are employed in the first quarter after exit divided by the number of participants who exit during the quarter.

    Operational Parameters

    Definitions
    Employed at the Date of Participation - An individual employed at the date of participation is one who, during the seven consecutive days prior to the date of participation:

    Employed in First Quarter After Exit Quarter - The individual is considered employed if wage records for the first quarter after exit show earnings greater than zero.

    Discussion
    ETA recognizes concerns that have been raised in regards to excluding individuals who are employed at the date of participation from this measure. However, including individuals with jobs t program entry does not provide an assessment of program impact in increasing employment and is not fully consistent with the concept of an entered employment indicator. The positive impact of services provided to incumbent workers and the underemployed can be demonstrated in both the retention and earnings measures. To ensure a comprehensive understanding of job attainment, ETA will also collect information on the post-program employment status of individuals who are employed at program entry.

    Retention

    Methodology
    Of those who are employed in the first quarter after exit: The number of participants who are employed in both the second and third quarters after exit divided by the number of participants who exit during the quarter.

    Operational Parameters

    Definitions
    Employed in First, Second, and Third Quarters After Exit Quarter - The individual is considered employed if wage records for these quarters show earnings greater than zero.

    Discussion
    A positive outcome on the retention measure does not necessarily indicate continuous employment. However, by defining a positive outcome as employment in the first, second, and third quarters after exit, the measure approximates retention for six months following entry into employment.

    Earnings Increase

    Methodology - Earnings Increase 1
    Of those who are employed in the first quarter after exit:
    Earnings in the first quarter after exit minus earnings in the quarter prior to the date of participation divided by earnings in the quarter prior to the date of participation.

    Methodology - Earnings Increase 2
    Of those who are employed in the first quarter after exit:
    Earnings in the third quarter after exit minus earnings in the first quarter after exit divided by earnings in the first quarter after exit.

    Operational Parameters

    Definition
    Employed in First Quarter After Exit Quarter - The individual is considered employed if wage records for the first quarter after exit show earnings greater than zero.

    Discussion
    The earnings increase measure has two parts. Earnings increase 1 is intended to evaluate program impact on earnings by comparing pre-program to post-program earnings. Earnings increase 2 is designed to capture earnings increases that occur as a result of raises, promotions, or other advancements following program exit.

    For purposes of the common measures, the pre-program earnings of dislocated workers will be determined as described above – one quarter prior to the date of participation – regardless of the date of dislocation. However, ETA will continue to collect the data necessary to determine a wage replacement rate for dislocated workers that is based on wages at the point of dislocation.

    C. Youth Measures

    Placement in Employment or Education

    Methodology
    Of those who are not in post-secondary education, employment, or the military at the date of participation:
    The number of participants who are in employment or the military or enrolled in post-secondary education and/or advanced training/occupational skills training in the first quarter after exit divided by the number of participants who exit during the quarter.

    Operational Parameters

    Definitions
    Advanced Training/Occupational Skills Training - An organized program of study that provides specific vocational skills that lead to proficiency in performing actual tasks and technical functions required by certain occupational fields at entry, intermediate, or advanced levels. Such training should: (1) be outcome-oriented and focused on a long-term goal, as specified in the Individual Service Strategy (ISS), (2) coincide with exit, rather than short-term training that is part of services received while enrolled in ETA-funded youth programs, and (3) result in attainment of a certificate, as defined on pages 10 and 11.

    Employed at the Date of Participation - An individual employed at the date of participation is one who, during the seven consecutive days prior to the date of participation:

    Military Status at the Date of Participation - An individual is considered to be in the military at the date of participation if: a) he/she currently is serving on active military duty and has NOT yet been provided with a firm date of separation from military service, or b) he/she is a member of the National Guard or one of the Military Reserves AND currently is serving in a mobilized (i.e., active military duty) status.

    Employed or in the Military in First Quarter After Exit Quarter - The individual is considered employed or in the military if wage records for the first quarter after exit show earnings greater than zero.

    Post-Secondary Education - A program at an accredited degree-granting institution that leads to an academic degree (e.g., A.A., A.S., B.A., B.S.). Programs offered by degree-granting institutions that do not lead to an academic degree (e.g., certificate programs) do not count as a placement in post-secondary education for this measure, but may count as "advanced training/occupational skills training."

    Discussion
    Participants who enter ETA-funded youth programs while already in post-secondary education, employment, or the military have achieved the desired outcome under this measure. Thus, these individuals are excluded from this measure.

    Successfully returning young people to school or alternative education is desirable, but secondary school enrollment does qualify as a placement under this measure. To avoid penalizing programs for serving these youth, this measure excludes youth participants who are in secondary school at exit and treats secondary school enrollment as a neutral outcome.

    Attainment of a Degree or Certificate

    Methodology
    Of those enrolled in education (at the date of participation or at any point during the program): The number of participants who attain a diploma, GED, or certificate by the end of the third quarter after exit divided by the number of participants who exit during the quarter.

    Operational Parameters

    Definitions
    Certificate - A certificate is awarded in recognition of an individual’s attainment of measurable technical or occupational skills necessary to gain employment or advance within an occupation. These technical or occupational skills are based on standards developed or endorsed by employers. Certificates awarded in recognition of the attainment of only generic pre-employment or work readiness skills are not included in this definition. A certificate is awarded in recognition of an individual’s attainment of technical or occupational skills by:

    Discussion
    Youth who exit ETA-funded programs while still in secondary school have not yet had the opportunity to attain a degree or certificate, and cannot demonstrate a positive outcome under this measure. Exclusion of youth participants in secondary school at exit treats secondary school enrollment as a neutral outcome and avoids penalizing programs for serving these youth.

    Literacy and Numeracy Gains
    The literacy and numeracy gains indicator measures the increase in the skills of participants through a common assessment tool administered at program entry and regular intervals thereafter.

    Methodology
    Of those who are basic skills deficient:
    The number of participants who increase one or more educational functioning levels divided by the number of participants who have completed a year in the program (i.e., one year from the date of program participation) plus the number of participants who exit before completing a year in the program.

    Operational Parameters

    National Reporting System:

    Educational Functioning Levels:

    Assessment Tool:

    Testing Interval:

    Definitions
    Basic Skills Deficient - The individual who computes or solves problems, reads, writes, or speaks English at or below the eighth grade level or is unable to compute or solve problems, read, write, or speak English at a level necessary to function on the job, in the individual’s family, or in society. In addition, states and grantees have the option of establishing their own definition, which must include the above language. In cases where states or grantees establish such a definition, that definition will be used for basic literacy skills determination.

    Pre-test - A test administered to a participant within 60 days of the date of participation.

    Post-test - A test administered to a participant at regular intervals during the program.

    Educational Gain - At post-test, participant completes or advances one or more educational functioning levels from the starting level measured on entry into the program (pre-test).

    Discussion
    Youth participants for whom the goal of literacy and/or numeracy gains is most appropriate are those with basic skills deficiencies. Participants who are not basic skills deficient cannot achieve a positive outcome because they have attained basic literacy and numeracy. Exclusion of individuals who are not basic skills deficient focuses the measure on the participants with the greatest need for remediation, and minimizes the burden of testing individuals who, by virtue of their existing skill level, will not achieve a positive outcome under this measure.

    Under a normal distribution of pre-test scores, most participants’ scores will place the individuals in a range where they have completed some of the skills in that particular educational functioning level. Therefore, for a majority of participants, a positive outcome for this measure (i.e., completion of one educational functioning level and an increase to the next level), is not likely to require the equivalent of completing two grade levels, but will average to the equivalent of one grade level.

    D. Efficiency Measure

    The efficiency measure applies to both adult and youth programs.

    Methodology
    Program appropriation level divided by the number of program participants.

    Definitions
    Appropriation Level - The amount of Federal funding authorized by Congress for a program.

    Program Participant - Individuals who are determined eligible and receive any service funded by the program in a physical location (e.g., a One-Stop career center) are participants. The criteria that are used to determine whether an individual is eligible to participate will be based on the guidelines for the program. (Program participation is discussed in more detail in Section 4).

    Discussion
    The efficiency measure will allow Congress and other Federal policymakers to make high-level assessments of the utilization of funds by programs based upon the number of individuals receiving program services. While cost is an important part of an assessment of program effectiveness, this element is only a starting point for analysis. Information on cost must be examined in the context of outcomes achieved, characteristics of individual participants, and the types of services provided. The efficiency measure is not intended to place greater value on low-cost services or to discourage the provision of services to individuals facing significant barriers to employment, such as individuals who have disabilities, have Limited English Proficiency, are age 55 or older, are homeless, or are migrant and seasonal farmworkers.

    The efficiency measure will be calculated at the Federal level to determine an efficiency outcome for programs as a whole. In addition, ETA may review efficiency levels as applied at the state and grantee levels based upon the allotment level of program funds.

  15. Data Sources. This section describes data sources and methods to collect data for the common measures. The data source(s) applicable to each measure are as follows:

    PERFORMANCE MEASURE DATA SOURCE
    Adult Measures
    Entered Employment Wage records Retention Wage records
    Earnings Increase Wage records Efficiency Administrative records
    Youth Measures
    Placement in Employment or Education Wage records for placement in employment and military

    Administrative records for placement in education or training
    Attainment of a Degree or Certificate Administrative records
    Literacy and Numeracy Gains Assessment instrument
    Efficiency Administrative records

    A. Wage Records

    To ensure comparability of the common measures on a national level, wage records are the data source for the employment-related measures (except as noted in this section).

    Unemployment Insurance Wage Records
    To the extent it is consistent with state law, Unemployment Insurance (UI) wage records will be the primary data source for tracking the adult entered employment, retention, and earnings increase measures and the employment portion of the youth placement in employment or education measure.

    Additional Wage Records
    While most forms of employment in a state’s workforce are "covered" and will be in the UI wage records, certain types of employers and employees are excluded by Federal unemployment law standards or are not covered under states’ UI laws. States may use record sharing and/or automated record matching with other employment and administrative data sources to determine and document employment and earnings for "uncovered" workers. These data sources include:

    ETA is currently working with the Office of Personnel Management, U.S. Postal Service, and the Department of Defense to create a data exchange system to provide access for all states to wage record information on Federal and military employment.

    B. Supplemental Sources of Data

    Currently, mechanisms are not in place to provide access to UI wage records for grantees operating the following programs: H-1B Technical Skills Training, Job Corps, Migrant and Seasonal Farmworkers, Native American Employment and Training, Responsible Reintegration of Youthful Offenders, and Senior Community Service Employment Program. For these grantees, supplemental sources of data, such as participant surveys or contacts with employers, will be permitted as an interim means of reporting on the common measures until all grantees in a program have access to wage records. Any adjustments necessary for the implementation of employment-related measures using supplemental information as the data source will be addressed in separate program guidance. Allowable sources of supplemental information for tracking employment and earnings in this interim period include case management, follow-up services, and surveys of participants and employers. All supplemental data and methods must be documented and are subject to audit.

    Supplemental sources of information will continue to be collected and reported by states and grantees. Supplemental data will be used for program management purposes and to gain a full understanding of program performance and activities. However, supplemental sources of information will not be used to calculate outcomes on the employment-related common measures.

    C. Administrative Records

    Administrative records will be the data source for the education and training portion of the placement in employment or education measure and the attainment of a degree or certificate measure. All data and methods used to determine placement in education and training or achievement of a degree or certificate must be documented and are subject to audit.

    Placement in Post-Secondary Education or Advanced Training/Occupational Skills Training
    The following data sources can be used to determine whether participants in youth programs are placed in post-secondary education and/or advanced training/occupational skills training:

    1. Case management, follow-up services, and surveys of participants to determine if the individual has been placed in post-secondary education and/or advanced training/occupational skills training; or

    2. Record-sharing agreements and/or automated record matching with administrative/other data sources to determine and document that the participant has been placed in post secondary education and/or advanced training/occupational skills training. These data sources may include:

      • State Board governing community colleges
      • State Board governing universities
      • State Education Associations
      • Integrated post-secondary or higher education reporting unit
      • Training institutions/providers

    Degree or Certificate
    The following data sources can be used to determine whether participants in youth programs attain degrees or certificates:

    1. Case management, follow-up services, and surveys of participants to determine that the individual has received a degree or certificate; or

    2. Record sharing agreements and/or automated record matching with administrative/other data sources to determine and document that the participant has received a degree or certificate. These data sources may include:

      • State Board of Education
      • State Board governing community colleges
      • State Board governing universities
      • State licensing boards for private schools
      • State Education Associations
      • Integrated post-secondary or higher education reporting unit
      • State Department of Professional or Occupational Regulation (possibly other units such as health care administration or specific boards like the "Board of Nursing")
      • Professional, industry, or employer organization or a product manufacturer or developer
      • Training institutions/providers
      • Adult Basic Education providers (GED/equivalent testing agencies)

  16. Action Required. Please make this information available to appropriate program staff.

  17. Attachments.

    Attachment A: Federal Job Training and Employment Programs Impacted by Common Measures
    Attachment B: Common Measures At-A-Glance
    Attachment C: Definitions of Key Terms
    Attachment D: Educational Functioning Level Descriptors