Position Profile


Job Title:  Assessment Analyst

Work Unit:  Apprenticeship and Student Aid / Industry Programs and Standards / Core Services

Ministry:   Alberta Advanced Education

Competition Number:  1040733

Date:  February 2017
 

Purpose

 

The Industry and Program Standards (IPS) Branch works with apprenticeship and industry training committees, stakeholders, and partners to develop, implement, and maintain training and certification standards for all of Alberta's designated trades and occupations. Part of the branch's mandate is the development, production, and release of theoretical and practical examinations and response keys for all trades and designated occupations. Exams are managed through the Certification Enterprise (CE) system, which is linked to ATOMS. The Assessment Analyst is accountable for guiding exam development, CE system integrity for examinations, as well as exam activation, suspension and de-activation on the system. This includes reviewing and assuring data integrity between the course outline objectives, examination items (questions) and item analysis. This position also manages exams on the Inter-provincial Computerized Examination Management System (ICEMS) for multiple Red Seal trades for which Alberta is the host province.

This is also the lead advisory position to provide CE system support between the IPS Managers (MIPS), users of the CE software, and the technical system software analysts. Addressing system software problems and issues from the branch user perspective is important in maintaining the validity and reliability of the assessments.

Responsibilities and Activities

  1.

Represent user needs for CE system management.

Activities:

  • Liaise with Business Integration regarding linkages between CE and ATOMS.
  • Identify examination software and hardware issues for remediation by Business Integration based on consultation and analysis of user needs and usage trends.
  • Test and provide feedback and functional approval for CE system changes.
  • Work with the CE/Exam Management System to link and bridge systems.
     
  2.

Lead content management and analysis to sustain high quality of exams on CE and ICEMS systems.

Activities:

  • Move items from active to draft EMS/CE for new course outlines.
  • Validate course outlines, class sections, Table of Specifications (TOS) and exams (theory and practical), ensuring alignment and integrity across products.
  • Oversee the production and maintenance of graphics for exams and other projects, using Corel Graphics Suite and Visio for Microsoft.
  • Perform data integrity and quality control on exams, TOS, and course outlines.
  • Activate and deactivate all exams on CE.
  • Conduct item analysis and evaluation.
  • Produce analysis of examination/assessment results to MIPS to contribute to the improvement of exam development and consistency in quality across exams.
     
  3.

Provide consultation, advice and training to branch staff to use the CE and ICEMS system most effectively.

Activities:

  • Develop processes, procedures and related supports to foster consistent and effective use of the CE and ICEMS systems.
  • Coach MIPS exam item development, labeling, item analysis and cross-analysis.
  • Consult with user groups to identify training needs and opportunities for system enhancement.
  • Develop and deliver training on the systems and exam development to various user groups.
     
  4.

Support the Senior Manager's leadership of the unit to deliver on business goals and priorities.

Activities:

  • Liaise with production personnel on the printing of exams so that there is consistency between paper and online exam versions.
  • Identify significant emerging issues and present options and recommendations for resolution.
  • Research and respond to requests for information, ARs, briefings, etc.
  • Participate with other provinces in Red Seal / ICEMS work as needed under the direction of another branch Senior Manager.

Scope

 

The work of this position directly impacts the implementation and evolution of the CE system in a manner that meets user, business and technology needs, and its linkages to ATOMS and the ICEMS for Red Seal trades. The Assessment Specialist also impacts the integrity of theoretical and practical assessment for trade and occupation certification, which is an essential part of the process to achieve certification in designated trades. The variety of analyses and validations completed by the Assessment Specialist provide assurance to the integrity and validity of exam items and exams; this is essential to ensure that exams reflect the knowledge required for trades. The guidance and advice provided by the Assessment Specialist ensures that all users are able to use the system, and that the system evolves to meet user and business needs.

The Assessment Analyst is accountable for the quality of assessment items in the CE system and for Red Seal exams for which Alberta is the host (currently 13 trades), the effective utilization of the systems and associated assessment (exam) management within the systems. This is the primary advisory role for MIPS and other staff on using the systems, and for providing training; this helps to increase the quality and consistency of integrity of the exams. This is also the primary position lead for liaison with Business Integration regarding the hardware and software of the CE system so that it meets business and user needs; judgement is applied to know when and how to engage Business Integration and how to reflect user needs for system development or enhancements. The Assessment Specialist has latitude and authority to determine areas of focus, priorities, and methods for achieving results (e.g. organizing the review of exam items and activation of tests on the CE system to meet overarching assessment timelines). The Assessment Specialist makes the decision to activate or deactivate exams based on consultation with MIPS and validation of relevant exam information.

The Assessment Analyst is challenged to identify and articulate multiple user needs for the CE system; there can be competing needs and potential conflicts with what the CE system is able to accommodate. Another challenge is ensuring the CE system is able to effectively disseminate assessment exams with minimal errors and with appropriate levels of security. Some trades have common training (i.e. the same training content is used for more than one trade) and so more complex analysis is required by the Assessment Analyst (e.g. to determine if exam questions are too easy or difficult). Items might be appropriate for some trade exams and not others. This work shapes ongoing exam development.

Resources available for guidance include applicable legislation, regulations, policies and procedures. MIPS provide guidance in the direction and requirements of individual trades and occupations. The Assessment Specialist is expected to research, refer to precedents and seek out guidance and information from other areas in the branch or sector as required. Precedents may need to be adjusted to fit a particular situation. Situations for which there is no apparent direct precedent are discussed with the Senior Manager, Core Supports. MIPS, and the Director, Industry Programs and Standards are also available for assistance and guidance as required.

Knowledge, Skills and Abilities

 
  • Thorough knowledge of the apprenticeship and industry training system (in Alberta and Red Seal Programs) and the role of assessment in the attainment of certification in designated trades and occupations.
  • Knowledge of legislation and policies that govern the AIT system and AIT sector (e.g. Apprenticeship and Industry Training Act; Government Accountability Act; Alberta Apprenticeship and Industry Training Board policies; Red Seal Policy and Procedures).
  • Thorough knowledge of adult learning and assessment principles, approaches and theories (e.g. exam development exam item integrity, alignment of exam items to course outline).
  • Thorough knowledge of the CE system and how it links to ATOMS and the Red Seal exam management system (ICEMS).
  • Knowledge of the Red Seal Program and related processes for all the trades, particularly for the trade exams for which Alberta is the host.
  • Knowledge of the project management processes used in technology system development to effectively represent use needs for system design, development and evolution (e.g. identification of business needs, user acceptance testing).
  • Proficient in working with computer programs such as the Microsoft Office suite including Word, Excel, Outlook and PowerPoint, in particular working with letters, reports and data spreadsheets.
  • Proficient in working with examination management software and graphics software, including Corel Graphics Suite and Visio.

Education:

  • Trade journeymen certification or university degree.

Skills and Abilities:

  • Strategic thinking to develop, assess, and recommend improvement options for the CE system.
  • Analytical skills to conduct item analysis and evaluation for exam quality assurance.
  • Consultation and advisory skills to work with multiple exam system users so that the systems are used effectively and consistently across users.
  • Systems thinking skills to understand how exam integrity impacts the whole AIT system - it's reputation and credibility and validity in assessing skills for designated trades and occupations.
  • Project management and coordination skills to work on multiple exams and with multiple stakeholders at one time across several exams; multi-tasking and prioritization of competing demands.
  • Ability to integrate, prioritize and represent user needs for the evolution of the CE system.
  • Ability to represent the province on issues related to ICEMS management and evolution.
  • Ability to blend business and technical knowledge to make recommendations for the evolution of the CE system.

Contacts

 
  • MIPS - Provide advice on CE; develop and provide results of item analyses to contribute to enhancements in exam quality; provide training and coaching as needed; identify AIT system user needs.
  • Administrative staff that work in CE - Oversee the linking of exam items to course outlines and coordination of the production of exams (e.g. formatting, editing, graphics); provide training and coaching as needed.
  • Senior Manager - Raise awareness to emerging issues of significance and identify solutions; provide reports on the utilization and results of the CE system.
  • Other sector units (e.g. Business Integration Unit) - Represent user needs in the evolution of the CE system design and development; test new developments; respond to queries.
  • Inter-provincial (Red Seal) exam representatives - Liaise regarding examination issues and best practices as needed to support the Senior Manager that leads this work; link Red Seal exam items in CE.

Supervision Exercised

  N/A

 

Back to Top