Design and Methodology
Feasibility study in three phases
The present project is designed to formulate and test a pilot model based on a three-step approach to developing a multi-dimensional instrument to measure and compare the achieved outcomes in a European context. The tests to be developed are tailored per domain and within a domain per specific subject area/discipline. Each test has to be multi-dimensional to take into account the difference in mission, orientation and profile of degree programmes, related to more theoretical approaches and more applied ones. The charm of a multi-dimensional test is that the scores of students can be valued against several elements. It will allow universities to analyze their students’ performance with respect to a series of key elements, and to analyze the results in an international framework: it will be possible to compare student achievement internationally, to see whether a given institution is performing on, below or above average and is meeting (threshold) standards. Furthermore the outcomes will offer important information for quality enhancement and quality assurance. They will also give important indications about whether certain TLA-approaches are more successful than others.
The first step is updating and refining the already quite sophisticated frameworks of reference points based on the work carried out by European Commission-funded pan-European Networks; the second is producing an assessment frameworks based on the first step. These two steps will be completed in the course of the present project and, if the results warrant, the stage has ben set for the third step: creation of multi-dimensional tests.
For the development of the three-step model a bottom-up approach will be applied. This means that for each domain and related subject area/discipline a workgroup of 15 renowned academics with experience in European and international networking will be established, representing 15 different countries and various types of higher education institutions, which together cover a variety of TLA-approaches. In the pilot phase five different academic domains and five related subject areas/disciplines will be covered: Engineering (Civil Engineering), Social Sciences (Education), Humanities (History), Health Care (Nursing) and Natural Sciences (Physics). This implies that 75 HEIs will be directly involved.
First phase – Update the frameworks of reference points
As part of the first phase 5 refined frameworks of reference points for the first (bachelor) and the second cycle (master) will be developed. This is required to assure a correct balance of learning outcomes between both cycles. Past experience has shown that this is the only way of defining reliable level descriptors for both levels. The frameworks will be based on a proven format, applied for all Tuning conceptual frameworks so far. They cover such items as typical degrees offered in the domain/ academic field, typical occupations with a first and a second cycle degree, agreed learning outcomes for the first and second cycle, if feasible both at sector and subject area level; and new approaches required in teaching, learning and assessment for outcome-based learning.
Given the positive experiences regarding the development of frameworks of reference points by the academic Tuning community as well as by Thematic Network Programmes (TNPs) as mentioned above, a good way forward is to utilize and build on the work that was successfully carried out in the period 2001-2010. The frameworks that were produced during those years will be updated and enhanced – in the framework of this initiative – on the basis of current knowledge and experience so that they can continue to serve as reliable references for the outcomes of programmes of study.
Second phase – Produce the assessment frameworks
Based on the outcomes of the first phase, during the second phase of the project an assessment framework per domain/subject area will be developed consisting of a) a clear set of assessment criteria based on the multi-dimensional approach doing justice to different types of institutions and profiles; b) a detailed test blue print for each of the assessments and c) a detailed work plan for the creation and implementation of the assessments. The assessment frameworks for Education, History, Nursing and Physics will be designed for the final stage of the first cycle (bachelor); for Engineering for either the end of the first cycle or the end of the second cycle (master). Both options are open and the decision will be made by the group of experts involved in the project.
Furthermore, a White paper will be prepared that lays out the costs/benefits for various assessment designs. This will allow for making evidence based decisions regarding next steps. Educational Testing Service (ETS) will in particular be involved in this phase of the project. This non-profit organization, which is also involved in the OECD PISA and PIAAC initiatives, has the high-level expertise and staff to implement its part of the project successfully.
It is expected the tests will be machine based to limit costs. This will allow for multiple-choice based testing but also for more sophisticated testing forms.
Third phase – Design the multi-dimensional tests
During this phase the actual multi-dimensional tests will be developed, which should offer reliable information about the strength and weaknesses of degree programmes and the approaches applied, given their profiles. It is intended that these assessments will be field tested in five institutions per country with one institution (represented by their academic expert) in the workgroup and therefore involved in the development of the test. This implies that each test will be taken in 75 higher education institutions and that a total of five times 75 institutions, that is 375, will be covered.
For budgetary reasons, the testing phase is expected to be realized in a follow-up project.
