This article was written in Oct 2008 and regularly updated. The most recent update was 23 Dec 2009.

Abstract

The Engineering Accreditation Council (EAC) Engineering Programme Accreditation Manual has undergone a number of reviews from its initial 1999 version, known as the blue book. The blue book incorporated the generic attributes (also known as outcomes) of engineering graduates. The 2003 version (orange book) incorporated minor changes which included making industrial training compulsory. In 2005 a review process was conducted to make the outcome based approach more explicit. The revised manual (compiled in the black file) and adopted in 2006 introduced sections 3 and 4 which discussed programme objectives and programme outcomes respectively. Despite the evaluation process was quantitative in nature with total marks above 70% to achieve full accreditation, the evaluation on programme objectives and outcomes were qualitative. Programmes must also achieve marks above 50% for each of the five criteria to receive full accreditation. Conditional accreditation would be accorded if there are shortcomings that are not classified as non-compliance. Declined accreditation is accorded without the need to triangulate the data. Despite the rigid structure for decision, there have been cases where decisions were on the contrary to that qualified by the marks obtained. The 2007 version introduced the qualitative evaluation with minor changes to the criteria. Changing from quantitative to qualitative evaluation complicates further the decision making as it becomes more subjective. Evaluators are expected to triangulate the information/evidence obtained before arriving at a decision. This document intends to provide some guidance for evaluators to identify strength, weakness, concern and opportunity for improvement of programmes and allows evaluators to make consistent decisions. Eventually, evaluators need to ascertain that the programme has the breadth and depth, the outcome based approached is implemented, and continual improvement is visible.

Introduction

Often some evaluators are commented on their lack of aptitude when delivering a recommendation from evaluation exercises. Evaluators have the tendency to focus on the problem areas and at time may be nitpicking. Thus a long list of weakness and/or concern is drawn, which may include unwarranted statement, such as, “the institution does not ensure the consistency of the colour of final year report cover”. Related issues (despite appearing under different criteria) could have been consolidated to provide a more meaningful evaluation. Usage of words in the evaluation report, such as, “inadequate” and “incomplete” may have far greater implication, as they imply non-compliance to the standard. Although there are several criteria to be satisfied for accreditation, that does not mean each criterion should be evaluated in isolation. There is a need to look at them across the board and determine the relationship between them.

Numeric (quantitative evaluation and at times prescriptive) with little comments provided, as practised by evaluators in the past, has to make way for qualitative evaluation. The quantitative approach no doubt simplifies decision making but is unable to provide a useful feedback to the programme concerned as evaluators would generally shy away from reporting specific problem areas and identifying strength that the institution/programme can build upon. Even institutions are expected to submit a qualitative self-assessment report, requiring them to evaluate their status on the implementation of the programme against the standard. Similarly, evaluators must also be qualitative in their evaluation and conclude then whether there is compliance to the standard.

At the end of an accreditation exercise it is imperative that evaluators must be able to determine whether the programme is sustainable, involved various stakeholders at appropriate time in reviewing it, has appropriate breadth and depth, demonstrated the outcome approach, and has appropriate resources and adequate quality system in place.

Evaluators must refer to the EAC Engineering Programme Accreditation Manual1 to determine compliance to the requirements whereas the document, Guidelines for Evaluators found in the Manual, is meant to provide some guidance through the examples and rubrics provided. Programmes cannot be penalised for not meeting the Guidelines unless there is a clear transgression against the Manual. The appendices of the Manual may detail out the requirements but they must not be taken as prescriptive. As an example, the Appendix B of the Manual which prescribes the areas and courses to be addressed in the conventional programmes is meant to guide institutions to consider them when providing the contents of the programme. It does not prescribe specific courses with the same title to be adopted by the institutions. Similarly sample tables provided in Appendix H of the Manual is to help institutions in developing their self assessment report, and they could be modified or replaced with better tables that could provide clearer and enhanced explanation. However, tables without evaluative comments are not adequate.

This document expounds on the issues that would allow evaluators to make an evaluation that is balanced, in relation to determining strength, weakness, concern and opportunities for improvement. This approach allows evaluators to confidently ascertain that the programme has the breadth and depth, and that the Institution of Higher Learning implements outcome-based approach significantly and continually improved.

What constitutes strength?

A programme is deemed to have strength when it exceeds the minimum standard set by the EAC Engineering Accreditation Manual. Achieving the minimum standard is deemed as satisfactory only. The following are examples of what constitutes strength, which can be defined as anything that exceeds expectations, with regards to the respective criteria of academic curriculum, student, academic and support staff, facilities and quality management system. The elements of programme objectives and outcomes which are an integral part of the criteria are discussed within the criteria.

Academic Curriculum

A curriculum is deemed as having the strength if it is keeping abreast with the development of engineering education, through extensive benchmarking (not only via the external examiners path) with more established programmes/institutions. Benchmarking of curriculum involves looking at the programme structure, balanced curriculum (appropriate breadth) and level of content (depth). The curriculum is built on strong fundamentals (engineering sciences) and appropriate engineering knowledge according to the discipline, which transcend national boundaries. Generic attributes (professional and/or interpersonal skills) should also be evident to prepare graduates for the advanced part of their career.

A curriculum that has been developed with clear (measurable) objective(s) and outcomes (that satisfies the ten (10) EAC stipulated outcomes) and had involved stakeholders, both internal and external, extensively are deemed as having the strength. The “extensive” participation here refers to regular and scheduled meetings (face-to-face and/or teleconferencing) with relevant (in the appropriate discipline) industry representatives including employers and alumni (in the case of external stakeholders) and involving majority of academic staff. Students’ participation (not through surveys only) in the development of the curriculum is an added strength.

An appropriate working load for students has been determined by the institution through a mechanism of extensive consultation with the academics. Usually a 15 – 16 credit per semester loading is considered excellent. Further, the curriculum should provide a blend of delivery methods such as cooperative learning, case method learning, problem based learning and project based learning (not just the typical lectures and laboratories).

Assessments that are open-ended, testing the creativity and innovativeness, challenging the students at the expected level (depth/insight) would be considered as strength. Appropriate assessment methods to measure cognitive, psychomotor and affective domains are employed. Observation techniques and use of rubrics should be evident to assess the psychomotor and the affective domains. For example, teamwork assessment cannot be simplistically associated to group work.

Having industrial attachment over and above the 4-year curriculum is strength, as it provides additional training apart from the 10 weeks exposure. However, the industrial attachment should not be conducted in a manner that would compromise the curriculum breadth and depth. For example, taking up one whole semester, or using up to 6 months within the 8 semesters for the purpose of students exposure during industrial attachment would amount to the ‘compromised situation’ especially if no evidence can be provided on students achieving the outcomes especially those in the cognitive domain.

Extensive professional exposure (industry visits and guest speakers) throughout the years students are at the institution (not limited to the specific years, e.g., in the first year or final year only) is considered strength.

Student

It would be difficult to determine student in-depth technical knowledge from the short interview session normally conducted during the accreditation visit. However, this could be ascertained from the output of the course assessments. It is expected that students’ answers demonstrate depth/insight that reflect attainment of relevant programme outcomes. Strength can be acknowledged if the findings show the programme challenges students to achieve greater heights than just satisfying the minimum standard. Ability to attain competency in the open-ended project based and problem oriented courses signifies strength in the student in attaining the appropriate outcomes.

Ability to give opinion and articulate with substance signifies student strength in communication. It is not merely ability to speak with correct tenses, intonation and confidence that should only be given emphasis. Students are clear of their goals upon graduation and highly motivated during their course of study. Motivation can be felt during the interview session where “constructive criticisms” are made on aspects related to their studies at the institutions, be it curriculum, resources and quality system. Widespread involvement of students in co-curricular activities signifies students’ enthusiasm towards a well balance education in producing a wholesome individual. Similarly students’ organisational and managerial involvement in co-curricular activities signifies leadership and managerial qualities. However, this must be widespread and not limited to a small group of students.

Students’ direct and regular involvement in the curriculum development as well as in the development of other related aspects would be seen as strength, rather than an ad-hoc approach or through survey questionnaires only.

Academic and Support Staff

If a programme just satisfies the minimum eight (8) academic staff as stipulated in the EAC Manual, it is only deemed as satisfactory. An example of strength with regard to the criterion of academic staff is that majority of the staff has PhD qualification and the number available indicates a low staff-student ratio (that enables greater contact with students). The academic staff also conduct research that permeates/contributes to teaching and learning. Majority of the academic staff are actively participating in professional activities (not just as members of professional societies). Greater than 30% of academic staff has Professional Engineer status (registered with national or international professional bodies) would be deemed exceeding the minimum standard. Design courses are taught by experienced academics (with consultancy experience or professional engineers). Majority of academic staff are aware and implementing the OBE curriculum effectively (implementing CQI at course level).

Technical support staff assigned to each laboratory and not expected to cover other laboratories indicates adequate manpower. This allows dedicated maintenance and ensures progressive development of the laboratory. However, if these laboratory staff (despite they are highly qualified based on academic qualification) are not competent in handling the equipment or laboratories are not maintained adequately including ensuring safety, it does not indicate strength of support staff.

Facilities

Up to date facilities are made available and they exceed the recommended student-equipment ratio appropriate to the relevant discipline. Ergonomics is taken seriously by the institution to reduce occupational hazard.

Safety is observed being practised and evident, for example medicine kept in the safety box is not outdated, safety signage are visible, safety markings are clear and according to standard, fire extinguishers are not outdated, safety items (eye wash, shower, hazardous disposal place/containers, ventilation etc) are available and maintained, and exits are not blocked or doors/grilles locked during sessions. Appropriate safety guidelines are made available and aware by staff and students.

Accessibility to extensive electronics publications for life long learning, project based courses and the final year project, among others, also signifies strength. How is “extensive” defined here? No complaints from staff and students in sourcing publication for the purpose of their research and studies, respectively, imply adequate resources. Any comments of commendation by staff and students signify exceeding the expectation. However, staff must be noted to be actively conducting research and students are stretched and challenged to solve complex problem during the course of their study and especially for their final year project. Otherwise the satisfactory level can be easily attained.

Quality Management System

Planning, implementation, monitoring and improvement are the essential elements of a quality management system (QMS). The QMS should provide appropriate quality control and assurance on the graduate demonstrated outcomes. Having a QMS also provide quality assurance or confidence to direct customer (students) and external stakeholders (industry and government) that the institution/programme would continually improve. If institutions show that they have the plan and implement it, and the completion of the quality cycles is widespread, it implies strength of the QMS. Awareness and monitoring of the QMS also indicates strength.

What constitutes weakness?

Accreditation is a peer review process that should provide institutions with “advice” (on the issues but stop short of providing solutions) so that improvement could be made. Despite the collegial approach to be taken, one has to “call a spade a spade”. Any transgression against the requirements of the EAC Manual has to be clearly reported. The incident and the contravention to the criteria should be highlighted. However, if the incident is isolated and could not be confirmed through a triangulation process, it would not be wise to make a conclusion. It would suffice with making a verbal comment to the institution.

What constitutes weakness? Sometimes evaluators tend to use the word “weakness” freely denoting any part of the policy, plan, activity, resources or system that does not quite satisfy the expected effectiveness. As an example evaluators may make the statement, “the programme has a weakness with regard to exposure to professional practice”, despite the programme has implemented activities related to it but not extensive enough to cover throughout the four (4) years. The word “weakness” here does not mean transgression to the requirement of the Manual. The term “weakness” should only be used when there is a clear transgression against the Manual. Even so it must be properly triangulated before arriving at using the word. Try using other words, such as, “shortcomings”, “improper”, “undesirable” and “dissatisfactory” to denote lapses in adhering to the criteria. Otherwise the outcome decision would be “decline accreditation”, as “weakness” refers to non-compliance to the EAC Manual. This is discussed further in this document (in the section on Decline Accreditation).

Classifying Concern

Usually a “concern” is that the programme/institution has not failed the criteria set under the EAC Manual, but if left unchecked it may lead to failure at a later date. Where there are lapses in observing the criteria of the EAC Manual, it would also appropriately be classified under “concern”. “Concern” can then be divided into two, namely, minor and major in nature, depending on the extent. Listing of the “concerns” without aggregating (triangulation) those that are closely related within the same group may depict a programme as having a serious problem. Whereas, the long list may also comprise extremely minor issues that could possibly fall into the category of “opportunity for improvement”. As such it may give the wrong perception to the Council. Examples of concerns are described as follows under the respective criteria:

Academic Curriculum

One or two courses are found missing or offered as electives in the curriculum, yet these courses are essential in providing a balanced curriculum for the relevant discipline. There is not enough depth/insight on the content for a few courses based on the teaching materials provided and course outline/plan written or from the evidence of direct assessment (e.g. final examination). The semester load is on the higher side for the students, is another possible concern, as it does not allow students to digest learning effectively, unless students are exceptionally excellent. The curriculum lacks examples related to the discipline. For example, a Medical engineering programme must introduce medical related examples.

Student

Students are not able to express themselves and unaware of the importance of sustainability, safety, and professional involvement etc, which reflect the lack in generic attributes expected of them. Lack of understanding on the outcome approach prevents students from approaching the learning effectively. Similarly they are not motivated when the programme objective is not clear, as they would not know where they are heading in the career later. Students’ motivation may also be affected if little or no ‘real world’ examples related to the discipline are covered in the programme.

Academic & Support Staff

The available academic staff are confined to a specific sub-discipline only instead of covering all the relevant sub-disciplines of the programme. For example the staff may be confined to only project management sub-discipline whereas there is no expertise in water, structures or geotechniques, in the case of civil engineering programmes. Many of the staff are not involved in research, and even if they do, research does not permeate to student’s learning. Availability of time to conduct research and involvement in professional activities for the academic staff should be critically evaluated. There is a tendency for institutions to provide a policy statement on this but there is no evidence of such practices taking place. Lack of grants obtained by the academic staff should not be used solely to conclude that research has not taken place. Further triangulation on evaluating the final year projects and through the interviews with staff and students would be needed to confirm.

Facilities

Difficulty of access to information is usually a common issue raised by staff and students at the interviews. However, evaluators need to triangulate with the funding provided for such purpose and the computer facilities available. It may be that the internet connection is poor but the electronic data and information subscribed is extensive. Sometimes this is reflected by the low number of references cited in the final year project reports. However, low citation could also be as a result of poor supervision and/or students’ attitude.

Quality Management System

An institution may have a well designed quality management system but its implementation does not include adequate monitoring. As such, the system is devoid of taking necessary actions (closing of the loop). Academic staff may not be teaching according to the syllabus, examining the students to the expected depth or employing appropriate delivery methods, and yet the institution is unaware of these concerns. Mapping and linking of programme objectives/outcomes to the course outcomes are just on paper and yet academic staff do not understand them and use in the planning. These are all signs of poorly implemented process. Moderation of examination questions/assessment does not capture lack of depth. Feedbacks to students/staff from complaints/comments/queries made are not provided. Even if they are provided, it may be too late in the case of students’ course assignments/assessment feedbacks.

Opportunity for Improvement

Opportunity for improvement refers to further improvement that an institution could consider despite already having the necessary strength or having already satisfied the minimum requirements of the EAC Manual. Institutions would not be penalised for not taking the necessary action to address the issue. However, it would be against the spirit of continual improvement that has been set by the EAC Manual.

Accord or Decline Accreditation

Evaluators are the eyes and ears of the EAC. As such, the comments or recommendations made must be explicit and with clear reasons of the incident of compliance or non-compliance. Otherwise it would be difficult to convince the EAC in their decision making. They are highly dependent upon the evaluators’ report to make their judgement. The presence of the team chair to deliver the report at the EAC meeting eases the decision making process.

Outcome based education (OBE) programmes should have been implemented with the issuance of the 1999 EAC Manual which specified the generic attributes that all programmes should provide. It was reiterated in the 2003 revision without much success in ensuring institutions to adopt the OBE approach. Some institutions began to migrate to OBE approach in a limited manner in 2004 with self interpretation. Generally the movement was slow. This has led to the EAC to include the element of OBE planning/implementation in their decision in 2005. Programmes could only be granted five (5) yeas accreditation provided they show plan for OBE implementation. In 2007 it was stepped up further to see some form of implementation before according accreditation. A widespread implementation is expected, not merely shown on paper or through pilot study, before a programme is accorded five (5) years accreditation, despite satisfying the numeric benchmark of overall 70% marks from the five criteria of evaluation.

Many programmes did not make the five years accreditation as the implementation was at an infancy stage (not significant) or there were serious flaws in the formulation of objectives and outcomes as well as the implementation was not widespread. Further, programmes are expected to make sure that continual quality improvement (CQI) is evident. Two (2) years accreditation thus becomes the standard decision.

“Significant” implementation has been misinterpreted as “effective” implementation by some evaluators. As time went by, there are institutions that had progressed very well but are still not effective in the implementation of the OBE approach. This made the evaluators to interpolate based on comparative evaluation between programmes/institutions, in deciding on the years of accreditation. It must be emphasised that accreditation is not a ranking exercise and therefore comparative evaluation should not be adopted. An institution or programme may perform better than others in many aspects but the fact is they still have the shortcomings. The two year period is only indicative of not meeting the full expectations.

If evaluators feel strongly to allocate three (3) or four (4) years accreditation due to comparative evaluation, then the evaluators should consider giving the full five (5) years accreditation with certain requirements attached and with an interim report to be submitted. The remaining one (1) year continuing accreditation in the case of receiving four (4) years accreditation is not a practical solution. However, the EAC at it Accreditation Decision Meeting held on 26 and 27 November 2008, had adopted a new stand of according three (3) years accreditation with two years (2) extension upon satisfying the conditions given. This decision was preferred to the five (5) years and with an interim report, as it would be difficult to withdraw accreditation if the programme fails to address the conditions satisfactorily.
Alternatively, the current difficulty in obtaining five (5) years accreditation may lead to institutions targeting for a two (2) year accreditation, instead of pushing the limits. Academic institutions, with their honesty and concern for quality would not easily succumb to such practice.

The 21-22 December 2009 ADM, the third and the last ADM for the year, reaffirmed the three (3) plus two (2), two (2), and two (2) with interim report. This time it also added one (1) year accreditation. Previously the EAC refrained from giving one (1) year as it was deemed unreasonably short for programmes to take actions. However, this time the EAC wanted to drive the message of urgency and thus accorded the one (1) year.

Significant implementation of outcome-based education (OBE)

It is expected that the programme/institution has a plan for the OBE approach and also implement it in a widespread manner. Pilot and limited implementation (through several courses) of OBE would not constitute as significant. Ideally efficient and effective implementation should be strived by the programme/institution. However, it is acceptable if the programme/institution acknowledges that the implementation is not effective and is taking actions to address it (CQI). Efficient and effective implementation implies the programme/institution exceeds the expectations (denoting strength).
Decline accreditation or (0) year

The EAC Manual allows evaluators to recommend a period of accreditation between zero (0) to five (5) years. The manual specifies that programmes with minor shortcoming(s) would be accorded less than five (5) years accreditation. Minor shortcoming(s) as stipulated in the Manual can be associated to the terms “WEAKNESS” and “CONCERN” used in the evaluation report.

Anytime “WEAKNESS” is invoked, it refers to non compliance with the minimum standard specified by the EAC Manual, and as such accreditation cannot be accorded, i.e., zero (0) year. Among those that can be categorised into “WEAKNESS” includes programme not meeting the qualifying requirements, curriculum does not provide the breadth and depth of engineering (technology programme or those that are training oriented would fall in this category) and OBE approach is not implemented at all. Similarly, despite excellent or adequate implementation of the OBE approach and curriculum has the breadth and depth, but could not demonstrate that the students are assessed to the expected depth or the students could not attain the expected depth, it would be as a “WEAKNESS”. This has to be widespread and not for isolated cases.

Evaluators must be fully convinced that the programme/institution is not in control of the situation and the outcomes are not evident, before recommending decline accreditation. There is a need to pursue on the problem issue with further investigation and obtain further evidence before making the decision. The programme/institution has to be explicitly clear of the contravention against the Manual or the issue raised by the evaluators.

Five years accreditation

The decision, five year accreditation, is accorded for a programme that has satisfied the minimum requirement of the EAC Manual. There is no shortcoming found except for the continual quality improvement issues (normally associated with Opportunities for Improvement). Generally all criteria would have been at a “satisfactory” or “good” category. Implementation of OBE approach is effective where academic staff are aware and fully implementing it at the course level, together with CQI. Overall the evaluation and CQI of the programme are visible.

Five years accreditation and with an interim report to be submitted or Three years accreditation with two years extension upon addressing the conditions satisfactorily

Five (5) years accreditation with interim report to be submitted is accorded to a programme that has satisfied the minimum requirement of the EAC Manual except for a few “CONCERN” found. Generally all criteria would have been at a “satisfactory” or “good” category. These concerns are isolated and minor in nature. Implementation of OBE approach is significant (widespread) with varied awareness among academic staff and varied implementation at the course level, including CQI. Overall the evaluation and CQI of the programme and courses are visible.

The EAC at its November 2008 meeting, as mentioned earlier, prefers the three (3) years accreditation with two (2) years extension instead of the five (5) years and with an interim report. The difficulty to equate the term “a few” was sighted as to why this decision is a preference, apart from the difficulty of having to withdraw accreditation when programmes failed or not meeting the satisfactory standard upon submitting the interim report.

Two years accreditation and with an interim report may be requested

Two (2) years accreditation and with an interim report to be submitted is accorded to a programme that has satisfied the minimum requirement of the EAC Manual except for the list of “CONCERN” found. Generally all criteria would have been at a “satisfactory” or “good” category. These concerns are mostly related and one/a few are major in nature. Implementation of OBE approach is at infancy or at pilot trials with varied awareness among academic staff and varied implementation at the course level. Overall CQI at programme and courses are not visible.

One year accreditation

One (1) year accreditation is introduced at the December 2009 ADM. It has been accorded to programmes that have not addressed the CONCERNS or shortcomings effectively or where there are numerous CONCERNS. In some cases the qualifying requirements are not fully met such as the provision of external examiners and without IAPs. One (1) year accreditation indicates the urgency of matter. Programmes must take immediate actions to rectify the shortcomings immediately or else losing the accreditation the following year.

Defer Accreditation

Deferment can be invoked to allow the institution to do the necessary corrective action due to non-compliance. Further evaluation would be required to ascertain compliance. The institution may also withdraw the application for accreditation as a face saving option when accreditation decision is deferred. The record of evaluation would then be expunged. The institution may reapply for accreditation when deemed ready. However, the cohorts of graduates that would be considered would be those that are affected through the changes made.

Conclusion

Benchmarks may vary with time. As time goes by there is always “the raising of the bar” to ensure that continual improvement is maintained. Similarly in accreditation the decision is also time based. What may be acceptable now may not be in the future due to needs and relevancy of the situation or environment. Accreditation has its benchmark and in the case of the EAC it is stipulated in the EAC Accreditation Manual. We have seen the evolution of the Manual and it has impacted the institutions considerably. The issues discussed in this document acts as a guide to evaluators as well as the Council members but they should be seen in a wide context or with a helicopter view, to establish the status of the programme as qualifying as an engineering programme, applying the outcome approach significantly and continually improved. It is the professional judgement and confidence that one has on the programme that ultimately tell us that the programme is in “safe hands” and has the necessary elements to provide engineering education.

Reference

1. Engineering Accreditation Council, Malaysia. Engineering Programme Accreditation Manual. 2007

Acknowledgement

The authors are indebted to EAC council members, panel evaluators, academics and all those who have contributed ideas that have been transformed into this document. Their valuable contributions facilitated the authors to lay down some principles that help evaluators and council members to decide and academics to be aware of the rules and interpretation.


Prepared by Megat Johari Megat Mohd Noor1, Mohd Saleh Jaafar2, Wan Hamidon Wan Badaruzzaman2, Azlan Abdul Aziz3, Suhaimi Abdul Talib4
1Director, Engineering Accreditation Department,
2Associate Director, Engineering Accreditation Department,
3Universiti Putra Malaysia,
4Universiti Teknologi MARA