Z Score and Advanced Level results: The intricate story

By Ranga CHANDRARATHNE

The release of results of the GCE Advanced level examination was delayed this time due to the controversy in calculation of Z score from two different examinations under two different syllabi held in August. The responsibility for the delay was thrown from the Department of Examinations to the University Grants Commission back and forth resulting in the students who sat for the examination to have sleepless nights.

The issue appears to be a statistical problem which has not been resolved quickly even by the panel appointed by the UGC. The Sunday Observer interviewed an eminent scientist who has specialised in Spatial Statistics for a clarification and answers for the unresolved problem which determines the fate of students in the university entrance. Prof. Ranjith Premalal De Silva, Vice Chancellor of Uva Wellassa University and author of the text book on 'Spatial Statistics: Theory and Practice' and several other publications related to spatial statistics, joins the discussion to answer our questions.

Q: Can you explain the background and the reasons led to the introduction of Z score for university admission in place of simple aggregate mark used previously?

A: There is a diverse degree of variability in the nature itself so is with the humans and their work. The scientific discipline called statistics provides the conceptual and procedural approaches to understand the variability and manage it in a fair and equitable manner. Due to the inherent variability present in question papers and evaluations, it was very well understood that the simple aggregate obtained from the marks of different subjects is not a fare criterion to determine the eligibility for university admission.

The degree of difficulty of the question papers in different subjects varies significantly and also there are variations of the aptitude measurement scale in different years. Therefore, it was not possible to have a fixed cut off mark or simple aggregate solution to determine eligibility for university admission.

For example, marks of one subject could have a range from 5 to 70 while the range for another subject spans from 45 to 99. A student who obtained 70 is the best among the lot in the first case and the one who obtained 70 in the second case is a mediocre student. When the simple aggregate is taken for admission, the best student in the first case is treated equally to a mediocre student who opted for a relatively easy subject as in the second case. Within the general assumption that students who sit for these two different subjects have more or less equal level of competence, the disparity in relative difficulty of the question papers or evaluation criteria definitely warrants some sort of an approach for standardisation.

This issue was the theme for a series of discussions in academic circles and in spite of several advanced methodologies available for standardisation, Z score approach was adopted about ten years back as the criterion for university admission due to its simplicity, computational convenience, application efficiency and the need for only two parametric estimates for its calculation.

Although the Ministry of Higher Education and the University Grants Commission took the initiative to device the methodology and introduce it for university admission, it is unfortunate that the school teachers and students have not been educated properly on the use of Z score standardisation approach. This has led to the circulation of several myths and misinterpretations in schools and even most of the university students (more than 90% is a very conservative estimation) are not aware how this magic number was calculated in the selection process.

Q: If the calculation of Z score involves a simple methodology, can you explain how it is computed?

A: The distribution of marks of the students in a given subject is generally arranged to take the theoretical range of 0 to 100. The calculation of Z score in fact does not need to have 0 as a lower bound or 100 as the upper bound. It can be used to standardise any range of values. Taking the individual mark of each and every student in a given subject, the simple average of marks (mean) and the standard deviation of the marks are calculated. These are the two parameters used for Z score calculation.

Secondly, from each and every mark of the students, the mean or the average value is subtracted and divided by the standard deviation. The answer of this calculation provides the Z score corresponding to each mark. Although, the calculation of the Z score is easy, its interpretation should be done with caution since it transforms the marks having a finite, discrete range to a new continuous distribution having positive and negative infinity as the upper and lower bounds.

However, most of the values are found around the middle of the Z distribution, which is 0 and 99.9% of the students have z score ranging from -3 to +3. Also the actual calculation could lead to Z score having several decimals, truncating to 4 decimals would not make any serious issue in locating the relative position of the students. Since the Z scores are standard values, simple the total of z scores in the three subjects is a fair determinant of university admission qualification.

However, it should be emphasised that Z score is not a substitute for district quota system and the disparities in accessing the education facilities are intended to be compensated through the district quota system although there are serious issues related to fairness in allocating university admission quota to districts based on district populations.

Q: If you have serious concerns on district quota system for university admission, would you have a better approach to resolve the issue of unfairness?

A: We still have a school-centered education system for Advanced Level students. Although tuition plays a considerable role, it is not possible to restrict individual's choice for tuition. Within a given district, there are schools having a wide disparity in their teaching and learning resources which are made available to the students in the school.

Although there are very good schools with all the facilities and having excellent teaching and learning environments, there are also resource poor schools within a very close proximity to the good schools. As far as the university admission criteria based on district quota is concerned, the students of resource rich and poor schools have to compete for the same admission slots made available for the same district. This is not a level playing field.

District quota system is based on the underlying assumption that all schools in a given district have the same standards of education facilities and it is obvious that this assumption is not realistic and hence, the quota system is also not fair. The schools should be categorised based on the available teaching and learning resources available evaluating the related attributes of schools and a stratification or a grading system should be introduced to rank the schools. The quota should be awarded for university admission not to districts but to each category of schools depending on the total number of students sitting for Advanced Level in each year from each stratum of schools. Private candidates should also be affiliated to their original school of education. A quota system based on the school classification is very much fair and equitable measure of the aptitude of the student for university admission.

Q: What is the issue for the delay in releasing results of the Advanced Level examination this year?

A: There were two separate examinations held this year for Advanced level based on old and new syllabi. For each subject, there have been two sets of questions papers one for old syllabus and the other for the new syllabus. The Department of Examinations has no intellectual capacity to determine how the Z scores should be calculated to determine university admission from these two separate examinations on a single continuous scale.

It is not fair to blame the Department of Examinations for not having the capability and capacity to propose an acceptable solution to the problem, yet they cannot be excused for not realising the problem until they were about to release the results.

The arrangements to hold two separate examinations were initiated more than 6 months ago and this problem should have been brought for discussion several months earlier rather than later, putting the innocent students in dilemma due to the delay in releasing the results.

The Director General of Examinations forwarded the question to University Grants Commission at the last minute expecting an immediate solution.

The Chairman of UGC, being a veteran academic wanted to have a consensus solution from a group of experts.

The Minister of Higher Education also demanded a solution which is fair and just for everybody and also emphasised the urgency in finding this solution.

An expert committee was appointed but UGC did not receive a prompt solution.

However, UGC was advised to request Z-score marks of the candidates in three different ways i.e. one for the old syllabus, one for the new syllabus, and a common Z-score for both the categories. The error free calculation marks with several cross checks in three different formats obviously take time and that made the delay in releasing the results.

Q: As a Professor involved in teaching spatial statistics, can you propose a fair solution to this problem?

A: It is not at all difficult to propose a solution for this problem provided you have the necessary conceptual and theoretical background and a sound understanding of the education system in Sri Lanka. I do not believe that anybody could propose a fair solution to the problem only reviewing marks in the requested three different formats. Let me explain this in detail. In exploring a solution to this problem, the underlying assumptions made in calculating the Z scores need to be re-examined.

One of the assumptions is that the aptitude of knowledge and skills of the student population opting for different subjects or subject streams are not significantly different.

This assumption should be validated now because almost all the students in the old syllabus except for those with valid medical or other reasons have made a failed attempt to enter university earlier. Those who qualify from the new syllabus are the fresh group who took the examination for the first time. Assuming the equal knowledge and skill background for these two diverse groups obviously introduce an error into the fair selection process.

Before processing marks, the hypothesis of statistically non significant difference between these two student groups needs to be validated with at least past data for a period of 10 years.

If this hypothesis cannot be proved, then the degree of dissimilarity needs to be assessed in order to determine a fair ratio for university admission from these two groups of students.

The second issue is whether the relative difficulty of two questions papers in both old and new syllabi is significantly different.

It is obvious that the calculated z scores are of no use to propose a solution for this problem and it is required to analyze the raw marks of the old and new question papers to make a mean comparison. If means are found to be different, then an adjustment needs to be made to equalize the two sets of marks.

The subject matter expert's opinion is also a choice that can be considered in decision making. In fact, in the fundamental level of statistics courses, we give a lesson to our students that they should propose a valid statistical methodology before they collect their data for research to ensure the compatibility of data with the statistical techniques to be employed in the analysis.

Unfortunately, the Department of Examinations could not stick to this advice and only seeks statistically valid solution after collecting the data from student examinations.

Once the data are collected, it is not possible to employ the best methodology but to reach a compromise to find a reasonable solution without creating a serious injustice to any group.

Reviewing the situation from its apparent background, it is obvious that the Z score calculated from the data combined from old and new syllabi or individual Z scores for each examination as requested by the expert panel have no practical relevance or validity in determining the university admission.