Education | ASER findings shocking, but how accurate is survey?

Since 2005, Pratham’s Annual Status of Education Reports – popularly referred to as ASER – have consistently pointed to ‘gaps’ in learning among children in rural India. The recently released ASER 2023 follows in the same path.

ASER 2023, titled Beyond Basics, focuses on the 14-18 year age group – older than the usually younger target age group. It deals with well-known metrics: basic reading; math and English abilities; application of basic skills to everyday calculations; reading and understanding written instructions; and financial calculations needed in real life.

Using a smartphone, the survey team also examined the youth's digital abilities and internet habits by having the youth perform tasks in front of them.

So, what does the survey say? Let’s begin with the downsides.

About 25% of the youth in 14-18 age group cannot read a Std II level text fluently in their regional language

Over half of all surveyed struggle with division (3-digit by 1-digit) problems

Foundational abilities do not always ensure adequate skills to complete real-life tasks. Females perform worse than males on almost all tasks.

That said, some figures don’t look too bad at first.

86.8 per cent of 14-18-year-olds are enrolled in school. Only slight enrolment gaps exist between boys and girls.

90 per cent of youth have and use smartphones at home. More importantly, two thirds say they used it for ‘education-related’ activities like viewing online study videos, solving questions, or exchanging notes.

Nearly half of respondents can calculate time and weights.

Just over half (57.3 per cent) can read English sentences. Of these, nearly three quarters (73.5 per cent) can explain them.

Are there problems with the ASER survey? Without a doubt, yes. Here are some:

Nearly 35,000 youth in 28 districts across 26 states participated in ASER 2023. Strangely, it is far smaller compared to ASER 2022, which surveyed 700,000 persons in 19,000 villages across 616 districts. How can a study of 35,000 in 28 districts claim to represent millions of youth in 806 districts of India?

Consider an analogy. Suppose we do a ‘state of youth’s health’ survey conducted by non-medical persons, across 26 districts in 28 states involving 35,000 youth, focused on just three parameters – blowing a balloon to test lung capacity, body-mass index and eye-sight.

Will you say such a survey represents the health of the age-group population in India? Will any doctor take this data seriously for treating youth’s illnesses?

India is perhaps the most diverse country in the world. Administering a standardised test in rural regions across most states to get a representative sample can be extremely challenging, if not impossible. For instance, youth in some part of the country or the other may not have seen a train, a plane, a television or a computer. How can they know words such as ‘kite’, ‘bat’ or ‘fun’?

Sure, they may know what these words mean in their language or local dialect, but the survey gives them low marks because these are in the passage evaluating English reading. Why should the lack of knowledge of English terms be seen as a ‘deficiency’? When asked in their tongue, they may describe in detail several types of ‘kite’ or ‘bat’ (the animal). The survey, however, will not consider that knowledge.

In the section ‘managing a budget’, for instance, the respondents are given a list of things that can be bought in a shop and asked what all they would buy exactly for ₹50. Now, it is possible that it doesn’t work like this in the respondent’s village. It is common among the rural poor to buy on weekly or monthly credit from small shops; they don’t really settle in full while buying. Asking the youth to spend exactly ₹50 will appear alien unless it is explained.

This is equally true of the question involving a girl going to bed at 8.30 pm and getting up at 4.30 am. The respondent must calculate the number of hours she slept. The picture shows sunrise. At 4.30 am? Unless the person is in Arunachal Pradesh, it not even imaginable. It may, in fact, confuse the respondent.

ASER has always focused on the respondents' knowledge gaps. What do survey respondents know or can do? ASER doesn’t tell us.

Rural youth may outperform even private school students in cities if the tests are about what they know based on the understanding of their ecosystem. Rural adolescents may be very resourceful and give acceptable responses to questions such as what they would do if they were thirsty or hungry in a forest or how to tell if a cow or goat is sick. This may give us a very useful perspective on their ‘abilities’ than what ASER tells us.

ASER says about 40 per cent males and 28 per cent females surveyed reported doing work other than household work, mostly in family farms, for at least 15 days during the preceding month. So, what do we know about their learnings from this farm work? We draw a blank from ASER.

After ASER consistently finds these ‘gaps’, what happens? Pretty much nothing.

Except for one large state attempting to set a target of coming within the top three in the next ASER a few years ago, other states don’t really have strategies to do better in the next ASER. It is like a child being weighed periodically and found malnourished or underweight. There is no follow-up action or strategy for improvement.

While ASER cannot be blamed for not being taken seriously, among many reasons, one can be that there are far more reliable large-scale surveys. Take for instance, the government’s National Achievement Survey (NAS).

It collects data on class 3, 5, 8, and 10 students' learning outcomes in both government and private schools. What’s more, the last NAS done in 2021 surveyed over 3.4 million children from 1.18 lakh schools in 720 districts, dwarfing ASER’s sample size.

Of course, NAS does not provide scores for individual students or schools. However, its findings help compare students’ performance across the spectrum and population and may be a more useful input to move in the desirable direction.

However, there is one agency which markets Learning Enhancement Programmes, Science Kits, Read India Campaign Programmes, Library Programmes, Teacher Training Programmes and more to help states – all for a charge. An indirect benefit can be that they may do better in some part of ASER in the future.

And that is Pratham, which promotes ASER.