Assessing the Effectiveness of Online Learning Methods: University of Delhi

Updated: Aug 24

Co-Author: Harnoor Kaur


The unexpected lockdown announced by the Government of India on March 21, 2020 with an aim to control the COVID-19 pandemic has had several effects on the education of school and college-going students. In this paper, we examine the effectiveness of teaching and preparedness of the stakeholders at the University of Delhi, and the impact of the lockdown on students currently enrolled in the Undergraduate program at the University. Our analysis draws on a regression study, which uses the data of 181 undergraduate students collected through an online survey. We find that average student participation and teaching regularity drops significantly in an online setting. From the regression model established, efficient curriculum coverage implies a reduction in the probability of adoption of ineffective online methods by 25 percent. A majority of students would prefer on-campus classes, throwing light on the efficacy of existing educational infrastructure.


Introduction


The Coronavirus pandemic has forced the education sector to move entirely to an online setting without a warning. Although the teacher quality, course requirements and material remain identical in an offline and online setting, noticeable differences arise among the two, which are bound to have effects on class participation as well as class performance.

Studies conducted on classroom participation of students in each of the two settings reveal interesting insights. Galyon. Heaton. Best, et al (2015) studied group cohesion, class participation, and student performance in a physical classroom as well as a hybrid setting (which included online teaching and physical examinations). The study found that participation in both settings was high, but group cohesion and student performance were significantly lower in a hybrid setting. Despite identical course requirements, the participants favored a conventional classroom over a hybrid one. Caspi. Chajut. and Saporta (2008) found certain gender differences in participation in the two modes of teaching, where men were more interactive in a physical classroom setting whereas women were more active in participating through written communication in an online setting. Their study also concluded that all in all, an online environment was not conducive for either gender. However, there is a dearth of studies to measure the effectiveness of a sudden transition to the online mode of teaching, forced by the pandemic, in the University of Delhi.


The University of Delhi is a collegiate public central university which witnesses students from diverse backgrounds each year. It offers a plethora of courses, such as language courses, arts courses in subjects including, but not limited to Philosophy, Psychology, Economics, History, Geography, Political Science, Sociology and Journalism, and commerce based courses such B.Com, Business Economics and Business Studies. These majors contain little to no practical elements and can have a somewhat smoother transition to an online platform. For certain other subjects, like Mathematics and Statistics, an online transition is less feasible but can be made possible with proper tools such as an online interface to type mathematical equations and formulae. However, for most scientific disciplines, including but not limited to, Applied Science, Biomedical Science, Botany, Chemistry, Physics, Zoology and Radiography require exposure to experimental lab work and self-practice of the same. For subjects such as Computer Science, Fine Arts, Music and Dance, it is important for students to have the required tools such as software compatible computers along with required software, paints and canvas, musical instruments, and an open space to practice, as these components form an integral part of their course learnings. Subjects such as Physical Education, Vocational Studies and Social Work also struggle with similar issues. Apart from a diverse array of subjects, regional differences among students impact their learning outcomes, as the problem of access to these online facilities comes into.

In this paper, we focus our attention on the perceptions of undergraduate students at the University on teaching effectiveness and class participation before and during the lockdown. The main contribution of this paper is to assess the participation of students in current times along with measuring the effectiveness of online teaching and preparedness in handling the transition to an online teaching environment. This paper is the first step towards acknowledging the problems in the e-learning process, which highlights how this seemingly progressive transition to online learning might, in reality, be a move that takes us back insofar as gender, regional, and income inequalities are concerned.


Methodology


Data Collection


Primary data was collected via an online survey and circulated among students and peers using social media platforms such as WhatsApp and LinkedIn. The survey was split into two questionnaires, one aimed at students and the other towards teachers.. The questions were carefully designed for students to emphasize on environments before and during the lockdown using a linear scale of attitudes and agreement:

  1. Basic description of college, course and year of study

  2. Number of relevant technological devices owned and family members who use them

  3. Pre-lockdown –

a. Participation in the classroom

b. Quality of teaching methodologies

c. Effectiveness of in-person classes

  1. Quarantine –

a. Efficient curriculum coverage

b. Teaching regularity

c. Teaching methods

d. Participation in the online setting

e. Effectiveness of administration grievance counselling

f. Perception of online classes as a permanent substitute

g. Expectations of a future culture of education

h. The overall effectiveness of Online Classes

Other questions of interest included the merits and demerits of using online methods and possible health concerns, both physical and mental.

A crucial objective of designing the survey for teachers was to attempt an assessment of personal attitudes towards the online learning culture using qualitative, anecdotal comments. The following questions were included:

  1. Rating personal comfort with regards to using digital platforms

  2. Effectiveness of communication received from the administration and grievance counselling system

  3. Issues faced with using online methods

  4. Views on the preparedness of students for examinations

  5. Expectations of future education culture

  6. The overall effectiveness of online classes

  7. Comments on a desired course of action to conduct semester examinations

Analysis of collected data includes a descriptive summary along with inferential statistics, and a Multinomial Logistic Regression Model fitted for the student sample using Python and Microsoft Excel.


Theoretical Construction


Inferential analysis of the data involved assessing sample means of ‘Participation in learning’ before and during the lockdown, under a null hypothesis that there is no change in average participation of students over the course of classroom teaching and online platforms, as stated below:

H0: m1 - m2 = 0

(No change in Average Participation)

HA: m1 - m2 ≠ 0

(Shift in Average Participation over the population), where

m1 = Average Participation before the lockdown

m2 = Average Participation during the lockdown

Various factors that shall have a hypothesized effect on the sample, such as stratification of respondents with respect to the course of study, college, year of study, and personal preferences over the college attended. Furthermore, financial and educational infrastructure of individual institutions concerning the objective of the study were not implicit in analysis owing to constraints of data collection brought on by government regulations of strict quarantine and social distancing, as an offline study of the aforementioned would be more effective. Said constraints also lead to the limited reach of the survey among the students of the University, raising questions about the sample chosen being truly representative. Thus, we do not have any assumed or known population variance of the variable for the hypotheses above. A Student’s t distribution was used to formulate the test statistic, with respective sample means as unbiased estimators of the variable under study, i.e. Participation.


Regression analysis as a Multinomial Logistic Model is made using Online Effectiveness as the dependent variable. The ordinal nature of this variable is assumed to have equal placement of the response categories and treated as nominal. Independent variables considered are summarized below:

  1. Curriculum – Efficiency of curriculum coverage during the lockdown

  2. Regularity – Regularity of teaching during the lockdown

  3. Participation – Student participation using online methods

  4. Grievance Counsel - Effectiveness of administration grievance counselling

  5. Future Culture - Expectations of a future culture of education, treated

  6. F, as a dummy variable for Future Culture - 1, if “On-campus classes”, and 0 Otherwise

The model used is a generalization of logistic regression in multiclass problems, where the dependent variable is categorically distributed with more than two responses, as is applicable in this study. The model predicts the probabilities of these categories, initially using a linear predictor function comprised of regression coefficients (weights) and respective independent variables

score (Xi, k) = bk. Xi , where

Xi is the vector of explanatory variables describing observation i, and bk is a vector of weights associated with the variables, and the score is associated with assigning observation i to category k. For all i ranging from 1 to N, there exists a set of M explanatory variables X1,i ….XM,i and a dependent variable Yi with k possible responses, assigning each with a number from 1 to K.


To arrive at the multinomial logistic regression equations for K categories, K-1 equations are formulated with one response category of Yi as the pivot, or reference category, and the remaining K-1 outcomes are individually regressed against the pivot. Thus, we have the following relative probabilities:

The sample was split into individual training and testing data sets to polish the regression model. A confusion matrix was formulated to obtain parameter counts such as true positives and false negatives. Accuracy, Recall, and Precision of the model were evaluated using the confusion matrix.

Results

Descriptive (Annexure 1 for a complete set of responses)

1. Overall Effectiveness – Average response of effective learning across the two periods shifts from “Somewhat Effective” (sd = 0.85) to “Somewhat Ineffective” (sd = 0.86). Regression results described below expand on the same.

2. Participation – Modal responses from classroom to lockdown conditions changed from “Somewhat Attentive” to “Somewhat Inattentive”, indicating a decline in attention span. Testing the hypothesis of unchanged average participation revealed high significance with reasonably low probability (t = 8.712, p ~ 0). Participation has significantly reduced due to a shift from pre-lockdown to online learning methods.


3. Curriculum Coverage – A majority of respondents agree with the absolute ineffectiveness of curriculum covered during the lockdown, which also conforms to a low frequency in the regularity of teaching using online methods as opposed to classrooms ( t = 13.25, p ~ 0).

4. Teaching Methodologies – A majority of lessons conducted are through assignments and projects, followed by self-compiled notes by the instructor and live lectures on online video conferencing applications.


5. Preferences – 76 percent of respondents do not prefer online learning as a substitute for classrooms, and about 59 percent would like on-campus classes in the future.

6. Features of online learning – Students agree on the characteristics of convenience (42%) and self-paced learning (36%). Respondents find difficulties in online methods with respect to a lack of interaction (70%) and issues with network and connectivity (60%). Responses also conformed to various health concerns, such as strain on eyesight (74%), headaches (53%), and physical inactivity.


Regression (Annexure 2 and 3 for the complete model)

The model predicted log odds coefficients for four regression equations, with the category “Extremely Ineffective” of the dependent variable, i.e. Online Effectiveness as a pivot. The regression equation for the probability of choosing “Extremely Effective” relative to the probability of falling in the pivot category is as follows: