A preprint reports that seven usability factors were associated with predicting ratings of M-commerce applications in Pakistan. The model combined effectiveness, learnability, communicativeness, consistency, operability, satisfaction and efficiency, and produced a reported R2 of 0.910. In the paper's terms, that represented 91% of the variation in user-rating prediction.
The result was stronger than that of the study's eight-factor hybrid model, whose reported R2 was 0.335. On the paper's measure of model fit, the selected seven-factor specification performed better on the analyzed data.
The paper also reports an MMRE of 0.0515, an error score used in its accuracy assessment. That was below the stated criterion of 0.25. PRED(25), another accuracy score, was reported as one for all folds, above the stated threshold of at least 0.75.
How the analysis was built
Researchers set out to examine which usability factors were associated with ratings of M-commerce applications and whether regression analysis could predict those ratings. The study focused on customers in Pakistan who used shopping applications to purchase products. The applications named in the analysis were daraz, shophive, home shopping, Symbios and yayvo, with 200 users reported across the five applications.
Participants assessed usability while carrying out specified tasks and answered a quantitative survey using a five-point Likert scale, running from 1, strongly disagree, to 5, strongly agree. Before the survey, the questionnaire was reviewed in a two-round Delphi process involving eight experts. The research-methodology section says that 144 initial questions were reduced to 60 through that process.
The paper reports 200 collected instances, of which 32 were rejected and 168 were used for analysis. It does not provide final application-specific analytic counts, so the reported total cannot show how evenly the retained data were distributed among the five applications.
For model building, the researchers used stepwise multiple linear regression, testing combinations from two through eight factors and examining both R2 and statistical significance. The paper reports significance values equal to or below 0.05 for all seven predictors in the final model. In separate simple linear regression, efficiency was described as the most reliable predictor, while human factors were described as the weakest.
A strong result with a narrow test
The validation used fixed eight-fold cross-validation, dividing the data into eight groups of 21 instances and considering folds from K=1 to K=8. The conclusion also reports significant results across fixed and randomly arranged folds, with R2 up to or more than 0.9.
Those checks provide evidence about performance within the reported dataset, but they do not establish that changing any one usability factor would raise an app's rating. The supplied analysis reports no independent external validation, confidence intervals or other uncertainty estimates. It remains unclear whether the model would work for new users, other applications, other countries or separate rating data.
One reporting issue concerns questionnaire length. The methodology account says Delphi reduced 144 initial questions to 60, while another part says 40 questions were finalized. The supplied analysis also says that the exact definition of the user-rating outcome and its connection to questionnaire scores are not fully described.
Taken together, the findings point to an exploratory rating-prediction model for the named applications and the 168 retained instances. They do not show improvements in sales, retention, satisfaction or other downstream outcomes, and they do not establish superiority under an independent external test. Independent replication with a clearly specified sample, questionnaire, rating definition and external validation would be needed to judge how broadly the model applies.
Paper data and sources
Original title: A Hybrid Usability Approach for Rating Evaluation of M-Commerce Applications
Authors: Ahmad Ibtisam, Bilal Khan, Arshad Ali
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-26
DOI: Not available
Original paper · Full text