The Design and Implementation of an Enculturated Web-Based Intelligent Tutoring System
Phaedra Mohammed Department of Computing and Information Technology The University of the West Indies phaedra.mohammed@gmail.com

ABSTRACT

Accommodating for learner diversity based on cultural backgrounds has not yet been a major personalisation focus until recently. Increasing numbers of Internet-ready devices have propelled e-Learning forward such that these deficiencies in cultural-awareness can no longer remain unattended. Despite being investigated from an instructional design standpoint, the enculturation of digital learning environments has largely been theorized, necessitating manual enculturation by experts, instructors and even students. Consequently, enculturated learning environments are limited in practice. In this paper, a preliminary design for building a web-based Intelligent Tutoring System (ITS) is described together with the features and intended functionality of the various components. This work contributes a practical approach that was implemented and evaluated using two concrete systems within the domain of Computer Science education. An analysis of the findings and empirical evidence reported in the study supports the viability of the approach taken and reveals that intelligent tutoring systems benefit from enculturation.
Author Keywords

considered to be more usable if they were designed without any culture-specific features. However, the development of culturally neutral content and tools is virtually impossible since cultural partiality pervades every design choice. The design of user interfaces, the selection of teaching strategies, the format and content of the educational material all vary depending on the cultural background of the developers [7]. Subtle cultural influences seep into the final product and this can be counter-productive to learning when stereotypes and personal interpretations clash with the practices and beliefs of the students. So, by internationalizing or localizing these systems [12], certain users may be included and others left out thereby working against the goal of providing individualized instruction to any learner at any time. This happens largely because the cultural background of a learner plays a significant role in shaping his/her learning habits, and cultural appropriateness can no longer be treated as an optional personalisation dimension. Despite being investigated from an instructional design standpoint, the enculturation of digital learning environments has largely been theorized, necessitating manual enculturation by experts, instructors and even students. Consequently, enculturated learning environments are limited in practice because many developers have shied away due to the complexity in reliably representing aspects of a particular culture [11, 2]. Young [12] points out that the dearth of culturally-aware ICT systems can also be attributed to the lack of guidance regarding the integration of culture-specific elements into present-day instructional design. Above all, the knowledge and processes for incorporating culture have not been clearly defined with automation in mind [2]. Interactive educational systems are in essence pieces of software and therefore any enculturated approach must be expressed in a well-defined, unambiguous manner. In this paper, a preliminary design for building an enculturated interactive educational system namely a web-based Intelligent Tutoring System (ITS) is described together with the features and intended functionality of the various

Intelligent Tutoring Systems; Culture; Computer Science Education; Software Design; Experimentation
INTRODUCTION

Interactive educational systems deliver instructional content to learners with the intention of providing a customised learning experience. This is achieved through personalization based on a variety of dimensions: learning styles, instructional objectives, devices used for content delivery and so on. Accommodating for learner diversity based on cultural backgrounds however has not yet been a major personalisation focus until recently. Increasing numbers of devices capable of accessing the Internet have propelled e-Learning forward such that deficiencies in cultural-awareness can no longer remain unattended. Educational content and online tools were originally

26

c components. Owing to the lack of compu O utationally-viab ble a approaches for building enculturated sys r stems [11], th his w work contribut a practical method that can be replicated tes c ir rrespective of the instruc f ctional domain chosen. Th n he im mplementation details and tools used to create tw n d wo c concrete syste ems based on this archit tecture for th he C Computer Scie ence domain ar discussed. The results of an re T e experimental study and qua s alitative evaluation conducted u using the syste are outline An analysis of the findin ems ed. s ngs a and empirical evidence rep ported in the study is don ne together with a discussion of the significanc of the results. f ce T The paper con ncludes with a summary of the research c contributions made and the future plans fo improving th m f or he p prototypes and advancing the research ideas expressed. s
E ENCULTURAT TED SYSTEM DESIGN AND COMPONENT D TS

CULTU URAL HEURIS STICS

E Enculturated in nstructional so oftware system must satis ms sfy s several require ements, and de esign decisions were made in o order to satisfy these require y ements. A mod dular design was w c chosen because of the compl e lexity involved in delivering a d g c culturally-relev vant instructi ional experience. Flexib ble a alteration and improvement of the compon i o nent features are a e easily accomm modated as a re esult. Many of the componen nts f featured in the design of the enculturated web-based IT e e TS a based on the following traditional ITS components: a are t S s student model, a domain mod and an expe model [8]. del, ert
Figure 1. Enc culturated web-based ITS syste architecture em e

A cult turally-relevan instructional approach req nt l quires that cultura references m al made by the so oftware system should be m applica able to the lea arning content, familiar to th students, he authen ntically rendere and integr ed, rated into the context of the ins structional mat terial [4, 6]. C Cultural rules m modify the textual portions of i l instructional c content such a question as descrip ptions, scenari ios, hints, and instructional feedback d l produc by the we ced eb-based syste The encult em. turation of visual portions of th hese systems, namely the m multimedia related to the learnin activities, i also handled by these d ng is rules. The textual m modifications include custom mizing the langua age of the i instructional feedback, wh hereas the multim media encultur ration involve swapping i cultural es in assets for generaliz zed assets su uch as images. Textual outputs of the encultu s uration process are sentences expressed s s in a c cultural dialec specifically mesolect fo ct y orms1, and equiva alent cultural lexical terms. The cultural target is . l determ mined by the st tudent’s cultur background (from the ral d student model). t

A client-server approach is ta aken. A cultura student mode al el, c cultural heurist tics, a content repository, an a pedagogic nd cal m module make up the major ar u rchitectural uni of the desig its gn. F Figure 1 shows how these co omponents are connected in a e n c cohesive system and it also shows the ex m xchange of da ata a amongst the co omponents. Web related com W mponents such as s server softwar and conte re ent aggregator will not be rs b d discussed since these are stan e ndard in web-ba ased systems.
C CULTURAL ST TUDENT MODEL

A student mode serves the tr el raditional purp pose of recordin ng th student’s kn he nowledge leve learning ac els, chievements, an nd learning goals. It stores logs of the pedag s gogical events in th web sessio The studen model store performanc he on. nt es cer related data such as pre erequisite kno owledge, topi ics c completed, su ubmitted answ wers, questio ons complete ed, s successful and failed attempts at learning ac s ctivities, numb ber o attempts, tim taken, sug of me ggested hints and instruction a nal g guidance give to the stu en udent. Econo omides [3] an nd B Blanchard et al. [1] recomm a mend storing cultural learne err related data in a student model and co n m onsequently th he c cultural learne model was assimilated into the stude er i ent m model since th are inhere hey ently related. The model also tr racks the pla ayer’s interact tion with the software, an e nd r records informa ation related to how the syste is being used o em s such as dwell time on areas of the screen for example. It is t o fo e essentially a snapshot of the player f r’s education nal e experiences. 27

CONTE ENT REPOSIT TORY

The c content reposi itory handles the organis s sation and distribu ution of all w web-related, in nstructional, an cultural nd assets to the web c content aggreg gator. Encultur rated webbased I ITSs rely on re eusable conten more than non-cultural nt ITSs b because of th additional dimension o cultural he of person nalisation; this was the basi for having a separate is asset repository - reusability. T The content repository primar hosts all o the educatio rily of onal and interf face-related materia used by th system and the student m al he model. For A v variety of lang guage in a Creole continuu that is um interme ediate betwee the standa form (acr en ard rolect) and forms t that diverge gr reatly from the standard form (basilect). m
1

example, multimedia files related to the interface’s look and feel, such as icons, logos, and those related to the learning exercises (scenario pictures, feedback pictures) are stored here together with educational material such as question descriptions, solutions, feedback files, topic hierarchies. Each of these assets is described by their asset metadata. The metadata descriptions define the context of use and the nature of the assets and are indispensable in the design because they facilitate reuse and exchange of compatible assets. Both the pedagogical module and cultural heuristic component use these descriptions when making instructional and enculturation-related decisions.
PEDAGOGICAL MODULE

student, the learning materials need to follow a template used internally or provide metadata descriptions that identify the corresponding parts identified in Table 1 below.
SYSTEM IMPLEMENTATION

Two web-based ITSs were implemented based on the software architecture described in the previous section. One system was enculturated for a Trinidad and Tobago context (Culturally Relevant Instructional Programming System – CRIPSY) while the other remained generic (Instructional Non-CulturAl Programming System – INCAPS). Both systems were built for the same educational domain, Computer Science programming, and were identical from a functional standpoint. The systems were implemented using Java-based tools and technology which facilitated seamless integration of the various components into one complete system. At the presentation layer, HTML, javascript, and css were used to create the web pages and graphical user interfaces. The Dynamic Web Content Aggregator was implemented using servlets that rendered and formatted the web pages. Additional servlets were used to deliver required functionality such as validation, verification of input data, and for handling log-ons and session management. Apache Tomcat 6.0 was used as the web server environment. The pedagogical module, student model, and cultural heuristic component were implemented using JESS (Java Expert System Shell) rule engines and rules. Intermediary java programs were used to connect each of these self-contained units. At the data level, simple file formats were used to manage the content repository since rule engines handle data manipulation primarily using facts. The development of enculturated assets for CRIPSY was done semi-automatically and manually. Computer Science (CS) education stresses the importance of analytical programming skills [5] because a large part of the curriculum involves reading, planning, and writing computer programs. These skills include being able to understand code written by other people such as libraries and full programs, and being able to detect and repair errors in the syntax and logic of program code. Proper development of these skills requires rigorous practice sessions with written problem sets in the form of code. Basic skills of understanding program code and detecting and repairing syntactical and logic/analytical errors were targeted using code snippets related to topics on the programming curriculum of the target student audience. Constructivism and situated cognition was selected as the major instructional strategy since analytical programming skills were being targeted and also because of the good fit between situated cognition and culturally-aware instruction The programming exercises’ descriptions, parts of the exercise code and instructional hints were enculturated using subtle, careful use of cultural semiotics, specifically familiar language and cultural names of objects and foods. As shown in the screenshots in Figure 2 above, both systems used the same instructional content but differed in 28

The pedagogical module serves the same purpose as the expert model in an ITS but was separated from the student model for a cleaner design. Instructional rules constantly access and update the student model in response to input data from the student and events captured from the screen. They control how and when instructional feedback is given, and they manage the selection and transition of the learning activities. Any on-screen feedback given is stored in the student model so that records of student experiences are kept up to date. Name of Variable ID Topics_Tested Multimedia* Skill_Level Description of Learning Activity Variable Unique identifier Topic(s) tested or covered in the learning activity Filename(s) of multimedia content used in the activity The skill level that a student must possess for successful completion of the activity Skills or knowledge that a student should possess upon successful completion of the activity Descriptions, instructions, and/or scenarios that guide the student and set the context of the question Content that the student must manipulate, modify, or select from. Model answer/content expected from the student Further guidance corresponding to different parts of the model answer

Learning_Objectives

Question_Description*

Question Answer Hints*

Table 1. Learning material template used by the pedagogical module and cultural heuristics

In order for these rules to properly scaffold the learning activities and determine appropriate feedback for the

th expression of the content, that is, cu he ultural and no onc cultural. A min nimalist interf face design wa used in ord as der n to distract the students and to increa ease of us not ase se. I Instructional feedback con nsisted of id dentification of c correct/incorrec lines of code, hints for the incorrect line ct e es, a general gui and idance.
F Figure 2. Screenshots of CRIP PSY (top) and IN NCAPS (bottom m) lturated and no on-cultural vers sions of the sam me featuring encul programming exercise.

After 3 minutes, th sessions tim 30 he med-out autom matically to ensure that both gr roups used the systems for the same e r duratio A timed po on. ost-test was the administered followed en d, by an evaluation su urvey. At the end of the e experiment, usage l logs were retrieved from the server machin nes.
LTS RESUL

Initial examination o the pre-test (P1) and pos of t st-test (P2) scores indicated p positive chan nges in the post-test perform mance of both groups of s h students as illu ustrated in Table 2 below. Paire t-tests condu ed ucted on both s of presets test an post-test sc nd cores revealed statistically significant d increas in the stu ses udents’ programming skills after they used th systems. Ta he able 2 shows t that the differe ence in the post-te scores of th test group (C est he CRIPSY) was h higher than that of the control group (INCAP f PS). It should be noted d howev ver, the differ rence between the groups’ scores is n modest and not statis t stically signific (p= 0.3975 cant 5) Grou up CRIPS Y INCAP PS N 30 30 P1 8.0 0667 4333 8.4 P2 10.100 00 9.900 00 Diff 2.0333 1.4666 T-Test 0.0001 0.0045

Table 2. Changes in pre-test and po e ost-test scores fo the test or SY) PS) group (CRIPS and control group (INCAP

E EXPERIMENTA DESIGN AN RESULTS AL ND

A study was done to evalu uate whether the enculturated t s system, CRIPS SY, was mor effective th re han the contr rol s system, INCAP for increa PS, asing analytica programmin al ng s skill, and to gauge student opinion and interest in th g t he c culturally-enha anced approach taken. h
P Participants

Overal the students had roughly equivalent av ll, s verages for time-on n-task with th programmin exercises f the test he ng for and co ontrol groups. Greater varia ation was foun for the nd numbe of correct, incorrect and total attempt made at er ts comple eting the exe ercises betwee the test an control en nd groups as shown in T s Table 3 below. CRIP PSY Correct Q Questions Correct A Attempts Incorrect A Attempts m = 1.8 8077 s.d.= 1.4 4148 m = 9.0 0385 s.d.= 5.2 2267 m = 26 6.2692 s.d.= 12 2.3792 INCAPS m = 2.2174 s.d.=1.9059 m = 11.21 174 s.d.= 7.7222 m = 37.52 217 s.d.= 18.34 424 m = 48.73 3913 s.d.= 5204 22.05 m = 25.58 8684 s.d.= 3995 6.373

S Sixty (60) stud dents, 31 male and 29 females, enrolled in es th first year computer programming course at th he r p he U University of the West Indies (U.W f W.I.) voluntari ily p participated in the study. Ag between 18 and 47 yea ged 1 ars ( (mean=20.983, s.d.=4.835), 23.3% were of African descen 2 nt, 4 41.6% were of East-India descent, 1.66% were of an 1 C Caucasian desc cent, and 33.3% were of mixe ethnicities. % ed
P Procedure

To Attempts otal

m = 35. .3077 s.d.= 16 6.2327

T students were randomly assigned to a control grou The w y up ( (n=30) and a test group (n n=30). A timed pre-test was w a administered to both groups, then the stude logged on to o ents th test server in the Com he rs mputer Scienc laboratory at ce U U.W.I. Each student was given a unique username an s g e nd p password that activated the re a espective pre-a assigned system m. T student wa therefore un The as naware of whe ether the syste em w cultural or non-cultural prior to loggi on. The te was r ing est g group used CR RIPSY and the control group used INCAP e p PS. 29

T Time-on-task

m 23.52745 s.d.= 6.041183

=

Table 3 Summary sta 3. atistics extracted from session log files for the test group (CR e RIPSY) and con ntrol group (IN NCAPS)

Analys of the logg session dat revealed pos sis ged ta sitive, very signific cant linear co orrelations (p< <0.01) between the total n time sp pent on the e exercises in th systems and the total he d numbe of attempts made for bot the test (r=0 er th 0.584) and control groups (r=0 l 0.519). Strong extremely significant g,

correlations (p<0.001) were also found between the total number of correct attempts made and the total number of exercises completed successfully for both the test (r=0.899) and control groups (r=0.932). The total number of incorrect attempts and the total number of exercises completed successfully were very significantly correlated for the test group (r=0.519, p<0.01) but only weakly related for the control group (r=0.102, p>0.1). A strong significant correlation was found between the positivity of the system rating given by the test group students and the total number of attempts they made at the exercises (r=0.589, p< 0.01) whereas a weak negative correlation existed for the control group (r=-0.097, p>0.1). In the subjective assessment survey, 56.2% of the test group students rated CRIPSY as ‘really good’ and ‘pretty good’, 42.3% rated it as ‘ok’, and 11.5% rated it as ‘not good’. The most popular reasons for liking CRIPSY were helpful hints, interesting and funny problem contexts, encouraging understanding of programming errors, and the use of cultural language. The most common reasons for disliking CRIPSY included server glitches, lack of flexible answer formats, unreadable or “too much” cultural language, and confusing problem contexts. 43.5% of the control group students rated INCAPS as ‘really good’ and ‘pretty good’, and 56.5% rated it as ‘ok’. None of the control group students rated INCAPS as ‘not good’. The most popular reasons for liking INCAPS were interesting and funny problem contexts, encouraging understanding of programming errors, and helpful problem descriptions. The most common reasons for disliking INCAPS included server glitches, lack of flexible answer formats, unreadable cultural language, and distracting problem contexts. The difficulty of the exercises received mostly similar reviews. Around 4% described the exercises as challenging from both groups. 74% of the control group and 76% of the test group found the exercises to be neither easy nor difficult. The remaining 22% of the control group rated the exercises as easy compared to 16% of the test group. Interestingly, 4% of test group gave a rating of ‘very easy’. The instructional feedback was identical in both systems with the exception of the enculturated language used in CRIPSY. Students rated the feedback in CRIPSY as helpful (85.7%), encouraging (10.7%) and useless (3.6%) whereas INCAPS was rated as helpful (69%), encouraging (24.1%), and confusing (6.9%).
ANALYSIS OF RESULTS

seamless integration intended amongst the cultural software components and the traditional ITS components was achieved since similar usage patterns were found in both the test and control group session logs. Cultural elements did not detract away from student behaviour commonly reported for ITSs since similar correlations were found for between the total time spent on the exercises in the systems and the total number of attempts made, and between the total number of correct attempts made and the total number of exercises completed successfully for both experimental groups. The enculturated system however performed slightly better compared to the control system since the total number of incorrect attempts and the total number of exercises completed successfully were strongly correlated. This implies that both correct and incorrect attempts made with the enculturated system proved to be beneficial for completing the exercises. Making attempts also seemed to be encouraging for students in the test group since the positivity of their system ratings increased with the number of attempts made. This relationship was weaker for the control group which suggests that the culturally enhanced system may indeed have promoted a more relaxed learning atmosphere as theorized by proponents of culturally-aware instructional environments [6,7]. Both systems suffered from parallel problems related to server glitches and software bugs. Inflexible answer formats understandably caused confusion for both sets of students since correct alternative programming solutions were not accepted by the pedagogical module. This resulted in some students being blocked from advancing to the next exercise despite having correct answers. Surprisingly, these problems did not decrease the appeal of the tutoring systems because students were eager to have access to the systems when the experiment was over. Another design issue that affected students was the static cultural language used in the enculturated system. For some students the density of the cultural terms used distorted the text and made the sentences difficult to read quickly and therefore difficult to understand quickly. An initial suggestion of incorporating a customizable language density scale was proposed as way of dealing with this problem so that users may adjust the language to suit their own preferences. This feature was immediately confirmed as desirable by the students. Overall, the students liked the enculturated system primarily because of the reasons outlined in earlier studies [9, 10] enriching learning experiences and humour. The use of culture created a familiar setting and it was done in a way that was interesting to the students. Although both systems employed an anchored instructional approach, a larger percentage of students rated the programming exercises as easier for the enculturated system compared to the control system. The instructional feedback/hints were also deemed to be more helpful by the students using the enculturated system again despite providing similar guidance to those used in the control system.

The increases in test scores for both groups were significant and provide strong evidence for the success of Intelligent Tutoring Systems in increasing student performance. Although only marginally larger, the group that used the enculturated system, CRIPSY, produced larger gains compared to control group. This confirms the assumption that cultural interventions do indeed have positive effects on learners [6, 7] and provides empirical evidence in support of enculturated learning systems. In addition, the 30

CONCLUSION

Culture is rapidly becoming an important consideration in the design of eLearning software firstly because of the increase in the number of users accessing software over the Internet, and secondly because of the sheer diversity in the cultural backgrounds of these users. Conventional learning has often taken place in a localized setting with a teacher guiding one or more students in their search for knowledge. With the advent of the Internet, this traditional setting has changed drastically since students now have access to teachers and educational material from over wide distances. Consequently, these students are exposed to a variety of educational tools, teaching strategies and learning materials which were not developed with their own personal needs in mind. This has dramatic usability implications especially when the mainstream culture for which e-Learning materials are designed clashes with that of the users. Based on the encouraging evidence established by the study, the research discussed in this paper demonstrates a practical approach towards developing an enculturated webbased learning environment. By leveraging research from various fields such as Intelligent Tutoring Systems and culturally-aware instruction, this work shows how the complexity of enculturation can be managed and how aspects of intelligent tutoring can be enculturated. Empirical evidence indicates that enculturated systems perform as well as traditional tutoring systems and are potentially superior at creating relaxed, engaging learning atmospheres for the Computer Science programming domain. However care must be taken to ensure that the cultural enhancements match the tolerance level of the student users. Further refinement and improvements are planned for the systems described. A limited amount of cultural automation was undertaken, so expansion of the cultural coverage is necessary. Additional features such as deeper cultural learner profiling, adjustable language density, and greater tutoring flexibility are also part of the plans intended for this research.

Aware Tutoring Systems, Montreal, Canada, (2008), 2334. 3. Economides, A.A. Culture-aware collaborative learning. Multi-cultural Education & Technology Journal 3, 4 (2008), 243-267. 4. Fleer, M. Reflecting indigenous culture in educational software design. Journal of Reading 32, 7 (1989), 611619. 5. Gray, S., Edwards, S., Lewandowski, G., and Shende, A. Improving student programming skills by developing program comprehension abilities: Panel discussion. Journal of Computing Sciences in Colleges 20, 3 (2005), 235 – 237. 1. Henderson, L. Theorizing a multiple cultures instructional design model for e-Learning and teaching. In Edmundson A. (ed): Globalized E-Learning Cultural Challenges, Information Science Publishing: London, UK (2007), 130- 154 2. McLoughlin, C. Adapting e-Learning across cultural boundaries: A framework for quality learning, pedagogy, and interaction. In Edmundson A. (ed): Globalized E-Learning Cultural Challenges, Information Science Publishing: London, UK (2007), 223- 238. 3. Mills, C., Dalgarno, B. A Conceptual model for gamebased Intelligent Tutoring Systems. In Proc. Ascilite http://www.ascilite.org.au/conferences/singapore07/proc s/mills.pdf (2007) 4. Mohammed, P., and Mohan, P. Student attitudes towards using culturally-oriented educational games to improve programming proficiency: An exploratory study. In: Edutainment, LNCS 5670, Springer-Verlag: Berlin (2009), 196-207. 5. Mohammed, P., and Mohan, P. Combining digital games with culture: A novel approach towards boosting student interest and skill development in Computer Science programming. In: 2010 Second International Conference on Mobile, Hybrid, and On-Line Learning, IEEE Computer Society (2010), 60-65. 6. Rehm, M. Developing enculturated agents- Pitfalls and strategies. In E.G. Blanchard, D. Allard (eds): Handbook of Research on Culturally-Aware Information Technology: Perspectivies and Models. IGI Global (2010) 7. Young, P. Integrating culture in the design of ILTS. British Journal of Educational Technology 39, 1 (2008), 6-17.

REFERENCES

1. Blanchard, E., Razaki, R., Frasson C. Cross-cultural adaptation of e-Learning contents: A methodology. In: Richards, G. (ed.): ELearn 2005, AACE (2005), 18951902. 2. Blanchard, E.G., Mizoguchi, R. Designing culturallyaware tutoring systems: Toward an upper ontology of culture. In E Blanchard, and D Allard (eds.), Culturally

31