Test My Code: An Automatic Assessment Service for the Extreme Apprenticeship Method - 2nd International Workshop on Evidence-Based Technology Enhanced Learning - page 107

It is important that any work done is as “genuine” as possible as one should not
have to later unlearn learned skills and practices [4, 10]. In our context learning to
program starts with setting up an industry-strength programming environment that
professionals use, which emphasizes that the learners are on a path to become true
practitioners starting from day one.
Assessing the students' work when each student can work on over 30 program-
ming exercises during a single week is laborious. Although most of the effort can
be efficiently spent [14], we have noticed that a noticeable amount of time goes into
trivial scaffolding. In early XA courses, the practice was that when a student felt that
her exercise was done, it was manually checked by a course instructor and marked
down in a course logbook [17].
It is evident that XA can benefit from the use of automated tools. In order to focus
the advisors' time on purposeful scaffolding, a minimally intrusive assessment sys-
tem that supports XA workflow is essential. Our main requirements for the assess-
ment system were that the system (1) should not introduce any additional overhead
to the students' working process, (2) should be integrated to an industry-standard
programming environment, (3) must allow building of scaffolding into an exercise,
(4) must allow awarding points from completing smaller goals, not just from a com-
plete exercise, (5) should cause no additional overhead to course instructors from
the management perspective, and (6) should allow honing of software engineering
practices for the course personnel as well.
The solution that we describe in this article removes the need for trivial exercise
checking, provides basic scaffolding capabilities, and allows the instructors to focus
more on their actual work - mentoring and supporting the students. The system is
integrated to an IDE, and allows dissecting the students' working process even if
they cannot be present. In order to verify that TMC can be used in long-distance
education, we have used it in our introductory programming (CS1) courses as well
as in MOOCs in programming [16].
This document is organized as follows: Section 2 gives a more detailed explana-
tion of XA in the context of programming courses. Section 3 describes the Test My
Code-server from both students' and instructors' perspective, and section 4 gives an
explanation on how exercises can be crafted for TMC. Section 5 gives an overview
of our initial evaluations, and in the final section we describe future work that is
planned to further improve TMC.

2 Extreme Apprenticeship Method

The Extreme Apprenticeship (XA) method extends Cognitive Apprenticeship [8]
by bringing in practices from methods such as Extreme Programming (XP) [3], and
emphasizes students' personal effort and bi-directional communication between the
learner and the advisor. Core values in XA are [17]: (1) A craft can only be mastered
by actually practicing it, for as long as is necessary, and (2) continuous feedback
between the learner and the advisor. The advisor must be aware of the successes
and challenges of the learner throughout the course, which allows the advisor to
provide better scaffolding and further improve the course material.