Test My Code: An Automatic Assessment Service for the Extreme Apprenticeship Method - 2nd International Workshop on Evidence-Based Technology Enhanced Learning - page 108

Instruction is usually focused to labs where the students are scaffolded as they
work on course-related exercises. By scaffolding we mean that the students are sup-
ported in a way that they are not given direct answers, rather, just pushed into a
direction to discover the answers themselves. The course instructors focus on pro-
viding high-quality learning material, exercises, and enough guidance to help stu-
dents during their working process. The course material, for example, emphasizes
the working process using worked examples [7] and process recordings [5], both
explicitly describing how a program is crafted using stepwise sub-task division: one
must always start small to grow big. The scaffolding that is built into the exercises
and material makes it easier for the student to proceed and to a favorable direction:
learning achievements are made visible to the student, which further increases the
motivation to continue [6].
As a craft can only be mastered by actually practising it, our semester-length
programming course has over 350 programming tasks, ranging from simple “Hello
World's” to programming a snake game from scratch. A student must complete the
majority of the exercises in order to receive a good grade. In order to verify that a
student has worked on the exercises herself, we organize either a written or an oral
exam at the end of the course.
Most of the exercises are composed of small incremental tasks that combine into
bigger programs. Incremental tasks are used to imitate a typical problem solving
process: students explicitly practise programming but are constantly influenced by
the written-out thought process behind the pre-performed sub-task division. Exer-
cises are intentionally written out to be as informative as possible, and often contain
sample input/output descriptions and code snippets with expected outputs that pro-
vide further support for verifying the correctness of the program. This also helps
students learn to read code written by others.
Scaffolded exercises bring several benefits for the students. Getting started is eas-
ier since exercises are split into smaller tasks. By working her way through several
scaffolded exercises each week, the student starts to see how programs should be
structured in addition to practicing fundamental programming routines. Exercises
are also used to direct students away from bad programming habits such as the use
of unnecessary class attributes and unclear method names.
Since the ultimate goal of university level instruction should be that the stu-
dents can master problem solving by themselves, fading is important. Fading means
correctly timed gradual dismantling of scaffolds [8], and it is realized in our pro-
gramming courses via open ended exercises that do not hint or enforce any kind of
program structure, but only define how the application is supposed to work for a
given user input, e.g. by defining a UI in a relatively strict manner. Before venturing
into open ended exercises, the students have worked through similar problems with
scaffolding.
3TtMyCode
The first documented automatic assessment systems are from the 1960s, after which
a large body of assessment services have been created, see e.g. [9, 1, 12]. Several of