June 6, 2014
Thoughts on Thesis Testing

I just completed my first ever major user test of a self-developed system - in this case, the educational system that I am developing for my master's thesis. I won't go into the gritty details both to avoid being pedantic and because I am not ready to release these yet, but the basic idea is a visual programming system centered around writing algorithms to drive a robot through a maze...the goal of which is to be used to teach students to both create and understand algorithms, and to understand fully the fundamental statements in programming (if statements, variables, etc.) The test took place during ASU's annual Robotics Summer Camp - I was allotted three hours of the first three days to present my system to the students, to give them exercises to do, and to test them. The students were all 7th graders as well. Hopefully that sets the scene. I am happy with how the test went. Again, I won't go into details but we did see relatively impressive performance over the course of three days. I tried to design my pretest / posttest to be applicable to the general domain of programming logic / computational thinking, and I hope that this performance increase will be convincing that my program does in fact help students learn these fundamental concepts. Still, perhaps the more important takeaways from the camp are my observations and ideas on teaching these fundamental computational thinking / algorithmic development skills. Thoughts like the following...

  • These concepts are quite cerebral. They must be introduced slowly, explicitly, and clearly.
  • A single exercise or assignment is not enough...students must have many exercises before they can internalize a concept.
  • Students tend to attempt to solve problems by copying code from examples, throwing as much code into the editor window as they could, and then modifying it until it worked. They didn't bother to actually read what they were doing. I have read similar reports on young students developing in Scratch.
  • Due to the above, students must be forced to walk through a series of steps that will get them to really think about what algorithm they are developing. For example, to get students to see how to use loops effectively, we might force them to write out procedural code to drive the robot around, and then ask them to reduce the length of that code using a loop. This will force them to see the benefits of loops directly.
  • These concepts absolutely must be taught individually...we don't allow students to learn their times tables in a group, yet groupwork is highly prevalent in CS classes, from beginning to advanced. There are of course practical reasons for this, but they must be overcome if we're going to truly teach students these concepts.

I have some other thoughts, perhaps more lighthearted, about running tests...

  • Permission slips should be sent out far in advance of the camp...I really goofed on this one.
  • Of course we need a computer to teach students computer programming skills, but they function as a terrible distraction, especially once students are allowed to play video games or engage in other distracting activities on their downtime. This appears to lead to students rushing through examples and even written tests (!) in order to get back to the games.

The more I research the topic, the more convinced I am that computational thinking skills should be taught in their own class, dedicated entirely to them. I believe this both due to their increasing importance and ubiquity, and due to the great breadth of topics one must cover - and the depth in which one must cover them (and the time that all this will take!) in order to really prepare students for computational thinking. Donald Knuth himself wrote that out of a standard class of computing students, only a select few will really be successful, and this is due to their innate understanding of computational thinking concepts (he did not use the term of course - he predates it by quite a few years!) The remainder of the class will likely fail. It is obvious to me that these skills, if they are so crucial to success, must be taught in their own right, as a proper class. But this is something better handled in its own topic, so I will leave it off here.