A Dynamic Blocks Platform Based on Projective Augmented Reality and Tangible Interfaces for
Educational Activities
Rafael Roberto, Daniel Freitas, Francisco Sim˜oes, Veronica Teichrieb Voxar Labs Informatics Center - UFPE
Recife, Brazil
{rar3, dqf, fpms, vt}@cin.ufpe.br
Abstract—This paper describes a dynamic blocks platform, called ARBlocks, which is based on projective augmented reality and tangible user interfaces aiming early childhood educational activities development. This tool was conceived specifically for children, applying design techniques to determinate the best shape, material and typography for young students. The content is displayed by projectors, which exhibit the required information only on the blocks surface using an automatic projector calibration technique. The blocks are tracked through a frame marker using the moving edges approach with multiple hypotheses to improve robustness. ARBlocks was evaluated from three different perspectives: computational, educational and user experience. Technical results show that the platform achieves a real time frame rate and an accurate projector calibration, as well as precisely displays information over the blocks. Additionally, in the educational evaluation the teachers interviewed asserted that the ARBlocks has a great potential and can be a very useful tool to be used in classrooms. Regarding the user experience, both teachers and children were excited in using this system continuously.
Keywords—augmented reality; education; tangible user interface;
design for children
I. INTRODUCTION
For a long period of time, education was thought as transmission of knowledge. In this context, students were considered passive, being responsible only for the storage of the content transmitted by the teacher. Currently, however, these theories have been overturned in the sense that there is no teaching without learning and that knowledge is seen as a building process. This concept of learning, aroused by the student interest, changed what is understood as educational material. Thus, there are a lot of teaching materials that are facilitators of the learning process. From the perspective of Jean Piaget, we see how the activities carried out through these tools have become important and contribute to child development [1].
In this context, an important concept is playfulness education, in which a subject is taught without the child to realize that he/she is learning. This can be accomplished through activities that allow them to effectively participate in their knowledge construction, thus the learner can establish a positive relationship with knowledge. This aspect becomes
especially relevant when considering children with learning difficulties [1].
Therefore, tangible user interfaces are a good instrument for the creation of a material that satisfies important needs of the students. They provide means to create tangible tools that can help in the students development, contributing to their motor aspects’ improvement, sensory engagement, accessibility, collaborative activities and understanding of the world around them [2]. A material that stands out for stimulating these skills are the interactive blocks. Usually made of wood, they are used as a concrete instrument to teach abstract concepts, particularly for children under eight years old. Depending on the content displayed on the block, teachers can use it in order to help children to develop many abilities, such as creativity, logical reasoning, language skills and other capabilities [3].
For these materials to be successful in facilitating learning, they must be designed to this purpose. Besides attracting the interest of children, visually and tactile, for instance, they must be very intuitive to use, in a way they have an enjoyable experience with these artifacts. For this, graphic, technical, ergonomic and educational factors should have a huge importance in the design process of these products [4].
Other tool capable to fulfill some of the teacher’s needs are the computers. Recently computers have become widely used in the classroom, especially as a tool to simulate activities that are common to children on their daily bases. One area in computer science that has a great potential to contribute with early childhood education is augmented reality [5], also known as AR. The main reason for this is its potential to improve the quality of information visualization and its interactivity. These are very important characteristics, especially when it comes to children learning.
In this context, this work presents the ARBlocks, a dynamic blocks platform aiming educational activities. It is based on projective augmented reality and tangible user interfaces. The contributions of this paper are:
1) A platform designed specific for children that allows performing educational activities in which the information projected only on top of physical blocks are dynamic and can change as they are manipulated;
2) Runtime analysis of the platform showing that the
projection and interaction are accurate and run in real- time;
3) An educational evaluation in which the teachers interviewed asserted that this platform can substantially increase the possibilities to the teachers as they can use their creativity to produce several activities. The authors were not able to find other projective augmented reality based systems being used in the educational domain, being this feedback a very promising indicator that such technology can be positively used for educational purposes;
4) An user experience evaluation in which both teachers and students were excited to use the tool. The methodology used for this evaluation provides guidelines for the community to assess other augmented reality educational systems with teachers and pupil.
In order to explain the ARBlocks concept, this paper is organized as follows: Section 2 shows related work regarding the use of tangible user interfaces and augmented reality in early childhood education. Then, in Section 3, it is explained how this product prototype was developed. In Section 4 the computational, educational and user satisfaction results achieved are presented. Finally, Section 5 presents conclusions and future directions for this platform development.
II. RELATEDWORKS
The use of interactive blocks is not new in education and this type of activity soon became a great aid to the teachers.
Today they exist in several sizes, shapes and materials. For each proposed activity, there is a specific set of blocks.
Thus, if the teacher wants to instruct mathematics, he/she uses a material made exclusively for this purpose, containing numbers, mathematical and geometric symbols [6]. In another moment, when the activity has the objective to develop language skills, it will be used other blocks containing letters [7]. Thus, schools must have multiple sets of games, one for each purpose.
Recently computers have been used as a tool to simulate activities that children already perform in their daily life [8]
[9]. Despite holding their attention, those programs cannot stimulate as well as tangible objects some important skills, because they work primarily as a digital finger on a flat screen.
With the cheapening and development of new technologies, solutions that use tangible objects to provide input or output for educational activities in computers have begun to emerge. They are called digital manipulatives. However, it is still evident the lack of interesting applications that can satisfy all children educational needs [10].
A good example of digital manipulative is the Sifteo [11], which are computers with approximately 1.3 inches of size that have an LCD screen and sensors to interact with other Sifteos. Developed at MIT, it is an interesting digital tangible user interface solution for educational use. However its price makes it impractical for many schools, especially in under development countries. A package with three blocks is sold for US$ 129.00 without the importation taxes.
According to the educators consulted during this work, augmented reality has two characteristics that make it attractive
to be used as an educational tool: firstly, AR delivers an exciting way to visualize the school subjects and secondly, it stimulates the interaction between the people involved in the learning process. Many augmented reality applications have being developed for educational use [12]. Some good examples are the augmented book [13], as can be seen in Figure 1, or the Augmented Reality Scratch [14].
Fig. 1. Volcano on a Book is an example of an augmented book.
However, few applications combine the use of augmented reality with tangible interfaces. One example is the Augmented Chemistry [15]. With this application the user manipulates an object that simulates a claw to pick up elements in a catalog.
The student is able to combine various elements, creating molecules that can be viewed in three dimensions by moving a cube that indicates its orientation, as can be seen in Figure 2.
Fig. 2. Augmented Chemistry in use. The user holds a claw with the right hand and a cube for orientation with the left hand. On the bottom left corner there is a catalogue of periodic elements and over the marker on the right there is the molecule assembled by the student.
Even with a few number of tangible augmented reality applications addressed to the educational scenario, there are works that evaluate the user experience that the students have using this type of tools [16]. However, it is more common to find research aiming to evaluate the user satisfaction in educational augmented reality applications in which the interaction form is not with a tangible interface [17] [18].
III. ARBLOCKS
In this work it is proposed the ARBlocks, a dynamic blocks platform aiming to explore the benefits of tangible tools along with the flexibility of computer systems applied to educational activities, thereby minimizing the restrictions of current educational materials. The conception of this tool was divided into three stages: firstly the design of the product, in which design techniques were selected and applied in order to conceive a solution that fits the needs of early childhood education; after, an architecture was proposed and a prototype was developed; finally, an educational activity that uses the platform was also proposed.
A. ARBlocks Design
The design process for the conception of the ARBlocks was based on the cartesian method [19], subdividing the problem in three minimum parts: shape, typography and material. Being a project aiming a very specific public, the data collecting was a crucial point to find the optimal solutions to every subproblems, becoming the base for the entire process.
After studying, ideating, prototyping and testing several possibilities, it was defined that the measures of the blocks should have 6 x 6 centimeters for the top face and 2 centimeters high to fit an usual 7 years old child hand. The material chosen was the ABS plastic, which is harmless and is commonly used in toys for children. It is also inexpensive since the estimate cost to produce this block in large scale is between R$ 0,30 to R$ 0,60 1. This characteristics and the chosen typography can be seen in Figure 3. The full design process can be deeply observed in [20].
Fig. 3. Initial sketches, paper mock-ups and final solution rendered with a 3D modeling software.
1Estimative value given by a plastic company in Recife/PE, Brazil.
B. ARBlocks Development
Once the main issues in the product design had been defined, the decisions on the architecture and the computational development could be taken.
One of the most important points to guide the architecture definitions is regarding the interaction with the ARBlocks.
This needed to be as simple as possible, preferably using the own blocks since many teachers do not feel comfortable using computers. With the use of the blocks itself as interaction form it is possible to reduce most of the educator actions on the PC. From the child perspective, interactions with the physical blocks explore all the benefits of using manipulatives in their educational development. Other important aspect is the interaction between the blocks in order to provide feedbacks to the students. Thus, the system may be able to determine if a block is close to another one or even if it is next to a specific side of other blocks. It also has to be able to track other information from the blocks, such as where they are in the projection area or their rotation angle.
The interaction form was determinant to select the ARBlocks visualization method. In this context, the most suitable choice was to make students interact with physical objects and visualize the digital information through projective augmented reality. This is because the long time using a HMD would not be comfortable for children and increases the solution cost. The use of monitors was discarded since it is not natural to manipulate an object and look in other direction to see the action results.
One of the main benefits of using dynamic blocks is the fact that several different activities can be executed with the same set of blocks. This fact allows the teachers to easily choose which one of them need to be used during class. The teachers could also propose new activities as they identify new necessities of the classroom. Thus, the ARBlocks was designed as a framework in which the developers would have access to basic functionalities implemented, such as for tracking or camera-projector alignment. Thereby the programmer just needs to develop an application that invokes these functions and incorporates to the ARBlocks as a plug-in. It would make easier for developers to create new educational activities for educators.
All these requirements were used to define the relationship between modules proposed in the ARBlocks architecture, which is illustrated in Figure 4. The first module is the Calibration, which is also the first to be executed. It is responsible for making the alignment of the camera-projector system and delivers this information to the Projection module, which is the last module to be executed. In the main loop, the camera captures an image that will be sent to the Tracker module. This module finds all the blocks present on the scene, along with their respective positions and orientations.
The core structure of ARBlocks is the combination of these three modules. The last one is the Application, which works as a map indicating what happens when blocks are moved, rotated or placed close to others. The Application is also responsible for informing what should be displayed on each block, besides of managing the educational activity. The processing result is
Fig. 4. ARBlocks architecture.
sent to the Projection module, which is combined with the alignment data in order to project the information correctly on the blocks.
1) Tracker: One of the first questions to be addressed when developing an augmented reality system is how the tracking will occur. Since the proposed platform is a tangible user interface using projective augmented reality, the tracking, the information visualization and the input device involves the same cubic block object. As the blocks are the projection surface, they cannot be textured so they do not harm the image quality. They should be uniform and white so that minimum color interference occurs in the display. However, it is not possible to track a surface with such characteristics. If all blocks are uniformly white, it is impossible to even distinguish one from another by computer vision.
To solve this problem, the first solution thought was to add a colored border to the blocks. However, this approach has some flaws, being the main one a limitation of the maximum number of markers tracked simultaneously, which would be only fifteen. This is due to the fact that color sensing is very susceptible to the ambient lighting condition and in order to have a reliable measure colors should differ from each other, as occurs in the CGA palette [21] which has only sixteen colors.
Besides, the white color cannot be used since it is the same as the block itself.
Therefore, the final solution uses a frame marker. With this kind of pattern it is possible to have all the benefits of a common marker and still get a large region of the block that does not have any information, being free to display the projection. As far as the authors know, there is no free augmented reality library for desktop computers that supports frame marker. Thus, a specific marker for this platform was created as well as a tracking method. The marker is wrapped by a continuous thin edge. Inside this edge there is a code, which is a sequence of 10 bits, as can be seen in Figure 5. Each side of the block has the same code for redundancy purposes, except for the first and last bits, which indicate the orientation of the marker. Between those two are eight others that compose
the code itself, which gives us up to 256 different markers.
Fig. 5. Frame marker designed for the ARBlocks.
The tracker for the proposed marker executes in four steps.
First, the image is segmented in black and white. In the next step, the Canny edge detector [22] is used to find all the edges in the image. Then, all closed contours are extracted and analyzed to find all squares with a minimum area and minimum joint edges angle. Finally, the bit code is read and those who possess a valid identifier are assigned as successfully tracked markers, as illustrated in Figure 6. For optimization purpose if a quadrilateral is detected near or in the same position of a marker found in a previous frame and they share the same approximated area, then the quadrilateral is considered to be the new position of the marker on the current frame.
Due to several factors, such as noise or partial occlusion, in some frames is not possible to find a closed quadrilateral. To make the tracker more robust, when this occurs and a marker detected in a previous frame is not found in the current one, a Moving Edges [23] approach with multiple hypothesis is used.
The edges of the marker in the previous frame are sampled in image coordinates and for every sample, several matches are selected along the normal of the edge based on the intensity gradient. Thus, the system creates clusters from the multiple hypotheses by choosing subsets of matches for every side of the marker. Because of that, each side of the marker will have various possible edges correspondents, as shown in Figure 7.
In order to cluster the hypotheses, the points are rearranged based on a k-mean algorithm where the centroid is calculated as the lines approximated from the clusters points by using a least squares approach and the distance function is the distance from points to these lines. As pointed out by [24], this approach has a quick convergence rate if it uses the hypothesis find order in the moving edges as the first k-mean cluster, leading to convergence in most cases with less than 10 iterations. Figure 8 shows a block with approximately 65% of its area occluded and still tracking the marker.
2) Calibration: Different from the traditional AR displays, in projective augmented reality the camera and the display do not share the same coordinate system. Thereby, an alignment of the camera-projector system is mandatory. In the ARBlocks
Fig. 6. The original frame captured by the camera appears on the top left of the image and on the top right there is the result from the segmentation process; the row in the middle shows every line extracted by the Canny and only the quadrilaterals that could be markers; in the end, only the proposed marker is selected.
this is done in the application initialization. Moreover, it should be automatic, so that no technical knowledge is required from teachers and there is no need for the educator to spend time adjusting those parameters manually. Thus, a technique based on the projection of structured light was chosen, as shown in Figure 9. These patterns are displayed on the table and the corners of the projection are used to calculate the alignment [25].
The camera must be able to see the entire projection area.
It captures each one of the patterns projected and the four corners of this region are identified. These points are used to calculate the homography that will map the coordinates from both systems.
To find the homography it was used the direct linear transformation algorithm, or DLT [26]. The homography is calculated from a set of four corresponding points f(x)→x0. In this case, they are the corners from the projection area and the camera image, as shown in Figure 10.
Fig. 7. Every colored line represents different hypotheses for the markers’
edges.
Fig. 8. The original image, on the top, shows that most of the marker is occluded. Even though, in the bottom it is possible to see that the block is tracked and the augmented reality content is rendered over it.
Thus, the points have the following relationship:
x0i=Hxi, (1)
in which x0i represents the homogeneous points from the camera’s corners andxi are the homogeneous points from the corners of the projection area. Thus, if the jth line of H is denoted ashjT, we can tell that
Hx=
h1Txi
h2Txi
h3Txi
. (2)
Equation 1 can be rewritten as a vector product:
Fig. 9. Structured lights projected in sequence so the alignment can be calculated.
Fig. 10. Correspondent points used to align the camera-projector system.
x0i×Hxi=0. (3)
Thus, writing x0i in function as its coordinates, we have:
(a0i,b0i,c0i)T×Hxi=
b0ih3Txi−c0ih2Txi
c0ih1Txi−a0ih3Txi
a0ih2Txi−b0ih1Txi
. (4) Since
hjTxi=xTihj, (5) then
0T −c0ixTi b0ixTi c0ixTi 0T −a0ixTi
−b0ixTi a0ixiT 0T
h1 h2 h3
=0. (6) This equation is in the form ofAih=0, in whichAiis a 9x3 matrix and h is a 9 dimension vector which values compose the homograph matrixH.
The singular value decomposition ofAi is used to calculate the homograph [27]. Thus, with:
A=UDVT (7)
and the diagonal D having positive values in descending order,h is equal to the last column ofV.
Since the matrix H is invertible, the homography that transforms the image points into projection points is
xi=H−1x0i. (8) Since the camera and the projector will not move during the use of ARBlocks, the calibration can be performed only once, when the system initializes. If accidentally any of the devices change their position, realignment can be executed without the need to reboot the platform.
3) Projection: The correct alignment of the digital content with the physical blocks is ensured by a series of transformations to the markers captured by the camera. First, the homography calculated in the Calibration module is used to find the correspondent coordinate of the markers detected by the Tracker module in the projector coordinate system. Then, this point is mapped in the camera coordinate system using the inverted homography. In the end, the images to be displayed on the blocks are transformed to fill their empty area using the same DLT algorithm.
C. Case Study
In order the evaluate the platform, it was developed an application that explores the ARBlocks characteristics [20].
As a first application, it was developed a simple game aiming to help teachers in early childhood literacy.
The game is addressed to teachers who want to work the children’s vocabulary in a non-syllabic way. For this application, in half of blocks are exhibited drawings, like animals or objects, while their respective words appear in the others. The objective of this game is to place together the blocks with word and the drawing. When the children make the correct association, a positive feedback is given to them.
The Application module verifies if two correspondent blocks are close to each other. This is done by using the marker’s position information, which comes from the Tracker. If this verification turns to be positive, a green rectangle is projected around those blocks. Figure 12 shows this application running with the ARBlocks platform.
IV. ARBLOCKSEVALUATION
The ARBlocks was evaluated from three different perspectives: computational, educational and user experience.
From the computational analysis point of view it was possible to evaluate the performance, robustness and precision of the projection system, the tracker and the overall tool. On the educational analysis, the potential use of this tool in classrooms was evaluated by the teachers. Regarding the using experience, the excitement of both teachers and children in use this system continuously was measured.
A. Computational Analysis
The platform was implemented in C++ using data structures and basic functions from the OpenCV [28], VXL [29] and
ViSP [30] libraries. It was also used the fmod Ex [31] library to give the sonorous feedback of the application. The execution was performed on a Windows 7 platform running on an Intel Core i3 with 2.8 GHz, 4GB RAM and a nVidia GeForce GT 440 graphic card. The camera used was the Microsoft LifeCam Cinema and the projector was the Epson X10, both with a resolution of 800 x 600 pixels. The camera-projector system setup used a tripod, as can be seen in Figure 11.
Fig. 11. Camera-projector system setup, using a tripod. The projector illuminates the table where the children can use the system.
The calibration step was executed in an average time of 4.701 seconds. It’s an off-line phase performed automatically during the system initialization. Even though it is not an online step, it is faster than the case in which the users need to adjust these parameters manually.
In order to evaluate the calibration precision, an image with the same size of the captured frame and centered on its coordinate system origin was transformed using the calibration homography. This approach is the same as transforming all points from the camera coordinate system to the projection coordinate system and project them on camera plane. As a result, the image has to be distorted to fill the entire projection area.
Regarding the tracking time, one frame marker is tracked in 9.791 milliseconds. The proposed frame marker shows good results in comparison with the ARToolKit [32]. The tracker was 9.791% faster than the ARToolKit when tracking one marker. The tracker is also very scalable since 98 markers were tracked simultaneously in 10.613 milliseconds.
Other important characteristic is the robustness of the frame marker tracker. In 3.381% of the frames containing a marker, the block was not successfully tracked and only in 0.442% a digital content was superimposed over other surface besides a valid block. In comparison to the ARToolKit, those rates are 1.382% and 0.953%, respectively. The Moving Edges approach also showed good results. After the addition of this technique to the tracker the amount of false negatives and false positives
dropped from 7.845% and 0.792%, respectively.
To analyze performance of every module of the ARBlocks running simultaneously, the system was used in a real scenario.
A projector was mounted on a tripod and a camera was positioned over it. After the initialization phase, in tests with up to 16 blocks, the application performed in an average time of 17 milliseconds, achieving a rate of almost 59 frames per second, being adequate for augmented reality purposes.
Regarding precision, the ARBlocks was only visually evaluated. It was observed that the projection occurs only on the determined area, as can be seen in Figure 12. However, it is possible to notice a drift in the projection. This happens because of the latency of the projector to update the image it is displaying.
B. Educational Evaluation
To evaluate the educational benefits of ARBlocks and its applicability in classrooms, an interaction with eight teachers was promoted. They all teach children, being five from private schools and three from public schools. They also had no previous knowledge in augmented reality.
The evaluation was a three steps interactive session. First, the concept of the ARBlocks was explained to the teachers, then they had the opportunity to use the tool running the developed application and finally they passed through a semi- structured interview. The questions of the interview were the following:
1) What is your opinion about this concept?
2) Even knowing it is a prototype, did you like the experience of using it?
3) What was the difficult to use it?
4) Do you think that children would like to use this tool?
5) Do you think that this tool has the potential to improve your classroom activities?
6) What are the positive and negative aspects you see in this tool?
7) Can you propose any improvement or new functionality to this tool?
The teachers pointed out the benefits of the ARBlocks being a tool based on physical blocks, since when children can manipulate objects the subject becomes more attractive and meaningful to them. In addition, the tool was praised for being very interactive, since it is possible to give feedback when the students perform a correct movement with the blocks and its content changes during the manipulation.
The multidisciplinary aspect and versatility of ARBlocks were exalted by teachers since with the same set of blocks it is possible to create many different activities. The educators pointed out that the system allows them to use their creativity to create and guide the activities according to their needs. The teachers said that this freedom feature is found only in a few educational software these days.
According to the teachers, the fact that the ARBlocks is a technological tool can increase the students’ motivation.
This is due to the fact that they are used habituated to use technological equipment in their daily routine. Thus, initiatives such as this one is very effective in holding students attention.
Fig. 12. The information is projected only on the blocks empty area. The green line around some blocks denotes to positive feedback when the activity is executed correctly.
An important fact observed while the teachers were using the ARBlocks is that they did not need any training to use the system. By that, the ARBlocks shows itself as a simple and intuitive tool. This fact was confirmed during the interview.
Some of the teachers suggested that the tool could be improved if both faces of the blocks could be used to project the content. It would increase the possibility of educational activities for teachers.
From teachers evaluation it is possible to observe that the ARBlocks was very well accepted by the educators. According to them, the proposed tool has a great potential to be used in classrooms, as it can be seen in the following statement from one of the teachers: “ Every effort to combine the teaching of a school subject with more visual, technological and manipulative content has potential to improve learning and this tool does that”.
C. User Satisfaction Evaluation
Under eight years old, children’s cognitive characteristics that would allow them to verbalize their perceptions and feelings about the system are not well developed yet.
Therefore, the methods used to measure their satisfaction did not aim to have objective and solid data results as the ones used for older children, such as Think Aloud, Questionnaires or even Visual Analogue Scales [33].
According to [34], the perception of satisfaction for children can be translated as fun, which can be better analyzed by studying its three dimensions: expectations, engagement and endurability. Expectations refer to how much the students want to work with the system before and after using it. The engagement can be described by how involved the children are
while using the system. Endurability, means how much they want to keep using it over time. These dimensions were studied through the method of observation. For this purpose, all the sections were video and audio recorded and then analyzed to extract useful information about how the users interact with the system. This information could be anything they say, their expressions or anything else that evidences how they think and feel about this new learning activity.
Also according to Xu’s study [35], is highly desirable to apply more than one evaluation method for interactive products for children [35]. In the same paper, she presents a new method based in school activities, called Drawing Intervention, which can extract more subjective information about the perception of the system by the children. This method was applied immediately after all the sections, by asking children to draw anything related to what they had done using the ARBlocks and then having a small conversation about the drawings.
The drawing intervention was applied with thirteen students randomly chosen from one school in Recife/PE, Brazil. They also had no previous knowledge in augmented reality.
Since the teachers are also users of the system, they also need the ARBlocks to have characteristics supplying their requirements. Two methods were used to measure the system’s usability from the teacher’s point of view: Observation and Semi-Structured Interview. The observation was performed adopting the same techniques used for the children. The Semi- Structured Interview was applied with the teacher immediately after all the sections, and covered main usability questions through a 20 minutes conversation that had its audio recorded.
Below are described the collected data, based on each method used.
Through the Observation Method, it was easy to see the enthusiasm children demonstrated for the system. Even after the eight sections, they still preferred using ARBlocks to other educational material and they all wanted to be part of the first group of the day to use the system. This enthusiasm could also be noticed when some of them asked if they could bring it with them to their homes. The positive audio feedbacks used by the applications were created with a small happy melody accompanied by claps and whistles, during 6 seconds. These feedbacks showed themselves as extremely useful to, beyond communicating that the expected action was completed, stimulate the young students to try to provide the correct answer, wanting to listen and clap along with the audio.
Negative aspects of the system could also be noticed from this method. Each group used the ARBlocks during thirty to forty minutes. As the setup was designed to be used on a table, they all spent this time standing around the table, on which they could not fit if seated, and this fact caused fatigue on the students and also on the teacher, who needed to stand in a curved position for almost two hours on every section.
Regarding the audio feedbacks, some of them were muffled by other sounds in the room, making the users confused about what to do next. The observation also showed that the children were not concerned about the ARBlocks’ limitations, such as the small jitter, the projector’s drift and the partial occlusion.
Lastly, it was also possible to see that the teacher stayed worried about the possibility of the children tumbling the setup or making reckless movements that could harm the system.
This fact was splitting her attention preventing her to focus only on the teaching process, which ended up worsening her user experience.
The second method used was the Drawing Intervention.
Even being a recent method and with few guidelines, it was possible to collect useful information that probably could not be perceived through other methods. The ARBlocks was designed based on how children would interact with the blocks, the projected elements on the blocks and table and with the audio feedbacks. However, through the students’ drawings and the further conversations, it was noticed that they interact not only with these elements, but with all the setup. The system is perceived as a “robot” by them, having eye, body, head and voice, represented by the camera, tripod, projector and the audio feedbacks from the stereo boxes, respectively. In one of the conversations, one of the students said that she likes to use it because it’s the robot that teaches her, and not the teacher, as always. This simple feedback can be a great input for the system’s future redesign.
However, an execution problem of this method occurred during our experiments: from the thirteen students instructed to draw something related to the activity with the ARBlocks, only seven really did it. The other six did non-related illustrations and, as they were not interested in participating in the activity, we could not apply the post conversation with them.
A Semi-Structured Interview with the teacher was very useful to detect problems regarding her use, but also children’s use too. Hearing, from her point of view, what are the positive and negative aspects of the system provided us a very rich feedback material to work on.
As was also perceived during the observation procedure, the teacher added that at first, she was very afraid of using the system, thinking that it could crash or that she could let the children do it in some way. This feeling decreased on time, but remained until the last section.
Some of the children did not recognize yet some of the letters from the alphabet. For this reason, the teacher suggested that we tried to avoid that the words projected on the blocks stay upside down for the students. Likewise, the process of taking the children from their classroom to the library to start de activities was considered good and bad at the same time. It was good because this change stimulates them, but in counterpoint it is hard to make them quiet and focused to initiate another learning process. Another problem she detected was that some illustrations shown by the application were not easily recognized, confusing the students.
Regarding the positive aspects, she commented that the ARBlocks can be so attractive for the children that even the most inattentive ones could pay attention to the educational activities using the system. Besides it was mentioned that their self-esteem got higher at the days they could interact with the ARBlocks and they also always complained when the using sections finished.
As mentioned before, to measure the satisfaction of the young students regarding the system, it was used the analogy with the fun it provides. For this, the collected information was organized as characteristics from each one of the three dimensions of fun.
Regarding the dimensions of the expectations and endurability, it was perceived that children, even after the eight sections, still wanted and got anxious to use the system and always asked to use it more times. This behavior could be seen both on the observations as on the interview. The engagement was clearly noticed on the observation, seeing that the students were very enthusiastic, happy and focused on the activities, competing to be the one moving the blocks and to answer the questions correctly. This dimension was also validated on the interview when the teacher said that even the most inattentive ones were paying attention to these new activities, meaning how engaging can the applications be using the ARBlocks.
Therefore, analyzing the fun provided by the ARBlocks to the children was found that the system’s usability supplies their satisfaction necessities when using it, but it has also characteristics that need to be improved, making the learning activities more pleasant avoiding usability problems.
Analyzing the usability for the teacher, despite she could guide her students through all the activities, we understood that the system, from the setup to the way we introduced it to her needs to be improved, so the teaching activities can become more pleasant to her too.
Resulting from all the information collected, there are some clear changes the system must pass by to make it more usable.
The visual feedbacks must be improved, knowing that relying only on the audio confused the users at some times. Trying to make the use more comfortable, new tests are going to be made projecting on the floor, where students and teacher could interact with the blocks while seated, avoiding the fatigue. As the teacher mentioned that children could get too much agitated
when changing to another room, we will try making new sections in the same room they study. Regarding the confusion from the illustrations, all the new ones created for the system are going to be submitted to recognition tests before being applied. Also a result from the interview, new possibilities are going to be studied to understand how to avoid the upside down problem.
The drawing intervention showed how the children perceive and give importance to the entire setup. New studies are going to be made to understand what is the best approach: trying to hide the projector, tripod and camera to make children see only the blocks and projections; or, trying to take advantage of their global perception and maybe “dress the robot” or put more childish characteristics on the structures.
Finally, the problem of the teacher being afraid of using and crashing the system was understood as a flaw of introduction.
If we explained better, at the first visit, how to use the system, what she could do and explained also the few things that could crash it, she would be more comfortable when using it.
V. CONCLUSION
The proposed system consists in a dynamic blocks platform for educational activities based on projective augmented reality and tangible user interfaces. The ARBlocks shows that augmented reality can be an important pedagogical tool especially in association to tangible interfaces.
The blocks are tracked based on a frame marker proposed in this paper. The tracker achieved real-time results, being suitable for augmented reality applications. It was also developed a technique to display the information on the blocks surface, as well as a fast and efficient solution for an automatic camera-projector alignment.
The ARBlocks was well evaluated by teachers that tested the system. They pointed out as a principal feature the fact that the system is a tangible interactive tool for children focused educational activities. Despite that, the teachers can also be free to create their own applications.
The user experience study indicated that the system is already satisfactory from the users point of view. The evaluation methods also extracted data that should be used to refine the system in a variety of its aspects. By taking into account user feedbacks in the design process of the system, ARBlocks will be more easier and natural to use.
A. Future Works
For future versions of the ARBlocks some improvements should be done, such as the motion blur treatment to deal with fast movement of the blocks. Other important feature that should be added to this tool is the support for multiple projectors.
It is also a future work the development of a graphical editor in which teachers could easily create their own educational activities.
Regarding the educational evaluation of this work, an ongoing step is the test of the impact that the ARBlocks could have in the learning process of the children.
VI. ACKNOWLEDGMENTS
The authors would like to thank FACEPE, Propesq UFPE and CNPq for the scholarship of some members of this paper (process PBPG-0660-1.03/09, process 10012149 and process 141853/2011-5, respectively), as well as the funding of our MCT/CNPq Universal project (process 475975/2009-0).
REFERENCES
[1] J. Piaget, Play, Dreams and Imitation in Childhood, 1st ed. W. W.
Norton and Company, 1962.
[2] O. Zuckerman, S. Arida, and M. Resnick, “Extending tangible interfaces for education: Digital montessori-inspired manipulatives,” in Proceedings of CHI 2005. Press, 2005, pp. 859–868.
[3] G. M. Schwartz, Dinˆamica l´udica: novos olhares, 1st ed. Manole, 2003.
[4] C. A. W. Torrezzan and P. A. Behar, “Design pedag´ogico de materiais educacionais digitais,” inAnais do V Congresso Nacional de Educac¸˜ao Superior a Distˆancia, 2008.
[5] T. N. M. Consortium, “Horizon report,” California, USA, Tech. Rep., 2011.
[6] Holgatetoy. (2011, fev) Hickory dickory dock clock nursery rhyme wooden mouse ran time learning shapes colors classroom teaching aid.
http://bit.ly/PHeT85.
[7] Hape. (2011, fev) Alphabet abacus. http://bit.ly/L4gzke.
[8] J. W. d. Costa and M. A. M. Oliveira, Novas Linguagens e Novas Tecnologias: Educac¸˜ao e Sociabilidade, 1st ed. Vozes, 2004.
[9] M. Mayrath, P. Nihalani, and S. Perkins, “Digital texts and the future of education: Why books?”EDUCAUSE Quarterly, mar 2011.
[10] T. P. Falc˜ao, L. Meira, and A. S. Gomes, “Designing tangible interfaces for mathematics learning in elementary school,” inProceedings of III Latin American Conference on Human-Computer Interaction, 2007.
[11] D. Merrill, “Siftables: towards sensor network user interfaces,” in In Proceedings of the 1st international Conference on Tangible and Embedded interaction. TEI ’07. ACM Press, 2007, pp. 75–78.
[12] M. M. Silva, R. A. Roberto, and V. Teichrieb, “Um estudo de aplicac¸˜oes de realidade aumentada para educac¸˜ao,” inIX Workshop de Realidade Virtual e Aumentada, nov 2012.
[13] HITLabNZ. (2011, nov) Hitlabnz - project - mixed reality book. http:
//bit.ly/LJNaNR.
[14] I. Radu and B. MacIntyre, “Augmented-reality scratch: A children’s authoring environment for augmented-reality experiences,” in Proceedings of the 8th International Conference on Interaction Design and Children, 2009, pp. 210–213.
[15] J. Almgren, R. Carlsson, H. Erkkonen, J. Fredriksson, S. Moller, H. Rydgard, M. Osterberg, and M. Fjeld, “Tangible user interface for chemistry education: Visualization, portability, and database,” in Visualization, Portability, and Database. Proc. SIGRAD 2005, 2005, pp. 19–24.
[16] E. Hornecker and A. Dunser, “Of Pages and Paddles: Childrens Expectations and Mistaken Interactions with Physical-Digital Too,”
Interacting with Computers, vol. 21, no. 1-2, pp. 95–107, 2009.
[17] H. Kaufmann and A. Dunser, “Summary of usability evaluations of an educational augmented reality application,” in Proceedings of the 2nd international conference on Virtual reality, ser. ICVR’07. Berlin, Heidelberg: Springer-Verlag, 2007, pp. 660–669.
[18] G. Salvador, D. Prez, M. Ortega, E. Soto, M. Alcaiz, and M. Contero,
“Evaluation of an augmented reality enhanced tabletop system as a collaborative learning tool: A case study on mathematics at the primary school.” Cagliari, Sardinia, Italy: Eurographics Association, 2012, pp.
9–16.
[19] B. Munari,Das Coisas Nascem Coisas, 1st ed. Martins Fontes, 1998.
[20] R. Roberto, D. Freitas, J. Lima, V. Teichrieb, and J. Kelner, “Arblocks:
A concept for a dynamic blocks platform for educational activities,” in Virtual Reality (SVR), 2011 XIII Symposium on, may 2011, pp. 28 –37.
[21] I. B. M. C. IBM, “Color/graphics adapter,” Tech. Rep., 1981.
[22] J. Canny, “Finding edges and lines in images,” Cambridge, MA, USA, Tech. Rep., 1983.
[23] P. Bouthemy, “A maximum likelihood framework for determining moving edges,”IEEE Trans. Pattern Anal. Mach. Intell., vol. 11, no. 5, pp. 499–511, May 1989.
[24] C. Teuliere, E. March, and L. Eck, “Using multiple hypothesis in model-based tracking,” in International Conference onRobotics and Automation, may 2010, pp. 4559 – 4565.
[25] J. M. Teixeira, F. Sim˜oes, R. Roberto, V. Teichrieb, and J. Kelner,
“Reconstruc˜ao 3d usando luzes estruturadas,” inVirtual Reality (SVR), 2010 XII Symposium on, may 2010, pp. 1 –33.
[26] R. Hartley and A. Zisserman, Multiple View Geometry in Computer Vision, 2nd ed. New York, NY, USA: Cambridge University Press, 2003.
[27] G. Golub and C. Van Loan,Matrix Computations, ser. Johns Hopkins Studies in the Mathematical Sciences. Johns Hopkins University Press, 1996. [Online]. Available: http://books.google.com.br/books?id=
mlOa7wPX6OYC
[28] G. Bradski, “The opencv library,”Dr. Dobb’s Journal of Software Tools, 2000.
[29] VXL. (2012, may) Vision-something-libraries vxl. http:
//vxl.sourceforge.net.
[30] E. Marchand, F. Spindler, and F. Chaumette, “ViSP for visual servoing:
a generic software platform with a wide class of robot control skills,”
IEEE Robotics and Automation Magazine, vol. 12, no. 4, pp. 40–52, 2005.
[31] fmod Ex. (2013, jan) fmod interactive audio. http://www.fmod.org.
[32] H. Kato and M. Billinghurst, “Marker tracking and hmd calibration for a video-based augmented reality conferencing system,” inProceedings of the 2nd IEEE and ACM International Workshop on Augmented Reality, ser. IWAR ’99. Washington, DC, USA: IEEE Computer Society, 1999, pp. 85–.
[33] W. Barendregt and M. Bekker, “Guidelines for user testing with children,” Eindhoven, The Netherlands, Tech. Rep., 2003.
[34] J. C. Read, S. J. MacFarlane, and C. Casey, “Endurability, engagement and expectations: Measuring children’s fun,” inInteraction Design and Children, 2002, pp. 189–198.
[35] D. Xu, “Design and evaluation of tangible interfaces for primary school children,” in Proceedings of the 6th international conference on Interaction design and children, ser. IDC ’07. New York, NY, USA: ACM, 2007, pp. 209–212. [Online]. Available: http:
//doi.acm.org/10.1145/1297277.1297331