Feet, hand, card, wand or voice: in a multisensory environment children can take part in many ways. A field study by the SENSEi research group, with 175 children, compares five interaction modalities and describes their benefits, challenges and trade-offs.

For teachers this is good news. The way the class interacts is not a technical detail: it is a teaching lever to be chosen with care.

What is the research question?

Activities that involve the body have the potential to improve learning and play. But in a multisensory environment children can interact in many different ways, and each one asks for different things.

The question the group asks is precise: what are the benefits, challenges and trade-offs between the various interaction modalities in educational multisensory environments? The work is presented at IDC, the ACM conference on interaction design and children.

The study in brief

It is an in situ study, that is, in the field rather than in the laboratory. 175 children aged 6 to 10 take part, using MOVES, a mobile multisensory environment that goes beyond “hardwired” rooms (we tell its story in The room that moves).

Each child tries all five modalities:

• feet: moving and stepping on the projected areas;
• hand: using gestures;
• card: bringing physical cards close so the system recognises them;
• wand: using an object as a pointer;
• voice: answering by speaking.

The group collects both what the children perceive and what they actually do: the actions logged by the system, physiological and movement signals, and picture-based self-assessments suited to their age.

What emerges?

The results describe the differences between the interaction modalities. The group also proposes design guidelines to facilitate learning in a multisensory environment.

The underlying message is already in the title: each modality involves challenges and trade-offs. There is no single right way for everyone. The quantitative details are in the full article.

Why does it matter today?

This research raises some useful questions to ask before preparing a lesson:

• What is the goal? Moving through space, choosing, naming, collaborating: each goal pairs better with certain modalities.
• Who is in the group? A child who does not use verbal language can take part with their body or with a card.
• How much physical activity is needed? Alternating more active and calmer modalities helps manage the group’s energy.
• How do I observe? Recording what the children do tells you much more than an impression at the end of the activity.

In the SENSEi magic room these choices become everyday practice. The room picks up the body moving, the hand pointing, touch and voice, even from several children at the same time, with nothing to wear. Within the same activity, the educator can therefore offer each child the most suitable way in.

It is a concrete way of applying Universal Design for Learning: several ways into the same activity, so that no child is left watching just because one channel does not suit them. We discuss this in the guide to multisensory teaching and on the Education page.

For those who design activities the consequence is direct: design them from the start for several modalities, not just one. Activities are created and adapted with Eureka. The original article is in the Research section: Interaction Modalities and Children’s Learning in Multisensory Environments.

Frequently asked questions

How many children take part in the study?

175 children aged 6 to 10, in a field study with the MOVES mobile environment.

Is one modality better than the others?

The study speaks of benefits, challenges and trade-offs. The choice depends on the goal of the activity and on the children involved.

Can several modalities be combined in SENSEi?

Yes. In the same activity one child can move, another point and another answer aloud: it is the educator who chooses the way in for each child.

How much does an environment like this cost?

It depends on the configuration: space, experiences, training and services. Contact us for an assessment.

Do you work in a school, a centre or a museum and want an environment like this? Tell us about your project: we start from the needs of the people who will use it.

Sources

• Cosentino G., Gelsomini M., Sharma K., Giannakos M. (2023). Interaction Modalities and Children’s Learning in Multisensory Environments: Challenges and Trade-offs. IDC 2023, ACM
• Cosentino G., Gelsomini M., Giannakos M. (2023). MOVES: Going beyond hardwired multisensory environments for children. IDC 2023, ACM