Spatial Concepts in the Conversation With a Computer

Human interactions with the physical environment are often mediated through information services, and sometimes depend on them. These human interactions with their environment relate to a range of scales,²⁸ in the scenario here from the “west of the city” to the “back of the store,” or beyond the scenario to “the cat is under the sofa.” These interactions go far beyond references to places that are recorded in geographic gazetteers,³⁷ both in scale (the place where the cat is) and conceptualization (the place that forms the west of the city²⁹), or that fit to the classical coordinate-based representations of digital maps. And yet, these kinds of services have to use such digital representations of environments, such as digital maps, building information models, knowledge bases, or just text/documents. Also, their abilities to interact are limited to either fusing with the environment,⁴⁴ or using media such as maps, photos, augmented reality, or voice. These interactions also happen in a vast range of real-world contexts, or in situ, in which conversation partners typically adapt their conversational strategies to their interlocutor, based on mutual information, activities, and the shared situation.² Verbal information sharing and conversations about places may also be more suitable when visual communication through maps or imagery is inaccessible, distracting, or irrelevant, such as when navigating in a familiar shopping mall.

Back to Top

Much of human conversation is about places without well-known names or with only local significance, such as the cinemas (see the scenario). They are often identified not by names at all, but by qualities (the rear car park; more examples are depicted in the accompanying table). Sometimes, places lack a well-established location or extent, such as the West of the city, or are characterized in their location just by spatial relations (the elevators at the cinema).

Figure.

Thus, we highlight here to broader concept of place we are confronted with. Borrowing from the core concepts,²⁴ we say that every object (in the core concepts) is located. And places are objects.³² To be more precise, references to objects make them places if the conversation is about localization—in contrast to other object properties. But localization of an object can happen in a variety of ways, and geometric localization—that is, in a spatial reference frame—is only one of them. Another one is the one by qualitative spatial relations. In this article we focus on the latter.

But while vital for human information sharing, decision-making, and co-ordination, conversations about places are still poorly supported by information services and situated robotic assistants such as self-driving vehicles, or, in our scenario, by Marvin. Typical conversational contexts about places include:

• Partners establishing a shared situation awareness, such as in command and control¹⁰ for emergency response (Point the firehose more to the right), in defense (The sniper must be on the roof of the building next to the tree), and in collaborative scenarios in games.
• Conversational location-aware services used as recommender systems (Where can I buy these sneakers?) or self-driving vehicles (Go to the car park behind the cinemas).
• Services that obfuscate or abandon geometric descriptions in favor of vague and qualitative descriptions to protect private or sensitive information (Kim lives in the city’s East is protecting the exact home address by referring to a vague and coarser place).

The wide-ranging opportunities to engage users in a verbal information exchange when seeking and providing place information extend beyond pure information seeking, and include commands as well the provision of explanations to machines. Here, we outline the underlying challenges of conversing about places with computers, and report on recent progress in this space. We also identify salient areas for immediate and longer-term future work.

Back to Top

Scope and Challenges

Communication about our physical environment—small scale and large scale—is special and difficult: the content of the communication strongly depends on the personal perspectives and preferences of the individuals engaged in the conversation, and may contain high levels of ambiguity that can only be resolved via mutual information and local context. Yet, this communication needs to enable physical interaction with the environment and others in it, such as coordination or planning of movement.⁵ In conversations with a computer, references to both places and their relations are difficult to parse and represent formally for a variety of reasons, among them the inherent flexibility, vagueness, and ambiguity of language (see the table), the context-dependency impacting on language grounding and disambiguation, and the interpretation of part-of relations (scale) and other spatial relations.

Table. Illustrations of additional complexities in references to places.

Language and facets of place. When conversing about places, the expressions used in linguistic strategies convey place qualities (the beautiful beach), relations between places (the river crossing Austria and Slovakia) and relations between people and places, or roles of places. Such functional, social, and operational roles of places are imposed by individuals, or groups. A function of a place can be tied to what it affords to an individual. The social role is derived from ownership. And the operational role is derived from intention (and spatial configuration). A recent comprehensive review,¹⁵ using facet theory,³ categorized these means of expression into three top-level categories of facets of places. Anthropocentric facets relate to the relations between people and places. They include the affordances of places and activities linked to them, as well as emotive facets that cover the sense of place and cultural attachment. Geographic facets describe the spatial and physical properties of places. Spatial facets enable localization of a place in space and describe the meaning of the place in relation to space and other places, while physical facets capture the structure, material, or form of a place. And finally, linguistic facets present a distinct category, capturing purely linguistic manifestations of place through place-names and identifiers. Linguistic facets can pick up anthropocentric facets (the sneaker store) or geographic facets (the Lower Eastside), but do not have to (Hype).

While the bulk of research and development in assistance systems thus far has focused only on linguistic facets (for example, place names), it falls short in the interpretation of references to place in natural language conversations. Situated assistants that are able to interact with the physical environment such as self-driving vehicles will require richer support for multifaceted natural language interaction about places. In this direction points already work in controlling robots by spatial references, for example identifying objects in an environment that can be referred to on the basis of a range of spatial reference systems,^30,41 or reflecting on deductive human reasoning with preferred models instead of formal qualitative spatial reasoning.³⁴

Vagueness. References to places are inherently vague in their specification of a location.³² Since place descriptions are usually referring expressions—expressions uniquely identifying a referent in a set of possibilities—they are typically as specific as necessary.⁴⁰ This specificity is given not by coordinates—that is, neither by crisp nor by fuzzy geometric descriptors (for example, Derungs et al.⁸)—but by contrast.⁴⁵ For example, the location of the West of the city^29,33 is sufficiently described as in contrast to the East, South, or North, and the back of the store is sufficiently specified in contrast to its front. Vagueness also applies to the qualities of places (the full car park), and to the relations between places (elevators at the cinema), each with an intent to be only as distinctive as necessary to be relevant in the given context.

Ambiguity. Most references to places are ambiguous. For example, while country names are globally unique, the same is already no longer true for city names (Paris) or landscape names (Alps). The sneakers shop on the second floor is not a globally unique reference. Disambiguation requires further locative discriminators, which are enabling the popular cluster-based disambiguation, further non-locative discriminators, which are enabling iterative identification such as through dialogue, and references to spatial relations, which are enabling disambiguation by spatial reasoning. Popular spatial relations are containment hierarchies, emphasizing the importance of scale (the sneakers shop in the Imperial Shopping Centre [in this city, in this country]), but may also be a relation with a local landmark (the sneakers shop near the food court). Dialog-based disambiguation methods⁴⁶ still require intense research, as they flip the roles of the computer and the human user, in that the human user becomes the source of information provided to the computer. Verbal disambiguation by all three approaches is easier for a situated than for a non-situated conversational agent.

Scale, or containment, is an intrinsic property of places that is impacting on spatial reasoning and enabling to adapt to spatial context. The question Where can I buy those sneakers can be answered correctly but of varying relevance by in the West of the city (for example, telling Kim the general direction), at the shopping mall (for example, if Kim is in the neighborhood), on the second floor (if Kim is in the car park), or by in the back of the shop (if Kim is entering the shop). These descriptions range between scales, and benefit from containment hierarchies that allow zooming and reasoning between levels of granularity.³⁶

Context. Conversational context provides implicit information taken into account when generating or interpreting place references. While this elementary function of context is well understood and undisputed, the concept of context itself is still ill-defined.⁹ Context is broader than the immediate perceptual cues of a situated agent, such as positioning,³⁹ orientation and vista analysis for meaningful salient features. It includes a consideration of the human communication partner, of the physical environment beyond the vista, and of social and technical aspects.¹⁹ Despite of this common broader understanding, and in tacit recognition of the ill-defined challenge, existing research on context-aware computing often links context only to position or any semantically richer notion of location. For example, some work matches a position with points-of-interest and the activities they afford (for example, Horvitz et al.¹⁶ and Schilit et al.³⁸), and on the smaller scales of smart environments sensors are used for activity recognition, which links to location,¹ or a broader notion of location is introduced to deal with positioning uncertainty in these smart environments.⁴ Semantically richer notions of location can enable assuming semantically meaningful places, including their relative description by selecting salient facets.³⁶ It also includes the choice of a degree of specificity that is sufficient for the communication purpose. It requires selection of relevant facets to avoid unnecessary ambiguity. And it requires a choice of scale that relates to the current decision making. Human conversation partners naturally adapt to context even when they are physically removed (for example, speaking on the phone, a form of co-presence⁴⁷).

Descriptions can be personalized for familiarity (in contrast to Kim, a newcomer to the shopping mall would also need instructions on how to get to the second floor); ability (a person in a wheelchair might require nuanced instructions on how to access the second floor); role (a cleaner might have access to staff-only entrances); and adapted to environmental characteristics (a complex environment will generally require a more detailed description) and dynamics (think of the full car park). This context-dependency in question-answering is currently avoided in most route direction services that maximally enrich instructions by a few personalized landmarks selected from the user’s history.

---

Since natural language uses predominantly qualitative relations, the preferred reasoning is qualitative reasoning.

---

Similarly, the nearness relation is known to be context-dependent and occasionally even asymmetric. For example, the sneaker shop near the food court and the airport near the city centre require different handling of the proximal relation. The interpretation of proximity prepositions is a persisting challenge.

Back to Top

Anatomy of a Place Assistance System

In order to converse about place, an assistance system needs to manage the challenges in handling place information along the four interleaved tasks of capturing an individual’s expression as it refers to places (covering the parsing of natural language), modelling extracted knowledge about places in some representation, processing such knowledge, and, thus, finally, interacting with the individual.

Capture. Place knowledge expressed in common language—questions or answers, commands, or explanations—can be parsed, and place references extracted and interpreted within their conversational context. This necessarily includes the ability to parse and interpret the rich means of referring to the various facets of place, including but not limited to toponym resolution and disambiguation.

The intricacies of handling context, scale, and vagueness become salient when locative expressions from natural language interactions are matched against the content of a knowledge base, with the main challenge of resolving ambiguities. When these knowledge bases are geometric ones (spatial databases) this matching is called geocoding.^12,13,27 The knowledge base can also be a qualitative one; see next section.

Some challenges in this matching process concern the unification between different references to the same place (for example, finding out that the sneakers shop and Hype are actually one and the same place in particular contexts), and the disambiguation where the same term refers to different places (for example, finding out that Hype in one conversation is a different place to a Hype store in another context). A related challenge is to establish that such a reference refers to an object missing in the knowledge base (for example, if the mall is in the knowledge base, but no Hype), or to a changed one (for example, if Hype is referring to the place that is stored in the knowledge base still as Pens&Paper).^6,33,35

Modeling. Place knowledge is inherently relational, structured by encoding primarily qualitative relations between places encoded in place references, instead of coordinate-based grounding. The representation of such relational place knowledge—whether inherent in the questions, captured in knowledge bases or encoded in answers—requires relational data structures adapted for place reasoning. Such a representation is by so-called place graphs⁴³—as illustrated in the figure here—an application of the concept of property graphs. Property graphs have recently received intensive attention in the database community, and are the basis of the first new query language standard considered by ISO in 35 years (https://www.gqlstandards.org/resources/committees-and-processes). Place graphs also provide the means to support the capture of conversational context, scale and rich facets of place.⁷

Figure. A graph of the place knowledge extracted from the conversation in the scenario.

Place graph structures enable the capture of knowledge about relations between places, even in cases where places have vague extents. As place graphs do not need to be anchored into a geographical coordinate reference system, the knowledge captured enables the reasoning flexibility observed when humans reason about places.⁶ In particular, the matching of the place knowledge against the knowledge gathered from additional utterances during conversations enables local consistency checking, without requirements for perfect matching against a globally consistent place knowledge base.

Reasoning. Reasoning refers to the task of inferring unknown information from known or given information. Since natural language uses predominantly qualitative relations, the preferred reasoning is qualitative reasoning. For example, if the sneakers shop is inside the shopping center, and the shoes are inside the sneakers shop, it is possible to infer with certainty that the shoes are also inside the shopping center. This kind of compositional transitive reasoning relies on formalisms that precisely define vague spatial relations between entities.²⁶

Yet, individuals may have their own vague conceptualizations of common spatial relations,^22,23 and a richer set of spatial prepositions than a formal model of relations provides.³⁰ They also can flexibly choose spatial reference frames.^21,25,41 For example, telling Kim to the right can refer to the walking direction, the actual body orientation, or the shop layout. The interpretation of those projective prepositions requires access to viewpoints, and hence a situated assistant. Its reasoning has to carry the reference frame explicitly, and the determination whether new information is consistent with previous information remains a challenging problem.

While reasoning with predefined sets of relations with well-described compositions is relatively simple, reasoning between different types of spatial relations (for example, a direction relation and a distance relation) is still an open problem. Furthermore, in certain place-related assistance tasks, the standard way of doing spatial reasoning is not applicable. For example, if the food court is left from the elevator, and Hype is left from the food court, for formal reasoning it remains ambiguous what the relation between Hype and the elevator is (left or back). But even if it were known, the composition is typically not helpful since Kim has to pass the food court first in this example and cannot walk directly in the direction of Hype. Instead, a valid inference needs to concatenate the two paths. In order to deal with the problem of context and user location, we have developed a unified representation of direction relations that allows us to disambiguate relative direction terms and to reason about them uniformly.¹⁸

User interaction. The initial—and still dominant—focus of place assistance systems about places.²⁰ This is now mostly happening ubiquitously and pervasively, including through conversational assistants such as Cortana and Siri. Yet, with the nearing commodification of autonomous systems, users will no longer only serve as consumers of knowledge served by computers, but will increasingly also communicate knowledge to computers that will act on it, including statements about preferences and refined specifications (Get me to the shopping center—Which entrance would you prefer?).

This transition to bidirectional interaction will pose new challenges to context handling and language grounding, personalization and privacy,⁴² and consideration of the physical affordances of the places as experienced by the combined user-machine system (The rear entrance, please. Sorry, we lack after-hours access to the rear car park). Also, language grounding and question-answering systems have thus far focused only on supporting the verbal interaction of users and machines either at the rlocal scale (vista space),^5,30 or conversely, the global geographical scale with relatively coarse place knowledge,³¹ while place assistance as in our scenario interacts across all scales in an integrated manner. Furthermore, while situated interaction offers significant contextual clues for refined interpretations of questions, the modalities for machines to capture the situation and to seek additional feedback from users are still in their infancy. This is related to the ability to converse in dialogue as well as to capture facets of the environment that go beyond location and personalization of the user. In particular, the ability of machines to reflect on vague and multifaceted place knowledge is thus far weak both in construction (West of the city) and in reasoning (behind the cinemas).

Back to Top

A Roadmap

The challenges of conversing about place have not been resolved by artificial intelligence so far, neither conceptually nor technologically. This is primarily caused by:

• Spatial databases (aka geographic information systems) remain strongly grounded in geometry and therefore lack capabilities to handle either the above challenges of capturing information about places (instead enforcing a crisp, scale-dependent, geometry-grounded spatial extent), or for computing and reasoning with qualitative spatial relations.
• Qualitative spatial representations and reasoning²⁶ capture some of the qualitative spatial relations referred to in language, but in formal interpretations that are not designed to directly cope with the flexibility of language and the embedding of the relations in conversational context. For example, we can reason with near yet we lack the ability to ground its meaning.
• Machine learning methods are neither suited to disambiguate places nor to interpret spatial relations. Due to the cross-cutting factors described above (context-dependence, scale, facets, vagueness, and ambiguity), statistical models often fail to detect significant correlations, or conversely, find spurious correlations—the challenge to interpret the meaning of near remains an excellent example.

In this situation, the anatomy of an idealised place-based assistance system as sketched in our scenario is intended to inspire a vision of a true conversational interaction with a computational assistant that can act in the physical environment, and interact with the user in a natural manner. This vision also enabled us to review some of the progress toward this vision, and to demonstrate that the state-of-the-art approaches are neither sufficient nor have we sufficiently tried to integrate the wide field of research involved.

A first, and enabling stepping stone is the establishment of a knowledge base of place knowledge. The obvious choice for a gold-standard of place knowledge is human verbal expressions—they are qualitative, contextualized (hence, sufficiently disambiguated), and independent from geometric descriptions, from commonly agreed collections of places such as gazetteers, or from named entities in general. Instead, the knowledge base is set up by places committing to an ontology of objects used to refer to (abstract) locations. Note that others have pointed to the contested nature of such extraction.¹¹

The next step is extending qualitative spatial representations and reasoning models to enable the handling of context. For example, Hua et al.¹⁷ have shown that reasoning with relative directions (left, right) involves a non-trivial ternary relation of two explicit referents and a third, possibly implicit anchor point which has to be identified from the context of the conversation.

A third critical step toward the vision presented here is the ability to effectively query place knowledge bases in natural language. This step profits from learning human answering patterns on place-related questions.¹⁴ The translation of natural language questions into queries (for example, Geo-SPARQL) can already be achieved. Thus, the only unresolved dependency for this last step is knowledge bases that contain place knowledge including qualitative spatial relations, which cannot be computed from geometry without context.

More importantly even, this paper makes a case that it is the integration of these areas that will be required to make real progress. In that sense we encourage work on the intersection between spatial databases and machine learning, or natural language processing, in the construction of knowledge bases such as place graphs, and then further between spatial databases, machine learning, and qualitative spatial reasoning for approaching contextualized reasoning.

In summary, situated conversations about places require the machine to understand verbal place-based expressions (commands, descriptions, or questions) and to respond adequately, accessing place knowledge bases. We will soon no longer just ask conversational assistants about well-known place knowledge (What is the capital of Slovakia?), but will require systems to understand complex interactions about places, enable machines to seek additional specifications to resolve uncertainty or ambiguity, and have the facility to add new facts extracted from such conversations to its knowledge base. To achieve this, however, a concerted research effort in spatial information science, natural language processing, database management, qualitative spatial reasoning, and AI is necessary.

Acknowledgment. This work was supported by a Discovery Project funded by the Australian Research Council, Grant DP170190109.