Organizing Argumentation Statements to Support Intelligence Analysis

Intelligence Analysis

Tim Chklovski

USC/ISI

4676 Admiralty Way Marina del Rey, CA 90292

001-310-822-1511

Jihie Kim

USC/ISI

4676 Admiralty Way Marina del Rey, CA 90292

001-310-822-1511

Yolanda Gil

USC/ISI

4676 Admiralty Way Marina del Rey, CA 90292

001-310-822-1511

timc@isi.edu

jihie@isi.edu gil@isi.edu

ABSTRACT

Understanding how to organize argumentation statements is important to effectively support intelligence analysis. One common approach to supporting analysis is with semi-structured argumentation interfaces, in which the user interrelates and organizes individual free-text analysis units to form a coherent overall argument. An important issue in the semi-structured argumentation approach is what expressivity and structuring assistance should be provided to the users in composing statements into a coherent analysis. At the same time, such interfaces can benefit from being easy to learn while being able to support analyses of different structures and in different domains. Based on our experiences of designing such easy-to-use, yet expressive interfaces, we argue for a particular approach to capturing overall argument structure: as a hierarchy of potentially heterogeneous nodes, with each node using a representation which matches the type of the analysis at that node. Node-level representations we have found promising include pro/con analysis and comparison of alternative competing hypotheses. To support additional analysis types suggested by our experience, we anticipate that additional representations, e.g., an event-structure based representation, may need to be added. We conclude by discussing opportunities for automatic assistance with specific analysis tasks, given this “hierarchy of heterogeneous nodes” approach.

Categories and Subject Descriptors

H.5.2 [Information Interfaces and Presentation]: User Interfaces. H.1.2 [User/Machine Systems]

General Terms: Design, Human Factors.

Keywords:

Structured

argumentation,

intelligent

user

interfaces

1. INTRODUCTION

Due to the emergence of asymmetric threats, intelligence analysis increasingly needs to support rapid analyses on a broad variety of topics. One common approach to supporting such analyses is using semi-structured argumentation interfaces, in which the user

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.

IUI workshop on Intelligence Analysis, Jan 29, 2006, Sydney, Australia. Copyright 2006 ACM 1-58113-000-0/00/0004…$5.00.

interrelates and organizes individual analysis units which denote pieces of evidence and working hypotheses to form the overall argument [e.g., 7,14,8].

An important issue in the semi-structured argumentation approach is what expressivity and structuring assistance should be provided to the users in composing statements into a coherent whole. Over the past several years, our group has explored a number of designs in this space [5,8]. Based on our experiences in designing such approaches and on feedback from intelligence analysts and other subjects who used our tools, we argue for a particular approach to capturing argumentation structure, namely a hierarchy of potentially heterogeneous nodes, with each node using a representation which matches the type of the analysis at that node.

Related work on structured argumentation often aims to support capture of a finished argument or incrementally adding to an argument [12,14,7]. It typically does not emphasize a more bottom-up approach of restructuring and modification in the process of evolving an argument from disconnected but potentially related pieces. Our “hierarchy of heterogeneous nodes” approach stems from our emphasis on ease of understanding, managing, and evolving the created argument. Other related work has also looked at improving activities related to sense-making, e.g. extracting summary statements from textual sources [11] or visualization of analyses (e.g. [15]). Such investigations also influenced our choice of proposed approach, as it is desirable for the approach to be integrated with these related techniques.

The rest of the paper points out some difficulties with creating, interpreting and updating highly expressive but elaborate argument structures, details our proposal for and experiences with structuring arguments, and finally discusses the opportunities for supporting the user in sense-making and related tasks afforded by the approach we advocate.

2. STRUCTURING ANALYSIS UNITS

We argue that overly specific semantic relations between analysis units, while possibly useful for capturing the details of an analysis, is not necessarily desirable in realistic settings. We also argue that interconnections in an argumentation system should be restricted to make the structure easier to grasp and to simplify updating of an argument.

We have started by exploring an approach supporting a variety of expressive, semantically defined connectors between individual statements. The connectors were motivated by rhetorical structures, logical relations, temporal relations, and so on. The approach also did not impose any restrictions or provide guidance on which statements could be interconnected, allowing formation

of complex networks of connections between arbitrary statements, as illustrated in Figure 1. In this system, the structure of the argument was extremely permissive and expressive. The lack of restrictions could make it difficult to interpret or update the structure of the argument.

The issues encountered included a significant learning curve due to richness of vocabulary of connectors. Also, in practice, certain relevant aspects were left implicit, complicating decomposition using precise connectors. Our experience also suggested that the arbitrary interconnections could make the structure difficult to modify or extend, as modification could affect any parts of the analysis.

Macintosh is more usable than Windows Macintosh is more usable than Windows is supported by Macintosh platform has a more stable OS AND Macintosh, as compared to Windows, has a friendlier UI Macintosh platform has a more stable OS stands though contradicted by Windows aims to surpass other platforms in security and stability Figure 1. Initial approach we explored allowed a rich set of structuring connectors with prespecificed semantics (shown in bold). Structurally, arbitrary interconnections are allowed, which made

interpreting argument structure more laborious.

Given these issues, we shifted away from expressive connectors and allowing arbitrary interconnections or ill-defined units. Instead, we required that each unit must be a complete statement, and connections between units to be structurally and semantically very simple – the analysis was to be represented as a hierarchy. A given free-text statement could be connected only to a single parent statement, and the only allowed connectors were “pro” and “con” (so for every statement several pros an cons could be specified, and so on recursively). Figure 2 gives an example of analysis in this format.

Macintosh is more usable than Windows pro: Macintosh platform has a more stable OS con: Windows aims to surpass other platforms in security and stability pro: Macintosh, as compared to Windows, has a friendlier UI Figure 2. Hierarchical approach we explored next. Only “pro” and “con” connectors are allowed, and the hierarchy takes the form of a pro/con tree. The argument structure is easier to create, understand

and update.

The hierarchy sub-levels can be collapsed or expanded to focus on all statements at a given level or drill down on a specific branch. The subtrees can also be dragged and dropped in place, a feature found useful and explicitly requested by users in our pilot studies of the system. This suggests the need for ease of changing the organization during the sense-making process, which may be more important than the precision that expressive connectors afford.

Our experience also supports the view that a hierarchy with multiple pros and cons at each level is a viable structure for capturing arguments on a wide variety of topics. In one collection exercise, more than 60 volunteers have, over the Web, entered more than 400 statements to create more than 20 arguments on topics ranging from computers to politics. In the following section, we review our additional findings and user comments on the usability of the approach in more detail.

2.1 Studies of Updating Pro/Con Hierarchies

We compared the pro/con structure with a more traditional analyst report format. We focused on settings in which an existing decision (possibly made by other users) needs to be updated or revised in light of new evidence. Specifically, we formed the following hypothesis:

Pro/con annotations of decisions help users modify existing decisions (possibly made by another person) more correctly over a traditional report because 1) pro/con annotations expose the rationale for the analyses in terms of intermediate hypotheses and supporting sources 2) pro/con annotations transmit this rationale to collaborating users

We have designed a controlled experiment comparing the pro/con hierarchy approach and an ablated version, which was modeled after traditional analyst reports. We tested our evaluation design with several subjects on the domain of computer purchase. The statements communicated the constraints on computer features and rationale for choosing the most likely one. They also pointed to potential weaknesses (cons) of intermediate alternatives and how the existing decision could be updated.

With the ablated tool, none of the subjects were able to produce a fully compliant (correct) result which satisfied all the decision constraints. With pro/con annotations, two of the subjects produced correct results. We have found that two ‘hasty’ subjects overlooked one decision constraint each. The subjects commented that with the pro/con annotations, the supporting sources are located close to the statements, which helped them find relevant information more easily. Some subjects also commented that pro/con annotations make tracking decision constraints easier than traditional reports.

For a controlled study with analysts, we compared pro/con annotations with a traditional report format. The scenarios examined were mockups of analyses of terrorist activities. Four analysts participated in this study. Each subject worked with a pro-con analysis on one topic, and a traditional report form on another topic. The subjects were given new sources for each analysis and were asked to update the analyses and select the most likely resultant alternative.

With the ablated tool (without the pro/con annotations), none of the subjects produced a correct result. With pro/con annotation, three out of four subjects produced correct results, satisfying all the decision constraints. We have found that interpretations of one particular source we used varied widely, which contributed to one of the subjects reaching an alternative conclusion. Overall, the pro/con structure was helpful in capturing the relevant features of the analysis, and users were able to insert new evidence into it and update their conclusion correctly.

Because the success of a tool depends on its usability to the analyst on practical tasks, we also investigated less guided use of the system with two analysts. They chose their own topics and collaboratively created a joint pro/con analysis. As the report was being built, each one updated the same analysis based on his/her own sources and shared the intermediate results with each other. Two reports were created from this study: 1) “What is the current status of the Iranian Nuclear Program” (23 pro/con statements, 26 uses of evidences from 17 unique sources) 2) “How likely is it that Al -Qaeda will attack a major city in the US using WMD such as chemical or biological weapons” (18 pro/con statements, 24 uses of evidences from 16 unique sources). The subjects generated each report in less than one hour.

Based on this experience, the subjects commented that our approach is easy to use and learn, estimating that it would take only 0.5 to 1 hour for an analyst to learn. They commented that the format makes it easy to understand quickly why and how different alternatives are likely or not likely. It was also easier to notice the need for supporting evidence and need for rationale. It was also mentioned that the pro/con annotations could be useful for ‘all-source’ reports, highlighting what sources contribute and how. The system could be used in analyses of complex issues which feature significant amounts of both relevant and irrelevant information, and could also support discussions among many analysts. Subjects also mentioned that the system could be a useful training tool in analyst schools, which suggests that the simple rather than expressive connectors and simple structure are sufficient to capture the important aspects of an argument well enough even to fully convey professional-grade analysis to trainees.

3. ARGUMENT-LEVEL STRUCTURE

We proposed that, although the overall argument structure is potentially arbitrarily complex, it may be well approximated with a hierarchy of heterogeneous nodes, with each node captured with one of few representations selected to match the overall structure of the analysis at that level. We briefly discuss merits of the hierarchical approach and then discuss the specific node-level representations.

Hierarchical organization. Structuring the issues hierarchically rather than as an arbitrary network gives up some expressivity, but greatly simplifies for the user the tasks of understanding managing and updating of the evolving structure. Hierarchical organization also supports management of attention, allowing the user to focus on the relevant entities at the same level, and drill down into the details when needed. For example, a hierarchy of pros and cons results in the pro/con tree structure which we have explored. Disallowing complicated interconnections also increases modularity of the structure, allowing insertion, modification, and removal of statements without need to first understand arbitrarily intricate implications. In our experience, this structure has been sufficient to capture the pertinent aspects of the analyses, being helpful by emphasizing the key interrelations.

Argument “nodes” and promising node-level representations. A key feature of the proposed approach is to capture analysis in terms of a hierarchy of nodes, with each node expressing organizing multiple statements and capturing relations between them. For example, a pro/con node contains multiple pro and con statements for a give statement. This approach allows introduction of entire structures rather than assembling an argument out of individual statements and connectors. We discuss particular node-level representations which we either found useful or believe should be added.

The aim of a given node-level representation is to be able to match the underlying structure of the analysis, thus relieving the burden of specifying specific argument connectors. A well-chosen representation can not only make the diverse factors “fall into place”, but can also provide an interface which simplifies understanding or extending an existing argument. We think it is desirable to have only few (carefully chosen) such representations, to avoid overwhelming or confusing the user about choice of one. We have already described one representation which we found useful in our work, the pro/con

decomposition. We describe a second representation which we have found promising in preliminary evaluations, and suggest a third that we think can also be very useful.

Our second representation supports comparison of multiple alternatives by features. The applicability of this representation and the associated non-assisted methodology to intelligence analysis has been discussed in prior literature as the analysis of competing hypotheses (ACH) approach [10, ch. 8]. We have implemented this representation, extending it with support for breaking out features hierarchically into sub-tables. Figure 3 presents an example.

Figure 3. Example of comparing alternatives based on several

selected features.

Individual cells in the table represent assessments of whether a given alternative has a given feature, and to what degree. Users enter alternatives, features, and assessments in free text, and can use graphical icons to indicate judgments of degree.

We conducted a preliminary comparison of this representation against pro/con for an example where four alternatives needed to be compared. In the pro-con case, the pros of the strongest alternatives with respect to a given feature were recorded. The user task consisted of identifying additional alternatives which would improve on current options and satisfy the constraints on the stated preferences and importance about the individual features. We found that using the multiple-alternatives representation users completed a test task both more correctly (not omitting important desired features) and in half the time. On the other hand, users also requested support for pro/con sub-levels to provide detail on statements entered in individual cells.

Organizing Based on Event Structure. Being able to structure and relate event-based information is important to capture structure of some intelligence analysis scenarios [16]. Analyses which concern events can have a number of steps which may or may not have happened, and can include alternate or parallel pathways. For example, development of biological weapons (BW) involves acquiring seed stock, acquiring production material, establishing production facilities, acquiring equipment, establishing worker safety, and producing BW agent (detailed process model available in [1]). The steps are related to each other temporally, e.g. developing BW agent should be done before weaponizing it. Hierarchical decomposition can introduce sub-steps for a given step. Such analyses can be supported with process models,

specialized by the user to represent and organize specific event related information at hand. The system may assist the user in modifying or validating the analysis based on the steps and constraints present in the models.

For an overall argument, it may be beneficial to allow mixing of types of structures, for example switching to pro/con analyses for specific event structure nodes.

4. OPPORTUNITIES FOR AUTOMATED SUPPORT

Given our proposed approach of a hierarchy of heterogeneous nodes, we identify several potentially high-payoff opportunities for automated support.

Assist with making a decision. Individual statements are organized into an overall structure so that a decision or assessment of evidence can be made. Having annotations such as pro/con or having alternatives evaluated with respect to specific features supports the process of weighing and combining available observations to draw a conclusion. For instance, once alternatives are listed and evaluated with respect to their features, the representation can support identifying the top alternatives, hiding the unlikely, “dormant” alternatives, or propagating the dismissal and acceptance conclusions from lower levels up the hierarchy. Also, because the impact of an individual ground-level statement in any representation depends on whether the statement is from a reliable source, it would be helpful to incorporate mechanisms for automated ratings of content trust on the sources. We have explored this issue [9] and extracted summary lessons from the extensive and diverse computer science literature on trust rating. We have identified more than a dozen factors which impact a user’s assessment of content trust [2].

Assist with collaboration. Because the overall structure we advocate is based on only few node-level representations, extending the system to support the collaboration and particular workflow of analysis is relatively simple. Only the individual node-level representations need to be augmented. One example of such augmentation which we have already implemented is extension of the pro/con analysis with collaboration features for polling other users’ agreement with hypotheses at various levels in the hierarchy (a feature that was also central to the SEAS system [14]). Future work can support determining what task to work on next, by adding visualizations of what part of the analysis may be unsupported (e.g. have only cons) or has gathered high levels of disagreement. Delegation of certain parts and sublevels of analysis has been explicitly requested by analysts who used our system. Such delegation can also be supported, with the outsourced sub-analyses being automatically included as they are completed.

Assist with finding evidence. Use of representations spanning argument levels rather than expressive individual connectors creates opportunities to automatically locate information to populate the specified structures. Partially filled-in representations can be used to formulate questions to be issued to information extraction and question answering systems. For instance we developed GrainPile [4], an extraction system which aggregates information from large corpora and quantifies expressions of degree, and could be used to suggest possible features for the “comparison of multiple alternatives” representation.

For process models, resources such as CYC [13] and VERBOCEAN [6] can aid in formulating additional question answering queries

to opportunistically look for information in raw text sources based on possible process stages. The user would be notified if a high-scoring match was found for a plausible question (e.g. evidence of purchase of raw materials for a dirty bomb).

Assist with generating a report based on the analysis. Finally, the task of generating a report based on the analysis emphasizes communicating major points concisely [3]. Identifying the key elements in a hierarchy containing pro/con analysis, comparison of multiple alternatives, or process models can be used to identify key points for the final report, together with their plausible ordering. This structure could then be adjusted and polished by the user.

In summary, we have argued for a particular approach to structuring arguments, based on our experiences as well as based on formal and informal evaluations. We have proposed that intelligent argumentation interfaces which aim to support sense-making may benefit from hierarchical decomposition, with several specific dedicated argumentation structures as the nodes (pro/con, feature-based comparison of multiple alternatives, and event-based structure). Finally, we have outlined several high-payoff opportunities for supporting argumentation based on the proposed structure.

REFERENCES

1. M. Archuleta; M. Bland; T. Duann; A.. Tucker and C. Widman, Proliferation Profile Assessment of Emerging Biological Weapons Threats, Technical Report, Air Command and Staff College, 1996

2. D. Artz, Y. Gil. A Survey of Research on Trust in Computer Science. Internal Report, Nov 2005 (publication forthcoming).

3. CIA. “Analytical Thinking & Presentation for Intelligence Producers”

4. T. Chklovski. Deriving Quantitative Overviews of Free Text Assessments on the Web. Conf. Intelligent User Interfaces (IUI06).

5. T. Chklovski, V. Ratnakar, Y. Gil. 2005. User Interfaces with Semi-Formal Representations: a Study of Designing Argumentation Structures. In Conf Intelligent User Interfaces (IUI 2005).

6. Chklovski, T., and Pantel, P. 2004. VERBOCEAN: Mining the Web for Fine-Grained Semantic Verb Relations. In 2004 Conference on Empirical Methods in Natural Language Processing (EMNLP)

7. Conklin, J. and Begeman, M. gIBIS: A hypertext tool for exploratory policy discussion. ACM Transactions on Office Information Systems, 6(4):303–331, October 1988.

8.

Y. Gil, V. Ratnakar: TRELLIS: An Interactive Tool for Capturing Information Analysis and Decision Making. In Proceedings of the 13th International Conference on Knowledge Engineering and Knowledge Management, Siguenza, Spain, October 1-4, 2002.

9. Y. Gil, V. Ratnakar: Trusting Information Sources One Citizen at a Time. In Proceedings of the First International Semantic Web Conference (ISWC), Sardinia, Italy, June 2002 : 162-176

10. R. Heuer, Jr.: Psychology of Intelligence Analysis. 1999.

11. R. Kaplan, R. Crouch, T. King, M. Tepper, D. Bobrow. A Note-taking Appliance for Intelligence Analysts. 2005 Intl Conf on Intelligence Analysis

12.

Li, G., Uren, V., Motta, E., Buckingham Shum, S. and Domingue, J. ClaiMaker: Weaving a Semantic Web of Research Papers. In Proceedings of the First International Semantic Web Conference on The Semantic Web (ISWC 2002). 2002.

13. D. Lenat. CYC: A large-scale investment in knowledge infrastructure. Communications of the ACM, 1995

14. Lowrance, J., Harrison, I., and Rodriguez, A. Capturing Analytic Thought. First Int. Conf. on Knowledge Capture (K-CAP 2001), 2001. 15. M. Maybury, J. Griffith, R. Holland, L. Damianos, Q. Hu, R. Fish. Virtually Integrated Visionary Intelligence Demonstration (VIVID). 2005 Intl Conf on Intelligence Analysis

16.

J. Pustejovsky, I. Mani. Event and Temporal Awareness for Intelligence Analysis. Tutorial at 2005 Intl Conf on Intelligence Analysis