Designing For Collective Intelligence

A collective intelligence application is one that harnesses the knowledge and work of its users to provide the data for the application and to improve its usefulness. The most hyped examples of collective intelligence applications have been labeled as “Web 2.0” applications. Web 2.0 is an amorphous term used to define a computing paradigm that uses the Web as the application platform and facilitates collaboration and information sharing between users.⁷ Classic examples of Web 2.0 applications include: wikis, blogs (or Weblogs), social network services, and social bookmarking.²

Collective intelligence is not a new concept. As long ago as 1968, computer visionaries foresaw the ability of computers to be applied to cooperation in creative endeavors by allowing people capable of solving specific problems to share their ideas.⁹ However, collective intelligence has been gaining momentum as new tools supporting collaboration have become available. The concept of collective intelligence is now being explored by businesses interested in using it for collaborative innovation³ and by researchers interested in addressing systemic problems like climate change.⁶

Collective intelligence is a fundamentally different way of viewing how applications can support human interaction and decision making. Most traditional applications have focused in improving the productivity or decision making of the individual user. The emphasis has been on providing the tools and data necessary to fulfill a specific job function. Under the collective intelligence paradigm, the focus is on harnessing the intelligence of groups of people to enable greater productivity and better decisions than are possible by individuals working in isolation.

The shift to a collective intelligence paradigm requires software developers to have different ways of thinking about how their how software might be used and what features would enable better visualization and use of information among groups of people. The new breed of collective intelligence applications needs to center around user defined data that can be reused to support decision making, team building, or to improve understanding of the world around us. The users of these systems should play a central role in defining what data is important and how the data is used. The essential features of collective intelligence applications are similar to the design patterns for Web 2.0 applications except that collective intelligence applications can be custom applications designed for small highly specialized domains instead of the larger Web audience served by most Web 2.0 applications.⁷ The seven principle collective intelligence application requirements are (adapted from O’Reilly⁷):

1. Task specific representations: Domain specific collective intelligence applications should support views of the task that are tailored to the particular domain.
2. Data is the key: Collective intelligence applications are data centric and should be designed to collect and share data among users.
3. Users add value: Users of collective intelligence applications know the most about the value of the information it contains. The application should provide mechanisms for them to add to, modify, or otherwise enhance the data to improve its usefulness.
4. Facilitate data aggregation: The ability to aggregate data adds value. Collective intelligence applications should be designed such that data aggregation occurs naturally through regular use.
5. Facilitate data access: The data in collective intelligence applications can have use beyond the boundaries of the application. Collective intelligence applications should offer Web services interfaces and other mechanisms to facilitate the re-use of data.
6. Facilitate access for all devices: The PC is no longer the only access device for internet applications. Collective intelligence applications need to be designed to integrate services across handheld devices, PCs, and internet servers.
7. The perpetual beta: Collective intelligence applications are ongoing services provided to its users thus new features should be added on a regular basis based on the changing needs of the user community.

The processes involved in designing and implementing specialized collective intelligence applications are discussed below in the context of DDtrac, a Web-based application that allows for the easy collection and summary of special education data.

Back to Top

The DDtrac Application

To illustrate how features of collective intelligence systems can be combined and used in special purpose applications, a system that combines distributed data capture, commenting, tagging, a wiki and a blog was created. DDtrac is an information system designed to collect and summarize information that is used to improve decisions related to education and therapy options for special needs children. The heart of DDtrac is a Web-based application that allows for the easy collection and summary of special education data. Figure 1 shows the DDtrac system architecture follows a basic hierarchical structure with functionality grouped into four major areas: data entry, creating goals and objectives, data analysis, and administration. The program is designed to allow data input from a Mac, a PC or a handheld device with a connection to the Internet.

The special education domain was selected for this study for three reasons.

• First, special education students often have many people involved in their education and therapy team. Students can receive services in school, at home, in clinical outpatient environments and from consultants from other regions of the country. These practitioners rarely have time to communicate details related to student progress which can lead to an inefficient duplication of effort, gaps in treatment, or team members working towards conflicting goals.
• Second, special education literature emphasizes the importance of using data collection and analysis procedures to monitor academic, social and behavior progress of students with intensive special education needs.^{1, 4, 5} However, currently there are no applications that support the data collection and analysis needs of this sector.
• Third, there is the opportunity to harness the collective intelligence of these practitioners to identify patterns of behavior and best practices for both individual students as well as groups of students with similar disabilities.

This suggests a need for software applications that can simplify the data collection and analysis activities of special education practitioners and harness their collective intelligence to improve their ability to make decisions about the children they serve.

The DDtrac collective intelligence application serves two primary purposes. First it serves as a communication medium for therapists and teachers so that they know what to do when they sit down to work with the special needs child. Second it collects data and provides data analysis tools to enhance the ability to assess the adequacy of student progress and determine whether and when instructional adjustments are necessary. The DDtrac system, shown in Figure 1, meets the needs of its users by supporting all seven collective intelligence application requirements.

Task-specific representation. The education programs of developmentally disabled children are highly individualized. Each special education student has an Individual Education Program (IEP), which establishes long-term goals and short-term objectives tailored to the needs of the individual student. The IEP is an important document because it defines the direction for treatment to be taken for the upcoming year. The IEP also should define the types of data the instructional team will collect to demonstrate the student is making adequate progress towards his or her goals.

DDtrac includes task specific representations that allow the special education team to define all of the specialized parameters necessary for an IEP. A goals and objectives wiki allows the education team to define the goals, objectives, how the objectives will be taught, and what types of data will be collected. The goals and objectives are then automatically translated into a set of customized data collection screens tailored to the individualized data collection needs of the student. Figure 2 shows a series of data entry screens as would be displayed on a PC.

Data is key. The purpose of the DDtrac application is collecting, sharing and analyzing special education data. For example, the customized data collection screens shown in Figure 2 are designed to make it easy for team members to collect the data related to the students’ performance on academic or other structured tasks (such as life skills). This data is used to evaluate student performance on instructional tasks and determine if the instructional methods being used are allowing the student to make adequate progress towards their IEP goals. DDtrac also supports the collection of quantitative data related to the student’s progress on social and behavior goals.

Qualitative data is also an important part of the information exchanged in many collective intelligence environments. Qualitative data in special education programs is frequently the type of data that capture the complexity and the transactional interaction between the setting and the student’s performance or behavior.⁸ DDtrac allows the capture of observation notes related to instructional activities, during social interactions and following behavior episodes. These observation notes are stored in a student-centric blog that includes the date and time they were recorded, the name of the practitioner making the comment, and the name of the student the comment is being made about. The descriptive notes represent the practitioner’s best efforts to record what is occurring in the context of the therapy session (for example, describe student mood and overall performance on tasks). The qualitative observations can also include the practitioner’s interpretation of how or why certain behaviors unfolded as they did.⁸

Users add value. Similar to traditional blogs, the commenting feature allows DDtrac users to comment on observations made by others. The comments are attached to a particular student-centric blog post and allow users to share experiences and provide suggestions related to issues raised about a student. It allows for a dialog between users so that the approaches that work best with a given student can be identified and adopted by the entire education team. The ability to comment and share insights is critical to DDtrac’s support of the collective intelligence of special education teams.

Facilitate data aggregation. In special education programs student progress is often uneven and can be agonizingly slow. Simply evaluating the student’s scores on their instructional tasks is not sufficient to understand how they are progressing and whether the educational interventions are providing the support necessary to allow the student to meet his or her goals. DDtrac includes a number of reporting and charting features that make it easier for special education teachers and therapists to examine student progress and modify student’s objectives and targets to maximize a student’s learning outcomes. For example, the unmastered target report shows all of the instructional targets the student has not yet mastered along with the number of days they have been worked on, the number of times they have been practiced and the student’s average performance on the target in the past month. This makes it easier for the teacher to identify the things the student is having the most difficulty learning.

Figure 3 shows three charts provided that can be used to aggregate student data to facilitate assessment of student progress. The stacked bar chart shows how individual behaviors contribute to the overall number or duration of behaviors observed for a student. The multi-line chart shows how a student is performing on multiple instructional targets. Finally, the mastery chart shows the rate a student is mastering targets as he or she works on an IEP goal.

In paper-based special education data collection environments much of the qualitative data found in the daily comments are lost within days of capturing it. Performing any meaningful analysis of this qualitative data (which can be accumulated over periods of years) is virtually impossible.

Tagging is one mechanism that can be used in collective intelligence applications to facilitate the aggregation of qualitative data. For example, DDtrac allows the semantic tagging of the narrative comments taken as a part of the daily instructional, behavior or socialization observation notes taken by practitioners. These semantic tags closely resemble the tags common on many Web 2.0 sites (such as Flickr, Delicious, Blogger, and so on). They are freely chosen keywords which allow for overlapping associations and that can be used for later retrieval and analysis of specific comments. For example, a student may exhibit a finger flicking behavior infrequently. The practitioner might note this in the daily notes along with other observations. Then, if the behavior becomes a problem, the practitioner could retrieve all of the comments tagged “flicking” to look for any patterns.

Facilitate data access. In today’s data intensive world, no application can afford to exist in isolation. This is as true for special education applications as it is for business applications. Student and staff data exists in existing school databases and needs to be easily imported into DDtrac. Detailed data collected in DDtrac also needs to be able to be exported so that the special education teams have the freedom to analyze it in ways not currently supported by DDtrac, or to combine it with other data not available in the system. The initial prototype DDtrac system supports these needs by allowing the import and export of data using comma delimited files. However, to more fully support this important collective intelligence application requirement, future versions of the system should also support the automatic exchange of XML data using Web services.

Facilitate access for all devices (and locations). Special education instruction frequently occurs in locations throughout a school (such as the special education classroom, specialized therapy rooms, the regular education classroom, in the gym or on the playground) and can also occur at home and in the community. Similar to traditional Web 2.0 applications, DDtrac uses the Web as the application platform which allows data entry forms customized to the individual student to be accessed from anywhere there is an internet connection. DDtrac users can choose to take data in two different ways: either online using a networked PC or offline, using a downloadable Web form which can be uploaded later when a network is available.

Another way applications can support distributed data entry is by supporting a wide variety of data input devices. Using the Web as a delivery platform is one way to support multiple platforms. However, to support handheld devices in addition to traditional PCs, DDtrac also uses style sheets targeted at different devices (such as mobile devices) and has data entry screens designed for devices as small as 320 pixels wide. In addition, the quantitative data entry is all accomplished using standard HTML form elements and were optimized for use with a stylus, the common data entry tool on many handheld devices.

Perpetual beta. The DDtrac system was designed so that it is easily extensible when new features are identified. The ability of the system to evolve in response to the needs of users was considered in the very early stages of the project. It was designed using simple modules that are easy to adapt or extend as needs change. In addition, communication channels were established to facilitate obtaining feedback from users. A wiki is being used for all software documentation and a blog is used to communicate with users. Both are essential for feeding the collective intelligence of users back into the software.

Back to Top

Collective Intelligence in Practice

DDtrac was deployed in an eighteen month field trial with one student with autism. The student participated in speech therapy, occupational therapy, ABA therapy, and socialization therapy known as Relationship Development Intervention in a home environment as well as receiving special education services at school. The practitioners working with the student rarely met in person and instead used the Web-based DDtrac application for data collection and communication. The field study participants used the initial prototype within days of it first being developed and their feedback helped guide the ongoing software development. The advantage of this approach was that it allowed the developers to start small and add features to the software as the volume of data grew and additional needs were identified. Over the course of the eighteen month trial data was captured for 481 separate work sessions and included more than 50,000 individual pieces of data. The wiki and the blog were active for the final 6 months of the project. During the six months the wiki had 163 pages added and the blog had 33 posts documenting 12 software upgrades.

All participants in the field study felt that DDtrac significantly enhanced their ability to take data and evaluate the performance of the student they were working with. They reported the following benefits to using DDtrac:

1. The task specific data collection features made data collection easier and faster.
2. The student centric blog enabled them to quickly understand the student’s recent behavior trends and better prepare for their own work sessions.
3. The ability to analyze the data in a wide variety of ways enhanced their ability to assess student progress and made it easier to comply with mandatory reporting requirements.

Back to Top

Conclusion

This article provides a framework for designing specialized collective intelligence applications and demonstrates the applicability of this framework through the development of a special education collective intelligence prototype. The prototype development and subsequent field trial makes two primary contributions to our understanding of collective intelligence applications. First, it demonstrates the benefits specialized collective intelligence applications can provide in specialized domains. Second, it maps the design attributes and features of a specialized collective intelligence application back to the design principles proposed as a part of the collective intelligence framework. This allows future developers to conceptualize how these abstract guidelines can be used to guide practice.

Use of a specialized collective intelligence application, like DDtrac, can potentially benefit organizations in a wide variety of domains (such as health care, outsourcing environments). The ability to apply the collective intelligence of individuals working on similar problems is an area that has just begun to be addressed by software developers; however, these systems will change the way information is shared and used and has the potential to dramatically improve decision making.

Additional information on the DDtrac project and software can be found at: http://developingmindssoftware.com.

Back to Top

Back to Top

Back to Top

Back to Top

Back to top