Representation Issues in IR Systems

LIS 4/5523: Online Information Retrieval

Dr. Manika Lamba

Introduction

Welcome to the second lecture of Week 3, Representation Issues in IR Systems.

One of the most important aspects of being a good searcher is choosing appropriate search terms. Understanding more about how the terms used in the records of the system are chosen, the sources of terms, and the impact on retrieval can help you utilize the features of the system to the fullest extent.

Recap

• In Part 1, we looked at information systems and database structures. We also addressed representation and indexing from the data modeling and technical side

• In this lecture, Part 2, we are going to look at representation and indexing from the human/creator and retrieval side

  • Indexing as process and effects related to human indexing
  • Effects of indexing decisions on retrieval

In Part 1, we learned about systems from a systems analysis perspective. We also defined some essential terms for understanding databases and systems and worked through the process of data modeling. We addressed representation from the system/technical side and discussed how inverted indexes work within a database. We also examined the system structure in some common systems you may use.

In Part 2, we are going to address representation and indexing from the human and retrieval side. We will discuss the process of subject analysis and choosing index terms to represent subjects in catalog or database records. We will also discuss some human factors that have an impact on retrieval.

Information Retrieval

• The central problem of IR is how to represent documents for retrieval
• To be more successful, document representation must be used in ways similar to the ways ordinary language is used
  • document representations should take context into account
  • document representations should take users into account

Let’s begin by looking at how IR and representation are linked. The central problem of IR is, as you have seen from the models we have examined in the last two lectures, how to represent documents for retrieval. We can also think about IR from the system’s side of how documents are represented by human agencies as well as computerized agency within the system itself.

To be more successful, we’ve found in the research, document representation must be used in ways similar to the way that ordinary language is used, which, again, presents some problems. We’ll talk later in this class about the natural language processing model of IR and some of the issues associated with the NLP method of information retrieval. Document representation should also take context into account, the domain of the system into account, as well as the users of that system. The combination of these factors can make representation quite a complex problem.

In Library IR Systems

• Descriptive Data
  • We use MARC records and MARC record encoding standard to hold the bibliographic indexing of the objects
  • We use RDA (Resource Description and Access) content standard
• Subject Representation
  • We use 6XX fields in the MARC record structure
  • We use subject headings list, thesauri (controlled vocabularies) to construct subject indexing
  • We use classification systems (eg. Library of Congress Classification, Dewey Decimal Classification) to assign classification codes

In library IR systems, and really in any system representing a collection of information objects, whether it be a physical collection, one that is online, or even one comprised of humans, contains two “buckets” that are represented in the record of the database the container and the content buckets as we discussed in Part 1.

In library systems the records include descriptive data (container oriented) and subject data or representations (such as subject headings). In the library catalogs we use MARC records and the structure of MARC to hold the bibliographic data that represents the items in the collection. We use the RDA content standard to form the entries in each field of the MARC record.

For subject representation, we use another standard or a list of subject terms, or what we call headings, to represent subjects. There are many different controlled vocabularies a library may use. The most common in public and academic libraries is the Library of Congress Subject Headings (LCSH), while Sears may be used in school libraries, or LC’s Subject Headings for Children might also be in use.

We also represent subject (content aspects) of the items in the collection using classification schemes such as Dewey Decimal Classification or LoC classification. If you have taken 5043 you were introduced to RDA, subject headings lists, and classification schemes.

In databases such as Ebsco or Proquest , they do not necessarily follow the same standards for creating records in the database. An indexer may use a controlled vocabulary, such as LCSH but depending on the domain/discipline of the database a different controlled vocabulary may be in use. Also, some database companies develop their own controlled vocabularies that are shown to the user through the Thesaurus feature in the system.

So, there are several standards, or sets of rules, that are used when creating representations in either library catalogs or in a specialized proprietary database. Oftentimes the user does not know which standards are used in the system unless there is a feature that allows the user to view the controlled vocabulary or list of subject terms or facets used in the system. As noted earlier, this feature is often called the Thesaurus.

Other IR Systems

• Digital libraries/institutional repositories
  • Use metadata schemes like Dublin Core, Darwin Core, MODS, METS, etc
• Online databases
  • Follow ANSI/NISO standards for field structures in databases
  • Follow ANSI/NISO standards for content standards–related to controlled vocabulary use in databases
• Web??

There are a host of other IR systems that you use all the time such as digital libraries/collections, institutional repositories, such as OU’s ShareOK system, and online databases of many varieties. We will use many of these systems later in the semester when we get into the searching portion of class. Each of these different systems has a metadata scheme or database scheme in use to create the representations or records of the objects. For example, digital libraries or institutional repositories might use a metadata standard such as Dublin Core, MODS, or METS to describe the objects in the collection. In archives MODS is frequently used but also a standard called EAD (Encoded Archival Description).

Online databases such as Proquest or Ebsco , use standards developed by national standards organizations (ANSI/NISO) to determine both the data schema and the controlled vocabulary to use in the system.

What does the Web use? Does the Web use any standards or controlled vocabularies? In intranets, the systems within corporations or businesses, they may have developed an ontology to represent the items in their organization. Ontologies can include both the database scheme and a specialized controlled vocabulary or categorization developed specifically for that organization.

Online retails sites, such as JC Penney, for example, have specialized ontologies to represent their retail collections. Many of these ontologies online serve both to represent the physical and subject aspects of objects, through providing descriptions of the objects but also categorizing the items into faceted categories for retrieval. Take a closer look at Amazon sometime to see how it is structured and the categories used in the system.

What about search engines? Search engines do not use any predetermined standards like we use in libraries or other information organizations. As the creators of webpages and sites may be companies or individuals, no standards are imposed when they set up a system or online database. There are sets of best practices that are used in interface design and techniques for making the website more accessible, such as search engine optimization, but little is known about what big companies like Google follow.

Representation

Definition

A system for choosing or highlighting some characteristics (attributes), together with a specification of the rules for selection (code)

This implies a tradeoff – if some characteristics are highlighted, other characteristics are left behind

Next, let’s define ‘representation’ within the broader context so we can talk about the process of representation. We talked about representation in Part 1 from the system side, but let’s look even more specifically about some of the issues related to representation within IR systems.

Representation can be thought of as a system for choosing or highlighting some characteristics, or attributes that are general characteristics, together with the specification of the rules for selection and the code, meaning that we should understand a little bit about how the system is structured and how data exists within the system’s records.

This selection (or surrogation ) process implies a trade off of some kind. If some characteristics are highlighted, then we have to assume that other characteristics are left behind.

In the data modeling process the database developer decides what to highlight and what not to include in the system.

So, for example, in library catalogs we made decisions on what attributes of the information objects in the collection were important to describe in the records for users.

In early catalog records these attributes included, Title, Author, Subject, and Classification. In order to create those representations, the card catalog records, we had to decide which characteristics to highlight and which to leave behind.

More Definitions

• ENTITIES: objects or concepts
• ATTRIBUTES: characteristics of entities
• DIACHRONIC Attribute: stable across time
• SYNCHRONIC Attribute: changes across time

Okay, let’s also look at some other definitions that we need to consider. In terms of representations, we have to consider the term of ‘entities.’ Entities are those objects, or even concepts, that we’re going to represent.

‘Attributes.’ And we talked about these earlier as the general characteristics of information objects. So, we can also think of them as characteristics of the entities that we’re representing.

There are different types of attributes; there are both ‘diachronic’ and ‘synchronic’ attributes. Diachronic attributes are those that remain stable across time. In other words, they do not change, and their meanings and uses also do not change. And then there are synchronic attributes, and those are attributes that change across time. And that could be as simple as the use of a particular term or concept within a specific language.

Diachronic Attributes

To illustrate the concepts of diachronic and synchronic attributes let’s use the example of a portrait pained by John Singer Sargent in 1883 or 1884. The model is Virginie Gautreau and the painting was displayed in the Paris Salon in 1884. The diachronic attributes include: the painter’s name, the model’s name, and the date of creation. Provenance of where it was displayed is another example of a diachronic attribute.

Synchronic Attributes

Apparently the painting was quite the scandal in Paris at the time due to the “lack of clothing” on the model and the fact that more than just the head, neck and shoulders were shown. Gautreaus’s mother was outraged and wanted Sargent arrested. This action severely crippled his career as a portrait painter but then just a few years later when the scandal died down he was admired for his technique, ground breaking subject matter, and became the most sought after painter in Britain and the US. If you were to describe the subject of this painting in 1884 versus a few years later, the subject terms would be different due to the stigma associated with it at the time it was painted.

This is just one example of synchronic attributes. The human language changes constantly and terms to represent subjects fall out of favor or take on new meaning. Take for example the lowly pound sign on a phone or keyboard, It has now been elevated to be the symbol for hashtags. Terms also can become politically incorrect, such as the term Indian, to represent Native American, which now is also represented by American Indian or Indigenous People.

Human Indexing

• Diachronic attributes (don’t change)
  • author, title, publisher, number of pages
• Synchronic attributes (change with time, context)
  • eg. meaning/uses of words over time
• Rules not evident to users

The problem here is that there are several factors related to human indexing of particular documents within collections. As I’ve said, we have diachronic attributes, and those are the ones that don’t change. They’re always the same when representing something in the system. Examples of diachronic attributes would be author, title, publisher, number of pages.

The problem is that oftentimes representation systems have only the most general thought about the users. We’re designing this system for retrieval, and the different agencies have different specifications for how or what they want to be retrievable within their system. But we’re oftentimes unaware of how the users actually use those particular systems or how they actually search for items within the systems.

Another problem that comes up with human indexing is that the rules in which we create representations are not evident to the users. How many times have you used a library catalog, only to have results come back to you that are really confusing with abbreviations you don’t understand or no information on how to access the resources?

So, again, there are different aspects of representation that are not evident to users or are confusing to the users.

Brian O’Connor, likes to say that when we’re thinking about representation and the different factors from a human side that impact retrieval, “there’s a great vagueness and generality resting on a foundation of shifting quicksand.”

Because if you think back to the IR/representation model that we looked at earlier, users are only concerned with one side of the model; they’re concerned with how they actually input their information need or their query into the system.

As the person who is creating the representation, we have to be concerned with both sides. We have to know the rules for creating the representation, including the structure of the system, but we also have to understand how the users are interacting with and/ or searching within that particular system . As a searcher you have to also be concerned with both sides you have to understand how the representations may be structured in the system but also have a good understanding of the information need so that a match can occur between the two representations.

What is Subject Analysis?

Definition

Determining intellectual content or subject content or aboutness

Types

• Document Analysis: information professional (cataloger, indexer) studies document to determine document surrogate for system

• Query Analysis: information professional (intermediary) or end-user studies user request to determine search terms

When a cataloger/indexer decides on how to represent the subject of the item in the system, they go through a process called subject analysis. This process is defined as “determining the intellectual content or subject content or aboutness of the object”.

There are various types of subject analysis such as document analysis that the cataloger/indexer conducts to determine the subject(s) of a book or journal article.

A web developer might also conduct document analysis when determining categories for an online retail site. The second type of subject analysis is query analysis that an information professional conducts prior to conducting a search. We will learn more about query analysis in a later module.

Purpose of Subject Analysis

• Clarify and organize subjects of docs and queries
• Express subjects precisely
• Achieve consistency between document and search terms

There are several purposes for subject analysis as listed on this slide. The primary purpose is, of course, to determine the subject(s) of an item and to choose appropriate terms to use to represent the subject(s) within the record. A very important aspect of this process is to represent subjects consistently using the same term throughout the system when describing a specific subject.

The How of Subject Analysis

1. Familiarization: Acquainting oneself with general content of document and query

2. Extraction: Identifying pulling out significant concepts and natural- langugage terms

3. Translation: Converting extracted terms into controlled vocabulary of system

4. Formalization: Applying rules for exact format, spelling, punctuation, codes, etc. for input to system

There are either 4 or 5 steps to subject analysis, depending on what source you are referring to. The approach shown on the slide is preferred.

1. The indexer/cataloger familiarizes themselves with the content of the document or if we are talking about searching, with the content of the query or information need.

2. They would then identify or extract terms that are used within the document by the author. These are called natural language or derived terms. A searcher would identify a set of terms that might be used to represent the query of the user.

3. In the third stage, the translation stage, the cataloger/indexer would translate the terms they extracted in step 2 into controlled vocabulary terms used by the system but consulting the subject heading list or controlled vocabulary. A searcher would conduct the same step but would translate the query terms into system terms by using the Thesaurus feature of the system, if one is available.

4. In the last step the cataloger/indexer would use the rules of the system (such as a cataloging standard or code) to form the subject heading. A searcher, would consult the help screens to determine how to enter the terms into the system or if using a thesaurus feature, they would follow the screen prompts to enter the terms.

The How of Subject Analysis

Subject analysis is a dance….

• based on literary warrant (information objects/author’s intentions) and on user warrant (user needs)

• requiring evaluation and verification at every stage in a continuous, iterative process

I like to think of subject analysis as a dance between the user and the author and the record creator. The searcher/indexer has to take into account the literary warrant of the author (the author’s intention) for the content of the document along with user warrant (the user’s needs for the document).

In searching this dance requires evaluation and verification at every stage in an iterative process as searches are constructed and conducted choosing appropriate search terms that meet the user’s idea of what they need and how the author and indexer may have represented the document in the system. The dance then requires the searcher to think about the types of document within a system, how the author(s) may have represented the content, and how the indexer/cataloger may have represented the document in the records.

The When of Subject Analyis

• During production of primary document

  Author's abstract and/or index
  
  Indexing commissioned by publisher
  
  Cataloging in publication (CIP)

• Prior to storage for retrieval

  Cataloging or indexing by bibliographic utility
  
  Cataloging or indexing by individual library

• During information retrieval

  Problem statement or question from user
  
  Query formulation by intermediary or user

This dance, and the process of subject analysis takes place

1. during production of the document,

2. prior to storage for retrieval when the cataloger/indexer creates the record, and

3. during the information retrieval process.

Indexing

Indexing

Process of creating index for purpose of representing and providing access to information objects

May be performed by humans or computers

Index Entry

Any pointer or indicator included in an index

This indexing process is conducted by either a library cataloger creating MARC records in a library catalog or by a database indexer describing articles in a database like Proquest. The indexing process also is conducted by website developers or content managers of online retail or other online sites.

Indexing can be performed by humans or by computers. The Chowdhury readings discuss computerized indexing in more depth than we will go into now but we will address computerized indexing and abstracting in later module.

What is produced by the indexing process is a pointer or indicator, usually a record that includes the physical description and subject representation of one object, whether a book, journal article, website, etc.

Index Entry and Term

Index Term

Any word/phrase used for physical or subject description

Any word/phrase used to search for and retrieve document

May describe any attribute of document (author, title, year, subject, etc.)

Within the record or index entry is the index term or terms used to represent the document.

Guessing Game

So, the dance begins between the indexer and the user. The indexer describes the document, where the user is trying to predict the document that they need that will help them resolve an information need. The indexer predicts use of the particular items that they’re representing. The user, on the other hand, has to somehow describe how they’re going to use the document or the object in your collection to satisfy their information need. They describe this use through search or query terms they input into the system.

So, if you look at it from this perspective, you can see that there are a lot of places where retrieval will break down. If the indexer and the user do not correspond in terms of how the document is described and how they’ll use the documents or how the indexer believes the document will be used versus how the user is actually describing their use, then there can be problems in retrieving documents from the collection.

Indexing Factors Affecting IR Performance

Indexing	Consistency Subject expertise Indexing experience
Types of Knowledge	Search Experience System Knowledge Domain Knowledge
Affective/Cognitive	Motivation Level Emotional State

We should also consider human factors, which can be associated with the indexing itself, such as the human indexer’s types and levels of knowledge, as well as affective and cognitive variables that affect humans on a daily basis.

We can also discuss the measure of inter-indexer consistency and inter-indexer inconsistency. The research has shown that there’s really at the highest level about 30% of consistency between any two indexers that are creating subject terms for the same document. So, consistency can be an issue. Also, subject expertise and indexing experience play into whether or not the index terms chosen are appropriate, accurate, and complete. If a person has a high degree of subject expertise, or domain knowledge, in a particular area, such as chemistry or ethnobotany, then they’re probably going to know the literature of the field and they’re going to know the appropriate levels of specificity and exhaustivity when they’re indexing.

There are also, as I said, types of knowledge, for example, I just mentioned domain knowledge and the indexer’s understanding of the domain itself, but also their search experience. Generally people that have a lot of experience with different search systems and have looked at the different structures within them and also use different techniques for searching are better indexers because they understand the effects that their decisions play on retrieval.

There are also factors related to motivation or emotional state. Research has looked at motivation level of indexers and some of the errors they make if they’re unhappy in their job, or the emotional state of individuals and how that might impact their indexing as well. There are not any recent studies in this area, but we do know that motivation and emotional state do have an effect on indexing. So, these human factors and the decisions an indexer makes when choosing terms to represent a subject of a document can potentially impact retrieval.

Indexing Factors Affecting IR Performance

System Factors

Index Language	Specificity Level of coordination
Indexing Assignment	Exhaustivity Specificity of term Accuracy

We had talked in an earlier lecture about some of the indexing factors that affect information retrieval performance. These are also evaluation measures or metrics that we use when we evaluate information retrieval systems. Some of those system factors are related to, say, index language and the indexing assignment within the system itself.

So, when we discuss system factors related to the index language, one aspect we evaluate is if the indexer is choosing the appropriate vocabulary to use within that system. And then also have you chosen the appropriate levels of specificity in terms that you’re using to assign subject terms to particular documents. Also, is the level of coordination of terms within the system accurate? Do you have similar concepts being represented with the same subject terms, for example? Or is the specificity level consistent throughout the system? And is it also consistent with the users who are interacting with the system?

In terms of indexing assignment, we evaluate if the level of exhaustivity is appropriate. For example, how are we representing the items within the system itself? Are we representing a broad, overall subject, or are we representing multiple topics and subtopics for each document? Also, how many different terms are used to represent the subjects within documents. Specificity is also part of this. Are we using the accurate level of specificity in the terms that meets the level of specificity in the documents themselves? Are accurate terms being assigned to represent those subjects? Are we representing the subjects themselves accurately and completely within the document?

These factors all require a very good understanding of the documents in the collection and the potential users who will use the system. For example, would it be appropriate to use LCSH as the controlled vocabulary for the system when its users are very young children? What would your level of specificity be when representing books for children?

As a searcher, it is important to understand these factors when constructing queries within the system.

Considerations when Choosing Terms

• Attributes of documents and users
• Record field that will contain the data
  • out of several potential kinds of subject fields
• Domain
• Scope
• Specificity
• Exhaustivity

There are other considerations catalogers/indexers have to make when choosing terms to represent subjects. And that searchers should consider when choosing terms for their queries. We will discuss each of these in turn.

For example, the attributes of the documents and the attributes of the users and their queries can impact the choices we make regarding choice of subject terms. Knowing about your users types/levels of knowledge will help you determine age-appropriate, subject-specific terms to use in subject description.

The field within the record that will contain the data also has bearing on which term(s) to choose.

Domain, Scope, Specificity and Exhaustivity are also terms we need to discuss when talking about considerations catalogers make when choosing subject terms.

Domain refers to the specific subject area or discipline of the subject. It would also include an understanding of how a specific discipline is organized, what terms are used in the discipline, what subjects are studied, who the main authors are, etc.

Scope refers to the limits of the discipline within the specific collection. In other words, what is included in the discipline or collection and what is NOT included.

Domain and Scope

Domain: overall subject, topic, discipline, or theme

Scope: Extent or limitations of domain

Considerations

• Breadth and complexity of subject area/discipline
• Users’ vocabulary level
• Terminology used in documents

Domain is particularly important for searchers. One of the first decisions you will make when beginning a search is to choose the appropriate resource to use, such as a library catalog, subject specific database or repository. Knowing the domain of the topic will help in this decision. Also, knowing the domain will help you determine terms to enter into the system that would be in use within the documents and by the authors (literary warrant).

Scope helps you determine if the system includes not just the topic you are interested in but also the extent of the database, such as the years covered, whether there is complete coverage of the topic area and date range that will be useful for your query.

Other considerations of domain and scope are listed on the slide, such as the breadth and complexity of the subject area, This will give you an idea of the level of language used by the authors that might be appropriate search terms.

Specificity and Exhaustivity

Specificity: extent to which index terms precisely represent the subject of the document. Can be general or more specific.

Exhaustivity: extent to which indexing represents all concepts in a document.

Considerations

► Level and complexity of terms in subject area/ discipline

► Users’ vocabulary level

► Terminology used in documents

Example

► Specificity is high if detailed math topics covered, e.g., set operations

► Exhaustivity is high if all math operations in textbook covered

Other considerations include specificity and exhaustivity levels that we represent a subject at within the records.

Specificity is the extent to which index terms precisely represent the subject of the document. Can be general or more specific.

Exhaustivity is the extent to which indexing represents all concepts in a document. The exhaustivity level may vary from one of Summarization (overall concept represented) to Exhaustive (all concepts represented)

We need to consider:

• Level and complexity of terms in subject area/discipline
• Users’ vocabulary level
• Terminology used in documents

Each of these must be considered so that we can choose ageappropriate, subject-specific terms to represent the concept or concepts of the document or when searching within the system.

Here is an example. We have a collection in a middle school which consists of mathrelated objects. It is limited to only those required by the state’s curriculum standards, so only subjects on those topics would be included. Specificity is high if detailed math topics covered, e.g., set operations, Boolean logic, etc. and exhaustivity is high if all math operations in textbook covered.

A closer look: Exhaustivity

Specificity and exhaustivity can really be thought of as a continuum. On one side, we have Summarization where we represent ONLY the overall subject of the object with few terms (usually 1-5 subject headings). On the other side, we represent the overall subject but also any other subtopics we deem necessary using MANY terms. Usually we are close to the middle ground in subject representation, describing the overall subject and a few subtopics with a moderate amount of terms.

Recall

RECALL= Relevant Documents in a IR Set/All relevant documents in the database

• A measure of how good a system is at retrieving all the relevant documents

• Inversely related to precision

• Dependent upon the users expectations and objectives

• Difficult to estimate. Need to know the number of relevant documents in the entire collection

Domain, scope, specificity and exhaustivity can affect the results of the search. Two measures used in IR are recall and precision. We will address each briefly here and come back to them in Modules 4 and 5.

Recall is the measure of how good a system is at retrieving all the relevant documents that are part of that system’s collection. The equation used to evaluate recall is included on the slide for you. We do know that in most cases and in most retrieval systems, recall is inversely related to precision. So, if you have a high level of recall, you have a low level of precision.

Precision

PRECISION = Relevant documents in a retrieved set/All documents in the retrieved set

• Measures how good the system is at “not” retrieving non-relevant documents
• Dependent upon user’s expectations and objectives

Precision, then, measures how good the system is at not retrieving those nonrelevant documents–in other words, bringing back documents that are only related to your query and nothing that’s irrelevant. This again, is a measurement that is dependent upon users’ expectations and objectives. The user is the only one that can really evaluate whether or not a document is useful to them. And again, it’s difficult to measure precision because we have to know how many documents in the collection are on particular topics or related to particular topics in a database.

Some of the largescale studies run precision and recall measures by setting out document sets in which they know every topic that’s included in that document set. And then they run queries that have been developed for those document sets against the collection using different kinds of search tools and different algorithms that they’ve developed in order to test precision and recall.

The TREC studies are still conducted today. Researchers compete developing and using new algorithms to test the precision and recall of their system. They also share information with each other about how to improve retrieval metrics, such as precision and recall in the IR systems. There are a lot of different TREC studies you can read online. And TREC deals not just with English language; it does also do some cross language retrieval as well, which is called CLIR, cross language information retrieval. There is an entire community for just CLIR and competitions related to CLIR as well. TREC itself includes user studies and competitions for not just text based documents, but also for images and multimedia and blogs and social networks and all kinds of different information retrieval systems. It is really interesting to see the problems and the issues that they trying to resolve by developing better retrieval mechanisms.

Effects on Precision/Recall

• Higher specificity means higher P or R? and lower P or R?

• Higher exhaustivity means higher P or R? and lower P or R?

• How do users information needs/expectations fit in?

A further consideration a searcher will make is related to specificity of the terms they choose. For example, if the system you are searching in has very specific terminology in the documents and in the indexing, how will using a higher level of specificity affect your search? Likely you would have more precise searches, right, but lower recall. What about exhaustivity? If the indexer represents all of the subjects in the document using many terms, this may assure a higher degree of precision, right? But what about recall? Would recall be higher if the searcher uses more terms to represent many subjects within the query or will recall suffer as a result of higher precision. Remember, recall and precision have an inverse relationship in the system; if one is high the other is low.

As a searcher you will need to determine which is important to your search results, finding everything that might mention your topic OR finding fewer documents that are more precisely on topic. It is also good practice to use the same level of specificity as the author(s) in the system if possible.

Putting it all together

• Subject representation must take into account:
  • Users and their information seeking/needs
  • Understanding of the changing nature of language
    • DIACHRONIC, SYNCHRONIC ATTRIBUTES
    • CULTURAL/GENDER ISSUES
    • NEW WORDS IN CURRENT USE
  • Author’s intentions
    • Subject(s) covered
    • Biases? Perspectives? Place within larger body of knowledge
  • Standards for creation of indexes/abstracts
  • Tools for term selection
  • Indexer’s experience/domain/system knowledge

Putting it all together, it is easy to see that subject representation, well document representation as a whole, is a very complex process. Many different factors such as those listed on the slide must be taken into account. Of primary importance to searching are an understanding of the user’s information need, being able to express this need (or helping them express this need) in terms that might be used by the author, the indexer, the controlled vocabulary of the system, and knowing how best to implement the terms in the system.

The domain of the query is also important as it will ensure that the searcher chooses the appropriate resource(s) in which to conduct the search but perhaps also the level of specificity of the terms used by the author/indexer. Understanding how the document may be represented within the system structure helps the searcher choose fields that might include the terms they feel the author or indexer used to represent the content of the documents they need to find. An indexer/searcher’s knowledge of the domain, their search experience, and understanding of the system structure all come into play when retrieving documents in the system.

The user’s expectations for retrieval are also important and can affect the choice of terms and level of specificity and exhaustivity of terms when conducting a search. In a few weeks we will look at controlled vocabulary searching which takes advantage of the domain’s terms and the controlled vocabulary feature of the system as one method to increase the precision and/or recall of a search.