Creating an Inverted Index for Information Retrieval | Desklib

This article explains how to create an inverted index for information retrieval using Desklib's study material. It covers removing stop words, using Porter Stemming algorithm, merging documents, sorting words alphabetically, and testing with Boolean and vector queries. The article also includes a comparison of search engines and evaluation of their performance.

COVER PAGE (ENTER YOUR DETAILS)

Contents
COVER PAGE (ENTER YOUR DETAILS)..................................................................................................1
Question 1...................................................................................................................................................3
Creating an inverted index......................................................................................................................3
A Remove stop words..........................................................................................................................3
Porter Stemming algorithm.................................................................................................................3
B The three documents merged..............................................................................................................4
Sorting words in alphabetical order for all documents........................................................................5
Word frequency for every document..................................................................................................6
C Dictionary and related posting file....................................................................................................7
D Testing..............................................................................................................................................8
Boolean and vector queries.....................................................................................................................8
Question 2 IR evaluation.............................................................................................................................9
A Search engines.....................................................................................................................................9
B Targets..................................................................................................................................................9
Search queries.........................................................................................................................................9
Google search Engine..........................................................................................................................9
Bing Search Engine............................................................................................................................10
C Average for Google and Bing..........................................................................................................11
Bibliography...............................................................................................................................................11

Question 1
Creating an inverted index
 Document 1
Information retrieval is the activity of obtaining information resources relevant to an
information need from a collection of information resources. Searches can be based on full-text
or other content-based indexing.
 Document 2
Information retrieval is finding material of an unstructured nature that satisfies an information
need from within large collections
 Document 3
Information systems is the study of complementary networks of hardware and software that
people and organizations use to collect, filter, process, create, and distribute data.
A Remove stop words
Results
 Document 1
Information retrieval activity obtaining information resources relevant information collection
information resources Searches based full-text content-based indexing
 Document 2
Information retrieval finding material unstructured nature satisfies information within large
collections
 Document 3
Information systems study complementary networks hardware software people organizations
collect filter process create distribute data
Porter Stemming algorithm
Results
 Document 1
Information retrieve active obtain inform resource relevant information collect information
resource Search base full text content base index
 Document 2
Information retrieve find material unstructure nature satisfy information within large collect
 Document 3
Information system study complement network hardware software people organ collect filter
process create distribute data

B The three documents merged
Term Document
Information 1
Retrieve 1
Active 1
Obtain 1
Inform 1
Resource 1
Relevant 1
Information 1
Resource 1
Search 1
Base 1
Full 1
Text 1
Content 1
Base 1
index 1
Information 2
Retrieve 2
Find 2
Material 2
Unstructure 2
Nature 2
Satisfy 2
Information 2
Within 2
Large 2
collect 2
Information 3
System 3
Study 3
Complement 3
Network 3
Hardware 3
Software 3
People 3
Organ 3
collect 3
Filter 3
Process 3
create 3
Distribute 3
data 3

Sorting words in alphabetical order for all documents
Term Doc ID
Active 1
Base 1
Base 1
collect 2
Complemen
t
3
Content 1
Find 2
Full 1
Hardware 3
index 1
Information 1
Information 2
Information 1
Information 1
Information 2
Information 3
Large 2
Material 2
Nature 2
Network 3
Obtain 1
People 3
Relevant 1
Resource 1
Resource 1
Retrieve 1
Retrieve 2
Satisfy 2
Search 1
Software 3
Study 3
System 3
Text 1
Unstructure 2

Word frequency for every document
Term Doc ID Frequency
Active 1 1
Base 1 2
collect 2 1
Complement 3 1
Content 1 1
Find 2 1
Full 1 1
Hardware 3 1
index 1 1
Information 1 3
Information 2 2
Information 3 1
Large 2 1
Material 2 1
Nature 2 1
Network 3 1
Obtain 1 1
People 3 1
Relevant 1 1
Resource 1 2
Retrieve 1 1
Retrieve 2 1
Satisfy 2 1
Search 1 1
Software 3 1
Study 3 1
System 3 1
Text 1 1
Unstructure 2 1

C Dictionary and related posting file

D Testing
Testing the inverted index was done using google search engine and the results returned by the search
engine included the three documents. The keywords used for the search are information, system and
index.
Boolean and vector queries
e. Boolean queries
1) information AND system AND index= Doc1, Doc2 , DOC3
2) System AND index= Doc1, Doc3
3) Information AND Index= Doc1, Doc2 , DOC3
f. Vector queries
Getting the cosine similarity based on;
Query Q= (Information, system, index)
Document 1
D1=<information, information, information, system, index> = <3, 1, 0>
Q=< information, system, index> = <1, 1, 1>
3 x 1+1 x 1+ 0 x 1
√ 32+12 +02 √ 12+12+12 = 4
√ 7 √ 3 = 1.15
Document 2
D2=<information, information> = <2, 0, 0>
Q=<information, information, system, index> = <1, 1, 1>
2 x 1+0 x 1+0 x 1
√22 +02+ 02 √12 +12+12 = 2
√4 √3 = 0.76
Document 3
D=<information, system>= <1, 1, 0>
Q=<information, information, system, index> <1, 1, 1>
σ ( D3 , Q)= 1 x 1+ 1 x 1+ 0 x 1
√12 +12 +02 √12+12 +12 = 2
√2 √3 = 1.07

Based on the results of the cosine similarity the order in which the documents appear is a
search engine is;
1) Document 1
2) Document 2
3) Document 3
Question 2 IR evaluation
A Search engines
 Google
 Bing
B Targets
Target 5: obtain the price of new Xbox one.
Search queries
 Query 1= Xbox one price
 Query 2= Price xbox one
Google search Engine
Figure 1: Google Search Engine

Bing Search Engine
Figure 2: Yahoo search engine

C Average for Google and Bing
Figure 3: Comparison by average
Based on the chart shown on figure 3 above Google is more powerful than Bing based on precision and
recall. Google has a higher recall value meaning that the number of results is greater than Bing. Google
has a precision meaning that the number of results related to the search query is higher compared to
Bing.
Bibliography
Brasetvik, A. (2013). Elasticsearch from the Bottom Up, Part 1. [online] Elastic. Available at:
https://www.elastic.co/blog/found-elasticsearch-from-the-bottom-up [Accessed 27 May 2018].