Information retrieval 1
Is finding material of an unstructured nature that satisfies an information need from within
large collections (usually stored on computers)
First idea for an automated system was 1945 by Vannevar Bush in As We May Think
In 1960s the field of Information Retreival emerged
Evolution of IR
1960-70s: era of Boolean Retrieval
1975s: first Vector Space Model
1980s: large document database systems run by companies became available (LexisNexis,
MedLine)
1990s: FTP search and the dawn of Web search (lycos, Yahoo)
IR in 2000s
Google
- Link analysis & ranking
- Multimedia IR (image and video analysis)
- Cross-language IR
- Semantic Web Technologies (DBPedia)
IR since 2010s
Categorization and clustering, and recommendation:
- iTunes “Top Songs”
- Amazon “people who bought this also bought …”
- IBMs Watson system (business related: predict future outcomes)
- Recommendations in Netflix, spotify, youtube
IR versus DB
IR DB (databases)
Unstructured data structured
Set of keywords (loose semantics) well defined query (SQL)
Incomplete query specification, partial matching complete query specification, exact matc
Relevant items for result, errors tolerable single error results in failure
Probabilistic models deterministic models
,What is needed to build a search engine
What makes a search engine good?
Speed + User happiness
Which of following actions if fastest and slowest?
1 – main memory reference (read random byte from memory) 1
2 – Hard disk seek (read random byte from hard disk) 5
3 – SSD random read (read random byte from solid-state drive) 3
4 – Zip 1KB of data (compress 1000 bytes in memory) 2
5 – Round trip within same datacenter (send one byte to another computer in same fast
datacenter network and back) 4
6 – Send one byte from Netherlands to California and back 6
,2:
In memory, can use linked lists or variable-length arrays
Token = an instance of a sequence of
characters in some particular document that are grouped together as a useful semantic unit
for processing
Type = the class of all tokens consisting of exactly the same character sequence
Term = a (perhaps normalized) type that is included in the IR systems dictionary
Bottleneck: sorting
Sorting lot of records on disk is much too slow – in particular for hard disks but also for SSDs
but data is too large for memory, so we need an external sorting algo
, Summary
Boolean retrieval:
A simple and well-understood retrieval model
Inverted indexes:
Inverting an index demands a lot of resources
Sorting the index is the critical step
3:
Biword indexes: Index every consecutive pair of terms in the text as a phrase
So “Friends, Romans, Countrymen would generate biwrods: friends romans & romans
countrymen
So we can now process 2-word phrase queries in a straightforward manner
But with longer phrase queries could be false positives, we cannot verify that it contains
query
Problems:
False positives in answer set
In particular for phrases with frequent words like “beer of the month”
Index blow-up due to bigger dictionary
Infeasible for more than biwords
Biword indexes are therefore not the standard solution, but they can be part of a compound
strategy
Is finding material of an unstructured nature that satisfies an information need from within
large collections (usually stored on computers)
First idea for an automated system was 1945 by Vannevar Bush in As We May Think
In 1960s the field of Information Retreival emerged
Evolution of IR
1960-70s: era of Boolean Retrieval
1975s: first Vector Space Model
1980s: large document database systems run by companies became available (LexisNexis,
MedLine)
1990s: FTP search and the dawn of Web search (lycos, Yahoo)
IR in 2000s
- Link analysis & ranking
- Multimedia IR (image and video analysis)
- Cross-language IR
- Semantic Web Technologies (DBPedia)
IR since 2010s
Categorization and clustering, and recommendation:
- iTunes “Top Songs”
- Amazon “people who bought this also bought …”
- IBMs Watson system (business related: predict future outcomes)
- Recommendations in Netflix, spotify, youtube
IR versus DB
IR DB (databases)
Unstructured data structured
Set of keywords (loose semantics) well defined query (SQL)
Incomplete query specification, partial matching complete query specification, exact matc
Relevant items for result, errors tolerable single error results in failure
Probabilistic models deterministic models
,What is needed to build a search engine
What makes a search engine good?
Speed + User happiness
Which of following actions if fastest and slowest?
1 – main memory reference (read random byte from memory) 1
2 – Hard disk seek (read random byte from hard disk) 5
3 – SSD random read (read random byte from solid-state drive) 3
4 – Zip 1KB of data (compress 1000 bytes in memory) 2
5 – Round trip within same datacenter (send one byte to another computer in same fast
datacenter network and back) 4
6 – Send one byte from Netherlands to California and back 6
,2:
In memory, can use linked lists or variable-length arrays
Token = an instance of a sequence of
characters in some particular document that are grouped together as a useful semantic unit
for processing
Type = the class of all tokens consisting of exactly the same character sequence
Term = a (perhaps normalized) type that is included in the IR systems dictionary
Bottleneck: sorting
Sorting lot of records on disk is much too slow – in particular for hard disks but also for SSDs
but data is too large for memory, so we need an external sorting algo
, Summary
Boolean retrieval:
A simple and well-understood retrieval model
Inverted indexes:
Inverting an index demands a lot of resources
Sorting the index is the critical step
3:
Biword indexes: Index every consecutive pair of terms in the text as a phrase
So “Friends, Romans, Countrymen would generate biwrods: friends romans & romans
countrymen
So we can now process 2-word phrase queries in a straightforward manner
But with longer phrase queries could be false positives, we cannot verify that it contains
query
Problems:
False positives in answer set
In particular for phrases with frequent words like “beer of the month”
Index blow-up due to bigger dictionary
Infeasible for more than biwords
Biword indexes are therefore not the standard solution, but they can be part of a compound
strategy
