Data Hungry Techniques- More Efficient Deep Neural Networks for IR

Introduction

Information Retrieval is a way of satisfying the user needs of the query specific documents. Ranking the documents in information retrieval has changed from Theoretical models to the neural models.The area of deep learning is growing in terms of the number of new architectures and training regimes, we can try combinations of every model on every IR task. Neural IR refers to the application of shallow or deep neural networks to these retrieval tasks[1].Though machine learning has been applied in IR but such supervised algorithms tend to overfit the classification data.deep learning enables representation learning and matching in IR.

Advantages of Deep Neural Models:
1.To solve the key problem of Matching and Representation Deep Learning is being used.
2.Solve the problem separately for short and long text matching.
3.Getting more Impactful and insightful results from IR.

(2017 -B. Mitra and N. Craswell)

Deep Learning Architectures in IR:

Deep Rank: 

It solved the core problem of relevance ranking in IR. Instead of directly applying the neural networks,understanding the concept of relevance in IR is more important. DeepRank[3] proposed by Liang Pang et al. captures important characteristics of IR  such as  proximity heuristics, query term importance,exact/semantic matching signals It works in three  simple steps:

1.A detection strategy :To extract query centric contexts. 

2.A measure network :To determine the local relevance between query and each query-centric context, by using CNN or 2D-GRU.

i) A tensor is constructed as the input. 

ii) CNN or 2D-GRU is applied on the tensor to output a vector  which stands for the representation of local relevance

3.Aggregation network: To produce the global relevance score, via RNN and a term gating network.

Deep Rank not only well simulates the relevance generation process in human judgement, but also captures important IR characteristics, i.e. exact/semantic matching signals, proximity heuristics, query term importance, and diverse relevance requirement.The Illustration of Deep Rank is shown below[3].

Deep Rank Illustration

DSRIM: A Deep Neural Information Retrieval Model 

•  The input is modeled as vector is divided into two parts:                                                           i)represented as paragraph vector                                                                                                   ii) Semantic relation expressed as Knowledge resource which is built upon relation mapping method.[4]
• Then the latent representation is being learnt for query and vector representation
• To solve the ranking problem,Loss Function is applied.
• It  combines the distributional and the relational semantics through the DSRIMkr+p2v model, the MAP value slightly increases, with some significant improvement[4].

Siamese Network:

1.  Deep Semantic Similarity Model (DSSM) is used.It has great potential for Ad-Hoc retrieval.
It resembles the auto encoder architecture.The Siamese model can be utilised through the equation given below as proposed in [1].

Lsiamese( vq, vd1, vd2) = log(1 + e −γ(sim( vq, vd1)−sim( vq, vd2))

Auto Encoders:
It is based on Information Bottleneck method(Tishby et al. 2000).

Both auto-encoders and Siamese networks learn compressed representation that retains the minimal sufficient statistics of the input. In case of auto-encoders, the learnt representation is optimized for reducing reconstruction errors, whereas the representation learnt by Siamese networks are optimized to better discriminate similar pairs from dissimilar ones.Both are shown below[1].

Semantic Matching and Lexical Matching:

It can be used for long text.For measuring the relevance rare terms can be more informative.So lexical patterns help us to estimating by training the DNN using frequency histograms of query or by learning interesting patterns through binary interaction matrix(Pang et al. 2016).

Conclusion:

Deep Neural Networks are effective than the traditional IR techniques.Some problems of ranking,matching,handling short text queries,long text queries and query document mismatch problem has been  are handled with the combination of  modern neural models.Different combinatorics of the model could generate more efficient models.

References:- 

1. B. Mitra and N. Craswell, An Introduction to Neural Information Retrieval.

2.Hang Li , Deep Learning for NLP 

3.Liang Pang, Yanyan et al. ,DeepRank:A New Deep Architecture for Relevance Ranking in Information Retrieval 
4.Gia-Hung Nguyen et al. , DSRIM: A Deep Neural Information Retrieval Model Enhanced by a Knowledge Resource Driven Representation of Documents