Skip to main content

A Demo For Understand Search Relevance In ElasticSearch Using Kibana

In this article, we will understand Relevance Search in ElasticSearch using Kibana.

In Elasticsearch relevance search means the output of the search can be done in 2 different ways like 'Precision', 'Recall'.

The 'Precision' result means the most accurate results from the ElasticSearch. This means it fetches exact matches of the search value. So in this search output will be more accurate, but the total output results count might be very very less.
  • Here all green color docs are more accurate documents for the search.
  • Here all red color docs are either less accurate documents or not matching documents for the search.
  • Here we can observe search results contain only accurate documents which means it is a 'Precision' result.
The 'Recall' results mean more in the count of the output results because it fetches partially matched data along with accurate data. Since it is fetching high count results, sometimes it skips the few accurate documents as well.
  • Here we can observe that the output search result contains both accurate and partially(not so accurate) documents.
  • Here we can observe a few accurate results left out from the search results.
  • So this search result is called the 'Recall' result.

Run ElasticSearch And Kibana Docker Containers:

Create a network first that can help to connect our services like 'Elasticsearch' & 'Kibana' under it.
Command to create a network:
docker network create your_network_name_any_name

Let's pull and create the Elasticsearch docker container.
Command To Create Elasticsearch Docker Container:

docker run -d --name your_container_name_any_name --net network_name_just_created -p 9200:9200 -e "discovery.type=single-node" elasticsearch:7.16.3
  • [ -d ] run the docker command in detach mode, which means runs as a background service.
  • [ --name your_container_name_any_name] define the name to the docker container.
  • [ --net network_name_just_created] specify the network name under which our service should run.
  • [ -p 9200:9200] right-hand side port number(fixed port number) is the default port number for the 'ElasticSearch', the left-hand side port number is exposing port number we can define any of our custom port numbers.
  • [-e "discovery.type=singlenode"] setting the environment variable to run our Elastic search on a single node. it can be changed for production applications.
  • [ elasticsearch:7.16.3] name of service and its version.
Let's pull and create the Kibana docker container.
Command To Create Kibana Docker Container:

docker run -d --name your_container_name_any_name --net network_name_just_created -p 5601:5601 kibana:7.16.3
  • [ -d ] run the docker command in detach mode, which means runs as a background service.
  • [ --name your_container_name_any_name] define the name to the docker container.
  • [ --net network_name_just_created] specify the network name under which our service should run.
  • [ -p 5601:5601] right-hand side port number(fixed port number) is the default port number for the 'Kibana', the left-hand side port number is exposing port number we can define any of our custom port numbers.
  • [ kibana:7.16.3] name of service and its version.

Upload Some Test Data Into Our ElasticSearch:

(1)Let's upload some test data into our elastic search. So from "https://perso.telecom-paristech.fr/eagan/class/igr204/datasets", this website downloads the 'Films' CSV file(any sample CSV file of your choice).

(2)Now open the 'Kibana' tool 'http://localhost:5601/app/home' and then click on the 'Upload A File' option.
(3) Now select the 'Film' CSV file and then click on the 'Import' Button.
(4) Create an index name for our test data.

Query To Retrieve Information About Document In An Index:

The syntax for Query to retrieve information about the document in an index.
GET Name_of_your_Index/_search

The sample query to retrieve information about the document in an index
GET film_info/_search
  • 'GET' - HTTP verb
  • 'film_info' - the name of the index.
  • '_search' - keyword of Elasticsearch.

Identify A Most Significant Text:

This section is totally an optional section, where I'm going to explain a query to get significant keywords from our Elastic store so that using those significant keywords we can frame a keyword to search against Elastic store so that we can understand the Relevance search results.

A sample query to get the Significant keywords.
GET film_info/_search
{
 "query": {
   "match": {
   "Subject": "Drama"
   }
  },
 "aggs": {
  "popular_kewords": {
    "significant_text": {
       "field": "Title"
     }
   }
  }
 }
  • GET - HTTP verbs
  • 'film_info' - the name of the index.
  • 'query' - Elasticsearch keyword.
  • 'match' - Elasticsearch keyword, inside match object, define 'Property Name' and its 'Value' for search against the documents.
  • 'aggs' - Elasticsearch keyword 'aggregation', the 'aggregation' summarizes our data as metrics, statics, or other analytics.
  • 'popular_keywords' - name of the 'aggregation'.
  • 'significant_text' - Elasticsearch keyword, inside of its define 'field' whose value must be property name of the document.

Now frame a sentence by using the above words to search against ElasticSearch to understand the Relevance search.

Recall Search Results:

Recall search result means it will bring the result if at least one word matched that means it will bring accurate and partial accurate data as well.

The sample query returns 'Recall Search Results'.
GET film_info/_search
{
   "query":
   {
      "match": {
          "Title":
             {
              "query": "until the last spring"
             }
      }
   }
}
  • GET - HTTP verb
  • film_info - the name of the index
  • _search - Elasticsearch keyword
  • query - Elasticsearch keyword
  • match - Elasticsearch keyword used to match data inside of it. Inside 'match' create an object with 'Property Name' of the document on which we want to search, inside of it add 'query' to it we have assigned our search keyword(eg: 'until the last spring').

Precision Search Result:

Precision search result, try to match every word in the search text against ElasticSearch data, here the position of the words don't matter. So, in this case, we will receive very less count results since every word must match.

The sample query returns 'Precision Search Result'.
GET film_info/_search
{
   "query":
     {
      "match":
      {
        "Title":
         {
          "query": "until the last spring",
         "operator": "and"
         }
       }
      }
}
  • 'operator' - Elasticsearch keyword and value is 'and' which means that need to match every word in the 'query' against the documents of the Elasticsearch.

Combine Precision & Recall Results:

To get more reasonable results we can combine 'Precision' & 'Recall' search results.

The sample query
GET film_info/_search
{
    "query":
   {
   "match":
  {
   "Title":
   {
   "query": "until the last spring",
   "minimum_should_match": 2
   }
   }
   }
}
  • 'minimum_should_match' - ElasticSearch keyword specifies minimum words of search keyword need to be matched against the documents.

Support Me!
Buy Me A Coffee PayPal Me

Video Session:

Wrapping Up:

Hopefully, I think this article delivered some useful information on Relevance Search in ElasticSearch. using I love to have your feedback, suggestions, and better techniques in the comment section below.

Follow Me:

Comments

Popular posts from this blog

Angular 14 Reactive Forms Example

In this article, we will explore the Angular(14) reactive forms with an example. Reactive Forms: Angular reactive forms support model-driven techniques to handle the form's input values. The reactive forms state is immutable, any form filed change creates a new state for the form. Reactive forms are built around observable streams, where form inputs and values are provided as streams of input values, which can be accessed synchronously. Some key notations that involve in reactive forms are like: FormControl - each input element in the form is 'FormControl'. The 'FormControl' tracks the value and validation status of form fields. FormGroup - Track the value and validate the state of the group of 'FormControl'. FormBuilder - Angular service which can be used to create the 'FormGroup' or FormControl instance quickly. Form Array - That can hold infinite form control, this helps to create dynamic forms. Create An Angular(14) Application: Let'

.NET 7 Web API CRUD Using Entity Framework Core

In this article, we are going to implement a sample .NET 7 Web API CRUD using the Entity Framework Core. Web API: Web API is a framework for building HTTP services that can be accessed from any client like browser, mobile devices, and desktop apps. In simple terminology API(Application Programming Interface) means an interface module that contains programming functions that can be requested via HTTP calls either to fetch or update data for their respective clients. Some of the Key Characteristics of API: Supports HTTP verbs like 'GET', 'POST', 'PUT', 'DELETE', etc. Supports default responses like 'XML' and 'JSON'. Also can define custom responses. Supports self-hosting or individual hosting, so that all different kinds of apps can consume it. Authentication and Authorization are easy to implement. The ideal platform to build the REST full services. Install The SQL Server And SQL Management Studio: Let's install the SQL server on our l

ReactJS(v18) JWT Authentication Using HTTP Only Cookie

In this article, we will implement the ReactJS application authentication using the HTTP-only cookie. HTTP Only Cookie: In a SPA(Single Page Application) Authentication JWT token either can be stored in browser 'LocalStorage' or in 'Cookie'. Storing the JWT token inside of the cookie then the cookie should be HTTP Only. The HTTP-ONly cookie nature is that it will be only accessible by the server application. Client apps like javascript-based apps can't access the HTTP-Only cookie. So if we use the authentication with HTTP-only JWT cookie then we no need to implement the custom logic like adding authorization header or storing token data, etc at our client application. Because once the user authenticated cookie will be automatically sent to the server by the browser on every API call. Authentication API: To authenticate our client application with JWT HTTP-only cookie, I developed a NetJS(which is a node) Mock API. Check the GitHub link and read the document on G

.NET6 Web API CRUD Operation With Entity Framework Core

In this article, we are going to do a small demo on AspNetCore 6 Web API CRUD operations. What Is Web API: Web API is a framework for building HTTP services that can be accessed from any client like browser, mobile devices, desktop apps. In simple terminology API(Application Programming Interface) means an interface module that contains a programming function that can be requested via HTTP calls to save or fetch the data for their respective clients. Some of the key characteristics of API: Supports HTTP verbs like 'GET', 'POST', 'PUT', 'DELETE', etc. Supports default responses like 'XML' and 'JSON'. Also can define custom responses. Supports self-hosting or individual hosting, so that all different kinds of apps can consume it. Authentication and Authorization are easy to implement. The ideal platform to build REST full services. Create A .NET6 Web API Application: Let's create a .Net6 Web API sample application to accomplish our

Angular 14 State Management CRUD Example With NgRx(14)

In this article, we are going to implement the Angular(14) state management CRUD example with NgRx(14) NgRx Store For State Management: In an angular application to share consistent data between multiple components, we use NgRx state management. Using NgRx state helps to avoid unwanted API calls, easy to maintain consistent data, etc. The main building blocks for the NgRx store are: Actions - NgRx actions represents event to trigger the reducers to save the data into the stores. Reducer - Reducer's pure function, which is used to create a new state on data change. Store - The store is the model or entity that holds the data. Selector - Selector to fetch the slices of data from the store to angular components. Effects - Effects deals with external network calls like API. The effect gets executed based the action performed Ngrx State Management flow: The angular component needs data for binding.  So angular component calls an action that is responsible for invoking the API call.  Aft

Angular 14 Crud Example

In this article, we will implement CRUD operation in the Angular 14 application. Angular: Angular is a framework that can be used to build a single-page application. Angular applications are built with components that make our code simple and clean. Angular components compose of 3 files like TypeScript File(*.ts), Html File(*.html), CSS File(*.cs) Components typescript file and HTML file support 2-way binding which means data flow is bi-directional Component typescript file listens for all HTML events from the HTML file. Create Angular(14) Application: Let's create an Angular(14) application to begin our sample. Make sure to install the Angular CLI tool into our local machine because it provides easy CLI commands to play with the angular application. Command To Install Angular CLI npm install -g @angular/cli Run the below command to create the angular application. Command To Create Angular Application ng new name_of_your_app Note: While creating the app, you will see a noti

Unit Testing Asp.NetCore Web API Using xUnit[.NET6]

In this article, we are going to write test cases to an Asp.NetCore Web API(.NET6) application using the xUnit. xUnit For .NET: The xUnit for .Net is a free, open-source, community-focused unit testing tool for .NET applications. By default .Net also provides a xUnit project template to implement test cases. Unit test cases build upon the 'AAA' formula that means 'Arrange', 'Act' and 'Assert' Arrange - Declaring variables, objects, instantiating mocks, etc. Act - Calling or invoking the method that needs to be tested. Assert - The assert ensures that code behaves as expected means yielding expected output. Create An API And Unit Test Projects: Let's create a .Net6 Web API and xUnit sample applications to accomplish our demo. We can use either Visual Studio 2022 or Visual Studio Code(using .NET CLI commands) to create any.Net6 application. For this demo, I'm using the 'Visual Studio Code'(using the .NET CLI command) editor. Create a fo

Part-1 Angular JWT Authentication Using HTTP Only Cookie[Angular V13]

In this article, we are going to implement a sample angular application authentication using HTTP only cookie that contains a JWT token. HTTP Only JWT Cookie: In a SPA(Single Page Application) Authentication JWT token either can be stored in browser 'LocalStorage' or in 'Cookie'. Storing JWT token inside of the cookie then the cookie should be HTTP Only. The HTTP-Only cookie nature is that it will be only accessible by the server application. Client apps like javascript-based apps can't access the HTTP-Only cookie. So if we use authentication with HTTP only JWT cookie then we no need to implement custom logic like adding authorization header or storing token data, etc at our client application. Because once the user authenticated cookie will be automatically sent to the server by the browser on every API call. Authentication API: To implement JWT cookie authentication we need to set up an API. For that, I had created a mock authentication API(Using the NestJS Se

ReactJS(v18) Authentication With JWT AccessToken And Refresh Token

In this article, we are going to do ReactJS(v18) application authentication using the JWT Access Token and Refresh Token. JSON Web Token(JWT): JSON Web Token is a digitally signed and secured token for user validation. The JWT is constructed with 3 important parts: Header Payload Signature Create ReactJS Application: Let's create a ReactJS application to accomplish our demo. npx create-react-app name-of-your-app Configure React Bootstrap Library: Let's install the React Bootstrap library npm install react-bootstrap bootstrap Now add the bootstrap CSS reference in 'index.js'. src/index.js: import 'bootstrap/dist/css/bootstrap.min.css' Create A React Component 'Layout': Let's add a React component like 'Layout' in 'components/shared' folders(new folders). src/components/shared/Layout.js: import Navbar from "react-bootstrap/Navbar"; import { Container } from "react-bootstrap"; import Nav from "react-boot

A Small Guide On NestJS Queues

NestJS Application Queues helps to deal with application scaling and performance challenges. When To Use Queues?: API request that mostly involves in time taking operations like CPU bound operation, doing them synchronously which will result in thread blocking. So to avoid these issues, it is an appropriate way to make the CPU-bound operation separate background job.  In nestjs one of the best solutions for these kinds of tasks is to implement the Queues. For queueing mechanism in the nestjs application most recommended library is '@nestjs/bull'(Bull is nodejs queue library). The 'Bull' depends on Redis cache for data storage like a job. So in this queueing technique, we will create services like 'Producer' and 'Consumer'. The 'Producer' is used to push our jobs into the Redis stores. The consumer will read those jobs(eg: CPU Bound Operations) and process them. So by using this queues technique user requests processed very fastly because actually