🔍 Semantic Search

Semantic searching, which involves understanding the intent and contextual meaning behind search queries, is yet another popular use-case of RAG. It has several popular use cases across various domains:

• Information Retrieval: Enhances search accuracy in databases and websites
• E-commerce: Improves product discovery in online shopping
• Customer Support: Powers smarter chatbots for effective responses
• Content Discovery: Aids in finding relevant media content
• Knowledge Management: Streamlines document and data retrieval in enterprises
• Healthcare: Facilitates medical research and literature search
• Legal Research: Assists in legal document and case law search
• Academic Research: Aids in academic paper discovery
• Language Processing: Enables multilingual search capabilities

EmbedJs offers a simple yet customizable search() API that you can use for semantic search. See the example in the next section to know more.

Example: Semantic Search over Next.JS Website + Forum

Step 1: Set Up Your RAG Pipeline

First, let’s create your RAG pipeline. Open your NodeJs application and add the following code:

Create pipeline

import { RAGApplicationBuilder } from '@llm-tools/embedjs';
import { OpenAiEmbeddings } from '@llm-tools/embedjs-openai';
import { HNSWDb } from '@llm-tools/embedjs-hnswlib';

const app = await new RAGApplicationBuilder()
.setModel(SIMPLE_MODELS.OPENAI_GPT4_O)
.setEmbeddingModel(new OpenAiEmbeddings())
.setVectorDatabase(new HNSWDb())
.build();

This initializes your application.

Step 2: Populate Your Pipeline with Data

Now, let’s add data to your pipeline. We’ll include the Next.JS website and its documentation:

Ingest data sources

import { SitemapLoader } from '@llm-tools/embedjs-loader-sitemap';

//Add Next.JS Website and docs
app.addLoader(new SitemapLoader({ url: "https://nextjs.org/sitemap.xml" }))

//Add Next.JS Forum data
app.addLoader(new SitemapLoader({ url: "https://nextjs-forum.com/sitemap.xml" }))

This step incorporates over 15K pages from the Next.JS website and forum into your pipeline. For more data source options, check the EmbedJs data sources overview.

Step 3: Local Testing of Your Pipeline

Test the pipeline on your local machine:

Search App

app.search("Summarize the features of Next.js 14?")
/*[
  {
    'score': 0.99,
    'pageContent': 'Next.js 14 | Next.jsBack to BlogThursday, October 26th 2023Next.js 14Posted byLee Robinson@leeerobTim Neutkens@timneutkensAs we announced at Next.js Conf, Next.js 14 is our most focused release with: Turbopack: 5,000 tests passing for App & Pages Router 53% faster local server startup 94% faster code updates with Fast Refresh Server Actions (Stable): Progressively enhanced mutations Integrated with caching & revalidating Simple function calls, or works natively with forms Partial Prerendering',
    'metadata': {
      'id': '6c8d1a7b-ea34-4927-8823-daa29dcfc5af',
      'uniqueLoaderId': '6c8d1a7b-ea34-4927-8823-xba29dcfc5ac',
      'source': 'https://nextjs.org/blog/next-14'
    }
  },
  {
    'score': 0.98,
    'pageContent': 'Next.js 13.3 | Next.jsBack to BlogThursday, April 6th 2023Next.js 13.3Posted byDelba de Oliveira@delba_oliveiraTim [e-mail] 13.3 adds popular community-requested features, including: File-Based Metadata API: Dynamically generate sitemaps, robots, favicons, and more. Dynamic Open Graph Images: Generate OG images using JSX, HTML, and CSS. Static Export for App Router: Static / Single-Page Application (SPA) support for Server Components. Parallel Routes and Interception: Advanced',
    'metadata': {
      'id': '6c8d1a7b-ea34-4927-8823-daa29dcfc5a1',
      'uniqueLoaderId': '6c8d1a7b-ea34-4927-8823-xba29dcfc5ae',
      'source': 'https://nextjs.org/blog/next-13-3'
    }
  },
  {
    'score': 0.98,
    'pageContent': 'Upgrading: Version 14 | Next.js MenuUsing App RouterFeatures available in /appApp Router.UpgradingVersion 14Version 14 Upgrading from 13 to 14 To update to Next.js version 14, run the following command using your preferred package manager: Terminalnpm i next@latest react@latest react-dom@latest eslint-config-next@latest Terminalyarn add next@latest react@latest react-dom@latest eslint-config-next@latest Terminalpnpm up next react react-dom eslint-config-next -latest Terminalbun add next@latest',
    'metadata': {
      'id': '6c8d1a7b-ea34-4927-8823-daa29dcfc5a2',
      'uniqueLoaderId': '6c8d1a7b-ea34-4927-8823-xba29dcfc5ad',
      'source': 'https://nextjs.org/docs/app/building-your-application/upgrading/version-14'
    }
  }
]*/

The source key contains the url of the document that yielded that document chunk.

If you are interested in configuring the search further, refer to our API documentation.

---

This guide will help you swiftly set up a semantic search pipeline with EmbedJs, making it easier to access and analyze specific information from large data sources.

Need help?

In case you run into issues, feel free to contact us via any of the following methods:

Slack

Join our slack community