04-data-processing/notebooks/02-Redis-add-HTML-to-existing-index.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating an index and populating it with documents using Redis\n",
    "\n",
    "Simple example on how to ingest HTML documents / web pages content into a Redis VectorStore.\n",
    "\n",
    "Requirements:\n",
    "- A Redis cluster\n",
    "- A Redis database with at least 2GB of memory (to match with the initial index cap)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Base parameters, the Redis info"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "redis_url = \"redis://server:port\"\n",
    "index_name = \"dellwebdocs\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Imports"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "from langchain.document_loaders import WebBaseLoader\n",
    "from langchain.text_splitter import RecursiveCharacterTextSplitter\n",
    "from langchain.embeddings.huggingface import HuggingFaceEmbeddings\n",
    "from langchain.vectorstores.redis import Redis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Ingesting new documents"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "loader = WebBaseLoader([\"https://infohub.delltechnologies.com/l/design-guide-sql-server-2022-database-solution-with-object-storage-on-dell-hardware-stack/business-challenge-193/\",\n",
    "                        \"https://infohub.delltechnologies.com/l/design-guide-sql-server-2022-database-solution-with-object-storage-on-dell-hardware-stack/solution-introduction-81/\",\n",
    "                        \"https://infohub.delltechnologies.com/l/design-guide-sql-server-2022-database-solution-with-object-storage-on-dell-hardware-stack/design-guide-introduction-28/\",\n",
    "                        \"https://infohub.delltechnologies.com/l/design-guide-sql-server-2022-database-solution-with-object-storage-on-dell-hardware-stack/terminology-279/\"\n",
    "                        \"https://infohub.delltechnologies.com/l/design-guide-sql-server-2022-database-solution-with-object-storage-on-dell-hardware-stack/physical-architecture-69/\",\n",
    "                        \"https://infohub.delltechnologies.com/l/design-guide-sql-server-2022-database-solution-with-object-storage-on-dell-hardware-stack/logical-architecture-106/\",\n",
    "                        \"https://infohub.delltechnologies.com/l/design-guide-sql-server-2022-database-solution-with-object-storage-on-dell-hardware-stack/virtualization-design-10/\",\n",
    "                        \"https://infohub.delltechnologies.com/l/design-guide-sql-server-2022-database-solution-with-object-storage-on-dell-hardware-stack/container-design/\",\n",
    "                        \"https://infohub.delltechnologies.com/l/design-guide-sql-server-2022-database-solution-with-object-storage-on-dell-hardware-stack/software-919/\",\n",
    "                        \"https://infohub.delltechnologies.com/l/design-guide-sql-server-2022-database-solution-with-object-storage-on-dell-hardware-stack/overview-4230/\",\n",
    "                        \"https://infohub.delltechnologies.com/l/design-guide-sql-server-2022-database-solution-with-object-storage-on-dell-hardware-stack/initial-setup/\",\n",
    "                        \"https://infohub.delltechnologies.com/l/design-guide-sql-server-2022-database-solution-with-object-storage-on-dell-hardware-stack/haproxy-loadbalancer-for-dell-ecs-storage/\",\n",
    "                        \"https://infohub.delltechnologies.com/l/design-guide-sql-server-2022-database-solution-with-object-storage-on-dell-hardware-stack/certificate-creation-and-installation/\",\n",
    "                        \"https://infohub.delltechnologies.com/l/design-guide-sql-server-2022-database-solution-with-object-storage-on-dell-hardware-stack/setup-access-to-dell-ecs-storage-cluster/\",\n",
    "                        \"https://infohub.delltechnologies.com/l/design-guide-sql-server-2022-database-solution-with-object-storage-on-dell-hardware-stack/configuring-vvols-on-dell-powerstore-storage/\",\n",
    "                        \"https://infohub.delltechnologies.com/l/design-guide-sql-server-2022-database-solution-with-object-storage-on-dell-hardware-stack/virtual-environment-setup/\",\n",
    "                        \"https://infohub.delltechnologies.com/l/design-guide-sql-server-2022-database-solution-with-object-storage-on-dell-hardware-stack/containerized-environment-set-up/\",\n",
    "                        \"https://infohub.delltechnologies.com/l/design-guide-sql-server-2022-database-solution-with-object-storage-on-dell-hardware-stack/introduction-3357/\",\n",
    "                        \"https://infohub.delltechnologies.com/l/design-guide-sql-server-2022-database-solution-with-object-storage-on-dell-hardware-stack/backup-and-restore-use-case-2/\",\n",
    "                        \"https://infohub.delltechnologies.com/l/design-guide-sql-server-2022-database-solution-with-object-storage-on-dell-hardware-stack/data-virtualization-use-case-2/\",\n",
    "                        \"https://infohub.delltechnologies.com/l/design-guide-sql-server-2022-database-solution-with-object-storage-on-dell-hardware-stack/data-tiering-use-case/\",\n",
    "                        \"https://infohub.delltechnologies.com/l/design-guide-sql-server-2022-database-solution-with-object-storage-on-dell-hardware-stack/new-t-sql-functions-use-cases/\",\n",
    "                        \"https://infohub.delltechnologies.com/l/design-guide-sql-server-2022-database-solution-with-object-storage-on-dell-hardware-stack/deployment-automation-use-case/\",\n",
    "                        \"https://infohub.delltechnologies.com/l/design-guide-sql-server-2022-database-solution-with-object-storage-on-dell-hardware-stack/summary-1165/\",\n",
    "                        \"https://infohub.delltechnologies.com/l/design-guide-sql-server-2022-database-solution-with-object-storage-on-dell-hardware-stack/conclusion-616/\",\n",
    "                        \"https://infohub.delltechnologies.com/l/design-guide-sql-server-2022-database-solution-with-object-storage-on-dell-hardware-stack/request-for-feedback/\",\n",
    "                        \"https://infohub.delltechnologies.com/l/design-guide-sql-server-2022-database-solution-with-object-storage-on-dell-hardware-stack/automation-scripts/\"\n",
    "                       ])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = loader.load()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "text_splitter = RecursiveCharacterTextSplitter(chunk_size=1024,\n",
    "                                               chunk_overlap=40)\n",
    "all_splits = text_splitter.split_documents(data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "embeddings = HuggingFaceEmbeddings()\n",
    "rds = Redis.from_existing_index(embeddings,\n",
    "                                redis_url=redis_url,\n",
    "                                index_name=index_name,\n",
    "                                schema=\"dellwebdocs_redis_schema.yaml\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rds.add_documents(all_splits)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Write the schema to a yaml file to be able to open the index later on"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rds.write_schema(\"redis_schema.yaml\")"
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}