Skip to main content

Vector Store Writer

Writes documents to a vector store in plain and vector data form.

What is the Vector Store Writer?​

A dark-themed user interface for a 'File Writer' utility. It has dropdown menus for 'Storage Provider' set to 'AWS S3', 'Credentials', and 'Region' set to 'US East (Virginia) - us-east-1'. There are input fields labeled 'File', 'File Path', and 'Bucket Name'.

The Vector Store Writer is an integral component in workflows that require storing and managing document embeddings in a vector store. This node facilitates the storage of documents, both in their plain text form and in their vector representations, to a designated vector store. This ensures that documents are properly stored and indexed, making them accessible for subsequent retrieval and analysis tasks.

How to use it?​

To use the Vector Store Writer in your workflow, follow these steps:

  1. Add the Vector Store Writer to Your Workflow:

    • Drag and drop the Vector Store Writer from the node library into your workflow canvas.

  2. Connect Input Anchors:

    • For a Vector Store Writer ensure you have a Vector Store, a Document Splitter and a Document to embed with the writer.
    • Vector Store: Connect the Vector Store to the Vector Store input anchor of the Vector Store Writer. This establishes the connection to the Vector Store where the documents will be written.
    • Document Splitter: Connect the Document Splitter to the Document Splitter input anchor of the Vector Store Writer.
    • Document: Connect a node that outputs a string (text) to the Document input anchor of the Vector Store Writer. This is the document that will be written to the Vector Store.
    • Metadata: Optionally provide some metadata to be stored along side the document inside the vector store writer

  3. Configure the Node:

    • Add a reference, this is used for makeing sure a document will not be added mutlipe times, it is optional, but recommended.
    • Select what kind of Document will be the input (either Text or Image)

  4. Complete the Workflow:

    • Ensure that all necessary nodes are connected and configured properly.
    • Validate the workflow to check for any errors or missing connections.
    • Execute the workflow to write the documents to the vector store.

Example of usage​

Let’s create a simple workflow to demonstrate how to use the Vector Store Writer:

Example Task: Storing Website Content​

Objective: Create a workflow to write the content of a website to a vector store using the Vector Store Writer.

Step-by-Step Setup:​

  1. Website Scraper:

    • Drag the Website Scraper from the Tool category to your workflow canvas.
    • Set the "Source" to "Custom URL" and enter the URL in the "Custom URL" field.

  2. Set Up OpenAI Embedder:

    • Drag the OpenAI Embedder from the Knowledge category to your workflow canvas.
    • Configure the embedder with your OpenAI API key.

  3. Set Up OpenSearch Client:

    • Drag the OpenSearch Client from the Knowledge category to your workflow canvas.
    • Configure the client node with the endpoint URL of your OpenSearch cluster and connect it to an AWS Config providing the necessary AWS credentials.

  4. Set Up Vector Store:

    • Drag the Vector Store from the Knowledge category to your workflow canvas.
    • Enter a name for your index, such as "website_content".

  5. Set Up Vector Store Writer:

    • Drag the Vector Store Writer from the Knowledge category to your workflow canvas.
    • Enter a reference to identify your documents, this is optional, but with this you can ensure that the same document does not get added multiple times

  6. Document Splitter:

    • Drag the Document Splitter from the Knowledge category to your workflow canvas.
    • Select the type of Splitter you want to use, we will use the HTML Splitter.

  7. Create Connections:

    • Connect the output of the Website Scraper (Website Content) to the input of the Vector Store Writer (Document).
    • Connect the output of the OpenAI Embedder (Embedder) to the input of the Vector Store (Embedder).
    • Connect the output of the OpenSearch Client (Database Client) to the input of the Vector Store (Database Client).
    • Connect the output of the Splitter (Document Splitter) with the input of your Vector Store Writer (Document Splitter).
    • Connect the output of the Vector Store (Vector Store) with the input of your Vector Store Writer (Vector Store).

  8. Execute the Workflow:

    • Run the workflow to scrape the website, generate vector embeddings for the content, and store these embeddings in the OpenSearch index.
Flowchart showing a data processing workflow with six nodes: 'OpenSearch Client', 'OpenAI Embedder', 'Vector Store', 'Website Scraper', 'Document Splitter', and 'Vector Store Writer'. Arrows indicate connections between nodes, demonstrating data flow. Black background with white text.

Additional Information​

OpenSearch Index Management​

  • Ensure that the OpenSearch cluster is properly configured and accessible.
  • The index name should be unique within the OpenSearch cluster to avoid conflicts.

Embedder Configuration​

  • The embedder (e.g., OpenAI Embedder) must be configured with valid API credentials to generate embeddings.
  • Make sure that the embedder is correctly connected to the Vector Store to ensure seamless data flow.

By following these steps, you can effectively set up and use the Vector Store to manage and store vector embeddings for various applications such as semantic search, recommendation systems, and more.