Use Your Own Data To Get Response From GPT like ChatGPT | Python
In this article, I’ll show you how you can use your locally stored text files to get response using GPT-3. You can ask questions and get response like ChatGPT.
On technology front, we will be using:
- OpenAI
- Langchain
- Python
Input files
You can take bunch of text files and store them in a directory on your local machine. I’ve grabbed input data from here and created 5 text files. My files are all about ‘Cause And Effect Of Homelessness’ and are placed in a directory named Store.
Import Required Packages
As we are using Python, let’s go ahead and import the required packages.
Once required packages are imported, we need to get OpenAI API key.
Get OpenAI API Key
To get the OpenAI key, you need to go to https://openai.com/, login and then grab the keys using highlighted way:
Once you got the key, set that inside an environment variable(I’m using Windows).
Load Input Data
In order to load our text files, we need to instantiate DirectoryLoader and that can be done as shown below:
In above code, glob needs to be mentioned so that it will pick only the text files. This is particularly useful, when your input directory contains mix of different-different types of files.
Split Data
As input data could be very long, we need to split our data into small chunks and here I’m taking chunk size as 1000.
After splitting, this is how the text looks like:
Checkout my complete article here
Using locally stored documents with AI models makes it possible to build useful question-answering applications without manually reading every file. A typical workflow starts with a collection of text documents, loads the files into a Python application, divides lengthy content into manageable sections, and then uses language-model capabilities to retrieve relevant information and generate responses. This approach is particularly useful when the information is specific to a project, organization, research collection, or private document repository.
ReplyDeletePython provides a practical foundation for implementing this type of document-based AI workflow because it has libraries for file processing, text manipulation, embeddings, vector search, and language-model integration. Developers can organize documents in a local directory, load selected file types, split large documents into smaller chunks, and process those chunks before sending relevant context to an AI model. For learners who want to strengthen their Python skills before building such applications, resources such as ChatGPT Developer Training can provide a useful connection between Python programming and generative AI application development.
The document-processing stage is important because large files may contain much more information than can conveniently be handled in a single request. A loader can collect files from a directory, while a text splitter can divide the content into smaller chunks with an appropriate overlap so that related information is not unnecessarily separated. Frameworks such as LangChain can help connect document loading, text splitting, retrieval, and language-model operations into a structured workflow. Developers interested in broader AI tooling can also explore AI Tools Training to understand how these components can be combined into practical AI-powered applications.
ReplyDeleteOnce the documents have been loaded and divided into meaningful chunks, the application can retrieve relevant sections when a user asks a question and provide those sections as context for generating an answer. This creates a document-questioning workflow in which users can interact with their own collection of information rather than relying only on general model knowledge. Such an approach can be extended to research documents, technical manuals, project reports, educational materials, and other structured collections, making it a useful foundation for Generative AI Projects for Final Year.
ReplyDelete