Suramya's Blog : Welcome to my crazy life…

One of the things that I have been wanting for a while is the ability to look up stuff about a book or series so that when a new book in the series comes out I don’t have to re-read the old books before starting the new one. I end up doing that because otherwise since I have forgotten a large part of the backstory it would make the new book a lot less enjoyable. There are some authors who solve this problem by summarizing the backstory when the book starts and it is really useful. When ChatGPT and associated AI bots started picking up one of the ideas I came up with was to create a chatbot trained on a book series so that I can ask it a question about the book and get an answer. Would be really useful when dealing with book series with a large cast and multiple overlying stories.

Recently privateGPT showed up in my feed with the following claim: “Ask questions to your documents without an internet connection, using the power of LLMs. 100% private, no data leaves your execution environment at any point. You can ingest documents and ask questions without an internet connection!” and it seemed like it was the perfect tool for what I wanted to do so I downloaded the program from GitHub and set it up on my machine.

Once the software was installed, I copied a single book from the Mercedes Lackey Valdemar series called Darian’s Tale 01: Owlflight into a temp directory and told privateGPT to ingest the contents. The ingestion took hours even though the file was only ~600kb in size, I left it running overnight and it completed sometime during the night. Then I ran the prompt to be able to query the chatbot. The first few times I tried querying it, the app crashed because I ran out of memory. So I enabled SWAP on my system and then it finally ran the query and answered.

Unfortunately the answers I got were complete nonsense. You can see the questions I asked and the answers I got below:

Enter a query: what is a heartstone

llama_print_timings:        load time =   974.93 ms
llama_print_timings:      sample time =     0.00 ms /     1 runs   (    0.00 ms per run)
llama_print_timings: prompt eval time =   974.77 ms /     7 tokens (  139.25 ms per token)
llama_print_timings:        eval time =     0.00 ms /     1 runs   (    0.00 ms per run)
llama_print_timings:       total time =   977.55 ms

A heartstone is a type of music that is typically composed in the key of F major. It is characterized by a steady rhythm and a melody that is often slow and mournful. 
Heartstones are often used in classical music and are often associated with love and loss. They are also used in some contemporary music genres, such as folk and world music.

The correct answer is (as per Valdemar Fandom Wiki):

A Heartstone is both an artificial node produced by the Tayledras and the node’s physical anchor. The knowledge of how to create a Heartstone was given to the Tayledras by the Star-Eyed after the Sundering of the Clans. She tasked the Tayledras to cleanse the untamed magic caused by the Mage Wars. A Heartstone is used as a vessel to contain the cleansed magic. Ley-Lines are re-routed by Adepts and Masters to drain entirely into the Heartstone, bringing the surrounding area back to normal. Once their work is finished at that location, the Clan will establish a new Vale and Heartstone in a new area that needs cleansing. They drain the old Heartstone entirely and transfer the energy to the new Heartstone where they will begin their work anew.

So I asked another question:

Enter a query: who are the Hertasi

llama_print_timings:        load time =   974.93 ms
llama_print_timings:      sample time =     0.00 ms /     1 runs   (    0.00 ms per run)
llama_print_timings: prompt eval time = 26948.27 ms /     7 tokens ( 3849.75 ms per token)
llama_print_timings:        eval time =     0.00 ms /     1 runs   (    0.00 ms per run)
llama_print_timings:       total time = 26976.13 ms

The Herta si are a type of ghostly beings in the world of Eri. They are often associated with love and romance, and they are known for their ability to control and manipulate emotions. 
They are often depicted as beautiful, ethereal creatures, with long, flowing hair and eyes that glow with an otherworldly light. 

In this result it is talking about characters from a completely different series called the Herta. The correct answer is:

Hertasi are popularly considered to be one of the five non-human sentient races created by Urtho, but the work to “uplift” the hertasi was begun earlier, by Sorcerer-Adept Khal Herta. When he died, his work passed to Urtho.

The wild hertasi began as large, solitary lizards living in swamps, with the intelligence of a dog. They were “mild-tempered reptiles, available in large quantities, living fairly simple lives.”[1] They could have been described as semi-sentient before the Adepts took a hand. After Khal Herta’s efforts, Hertasi “had simple structures, organized hunting and fishing, and rudimentary medicine.”[1] Today, they are extremely active, intelligent, cave and tunnel-dwelling, social creatures.

Since the results were completely useless like all the other AI chatbots that I have tried so far, I have removed the software on my machine and will come back to this at somepoint later in time by when things would have hopefully improved.

– Suramya