CTC29 Local LLM — BeeBee AI Help
Large Language Models (LLM’s) are the talk of not just the tech world but main stream news channels around the world in the last 6 months. These models ‘read’ the internet and then models are trained based on text transformer inputs. The model can then be queried via a website in the cloud via a chatbot interface e.g. from openAI chatGPT or open-assistant etc. The list grows weekly.
Can these LLM’s be run locally on a laptop? That was the question posed at Code The City’s CTC29 health hackathon. The open sourcing of LLM’s has split opinions: this technology is potentailly too danergous to be released openly or the by open sourcing the code we will have more democratic access and have a wider scrunity of the code. One entity taking the latter view is nomicAI that have opensourced GTP4all. This open source code was downloaded and installed on Ubuntu.
Various models are offered, some open source licenses other just for reasearch. An open source license model was downloaded 3.7GB and installed via their desktop application. The chat interface was used and it was very very slow to reply. Most of these chatbots run on GPU clusters in the cloud but GTP4All runs on a computers CPU but there is an option to use the GPU. Model can be ‘fine tuned’ by providing local documents. The toolkit allowed for text documents to be added to a folder for LLM to access. Queries relevant to the document, help instructions in our case, did get referenced when a natural query was asked, How to upload a file to BentoBox-DS?
Next the challenge was to run the model via code, python was used but other libraires are offered. This ran just about as slow to respond but adding ‘fine tuning’ documents was not achived as the documentation is limited. Under the hood, LangChain is used to setup and manage the LLM. As the name suggests this give the ability to chain together many LLM’s. BentoBox-DS wants to extract the datatype context, time period, type of visualisation and compute to perform. So, one master agent will mange the overall goal while seperate LLM’s will work on each of the individual extraction of context. These can then all be brought together to form a query for HOP to get the source data and produce a chart. As the Peer provides feedback it can learn how to map queries to chart better over time.
Overall, running a LLM’s is not practical for any application base on this weekends experience but with small training dataset e.g. limited to wikipedia and help documentation you can envision a 10x smaller model but with 100x more accuracy being built. Or just wait 2–3 years and the ‘general’ foundation LLM’s will do it for you?
Summary slides and links below.
GPT4all https://gpt4all.io/index.html
pandasAI https://levelup.gitconnected.com/introducing-pandasai-the-generative-ai-python-library-568a971af014
Langchain https://python.langchain.com/docs/modules/model_io/models/llms/integrations/gpt4all.html
Open-Assistant https://open-assistant.io/
BentoBox-DS https://bentoboxds.org/ discord channel or BeeBeeAI
