As I delved into the world of AI and machine learning, I knew I wanted to take things into my own hands. Literally. Instead of relying on cloud-based services or remote servers, I decided to set up a local Large Language Model (LLM) instance on my Synology NAS.

Step 1: Selecting the Right Tools

My goal is to experiment with large language models for text analysis and generation, specifically in the context of digital health policy analysis. I want to explore how these models can be used to identify trends, patterns, and insights from large datasets of health-related texts.

In my search for suitable tools, I came across three options: h2o.ai, Jan.AI, and AnythingLLM. Each has its own strengths and weaknesses, which I’ll outline below:

• h2o.ai: H2O is a machine learning platform that provides pre-trained models for various NLP tasks, including text classification, sentiment analysis, and language modeling. Its benefits include ease of use, flexibility, and scalability. However, it may not be as comprehensive as other options.
• Jan.AI: Jan.AI is an AI-powered research platform that enables users to analyze and generate text based on large datasets. It offers features such as natural language processing, entity recognition, and topic modeling. Its benefits include ease of use, speed, and scalability. However, it may not be as customizable as other options.
• AnythingLLM: AnythingLLM is an open-source LLM framework that allows users to train and fine-tune their own models for specific tasks. Its benefits include customization, flexibility, and adaptability. However, it may require more technical expertise and computational resources.

Comparing these tools with my use case requirements:

• h2o.ai: While h2o.ai provides pre-trained models, they may not be tailored to the specific task of digital health policy analysis.
• Jan.AI: Jan.AI is easy to use and scalable, but it may not offer the level of customization I need for my specific use case.
• AnythingLLM: AnythingLLM offers customization and flexibility, which aligns with my requirements.

I decided to opt for AnythingLLM due to its ability to be customized for my specific use case. However, before setting up AnythingLLM, I also considered the importance of data privacy and security.

Step 2: Preparing My Synology NAS

Before setting up the LLM instance on my NAS, I needed to upgrade the RAM. According to the Reddit community, upgrading the DS920+ with 16GB of RAM is possible, but requires careful attention to the specific requirements for the RAM module.

To upgrade the RAM, you will need:

• RAM Module: The Synology DS920+ supports up to 32GB of RAM, but only in specific configurations. You can use a single 16GB DDR4 SO-DIMM module or two 8GB DDR4 SO-DIMM modules.
• Frequency and Voltage: The RAM module must have a frequency of 2400MHz and a voltage of 1.2V.
• Timings: The RAM module must support the following timings: CAS Latency (CL) 16, RAS to CAS (tRCD) 10, Write Recovery Time (tWR) 20.

Failure to meet these requirements may result in compatibility issues or even damage to the NAS. Make sure to check the specifications of your RAM module before upgrading.

Step 3: Setting Up AnythingLLM

After installing Docker on my NAS, I focused on setting up AnythingLLM. According to the official documentation, the installation process involves several steps:

1. Pulling the Docker Image: Run the command docker pull anythingllm/anything-llm to pull the latest version of the AnythingLLM Docker image.
2. Creating a New Container: Run the command docker run -d --name anything-llm -p 3001:3001 anythingllm/anything-llm to create a new container and map port 3001 to the host machine.
3. Configuring the Environment Variables: You need to set up environment variables for the LLM instance, such as LLM_ MODEL and LLM_TOKEN. You can do this by adding the following lines to your docker-compose.yml file:

version: '3'
services:
  anything-llm:
    image: anythingllm/anything-llm
    ports:
       - "3000:3000"
    environment:
       - LLM_MODEL=your-model-name
       - LLM_TOKEN=your-token

4. Running the Container: Run the command docker-compose up to start the container and make it available on port 3001.

Loading LLM Models

After setting up AnythingLLM, I encountered another challenge:

• Challenge 4: Loading LLM Models into Ollama: To load the desired LLM models into Ollama, I had to SSH into the Docker instance and set up secure access rights for my NAS. Once I did this, I was able to load the Llama3.2 3b model into Ollama.

The Failure

Unfortunately, after completing these steps, I discovered that the system as I set it up does not work. According to an issue report on GitHub (https://github.com/Mintplex-Labs/anything-llm/issues/1331), there is a known bug in the AnythingLLM Docker image that prevents it from loading LLM models correctly.

As I delved deeper into the issue, I discovered that the problem wasn’t with the LLM model loading or processing, but rather with the underlying hardware requirements of AnythingLLM. Specifically, it relies on a database called Lancedb, which has specific CPU requirements.

What is Lancedb?

Lancedb is a high-performance database designed for large-scale natural language processing tasks. It’s used by AnythingLLM to store and manage its models, allowing for efficient querying and retrieval of information. However, Lancedb requires a specific set of CPU instructions to function properly.

AVX (Advanced Vector Extensions)

AVX is a set of instructions introduced by Intel in 2011 that enables CPUs to perform complex vector operations more efficiently. It’s designed to improve the performance of scientific simulations, data compression, and other compute-intensive tasks. AVX allows for wider registers, which can store multiple sets of data simultaneously, making it possible to perform calculations on large datasets more quickly.

AVX2 (Advanced Vector Extensions 2)

AVX2 is an extension of the original AVX instruction set, introduced by Intel in 2013. It offers improved performance and functionality over its predecessor, with support for additional instructions and features. AVX2 enables CPUs to perform even more complex vector operations, making it essential for many modern applications, including machine learning and data science.

The Issue

The problem arises because the Synology NAS’s Intel Celeron 4125 processor does not support AVX or AVX2 instructions. This means that the Lancedb database, which relies on these instructions to function, cannot be used with this hardware configuration. As a result, the Docker image crashes and fails to start, causing the AnythingLLM instance to stop working.

Conclusion

In conclusion, the failure of the AnythingLLM instance was due to the CPU requirements of Lancedb, which demands support for AVX2 instructions. Unfortunately, the Synology NAS’s hardware does not meet these requirements, making it incompatible with Lancedb and preventing the Docker image from running successfully. This highlights the importance of considering the specific hardware requirements of software applications when setting up complex systems like LLMs.