release v1.1.0

This commit is contained in:
will.yang
2024-10-10 20:25:23 +08:00
parent 92fce84bdd
commit 71773f050a
28 changed files with 1717 additions and 650 deletions

View File

@@ -8,8 +8,8 @@ Before running the demo, you need to prepare the following files:
### Build
You can run the demo with the only command:
```bash
# ./build_rkllm_server_flask.sh [target_platform:rk3588/rk3576] [RKLLM-Server workshop] [transformed_rkllm_model_path in borad]
./build_rkllm_server_flask.sh rk3588 /user/data/rkllm_server /user/data/rkllm_server/model.rkllm
# Usage: ./build_rkllm_server_flask.sh --workshop [RKLLM-Server Working Path] --model_path [Absolute Path of Converted RKLLM Model on Board] --platform [Target Platform: rk3588/rk3576] --npu_num [NPU Core Count] [--lora_model_path [Lora Model Path]] [--prompt_cache_path [Prompt Cache File Path]]
./build_rkllm_server_flask.sh --workshop /user/data --model_path /user/data/model.rkllm --platform rk3588 --npu_num 3
```
### Access with API
After building the RKLLM-Server-Flask, You can use ‘chat_api_flask.py’ to access the RKLLM-Server-Flask and get the answser of RKLLM models.
@@ -20,8 +20,8 @@ Attention: you should check the IP address of the board with 'ifconfig' command
### Build
You can run the demo with the only command:
```bash
# ./build_rkllm_server_gradio.sh [target_platform:rk3588/rk3576] [RKLLM-Server workshop] [transformed_rkllm_model_path in borad]
./build_rkllm_server_gradio.sh rk3588 /user/data/rkllm_server /user/data/rkllm_server/model.rkllm
# Usage: ./build_rkllm_server_gradio.sh --workshop [RKLLM-Server Working Path] --model_path [Absolute Path of Converted RKLLM Model on Board] --platform [Target Platform: rk3588/rk3576] --npu_num [NPU Core Count] [--lora_model_path [Lora Model Path]] [--prompt_cache_path [Prompt Cache File Path]]
./build_rkllm_server_gradio.sh --workshop /user/data --model_path /user/data/model.rkllm --platform rk3588 --npu_num 3
```
### Access the Server
After running the demo, You can access the RKLLM-Server-Gradio with two ways:

View File

@@ -1,61 +1,110 @@
#!/bin/bash
#*****************************************************************************************#
# 该脚本为 RKLLM-Server-Flask 服务的一键设置脚本
# 用户可以运行该脚本实现Linux板端的 RKLLM-Server-Flask 服务的自动化部署。
# 使用说明: ./build_rkllm_server_flask.sh [目标平台:rk3588/rk3576] [RKLLM-Server工作路径] [已转换的rkllm模型在板端的绝对路径]
# example: ./build_rkllm_server_flask.sh rk3588 /user/data/rkllm_server /user/data/rkllm_server/model.rkllm
# This script is an automated setup script for the RKLLM-Server-Flask service.
# Users can run this script to automate the deployment of the RKLLM-Server-Flask service on a Linux board.
# Usage: ./build_rkllm_server_flask.sh --workshop [RKLLM-Server Working Path] --model_path [Absolute Path of Converted RKLLM Model on Board] --platform [Target Platform: rk3588/rk3576] --npu_num [NPU Core Count] [--lora_model_path [Lora Model Path]] [--prompt_cache_path [Prompt Cache File Path]]
# example: ./build_rkllm_server_flask.sh --workshop /user/data --model_path /user/data/model.rkllm --platform rk3588 --npu_num 3
#*****************************************************************************************#
#################### 检查板端是否已经安装了 pip/gradio 库 ####################
# 1.准备板端的gradio环境
LORA_PATH=""
PROMPT_FILE_PATH=""
# Function to display help
function show_help {
echo "Usage: ./build_rkllm_server_flask.sh --workshop [RKLLM-Server Working Path] --model_path [Absolute Path of Converted RKLLM Model on Board] --platform [Target Platform: rk3588/rk3576] --npu_num [NPU Core Count] [--lora_path [Lora Model Path]] [--prompt_cache_path [Prompt Cache File Path]]"
}
# Parse command-line options
while [[ $# -gt 0 ]]; do
case "$1" in
--workshop)
WORKING_PATH="$2"
shift 2
;;
--model_path)
MODEL_PATH="$2"
shift 2
;;
--platform)
TARGET_PLATFORM="$2"
shift 2
;;
--npu_num)
NPU_CORE_COUNT="$2"
shift 2
;;
--lora_model_path)
LORA_PATH="$2"
shift 2
;;
--prompt_cache_path)
PROMPT_FILE_PATH="$2"
shift 2
;;
--help)
show_help
exit 0
;;
*)
echo "无效的选项: $1" 1>&2
show_help
exit 1
;;
esac
done
#################### Check if pip and the Flask library are already installed on the board. ####################
adb shell << EOF
# 检查是否安装了 pip3
if ! command -v pip3 &> /dev/null; then
echo "-------- pip3 未安装,将进行安装... --------"
# 安装 pip3
echo "-------- pip3 is not installed. Installing it now... --------"
sudo apt update
sudo apt install python3-pip -y
else
echo "-------- pip3 已经安装 --------"
echo "-------- pip3 is already installed. --------"
fi
# 检查是否安装了 flask
if ! python3 -c "import flask" &> /dev/null; then
echo "-------- flask 未安装,将进行安装... --------"
echo "-------- flask is not installed. Installing it now... --------"
# 安装 flask
pip install flask==2.2.2 Werkzeug==2.2.2 -i https://pypi.tuna.tsinghua.edu.cn/simple
pip install flask==2.2.2 Werkzeug==2.2.2 -i https://pypi.tuna.tsinghua.edu.cn/simple --break-system-packages
else
echo "-------- flask 已经安装 --------"
echo "-------- flask is already installed. --------"
fi
exit
EOF
#################### 推送 server 运行的相关文件进入板端 ####################
# 2.检查需要推送进板端的路径是否存在
adb shell ls $2 > /dev/null 2>&1
#################### Push the relevant files for the server to the board. ####################
adb shell ls $WORKING_PATH > /dev/null 2>&1
if [ $? -ne 0 ]; then
# 如果路径不存在,则创建路径
adb shell mkdir -p $2
echo "-------- rkllm_server 工作目录不存在,已创建目录 --------"
adb shell mkdir -p $WORKING_PATH
echo "-------- The rkllm_server working directory does not exist, so it has been created. --------"
else
echo "-------- rkllm_server 工作目录已存在 --------"
echo "-------- The rkllm_server working directory already exists. --------"
fi
# 3.更新 ./rkllm_server/lib 中的 librkllmrt.so 文件
# Update the `librkllmrt.so` file in the `./rkllm_server/lib` directory.
cp ../../runtime/Linux/librkllm_api/aarch64/librkllmrt.so ./rkllm_server/lib/
# 4.推送文件到 Linux 板端
adb push ./rkllm_server $2
adb push ./rkllm_server $WORKING_PATH
#################### Enter the board terminal and start the server service. ####################
CMD="python3 flask_server.py --rkllm_model_path $MODEL_PATH --target_platform $TARGET_PLATFORM --num_npu_core $NPU_CORE_COUNT"
if [[ -n "$LORA_PATH" ]]; then
CMD="$CMD --lora_model_path $LORA_PATH"
fi
if [[ -n "$PROMPT_FILE_PATH" ]]; then
CMD="$CMD --prompt_cache_path $PROMPT_FILE_PATH"
fi
#################### 进入板端并启动 server 服务 ####################
# 5.进入板端启动 server 服务
adb shell << EOF
cd $2/rkllm_server/
python3 flask_server.py --target_platform $1 --rkllm_model_path $3
cd $WORKING_PATH/rkllm_server/
$CMD
EOF

View File

@@ -1,61 +1,110 @@
#!/bin/bash
#*****************************************************************************************#
# 该脚本为 RKLLM-Server-Gradio 服务的一键设置脚本
# 用户可以运行该脚本实现Linux板端的 RKLLM-Server-Gradio 服务的自动化部署。
# 使用说明: ./build_rkllm_server_gradio.sh [目标平台:rk3588/rk3576] [RKLLM-Server工作路径] [已转换的rkllm模型在板端的绝对路径]
# example: ./build_rkllm_server_gradio.sh rk3588 /user/data/rkllm_server /user/data/rkllm_server/model.rkllm
# This script is an automated setup script for the RKLLM-Server-Gradio service.
# Users can run this script to automate the deployment of the RKLLM-Server-Gradio service on a Linux board.
# Usage: ./build_rkllm_server_gradio.sh --workshop [RKLLM-Server Working Path] --model_path [Absolute Path of Converted RKLLM Model on Board] --platform [Target Platform: rk3588/rk3576] --npu_num [NPU Core Count] [--lora_model_path [Lora Model Path]] [--prompt_cache_path [Prompt Cache File Path]]
# example: ./build_rkllm_server_gradio.sh --workshop /user/data --model_path /user/data/model.rkllm --platform rk3588 --npu_num 3
#*****************************************************************************************#
#################### 检查板端是否已经安装了 pip/gradio 库 ####################
# 1.准备板端的gradio环境
LORA_PATH=""
PROMPT_FILE_PATH=""
# Function to display help
function show_help {
echo "Usage: ./build_rkllm_server_gradio.sh --workshop [RKLLM-Server Working Path] --model_path [Absolute Path of Converted RKLLM Model on Board] --platform [Target Platform: rk3588/rk3576] --npu_num [NPU Core Count] [--lora_path [Lora Model Path]] [--prompt_cache_path [Prompt Cache File Path]]"
}
# Parse command-line options
while [[ $# -gt 0 ]]; do
case "$1" in
--workshop)
WORKING_PATH="$2"
shift 2
;;
--model_path)
MODEL_PATH="$2"
shift 2
;;
--platform)
TARGET_PLATFORM="$2"
shift 2
;;
--npu_num)
NPU_CORE_COUNT="$2"
shift 2
;;
--lora_model_path)
LORA_PATH="$2"
shift 2
;;
--prompt_cache_path)
PROMPT_FILE_PATH="$2"
shift 2
;;
--help)
show_help
exit 0
;;
*)
echo "无效的选项: $1" 1>&2
show_help
exit 1
;;
esac
done
#################### Check if pip and the Gradio library are already installed on the board. ####################
adb shell << EOF
# 检查是否安装了 pip3
if ! command -v pip3 &> /dev/null; then
echo "-------- pip3 未安装,将进行安装... --------"
# 安装 pip3
echo "-------- pip3 is not installed. Installing it now... --------"
sudo apt update
sudo apt install python3-pip -y
else
echo "-------- pip3 已经安装 --------"
echo "-------- pip3 is already installed. --------"
fi
# 检查是否安装了 gradio
if ! python3 -c "import gradio" &> /dev/null; then
echo "-------- Gradio 未安装,将进行安装... --------"
# 安装 Gradio
pip3 install gradio>=4.24.0 -i https://pypi.tuna.tsinghua.edu.cn/simple/
echo "-------- gradio is not installed. Installing it now... --------"
pip3 install gradio>=4.24.0 -i https://pypi.tuna.tsinghua.edu.cn/simple/ --break-system-packages
else
echo "-------- Gradio 已经安装 --------"
echo "-------- gradio is already installed. --------"
fi
exit
EOF
#################### 推送 server 运行的相关文件进入板端 ####################
# 2.检查需要推送进板端的路径是否存在
adb shell ls $2 > /dev/null 2>&1
#################### Push the relevant files for the server to the board. ####################
adb shell ls $WORKING_PATH > /dev/null 2>&1
if [ $? -ne 0 ]; then
# 如果路径不存在,则创建路径
adb shell mkdir -p $2
echo "-------- rkllm_server 工作目录不存在,已创建目录 --------"
adb shell mkdir -p $WORKING_PATH
echo "-------- The rkllm_server working directory does not exist, so it has been created. --------"
else
echo "-------- rkllm_server 工作目录已存在 --------"
echo "-------- The rkllm_server working directory already exists. --------"
fi
# 3.更新 ./rkllm_server/lib 中的 librkllmrt.so 文件
# Update the `librkllmrt.so` file in the `./rkllm_server/lib` directory.
cp ../../runtime/Linux/librkllm_api/aarch64/librkllmrt.so ./rkllm_server/lib/
# 4.推送文件到 Linux 板端
adb push ./rkllm_server $2
adb push ./rkllm_server $WORKING_PATH
#################### Enter the board terminal and start the server service. ####################
CMD="python3 gradio_server.py --rkllm_model_path $MODEL_PATH --target_platform $TARGET_PLATFORM --num_npu_core $NPU_CORE_COUNT"
if [[ -n "$LORA_PATH" ]]; then
CMD="$CMD --lora_model_path $LORA_PATH"
fi
if [[ -n "$PROMPT_FILE_PATH" ]]; then
CMD="$CMD --prompt_cache_path $PROMPT_FILE_PATH"
fi
#################### 进入板端并启动 server 服务 ####################
# 5.进入板端启动 server 服务
adb shell << EOF
cd $2/rkllm_server/
python3 gradio_server.py --target_platform $1 --rkllm_model_path $3
cd $WORKING_PATH/rkllm_server/
$CMD
EOF

View File

@@ -2,53 +2,53 @@ import sys
import requests
import json
# 设置 Server 服务器的地址
server_url = 'http://172.16.10.102:8080/rkllm_chat'
# 设置是否开启流式对话
# Set the address of the Server.
server_url = 'http://172.16.10.79:8080/rkllm_chat'
# Set whether to enable streaming mode.
is_streaming = True
# 创建一个会话对象
# Create a session object.
session = requests.Session()
session.keep_alive = False # 关闭连接池,保持长连接
session.keep_alive = False # Close the connection pool to maintain a long connection.
adapter = requests.adapters.HTTPAdapter(max_retries=5)
session.mount('https://', adapter)
session.mount('http://', adapter)
if __name__ == '__main__':
print("============================")
print("在终端中输入您的问题,即可与 RKLLM 模型进行对话....")
print("Input your question in the terminal to start a conversation with the RKLLM model...")
print("============================")
# 进入循环,持续获取用户输入,并与RKLLM模型进行对话
# Enter a loop to continuously get user input and converse with the RKLLM model.
while True:
try:
user_message = input("请输入您的问题:")
user_message = input("\n*Please enter your question:")
if user_message == "exit":
print("============================")
print("程序正在退出......")
print("The RKLLM Server is stopping......")
print("============================")
break
else:
# 设置请求头,此处的请求头实际并无作用,仅为模拟OpenAI接口设计
# Set the request headers; in this case, the headers have no actual effect and are only used to simulate the OpenAI interface design.
headers = {
'Content-Type': 'application/json',
'Authorization': 'not_required'
}
# 准备要发送的数据
# model: 为用户在设置RKLLM-Server时定义的模型,此处并无作用
# messages: 用户输入的问题,RKLLM-Server将会把它作为输入,并返回模型的回复;支持在 messags 加入多个问题
# stream: 是否开启流式对话,与OpenAI接口相同
# Prepare the data to be sent
# model: The model defined by the user when setting up RKLLM-Server; this has no effect here
# messages: The user's input question, which RKLLM-Server will use as input and return the model's reply; multiple questions can be added to messages
# stream: Whether to enable streaming conversation, similar to the OpenAI interface
data = {
"model": 'your_model_deploy_with_RKLLM_Server',
"messages": [{"role": "user", "content": user_message}],
"stream": is_streaming
}
# 发送 POST 请求
# Send a POST request
responses = session.post(server_url, json=data, headers=headers, stream=is_streaming, verify=False)
if not is_streaming:
# 解析响应
# Parse the response
if responses.status_code == 200:
print("Q:", data["messages"][-1]["content"])
print("A:", json.loads(responses.text)["choices"][-1]["message"]["content"])
@@ -66,16 +66,13 @@ if __name__ == '__main__':
sys.stdout.flush()
else:
print('Error:', responses.text)
except KeyboardInterrupt:
# 捕获 Ctrl-C 信号,关闭会话
# Capture Ctrl-C signal to close the session
session.close()
print("\n")
print("============================")
print("程序正在退出......")
print("The RKLLM Server is stopping......")
print("============================")
break

View File

@@ -1,44 +1,43 @@
from gradio_client import Client
# 该函数通过调用Gradio Client API与RKLLM模型进行交互
# This function interacts with the RKLLM model by calling the Gradio Client API.
def chat_with_rkllm(user_message, history=[]):
# 实例化Gradio Client,用户需要根据自己部署的具体网址进行修改
client = Client("http://172.16.10.102:8080/")
# Instantiate the Gradio Client. Users need to modify according to their specific deployment URL.
client = Client("http://172.16.10.79:8080")
# 调用Gradio Client API进行交互,内部的API主要包括:
# /get_user_input:模型获取用户输入,并将输入添加至历史记录history
# /get_RKLLM_output:RKLLM利用已包含输入的历史记录history生成回复
# Call the Gradio Client API for interaction. The internal APIs mainly include:
# get_user_input: The model retrieves user input and adds it to the history record 'history'.
# get_RKLLM_output: RKLLM generates a response using the historical record 'history' that contains the input.
_, history = client.predict(user_message=user_message, history=history, api_name="/get_user_input")
result_history = client.predict(history=history, api_name="/get_RKLLM_output")
return result_history
if __name__ == '__main__':
#初始化聊天记录
result_history = []
print("============================")
print("在终端中输入您的问题,即可与 RKLLM 模型进行对话....")
print("Enter your question in the terminal to have a conversation with the RKLLM model...")
print("============================")
# 进入循环,持续获取用户输入,并与RKLLM模型进行对话
# Enter a loop to continuously receive user input and have a conversation with the RKLLM model...
while True:
try:
user_message = input("请输入您的问题:")
user_message = input("Please enter your question:")
if user_message == "exit":
print("============================")
print("程序正在退出......")
print("The RKLLM Server is stopping......")
print("============================")
break
else:
# 调用chat_with_rkllm函数,获取模型的回复
# Call the `chat_with_rkllm` function to get the model's response.
result_history = chat_with_rkllm(user_message, result_history)
# 打印模型输出
# Print the history of chatting
print("Q:", result_history[-1][0])
print("A:", result_history[-1][1])
except KeyboardInterrupt:
print("\n")
print("============================")
print("程序正在退出......")
print("The RKLLM Server is stopping......")
print("============================")
break

View File

@@ -11,68 +11,41 @@ from flask import Flask, request, jsonify, Response
app = Flask(__name__)
# 创建一个锁,用于控制多人访问Server
lock = threading.Lock()
PROMPT_TEXT_PREFIX = "<|im_start|>system You are a helpful assistant. <|im_end|> <|im_start|>user"
PROMPT_TEXT_POSTFIX = "<|im_end|><|im_start|>assistant"
# 创建一个全局变量,用于标识服务器当前是否处于阻塞状态
is_blocking = False
# 设置动态库路径
# Set the dynamic library path
rkllm_lib = ctypes.CDLL('lib/librkllmrt.so')
# 定义全局变量,用于保存回调函数的输出,便于在gradio界面中输出
global_text = []
global_state = -1
split_byte_data = bytes(b"") # 用于保存分割的字节数据
# Define the structures from the library
RKLLM_Handle_t = ctypes.c_void_p
userdata = ctypes.c_void_p(None)
# 定义动态库中的结构体
class Token(ctypes.Structure):
_fields_ = [
("logprob", ctypes.c_float),
("id", ctypes.c_int32)
]
LLMCallState = ctypes.c_int
LLMCallState.RKLLM_RUN_NORMAL = 0
LLMCallState.RKLLM_RUN_FINISH = 1
LLMCallState.RKLLM_RUN_ERROR = 2
LLMCallState.RKLLM_RUN_GET_LAST_HIDDEN_LAYER = 3
class RKLLMResult(ctypes.Structure):
_fields_ = [
("text", ctypes.c_char_p),
("tokens", ctypes.POINTER(Token)),
("num", ctypes.c_int32)
]
RKLLMInputMode = ctypes.c_int
RKLLMInputMode.RKLLM_INPUT_PROMPT = 0
RKLLMInputMode.RKLLM_INPUT_TOKEN = 1
RKLLMInputMode.RKLLM_INPUT_EMBED = 2
RKLLMInputMode.RKLLM_INPUT_MULTIMODAL = 3
RKLLMInferMode = ctypes.c_int
RKLLMInferMode.RKLLM_INFER_GENERATE = 0
RKLLMInferMode.RKLLM_INFER_GET_LAST_HIDDEN_LAYER = 1
# 定义回调函数
def callback(result, userdata, state):
global global_text, global_state, split_byte_data
if state == 0:
# 保存输出的token文本及RKLLM运行状态
global_state = state
# 需要监控当前的字节数据是否完整,不完整则进行记录,后续进行解析
try:
global_text.append((split_byte_data + result.contents.text).decode('utf-8'))
print((split_byte_data + result.contents.text).decode('utf-8'), end='')
split_byte_data = bytes(b"")
except:
split_byte_data += result.contents.text
sys.stdout.flush()
elif state == 1:
# 保存RKLLM运行状态
global_state = state
print("\n")
sys.stdout.flush()
else:
print("run error")
# Python端与C++端的回调函数连接
callback_type = ctypes.CFUNCTYPE(None, ctypes.POINTER(RKLLMResult), ctypes.c_void_p, ctypes.c_int)
c_callback = callback_type(callback)
# 定义动态库中的结构体
class RKNNllmParam(ctypes.Structure):
class RKLLMParam(ctypes.Structure):
_fields_ = [
("model_path", ctypes.c_char_p),
("license_path", ctypes.c_char_p),
("num_npu_core", ctypes.c_int32),
("max_context_len", ctypes.c_int32),
("n_prefill_batch", ctypes.c_int32),
("max_new_tokens", ctypes.c_int32),
("skip_special_token", ctypes.c_bool),
("top_k", ctypes.c_int32),
("top_p", ctypes.c_float),
("temperature", ctypes.c_float),
@@ -84,58 +57,234 @@ class RKNNllmParam(ctypes.Structure):
("mirostat_eta", ctypes.c_float),
("logprobs", ctypes.c_bool),
("top_logprobs", ctypes.c_int32),
("use_gpu", ctypes.c_bool)
("is_async", ctypes.c_bool),
("img_start", ctypes.c_char_p),
("img_end", ctypes.c_char_p),
("img_content", ctypes.c_char_p),
]
# 定义RKLLM_Handle_t和userdata
RKLLM_Handle_t = ctypes.c_void_p
userdata = ctypes.c_void_p(None)
class RKLLMLoraAdapter(ctypes.Structure):
_fields_ = [
("lora_adapter_path", ctypes.c_char_p),
("lora_adapter_name", ctypes.c_char_p),
("scale", ctypes.c_float)
]
# 设置提示文本
PROMPT_TEXT_PREFIX = "<|im_start|>system You are a helpful assistant. <|im_end|> <|im_start|>user"
PROMPT_TEXT_POSTFIX = "<|im_end|><|im_start|>assistant"
class RKLLMLoraParam(ctypes.Structure):
_fields_ = [
("lora_adapter_name", ctypes.c_char_p)
]
# 定义Python端的RKLLM类,其中包括了对动态库中RKLLM模型的初始化、推理及释放操作
class RKLLMPromptCacheParam(ctypes.Structure):
_fields_ = [
("save_prompt_cache", ctypes.c_int),
("prompt_cache_path", ctypes.c_char_p)
]
class RKLLMEmbedInput(ctypes.Structure):
_fields_ = [
("embed", ctypes.POINTER(ctypes.c_float)),
("n_tokens", ctypes.c_size_t)
]
class RKLLMTokenInput(ctypes.Structure):
_fields_ = [
("input_ids", ctypes.POINTER(ctypes.c_int32)),
("n_tokens", ctypes.c_size_t)
]
class RKLLMMultiModelInput(ctypes.Structure):
_fields_ = [
("prompt", ctypes.c_char_p),
("image_embed", ctypes.POINTER(ctypes.c_float)),
("n_image_tokens", ctypes.c_size_t)
]
class RKLLMInputUnion(ctypes.Union):
_fields_ = [
("prompt_input", ctypes.c_char_p),
("embed_input", RKLLMEmbedInput),
("token_input", RKLLMTokenInput),
("multimodal_input", RKLLMMultiModelInput)
]
class RKLLMInput(ctypes.Structure):
_fields_ = [
("input_mode", ctypes.c_int),
("input_data", RKLLMInputUnion)
]
class RKLLMInferParam(ctypes.Structure):
_fields_ = [
("mode", RKLLMInferMode),
("lora_params", ctypes.POINTER(RKLLMLoraParam)),
("prompt_cache_params", ctypes.POINTER(RKLLMPromptCacheParam))
]
class RKLLMResultLastHiddenLayer(ctypes.Structure):
_fields_ = [
("hidden_states", ctypes.POINTER(ctypes.c_float)),
("embd_size", ctypes.c_int),
("num_tokens", ctypes.c_int)
]
class RKLLMResult(ctypes.Structure):
_fields_ = [
("text", ctypes.c_char_p),
("size", ctypes.c_int),
("last_hidden_layer", RKLLMResultLastHiddenLayer)
]
# Create a lock to control multi-user access to the server.
lock = threading.Lock()
# Create a global variable to indicate whether the server is currently in a blocked state.
is_blocking = False
# Define global variables to store the callback function output for displaying in the Gradio interface
global_text = []
global_state = -1
split_byte_data = bytes(b"") # Used to store the segmented byte data
# Define the callback function
def callback_impl(result, userdata, state):
global global_text, global_state, split_byte_data
if state == LLMCallState.RKLLM_RUN_FINISH:
global_state = state
print("\n")
sys.stdout.flush()
elif state == LLMCallState.RKLLM_RUN_ERROR:
global_state = state
print("run error")
sys.stdout.flush()
elif state == LLMCallState.RKLLM_RUN_GET_LAST_HIDDEN_LAYER:
'''
If using the GET_LAST_HIDDEN_LAYER function, the callback interface will return the memory pointer: last_hidden_layer, the number of tokens: num_tokens, and the size of the hidden layer: embd_size.
With these three parameters, you can retrieve the data from last_hidden_layer.
Note: The data needs to be retrieved during the current callback; if not obtained in time, the pointer will be released by the next callback.
'''
if result.last_hidden_layer.embd_size != 0 and result.last_hidden_layer.num_tokens != 0:
data_size = result.last_hidden_layer.embd_size * result.last_hidden_layer.num_tokens * ctypes.sizeof(ctypes.c_float)
print(f"data_size: {data_size}")
global_text.append(f"data_size: {data_size}\n")
output_path = os.getcwd() + "/last_hidden_layer.bin"
with open(output_path, "wb") as outFile:
data = ctypes.cast(result.last_hidden_layer.hidden_states, ctypes.POINTER(ctypes.c_float))
float_array_type = ctypes.c_float * (data_size // ctypes.sizeof(ctypes.c_float))
float_array = float_array_type.from_address(ctypes.addressof(data.contents))
outFile.write(bytearray(float_array))
print(f"Data saved to {output_path} successfully!")
global_text.append(f"Data saved to {output_path} successfully!")
else:
print("Invalid hidden layer data.")
global_text.append("Invalid hidden layer data.")
global_state = state
time.sleep(0.05) # Delay for 0.05 seconds to wait for the output result
sys.stdout.flush()
else:
# Save the output token text and the RKLLM running state
global_state = state
# Monitor if the current byte data is complete; if incomplete, record it for later parsing
try:
global_text.append((split_byte_data + result.contents.text).decode('utf-8'))
print((split_byte_data + result.contents.text).decode('utf-8'), end='')
split_byte_data = bytes(b"")
except:
split_byte_data += result.contents.text
sys.stdout.flush()
# Connect the callback function between the Python side and the C++ side
callback_type = ctypes.CFUNCTYPE(None, ctypes.POINTER(RKLLMResult), ctypes.c_void_p, ctypes.c_int)
callback = callback_type(callback_impl)
# Define the RKLLM class, which includes initialization, inference, and release operations for the RKLLM model in the dynamic library
class RKLLM(object):
def __init__(self, model_path, target_platform):
rknnllm_param = RKNNllmParam()
rknnllm_param.model_path = bytes(model_path, 'utf-8')
if target_platform == "rk3588":
rknnllm_param.num_npu_core = 3
elif target_platform == "rk3576":
rknnllm_param.num_npu_core = 1
rknnllm_param.max_context_len = 320
rknnllm_param.max_new_tokens = 512
rknnllm_param.top_k = 1
rknnllm_param.top_p = 0.9
rknnllm_param.temperature = 0.8
rknnllm_param.repeat_penalty = 1.1
rknnllm_param.frequency_penalty = 0.0
rknnllm_param.presence_penalty = 0.0
rknnllm_param.mirostat = 0
rknnllm_param.mirostat_tau = 5.0
rknnllm_param.mirostat_eta = 0.1
rknnllm_param.logprobs = False
rknnllm_param.top_logprobs = 5
rknnllm_param.use_gpu = True
def __init__(self, model_path, num_npu_core, lora_model_path = None, prompt_cache_path = None):
rkllm_param = RKLLMParam()
rkllm_param.model_path = bytes(model_path, 'utf-8')
rkllm_param.license_path = None
rkllm_param.num_npu_core = num_npu_core
rkllm_param.max_context_len = 512
rkllm_param.n_prefill_batch = 512
rkllm_param.max_new_tokens = -1
rkllm_param.skip_special_token = True
rkllm_param.top_k = 1
rkllm_param.top_p = 0.9
rkllm_param.temperature = 0.8
rkllm_param.repeat_penalty = 1.1
rkllm_param.frequency_penalty = 0.0
rkllm_param.presence_penalty = 0.0
rkllm_param.mirostat = 0
rkllm_param.mirostat_tau = 5.0
rkllm_param.mirostat_eta = 0.1
rkllm_param.logprobs = False
rkllm_param.top_logprobs = 5
rkllm_param.is_async = False
rkllm_param.img_start = "".encode('utf-8')
rkllm_param.img_end = "".encode('utf-8')
rkllm_param.img_content = "".encode('utf-8')
self.handle = RKLLM_Handle_t()
self.rkllm_init = rkllm_lib.rkllm_init
self.rkllm_init.argtypes = [ctypes.POINTER(RKLLM_Handle_t), ctypes.POINTER(RKNNllmParam), callback_type]
self.rkllm_init.argtypes = [ctypes.POINTER(RKLLM_Handle_t), RKLLMParam, callback_type]
self.rkllm_init.restype = ctypes.c_int
self.rkllm_init(ctypes.byref(self.handle), rknnllm_param, c_callback)
self.rkllm_init(ctypes.byref(self.handle), rkllm_param, callback)
self.rkllm_run = rkllm_lib.rkllm_run
self.rkllm_run.argtypes = [RKLLM_Handle_t, ctypes.POINTER(ctypes.c_char), ctypes.c_void_p]
self.rkllm_run.argtypes = [RKLLM_Handle_t, ctypes.POINTER(RKLLMInput), ctypes.POINTER(RKLLMInferParam), ctypes.c_void_p]
self.rkllm_run.restype = ctypes.c_int
self.rkllm_destroy = rkllm_lib.rkllm_destroy
self.rkllm_destroy.argtypes = [RKLLM_Handle_t]
self.rkllm_destroy.restype = ctypes.c_int
self.lora_adapter_path = None
self.lora_model_name = None
if lora_model_path:
self.lora_adapter_path = lora_model_path
self.lora_adapter_name = "test"
lora_adapter = RKLLMLoraAdapter()
ctypes.memset(ctypes.byref(lora_adapter), 0, ctypes.sizeof(RKLLMLoraAdapter))
lora_adapter.lora_adapter_path = ctypes.c_char_p((self.lora_adapter_path).encode('utf-8'))
lora_adapter.lora_adapter_name = ctypes.c_char_p((self.lora_adapter_name).encode('utf-8'))
lora_adapter.scale = 1.0
rkllm_load_lora = rkllm_lib.rkllm_load_lora
rkllm_load_lora.argtypes = [RKLLM_Handle_t, ctypes.POINTER(RKLLMLoraAdapter)]
rkllm_load_lora.restype = ctypes.c_int
rkllm_load_lora(self.handle, ctypes.byref(lora_adapter))
self.prompt_cache_path = None
if prompt_cache_path:
self.prompt_cache_path = prompt_cache_path
rkllm_load_prompt_cache = rkllm_lib.rkllm_load_prompt_cache
rkllm_load_prompt_cache.argtypes = [RKLLM_Handle_t, ctypes.c_char_p]
rkllm_load_prompt_cache.restype = ctypes.c_int
rkllm_load_prompt_cache(self.handle, ctypes.c_char_p((prompt_cache_path).encode('utf-8')))
def run(self, prompt):
prompt = bytes(PROMPT_TEXT_PREFIX + prompt + PROMPT_TEXT_POSTFIX, 'utf-8')
self.rkllm_run(self.handle, prompt, ctypes.byref(userdata))
rkllm_lora_params = None
if self.lora_model_name:
rkllm_lora_params = RKLLMLoraParam()
rkllm_lora_params.lora_adapter_name = ctypes.c_char_p((self.lora_model_name).encode('utf-8'))
rkllm_infer_params = RKLLMInferParam()
ctypes.memset(ctypes.byref(rkllm_infer_params), 0, ctypes.sizeof(RKLLMInferParam))
rkllm_infer_params.mode = RKLLMInferMode.RKLLM_INFER_GENERATE
rkllm_infer_params.lora_params = ctypes.byref(rkllm_lora_params) if rkllm_lora_params else None
rkllm_input = RKLLMInput()
rkllm_input.input_mode = RKLLMInputMode.RKLLM_INPUT_PROMPT
rkllm_input.input_data.prompt_input = ctypes.c_char_p((PROMPT_TEXT_PREFIX + prompt + PROMPT_TEXT_POSTFIX).encode('utf-8'))
self.rkllm_run(self.handle, ctypes.byref(rkllm_input), ctypes.byref(rkllm_infer_params), None)
return
def release(self):
@@ -143,62 +292,81 @@ class RKLLM(object):
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument('--target_platform', help='目标平台: 如rk3588/rk3576;')
parser.add_argument('--rkllm_model_path', help='Linux板端上已转换好的rkllm模型的绝对路径')
parser.add_argument('--rkllm_model_path', type=str, required=True, help='Absolute path of the converted RKLLM model on the Linux board;')
parser.add_argument('--target_platform', type=str, required=True, help='Target platform: e.g., rk3588/rk3576;')
parser.add_argument('--num_npu_core', type=int, required=True, help='Target num_npu_core;')
parser.add_argument('--lora_model_path', type=str, help='Absolute path of the lora_model on the Linux board;')
parser.add_argument('--prompt_cache_path', type=str, help='Absolute path of the prompt_cache file on the Linux board;')
args = parser.parse_args()
if not (args.target_platform in ["rk3588", "rk3576"]):
print("====== Error: 请指定正确的目标平台: rk3588/rk3576 ======")
sys.stdout.flush()
exit()
if not os.path.exists(args.rkllm_model_path):
print("====== Error: 请给出准确的rkllm模型路径,需注意是板端的绝对路径 ======")
print("Error: Please provide the correct rkllm model path, and ensure it is the absolute path on the board.")
sys.stdout.flush()
exit()
# 定频设置
if not (args.target_platform in ["rk3588", "rk3576"]):
print("Error: Please specify the correct target platform: rk3588/rk3576.")
sys.stdout.flush()
exit()
if not (args.num_npu_core in [1, 2, 3]):
print("Error: rk3576 supports 1/2 cores, rk3588 supports 1/2/3 cores, please specify the correct number of cores.")
sys.stdout.flush()
exit()
if args.lora_model_path:
if not os.path.exists(args.lora_model_path):
print("Error: Please provide the correct lora_model path, and advise it is the absolute path on the board.")
sys.stdout.flush()
exit()
if args.prompt_cache_path:
if not os.path.exists(args.prompt_cache_path):
print("Error: Please provide the correct prompt_cache_file path, and advise it is the absolute path on the board.")
sys.stdout.flush()
exit()
# Fix frequency
command = "sudo bash fix_freq_{}.sh".format(args.target_platform)
subprocess.run(command, shell=True)
# 设置文件描述符限制
# Set resource limit
resource.setrlimit(resource.RLIMIT_NOFILE, (102400, 102400))
# 初始化RKLLM模型
# Initialize RKLLM model
print("=========init....===========")
sys.stdout.flush()
target_platform = args.target_platform
model_path = args.rkllm_model_path
rkllm_model = RKLLM(model_path, target_platform)
print("RKLLM初始化成功!")
num_npu_core = args.num_npu_core
rkllm_model = RKLLM(model_path, num_npu_core, args.lora_model_path, args.prompt_cache_path)
print("RKLLM Model has been initialized successfully!")
print("==============================")
sys.stdout.flush()
# 创建一个函数用于接受用户使用 request 发送的数据
# Create a function to receive data sent by the user using a request
@app.route('/rkllm_chat', methods=['POST'])
def receive_message():
# 链接全局变量,获取回调函数的输出信息
# Link global variables to retrieve the output information from the callback function
global global_text, global_state
global is_blocking
# 如果服务器正在阻塞状态,则返回特定响应
# If the server is in a blocking state, return a specific response.
if is_blocking or global_state==0:
return jsonify({'status': 'error', 'message': 'RKLLM_Server is busy! Maybe you can try again later.'}), 503
# 加锁
lock.acquire()
try:
# 设置服务器为阻塞状态
# Set the server to a blocking state.
is_blocking = True
# 获取 POST 请求中的 JSON 数据
# Get JSON data from the POST request.
data = request.json
if data and 'messages' in data:
# 重置全局变量
# Reset global variables.
global_text = []
global_state = -1
# 定义返回的结构体
# Define the structure for the returned response.
rkllm_responses = {
"id": "rkllm_chat",
"object": "rkllm_chat",
@@ -212,18 +380,18 @@ if __name__ == "__main__":
}
if not "stream" in data.keys() or data["stream"] == False:
# 在这里处理收到的数据
# Process the received data here.
messages = data['messages']
print("Received messages:", messages)
for index, message in enumerate(messages):
input_prompt = message['content']
rkllm_output = ""
# 创建模型推理的线程
# Create a thread for model inference.
model_thread = threading.Thread(target=rkllm_model.run, args=(input_prompt,))
model_thread.start()
# 等待模型运行完成,定时检查模型的推理线程
# Wait for the model to finish running and periodically check the inference thread of the model.
model_thread_finished = False
while not model_thread_finished:
while len(global_text) > 0:
@@ -245,7 +413,6 @@ if __name__ == "__main__":
)
return jsonify(rkllm_responses), 200
else:
# 在这里处理收到的数据
messages = data['messages']
print("Received messages:", messages)
for index, message in enumerate(messages):
@@ -253,11 +420,9 @@ if __name__ == "__main__":
rkllm_output = ""
def generate():
# 创建模型推理的线程
model_thread = threading.Thread(target=rkllm_model.run, args=(input_prompt,))
model_thread.start()
# 等待模型运行完成,定时检查模型的推理线程
model_thread_finished = False
while not model_thread_finished:
while len(global_text) > 0:
@@ -282,16 +447,14 @@ if __name__ == "__main__":
else:
return jsonify({'status': 'error', 'message': 'Invalid JSON data!'}), 400
finally:
# 释放锁
lock.release()
# 将服务器状态设置为非阻塞
is_blocking = False
# 启动 Flask 应用程序
# Start the Flask application.
# app.run(host='0.0.0.0', port=8080)
app.run(host='0.0.0.0', port=8080, threaded=True, debug=False)
print("====================")
print("RKLLM模型推理结束, 释放RKLLM模型资源...")
print("RKLLM model inference completed, releasing RKLLM model resources...")
rkllm_model.release()
print("====================")

View File

@@ -8,65 +8,45 @@ import time
import gradio as gr
import argparse
# 设定环境变量
PROMPT_TEXT_PREFIX = "<|im_start|>system You are a helpful assistant. <|im_end|> <|im_start|>user"
PROMPT_TEXT_POSTFIX = "<|im_end|><|im_start|>assistant"
# Set environment variables
os.environ["GRADIO_SERVER_NAME"] = "0.0.0.0"
os.environ["GRADIO_SERVER_PORT"] = "8080"
# 设置动态库路径
# Set the dynamic library path
rkllm_lib = ctypes.CDLL('lib/librkllmrt.so')
# 定义全局变量,用于保存回调函数的输出,便于在gradio界面中输出
global_text = []
global_state = -1
split_byte_data = bytes(b"") # 用于保存分割的字节数据
# Define the structures from the library
RKLLM_Handle_t = ctypes.c_void_p
userdata = ctypes.c_void_p(None)
# 定义动态库中的结构体
class Token(ctypes.Structure):
_fields_ = [
("logprob", ctypes.c_float),
("id", ctypes.c_int32)
]
LLMCallState = ctypes.c_int
LLMCallState.RKLLM_RUN_NORMAL = 0
LLMCallState.RKLLM_RUN_FINISH = 1
LLMCallState.RKLLM_RUN_ERROR = 2
LLMCallState.RKLLM_RUN_GET_LAST_HIDDEN_LAYER = 3
class RKLLMResult(ctypes.Structure):
_fields_ = [
("text", ctypes.c_char_p),
("tokens", ctypes.POINTER(Token)),
("num", ctypes.c_int32)
]
RKLLMInputMode = ctypes.c_int
RKLLMInputMode.RKLLM_INPUT_PROMPT = 0
RKLLMInputMode.RKLLM_INPUT_TOKEN = 1
RKLLMInputMode.RKLLM_INPUT_EMBED = 2
RKLLMInputMode.RKLLM_INPUT_MULTIMODAL = 3
# 定义回调函数
def callback(result, userdata, state):
global global_text, global_state, split_byte_data
if state == 0:
# 保存输出的token文本及RKLLM运行状态
global_state = state
# 需要监控当前的字节数据是否完整,不完整则进行记录,后续进行解析
try:
global_text.append((split_byte_data + result.contents.text).decode('utf-8'))
print((split_byte_data + result.contents.text).decode('utf-8'), end='')
split_byte_data = bytes(b"")
except:
split_byte_data += result.contents.text
sys.stdout.flush()
elif state == 1:
# 保存RKLLM运行状态
global_state = state
print("\n")
sys.stdout.flush()
else:
print("run error")
RKLLMInferMode = ctypes.c_int
RKLLMInferMode.RKLLM_INFER_GENERATE = 0
RKLLMInferMode.RKLLM_INFER_GET_LAST_HIDDEN_LAYER = 1
# Python端与C++端的回调函数连接
callback_type = ctypes.CFUNCTYPE(None, ctypes.POINTER(RKLLMResult), ctypes.c_void_p, ctypes.c_int)
c_callback = callback_type(callback)
# 定义动态库中的结构体
class RKNNllmParam(ctypes.Structure):
class RKLLMParam(ctypes.Structure):
_fields_ = [
("model_path", ctypes.c_char_p),
("license_path", ctypes.c_char_p),
("num_npu_core", ctypes.c_int32),
("max_context_len", ctypes.c_int32),
("n_prefill_batch", ctypes.c_int32),
("max_new_tokens", ctypes.c_int32),
("skip_special_token", ctypes.c_bool),
("top_k", ctypes.c_int32),
("top_p", ctypes.c_float),
("temperature", ctypes.c_float),
@@ -78,58 +58,227 @@ class RKNNllmParam(ctypes.Structure):
("mirostat_eta", ctypes.c_float),
("logprobs", ctypes.c_bool),
("top_logprobs", ctypes.c_int32),
("use_gpu", ctypes.c_bool)
("is_async", ctypes.c_bool),
("img_start", ctypes.c_char_p),
("img_end", ctypes.c_char_p),
("img_content", ctypes.c_char_p),
]
# 定义RKLLM_Handle_t和userdata
RKLLM_Handle_t = ctypes.c_void_p
userdata = ctypes.c_void_p(None)
class RKLLMLoraAdapter(ctypes.Structure):
_fields_ = [
("lora_adapter_path", ctypes.c_char_p),
("lora_adapter_name", ctypes.c_char_p),
("scale", ctypes.c_float)
]
# 设置提示文本
PROMPT_TEXT_PREFIX = "<|im_start|>system You are a helpful assistant. <|im_end|> <|im_start|>user"
PROMPT_TEXT_POSTFIX = "<|im_end|><|im_start|>assistant"
class RKLLMLoraParam(ctypes.Structure):
_fields_ = [
("lora_adapter_name", ctypes.c_char_p)
]
# 定义Python端的RKLLM类,其中包括了对动态库中RKLLM模型的初始化、推理及释放操作
class RKLLMPromptCacheParam(ctypes.Structure):
_fields_ = [
("save_prompt_cache", ctypes.c_int),
("prompt_cache_path", ctypes.c_char_p)
]
class RKLLMEmbedInput(ctypes.Structure):
_fields_ = [
("embed", ctypes.POINTER(ctypes.c_float)),
("n_tokens", ctypes.c_size_t)
]
class RKLLMTokenInput(ctypes.Structure):
_fields_ = [
("input_ids", ctypes.POINTER(ctypes.c_int32)),
("n_tokens", ctypes.c_size_t)
]
class RKLLMMultiModelInput(ctypes.Structure):
_fields_ = [
("prompt", ctypes.c_char_p),
("image_embed", ctypes.POINTER(ctypes.c_float)),
("n_image_tokens", ctypes.c_size_t)
]
class RKLLMInputUnion(ctypes.Union):
_fields_ = [
("prompt_input", ctypes.c_char_p),
("embed_input", RKLLMEmbedInput),
("token_input", RKLLMTokenInput),
("multimodal_input", RKLLMMultiModelInput)
]
class RKLLMInput(ctypes.Structure):
_fields_ = [
("input_mode", ctypes.c_int),
("input_data", RKLLMInputUnion)
]
class RKLLMInferParam(ctypes.Structure):
_fields_ = [
("mode", RKLLMInferMode),
("lora_params", ctypes.POINTER(RKLLMLoraParam)),
("prompt_cache_params", ctypes.POINTER(RKLLMPromptCacheParam))
]
class RKLLMResultLastHiddenLayer(ctypes.Structure):
_fields_ = [
("hidden_states", ctypes.POINTER(ctypes.c_float)),
("embd_size", ctypes.c_int),
("num_tokens", ctypes.c_int)
]
class RKLLMResult(ctypes.Structure):
_fields_ = [
("text", ctypes.c_char_p),
("size", ctypes.c_int),
("last_hidden_layer", RKLLMResultLastHiddenLayer)
]
# Define global variables to store the callback function output for displaying in the Gradio interface
global_text = []
global_state = -1
split_byte_data = bytes(b"") # Used to store the segmented byte data
# Define the callback function
def callback_impl(result, userdata, state):
global global_text, global_state, split_byte_data
if state == LLMCallState.RKLLM_RUN_FINISH:
global_state = state
print("\n")
sys.stdout.flush()
elif state == LLMCallState.RKLLM_RUN_ERROR:
global_state = state
print("run error")
sys.stdout.flush()
elif state == LLMCallState.RKLLM_RUN_GET_LAST_HIDDEN_LAYER:
'''
If using the GET_LAST_HIDDEN_LAYER function, the callback interface will return the memory pointer: last_hidden_layer, the number of tokens: num_tokens, and the size of the hidden layer: embd_size.
With these three parameters, you can retrieve the data from last_hidden_layer.
Note: The data needs to be retrieved during the current callback; if not obtained in time, the pointer will be released by the next callback.
'''
if result.last_hidden_layer.embd_size != 0 and result.last_hidden_layer.num_tokens != 0:
data_size = result.last_hidden_layer.embd_size * result.last_hidden_layer.num_tokens * ctypes.sizeof(ctypes.c_float)
print(f"data_size: {data_size}")
global_text.append(f"data_size: {data_size}\n")
output_path = os.getcwd() + "/last_hidden_layer.bin"
with open(output_path, "wb") as outFile:
data = ctypes.cast(result.last_hidden_layer.hidden_states, ctypes.POINTER(ctypes.c_float))
float_array_type = ctypes.c_float * (data_size // ctypes.sizeof(ctypes.c_float))
float_array = float_array_type.from_address(ctypes.addressof(data.contents))
outFile.write(bytearray(float_array))
print(f"Data saved to {output_path} successfully!")
global_text.append(f"Data saved to {output_path} successfully!")
else:
print("Invalid hidden layer data.")
global_text.append("Invalid hidden layer data.")
global_state = state
time.sleep(0.05) # Delay for 0.05 seconds to wait for the output result
sys.stdout.flush()
else:
# Save the output token text and the RKLLM running state
global_state = state
# Monitor if the current byte data is complete; if incomplete, record it for later parsing
try:
global_text.append((split_byte_data + result.contents.text).decode('utf-8'))
print((split_byte_data + result.contents.text).decode('utf-8'), end='')
split_byte_data = bytes(b"")
except:
split_byte_data += result.contents.text
sys.stdout.flush()
# Connect the callback function between the Python side and the C++ side
callback_type = ctypes.CFUNCTYPE(None, ctypes.POINTER(RKLLMResult), ctypes.c_void_p, ctypes.c_int)
callback = callback_type(callback_impl)
# Define the RKLLM class, which includes initialization, inference, and release operations for the RKLLM model in the dynamic library
class RKLLM(object):
def __init__(self, model_path, target_platform):
rknnllm_param = RKNNllmParam()
rknnllm_param.model_path = bytes(model_path, 'utf-8')
if target_platform == "rk3588":
rknnllm_param.num_npu_core = 3
elif target_platform == "rk3576":
rknnllm_param.num_npu_core = 2
rknnllm_param.max_context_len = 320
rknnllm_param.max_new_tokens = 512
rknnllm_param.top_k = 1
rknnllm_param.top_p = 0.9
rknnllm_param.temperature = 0.8
rknnllm_param.repeat_penalty = 1.1
rknnllm_param.frequency_penalty = 0.0
rknnllm_param.presence_penalty = 0.0
rknnllm_param.mirostat = 0
rknnllm_param.mirostat_tau = 5.0
rknnllm_param.mirostat_eta = 0.1
rknnllm_param.logprobs = False
rknnllm_param.top_logprobs = 5
rknnllm_param.use_gpu = True
def __init__(self, model_path, num_npu_core, lora_model_path = None, prompt_cache_path = None):
rkllm_param = RKLLMParam()
rkllm_param.model_path = bytes(model_path, 'utf-8')
rkllm_param.license_path = None
rkllm_param.num_npu_core = num_npu_core
rkllm_param.max_context_len = 512
rkllm_param.n_prefill_batch = 512
rkllm_param.max_new_tokens = -1
rkllm_param.skip_special_token = True
rkllm_param.top_k = 1
rkllm_param.top_p = 0.9
rkllm_param.temperature = 0.8
rkllm_param.repeat_penalty = 1.1
rkllm_param.frequency_penalty = 0.0
rkllm_param.presence_penalty = 0.0
rkllm_param.mirostat = 0
rkllm_param.mirostat_tau = 5.0
rkllm_param.mirostat_eta = 0.1
rkllm_param.logprobs = False
rkllm_param.top_logprobs = 5
rkllm_param.is_async = False
rkllm_param.img_start = "".encode('utf-8')
rkllm_param.img_end = "".encode('utf-8')
rkllm_param.img_content = "".encode('utf-8')
self.handle = RKLLM_Handle_t()
self.rkllm_init = rkllm_lib.rkllm_init
self.rkllm_init.argtypes = [ctypes.POINTER(RKLLM_Handle_t), ctypes.POINTER(RKNNllmParam), callback_type]
self.rkllm_init.argtypes = [ctypes.POINTER(RKLLM_Handle_t), RKLLMParam, callback_type]
self.rkllm_init.restype = ctypes.c_int
self.rkllm_init(ctypes.byref(self.handle), rknnllm_param, c_callback)
self.rkllm_init(ctypes.byref(self.handle), rkllm_param, callback)
self.rkllm_run = rkllm_lib.rkllm_run
self.rkllm_run.argtypes = [RKLLM_Handle_t, ctypes.POINTER(ctypes.c_char), ctypes.c_void_p]
self.rkllm_run.argtypes = [RKLLM_Handle_t, ctypes.POINTER(RKLLMInput), ctypes.POINTER(RKLLMInferParam), ctypes.c_void_p]
self.rkllm_run.restype = ctypes.c_int
self.rkllm_destroy = rkllm_lib.rkllm_destroy
self.rkllm_destroy.argtypes = [RKLLM_Handle_t]
self.rkllm_destroy.restype = ctypes.c_int
self.lora_adapter_path = None
self.lora_model_name = None
if lora_model_path:
self.lora_adapter_path = lora_model_path
self.lora_adapter_name = "test"
lora_adapter = RKLLMLoraAdapter()
ctypes.memset(ctypes.byref(lora_adapter), 0, ctypes.sizeof(RKLLMLoraAdapter))
lora_adapter.lora_adapter_path = ctypes.c_char_p((self.lora_adapter_path).encode('utf-8'))
lora_adapter.lora_adapter_name = ctypes.c_char_p((self.lora_adapter_name).encode('utf-8'))
lora_adapter.scale = 1.0
rkllm_load_lora = rkllm_lib.rkllm_load_lora
rkllm_load_lora.argtypes = [RKLLM_Handle_t, ctypes.POINTER(RKLLMLoraAdapter)]
rkllm_load_lora.restype = ctypes.c_int
rkllm_load_lora(self.handle, ctypes.byref(lora_adapter))
self.prompt_cache_path = None
if prompt_cache_path:
self.prompt_cache_path = prompt_cache_path
rkllm_load_prompt_cache = rkllm_lib.rkllm_load_prompt_cache
rkllm_load_prompt_cache.argtypes = [RKLLM_Handle_t, ctypes.c_char_p]
rkllm_load_prompt_cache.restype = ctypes.c_int
rkllm_load_prompt_cache(self.handle, ctypes.c_char_p((prompt_cache_path).encode('utf-8')))
def run(self, prompt):
prompt = bytes(PROMPT_TEXT_PREFIX + prompt + PROMPT_TEXT_POSTFIX, 'utf-8')
self.rkllm_run(self.handle, prompt, ctypes.byref(userdata))
rkllm_lora_params = None
if self.lora_model_name:
rkllm_lora_params = RKLLMLoraParam()
rkllm_lora_params.lora_adapter_name = ctypes.c_char_p((self.lora_model_name).encode('utf-8'))
rkllm_infer_params = RKLLMInferParam()
ctypes.memset(ctypes.byref(rkllm_infer_params), 0, ctypes.sizeof(RKLLMInferParam))
rkllm_infer_params.mode = RKLLMInferMode.RKLLM_INFER_GENERATE
rkllm_infer_params.lora_params = ctypes.byref(rkllm_lora_params) if rkllm_lora_params else None
rkllm_input = RKLLMInput()
rkllm_input.input_mode = RKLLMInputMode.RKLLM_INPUT_PROMPT
rkllm_input.input_data.prompt_input = ctypes.c_char_p((PROMPT_TEXT_PREFIX + prompt + PROMPT_TEXT_POSTFIX).encode('utf-8'))
self.rkllm_run(self.handle, ctypes.byref(rkllm_input), ctypes.byref(rkllm_infer_params), None)
return
def release(self):
@@ -137,92 +286,112 @@ class RKLLM(object):
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument('--target_platform', help='目标平台: 如rk3588/rk3576;')
parser.add_argument('--rkllm_model_path', help='Linux板端上已转换好的rkllm模型的绝对路径')
parser.add_argument('--rkllm_model_path', type=str, required=True, help='Absolute path of the converted RKLLM model on the Linux board;')
parser.add_argument('--target_platform', type=str, required=True, help='Target platform: e.g., rk3588/rk3576;')
parser.add_argument('--num_npu_core', type=int, required=True, help='Target num_npu_core;')
parser.add_argument('--lora_model_path', type=str, help='Absolute path of the lora_model on the Linux board;')
parser.add_argument('--prompt_cache_path', type=str, help='Absolute path of the prompt_cache file on the Linux board;')
args = parser.parse_args()
if not (args.target_platform in ["rk3588", "rk3576"]):
print("====== Error: 请指定正确的目标平台: rk3588/rk3576 ======")
sys.stdout.flush()
exit()
if not os.path.exists(args.rkllm_model_path):
print("====== Error: 请给出准确的rkllm模型路径,需注意是板端的绝对路径 ======")
print("Error: Please provide the correct rkllm model path, and ensure it is the absolute path on the board.")
sys.stdout.flush()
exit()
# 定频设置
if not (args.target_platform in ["rk3588", "rk3576"]):
print("Error: Please specify the correct target platform: rk3588/rk3576.")
sys.stdout.flush()
exit()
if not (args.num_npu_core in [1, 2, 3]):
print("Error: rk3576 supports 1/2 cores, rk3588 supports 1/2/3 cores, please specify the correct number of cores.")
sys.stdout.flush()
exit()
if args.lora_model_path:
if not os.path.exists(args.lora_model_path):
print("Error: Please provide the correct lora_model path, and advise it is the absolute path on the board.")
sys.stdout.flush()
exit()
if args.prompt_cache_path:
if not os.path.exists(args.prompt_cache_path):
print("Error: Please provide the correct prompt_cache_file path, and advise it is the absolute path on the board.")
sys.stdout.flush()
exit()
# Fix frequency
command = "sudo bash fix_freq_{}.sh".format(args.target_platform)
subprocess.run(command, shell=True)
# 设置文件描述符限制
# Set resource limit
resource.setrlimit(resource.RLIMIT_NOFILE, (102400, 102400))
# 初始化RKLLM模型
# Initialize RKLLM model
print("=========init....===========")
sys.stdout.flush()
target_platform = args.target_platform
model_path = args.rkllm_model_path
rkllm_model = RKLLM(model_path, target_platform)
print("RKLLM初始化成功!")
num_npu_core = args.num_npu_core
rkllm_model = RKLLM(model_path, num_npu_core, args.lora_model_path, args.prompt_cache_path)
print("RKLLM Model has been initialized successfully!")
print("==============================")
sys.stdout.flush()
# 记录用户输入的prompt
# Record the user's input prompt
def get_user_input(user_message, history):
history = history + [[user_message, None]]
return "", history
# 获取RKLLM模型的输出并进行流式打印
# Retrieve the output from the RKLLM model and print it in a streaming manner
def get_RKLLM_output(history):
# 链接全局变量,获取回调函数的输出信息
# Link global variables to retrieve the output information from the callback function
global global_text, global_state
global_text = []
global_state = -1
# 创建模型推理的线程
# Create a thread for model inference
model_thread = threading.Thread(target=rkllm_model.run, args=(history[-1][0],))
model_thread.start()
# history[-1][1]表示当前的输出对话
# history[-1][1] represents the current dialogue
history[-1][1] = ""
# 等待模型运行完成,定时检查模型的推理线程
# Wait for the model to finish running and periodically check the inference thread of the model
model_thread_finished = False
while not model_thread_finished:
while len(global_text) > 0:
history[-1][1] += global_text.pop(0)
time.sleep(0.005)
# gradio在调用then方法式自动将yield返回的结果推进行输出
# Gradio automatically pushes the result returned by the yield statement when calling the then method
yield history
model_thread.join(timeout=0.005)
model_thread_finished = not model_thread.is_alive()
# 创建gradio界面
# Create a Gradio interface
with gr.Blocks(title="Chat with RKLLM") as chatRKLLM:
gr.Markdown("<div align='center'><font size='70'> Chat with RKLLM </font></div>")
gr.Markdown("### 在 inputTextBox 输入您的问题,按下 Enter 键,即可与 RKLLM 模型进行对话。")
# 创建一个Chatbot组件,用于显示对话历史
gr.Markdown("### Enter your question in the inputTextBox and press the Enter key to chat with the RKLLM model.")
# Create a Chatbot component to display conversation history
rkllmServer = gr.Chatbot(height=600)
# 创建一个Textbox组件,让用户输入消息
# Create a Textbox component for user message input
msg = gr.Textbox(placeholder="Please input your question here...", label="inputTextBox")
# 创建一个Button组件,用于清除聊天历史
clear = gr.Button("清除")
# Create a Button component to clear the chat history.
clear = gr.Button("Clear")
# 将用户输入的消息提交给get_user_input函数,并且立即更新聊天历史
# 然后调用get_RKLLM_output函数,进一步更新聊天历史
# queue=False参数确保这些更新不会被放入队列,而是立即执行
# Submit the user's input message to the get_user_input function and immediately update the chat history.
# Then call the get_RKLLM_output function to further update the chat history.
# The queue=False parameter ensures that these updates are not queued, but executed immediately.
msg.submit(get_user_input, [msg, rkllmServer], [msg, rkllmServer], queue=False).then(get_RKLLM_output, rkllmServer, rkllmServer)
# 当点击清除按钮时,执行一个空操作(lambda: None),并且立即清除聊天历史
# When the clear button is clicked, perform a no-operation (lambda: None) and immediately clear the chat history.
clear.click(lambda: None, None, rkllmServer, queue=False)
# 启用事件队列系统
# Enable the event queue system.
chatRKLLM.queue()
# 启动Gradio应用程序
# Start the Gradio application.
chatRKLLM.launch()
print("====================")
print("RKLLM模型推理结束, 释放RKLLM模型资源...")
print("RKLLM model inference completed, releasing RKLLM model resources...")
rkllm_model.release()
print("====================")