diff --git a/CHANGELOG.md b/CHANGELOG.md index 00369d9..cdd69d2 100755 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,4 +1,17 @@ # CHANGELOG +## v1.1.0 +- Support group-wise quantization (w4a16 group sizes of 32/64/128, w8a8 group sizes of 128/256/512). +- Support joint inference with LoRA model loading +- Support storage and preloading of prompt cache. +- Support gguf model conversion (currently only support q4_0 and fp16). +- Optimize initialization, prefill, and decode time. +- Support four input types: prompt, embedding, token, and multimodal. +- Add PC-based simulation accuracy testing and inference interface support for rkllm-toolkit. +- Add gdq algorithm to improve 4-bit quantization accuracy. +- Add mixed quantization algorithm, supporting a combination of grouped and non-grouped quantization based on specified ratios. +- Add support for models such as Llama3, Gemma2, and MiniCPM3. +- Resolve catastrophic forgetting issue when the number of tokens exceeds max_context. + ## v1.0.1 - Optimize model conversion memory occupation - Optimize inference memory occupation @@ -11,7 +24,7 @@ - Add logprob and token_id to the return value ## v1.0.0 - - Supports the conversion and deployment of LLM models on RK3588/RK3576 platforms + - Support the conversion and deployment of LLM models on RK3588/RK3576 platforms - Compatible with Hugging Face model architectures - - Currently supports the models Llama, Qwen, Qwen2, and Phi-2 - - Supports quantization with w8a8 and w4a16 precision \ No newline at end of file + - Currently support the models Llama, Qwen, Qwen2, and Phi-2 + - Support quantization with w8a8 and w4a16 precision \ No newline at end of file diff --git a/README.md b/README.md old mode 100644 new mode 100755 index aeb6ebf..2e6f57e --- a/README.md +++ b/README.md @@ -66,6 +66,7 @@ There are dedicated branch containing the latest commit done by this fork before
# Description + RKLLM software stack can help users to quickly deploy AI models to Rockchip chips. The overall framework is as follows:
@@ -81,38 +82,97 @@ There are dedicated branch containing the latest commit done by this fork before - RKNPU kernel driver is responsible for interacting with NPU hardware. It has been open source and can be found in the Rockchip kernel code. # Support Platform - - RK3588 Series - - RK3576 Series + +- RK3588 Series +- RK3576 Series # Support Models - - [X] [TinyLLAMA 1.1B](https://huggingface.co/TinyLlama/TinyLlama-1.1B-Chat-v1.0/tree/fe8a4ea1ffedaf415f4da2f062534de366a451e6) - - [X] [Qwen 1.8B](https://huggingface.co/Qwen/Qwen-1_8B-Chat/tree/1d0f68de57b88cfde81f3c3e537f24464d889081) - - [X] [Qwen2 0.5B](https://huggingface.co/Qwen/Qwen1.5-0.5B/tree/8f445e3628f3500ee69f24e1303c9f10f5342a39) - - [X] [Phi-2 2.7B](https://hf-mirror.com/microsoft/phi-2/tree/834565c23f9b28b96ccbeabe614dd906b6db551a) - - [X] [Phi-3 3.8B](https://huggingface.co/microsoft/Phi-3-mini-4k-instruct/tree/291e9e30e38030c23497afa30f3af1f104837aa6) - - [X] [ChatGLM3 6B](https://huggingface.co/THUDM/chatglm3-6b/tree/103caa40027ebfd8450289ca2f278eac4ff26405) - - [X] [Gemma 2B](https://huggingface.co/google/gemma-2b-it/tree/de144fb2268dee1066f515465df532c05e699d48) - - [X] [InternLM2 1.8B](https://huggingface.co/internlm/internlm2-chat-1_8b/tree/ecccbb5c87079ad84e5788baa55dd6e21a9c614d) - - [X] [MiniCPM 2B](https://huggingface.co/openbmb/MiniCPM-2B-sft-bf16/tree/79fbb1db171e6d8bf77cdb0a94076a43003abd9e) + +- [x] [LLAMA models](https://huggingface.co/meta-llama) +- [x] [TinyLLAMA models](https://huggingface.co/TinyLlama) +- [x] [Qwen models](https://huggingface.co/models?search=Qwen/Qwen) +- [x] [Phi models](https://huggingface.co/models?search=microsoft/phi) +- [x] [ChatGLM3-6B](https://huggingface.co/THUDM/chatglm3-6b/tree/103caa40027ebfd8450289ca2f278eac4ff26405) +- [x] [Gemma models](https://huggingface.co/collections/google/gemma-2-release-667d6600fd5220e7b967f315) +- [x] [InternLM2 models](https://huggingface.co/collections/internlm/internlm2-65b0ce04970888799707893c) +- [x] [MiniCPM models](https://huggingface.co/collections/openbmb/minicpm-65d48bf958302b9fd25b698f) + +# Model Performance Benchmark + +| model | dtype | seqlen | max_context | new_tokens | TTFT(ms) | Tokens/s | memory(G) | platform | +|:-------------- |:---------- |:------:|:-----------:|:----------:|:--------:|:--------:|:---------:|:--------:| +| TinyLLAMA-1.1B | w4a16 | 64 | 320 | 256 | 345.00 | 21.10 | 0.77 | RK3576 | +| | w4a16_g128 | 64 | 320 | 256 | 410.00 | 18.50 | 0.8 | RK3576 | +| | w8a8 | 64 | 320 | 256 | 140.46 | 24.21 | 1.25 | RK3588 | +| | w8a8_g512 | 64 | 320 | 256 | 195.00 | 20.08 | 1.29 | RK3588 | +| Qwen2-1.5B | w4a16 | 64 | 320 | 256 | 512.00 | 14.40 | 1.75 | RK3576 | +| | w4a16_g128 | 64 | 320 | 256 | 550.00 | 12.75 | 1.76 | RK3576 | +| | w8a8 | 64 | 320 | 256 | 206.00 | 16.46 | 2.47 | RK3588 | +| | w8a8_g128 | 64 | 320 | 256 | 725.00 | 7.00 | 2.65 | RK3588 | +| Phi-3-3.8B | w4a16 | 64 | 320 | 256 | 975.00 | 6.60 | 2.16 | RK3576 | +| | w4a16_g128 | 64 | 320 | 256 | 1180.00 | 5.85 | 2.23 | RK3576 | +| | w8a8 | 64 | 320 | 256 | 516.00 | 7.44 | 3.88 | RK3588 | +| | w8a8_g512 | 64 | 320 | 256 | 610.00 | 6.13 | 3.95 | RK3588 | +| ChatGLM3-6B | w4a16 | 64 | 320 | 256 | 1168.00 | 4.62 | 3.86 | RK3576 | +| | w4a16_g128 | 64 | 320 | 256 | 1582.56 | 3.82 | 3.96 | RK3576 | +| | w8a8 | 64 | 320 | 256 | 800.00 | 4.95 | 6.69 | RK3588 | +| | w8a8_g128 | 64 | 320 | 256 | 2190.00 | 2.70 | 7.18 | RK3588 | +| Gemma2-2B | w4a16 | 64 | 320 | 256 | 628.00 | 8.00 | 3.63 | RK3576 | +| | w4a16_g128 | 64 | 320 | 256 | 776.20 | 7.40 | 3.63 | RK3576 | +| | w8a8 | 64 | 320 | 256 | 342.29 | 9.67 | 4.84 | RK3588 | +| | w8a8_g128 | 64 | 320 | 256 | 1055.00 | 5.49 | 5.14 | RK3588 | +| InternLM2-1.8B | w4a16 | 64 | 320 | 256 | 475.00 | 13.30 | 1.59 | RK3576 | +| | w4a16_g128 | 64 | 320 | 256 | 572.00 | 11.95 | 1.62 | RK3576 | +| | w8a8 | 64 | 320 | 256 | 205.97 | 15.66 | 2.38 | RK3588 | +| | w8a8_g512 | 64 | 320 | 256 | 298.00 | 12.66 | 2.45 | RK3588 | +| MiniCPM3-4B | w4a16 | 64 | 320 | 256 | 1397.00 | 4.80 | 2.7 | RK3576 | +| | w4a16_g128 | 64 | 320 | 256 | 1645.00 | 4.39 | 2.8 | RK3576 | +| | w8a8 | 64 | 320 | 256 | 702.18 | 6.15 | 4.65 | RK3588 | +| | w8a8_g128 | 64 | 320 | 256 | 1691.00 | 3.42 | 5.06 | RK3588 | +| llama3-8B | w4a16 | 64 | 320 | 256 | 1607.98 | 3.60 | 5.63 | RK3576 | +| | w4a16_g128 | 64 | 320 | 256 | 2010.00 | 3.00 | 5.76 | RK3576 | +| | w8a8 | 64 | 320 | 256 | 1128.00 | 3.79 | 9.21 | RK3588 | +| | w8a8_g512 | 64 | 320 | 256 | 1281.35 | 3.05 | 9.45 | RK3588 | + +- This performance data were collected based on the maximum CPU and NPU frequencies of each platform with version 1.1.0. +- The script for setting the frequencies is located in the scripts directory. # Download -- You can also download all packages, docker image, examples, docs and platform-tools from [RKLLM_SDK](https://console.zbox.filez.com/l/RJJDmB), fetch code: rkllm + +You can download the latest package, docker image, example, documentation, and platform-tool from [RKLLM_SDK](https://console.zbox.filez.com/l/RJJDmB), fetch code: rkllm + +# Note + +- The modifications in version 1.1 are significant, making it incompatible with older version models. Please use the latest toolchain for model conversion and inference. + +- The supported Python versions are: + + - Python 3.8 + + - Python 3.10 + +- Latest version: [ v1.1.1](https://github.com/airockchip/rknn-llm/releases/tag/release-v1.1.1) # RKNN Toolkit2 + If you want to deploy additional AI model, we have introduced a SDK called RKNN-Toolkit2. For details, please refer to: https://github.com/airockchip/rknn-toolkit2 # CHANGELOG -## v1.0.1 - - Optimize model conversion memory occupation - - Optimize inference memory occupation - - Increase prefill speed - - Reduce initialization time - - Improve quantization accuracy - - Add support for Gemma, ChatGLM3, MiniCPM, InternLM2, and Phi-3 - - Add Server invocation - - Add inference interruption interface - - Add logprob and token_id to the return value -For older version, please refer [CHANGELOG](CHANGELOG.md) +## v1.1.0 + +- Support group-wise quantization (w4a16 group sizes of 32/64/128, w8a8 group sizes of 128/256/512). +- Support joint inference with LoRA model loading +- Support storage and preloading of prompt cache. +- Support gguf model conversion (currently only support q4_0 and fp16). +- Optimize initialization, prefill, and decode time. +- Support four input types: prompt, embedding, token, and multimodal. +- Add PC-based simulation accuracy testing and inference interface support for rkllm-toolkit. +- Add gdq algorithm to improve 4-bit quantization accuracy. +- Add mixed quantization algorithm, supporting a combination of grouped and non-grouped quantization based on specified ratios. +- Add support for models such as Llama3, Gemma2, and MiniCPM3. +- Resolve catastrophic forgetting issue when the number of tokens exceeds max_context. + +for older version, please refer [CHANGELOG](CHANGELOG.md) diff --git a/doc/Rockchip_RKLLM_SDK_CN.pdf b/doc/Rockchip_RKLLM_SDK_CN.pdf deleted file mode 100644 index 42ca2b2..0000000 Binary files a/doc/Rockchip_RKLLM_SDK_CN.pdf and /dev/null differ diff --git a/doc/Rockchip_RKLLM_SDK_CN_1.1.0.pdf b/doc/Rockchip_RKLLM_SDK_CN_1.1.0.pdf new file mode 100644 index 0000000..f58a812 Binary files /dev/null and b/doc/Rockchip_RKLLM_SDK_CN_1.1.0.pdf differ diff --git a/doc/Rockchip_RKLLM_SDK_EN.pdf b/doc/Rockchip_RKLLM_SDK_EN.pdf deleted file mode 100644 index 4ef3cbc..0000000 Binary files a/doc/Rockchip_RKLLM_SDK_EN.pdf and /dev/null differ diff --git a/doc/Rockchip_RKLLM_SDK_EN_1.1.0.pdf b/doc/Rockchip_RKLLM_SDK_EN_1.1.0.pdf new file mode 100644 index 0000000..78be45e Binary files /dev/null and b/doc/Rockchip_RKLLM_SDK_EN_1.1.0.pdf differ diff --git a/rkllm-runtime/examples/rkllm_api_demo/CMakeLists.txt b/rkllm-runtime/examples/rkllm_api_demo/CMakeLists.txt index 0c0ccad..3c971ad 100644 --- a/rkllm-runtime/examples/rkllm_api_demo/CMakeLists.txt +++ b/rkllm-runtime/examples/rkllm_api_demo/CMakeLists.txt @@ -1,21 +1,26 @@ cmake_minimum_required(VERSION 3.10) -project(llm_demo) +project(rkllm_demo) set(CMAKE_CXX_STANDARD 11) set(CMAKE_CXX_STANDARD_REQUIRED ON) -set(SOURCE_FILES src/main.cpp) +set(SOURCE_FILES_1 src/llm_demo.cpp) +add_executable(llm_demo ${SOURCE_FILES_1}) -add_executable(${PROJECT_NAME} ${SOURCE_FILES}) +set(SOURCE_FILES_2 src/multimodel_demo.cpp) +add_executable(multimodel_demo ${SOURCE_FILES_2}) set(RKLLM_API_PATH "${CMAKE_SOURCE_DIR}/../../runtime/${CMAKE_SYSTEM_NAME}/librkllm_api") include_directories(${RKLLM_API_PATH}/include) if(CMAKE_SYSTEM_NAME STREQUAL "Android") set(RKLLM_RT_LIB ${RKLLM_API_PATH}/${CMAKE_ANDROID_ARCH_ABI}/librkllmrt.so) - target_link_libraries(${PROJECT_NAME} ${RKLLM_RT_LIB} log) + find_package(OpenMP REQUIRED) + target_link_libraries(llm_demo ${RKLLM_RT_LIB} log OpenMP::OpenMP_CXX) + target_link_libraries(multimodel_demo ${RKLLM_RT_LIB} log OpenMP::OpenMP_CXX) elseif(CMAKE_SYSTEM_NAME STREQUAL "Linux") set(RKLLM_RT_LIB ${RKLLM_API_PATH}/aarch64/librkllmrt.so) - target_link_libraries(${PROJECT_NAME} ${RKLLM_RT_LIB}) + target_link_libraries(llm_demo ${RKLLM_RT_LIB}) + target_link_libraries(multimodel_demo ${RKLLM_RT_LIB}) endif() diff --git a/rkllm-runtime/examples/rkllm_api_demo/README.md b/rkllm-runtime/examples/rkllm_api_demo/README.md index b514c08..ff1e3db 100644 --- a/rkllm-runtime/examples/rkllm_api_demo/README.md +++ b/rkllm-runtime/examples/rkllm_api_demo/README.md @@ -10,7 +10,7 @@ Push the compiled `llm_demo` file and `librkllmrt.so` file to the device: ```bash cp build/build_linux_aarch64_Release/llm_demo /usr/bin -cp ../runtime/Linux/librkllm_api/aarch64/librkllmrt.so /usr/local/lib +cp ../../runtime/Linux/librkllm_api/aarch64/librkllmrt.so /usr/local/lib ``` To exit the LLM enter either `exit` or `quit`. diff --git a/rkllm-runtime/examples/rkllm_api_demo/src/llm_demo.cpp b/rkllm-runtime/examples/rkllm_api_demo/src/llm_demo.cpp new file mode 100644 index 0000000..b7c0bb6 --- /dev/null +++ b/rkllm-runtime/examples/rkllm_api_demo/src/llm_demo.cpp @@ -0,0 +1,194 @@ +// Copyright (c) 2024 by Rockchip Electronics Co., Ltd. All Rights Reserved. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +#include +#include +#include +#include "rkllm.h" +#include +#include +#include +#include + +#define PROMPT_TEXT_PREFIX "<|im_start|>system You are a helpful assistant. <|im_end|> <|im_start|>user" +#define PROMPT_TEXT_POSTFIX "<|im_end|><|im_start|>assistant" + + +using namespace std; +LLMHandle llmHandle = nullptr; + +void exit_handler(int signal) +{ + if (llmHandle != nullptr) + { + { + cout << "程序即将退出" << endl; + LLMHandle _tmp = llmHandle; + llmHandle = nullptr; + rkllm_destroy(_tmp); + } + } + exit(signal); +} + +void callback(RKLLMResult *result, void *userdata, LLMCallState state) +{ + if (state == RKLLM_RUN_FINISH) + { + printf("\n"); + } else if (state == RKLLM_RUN_ERROR) { + printf("\\run error\n"); + } else if (state == RKLLM_RUN_GET_LAST_HIDDEN_LAYER) { + /* ================================================================================================================ + 若使用GET_LAST_HIDDEN_LAYER功能,callback接口会回传内存指针:last_hidden_layer,token数量:num_tokens与隐藏层大小:embd_size + 通过这三个参数可以取得last_hidden_layer中的数据 + 注:需要在当前callback中获取,若未及时获取,下一次callback会将该指针释放 + ===============================================================================================================*/ + if (result->last_hidden_layer.embd_size != 0 && result->last_hidden_layer.num_tokens != 0) { + int data_size = result->last_hidden_layer.embd_size * result->last_hidden_layer.num_tokens * sizeof(float); + printf("\ndata_size:%d",data_size); + std::ofstream outFile("last_hidden_layer.bin", std::ios::binary); + if (outFile.is_open()) { + outFile.write(reinterpret_cast(result->last_hidden_layer.hidden_states), data_size); + outFile.close(); + std::cout << "Data saved to output.bin successfully!" << std::endl; + } else { + std::cerr << "Failed to open the file for writing!" << std::endl; + } + } + } else if (state == RKLLM_RUN_NORMAL) { + printf("%s", result->text); + // for(int i=0; inum; i++) + // { + // printf("%d token_id: %d logprob: %f\n", i, result->tokens[i].id, result->tokens[i].logprob); + // } + } +} + +int main(int argc, char **argv) +{ + if (argc < 4) { + std::cerr << "Usage: " << argv[0] << " model_path max_new_tokens max_context_len\n"; + return 1; + } + + signal(SIGINT, exit_handler); + printf("rkllm init start\n"); + + //设置参数及初始化 + RKLLMParam param = rkllm_createDefaultParam(); + param.model_path = argv[1]; + param.top_k = 1; + param.max_new_tokens = std::atoi(argv[2]); + param.max_context_len = std::atoi(argv[3]); + + param.skip_special_token = true; + param.extend_param.base_domain_id = 0; + int ret = rkllm_init(&llmHandle, ¶m, callback); + if (ret == 0){ + printf("rkllm init success\n"); + } else { + printf("rkllm init failed\n"); + exit_handler(-1); + } + + vector pre_input; + pre_input.push_back("Summarize the text in 50 words: And the number one lesson I could offer you where your work is concerned is this. Become so skilled, so vigilant, so flat, out, fantastic at what you do that your talent cannot be dismissed. When I was a kid, I thought I was going to be a social worker or be a teacher or somehow be in an environment where I would be connected and speaking to people to honor that calling. I had no idea that God could dream this big dream of television life for myself. But everybody has a calling. and your real job in life is to figure out as soon as possible what that is, who you were meant to be and begin to honor that in the best way possible for yourself. We have seen over the years on the Oprah Show so many people who've been able to rise to success in their life and only able to rise to success because they answered the call. Lots of people think that it's about being famous or about being known or about doing big, big, incredible things. What you're going to hear today and see through the stories is that sometimes the calling is right in your own neighborhood. Sometimes the calling is something that was just a whisper to you. And when you begin to honor that whisper and to follow that, you'd end up being the best that you could be. I actually thought at the time that being thin made me better. My identification with form, my wanting so desperately to be in a size 10 gene was so prominent identifying with that. That shortly after this, I gained five pounds. And I was invited to a party at the time by Don Johnson, a member of Dreamy Don Johnson. I was invited to a holiday party. And I did not go to that party because I thought the five pounds made me too fat, not good enough to be a Don Johnson's party. Sounds kind of sick. Now I know. But that's what the ego does. It is, it is sick. It's wily, it's cunning, it's deceptive. It's an imposter imposing on the real you, making you often think that you're something that you're not. You're not the shape of your body. You're not your status. You're not your position in life. You're not the car you drive, no matter how fancy it is. You're not your house or your square footage. For all of you watching right now, here's another way to better understand ego. What I'm talking about, Eckhart taught me this. Everybody try this from home or wherever you are right now. Close your eyes and notice what you're thinking. Notice your thoughts. Now, who is thinking those thoughts? Can you notice that there is a space where you are observing the thoughts and where you are aware of the thoughts that you are observing? Eckhart says, you are that awareness disguised as a person. And when I got that, I understood the difference between my true self and my ego self. Over the years, we've heard invaluable information that in a moment of danger could be life-saving. Nothing I heard made a greater impression on me than back in 1991. I think we were beginning our fifth season and man named Sanford Strong came on and shared this. You never forgot it. Rule number one. And frankly, it's probably in my opinion the most important. Never allow them to take you somewhere else. Never. If everyone in this room and everyone watching this program has never drawn the line and made a decision on crime protection, you better make it when they decide to move you from crime scene number one to crime scene number two. Because the crime scene number two, it's going to be isolated, you won't choose it, you'll be the focus of the crime. I think what was so interesting about that piece of advice is at the time that we all heard that we had been trained as women people especially to believe that you just do whatever they say, whatever they say, go along with it. And what's important for those of you who are watching right now and Those of you who are on the internet with us right now to know is that it's all about using your gut in the moment and people who've survived horrific circumstances talk about listening to that intuition and every move being made calculated on listening to what your gut says. What the experts now say is do not allow yourself to be taken to the second location because anybody who is trying to harm you wants to get you to an isolated place where they can do that without other people seeing or knowing it. So in that moment of making the decision, oh, you're going to shoot me, if you're going to shoot me, You have to shoot me now rather than shoot me in isolation where nobody can see you. Whenever somebody has said something that made the little hairs on my arms or neck stand up, I know that if that's happening to me, that's also happening to somebody else too. The way to make movies is to do stuff that you love because, you know, for 25 years, I worked on the Oprah Show and Stemmin will tell you that there were nights that I came home and I almost, you know, it was hard to even take off my clothes because I knew I was gonna be getting up for hours later. But I never really felt exhausted, like, I felt exhausted, but I never felt depleted. So do the work that comes straight from the soul of you. from your background, from stories that you've grown up with, from stories that bring you passion, from stories that you not just urine to tell, but that if you don't tell them they won't get told. And when you are operating, you know, the single greatest wisdom, I think I've ever received other than when people show you they are, is that the key to full filament, success, happiness, contentment in life is when you align your personality with what your soul actually came to do. I believe everybody has a soul and has their own personal spiritual energy. So when you can use your personality to serve whatever that thing is, you can't help but be successful. So if you do films that come from the interior of your soul You do work, you do art that comes from the interior of you. You cannot miss. It's only when you're doing stuff that you think might make money. You think it may be a hit or you think it may bring you some level of attention or success. That isn't what does it. I would have to say that all of the great wonderful experiences of my life that have brought me to this moment have come from working from the interior of myself. And so that's why it feels so authentic because it actually is. So when you do that, you'll win. And everybody has a different talent. And the reason we're all so messed up is because you're looking at everybody else's talent. and wishing you had some of their talent. All the energy that you spend thinking about, wishing about being jealous of, invious of anybody else, is energy that you're not only putting out just going to come back to you negatively, but you're taking that away from you. All your energy should be forced on what do I have to offer, what do I I have to give how can I be used in service because Dr. King's message of not everybody can be famous but everybody can be great because greatness is determined by service and there is not a job in here that you can do that you don't switch the paradigm to service and not make that job more fulfilling. I don't care what the job is. If you say, I'm a singer, I'm a dancer, I'm an artist, I'm a teacher, I'm a nurse, I'm a doctor, I'm a janitor, I'm a clerk, if you say, if I look at this from, how do I use this in service to something bigger than myself? no longer becomes a job. It becomes an offering to the world. I first started making money and it was, you know, my salary or my earnings were published all over the place."); + pre_input.push_back("他不多久就睡熟了，梦见小时候见到的非洲，长长的金色海滩和白色海滩，白得刺眼，还有高耸的海岬和褐色的大山。他如今每天夜里都神游那道海岸，在梦中听见拍岸海浪的隆隆声，看见土人驾船穿浪而行。他睡着时闻到甲板上柏油和填絮的气味，还闻到早晨陆地上刮来的微风带来的非洲气息。通常一闻到陆地上刮来的微风，他就醒来，穿上衣服去叫醒那男孩。然而今夜陆地上刮来的微风的气息来得很早，他在梦中知道时间尚早，就继续把梦做下去，看见群岛间那些白色浪峰从海面上升起，随后梦见了加那利群岛的各个港湾和锚泊地。他不再梦见风暴，不再梦见妇女们，不再梦见发生过的大事，不再梦见大鱼，不再梦见打架，不再梦见角力，不再梦见他的妻子。他如今只梦见某些地方和海滩上的狮子。它们在暮色中像小猫一般嬉耍着，他爱它们，如同爱这男孩一样。他从没梦见过这男孩。他就这么醒过来，望望敞开的门外边的月亮，摊开长裤穿上。他在窝棚外撒了尿，然后顺着大路走去叫醒男孩。他被清晨的寒气弄得直哆嗦。但他知道哆嗦了一阵后会感到暖和，要不了多久就要去划船了。男孩住的那所房子的门没有上锁，他推开门，光着脚悄悄走进去。男孩在外间一张帆布床上熟睡着，老人靠着外面射进来的残月的光线，清楚地看见他。他轻轻握住男孩的一只脚，直到男孩醒来，转过脸来对他望着。老人点点头，男孩从床边椅子上拿起他的长裤，坐在床沿上穿裤子。老人走出门去，男孩跟在他背后。他还是昏昏欲睡，老人伸出胳臂搂住他的肩膀说，“对不起。”“哪里，”男孩说。“男子汉就该这么干。”他们顺着大路朝老人的窝棚走去，一路上，有些光着脚的男人在黑暗中走动，扛着他们船上的桅杆。他们走进老人的窝棚，男孩拿起装在篮子里的钓索卷儿，还有鱼叉和鱼钩，老人把绕着帆的桅杆扛在肩上。“我们把家什放在船里，然后喝一点吧。”他们在一家清早就营业的供应渔夫的小吃馆里，喝着盛在炼乳听里的咖啡。“你睡得怎么样，老大爷？”男孩问。他如今清醒过来了，尽管要他完全摆脱睡魔还不大容易。“睡得很好，马诺林，”老人说。“我感到今天挺有把握。”“我也这样，”男孩说。“现在我该去拿你我用的沙丁鱼，还有给你的新鲜鱼饵。那条船上的家什总是他自己拿的。他从来不要别人帮他拿东西。”“我们可不同，”老人说。“你还只五岁时我就让你帮忙拿东西来着。”“我记得，”男孩说。“我马上回来。再要杯咖啡吧。我们在这儿可以挂账。”他走了，光着脚在珊瑚石砌的走道上向保藏鱼饵的冷藏所走去。老人慢腾腾地喝着咖啡。这是他今儿一整天的饮食，他知道应该把它喝了。好久以来，吃饭使他感到厌烦，因此从来不带午饭。他在小帆船的船头上放着一瓶水，一整天只需要这个就够了。男孩这时带着沙丁鱼和两份包在报纸里的鱼饵回来了，他们就顺着小径走向小帆船，感到脚下的沙地里嵌着鹅卵石，他们抬起小帆船，让它溜进水里。“祝你好运，老大爷。”“祝你好运，”老人说。他把桨上的绳圈套在桨座的钉子上，身子朝前冲，抵消桨片在水中所遇到的阻力，在黑暗中动手划出港去。其他那些海滩上也有其他船只在出海，老人听到他们的桨落水和划动的声音，尽管此刻月亮已掉到了山背后，他还看不清他们。偶尔有条船上有人在说话。但是除了桨声外，大多数船只都寂静无声。它们一出港口就分散开来，每一条驶向指望能找到鱼的那片海面。老人知道自己要驶向远方，所以把陆地的气息抛在后方，划进海洋上清晨的清新气息中。他划到海里的某一片水域，看见果囊马尾藻闪出的磷光，渔夫们管这片水域叫“大井”，因为那儿水深突然达到七百英寻，海流冲击在海底深渊的峭壁上，激起了旋涡，种种鱼儿都聚集在那儿。这里集中着海虾和可作鱼饵的小鱼，在那些深不可测的水底洞穴里，有时还有成群的柔鱼，它们在夜间浮到紧靠海面的地方，所有在那儿漫游的鱼类都拿它们当食物。老人在黑暗中感觉到早晨在来临，他划着划着，听见飞鱼出水时的颤抖声，还有它们在黑暗中凌空飞走时挺直的翅膀所发出的咝咝声。他非常喜爱飞鱼，因为它们是他在海洋上的主要朋友。他替鸟儿伤心，尤其是那些柔弱的黑色小燕鸥，它们始终在飞翔，在找食，但几乎从没找到过，于是他想，鸟儿的生活过得比我们的还要艰难，除了那些猛禽和强有力的大鸟。既然海洋这样残暴，为什么像这些海燕那样的鸟儿生来就如此柔弱和纤巧？海洋是仁慈并十分美丽的。然而她能变得这样残暴，又是来得这样突然，而这些飞翔的鸟儿，从空中落下觅食，发出细微的哀鸣，却生来就柔弱得不适宜在海上生活。他每想到海洋，老是称她为la mar，这是人们对海洋抱着好感时用西班牙语对她的称呼。有时候，对海洋抱着好感的人们也说她的坏话，不过说起来总是拿她当女性看待的。有些较年轻的渔夫，用浮标当钓索上的浮子，并且在把鲨鱼肝卖了好多钱后置备了汽艇，都管海洋叫elmar，这是表示男性的说法。他们提起她时，拿她当做一个竞争者或一个去处，甚至当做一个敌人。可是这老人总是拿海洋当做女性，她给人或者不愿给人莫大的恩惠，如果她干出了任性或缺德的事儿来，那是因为她由不得自己。月亮对她起着影响，如同对一个女人那样，他想。他平稳地划着，对他说来并不费劲，因为他好好保持在自己的最高速度以内，而且除了水流偶尔打个旋儿以外，海面是平坦无浪的。他正让海流帮他干三分之一的活儿，这时天渐渐亮了，他发现自己已经划到比预期此刻能达到的地方更远了。我在这海底的深渊上转游了一个礼拜，可是一无作为，他想。今天，我要找到那些鲣鱼和长鳍金枪鱼群在什么地方，说不定会有条大鱼跟它们在一起。不等天色大亮，他就放出一个个鱼饵，让船随着海流漂去。有个鱼饵下沉到四十英寻的深处。第二个在七十五英寻的深处，第三个和第四个分别在一百英寻和一百二十五英寻深的蓝色海水中。每个由新鲜沙丁鱼做的鱼饵都是头朝下的，钓钩的钩身穿进小鱼的身子，给扎好，缝牢，因此钓钩的所有突出部分，弯钩和尖端，都给包在鱼肉里。每条沙丁鱼都用钓钩穿过双眼，这样鱼的身子在突出的钢钩上构成了半个环形。钓钩上就没有哪一部分不会叫一条大鱼觉得喷香而美味的。男孩给了他两条新鲜的小金枪鱼，或者叫做长鳍金枪鱼，它们正像铅锤般挂在那两根最深的钓索上，在另外两根上，他挂上一条蓝色大鱼和一条黄色金银鱼，它们已被使用过，但依然完好，而且还有出色的沙丁鱼给它们添上香味和吸引力。每根钓索都像一支大铅笔那么粗，一端给缠在一根青皮钓竿上，这样，只要鱼在鱼饵上一拉或一碰，就能使钓竿下垂，而每根钓索有两个四十英寻长的卷儿，它们可以牢系在其他备用的卷儿上，这一来，如果用得着的话，一条鱼可以拉出三百多英寻长的钓索。这时老人察看着那三根挑出在小帆船一边的钓竿有没有动静，一边缓缓地划着，使钓索保持上下笔直，停留在适当的水底深处。天相当亮了，太阳随时会升起来。淡淡的太阳从海上升起，老人看见其他的船只，低低地挨着水面，离海岸不远，和海流的方向垂直地展开着。跟着太阳越发明亮了，耀眼的阳光射在水面上，随后太阳从地平线上完全升起，平坦的海面把阳光反射到他眼睛里，使眼睛剧烈地刺痛，因此他不朝太阳看，顾自划着。他俯视水中，注视着那几根一直下垂到黑魆魆的深水里的钓索。他把钓索垂得比任何人更直，这样，在黑魆魆的湾流深处的几个不同的深度，都会有一个鱼饵刚好在他所指望的地方等待着在那儿游动的鱼来吃。别的渔夫让钓索随着海流漂动，有时候钓索在六十英寻的深处，他们却自以为在一百英寻的深处呢。不过，他想，我总是把它们精确地放在适当的地方的。问题只在于我的运气就此不好了。可是谁说得准呢？说不定今天就转运。每一天都是一个新的日子。走运当然更好。不过我情愿做到分毫不差。这样，运气来的时候，你就有所准备了。两小时过去了，太阳如今相应地升得更高了，他朝东望时不再感到那么刺眼了。眼前只看得见三条船，它们显得特别低矮，远在近岸的海面上。我这一辈子，初升的太阳老是刺痛我的眼睛，他想。然而眼睛还是好好的。傍晚时分，我可以直望着太阳，不会有眼前发黑的感觉。阳光的力量在傍晚要强一些。不过在早上它叫人感到眼痛。就在这时，他看见一只黑色的长翅膀军舰鸟在他前方的天空中盘旋飞翔。它倏地斜着后掠的双翅俯冲，然后又盘旋起来。“它逮住什么东西了，”老人说出声来。“它不光是找找罢了。”他慢慢划着，直朝鸟儿盘旋的地方划去。他并不着急，让那些钓索保持着上下笔直的位置。不过他还是挨近了一点儿海流，这样，他依然在用正确的方式捕鱼，尽管他的速度要比他不打算利用鸟儿来指路时来得快。军舰鸟在空中飞得高些了，又盘旋起来，双翅纹丝不动。它随即猛然俯冲下来，老人看见飞鱼从海里跃出，在海面上拚命地掠去。“鲯鳅，”老人说出声来。“大鲯鳅。”他把双桨从桨架上取下，从船头下面拿出一根细钓丝。钓丝上系着一段铁丝导线和一只中号钓钩，他拿一条沙丁鱼挂在上面。他把钓丝从船舷放下水去，将上端紧系在船艄一只拳头螺栓上。跟着他在另一根钓丝上安上鱼饵，把它盘绕着搁在船头的阴影里。他又划起船来，注视着那只此刻正在水面上低低地飞掠的长翅膀黑鸟。他看着看着，那鸟儿又朝下冲，为了俯冲，把翅膀朝后掠，然后猛地展开，追踪着飞鱼，可是没有成效。老人看见那些大鲯鳅追随在脱逃的鱼后面，把海面弄得微微隆起。鲯鳅在飞掠的鱼下面破水而行，只等飞鱼一掉下，就飞快地钻进水里。这群鲯鳅真大啊，他想。它们分布得很广，飞鱼很少脱逃的机会。那只鸟可没有成功的机会。飞鱼对它来说个头太大了，而且又飞得太快。他看着飞鱼一再地从海里冒出来，看着那只鸟儿的一无效果的行动。那群鱼从我附近逃走啦，他想。它们逃得太快，游得太远啦。不过说不定我能逮住一条掉队的，说不定我想望的大鱼就在它们周围转游着。我的大鱼总该在某处地方啊。陆地上空的云块这时像山冈般耸立着，海岸只剩下一长条绿色的线，背后是些灰青色的小山。海水此刻呈深蓝色，深得简直发紫了。他仔细俯视着海水，只见深蓝色的水中穿梭地闪出点点红色的浮游生物，阳光这时在水中变幻出奇异的光彩。他注视着那几根钓索，看见它们一直朝下没入水中看不见的地方，他很高兴看到这么多浮游生物，因为这说明有鱼。太阳此刻升得更高了，阳光在水中变幻出奇异的光彩，说明天气晴朗，而陆地上空的云块的形状也说明了这一点。可是那只鸟儿这时几乎看不见了，水面上没什么东西，只有几摊被太阳晒得发白的黄色马尾藻和一只紧靠着船舷浮动的僧帽水母，它那胶质的浮囊呈紫色，具有一定的外形，闪现出虹彩。它倒向一边，然后竖直了身子。它像个大气泡般高高兴兴地浮动着，那些厉害的紫色长触须在水中拖在身后，长达一码。请使用中文归纳上述文本"); + pre_input.push_back("把下面的现代文翻译成文言文: 到了春风和煦，阳光明媚的时候，湖面平静，没有惊涛骇浪，天色湖光相连，一片碧绿，广阔无际；沙洲上的鸥鸟，时而飞翔，时而停歇，美丽的鱼游来游去，岸上与小洲上的花草，青翠欲滴。"); + pre_input.push_back("以咏梅为题目，帮我写一首古诗，要求包含梅花、白雪等元素。"); + pre_input.push_back("上联: 江边惯看千帆过"); + pre_input.push_back("把这句话翻译成中文: Knowledge can be acquired from many sources. These include books, teachers and practical experience, and each has its own advantages. The knowledge we gain from books and formal education enables us to learn about things that we have no opportunity to experience in daily life. We can also develop our analytical skills and learn how to view and interpret the world around us in different ways. Furthermore, we can learn from the past by reading books. In this way, we won't repeat the mistakes of others and can build on their achievements."); + pre_input.push_back("把这句话翻译成英文: RK3588是新一代高端处理器，具有高算力、低功耗、超强多媒体、丰富数据接口等特点"); + cout << "\n**********************可输入以下问题对应序号获取回答/或自定义输入********************\n" + << endl; + for (int i = 0; i < (int)pre_input.size(); i++) + { + cout << "[" << i << "] " << pre_input[i] << endl; + } + cout << "\n*************************************************************************\n" + << endl; + + string text; + RKLLMInput rkllm_input; + + // RKLLMLoraAdapter lora_adapter; + // memset(&lora_adapter, 0, sizeof(RKLLMLoraAdapter)); + // lora_adapter.lora_adapter_path = "qwen0.5b_fp16_lora.rkllm"; + // lora_adapter.lora_adapter_name = "test"; + // lora_adapter.scale = 1.0; + // ret = rkllm_load_lora(llmHandle, &lora_adapter); + // if (ret != 0) { + // printf("\nload lora failed\n"); + // } + + // 加载第二个lora + // lora_adapter.lora_adapter_path = "Qwen2-0.5B-Instruct-all-rank8-F16-LoRA.gguf"; + // lora_adapter.lora_adapter_name = "knowledge_old"; + // lora_adapter.scale = 1.0; + // ret = rkllm_load_lora(llmHandle, &lora_adapter); + // if (ret != 0) { + // printf("\nload lora failed\n"); + // } + + // 初始化 infer 参数结构体 + RKLLMInferParam rkllm_infer_params; + memset(&rkllm_infer_params, 0, sizeof(RKLLMInferParam)); // 将所有内容初始化为 0 + + // 1. 初始化并设置 LoRA 参数（如果需要使用 LoRA） + RKLLMLoraParam lora_params; + lora_params.lora_adapter_name = "test"; // 指定用于推理的 lora 名称 + + // 2. 初始化并设置 Prompt Cache 参数（如果需要使用 prompt cache） + RKLLMPromptCacheParam prompt_cache_params; + prompt_cache_params.save_prompt_cache = true; // 是否保存 prompt cache + prompt_cache_params.prompt_cache_path = "./prompt_cache.bin"; // 若需要保存prompt cache, 指定 cache 文件路径 + + rkllm_infer_params.mode = RKLLM_INFER_GENERATE; + // rkllm_infer_params.lora_params = &lora_params; + // rkllm_infer_params.prompt_cache_params = &prompt_cache_params; + + // rkllm_load_prompt_cache(llmHandle, "./prompt_cache.bin"); + while (true) + { + std::string input_str; + printf("\n"); + printf("user: "); + std::getline(std::cin, input_str); + if (input_str == "exit") + { + break; + } + for (int i = 0; i < (int)pre_input.size(); i++) + { + if (input_str == to_string(i)) + { + input_str = pre_input[i]; + cout << input_str << endl; + } + } + // text = PROMPT_TEXT_PREFIX + input_str + PROMPT_TEXT_POSTFIX; + text = input_str; + rkllm_input.input_type = RKLLM_INPUT_PROMPT; + rkllm_input.prompt_input = (char *)text.c_str(); + printf("robot: "); + + // 若要使用普通推理功能,则配置rkllm_infer_mode为RKLLM_INFER_GENERATE或不配置参数 + rkllm_run(llmHandle, &rkllm_input, &rkllm_infer_params, NULL); + // rkllm_run(llmHandle, &rkllm_input, NULL); + } + rkllm_destroy(llmHandle); + + return 0; +} \ No newline at end of file diff --git a/rkllm-runtime/examples/rkllm_api_demo/src/multimodel_demo.cpp b/rkllm-runtime/examples/rkllm_api_demo/src/multimodel_demo.cpp new file mode 100644 index 0000000..9775d68 --- /dev/null +++ b/rkllm-runtime/examples/rkllm_api_demo/src/multimodel_demo.cpp @@ -0,0 +1,191 @@ +// Copyright (c) 2024 by Rockchip Electronics Co., Ltd. All Rights Reserved. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +#include +#include +#include +#include "rkllm.h" +#include +#include +#include +#include + +#define PROMPT_TEXT_PREFIX "<用户>" +#define PROMPT_TEXT_POSTFIX "" + + +using namespace std; +LLMHandle llmHandle = nullptr; + +void exit_handler(int signal) +{ + if (llmHandle != nullptr) + { + { + cout << "程序即将退出" << endl; + LLMHandle _tmp = llmHandle; + llmHandle = nullptr; + rkllm_destroy(_tmp); + } + } + exit(signal); +} + +void callback(RKLLMResult *result, void *userdata, LLMCallState state) +{ + if (state == RKLLM_RUN_FINISH) + { + printf("\n"); + } else if (state == RKLLM_RUN_ERROR) { + printf("\\run error\n"); + } else if (state == RKLLM_RUN_GET_LAST_HIDDEN_LAYER) { + /* ================================================================================================================ + 若使用GET_LAST_HIDDEN_LAYER功能,callback接口会回传内存指针:last_hidden_layer,token数量:num_tokens与隐藏层大小:embd_size + 通过这三个参数可以取得last_hidden_layer中的数据 + 注:需要在当前callback中获取,若未及时获取,下一次callback会将该指针释放 + ===============================================================================================================*/ + if (result->last_hidden_layer.embd_size != 0 && result->last_hidden_layer.num_tokens != 0) { + int data_size = result->last_hidden_layer.embd_size * result->last_hidden_layer.num_tokens * sizeof(float); + printf("\ndata_size:%d",data_size); + std::ofstream outFile("last_hidden_layer.bin", std::ios::binary); + if (outFile.is_open()) { + outFile.write(reinterpret_cast(result->last_hidden_layer.hidden_states), data_size); + outFile.close(); + std::cout << "Data saved to output.bin successfully!" << std::endl; + } else { + std::cerr << "Failed to open the file for writing!" << std::endl; + } + } + } else if (state == RKLLM_RUN_NORMAL) { + printf("%s", result->text); + // for(int i=0; inum; i++) + // { + // printf("%d token_id: %d logprob: %f\n", i, result->tokens[i].id, result->tokens[i].logprob); + // } + } +} + +int main(int argc, char **argv) +{ + if (argc < 4) { + std::cerr << "Usage: " << argv[0] << " model_path max_new_tokens max_context_len\n"; + return 1; + } + + signal(SIGINT, exit_handler); + printf("rkllm init start\n"); + + //设置参数及初始化 + RKLLMParam param = rkllm_createDefaultParam(); + param.model_path = argv[1]; + param.top_k = 1; + param.max_new_tokens = std::atoi(argv[2]); + param.max_context_len = std::atoi(argv[3]); + + // if use multimodel mode, need to set img_start,img_end and img_content + param.img_start = ""; + param.img_end = "\n"; + param.img_content = ""; + + param.skip_special_token = true; + + int ret = rkllm_init(&llmHandle, ¶m, callback); + if (ret == 0){ + printf("rkllm init success\n"); + } else { + printf("rkllm init failed\n"); + exit_handler(-1); + } + + vector pre_input; + pre_input.push_back("What is in the image?"); + cout << "\n**********************可输入以下问题对应序号获取回答/或自定义输入********************\n" + << endl; + for (int i = 0; i < (int)pre_input.size(); i++) + { + cout << "[" << i << "] " << pre_input[i] << endl; + } + cout << "\n*************************************************************************\n" + << endl; + + string text; + RKLLMInput rkllm_input; + + // 初始化 infer 参数结构体 + RKLLMInferParam rkllm_infer_params; + memset(&rkllm_infer_params, 0, sizeof(RKLLMInferParam)); // 将所有内容初始化为 0 + + // // 1. 初始化并设置 LoRA 参数（如果需要使用 LoRA） + // RKLLMLoraParam lora_params; + // lora_params.lora_adapter_name = "test"; // 指定用于推理的 lora 名称 + + // // 2. 初始化并设置 Prompt Cache 参数（如果需要使用 prompt cache） + // RKLLMPromptCacheParam prompt_cache_params; + // prompt_cache_params.save_prompt_cache = true; // 是否保存 prompt cache + // prompt_cache_params.prompt_cache_path = "./prompt_cache.bin"; // 若需要保存prompt cache, 指定 cache 文件路径 + + rkllm_infer_params.mode = RKLLM_INFER_GENERATE; + // rkllm_infer_params.lora_params = &lora_params; + // rkllm_infer_params.prompt_cache_params = &prompt_cache_params; + + // rkllm_load_prompt_cache(llmHandle, "./prompt_cache.bin"); + while (true) + { + std::string input_str; + printf("\n"); + printf("user: "); + std::getline(std::cin, input_str); + if (input_str == "exit") + { + break; + } + for (int i = 0; i < (int)pre_input.size(); i++) + { + if (input_str == to_string(i)) + { + input_str = pre_input[i]; + cout << input_str << endl; + } + } + + float * img_embed_data_ptr = (float *)malloc(64 * 2304 * sizeof(float)); + // 打开输入文件流（以二进制模式） + std::ifstream inFile("./img_vec.bin", std::ios::binary); + // 检查文件是否成功打开 + if (!inFile) { + printf("Failed to open file for reading: "); + } + // 读取数据 + float temp_data; + int idx = 0; + while (inFile.read(reinterpret_cast(&temp_data), sizeof(float))){ + img_embed_data_ptr[idx] = temp_data; + idx = idx + 1; + } + // 关闭文件 + inFile.close(); + + text = PROMPT_TEXT_PREFIX + input_str + PROMPT_TEXT_POSTFIX; + rkllm_input.input_type = RKLLM_INPUT_MULTIMODAL; + rkllm_input.multimodal_input.prompt = (char *)text.c_str(); + rkllm_input.multimodal_input.image_embed = img_embed_data_ptr; + rkllm_input.multimodal_input.n_image_tokens = 64; + printf("robot: "); + rkllm_run(llmHandle, &rkllm_input, &rkllm_infer_params, NULL); + free(img_embed_data_ptr); + } + rkllm_destroy(llmHandle); + + return 0; +} diff --git a/rkllm-runtime/examples/rkllm_server_demo/README.md b/rkllm-runtime/examples/rkllm_server_demo/README.md index 75f46c8..f5dd07b 100644 --- a/rkllm-runtime/examples/rkllm_server_demo/README.md +++ b/rkllm-runtime/examples/rkllm_server_demo/README.md @@ -8,8 +8,8 @@ Before running the demo, you need to prepare the following files: ### Build You can run the demo with the only command: ```bash -# ./build_rkllm_server_flask.sh [target_platform:rk3588/rk3576] [RKLLM-Server workshop] [transformed_rkllm_model_path in borad] -./build_rkllm_server_flask.sh rk3588 /user/data/rkllm_server /user/data/rkllm_server/model.rkllm +# Usage: ./build_rkllm_server_flask.sh --workshop [RKLLM-Server Working Path] --model_path [Absolute Path of Converted RKLLM Model on Board] --platform [Target Platform: rk3588/rk3576] [--lora_model_path [Lora Model Path]] [--prompt_cache_path [Prompt Cache File Path]] +./build_rkllm_server_flask.sh --workshop /user/data --model_path /user/data/model.rkllm --platform rk3588 ``` ### Access with API After building the RKLLM-Server-Flask, You can use ‘chat_api_flask.py’ to access the RKLLM-Server-Flask and get the answser of RKLLM models. @@ -20,8 +20,8 @@ Attention: you should check the IP address of the board with 'ifconfig' command ### Build You can run the demo with the only command: ```bash -# ./build_rkllm_server_gradio.sh [target_platform:rk3588/rk3576] [RKLLM-Server workshop] [transformed_rkllm_model_path in borad] -./build_rkllm_server_gradio.sh rk3588 /user/data/rkllm_server /user/data/rkllm_server/model.rkllm +# Usage: ./build_rkllm_server_gradio.sh --workshop [RKLLM-Server Working Path] --model_path [Absolute Path of Converted RKLLM Model on Board] --platform [Target Platform: rk3588/rk3576] [--lora_model_path [Lora Model Path]] [--prompt_cache_path [Prompt Cache File Path]] +./build_rkllm_server_gradio.sh --workshop /user/data --model_path /user/data/model.rkllm --platform rk3588 ``` ### Access the Server After running the demo, You can access the RKLLM-Server-Gradio with two ways: diff --git a/rkllm-runtime/examples/rkllm_server_demo/build_rkllm_server_flask.sh b/rkllm-runtime/examples/rkllm_server_demo/build_rkllm_server_flask.sh index 7f4c9ce..b89ec63 100755 --- a/rkllm-runtime/examples/rkllm_server_demo/build_rkllm_server_flask.sh +++ b/rkllm-runtime/examples/rkllm_server_demo/build_rkllm_server_flask.sh @@ -1,61 +1,106 @@ #!/bin/bash #*****************************************************************************************# -# 该脚本为 RKLLM-Server-Flask 服务的一键设置脚本 -# 用户可以运行该脚本实现Linux板端的 RKLLM-Server-Flask 服务的自动化部署。 -# 使用说明: ./build_rkllm_server_flask.sh [目标平台:rk3588/rk3576] [RKLLM-Server工作路径] [已转换的rkllm模型在板端的绝对路径] -# example: ./build_rkllm_server_flask.sh rk3588 /user/data/rkllm_server /user/data/rkllm_server/model.rkllm +# This script is an automated setup script for the RKLLM-Server-Flask service. +# Users can run this script to automate the deployment of the RKLLM-Server-Flask service on a Linux board. +# Usage: ./build_rkllm_server_flask.sh --workshop [RKLLM-Server Working Path] --model_path [Absolute Path of Converted RKLLM Model on Board] --platform [Target Platform: rk3588/rk3576] [--lora_model_path [Lora Model Path]] [--prompt_cache_path [Prompt Cache File Path]] +# example: ./build_rkllm_server_flask.sh --workshop /user/data --model_path /user/data/model.rkllm --platform rk3588 #*****************************************************************************************# -#################### 检查板端是否已经安装了 pip/gradio 库 #################### -# 1.准备板端的gradio环境 +LORA_PATH="" +PROMPT_FILE_PATH="" + +# Function to display help +function show_help { + echo "Usage: ./build_rkllm_server_flask.sh --workshop [RKLLM-Server Working Path] --model_path [Absolute Path of Converted RKLLM Model on Board] --platform [Target Platform: rk3588/rk3576] [--lora_path [Lora Model Path]] [--prompt_cache_path [Prompt Cache File Path]]" +} + +# Parse command-line options +while [[ $# -gt 0 ]]; do + case "$1" in + --workshop) + WORKING_PATH="$2" + shift 2 + ;; + --model_path) + MODEL_PATH="$2" + shift 2 + ;; + --platform) + TARGET_PLATFORM="$2" + shift 2 + ;; + --lora_model_path) + LORA_PATH="$2" + shift 2 + ;; + --prompt_cache_path) + PROMPT_FILE_PATH="$2" + shift 2 + ;; + --help) + show_help + exit 0 + ;; + *) + echo "无效的选项: $1" 1>&2 + show_help + exit 1 + ;; + esac +done + +#################### Check if pip and the Flask library are already installed on the board. #################### adb shell << EOF -# 检查是否安装了 pip3 if ! command -v pip3 &> /dev/null; then - echo "-------- pip3 未安装，将进行安装... --------" - # 安装 pip3 + echo "-------- pip3 is not installed. Installing it now... --------" sudo apt update sudo apt install python3-pip -y else - echo "-------- pip3 已经安装 --------" + echo "-------- pip3 is already installed. --------" fi -# 检查是否安装了 flask if ! python3 -c "import flask" &> /dev/null; then - echo "-------- flask 未安装，将进行安装... --------" + echo "-------- flask is not installed. Installing it now... --------" # 安装 flask - pip install flask==2.2.2 Werkzeug==2.2.2 -i https://pypi.tuna.tsinghua.edu.cn/simple + pip install flask==2.2.2 Werkzeug==2.2.2 -i https://pypi.tuna.tsinghua.edu.cn/simple --break-system-packages else - echo "-------- flask 已经安装 --------" + echo "-------- flask is already installed. --------" fi exit EOF -#################### 推送 server 运行的相关文件进入板端 #################### -# 2.检查需要推送进板端的路径是否存在 -adb shell ls $2 > /dev/null 2>&1 +#################### Push the relevant files for the server to the board. #################### +adb shell ls $WORKING_PATH > /dev/null 2>&1 + if [ $? -ne 0 ]; then - # 如果路径不存在，则创建路径 - adb shell mkdir -p $2 - echo "-------- rkllm_server 工作目录不存在，已创建目录 --------" + adb shell mkdir -p $WORKING_PATH + echo "-------- The rkllm_server working directory does not exist, so it has been created. --------" else - echo "-------- rkllm_server 工作目录已存在 --------" + echo "-------- The rkllm_server working directory already exists. --------" fi -# 3.更新 ./rkllm_server/lib 中的 librkllmrt.so 文件 +# Update the `librkllmrt.so` file in the `./rkllm_server/lib` directory. cp ../../runtime/Linux/librkllm_api/aarch64/librkllmrt.so ./rkllm_server/lib/ -# 4.推送文件到 Linux 板端 -adb push ./rkllm_server $2 +adb push ./rkllm_server $WORKING_PATH + +#################### Enter the board terminal and start the server service. #################### +CMD="python3 flask_server.py --rkllm_model_path $MODEL_PATH --target_platform $TARGET_PLATFORM" +if [[ -n "$LORA_PATH" ]]; then + CMD="$CMD --lora_model_path $LORA_PATH" +fi + +if [[ -n "$PROMPT_FILE_PATH" ]]; then + CMD="$CMD --prompt_cache_path $PROMPT_FILE_PATH" +fi -#################### 进入板端并启动 server 服务 #################### -# 5.进入板端启动 server 服务 adb shell << EOF -cd $2/rkllm_server/ -python3 flask_server.py --target_platform $1 --rkllm_model_path $3 +cd $WORKING_PATH/rkllm_server/ +$CMD EOF diff --git a/rkllm-runtime/examples/rkllm_server_demo/build_rkllm_server_gradio.sh b/rkllm-runtime/examples/rkllm_server_demo/build_rkllm_server_gradio.sh index 356968d..7998b5f 100755 --- a/rkllm-runtime/examples/rkllm_server_demo/build_rkllm_server_gradio.sh +++ b/rkllm-runtime/examples/rkllm_server_demo/build_rkllm_server_gradio.sh @@ -1,61 +1,106 @@ #!/bin/bash #*****************************************************************************************# -# 该脚本为 RKLLM-Server-Gradio 服务的一键设置脚本 -# 用户可以运行该脚本实现Linux板端的 RKLLM-Server-Gradio 服务的自动化部署。 -# 使用说明: ./build_rkllm_server_gradio.sh [目标平台:rk3588/rk3576] [RKLLM-Server工作路径] [已转换的rkllm模型在板端的绝对路径] -# example: ./build_rkllm_server_gradio.sh rk3588 /user/data/rkllm_server /user/data/rkllm_server/model.rkllm +# This script is an automated setup script for the RKLLM-Server-Gradio service. +# Users can run this script to automate the deployment of the RKLLM-Server-Gradio service on a Linux board. +# Usage: ./build_rkllm_server_gradio.sh --workshop [RKLLM-Server Working Path] --model_path [Absolute Path of Converted RKLLM Model on Board] --platform [Target Platform: rk3588/rk3576] [--lora_model_path [Lora Model Path]] [--prompt_cache_path [Prompt Cache File Path]] +# example: ./build_rkllm_server_gradio.sh --workshop /user/data --model_path /user/data/model.rkllm --platform rk3588 #*****************************************************************************************# -#################### 检查板端是否已经安装了 pip/gradio 库 #################### -# 1.准备板端的gradio环境 +LORA_PATH="" +PROMPT_FILE_PATH="" + +# Function to display help +function show_help { + echo "Usage: ./build_rkllm_server_gradio.sh --workshop [RKLLM-Server Working Path] --model_path [Absolute Path of Converted RKLLM Model on Board] --platform [Target Platform: rk3588/rk3576] [--lora_path [Lora Model Path]] [--prompt_cache_path [Prompt Cache File Path]]" +} + +# Parse command-line options +while [[ $# -gt 0 ]]; do + case "$1" in + --workshop) + WORKING_PATH="$2" + shift 2 + ;; + --model_path) + MODEL_PATH="$2" + shift 2 + ;; + --platform) + TARGET_PLATFORM="$2" + shift 2 + ;; + --lora_model_path) + LORA_PATH="$2" + shift 2 + ;; + --prompt_cache_path) + PROMPT_FILE_PATH="$2" + shift 2 + ;; + --help) + show_help + exit 0 + ;; + *) + echo "无效的选项: $1" 1>&2 + show_help + exit 1 + ;; + esac +done + +#################### Check if pip and the Gradio library are already installed on the board. #################### adb shell << EOF -# 检查是否安装了 pip3 if ! command -v pip3 &> /dev/null; then - echo "-------- pip3 未安装，将进行安装... --------" - # 安装 pip3 + echo "-------- pip3 is not installed. Installing it now... --------" sudo apt update sudo apt install python3-pip -y else - echo "-------- pip3 已经安装 --------" + echo "-------- pip3 is already installed. --------" fi -# 检查是否安装了 gradio if ! python3 -c "import gradio" &> /dev/null; then - echo "-------- Gradio 未安装，将进行安装... --------" - # 安装 Gradio - pip3 install gradio>=4.24.0 -i https://pypi.tuna.tsinghua.edu.cn/simple/ + echo "-------- gradio is not installed. Installing it now... --------" + pip3 install gradio>=4.24.0 -i https://pypi.tuna.tsinghua.edu.cn/simple/ --break-system-packages else - echo "-------- Gradio 已经安装 --------" + echo "-------- gradio is already installed. --------" fi exit EOF -#################### 推送 server 运行的相关文件进入板端 #################### -# 2.检查需要推送进板端的路径是否存在 -adb shell ls $2 > /dev/null 2>&1 +#################### Push the relevant files for the server to the board. #################### +adb shell ls $WORKING_PATH > /dev/null 2>&1 + if [ $? -ne 0 ]; then - # 如果路径不存在，则创建路径 - adb shell mkdir -p $2 - echo "-------- rkllm_server 工作目录不存在，已创建目录 --------" + adb shell mkdir -p $WORKING_PATH + echo "-------- The rkllm_server working directory does not exist, so it has been created. --------" else - echo "-------- rkllm_server 工作目录已存在 --------" + echo "-------- The rkllm_server working directory already exists. --------" fi -# 3.更新 ./rkllm_server/lib 中的 librkllmrt.so 文件 +# Update the `librkllmrt.so` file in the `./rkllm_server/lib` directory. cp ../../runtime/Linux/librkllm_api/aarch64/librkllmrt.so ./rkllm_server/lib/ -# 4.推送文件到 Linux 板端 -adb push ./rkllm_server $2 +adb push ./rkllm_server $WORKING_PATH + +#################### Enter the board terminal and start the server service. #################### +CMD="python3 gradio_server.py --rkllm_model_path $MODEL_PATH --target_platform $TARGET_PLATFORM" + +if [[ -n "$LORA_PATH" ]]; then + CMD="$CMD --lora_model_path $LORA_PATH" +fi + +if [[ -n "$PROMPT_FILE_PATH" ]]; then + CMD="$CMD --prompt_cache_path $PROMPT_FILE_PATH" +fi -#################### 进入板端并启动 server 服务 #################### -# 5.进入板端启动 server 服务 adb shell << EOF -cd $2/rkllm_server/ -python3 gradio_server.py --target_platform $1 --rkllm_model_path $3 +cd $WORKING_PATH/rkllm_server/ +$CMD EOF diff --git a/rkllm-runtime/examples/rkllm_server_demo/chat_api_flask.py b/rkllm-runtime/examples/rkllm_server_demo/chat_api_flask.py index f6d6d06..e96024f 100644 --- a/rkllm-runtime/examples/rkllm_server_demo/chat_api_flask.py +++ b/rkllm-runtime/examples/rkllm_server_demo/chat_api_flask.py @@ -2,53 +2,53 @@ import sys import requests import json -# 设置 Server 服务器的地址 -server_url = 'http://172.16.10.102:8080/rkllm_chat' -# 设置是否开启流式对话 +# Set the address of the Server. +server_url = 'http://172.16.10.79:8080/rkllm_chat' +# Set whether to enable streaming mode. is_streaming = True -# 创建一个会话对象 +# Create a session object. session = requests.Session() -session.keep_alive = False # 关闭连接池，保持长连接 +session.keep_alive = False # Close the connection pool to maintain a long connection. adapter = requests.adapters.HTTPAdapter(max_retries=5) session.mount('https://', adapter) session.mount('http://', adapter) if __name__ == '__main__': print("============================") - print("在终端中输入您的问题，即可与 RKLLM 模型进行对话....") + print("Input your question in the terminal to start a conversation with the RKLLM model...") print("============================") - # 进入循环，持续获取用户输入，并与RKLLM模型进行对话 + # Enter a loop to continuously get user input and converse with the RKLLM model. while True: try: - user_message = input("请输入您的问题：") + user_message = input("\n*Please enter your question:") if user_message == "exit": print("============================") - print("程序正在退出......") + print("The RKLLM Server is stopping......") print("============================") break else: - # 设置请求头，此处的请求头实际并无作用，仅为模拟OpenAI接口设计 + # Set the request headers; in this case, the headers have no actual effect and are only used to simulate the OpenAI interface design. headers = { 'Content-Type': 'application/json', 'Authorization': 'not_required' } - # 准备要发送的数据 - # model: 为用户在设置RKLLM-Server时定义的模型，此处并无作用 - # messages: 用户输入的问题，RKLLM-Server将会把它作为输入，并返回模型的回复；支持在 messags 加入多个问题 - # stream: 是否开启流式对话，与OpenAI接口相同 + # Prepare the data to be sent + # model: The model defined by the user when setting up RKLLM-Server; this has no effect here + # messages: The user's input question, which RKLLM-Server will use as input and return the model's reply; multiple questions can be added to messages + # stream: Whether to enable streaming conversation, similar to the OpenAI interface data = { "model": 'your_model_deploy_with_RKLLM_Server', "messages": [{"role": "user", "content": user_message}], "stream": is_streaming } - # 发送 POST 请求 + # Send a POST request responses = session.post(server_url, json=data, headers=headers, stream=is_streaming, verify=False) if not is_streaming: - # 解析响应 + # Parse the response if responses.status_code == 200: print("Q:", data["messages"][-1]["content"]) print("A:", json.loads(responses.text)["choices"][-1]["message"]["content"]) @@ -66,16 +66,13 @@ if __name__ == '__main__': sys.stdout.flush() else: print('Error:', responses.text) - - - - + except KeyboardInterrupt: - # 捕获 Ctrl-C 信号，关闭会话 + # Capture Ctrl-C signal to close the session session.close() print("\n") print("============================") - print("程序正在退出......") + print("The RKLLM Server is stopping......") print("============================") break diff --git a/rkllm-runtime/examples/rkllm_server_demo/chat_api_gradio.py b/rkllm-runtime/examples/rkllm_server_demo/chat_api_gradio.py index 1b8fdfb..b69888b 100644 --- a/rkllm-runtime/examples/rkllm_server_demo/chat_api_gradio.py +++ b/rkllm-runtime/examples/rkllm_server_demo/chat_api_gradio.py @@ -1,44 +1,43 @@ from gradio_client import Client -# 该函数通过调用Gradio Client API与RKLLM模型进行交互 +# This function interacts with the RKLLM model by calling the Gradio Client API. def chat_with_rkllm(user_message, history=[]): - # 实例化Gradio Client，用户需要根据自己部署的具体网址进行修改 - client = Client("http://172.16.10.102:8080/") + # Instantiate the Gradio Client. Users need to modify according to their specific deployment URL. + client = Client("http://172.16.10.79:8080") - # 调用Gradio Client API进行交互，内部的API主要包括： - # /get_user_input：模型获取用户输入，并将输入添加至历史记录history - # /get_RKLLM_output：RKLLM利用已包含输入的历史记录history生成回复 + # Call the Gradio Client API for interaction. The internal APIs mainly include: + # get_user_input: The model retrieves user input and adds it to the history record 'history'. + # get_RKLLM_output: RKLLM generates a response using the historical record 'history' that contains the input. _, history = client.predict(user_message=user_message, history=history, api_name="/get_user_input") result_history = client.predict(history=history, api_name="/get_RKLLM_output") return result_history if __name__ == '__main__': - #初始化聊天记录 result_history = [] print("============================") - print("在终端中输入您的问题，即可与 RKLLM 模型进行对话....") + print("Enter your question in the terminal to have a conversation with the RKLLM model...") print("============================") - # 进入循环，持续获取用户输入，并与RKLLM模型进行对话 + # Enter a loop to continuously receive user input and have a conversation with the RKLLM model... while True: try: - user_message = input("请输入您的问题：") + user_message = input("Please enter your question:") if user_message == "exit": print("============================") - print("程序正在退出......") + print("The RKLLM Server is stopping......") print("============================") break else: - # 调用chat_with_rkllm函数，获取模型的回复 + # Call the `chat_with_rkllm` function to get the model's response. result_history = chat_with_rkllm(user_message, result_history) - # 打印模型输出 + # Print the history of chatting print("Q:", result_history[-1][0]) print("A:", result_history[-1][1]) except KeyboardInterrupt: print("\n") print("============================") - print("程序正在退出......") + print("The RKLLM Server is stopping......") print("============================") break \ No newline at end of file diff --git a/rkllm-runtime/examples/rkllm_server_demo/rkllm_server/flask_server.py b/rkllm-runtime/examples/rkllm_server_demo/rkllm_server/flask_server.py index 2ceab37..a018a89 100644 --- a/rkllm-runtime/examples/rkllm_server_demo/rkllm_server/flask_server.py +++ b/rkllm-runtime/examples/rkllm_server_demo/rkllm_server/flask_server.py @@ -11,66 +11,42 @@ from flask import Flask, request, jsonify, Response app = Flask(__name__) -# 创建一个锁，用于控制多人访问Server -lock = threading.Lock() +PROMPT_TEXT_PREFIX = "<|im_start|>system You are a helpful assistant. <|im_end|> <|im_start|>user" +PROMPT_TEXT_POSTFIX = "<|im_end|><|im_start|>assistant" -# 创建一个全局变量，用于标识服务器当前是否处于阻塞状态 -is_blocking = False - -# 设置动态库路径 +# Set the dynamic library path rkllm_lib = ctypes.CDLL('lib/librkllmrt.so') -# 定义全局变量，用于保存回调函数的输出，便于在gradio界面中输出 -global_text = [] -global_state = -1 -split_byte_data = bytes(b"") # 用于保存分割的字节数据 +# Define the structures from the library +RKLLM_Handle_t = ctypes.c_void_p +userdata = ctypes.c_void_p(None) -# 定义动态库中的结构体 -class Token(ctypes.Structure): +LLMCallState = ctypes.c_int +LLMCallState.RKLLM_RUN_NORMAL = 0 +LLMCallState.RKLLM_RUN_WAITING = 1 +LLMCallState.RKLLM_RUN_FINISH = 2 +LLMCallState.RKLLM_RUN_ERROR = 3 +LLMCallState.RKLLM_RUN_GET_LAST_HIDDEN_LAYER = 4 + +RKLLMInputMode = ctypes.c_int +RKLLMInputMode.RKLLM_INPUT_PROMPT = 0 +RKLLMInputMode.RKLLM_INPUT_TOKEN = 1 +RKLLMInputMode.RKLLM_INPUT_EMBED = 2 +RKLLMInputMode.RKLLM_INPUT_MULTIMODAL = 3 + +RKLLMInferMode = ctypes.c_int +RKLLMInferMode.RKLLM_INFER_GENERATE = 0 +RKLLMInferMode.RKLLM_INFER_GET_LAST_HIDDEN_LAYER = 1 + +class RKLLMExtendParam(ctypes.Structure): _fields_ = [ - ("logprob", ctypes.c_float), - ("id", ctypes.c_int32) + ("base_domain_id", ctypes.c_int32), + ("reserved", ctypes.c_uint8 * 112) ] -class RKLLMResult(ctypes.Structure): - _fields_ = [ - ("text", ctypes.c_char_p), - ("tokens", ctypes.POINTER(Token)), - ("num", ctypes.c_int32) - ] - - -# 定义回调函数 -def callback(result, userdata, state): - global global_text, global_state, split_byte_data - if state == 0: - # 保存输出的token文本及RKLLM运行状态 - global_state = state - # 需要监控当前的字节数据是否完整，不完整则进行记录，后续进行解析 - try: - global_text.append((split_byte_data + result.contents.text).decode('utf-8')) - print((split_byte_data + result.contents.text).decode('utf-8'), end='') - split_byte_data = bytes(b"") - except: - split_byte_data += result.contents.text - sys.stdout.flush() - elif state == 1: - # 保存RKLLM运行状态 - global_state = state - print("\n") - sys.stdout.flush() - else: - print("run error") - -# Python端与C++端的回调函数连接 -callback_type = ctypes.CFUNCTYPE(None, ctypes.POINTER(RKLLMResult), ctypes.c_void_p, ctypes.c_int) -c_callback = callback_type(callback) - -# 定义动态库中的结构体 -class RKNNllmParam(ctypes.Structure): +class RKLLMParam(ctypes.Structure): _fields_ = [ ("model_path", ctypes.c_char_p), - ("num_npu_core", ctypes.c_int32), ("max_context_len", ctypes.c_int32), ("max_new_tokens", ctypes.c_int32), ("top_k", ctypes.c_int32), @@ -82,60 +58,234 @@ class RKNNllmParam(ctypes.Structure): ("mirostat", ctypes.c_int32), ("mirostat_tau", ctypes.c_float), ("mirostat_eta", ctypes.c_float), - ("logprobs", ctypes.c_bool), - ("top_logprobs", ctypes.c_int32), - ("use_gpu", ctypes.c_bool) + ("skip_special_token", ctypes.c_bool), + ("is_async", ctypes.c_bool), + ("img_start", ctypes.c_char_p), + ("img_end", ctypes.c_char_p), + ("img_content", ctypes.c_char_p), + ("extend_param", RKLLMExtendParam), ] -# 定义RKLLM_Handle_t和userdata -RKLLM_Handle_t = ctypes.c_void_p -userdata = ctypes.c_void_p(None) +class RKLLMLoraAdapter(ctypes.Structure): + _fields_ = [ + ("lora_adapter_path", ctypes.c_char_p), + ("lora_adapter_name", ctypes.c_char_p), + ("scale", ctypes.c_float) + ] -# 设置提示文本 -PROMPT_TEXT_PREFIX = "<|im_start|>system You are a helpful assistant. <|im_end|> <|im_start|>user" -PROMPT_TEXT_POSTFIX = "<|im_end|><|im_start|>assistant" +class RKLLMEmbedInput(ctypes.Structure): + _fields_ = [ + ("embed", ctypes.POINTER(ctypes.c_float)), + ("n_tokens", ctypes.c_size_t) + ] -# 定义Python端的RKLLM类，其中包括了对动态库中RKLLM模型的初始化、推理及释放操作 +class RKLLMTokenInput(ctypes.Structure): + _fields_ = [ + ("input_ids", ctypes.POINTER(ctypes.c_int32)), + ("n_tokens", ctypes.c_size_t) + ] + +class RKLLMMultiModelInput(ctypes.Structure): + _fields_ = [ + ("prompt", ctypes.c_char_p), + ("image_embed", ctypes.POINTER(ctypes.c_float)), + ("n_image_tokens", ctypes.c_size_t) + ] + +class RKLLMInputUnion(ctypes.Union): + _fields_ = [ + ("prompt_input", ctypes.c_char_p), + ("embed_input", RKLLMEmbedInput), + ("token_input", RKLLMTokenInput), + ("multimodal_input", RKLLMMultiModelInput) + ] + +class RKLLMInput(ctypes.Structure): + _fields_ = [ + ("input_mode", ctypes.c_int), + ("input_data", RKLLMInputUnion) + ] + +class RKLLMLoraParam(ctypes.Structure): + _fields_ = [ + ("lora_adapter_name", ctypes.c_char_p) + ] + +class RKLLMPromptCacheParam(ctypes.Structure): + _fields_ = [ + ("save_prompt_cache", ctypes.c_int), + ("prompt_cache_path", ctypes.c_char_p) + ] + +class RKLLMInferParam(ctypes.Structure): + _fields_ = [ + ("mode", RKLLMInferMode), + ("lora_params", ctypes.POINTER(RKLLMLoraParam)), + ("prompt_cache_params", ctypes.POINTER(RKLLMPromptCacheParam)) + ] + +class RKLLMResultLastHiddenLayer(ctypes.Structure): + _fields_ = [ + ("hidden_states", ctypes.POINTER(ctypes.c_float)), + ("embd_size", ctypes.c_int), + ("num_tokens", ctypes.c_int) + ] + +class RKLLMResult(ctypes.Structure): + _fields_ = [ + ("text", ctypes.c_char_p), + ("size", ctypes.c_int), + ("last_hidden_layer", RKLLMResultLastHiddenLayer) + ] + + +# Create a lock to control multi-user access to the server. +lock = threading.Lock() + +# Create a global variable to indicate whether the server is currently in a blocked state. +is_blocking = False + +# Define global variables to store the callback function output for displaying in the Gradio interface +global_text = [] +global_state = -1 +split_byte_data = bytes(b"") # Used to store the segmented byte data + +# Define the callback function +def callback_impl(result, userdata, state): + global global_text, global_state, split_byte_data + if state == LLMCallState.RKLLM_RUN_FINISH: + global_state = state + print("\n") + sys.stdout.flush() + elif state == LLMCallState.RKLLM_RUN_ERROR: + global_state = state + print("run error") + sys.stdout.flush() + elif state == LLMCallState.RKLLM_RUN_GET_LAST_HIDDEN_LAYER: + ''' + If using the GET_LAST_HIDDEN_LAYER function, the callback interface will return the memory pointer: last_hidden_layer, the number of tokens: num_tokens, and the size of the hidden layer: embd_size. + With these three parameters, you can retrieve the data from last_hidden_layer. + Note: The data needs to be retrieved during the current callback; if not obtained in time, the pointer will be released by the next callback. + ''' + if result.last_hidden_layer.embd_size != 0 and result.last_hidden_layer.num_tokens != 0: + data_size = result.last_hidden_layer.embd_size * result.last_hidden_layer.num_tokens * ctypes.sizeof(ctypes.c_float) + print(f"data_size: {data_size}") + global_text.append(f"data_size: {data_size}\n") + output_path = os.getcwd() + "/last_hidden_layer.bin" + with open(output_path, "wb") as outFile: + data = ctypes.cast(result.last_hidden_layer.hidden_states, ctypes.POINTER(ctypes.c_float)) + float_array_type = ctypes.c_float * (data_size // ctypes.sizeof(ctypes.c_float)) + float_array = float_array_type.from_address(ctypes.addressof(data.contents)) + outFile.write(bytearray(float_array)) + print(f"Data saved to {output_path} successfully!") + global_text.append(f"Data saved to {output_path} successfully!") + else: + print("Invalid hidden layer data.") + global_text.append("Invalid hidden layer data.") + global_state = state + time.sleep(0.05) # Delay for 0.05 seconds to wait for the output result + sys.stdout.flush() + else: + # Save the output token text and the RKLLM running state + global_state = state + # Monitor if the current byte data is complete; if incomplete, record it for later parsing + try: + global_text.append((split_byte_data + result.contents.text).decode('utf-8')) + print((split_byte_data + result.contents.text).decode('utf-8'), end='') + split_byte_data = bytes(b"") + except: + split_byte_data += result.contents.text + sys.stdout.flush() + +# Connect the callback function between the Python side and the C++ side +callback_type = ctypes.CFUNCTYPE(None, ctypes.POINTER(RKLLMResult), ctypes.c_void_p, ctypes.c_int) +callback = callback_type(callback_impl) + +# Define the RKLLM class, which includes initialization, inference, and release operations for the RKLLM model in the dynamic library class RKLLM(object): - def __init__(self, model_path, target_platform): - rknnllm_param = RKNNllmParam() - rknnllm_param.model_path = bytes(model_path, 'utf-8') - if target_platform == "rk3588": - rknnllm_param.num_npu_core = 3 - elif target_platform == "rk3576": - rknnllm_param.num_npu_core = 1 - rknnllm_param.max_context_len = 320 - rknnllm_param.max_new_tokens = 512 - rknnllm_param.top_k = 1 - rknnllm_param.top_p = 0.9 - rknnllm_param.temperature = 0.8 - rknnllm_param.repeat_penalty = 1.1 - rknnllm_param.frequency_penalty = 0.0 - rknnllm_param.presence_penalty = 0.0 - rknnllm_param.mirostat = 0 - rknnllm_param.mirostat_tau = 5.0 - rknnllm_param.mirostat_eta = 0.1 - rknnllm_param.logprobs = False - rknnllm_param.top_logprobs = 5 - rknnllm_param.use_gpu = True + def __init__(self, model_path, lora_model_path = None, prompt_cache_path = None): + rkllm_param = RKLLMParam() + rkllm_param.model_path = bytes(model_path, 'utf-8') + + rkllm_param.max_context_len = 512 + rkllm_param.max_new_tokens = -1 + rkllm_param.skip_special_token = True + + rkllm_param.top_k = 1 + rkllm_param.top_p = 0.9 + rkllm_param.temperature = 0.8 + rkllm_param.repeat_penalty = 1.1 + rkllm_param.frequency_penalty = 0.0 + rkllm_param.presence_penalty = 0.0 + + rkllm_param.mirostat = 0 + rkllm_param.mirostat_tau = 5.0 + rkllm_param.mirostat_eta = 0.1 + + rkllm_param.is_async = False + + rkllm_param.img_start = "".encode('utf-8') + rkllm_param.img_end = "".encode('utf-8') + rkllm_param.img_content = "".encode('utf-8') + + rkllm_param.extend_param.base_domain_id = 0 + self.handle = RKLLM_Handle_t() self.rkllm_init = rkllm_lib.rkllm_init - self.rkllm_init.argtypes = [ctypes.POINTER(RKLLM_Handle_t), ctypes.POINTER(RKNNllmParam), callback_type] + self.rkllm_init.argtypes = [ctypes.POINTER(RKLLM_Handle_t), ctypes.POINTER(RKLLMParam), callback_type] self.rkllm_init.restype = ctypes.c_int - self.rkllm_init(ctypes.byref(self.handle), rknnllm_param, c_callback) + self.rkllm_init(ctypes.byref(self.handle), ctypes.byref(rkllm_param), callback) self.rkllm_run = rkllm_lib.rkllm_run - self.rkllm_run.argtypes = [RKLLM_Handle_t, ctypes.POINTER(ctypes.c_char), ctypes.c_void_p] + self.rkllm_run.argtypes = [RKLLM_Handle_t, ctypes.POINTER(RKLLMInput), ctypes.POINTER(RKLLMInferParam), ctypes.c_void_p] self.rkllm_run.restype = ctypes.c_int self.rkllm_destroy = rkllm_lib.rkllm_destroy self.rkllm_destroy.argtypes = [RKLLM_Handle_t] self.rkllm_destroy.restype = ctypes.c_int + self.lora_adapter_path = None + self.lora_model_name = None + if lora_model_path: + self.lora_adapter_path = lora_model_path + self.lora_adapter_name = "test" + + lora_adapter = RKLLMLoraAdapter() + ctypes.memset(ctypes.byref(lora_adapter), 0, ctypes.sizeof(RKLLMLoraAdapter)) + lora_adapter.lora_adapter_path = ctypes.c_char_p((self.lora_adapter_path).encode('utf-8')) + lora_adapter.lora_adapter_name = ctypes.c_char_p((self.lora_adapter_name).encode('utf-8')) + lora_adapter.scale = 1.0 + + rkllm_load_lora = rkllm_lib.rkllm_load_lora + rkllm_load_lora.argtypes = [RKLLM_Handle_t, ctypes.POINTER(RKLLMLoraAdapter)] + rkllm_load_lora.restype = ctypes.c_int + rkllm_load_lora(self.handle, ctypes.byref(lora_adapter)) + + self.prompt_cache_path = None + if prompt_cache_path: + self.prompt_cache_path = prompt_cache_path + + rkllm_load_prompt_cache = rkllm_lib.rkllm_load_prompt_cache + rkllm_load_prompt_cache.argtypes = [RKLLM_Handle_t, ctypes.c_char_p] + rkllm_load_prompt_cache.restype = ctypes.c_int + rkllm_load_prompt_cache(self.handle, ctypes.c_char_p((prompt_cache_path).encode('utf-8'))) + def run(self, prompt): - prompt = bytes(PROMPT_TEXT_PREFIX + prompt + PROMPT_TEXT_POSTFIX, 'utf-8') - self.rkllm_run(self.handle, prompt, ctypes.byref(userdata)) + rkllm_lora_params = None + if self.lora_model_name: + rkllm_lora_params = RKLLMLoraParam() + rkllm_lora_params.lora_adapter_name = ctypes.c_char_p((self.lora_model_name).encode('utf-8')) + + rkllm_infer_params = RKLLMInferParam() + ctypes.memset(ctypes.byref(rkllm_infer_params), 0, ctypes.sizeof(RKLLMInferParam)) + rkllm_infer_params.mode = RKLLMInferMode.RKLLM_INFER_GENERATE + rkllm_infer_params.lora_params = ctypes.byref(rkllm_lora_params) if rkllm_lora_params else None + + rkllm_input = RKLLMInput() + rkllm_input.input_mode = RKLLMInputMode.RKLLM_INPUT_PROMPT + rkllm_input.input_data.prompt_input = ctypes.c_char_p((PROMPT_TEXT_PREFIX + prompt + PROMPT_TEXT_POSTFIX).encode('utf-8')) + self.rkllm_run(self.handle, ctypes.byref(rkllm_input), ctypes.byref(rkllm_infer_params), None) return def release(self): @@ -143,62 +293,74 @@ class RKLLM(object): if __name__ == "__main__": parser = argparse.ArgumentParser() - parser.add_argument('--target_platform', help='目标平台: 如rk3588/rk3576;') - parser.add_argument('--rkllm_model_path', help='Linux板端上已转换好的rkllm模型的绝对路径') + parser.add_argument('--rkllm_model_path', type=str, required=True, help='Absolute path of the converted RKLLM model on the Linux board;') + parser.add_argument('--target_platform', type=str, required=True, help='Target platform: e.g., rk3588/rk3576;') + parser.add_argument('--lora_model_path', type=str, help='Absolute path of the lora_model on the Linux board;') + parser.add_argument('--prompt_cache_path', type=str, help='Absolute path of the prompt_cache file on the Linux board;') args = parser.parse_args() - if not (args.target_platform in ["rk3588", "rk3576"]): - print("====== Error: 请指定正确的目标平台: rk3588/rk3576 ======") - sys.stdout.flush() - exit() - if not os.path.exists(args.rkllm_model_path): - print("====== Error: 请给出准确的rkllm模型路径，需注意是板端的绝对路径 ======") + print("Error: Please provide the correct rkllm model path, and ensure it is the absolute path on the board.") sys.stdout.flush() exit() - # 定频设置 + if not (args.target_platform in ["rk3588", "rk3576"]): + print("Error: Please specify the correct target platform: rk3588/rk3576.") + sys.stdout.flush() + exit() + + if args.lora_model_path: + if not os.path.exists(args.lora_model_path): + print("Error: Please provide the correct lora_model path, and advise it is the absolute path on the board.") + sys.stdout.flush() + exit() + + if args.prompt_cache_path: + if not os.path.exists(args.prompt_cache_path): + print("Error: Please provide the correct prompt_cache_file path, and advise it is the absolute path on the board.") + sys.stdout.flush() + exit() + + # Fix frequency command = "sudo bash fix_freq_{}.sh".format(args.target_platform) subprocess.run(command, shell=True) - # 设置文件描述符限制 + # Set resource limit resource.setrlimit(resource.RLIMIT_NOFILE, (102400, 102400)) - # 初始化RKLLM模型 + # Initialize RKLLM model print("=========init....===========") sys.stdout.flush() - target_platform = args.target_platform model_path = args.rkllm_model_path - rkllm_model = RKLLM(model_path, target_platform) - print("RKLLM初始化成功！") + rkllm_model = RKLLM(model_path, args.lora_model_path, args.prompt_cache_path) + print("RKLLM Model has been initialized successfully！") print("==============================") sys.stdout.flush() - # 创建一个函数用于接受用户使用 request 发送的数据 + # Create a function to receive data sent by the user using a request @app.route('/rkllm_chat', methods=['POST']) def receive_message(): - # 链接全局变量，获取回调函数的输出信息 + # Link global variables to retrieve the output information from the callback function global global_text, global_state global is_blocking - # 如果服务器正在阻塞状态，则返回特定响应 + # If the server is in a blocking state, return a specific response. if is_blocking or global_state==0: return jsonify({'status': 'error', 'message': 'RKLLM_Server is busy! Maybe you can try again later.'}), 503 - # 加锁 lock.acquire() try: - # 设置服务器为阻塞状态 + # Set the server to a blocking state. is_blocking = True - # 获取 POST 请求中的 JSON 数据 + # Get JSON data from the POST request. data = request.json if data and 'messages' in data: - # 重置全局变量 + # Reset global variables. global_text = [] global_state = -1 - # 定义返回的结构体 + # Define the structure for the returned response. rkllm_responses = { "id": "rkllm_chat", "object": "rkllm_chat", @@ -212,18 +374,18 @@ if __name__ == "__main__": } if not "stream" in data.keys() or data["stream"] == False: - # 在这里处理收到的数据 + # Process the received data here. messages = data['messages'] print("Received messages:", messages) for index, message in enumerate(messages): input_prompt = message['content'] rkllm_output = "" - # 创建模型推理的线程 + # Create a thread for model inference. model_thread = threading.Thread(target=rkllm_model.run, args=(input_prompt,)) model_thread.start() - # 等待模型运行完成，定时检查模型的推理线程 + # Wait for the model to finish running and periodically check the inference thread of the model. model_thread_finished = False while not model_thread_finished: while len(global_text) > 0: @@ -245,7 +407,6 @@ if __name__ == "__main__": ) return jsonify(rkllm_responses), 200 else: - # 在这里处理收到的数据 messages = data['messages'] print("Received messages:", messages) for index, message in enumerate(messages): @@ -253,11 +414,9 @@ if __name__ == "__main__": rkllm_output = "" def generate(): - # 创建模型推理的线程 model_thread = threading.Thread(target=rkllm_model.run, args=(input_prompt,)) model_thread.start() - # 等待模型运行完成，定时检查模型的推理线程 model_thread_finished = False while not model_thread_finished: while len(global_text) > 0: @@ -282,16 +441,14 @@ if __name__ == "__main__": else: return jsonify({'status': 'error', 'message': 'Invalid JSON data!'}), 400 finally: - # 释放锁 lock.release() - # 将服务器状态设置为非阻塞 is_blocking = False - # 启动 Flask 应用程序 + # Start the Flask application. # app.run(host='0.0.0.0', port=8080) app.run(host='0.0.0.0', port=8080, threaded=True, debug=False) print("====================") - print("RKLLM模型推理结束, 释放RKLLM模型资源...") + print("RKLLM model inference completed, releasing RKLLM model resources...") rkllm_model.release() print("====================") diff --git a/rkllm-runtime/examples/rkllm_server_demo/rkllm_server/gradio_server.py b/rkllm-runtime/examples/rkllm_server_demo/rkllm_server/gradio_server.py index 9e56446..351f344 100644 --- a/rkllm-runtime/examples/rkllm_server_demo/rkllm_server/gradio_server.py +++ b/rkllm-runtime/examples/rkllm_server_demo/rkllm_server/gradio_server.py @@ -8,63 +8,46 @@ import time import gradio as gr import argparse -# 设定环境变量 +PROMPT_TEXT_PREFIX = "<|im_start|>system You are a helpful assistant. <|im_end|> <|im_start|>user" +PROMPT_TEXT_POSTFIX = "<|im_end|><|im_start|>assistant" + +# Set environment variables os.environ["GRADIO_SERVER_NAME"] = "0.0.0.0" os.environ["GRADIO_SERVER_PORT"] = "8080" -# 设置动态库路径 +# Set the dynamic library path rkllm_lib = ctypes.CDLL('lib/librkllmrt.so') -# 定义全局变量，用于保存回调函数的输出，便于在gradio界面中输出 -global_text = [] -global_state = -1 -split_byte_data = bytes(b"") # 用于保存分割的字节数据 +# Define the structures from the library +RKLLM_Handle_t = ctypes.c_void_p +userdata = ctypes.c_void_p(None) -# 定义动态库中的结构体 -class Token(ctypes.Structure): +LLMCallState = ctypes.c_int +LLMCallState.RKLLM_RUN_NORMAL = 0 +LLMCallState.RKLLM_RUN_WAITING = 1 +LLMCallState.RKLLM_RUN_FINISH = 2 +LLMCallState.RKLLM_RUN_ERROR = 3 +LLMCallState.RKLLM_RUN_GET_LAST_HIDDEN_LAYER = 4 + +RKLLMInputMode = ctypes.c_int +RKLLMInputMode.RKLLM_INPUT_PROMPT = 0 +RKLLMInputMode.RKLLM_INPUT_TOKEN = 1 +RKLLMInputMode.RKLLM_INPUT_EMBED = 2 +RKLLMInputMode.RKLLM_INPUT_MULTIMODAL = 3 + +RKLLMInferMode = ctypes.c_int +RKLLMInferMode.RKLLM_INFER_GENERATE = 0 +RKLLMInferMode.RKLLM_INFER_GET_LAST_HIDDEN_LAYER = 1 + +class RKLLMExtendParam(ctypes.Structure): _fields_ = [ - ("logprob", ctypes.c_float), - ("id", ctypes.c_int32) + ("base_domain_id", ctypes.c_int32), + ("reserved", ctypes.c_uint8 * 112) ] -class RKLLMResult(ctypes.Structure): - _fields_ = [ - ("text", ctypes.c_char_p), - ("tokens", ctypes.POINTER(Token)), - ("num", ctypes.c_int32) - ] - -# 定义回调函数 -def callback(result, userdata, state): - global global_text, global_state, split_byte_data - if state == 0: - # 保存输出的token文本及RKLLM运行状态 - global_state = state - # 需要监控当前的字节数据是否完整，不完整则进行记录，后续进行解析 - try: - global_text.append((split_byte_data + result.contents.text).decode('utf-8')) - print((split_byte_data + result.contents.text).decode('utf-8'), end='') - split_byte_data = bytes(b"") - except: - split_byte_data += result.contents.text - sys.stdout.flush() - elif state == 1: - # 保存RKLLM运行状态 - global_state = state - print("\n") - sys.stdout.flush() - else: - print("run error") - -# Python端与C++端的回调函数连接 -callback_type = ctypes.CFUNCTYPE(None, ctypes.POINTER(RKLLMResult), ctypes.c_void_p, ctypes.c_int) -c_callback = callback_type(callback) - -# 定义动态库中的结构体 -class RKNNllmParam(ctypes.Structure): +class RKLLMParam(ctypes.Structure): _fields_ = [ ("model_path", ctypes.c_char_p), - ("num_npu_core", ctypes.c_int32), ("max_context_len", ctypes.c_int32), ("max_new_tokens", ctypes.c_int32), ("top_k", ctypes.c_int32), @@ -76,60 +59,227 @@ class RKNNllmParam(ctypes.Structure): ("mirostat", ctypes.c_int32), ("mirostat_tau", ctypes.c_float), ("mirostat_eta", ctypes.c_float), - ("logprobs", ctypes.c_bool), - ("top_logprobs", ctypes.c_int32), - ("use_gpu", ctypes.c_bool) + ("skip_special_token", ctypes.c_bool), + ("is_async", ctypes.c_bool), + ("img_start", ctypes.c_char_p), + ("img_end", ctypes.c_char_p), + ("img_content", ctypes.c_char_p), + ("extend_param", RKLLMExtendParam), ] -# 定义RKLLM_Handle_t和userdata -RKLLM_Handle_t = ctypes.c_void_p -userdata = ctypes.c_void_p(None) +class RKLLMLoraAdapter(ctypes.Structure): + _fields_ = [ + ("lora_adapter_path", ctypes.c_char_p), + ("lora_adapter_name", ctypes.c_char_p), + ("scale", ctypes.c_float) + ] -# 设置提示文本 -PROMPT_TEXT_PREFIX = "<|im_start|>system You are a helpful assistant. <|im_end|> <|im_start|>user" -PROMPT_TEXT_POSTFIX = "<|im_end|><|im_start|>assistant" +class RKLLMEmbedInput(ctypes.Structure): + _fields_ = [ + ("embed", ctypes.POINTER(ctypes.c_float)), + ("n_tokens", ctypes.c_size_t) + ] -# 定义Python端的RKLLM类，其中包括了对动态库中RKLLM模型的初始化、推理及释放操作 +class RKLLMTokenInput(ctypes.Structure): + _fields_ = [ + ("input_ids", ctypes.POINTER(ctypes.c_int32)), + ("n_tokens", ctypes.c_size_t) + ] + +class RKLLMMultiModelInput(ctypes.Structure): + _fields_ = [ + ("prompt", ctypes.c_char_p), + ("image_embed", ctypes.POINTER(ctypes.c_float)), + ("n_image_tokens", ctypes.c_size_t) + ] + +class RKLLMInputUnion(ctypes.Union): + _fields_ = [ + ("prompt_input", ctypes.c_char_p), + ("embed_input", RKLLMEmbedInput), + ("token_input", RKLLMTokenInput), + ("multimodal_input", RKLLMMultiModelInput) + ] + +class RKLLMInput(ctypes.Structure): + _fields_ = [ + ("input_mode", ctypes.c_int), + ("input_data", RKLLMInputUnion) + ] + +class RKLLMLoraParam(ctypes.Structure): + _fields_ = [ + ("lora_adapter_name", ctypes.c_char_p) + ] + +class RKLLMPromptCacheParam(ctypes.Structure): + _fields_ = [ + ("save_prompt_cache", ctypes.c_int), + ("prompt_cache_path", ctypes.c_char_p) + ] + +class RKLLMInferParam(ctypes.Structure): + _fields_ = [ + ("mode", RKLLMInferMode), + ("lora_params", ctypes.POINTER(RKLLMLoraParam)), + ("prompt_cache_params", ctypes.POINTER(RKLLMPromptCacheParam)) + ] + +class RKLLMResultLastHiddenLayer(ctypes.Structure): + _fields_ = [ + ("hidden_states", ctypes.POINTER(ctypes.c_float)), + ("embd_size", ctypes.c_int), + ("num_tokens", ctypes.c_int) + ] + +class RKLLMResult(ctypes.Structure): + _fields_ = [ + ("text", ctypes.c_char_p), + ("size", ctypes.c_int), + ("last_hidden_layer", RKLLMResultLastHiddenLayer) + ] + +# Define global variables to store the callback function output for displaying in the Gradio interface +global_text = [] +global_state = -1 +split_byte_data = bytes(b"") # Used to store the segmented byte data + +# Define the callback function +def callback_impl(result, userdata, state): + global global_text, global_state, split_byte_data + if state == LLMCallState.RKLLM_RUN_FINISH: + global_state = state + print("\n") + sys.stdout.flush() + elif state == LLMCallState.RKLLM_RUN_ERROR: + global_state = state + print("run error") + sys.stdout.flush() + elif state == LLMCallState.RKLLM_RUN_GET_LAST_HIDDEN_LAYER: + ''' + If using the GET_LAST_HIDDEN_LAYER function, the callback interface will return the memory pointer: last_hidden_layer, the number of tokens: num_tokens, and the size of the hidden layer: embd_size. + With these three parameters, you can retrieve the data from last_hidden_layer. + Note: The data needs to be retrieved during the current callback; if not obtained in time, the pointer will be released by the next callback. + ''' + if result.last_hidden_layer.embd_size != 0 and result.last_hidden_layer.num_tokens != 0: + data_size = result.last_hidden_layer.embd_size * result.last_hidden_layer.num_tokens * ctypes.sizeof(ctypes.c_float) + print(f"data_size: {data_size}") + global_text.append(f"data_size: {data_size}\n") + output_path = os.getcwd() + "/last_hidden_layer.bin" + with open(output_path, "wb") as outFile: + data = ctypes.cast(result.last_hidden_layer.hidden_states, ctypes.POINTER(ctypes.c_float)) + float_array_type = ctypes.c_float * (data_size // ctypes.sizeof(ctypes.c_float)) + float_array = float_array_type.from_address(ctypes.addressof(data.contents)) + outFile.write(bytearray(float_array)) + print(f"Data saved to {output_path} successfully!") + global_text.append(f"Data saved to {output_path} successfully!") + else: + print("Invalid hidden layer data.") + global_text.append("Invalid hidden layer data.") + global_state = state + time.sleep(0.05) # Delay for 0.05 seconds to wait for the output result + sys.stdout.flush() + else: + # Save the output token text and the RKLLM running state + global_state = state + # Monitor if the current byte data is complete; if incomplete, record it for later parsing + try: + global_text.append((split_byte_data + result.contents.text).decode('utf-8')) + print((split_byte_data + result.contents.text).decode('utf-8'), end='') + split_byte_data = bytes(b"") + except: + split_byte_data += result.contents.text + sys.stdout.flush() + +# Connect the callback function between the Python side and the C++ side +callback_type = ctypes.CFUNCTYPE(None, ctypes.POINTER(RKLLMResult), ctypes.c_void_p, ctypes.c_int) +callback = callback_type(callback_impl) + +# Define the RKLLM class, which includes initialization, inference, and release operations for the RKLLM model in the dynamic library class RKLLM(object): - def __init__(self, model_path, target_platform): - rknnllm_param = RKNNllmParam() - rknnllm_param.model_path = bytes(model_path, 'utf-8') - if target_platform == "rk3588": - rknnllm_param.num_npu_core = 3 - elif target_platform == "rk3576": - rknnllm_param.num_npu_core = 2 - rknnllm_param.max_context_len = 320 - rknnllm_param.max_new_tokens = 512 - rknnllm_param.top_k = 1 - rknnllm_param.top_p = 0.9 - rknnllm_param.temperature = 0.8 - rknnllm_param.repeat_penalty = 1.1 - rknnllm_param.frequency_penalty = 0.0 - rknnllm_param.presence_penalty = 0.0 - rknnllm_param.mirostat = 0 - rknnllm_param.mirostat_tau = 5.0 - rknnllm_param.mirostat_eta = 0.1 - rknnllm_param.logprobs = False - rknnllm_param.top_logprobs = 5 - rknnllm_param.use_gpu = True + def __init__(self, model_path, lora_model_path = None, prompt_cache_path = None): + rkllm_param = RKLLMParam() + rkllm_param.model_path = bytes(model_path, 'utf-8') + + rkllm_param.max_context_len = 512 + rkllm_param.max_new_tokens = -1 + rkllm_param.skip_special_token = True + + rkllm_param.top_k = 1 + rkllm_param.top_p = 0.9 + rkllm_param.temperature = 0.8 + rkllm_param.repeat_penalty = 1.1 + rkllm_param.frequency_penalty = 0.0 + rkllm_param.presence_penalty = 0.0 + + rkllm_param.mirostat = 0 + rkllm_param.mirostat_tau = 5.0 + rkllm_param.mirostat_eta = 0.1 + + rkllm_param.is_async = False + + rkllm_param.img_start = "".encode('utf-8') + rkllm_param.img_end = "".encode('utf-8') + rkllm_param.img_content = "".encode('utf-8') + + rkllm_param.extend_param.base_domain_id = 0 + self.handle = RKLLM_Handle_t() self.rkllm_init = rkllm_lib.rkllm_init - self.rkllm_init.argtypes = [ctypes.POINTER(RKLLM_Handle_t), ctypes.POINTER(RKNNllmParam), callback_type] + self.rkllm_init.argtypes = [ctypes.POINTER(RKLLM_Handle_t), ctypes.POINTER(RKLLMParam), callback_type] self.rkllm_init.restype = ctypes.c_int - self.rkllm_init(ctypes.byref(self.handle), rknnllm_param, c_callback) + self.rkllm_init(ctypes.byref(self.handle), ctypes.byref(rkllm_param), callback) self.rkllm_run = rkllm_lib.rkllm_run - self.rkllm_run.argtypes = [RKLLM_Handle_t, ctypes.POINTER(ctypes.c_char), ctypes.c_void_p] + self.rkllm_run.argtypes = [RKLLM_Handle_t, ctypes.POINTER(RKLLMInput), ctypes.POINTER(RKLLMInferParam), ctypes.c_void_p] self.rkllm_run.restype = ctypes.c_int self.rkllm_destroy = rkllm_lib.rkllm_destroy self.rkllm_destroy.argtypes = [RKLLM_Handle_t] self.rkllm_destroy.restype = ctypes.c_int + self.lora_adapter_path = None + self.lora_model_name = None + if lora_model_path: + self.lora_adapter_path = lora_model_path + self.lora_adapter_name = "test" + + lora_adapter = RKLLMLoraAdapter() + ctypes.memset(ctypes.byref(lora_adapter), 0, ctypes.sizeof(RKLLMLoraAdapter)) + lora_adapter.lora_adapter_path = ctypes.c_char_p((self.lora_adapter_path).encode('utf-8')) + lora_adapter.lora_adapter_name = ctypes.c_char_p((self.lora_adapter_name).encode('utf-8')) + lora_adapter.scale = 1.0 + + rkllm_load_lora = rkllm_lib.rkllm_load_lora + rkllm_load_lora.argtypes = [RKLLM_Handle_t, ctypes.POINTER(RKLLMLoraAdapter)] + rkllm_load_lora.restype = ctypes.c_int + rkllm_load_lora(self.handle, ctypes.byref(lora_adapter)) + + self.prompt_cache_path = None + if prompt_cache_path: + self.prompt_cache_path = prompt_cache_path + + rkllm_load_prompt_cache = rkllm_lib.rkllm_load_prompt_cache + rkllm_load_prompt_cache.argtypes = [RKLLM_Handle_t, ctypes.c_char_p] + rkllm_load_prompt_cache.restype = ctypes.c_int + rkllm_load_prompt_cache(self.handle, ctypes.c_char_p((prompt_cache_path).encode('utf-8'))) + def run(self, prompt): - prompt = bytes(PROMPT_TEXT_PREFIX + prompt + PROMPT_TEXT_POSTFIX, 'utf-8') - self.rkllm_run(self.handle, prompt, ctypes.byref(userdata)) + rkllm_lora_params = None + if self.lora_model_name: + rkllm_lora_params = RKLLMLoraParam() + rkllm_lora_params.lora_adapter_name = ctypes.c_char_p((self.lora_model_name).encode('utf-8')) + + rkllm_infer_params = RKLLMInferParam() + ctypes.memset(ctypes.byref(rkllm_infer_params), 0, ctypes.sizeof(RKLLMInferParam)) + rkllm_infer_params.mode = RKLLMInferMode.RKLLM_INFER_GENERATE + rkllm_infer_params.lora_params = ctypes.byref(rkllm_lora_params) if rkllm_lora_params else None + + rkllm_input = RKLLMInput() + rkllm_input.input_mode = RKLLMInputMode.RKLLM_INPUT_PROMPT + rkllm_input.input_data.prompt_input = ctypes.c_char_p((PROMPT_TEXT_PREFIX + prompt + PROMPT_TEXT_POSTFIX).encode('utf-8')) + self.rkllm_run(self.handle, ctypes.byref(rkllm_input), ctypes.byref(rkllm_infer_params), None) return def release(self): @@ -137,92 +287,105 @@ class RKLLM(object): if __name__ == "__main__": parser = argparse.ArgumentParser() - parser.add_argument('--target_platform', help='目标平台: 如rk3588/rk3576;') - parser.add_argument('--rkllm_model_path', help='Linux板端上已转换好的rkllm模型的绝对路径') + parser.add_argument('--rkllm_model_path', type=str, required=True, help='Absolute path of the converted RKLLM model on the Linux board;') + parser.add_argument('--target_platform', type=str, required=True, help='Target platform: e.g., rk3588/rk3576;') + parser.add_argument('--lora_model_path', type=str, help='Absolute path of the lora_model on the Linux board;') + parser.add_argument('--prompt_cache_path', type=str, help='Absolute path of the prompt_cache file on the Linux board;') args = parser.parse_args() - if not (args.target_platform in ["rk3588", "rk3576"]): - print("====== Error: 请指定正确的目标平台: rk3588/rk3576 ======") - sys.stdout.flush() - exit() - if not os.path.exists(args.rkllm_model_path): - print("====== Error: 请给出准确的rkllm模型路径，需注意是板端的绝对路径 ======") + print("Error: Please provide the correct rkllm model path, and ensure it is the absolute path on the board.") sys.stdout.flush() exit() - # 定频设置 + if not (args.target_platform in ["rk3588", "rk3576"]): + print("Error: Please specify the correct target platform: rk3588/rk3576.") + sys.stdout.flush() + exit() + + if args.lora_model_path: + if not os.path.exists(args.lora_model_path): + print("Error: Please provide the correct lora_model path, and advise it is the absolute path on the board.") + sys.stdout.flush() + exit() + + if args.prompt_cache_path: + if not os.path.exists(args.prompt_cache_path): + print("Error: Please provide the correct prompt_cache_file path, and advise it is the absolute path on the board.") + sys.stdout.flush() + exit() + + # Fix frequency command = "sudo bash fix_freq_{}.sh".format(args.target_platform) subprocess.run(command, shell=True) - # 设置文件描述符限制 + # Set resource limit resource.setrlimit(resource.RLIMIT_NOFILE, (102400, 102400)) - # 初始化RKLLM模型 + # Initialize RKLLM model print("=========init....===========") sys.stdout.flush() - target_platform = args.target_platform model_path = args.rkllm_model_path - rkllm_model = RKLLM(model_path, target_platform) - print("RKLLM初始化成功！") + rkllm_model = RKLLM(model_path, args.lora_model_path, args.prompt_cache_path) + print("RKLLM Model has been initialized successfully！") print("==============================") sys.stdout.flush() - # 记录用户输入的prompt + # Record the user's input prompt def get_user_input(user_message, history): history = history + [[user_message, None]] return "", history - # 获取RKLLM模型的输出并进行流式打印 + # Retrieve the output from the RKLLM model and print it in a streaming manner def get_RKLLM_output(history): - # 链接全局变量，获取回调函数的输出信息 + # Link global variables to retrieve the output information from the callback function global global_text, global_state global_text = [] global_state = -1 - # 创建模型推理的线程 + # Create a thread for model inference model_thread = threading.Thread(target=rkllm_model.run, args=(history[-1][0],)) model_thread.start() - # history[-1][1]表示当前的输出对话 + # history[-1][1] represents the current dialogue history[-1][1] = "" - # 等待模型运行完成，定时检查模型的推理线程 + # Wait for the model to finish running and periodically check the inference thread of the model model_thread_finished = False while not model_thread_finished: while len(global_text) > 0: history[-1][1] += global_text.pop(0) time.sleep(0.005) - # gradio在调用then方法式自动将yield返回的结果推进行输出 + # Gradio automatically pushes the result returned by the yield statement when calling the then method yield history model_thread.join(timeout=0.005) model_thread_finished = not model_thread.is_alive() - # 创建gradio界面 + # Create a Gradio interface with gr.Blocks(title="Chat with RKLLM") as chatRKLLM: gr.Markdown("
Chat with RKLLM
") - gr.Markdown("### 在 inputTextBox 输入您的问题，按下 Enter 键，即可与 RKLLM 模型进行对话。") - # 创建一个Chatbot组件，用于显示对话历史 + gr.Markdown("### Enter your question in the inputTextBox and press the Enter key to chat with the RKLLM model.") + # Create a Chatbot component to display conversation history rkllmServer = gr.Chatbot(height=600) - # 创建一个Textbox组件，让用户输入消息 + # Create a Textbox component for user message input msg = gr.Textbox(placeholder="Please input your question here...", label="inputTextBox") - # 创建一个Button组件，用于清除聊天历史 - clear = gr.Button("清除") + # Create a Button component to clear the chat history. + clear = gr.Button("Clear") - # 将用户输入的消息提交给get_user_input函数，并且立即更新聊天历史 - # 然后调用get_RKLLM_output函数，进一步更新聊天历史 - # queue=False参数确保这些更新不会被放入队列，而是立即执行 + # Submit the user's input message to the get_user_input function and immediately update the chat history. + # Then call the get_RKLLM_output function to further update the chat history. + # The queue=False parameter ensures that these updates are not queued, but executed immediately. msg.submit(get_user_input, [msg, rkllmServer], [msg, rkllmServer], queue=False).then(get_RKLLM_output, rkllmServer, rkllmServer) - # 当点击清除按钮时，执行一个空操作（lambda: None），并且立即清除聊天历史 + # When the clear button is clicked, perform a no-operation (lambda: None) and immediately clear the chat history. clear.click(lambda: None, None, rkllmServer, queue=False) - # 启用事件队列系统 + # Enable the event queue system. chatRKLLM.queue() - # 启动Gradio应用程序 + # Start the Gradio application. chatRKLLM.launch() print("====================") - print("RKLLM模型推理结束, 释放RKLLM模型资源...") + print("RKLLM model inference completed, releasing RKLLM model resources...") rkllm_model.release() print("====================") \ No newline at end of file diff --git a/rkllm-runtime/runtime/Android/librkllm_api/arm64-v8a/librkllmrt.so b/rkllm-runtime/runtime/Android/librkllm_api/arm64-v8a/librkllmrt.so index 2f7e7e9..9cb37c7 100755 Binary files a/rkllm-runtime/runtime/Android/librkllm_api/arm64-v8a/librkllmrt.so and b/rkllm-runtime/runtime/Android/librkllm_api/arm64-v8a/librkllmrt.so differ diff --git a/rkllm-runtime/runtime/Android/librkllm_api/include/rkllm.h b/rkllm-runtime/runtime/Android/librkllm_api/include/rkllm.h index ce1b2ff..e565e6c 100644 --- a/rkllm-runtime/runtime/Android/librkllm_api/include/rkllm.h +++ b/rkllm-runtime/runtime/Android/librkllm_api/include/rkllm.h @@ -1,119 +1,271 @@ -#ifndef _LLM_H_ -#define _LLM_H_ +#ifndef _RKLLM_H_ +#define _RKLLM_H_ #ifdef __cplusplus extern "C" { #endif -typedef void* LLMHandle; /* Handle for an instance of a language model. */ +/** + * @typedef LLMHandle + * @brief A handle used to manage and interact with the large language model. + */ +typedef void* LLMHandle; /** - * @brief Structure for possible states of an inference call. - * + * @enum LLMCallState + * @brief Describes the possible states of an LLM call. */ typedef enum { - LLM_RUN_NORMAL = 0, /* Inference status is normal and inference has not yet finished. */ - LLM_RUN_FINISH = 1, /* Inference status is normal and inference has finished. */ - LLM_RUN_ERROR = 2 /* Inference status is abnormal. */ + RKLLM_RUN_NORMAL = 0, /**< The LLM call is in a normal running state. */ + RKLLM_RUN_WAITING = 1, /**< The LLM call is waiting for complete UTF-8 encoded character. */ + RKLLM_RUN_FINISH = 2, /**< The LLM call has finished execution. */ + RKLLM_RUN_ERROR = 3, /**< An error occurred during the LLM call. */ + RKLLM_RUN_GET_LAST_HIDDEN_LAYER = 4 /**< Retrieve the last hidden layer during inference. */ } LLMCallState; /** - * @brief Structure for setting up parameters for the language model - * + * @enum RKLLMInputType + * @brief Defines the types of inputs that can be fed into the LLM. + */ +typedef enum { + RKLLM_INPUT_PROMPT = 0, /**< Input is a text prompt. */ + RKLLM_INPUT_TOKEN = 1, /**< Input is a sequence of tokens. */ + RKLLM_INPUT_EMBED = 2, /**< Input is an embedding vector. */ + RKLLM_INPUT_MULTIMODAL = 3, /**< Input is multimodal (e.g., text and image). */ +} RKLLMInputType; + +/** + * @enum RKLLMInferMode + * @brief Specifies the inference modes of the LLM. + */ +typedef enum { + RKLLM_INFER_GENERATE = 0, /**< The LLM generates text based on input. */ + RKLLM_INFER_GET_LAST_HIDDEN_LAYER = 1, /**< The LLM retrieves the last hidden layer for further processing. */ +} RKLLMInferMode; + +/** + * @struct RKLLMExtendParam + * @brief The extend parameters for configuring an LLM instance. */ typedef struct { - const char* model_path; /* Path where the model file is located. */ - int32_t num_npu_core; /* Number of NPU cores used for model inference. */ - int32_t max_context_len; /* Maximum size of the context. */ - int32_t max_new_tokens; /* Maximum number of tokens to generate during model inference. */ - int32_t top_k; /* The number of highest probability tokens to consider for generation. */ - float top_p; /* Nucleus sampling: cumulative probability cutoff to use for token selection. */ - float temperature; /* Hyperparameter to control the randomness of predictions by scaling the logits before applying softmax. */ - float repeat_penalty; /* Penalty applied to the logits of previously generated tokens, helps prevent repetitive or monotonic text. */ - float frequency_penalty; /* Penalty for repeating the same word or phrase, reducing the likelihood of repeated content. */ - float presence_penalty; /* Penalty or reward for introducing new tokens into the generated text. */ - int32_t mirostat; /* Enables mirostat algorithm, where 0 = off, 1 = use mirostat algorithm, 2 = use mirostat 2.0 algorithm. */ - float mirostat_tau; /* Target entropy (perplexity) for mirostat algorithm, setting the desired complexity of the generated text. */ - float mirostat_eta; /* Learning rate for the mirostat algorithm. */ - bool logprobs; /* Whether to return the log probabilities for each output token along with their token ids. */ - int32_t top_logprobs; /* The number of top tokens for which to return log probabilities, along with their token ids. */ - bool use_gpu; /* Flag to indicate whether to use GPU for inference. */ + int32_t base_domain_id; /**< base_domain_id */ + uint8_t reserved[112]; /**< reserved */ +} RKLLMExtendParam; + +/** + * @struct RKLLMParam + * @brief Defines the parameters for configuring an LLM instance. + */ +typedef struct { + const char* model_path; /**< Path to the model file. */ + int32_t max_context_len; /**< Maximum number of tokens in the context window. */ + int32_t max_new_tokens; /**< Maximum number of new tokens to generate. */ + int32_t top_k; /**< Top-K sampling parameter for token generation. */ + float top_p; /**< Top-P (nucleus) sampling parameter. */ + float temperature; /**< Sampling temperature, affecting the randomness of token selection. */ + float repeat_penalty; /**< Penalty for repeating tokens in generation. */ + float frequency_penalty; /**< Penalizes frequent tokens during generation. */ + float presence_penalty; /**< Penalizes tokens based on their presence in the input. */ + int32_t mirostat; /**< Mirostat sampling strategy flag (0 to disable). */ + float mirostat_tau; /**< Tau parameter for Mirostat sampling. */ + float mirostat_eta; /**< Eta parameter for Mirostat sampling. */ + bool skip_special_token; /**< Whether to skip special tokens during generation. */ + bool is_async; /**< Whether to run inference asynchronously. */ + const char* img_start; /**< Starting position of an image in multimodal input. */ + const char* img_end; /**< Ending position of an image in multimodal input. */ + const char* img_content; /**< Pointer to the image content. */ + RKLLMExtendParam extend_param; /**< Extend parameters. */ } RKLLMParam; /** - * @brief Structure representing a token with its associated log probability. - * + * @struct RKLLMLoraAdapter + * @brief Defines parameters for a Lora adapter used in model fine-tuning. */ typedef struct { - float logprob; /* Log probability corresponding to the token ID. */ - int id; /* Token ID. */ -} Token; + const char* lora_adapter_path; /**< Path to the Lora adapter file. */ + const char* lora_adapter_name; /**< Name of the Lora adapter. */ + float scale; /**< Scaling factor for applying the Lora adapter. */ +} RKLLMLoraAdapter; /** - * @brief Structure to hold the results from the language model inference, including text and token details. - * + * @struct RKLLMEmbedInput + * @brief Represents an embedding input to the LLM. */ typedef struct { - const char* text; /* Decoded text from the inference output. */ - Token* tokens; /* Array of Token structures, each containing a log probability and a token ID. */ - int num; /* Number of top tokens returned, typically those with the highest probabilities. */ + float* embed; /**< Pointer to the embedding vector (of size n_tokens * n_embed). */ + size_t n_tokens; /**< Number of tokens represented in the embedding. */ +} RKLLMEmbedInput; + +/** + * @struct RKLLMTokenInput + * @brief Represents token input to the LLM. + */ +typedef struct { + int32_t* input_ids; /**< Array of token IDs. */ + size_t n_tokens; /**< Number of tokens in the input. */ +} RKLLMTokenInput; + +/** + * @struct RKLLMMultiModelInput + * @brief Represents multimodal input (e.g., text and image). + */ +typedef struct { + char* prompt; /**< Text prompt input. */ + float* image_embed; /**< Embedding of the image (of size n_image_tokens * n_image_embed). */ + size_t n_image_tokens; /**< Number of image tokens. */ +} RKLLMMultiModelInput; + +/** + * @struct RKLLMInput + * @brief Represents different types of input to the LLM via a union. + */ +typedef struct { + RKLLMInputType input_type; /**< Specifies the type of input provided (e.g., prompt, token, embed, multimodal). */ + union { + const char* prompt_input; /**< Text prompt input if input_type is RKLLM_INPUT_PROMPT. */ + RKLLMEmbedInput embed_input; /**< Embedding input if input_type is RKLLM_INPUT_EMBED. */ + RKLLMTokenInput token_input; /**< Token input if input_type is RKLLM_INPUT_TOKEN. */ + RKLLMMultiModelInput multimodal_input; /**< Multimodal input if input_type is RKLLM_INPUT_MULTIMODAL. */ + }; +} RKLLMInput; + +/** + * @struct RKLLMLoraParam + * @brief Structure defining parameters for Lora adapters. + */ +typedef struct { + const char* lora_adapter_name; /**< Name of the Lora adapter. */ +} RKLLMLoraParam; + +/** + * @struct RKLLMPromptCacheParam + * @brief Structure to define parameters for caching prompts. + */ +typedef struct { + int save_prompt_cache; /**< Flag to indicate whether to save the prompt cache (0 = don't save, 1 = save). */ + const char* prompt_cache_path; /**< Path to the prompt cache file. */ +} RKLLMPromptCacheParam; + +/** + * @struct RKLLMInferParam + * @brief Structure for defining parameters during inference. + */ +typedef struct { + RKLLMInferMode mode; /**< Inference mode (e.g., generate or get last hidden layer). */ + RKLLMLoraParam* lora_params; /**< Pointer to Lora adapter parameters. */ + RKLLMPromptCacheParam* prompt_cache_params; /**< Pointer to prompt cache parameters. */ +} RKLLMInferParam; + +/** + * @struct RKLLMResultLastHiddenLayer + * @brief Structure to hold the hidden states from the last layer. + */ +typedef struct { + const float* hidden_states; /**< Pointer to the hidden states (of size num_tokens * embd_size). */ + int embd_size; /**< Size of the embedding vector. */ + int num_tokens; /**< Number of tokens for which hidden states are stored. */ +} RKLLMResultLastHiddenLayer; + +/** + * @struct RKLLMResult + * @brief Structure to represent the result of LLM inference. + */ +typedef struct { + const char* text; /**< Generated text result. */ + int32_t token_id; /**< ID of the generated token. */ + RKLLMResultLastHiddenLayer last_hidden_layer; /**< Hidden states of the last layer (if requested). */ } RKLLMResult; /** - * @brief Callback function for handling inference results. - * - * @param result A pointer to an RKLLMResult struct containing the inference results. - * @param userdata A pointer to user-defined function or null if no user function was provided. - * @param state The state of the inference process, indicating success, failure, or completion. + * @typedef LLMResultCallback + * @brief Callback function to handle LLM results. + * @param result Pointer to the LLM result. + * @param userdata Pointer to user data for the callback. + * @param state State of the LLM call (e.g., finished, error). */ typedef void(*LLMResultCallback)(RKLLMResult* result, void* userdata, LLMCallState state); /** - * @brief Initializes RKLLMParam with default settings. - * - * @return RKLLMParam An RKLLMParam struct with default values set. + * @brief Creates a default RKLLMParam structure with preset values. + * @return A default RKLLMParam structure. */ RKLLMParam rkllm_createDefaultParam(); /** - * @brief Initializes the model with specified parameters. - * - * @param handle Pointer to a handle for the language model, which will be initialized by this function. - * @param param An RKLLMParam struct containing all the parameters needed for the model. - * @param callback A function pointer to the callback that handles the results of the inference. - * @return int Returns 0 on success, or a negative error code on failure. + * @brief Initializes the LLM with the given parameters. + * @param handle Pointer to the LLM handle. + * @param param Configuration parameters for the LLM. + * @param callback Callback function to handle LLM results. + * @return Status code (0 for success, non-zero for failure). */ -int rkllm_init(LLMHandle* handle, RKLLMParam param, LLMResultCallback callback); +int rkllm_init(LLMHandle* handle, RKLLMParam* param, LLMResultCallback callback); /** - * @brief Releases the model resources. - * - * @param handle The handle to the language model to be destroyed. - * @return int Returns 0 on successful release, or a negative error code if an error occurs. + * @brief Loads a Lora adapter into the LLM. + * @param handle LLM handle. + * @param lora_adapter Pointer to the Lora adapter structure. + * @return Status code (0 for success, non-zero for failure). + */ +int rkllm_load_lora(LLMHandle handle, RKLLMLoraAdapter* lora_adapter); + +/** + * @brief Loads a prompt cache from a file. + * @param handle LLM handle. + * @param prompt_cache_path Path to the prompt cache file. + * @return Status code (0 for success, non-zero for failure). + */ +int rkllm_load_prompt_cache(LLMHandle handle, const char* prompt_cache_path); + +/** + * @brief Releases the prompt cache from memory. + * @param handle LLM handle. + * @return Status code (0 for success, non-zero for failure). + */ +int rkllm_release_prompt_cache(LLMHandle handle); + +/** + * @brief Destroys the LLM instance and releases resources. + * @param handle LLM handle. + * @return Status code (0 for success, non-zero for failure). */ int rkllm_destroy(LLMHandle handle); /** - * @brief Runs model inference on the given prompt. - * - * @param handle The handle to the initialized language model. - * @param prompt The text prompt on which to perform inference. - * @param userdata Optional user-defined function that will be passed to the callback. - * @return int Returns 0 on success, or a negative error code if an error occurs during inference. + * @brief Runs an LLM inference task synchronously. + * @param handle LLM handle. + * @param rkllm_input Input data for the LLM. + * @param rkllm_infer_params Parameters for the inference task. + * @param userdata Pointer to user data for the callback. + * @return Status code (0 for success, non-zero for failure). */ -int rkllm_run(LLMHandle handle, const char* prompt, void* userdata); +int rkllm_run(LLMHandle handle, RKLLMInput* rkllm_input, RKLLMInferParam* rkllm_infer_params, void* userdata); /** - * @brief Aborts the current inference process. - * - * @param handle The handle to the language model whose inference is to be aborted. - * @return int Returns 0 if the process is successfully aborted, or a negative error code - * if no process was running or if the abort fails. + * @brief Runs an LLM inference task asynchronously. + * @param handle LLM handle. + * @param rkllm_input Input data for the LLM. + * @param rkllm_infer_params Parameters for the inference task. + * @param userdata Pointer to user data for the callback. + * @return Status code (0 for success, non-zero for failure). + */ +int rkllm_run_async(LLMHandle handle, RKLLMInput* rkllm_input, RKLLMInferParam* rkllm_infer_params, void* userdata); + +/** + * @brief Aborts an ongoing LLM task. + * @param handle LLM handle. + * @return Status code (0 for success, non-zero for failure). */ int rkllm_abort(LLMHandle handle); +/** + * @brief Checks if an LLM task is currently running. + * @param handle LLM handle. + * @return Status code (0 if a task is running, non-zero for otherwise). + */ +int rkllm_is_running(LLMHandle handle); + #ifdef __cplusplus -} //extern "C" +} #endif -#endif \ No newline at end of file +#endif diff --git a/rkllm-runtime/runtime/Linux/librkllm_api/aarch64/librkllmrt.so b/rkllm-runtime/runtime/Linux/librkllm_api/aarch64/librkllmrt.so index c913ab1..bfca051 100755 Binary files a/rkllm-runtime/runtime/Linux/librkllm_api/aarch64/librkllmrt.so and b/rkllm-runtime/runtime/Linux/librkllm_api/aarch64/librkllmrt.so differ diff --git a/rkllm-runtime/runtime/Linux/librkllm_api/include/rkllm.h b/rkllm-runtime/runtime/Linux/librkllm_api/include/rkllm.h index ce1b2ff..e565e6c 100644 --- a/rkllm-runtime/runtime/Linux/librkllm_api/include/rkllm.h +++ b/rkllm-runtime/runtime/Linux/librkllm_api/include/rkllm.h @@ -1,119 +1,271 @@ -#ifndef _LLM_H_ -#define _LLM_H_ +#ifndef _RKLLM_H_ +#define _RKLLM_H_ #ifdef __cplusplus extern "C" { #endif -typedef void* LLMHandle; /* Handle for an instance of a language model. */ +/** + * @typedef LLMHandle + * @brief A handle used to manage and interact with the large language model. + */ +typedef void* LLMHandle; /** - * @brief Structure for possible states of an inference call. - * + * @enum LLMCallState + * @brief Describes the possible states of an LLM call. */ typedef enum { - LLM_RUN_NORMAL = 0, /* Inference status is normal and inference has not yet finished. */ - LLM_RUN_FINISH = 1, /* Inference status is normal and inference has finished. */ - LLM_RUN_ERROR = 2 /* Inference status is abnormal. */ + RKLLM_RUN_NORMAL = 0, /**< The LLM call is in a normal running state. */ + RKLLM_RUN_WAITING = 1, /**< The LLM call is waiting for complete UTF-8 encoded character. */ + RKLLM_RUN_FINISH = 2, /**< The LLM call has finished execution. */ + RKLLM_RUN_ERROR = 3, /**< An error occurred during the LLM call. */ + RKLLM_RUN_GET_LAST_HIDDEN_LAYER = 4 /**< Retrieve the last hidden layer during inference. */ } LLMCallState; /** - * @brief Structure for setting up parameters for the language model - * + * @enum RKLLMInputType + * @brief Defines the types of inputs that can be fed into the LLM. + */ +typedef enum { + RKLLM_INPUT_PROMPT = 0, /**< Input is a text prompt. */ + RKLLM_INPUT_TOKEN = 1, /**< Input is a sequence of tokens. */ + RKLLM_INPUT_EMBED = 2, /**< Input is an embedding vector. */ + RKLLM_INPUT_MULTIMODAL = 3, /**< Input is multimodal (e.g., text and image). */ +} RKLLMInputType; + +/** + * @enum RKLLMInferMode + * @brief Specifies the inference modes of the LLM. + */ +typedef enum { + RKLLM_INFER_GENERATE = 0, /**< The LLM generates text based on input. */ + RKLLM_INFER_GET_LAST_HIDDEN_LAYER = 1, /**< The LLM retrieves the last hidden layer for further processing. */ +} RKLLMInferMode; + +/** + * @struct RKLLMExtendParam + * @brief The extend parameters for configuring an LLM instance. */ typedef struct { - const char* model_path; /* Path where the model file is located. */ - int32_t num_npu_core; /* Number of NPU cores used for model inference. */ - int32_t max_context_len; /* Maximum size of the context. */ - int32_t max_new_tokens; /* Maximum number of tokens to generate during model inference. */ - int32_t top_k; /* The number of highest probability tokens to consider for generation. */ - float top_p; /* Nucleus sampling: cumulative probability cutoff to use for token selection. */ - float temperature; /* Hyperparameter to control the randomness of predictions by scaling the logits before applying softmax. */ - float repeat_penalty; /* Penalty applied to the logits of previously generated tokens, helps prevent repetitive or monotonic text. */ - float frequency_penalty; /* Penalty for repeating the same word or phrase, reducing the likelihood of repeated content. */ - float presence_penalty; /* Penalty or reward for introducing new tokens into the generated text. */ - int32_t mirostat; /* Enables mirostat algorithm, where 0 = off, 1 = use mirostat algorithm, 2 = use mirostat 2.0 algorithm. */ - float mirostat_tau; /* Target entropy (perplexity) for mirostat algorithm, setting the desired complexity of the generated text. */ - float mirostat_eta; /* Learning rate for the mirostat algorithm. */ - bool logprobs; /* Whether to return the log probabilities for each output token along with their token ids. */ - int32_t top_logprobs; /* The number of top tokens for which to return log probabilities, along with their token ids. */ - bool use_gpu; /* Flag to indicate whether to use GPU for inference. */ + int32_t base_domain_id; /**< base_domain_id */ + uint8_t reserved[112]; /**< reserved */ +} RKLLMExtendParam; + +/** + * @struct RKLLMParam + * @brief Defines the parameters for configuring an LLM instance. + */ +typedef struct { + const char* model_path; /**< Path to the model file. */ + int32_t max_context_len; /**< Maximum number of tokens in the context window. */ + int32_t max_new_tokens; /**< Maximum number of new tokens to generate. */ + int32_t top_k; /**< Top-K sampling parameter for token generation. */ + float top_p; /**< Top-P (nucleus) sampling parameter. */ + float temperature; /**< Sampling temperature, affecting the randomness of token selection. */ + float repeat_penalty; /**< Penalty for repeating tokens in generation. */ + float frequency_penalty; /**< Penalizes frequent tokens during generation. */ + float presence_penalty; /**< Penalizes tokens based on their presence in the input. */ + int32_t mirostat; /**< Mirostat sampling strategy flag (0 to disable). */ + float mirostat_tau; /**< Tau parameter for Mirostat sampling. */ + float mirostat_eta; /**< Eta parameter for Mirostat sampling. */ + bool skip_special_token; /**< Whether to skip special tokens during generation. */ + bool is_async; /**< Whether to run inference asynchronously. */ + const char* img_start; /**< Starting position of an image in multimodal input. */ + const char* img_end; /**< Ending position of an image in multimodal input. */ + const char* img_content; /**< Pointer to the image content. */ + RKLLMExtendParam extend_param; /**< Extend parameters. */ } RKLLMParam; /** - * @brief Structure representing a token with its associated log probability. - * + * @struct RKLLMLoraAdapter + * @brief Defines parameters for a Lora adapter used in model fine-tuning. */ typedef struct { - float logprob; /* Log probability corresponding to the token ID. */ - int id; /* Token ID. */ -} Token; + const char* lora_adapter_path; /**< Path to the Lora adapter file. */ + const char* lora_adapter_name; /**< Name of the Lora adapter. */ + float scale; /**< Scaling factor for applying the Lora adapter. */ +} RKLLMLoraAdapter; /** - * @brief Structure to hold the results from the language model inference, including text and token details. - * + * @struct RKLLMEmbedInput + * @brief Represents an embedding input to the LLM. */ typedef struct { - const char* text; /* Decoded text from the inference output. */ - Token* tokens; /* Array of Token structures, each containing a log probability and a token ID. */ - int num; /* Number of top tokens returned, typically those with the highest probabilities. */ + float* embed; /**< Pointer to the embedding vector (of size n_tokens * n_embed). */ + size_t n_tokens; /**< Number of tokens represented in the embedding. */ +} RKLLMEmbedInput; + +/** + * @struct RKLLMTokenInput + * @brief Represents token input to the LLM. + */ +typedef struct { + int32_t* input_ids; /**< Array of token IDs. */ + size_t n_tokens; /**< Number of tokens in the input. */ +} RKLLMTokenInput; + +/** + * @struct RKLLMMultiModelInput + * @brief Represents multimodal input (e.g., text and image). + */ +typedef struct { + char* prompt; /**< Text prompt input. */ + float* image_embed; /**< Embedding of the image (of size n_image_tokens * n_image_embed). */ + size_t n_image_tokens; /**< Number of image tokens. */ +} RKLLMMultiModelInput; + +/** + * @struct RKLLMInput + * @brief Represents different types of input to the LLM via a union. + */ +typedef struct { + RKLLMInputType input_type; /**< Specifies the type of input provided (e.g., prompt, token, embed, multimodal). */ + union { + const char* prompt_input; /**< Text prompt input if input_type is RKLLM_INPUT_PROMPT. */ + RKLLMEmbedInput embed_input; /**< Embedding input if input_type is RKLLM_INPUT_EMBED. */ + RKLLMTokenInput token_input; /**< Token input if input_type is RKLLM_INPUT_TOKEN. */ + RKLLMMultiModelInput multimodal_input; /**< Multimodal input if input_type is RKLLM_INPUT_MULTIMODAL. */ + }; +} RKLLMInput; + +/** + * @struct RKLLMLoraParam + * @brief Structure defining parameters for Lora adapters. + */ +typedef struct { + const char* lora_adapter_name; /**< Name of the Lora adapter. */ +} RKLLMLoraParam; + +/** + * @struct RKLLMPromptCacheParam + * @brief Structure to define parameters for caching prompts. + */ +typedef struct { + int save_prompt_cache; /**< Flag to indicate whether to save the prompt cache (0 = don't save, 1 = save). */ + const char* prompt_cache_path; /**< Path to the prompt cache file. */ +} RKLLMPromptCacheParam; + +/** + * @struct RKLLMInferParam + * @brief Structure for defining parameters during inference. + */ +typedef struct { + RKLLMInferMode mode; /**< Inference mode (e.g., generate or get last hidden layer). */ + RKLLMLoraParam* lora_params; /**< Pointer to Lora adapter parameters. */ + RKLLMPromptCacheParam* prompt_cache_params; /**< Pointer to prompt cache parameters. */ +} RKLLMInferParam; + +/** + * @struct RKLLMResultLastHiddenLayer + * @brief Structure to hold the hidden states from the last layer. + */ +typedef struct { + const float* hidden_states; /**< Pointer to the hidden states (of size num_tokens * embd_size). */ + int embd_size; /**< Size of the embedding vector. */ + int num_tokens; /**< Number of tokens for which hidden states are stored. */ +} RKLLMResultLastHiddenLayer; + +/** + * @struct RKLLMResult + * @brief Structure to represent the result of LLM inference. + */ +typedef struct { + const char* text; /**< Generated text result. */ + int32_t token_id; /**< ID of the generated token. */ + RKLLMResultLastHiddenLayer last_hidden_layer; /**< Hidden states of the last layer (if requested). */ } RKLLMResult; /** - * @brief Callback function for handling inference results. - * - * @param result A pointer to an RKLLMResult struct containing the inference results. - * @param userdata A pointer to user-defined function or null if no user function was provided. - * @param state The state of the inference process, indicating success, failure, or completion. + * @typedef LLMResultCallback + * @brief Callback function to handle LLM results. + * @param result Pointer to the LLM result. + * @param userdata Pointer to user data for the callback. + * @param state State of the LLM call (e.g., finished, error). */ typedef void(*LLMResultCallback)(RKLLMResult* result, void* userdata, LLMCallState state); /** - * @brief Initializes RKLLMParam with default settings. - * - * @return RKLLMParam An RKLLMParam struct with default values set. + * @brief Creates a default RKLLMParam structure with preset values. + * @return A default RKLLMParam structure. */ RKLLMParam rkllm_createDefaultParam(); /** - * @brief Initializes the model with specified parameters. - * - * @param handle Pointer to a handle for the language model, which will be initialized by this function. - * @param param An RKLLMParam struct containing all the parameters needed for the model. - * @param callback A function pointer to the callback that handles the results of the inference. - * @return int Returns 0 on success, or a negative error code on failure. + * @brief Initializes the LLM with the given parameters. + * @param handle Pointer to the LLM handle. + * @param param Configuration parameters for the LLM. + * @param callback Callback function to handle LLM results. + * @return Status code (0 for success, non-zero for failure). */ -int rkllm_init(LLMHandle* handle, RKLLMParam param, LLMResultCallback callback); +int rkllm_init(LLMHandle* handle, RKLLMParam* param, LLMResultCallback callback); /** - * @brief Releases the model resources. - * - * @param handle The handle to the language model to be destroyed. - * @return int Returns 0 on successful release, or a negative error code if an error occurs. + * @brief Loads a Lora adapter into the LLM. + * @param handle LLM handle. + * @param lora_adapter Pointer to the Lora adapter structure. + * @return Status code (0 for success, non-zero for failure). + */ +int rkllm_load_lora(LLMHandle handle, RKLLMLoraAdapter* lora_adapter); + +/** + * @brief Loads a prompt cache from a file. + * @param handle LLM handle. + * @param prompt_cache_path Path to the prompt cache file. + * @return Status code (0 for success, non-zero for failure). + */ +int rkllm_load_prompt_cache(LLMHandle handle, const char* prompt_cache_path); + +/** + * @brief Releases the prompt cache from memory. + * @param handle LLM handle. + * @return Status code (0 for success, non-zero for failure). + */ +int rkllm_release_prompt_cache(LLMHandle handle); + +/** + * @brief Destroys the LLM instance and releases resources. + * @param handle LLM handle. + * @return Status code (0 for success, non-zero for failure). */ int rkllm_destroy(LLMHandle handle); /** - * @brief Runs model inference on the given prompt. - * - * @param handle The handle to the initialized language model. - * @param prompt The text prompt on which to perform inference. - * @param userdata Optional user-defined function that will be passed to the callback. - * @return int Returns 0 on success, or a negative error code if an error occurs during inference. + * @brief Runs an LLM inference task synchronously. + * @param handle LLM handle. + * @param rkllm_input Input data for the LLM. + * @param rkllm_infer_params Parameters for the inference task. + * @param userdata Pointer to user data for the callback. + * @return Status code (0 for success, non-zero for failure). */ -int rkllm_run(LLMHandle handle, const char* prompt, void* userdata); +int rkllm_run(LLMHandle handle, RKLLMInput* rkllm_input, RKLLMInferParam* rkllm_infer_params, void* userdata); /** - * @brief Aborts the current inference process. - * - * @param handle The handle to the language model whose inference is to be aborted. - * @return int Returns 0 if the process is successfully aborted, or a negative error code - * if no process was running or if the abort fails. + * @brief Runs an LLM inference task asynchronously. + * @param handle LLM handle. + * @param rkllm_input Input data for the LLM. + * @param rkllm_infer_params Parameters for the inference task. + * @param userdata Pointer to user data for the callback. + * @return Status code (0 for success, non-zero for failure). + */ +int rkllm_run_async(LLMHandle handle, RKLLMInput* rkllm_input, RKLLMInferParam* rkllm_infer_params, void* userdata); + +/** + * @brief Aborts an ongoing LLM task. + * @param handle LLM handle. + * @return Status code (0 for success, non-zero for failure). */ int rkllm_abort(LLMHandle handle); +/** + * @brief Checks if an LLM task is currently running. + * @param handle LLM handle. + * @return Status code (0 if a task is running, non-zero for otherwise). + */ +int rkllm_is_running(LLMHandle handle); + #ifdef __cplusplus -} //extern "C" +} #endif -#endif \ No newline at end of file +#endif diff --git a/rkllm-toolkit/examples/huggingface/test.py b/rkllm-toolkit/examples/huggingface/test.py deleted file mode 100644 index 11519fd..0000000 --- a/rkllm-toolkit/examples/huggingface/test.py +++ /dev/null @@ -1,28 +0,0 @@ -from rkllm.api import RKLLM - -''' -https://huggingface.co/Qwen/Qwen-1_8B-Chat -Download the Qwen model from the above website. -''' - -# Default option. Works only if in the same directory -modelpath = 'Qwen-1_8B-Chat' -llm = RKLLM() - -# Load model -ret = llm.load_huggingface(model = modelpath) -if ret != 0: - print('Load model failed!') - exit(ret) - -# Build model -ret = llm.build(do_quantization=True, optimization_level=1, quantized_dtype='w8a8', target_platform='rk3588') -if ret != 0: - print('Build model failed!') - exit(ret) - -# Export rknn model -ret = llm.export_rkllm("./qwen.rkllm") -if ret != 0: - print('Export model failed!') - exit(ret) diff --git a/rkllm-toolkit/examples/test.py b/rkllm-toolkit/examples/test.py new file mode 100755 index 0000000..a160b25 --- /dev/null +++ b/rkllm-toolkit/examples/test.py @@ -0,0 +1,86 @@ +from rkllm.api import RKLLM +from datasets import load_dataset +from transformers import AutoTokenizer +from tqdm import tqdm +import torch +from torch import nn +import os +# os.environ['CUDA_VISIBLE_DEVICES']='1' + +''' +https://huggingface.co/Qwen/Qwen-1_8B-Chat +从上面网址中下载Qwen模型 +''' + +modelpath = './path/to/model' +# modelpath = "./path/to/Qwen-1.8B-F16.gguf" +llm = RKLLM() + +# Load model +# Use 'export CUDA_VISIBLE_DEVICES=2' to specify GPU device +# options ['cpu', 'cuda'] +ret = llm.load_huggingface(model=modelpath, model_lora = None, device='cpu') +# ret = llm.load_gguf(model = modelpath) +if ret != 0: + print('Load model failed!') + exit(ret) + +# Build model +dataset = "./data_quant.json" +# Json file format, please note to add prompt in the input，like this: +# [{"input":"Human: 你好！\nAssistant: ", "target": "你好！我是人工智能助手KK！"},...] + +qparams = None +# qparams = 'gdq.qparams' # Use extra_qparams +ret = llm.build(do_quantization=True, optimization_level=1, quantized_dtype='w8a8', + quantized_algorithm='normal', target_platform='rk3588', num_npu_core=3, extra_qparams=qparams, dataset=dataset) + +if ret != 0: + print('Build model failed!') + exit(ret) + +# Evaluate Accuracy +def eval_wikitext(llm): + seqlen = 512 + tokenizer = AutoTokenizer.from_pretrained( + modelpath, trust_remote_code=True) + # Dataset download link: + # https://huggingface.co/datasets/Salesforce/wikitext/tree/main/wikitext-2-raw-v1 + testenc = load_dataset( + "parquet", data_files='./wikitext/wikitext-2-raw-1/test-00000-of-00001.parquet', split='train') + testenc = tokenizer("\n\n".join( + testenc['text']), return_tensors="pt").input_ids + nsamples = testenc.numel() // seqlen + nlls = [] + for i in tqdm(range(nsamples), desc="eval_wikitext: "): + batch = testenc[:, (i * seqlen): ((i + 1) * seqlen)] + inputs = {"input_ids": batch} + lm_logits = llm.get_logits(inputs) + if lm_logits is None: + print("get logits failed!") + return + shift_logits = lm_logits[:, :-1, :] + shift_labels = batch[:, 1:].to(lm_logits.device) + loss_fct = nn.CrossEntropyLoss().to(lm_logits.device) + loss = loss_fct( + shift_logits.view(-1, shift_logits.size(-1)), shift_labels.view(-1)) + neg_log_likelihood = loss.float() * seqlen + nlls.append(neg_log_likelihood) + ppl = torch.exp(torch.stack(nlls).sum() / (nsamples * seqlen)) + print(f'wikitext-2-raw-1-test ppl: {round(ppl.item(), 2)}') + +# eval_wikitext(llm) + + +# Chat with model +messages = "<|im_start|>system You are a helpful assistant.<|im_end|><|im_start|>user你好！\n<|im_end|><|im_start|>assistant" +kwargs = {"max_length": 128, "top_k": 1, "top_p": 0.8, + "temperature": 0.8, "do_sample": True, "repetition_penalty": 1.1} +# print(llm.chat_model(messages, kwargs)) + + +# Export rkllm model +ret = llm.export_rkllm("./qwen.rkllm") +if ret != 0: + print('Export model failed!') + exit(ret) diff --git a/rkllm-toolkit/packages/md5sum.txt b/rkllm-toolkit/packages/md5sum.txt index d115b4f..02c1b05 100644 --- a/rkllm-toolkit/packages/md5sum.txt +++ b/rkllm-toolkit/packages/md5sum.txt @@ -1 +1,2 @@ -cf82f9756844793bbcd132aae58d0b81 rkllm_toolkit-1.0.1-cp38-cp38-linux_x86_64.whl +696070d1ec134ff68fd5b03b873cd678 rkllm_toolkit-1.1.1-cp38-cp38-linux_x86_64.whl +e8e37c6bf0af799f9a0187c9864b2450 rkllm_toolkit-1.1.1-cp310-cp310-linux_x86_64.whl diff --git a/rkllm-toolkit/packages/rkllm_toolkit-1.0.1-cp38-cp38-linux_x86_64.whl b/rkllm-toolkit/packages/rkllm_toolkit-1.0.1-cp38-cp38-linux_x86_64.whl deleted file mode 100644 index 0f6cd35..0000000 Binary files a/rkllm-toolkit/packages/rkllm_toolkit-1.0.1-cp38-cp38-linux_x86_64.whl and /dev/null differ diff --git a/rkllm-toolkit/packages/rkllm_toolkit-1.1.1-cp310-cp310-linux_x86_64.whl b/rkllm-toolkit/packages/rkllm_toolkit-1.1.1-cp310-cp310-linux_x86_64.whl new file mode 100644 index 0000000..e4d5cfa Binary files /dev/null and b/rkllm-toolkit/packages/rkllm_toolkit-1.1.1-cp310-cp310-linux_x86_64.whl differ diff --git a/rkllm-toolkit/packages/rkllm_toolkit-1.1.1-cp38-cp38-linux_x86_64.whl b/rkllm-toolkit/packages/rkllm_toolkit-1.1.1-cp38-cp38-linux_x86_64.whl new file mode 100644 index 0000000..5a1cd5e Binary files /dev/null and b/rkllm-toolkit/packages/rkllm_toolkit-1.1.1-cp38-cp38-linux_x86_64.whl differ diff --git a/rknpu-driver/rknpu_driver_0.9.6_20240322.tar.bz2 b/rknpu-driver/rknpu_driver_0.9.6_20240322.tar.bz2 deleted file mode 100644 index 64fa3ec..0000000 Binary files a/rknpu-driver/rknpu_driver_0.9.6_20240322.tar.bz2 and /dev/null differ diff --git a/rknpu-driver/rknpu_driver_0.9.8_20241009.tar.bz2 b/rknpu-driver/rknpu_driver_0.9.8_20241009.tar.bz2 new file mode 100644 index 0000000..a120695 Binary files /dev/null and b/rknpu-driver/rknpu_driver_0.9.8_20241009.tar.bz2 differ diff --git a/scripts/fix_freq_rk3576.sh b/scripts/fix_freq_rk3576.sh new file mode 100755 index 0000000..e2517a1 --- /dev/null +++ b/scripts/fix_freq_rk3576.sh @@ -0,0 +1,43 @@ +#!/system/bin/sh + +echo 1 > /sys/devices/system/cpu/cpu0/cpuidle/state1/disable +echo 1 > /sys/devices/system/cpu/cpu1/cpuidle/state1/disable +echo 1 > /sys/devices/system/cpu/cpu2/cpuidle/state1/disable +echo 1 > /sys/devices/system/cpu/cpu3/cpuidle/state1/disable +echo 1 > /sys/devices/system/cpu/cpu4/cpuidle/state1/disable +echo 1 > /sys/devices/system/cpu/cpu5/cpuidle/state1/disable +echo 1 > /sys/devices/system/cpu/cpu6/cpuidle/state1/disable +echo 1 > /sys/devices/system/cpu/cpu7/cpuidle/state1/disable + +echo "NPU available frequencies:" +cat /sys/class/devfreq/27700000.npu/available_frequencies +echo "Fix NPU max frequency:" +echo userspace > /sys/class/devfreq/27700000.npu/governor +echo 1000000000 > /sys/class/devfreq/27700000.npu/userspace/set_freq +cat /sys/class/devfreq/27700000.npu/cur_freq + +echo "CPU available frequencies:" +cat /sys/devices/system/cpu/cpufreq/policy0/scaling_available_frequencies +cat /sys/devices/system/cpu/cpufreq/policy4/scaling_available_frequencies +echo "Fix CPU max frequency:" +echo userspace > /sys/devices/system/cpu/cpufreq/policy0/scaling_governor +echo 2208000 > /sys/devices/system/cpu/cpufreq/policy0/scaling_setspeed +cat /sys/devices/system/cpu/cpufreq/policy0/scaling_cur_freq +echo userspace > /sys/devices/system/cpu/cpufreq/policy4/scaling_governor +echo 2304000 > /sys/devices/system/cpu/cpufreq/policy4/scaling_setspeed +cat /sys/devices/system/cpu/cpufreq/policy4/scaling_cur_freq + +echo "GPU available frequencies:" +cat /sys/class/devfreq/27800000.gpu/cur_freq +cat /sys/class/devfreq/27800000.gpu/available_frequencies +echo "Fix GPU max frequency:" +echo userspace > /sys/class/devfreq/27800000.gpu/governor +echo 950000000 > /sys/class/devfreq/27800000.gpu/userspace/set_freq +cat /sys/class/devfreq/27800000.gpu/cur_freq + +echo "DDR available frequencies:" +cat /sys/class/devfreq/dmc/available_frequencies +echo "Fix DDR max frequency:" +echo userspace > /sys/class/devfreq/dmc/governor +echo 2112000000 > /sys/class/devfreq/dmc/userspace/set_freq +cat /sys/class/devfreq/dmc/cur_freq \ No newline at end of file diff --git a/scripts/fix_freq_rk3588.sh b/scripts/fix_freq_rk3588.sh new file mode 100755 index 0000000..ba223c2 --- /dev/null +++ b/scripts/fix_freq_rk3588.sh @@ -0,0 +1,46 @@ +#!/system/bin/sh + +echo 1 > /sys/devices/system/cpu/cpu0/cpuidle/state1/disable +echo 1 > /sys/devices/system/cpu/cpu1/cpuidle/state1/disable +echo 1 > /sys/devices/system/cpu/cpu2/cpuidle/state1/disable +echo 1 > /sys/devices/system/cpu/cpu3/cpuidle/state1/disable +echo 1 > /sys/devices/system/cpu/cpu4/cpuidle/state1/disable +echo 1 > /sys/devices/system/cpu/cpu5/cpuidle/state1/disable +echo 1 > /sys/devices/system/cpu/cpu6/cpuidle/state1/disable +echo 1 > /sys/devices/system/cpu/cpu7/cpuidle/state1/disable + +echo "NPU available frequencies:" +cat /sys/class/devfreq/fdab0000.npu/available_frequencies +echo "Fix NPU max frequency:" +echo userspace > /sys/class/devfreq/fdab0000.npu/governor +echo 1000000000 > /sys/class/devfreq/fdab0000.npu/userspace/set_freq +cat /sys/class/devfreq/fdab0000.npu/cur_freq + +echo "CPU available frequencies:" +cat /sys/devices/system/cpu/cpufreq/policy0/scaling_available_frequencies +cat /sys/devices/system/cpu/cpufreq/policy4/scaling_available_frequencies +cat /sys/devices/system/cpu/cpufreq/policy6/scaling_available_frequencies +echo "Fix CPU max frequency:" +echo userspace > /sys/devices/system/cpu/cpufreq/policy0/scaling_governor +echo 1800000 > /sys/devices/system/cpu/cpufreq/policy0/scaling_setspeed +cat /sys/devices/system/cpu/cpufreq/policy0/scaling_cur_freq +echo userspace > /sys/devices/system/cpu/cpufreq/policy4/scaling_governor +echo 2352000 > /sys/devices/system/cpu/cpufreq/policy4/scaling_setspeed +cat /sys/devices/system/cpu/cpufreq/policy4/scaling_cur_freq +echo userspace > /sys/devices/system/cpu/cpufreq/policy6/scaling_governor +echo 2352000 > /sys/devices/system/cpu/cpufreq/policy6/scaling_setspeed +cat /sys/devices/system/cpu/cpufreq/policy6/scaling_cur_freq + +echo "GPU available frequencies:" +cat /sys/class/devfreq/fb000000.gpu/available_frequencies +echo "Fix GPU max frequency:" +echo userspace > /sys/class/devfreq/fb000000.gpu/governor +echo 1000000000 > /sys/class/devfreq/fb000000.gpu/userspace/set_freq +cat /sys/class/devfreq/fb000000.gpu/cur_freq + +echo "DDR available frequencies:" +cat /sys/class/devfreq/dmc/available_frequencies +echo "Fix DDR max frequency:" +echo userspace > /sys/class/devfreq/dmc/governor +echo 2112000000 > /sys/class/devfreq/dmc/userspace/set_freq +cat /sys/class/devfreq/dmc/cur_freq