mirror of
https://github.com/DrHo1y/ezrknn-llm.git
synced 2026-03-23 09:06:47 +07:00
Update README.md with conversion
This commit is contained in:
20
README.md
20
README.md
@@ -21,6 +21,26 @@ Run (cd is required):
|
|||||||
# TODO
|
# TODO
|
||||||
```
|
```
|
||||||
|
|
||||||
|
## Converting LLMs for Rockchip's NPUs
|
||||||
|
### Docker
|
||||||
|
In order to do this, you need a Linux PC x86 (Intel or AMD). Currently, Rockchip does not provide ARM support for converting models, so can't be done on a Orange Pi or similar.
|
||||||
|
Run:
|
||||||
|
|
||||||
|
`docker run -it pelochus/ezrkllm-toolkit:1.0 bash`
|
||||||
|
|
||||||
|
Then, inside the Docker container:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
apt install -y python3-tk # This needs some configuring from your part
|
||||||
|
cd ezrknn-llm/rkllm-toolkit/examples/huggingface/
|
||||||
|
```
|
||||||
|
|
||||||
|
Now change the `test.py` with your preferred model. This container provides Qwen-1.8B and LLaMa2 Uncensored. By default, Qwen-1.8B is selected. To convert the model, run:
|
||||||
|
|
||||||
|
`python3 test.py`
|
||||||
|
|
||||||
|
I currently cannot convert the models, so I don't know what the output will be. I believe this is Rockchip's fault. Let me know if you could or what error gives you.
|
||||||
|
|
||||||
# Original README starts below
|
# Original README starts below
|
||||||
|
|
||||||
<hr>
|
<hr>
|
||||||
|
|||||||
Reference in New Issue
Block a user