cross-posted from: https://lemmy.dbzer0.com/post/36841328
Hello, everyone! I wanted to share my experience of successfully running LLaMA on an Android device. The model that performed the best for me was llama3.2:1b on a mid-range phone with around 8 GB of RAM. I was also able to get it up and running on a lower-end phone with 4 GB RAM. However, I also tested several other models that worked quite well, including qwen2.5:0.5b , qwen2.5:1.5b , qwen2.5:3b , smallthinker , tinyllama , deepseek-r1:1.5b , and gemma2:2b. I hope this helps anyone looking to experiment with these models on mobile devices!
Step 1: Install Termux
- Download and install Termux from the Google Play Store or F-Droid
Step 2: Set Up proot-distro and Install Debian
Open Termux and update the package list:
pkg update && pkg upgradeInstall proot-distro
pkg install proot-distroInstall Debian using proot-distro:
proot-distro install debianLog in to the Debian environment:
proot-distro login debianYou will need to log-in every time you want to run Ollama. You will need to repeat this step and all the steps below every time you want to run a model (excluding step 3 and the first half of step 4).
Step 3: Install Dependencies
Update the package list in Debian:
apt update && apt upgradeInstall curl:
apt install curl
Step 4: Install Ollama
Run the following command to download and install Ollama:
curl -fsSL https://ollama.com/install.sh | shStart the Ollama server:
ollama serve &After you run this command, do ctrl + c and the server will continue to run in the background.
Step 5: Download and run the Llama3.2:1B Model
- Use the following command to download the Llama3.2:1B model:
This step fetches and runs the lightweight 1-billion-parameter version of the Llama 3.2 model .ollama run llama3.2:1b
Running LLaMA and other similar models on Android devices is definitely achievable, even with mid-range hardware. The performance varies depending on the model size and your device’s specifications, but with some experimentation, you can find a setup that works well for your needs. I’ll make sure to keep this post updated if there are any new developments or additional tips that could help improve the experience. If you have any questions or suggestions, feel free to share them below!
– llama
you only fry your phone with this. very bad idea
Not true. If you load a model that is below your phone’s hardware capabilities it simply won’t open. Stop spreading fud.
@[email protected] Depends on the inference engine. Some of them will try to load the model until it blows up and runs out of memory. Which can cause its own problems. But it won’t overheat the phone, no. But if you DO use a model that the phone can run, like any intense computation, it can cause the phone to heat up. Best not run a long inference prompt while the phone is in your pocket, I think.
Thanks for your comment. That for sure is something to look out for. It is really important to know what you’re running and what possible limitations there could be. Not what the original comment said, though.
that’s not how it works. Your phone can easily overheat if you use it too much, even if your device can handle it. Smartphones don’t have cooling like pcs and laptops (except some rog phone and stuff). If you don’t want to fry your processor, only run LLMs on high-end gaming pcs with All in one water cooling
This is so horrifically wrong, I don’t even know where to start.
The short version is that phone and computer makers aren’t stupid and they will kill things or shutdown when overheating happens. If you were a phone maker, why tf would you allow someone to fry their own phone?
My laptop has shut itself off when I was trying to compile code while playing video games, while watching twitch. My android phone has killed apps when I try to do too much as well.
This is all very nuanced and there isn’t a clear cut answer. It really depends on what you’re running, for how long you’re running it, your device specs, etc. The LLMs I mentioned in the post did just fine and did not cause any overheating if not used for extended periods of time. You absolutely can run a SMALL LLM and not fry your processor if you don’t overdo it. Even then, I find it extremely unlikely that you’re going to cause permanent damage to your hardware components.
Of course that is something to be mindful of, but that’s not what the person in the original comment said. It does run, but you need to be aware of the limitations and potential consequences. That goes without saying, though.
Just don’t overdo it. Or do, but the worst thing that will happen is your phone getting hella hot and shutting down.
my phone was fried last week, it needed soc reballing. From watching videos and browsing the web at the same time. Most hardware developers don’t pay attention to cooling and these stuff run on hopes and dreams. Plus auto switchoff is only a software solution, and software can have bugs