I have a collection of ebooks(8 GB atm) that i want to backup to google drive. It usually takes me 2-3 hrs to upload a 2 GB zip file(containing only the most important documents and files).
Tried to upload my ebooks, left it for 12 hrs and it still hasn’t uploaded. Thought maybe its because i am using Rclone shared client ID(heavily rate limited ), so i set up my own client ID and tried again and its been 3 hrs and still hasn’t upload
I am realizing now that its probably has nothing to do with rclone but rather my internet speed, specifically my upload speed(download 15-30 mps & upload 5-8 mps).
So i am wondering if anyone has solutions for people in countries with abysmal internet speeds.
The only real solution to do it faster is to use faster internet or compress it further. Then alternatives are chunk it or deal with a long maybe even 24 hour upload.
Have you considered a torrent based solution?
Setup a torrenting server hosted online somewhere with fast speeds (e.g. Netherlands), sync it with gdrive, create a torrent for your books, and “download” your books to the toreenting machine.
This approach won’t help with upload speed, but it should help with reliability/continuity. As long as you seed from your home computer, your books will eventually make it into your gdrive.
Although with this setup it might be easier to just torrent the ebooks to the online server from sources other than yourself.
- uploading from a phone if it stays on all the time
- uploading one at a time
- uploading in smaller batches
- uploading from an internet café, a library or some other place if it has internet faster than yours
- hire a VPS, use some transfer method with a more generous timeout tolerance to transfer the files uncompressed to the VPS, and from there leave the ebooks uploading with your preferred method
- hire a faster service if possible
Mail a physical copy to someone with a faster connection and ask them to upload for you
uploading one at a time, uploading in smaller batches
Yup, that’s where I’d start. Creating a tar and splitting it into parts. Just as an example. Not sure if that guide is good or not. It was just the first thing that popped up in the search engine.
It’s gonna take a long time, just deal with it. 12 hours is expected, right? Get some sleep and check it afterwards.
It didn’t even upload after 12 hours. I left it running the whole night and woke up in the morning to it still running
Where will you be using this upload?
If you’re doing it to make it accessible on your own devices from anywhere, try Syncthing on the various devices instead. If the devices are on the same LAN from time to time, your modem won’t be a bottleneck.
If you still have the sources from which you originally acquired the books, you could use a VPS to re-acquire them, and then push them to the google drive directly from the VPS. They never pass through your modem; your modem can’t be the bottleneck.
Sounds like you need a client that does “multiparty uploads”. Meaning interruptions should continue if there is a break in comms.
All the Cloud endpoints support this, you just need a better client. Do some research I guess.
At 5mbps it should take about 1 hour for 2GB. It sounds like your actual speed is 2-3x lower. Can you take that up with your ISP? Are you certain your machine has the best connection within your control, i.e. directly wired into the router? Network equipment is not faulty? Have you tested with iperf within your network? Just in case there’s another issue beside the slow external speed…
Another thing that springs to mind is to use a backup tool like
resticthat will not only compress but deduplicate your data into hundreds of small files that might make upload faster. Dedup can save significant space and you can try it out locally first. Just dorestic initthenrestic backup PATH.Restic can use rclone as a backend also and upload straight to google: https://restic.readthedocs.io/en/stable/030_preparing_a_new_repo.html#other-services
Finally there’s sometimes nothing faster than physically moving data. A person jogging with a 100gb drive has great bandwidth! Is there a location with better internet within reach? A library or school perhaps?
