I want to have a mirror of my local music collection on my server, and a script that periodically updates the server to, well, mirror my local collection.
But crucially, I want to convert all lossless files to lossy, preferably before uploading them.
That’s the one reason why I can’t just use git - or so I believe.
I also want locally deleted files to be deleted on the server.
Sometimes I even move files around (I believe in directory structure) and again, git deals with this perfectly. If it weren’t for the lossless-to-lossy caveat.
It would be perfect if my script could recognize that just like git does, instead of deleting and reuploading the same file to a different location.
My head is spinning round and round and before I continue messing around with find
and scp it’s time to ask the community.
I am writing in bash but if some python module could help with it I’m sure I could find my way around it.
TIA
additional info:
- Not all files in the local collection are lossless. A variety of formats.
- The purpose of the remote is for listening/streaming with various applications
- The lossy version is for both reducing upload and download (streaming) bandwidth. On mobile broadband FLAC tends to buffer a lot.
- The home of the collection (and its origin) is my local machine.
- The local machine cannot act as a server
Speaking as a data engineer, you’re having trouble because git is the wrong tool for the job. You can make it work if you use git-lfs + custom hooks — but if you choose to go that route, be aware you’re making things unnecessarily hard for yourself.
If you want to make this easy, separate out your concerns:
- Versioning: take periodic snapshots of your unconverted files with a binary-friendly diffing tool like restic or borg. Alternatively, ZFS/btrfs snapshots are an excellent way to handle this.
- Conversion: keep your original files in their own directory. Set up a small script that searches your directory of original files recursively, passes the files to lame to encode to V0 or V2, and outputs them to a separate directory of lossy mp3 files.
- Syncing: use rsync with the --delete flag to copy your lossy files to the server + clear out files you’ve removed locally.
Thank you, this is very comprehensive! Yes, after some thought git is no good here.
I am coming round to a similar concept now: one job for lossy files - relatively easy with rsync I guess - and another for the lossless.
Yeah just encode the files locally and rsync them to the server. You could even use a Makefile to do the conversation.
For the conversion to lossy, git smudge/clean filters might work. Afaik git-lfs works with them. In general git-lfs might be something interesting for your use case.
Wow, TIL, I never heard of git smudge/clean, or git large filesystem I’m not OP, but just amazed about another corner of git I hadn’t known about.
You might be able to achieve what you want with pre-commit git hooks. You might be able to write a pre-commit script that converts your loseless files to lossy before commiting them to the git repo.
A quick googling gave me this demo python pre-commit hook guide, but I’m sure you can write it in other languages if you’d like, could probably even use a shell script I would think.
https://dev.to/jalvaradosegura/create-your-own-pre-commit-hook-3kh
