Home
bitmagnet.io

cross-posted from: https://lemmy.world/post/6301281

I’m excited to announce the first alpha preview of this project that I’ve been working on for the past 4 months. I’m initially posting about this in a few small communities, and hoping to get some input from early adopters and beta testers.

What is a DHT crawler?

The DHT crawler is Bitmagnet’s killer feature that (currently) makes it unique. Well, almost unique, read on…

So what is it? You might be aware that you can enable DHT in your BitTorrent client, and that this allows you find peers who are announcing a torrent’s hash to a Distributed Hash Table (DHT), rather than to a centralized tracker. DHT’s lesser known feature is that it allows you to crawl the info hashes it knows about. This is how Bitmagnet’s DHT crawler works works - it crawls the DHT network, requesting metadata about each info hash it discovers. It then further enriches this metadata by attempting to classify it and associate it with known pieces of content, such as movies and TV shows. It then allows you to search everything it has indexed.

This means that Bitmagnet is not reliant on any external trackers or torrent indexers. It’s a self-contained, self-hosted torrent indexer, connected via the DHT to a global network of peers and constantly discovering new content.

The DHT crawler is not quite unique to Bitmagnet; another open-source project, magnetico was first (as far as I know) to implement a usable DHT crawler, and was a crucial reference point for implementing this feature. However that project is no longer maintained, and does not provide the other features such as content classification, and integration with other software in the ecosystem, that greatly improve usability.

Currently implemented features of Bitmagnet:

  • A DHT crawler
  • A generic BitTorrent indexer: Bitmagnet can index torrents from any source, not only the DHT network - currently this is only possible via the /import endpoint; more user-friendly methods are in the pipeline
  • A content classifier that can currently identify movie and television content, along with key related attributes such as language, resolution, source (BluRay, webrip etc.) and enriches this with data from The Movie Database
  • An import facility for ingesting torrents from any source, for example the RARBG backup
  • A torrent search engine
  • A GraphQL API: currently this provides a single search query; there is also an embedded GraphQL playground at /graphql
  • A web user interface implemented in Angular: currently this is a simple single-page application providing a user interface for search queries via the GraphQL API
  • A Torznab-compatible endpoint for integration with the Serverr stack

Interested?

If this project interests you then I’d really appreciate your input:

  • How did you get along with following the documentation and installation instructions? Were there any pain points?
  • There’s a roadmap of high-priority features on the website - what do you see as the highest priority for near-term development?
  • If you’re a developer, are you interested in contributing to the project?

Thanks for your attention. If you’re interested in this project and would like to help it gain momentum then please give it a star on GitHub, and expect further updates soon!

  • taaz@biglemmowski.winEnglish
    141·
    1 year ago

    Slight off-topic but do people not know about Tixati ? It’s a torrent client but it has searchable DHT (by torrent name) and full decentralised forums (with audio streaming, for whatever reason lol) for a long time now.
    Except for being closed source it’s pretty interesting. A bit baffled there isn’t more talk about it, or at least about the features.

    • else@lemmy.fmhy.netEnglish
      6·
      1 year ago

      Great client but unfortunately for those of us on private trackers it isn’t going to be whitelisted on many of them, or at least the most prominent ones. At least that was the case last I checked, and I gave up on it after.

      • AdmiralShatEnglish
        2·
        1 year ago

        What’s the actual benefit for private trackers? I’ve never failed to find a torrent for something I wanted, so I don’t know what they actually bring to the table

        • else@lemmy.fmhy.netEnglish
          3·
          1 year ago

          1. Always high quality content and easy as hell to find. Flacs for music, Blu-ray sourced movies/tv with high bitrate, etc. No sifting through torrents of questionable quality.

          2. That’s because good private trackers have torrent checkers, report systems, and staff/dedicated users making sure content is up to par in terms of quality and safety against malware.

          3. They are filled with seed boxing users making all of your downloads very fast. On the flip side these users often cannabalize upload ratio for torrents so individuals on normal connections might have to rely on systems outside purely seeding to maintain a proper ratio. (This is exaggerated on places like r/trackers though, it’s not hard at all imo.)

          4. The request sections can get you pretty much anything that’s reasonably obtainable if you out a bounty up. For me, I have niche tastes in music so that’s been helpful more than a few times.

          5. I never have to worry about my isp complaining about what I download. I’ve never received a complaint once, and I’ve never used a VPN.

          • AdmiralShatEnglish
            1·
            1 year ago

            Thanks for the response

            On that last point, is that still a thing? I haven’t gotten a torrent complaint in over a decade now, I honestly that was like a Comcast only thing at this point.

            • else@lemmy.fmhy.netEnglish
              2·
              1 year ago

              Comcast is the only option where I live. And yes, there are more strict rules in certain other regions like I think some European countries. Many people I’ve talked to have gotten isp letters just where I live though.

              • AdmiralShatEnglish
                1·
                1 year ago

                Does your power company have any plans for internet? Mine installed fiber and I know many across the US have also done the same.

  • frozen@lemmy.frozeninferno.xyzEnglish
    8·
    1 year ago

    This is dope! There are a lot more important things on the roadmap, but a qBitTorrent plugin or compatible endpoint would be amazing.

  • TheOneCurly@lemmy.theonecurly.pageEnglish
    7·
    1 year ago

    The included docker compose was very easy to use. I was up and running in just a few minutes.

    DHT crawling started immediately which was pretty cool to see.

    Sometime later this week I’ll try integrating it into my arr stack.

  • Daniel Quinn@lemmy.caEnglish
    4·
    1 year ago

    Very nice work, and thanks for sharing it!

    I haven’t spent much time digging through the code yet but noted that you’re running your application in the ENTRYPOINT rather than the CMD. This can make overriding or extending parts of your container difficult, so you may wanna change that.

    I may try setting this on my k8s cluster… if it can build on aarm64 ;-)

  • privadesco@feddit.nlEnglish
    3·
    1 year ago

    Please, with sugar on top: make this a plugin that can be integrated to qbittorrent or rtorrent.

    Try to make your idea reach the seedboxes. This would be great for open-tracker seedboxes!