• @[email protected]OP
      link
      fedilink
      2171 month ago

      The best I can do is an ML model running on an NPU that parses JSON in subtly wrong and impossible to debug ways

      • @[email protected]
        link
        fedilink
        551 month ago

        Just make it a LJM (Large JSON Model) capable of predicting the next JSON token from the previous JSON tokens and you would have massive savings in file storagre and network traffic from not having to store and transmit full JSON documents all in exchange for an “acceptable” error rate.

      • @[email protected]
        link
        fedilink
        English
        531 month ago

        You need to make sure to remove excess whitespace from the JSON to speed up parsing. Have an AI read the JSON as plaintext and convert it to a handwriting-style image, then another one to use OCR to convert it back to text. Trailing whitespace will be removed.

        • @[email protected]
          link
          fedilink
          31 month ago

          Did you know? By indiscriminately removing every 3rd letter, you can ethically decrease input size by up to 33%!

      • Terrasque
        link
        fedilink
        71 month ago

        So you’re saying it’s already feature complete with most json libraries out there?

      • I Cast Fist
        link
        41 month ago

        Finally, something to process “databases” that ditched excel for json!

      • JackbyDev
        link
        English
        31 month ago

        60 million CLOPS? No way!

        • ChaoticNeutralCzech
          link
          fedilink
          14
          edit-2
          1 month ago

          The website title says “Arm Developer”, not “ARM Developer”, in a clearly non-acronym way so it’s a guide for making prosthetic hardware. Of course you want a cyborg arm to parse JS natively, why else even get one?

        • @[email protected]
          link
          fedilink
          41 month ago

          Nope it’s still a register-register op, that’s very much load-store architecture.

          It’s reduced, not minimalist, otherwise every RISC CPU out there would only have one instruction like decrement and branch if nonzero. RISC-V would not have an extension mechanism. The instruction exists because it makes things faster because you don’t have to do manual bit-fiddling over 10 instructions to achieve a thing already-existing ALU logic can do in a single cycle. A thing that isn’t even javascript-specific (or terribly relevant to json), it’s a specific float to int cast with specific rounding and overflow mode. Would it more palatable to your tastes if the CPU were to do macro-op fusion on 10(!) instructions to get the same result?

        • @[email protected]
          link
          fedilink
          7
          edit-2
          1 month ago

          No, that’s not what RISC is about. There was some early attempts to keep the number of instructions low–originally, ARM didn’t have a multiply instruction, and there’s still a bunch of microcontrollers you can buy that don’t have a divide instruction–but it was quickly abandoned as it’s just not that useful. It only holds back instructions that optimize common cases. Your compiler can implement multiplication by doing addition in a loop, but that’s not very efficient.

          What really worked about it was keeping a separation between how memory is accessed. You don’t have an ADD instruction that can fetch from both registers or main memory. You have a MOV instruction that can fetch from memory into a register, and you have an ADD instruction that can work on registers.

          ARM still does this just fine.

          • DumbAceDragon
            link
            fedilink
            English
            8
            edit-2
            1 month ago

            I’m a computer engineering major (still a student tbf), I’m well aware of the difference between CISC and RISC, I was making a joke.

            Also, I understand your point, but you should know though that a load-store architecture and a RISC instruction set are not the same thing. The vast majority of RISC ISAs are load-store, but not all load-store architectures are RISC.

            • @[email protected]
              link
              fedilink
              91 month ago

              http://www.quadibloc.com/arch/sriscint.htm

              The RISC architecture contains several common elements. Some of them are no longer present in most chips that still call themselves RISC:

              • All instructions execute in a single cycle.
              • Floating-point operations, specifically, are therefore excluded.

              But most of the defining characteristics of RISC do remain in force:

              • All instructions occupy the same amount of space in memory.
              • Only load, store, and jump instructions directly address memory. Calculations are performed only between operands in registers.

              https://groups.google.com/g/comp.arch/c/IZP5KUJprHw?pli=1

              MOST RISCs:
              3a) Have 1 size of instruction in an instruction stream
              3b) And that size is 4 bytes
              3c) Have a handful (1-4) addressing modes) (* it is VERY hard to count these things; will discuss later).
              3d) Have NO indirect addressing in any form (i.e., where you need one memory access to get the address of another operand in memory)
              4a) Have NO operations that combine load/store with arithmetic, i.e., like add from memory, or add to memory. (note: this means especially avoiding operations that use the value of a load as input to an ALU operation, especially when that operation can cause an exception. Loads/stores with address modification can often be OK as they don’t have some of the bad effects)
              4b) Have no more than 1 memory-addressed operand per instruction
              5a) Do NOT support arbitrary alignment of data for loads/stores
              5b) Use an MMU for a data address no more than once per instruction
              6a) Have >=5 bits per integer register specifier
              6b) Have >= 4 bits per FP register specifier

              Note that none of this has to do with reducing the number of instructions, which is what people tend to think of when they hear the name.

              • @[email protected]
                link
                fedilink
                21 month ago

                All instructions occupy the same amount of space in memory.

                Both ARM and RISC-V have compressed instructions. Dunno how ARM works but with RISC-V the 16-bit instruction set is freely interspersable with the 32 bit one, which also get their alignment reduced to 16 bits. Gets like 95% of the space reduction possible with full variable-width instructions without overcomplicating the insn decoder.

                As to addressing and loads and arithmetic: No such instructions, but every CPU but the tiniest ones are expected to do macro-op fusion for things like indexed loads. Here’s an overview.

                The MMU thing… well the vector extension can do gather/scatter, I guess it could stay within the letter of “use the MMU once” but definitely not the spirit.

      • @[email protected]
        link
        fedilink
        English
        251 month ago

        106 Gbps

        They get to this result on 0.6 MB of data (paper, page 5)

        They even say:

        Moreover, there is no need to evaluate our design with datasets larger than the ones we have used; we achieve steady state performance with our datasets

        This requires an explanation. I do see the need - if you promise 100Gbps you need to process at least a few Tbs.

        • @[email protected]
          link
          fedilink
          121 month ago

          Imagine you have a car powered by a nuclear reactor with enough fuel to last 100 years and a stable output of energy. Then you put it on a 5 mile road that is comprised of the same 250 small segments in various configurations, but you know for a fact that starts and ends at the same elevation. You also know that this car gains exactly as much performance going downhill as it loses going uphill.

          You set the car driving and determine that, it takes 15 minutes to travel 5 miles. You reconfigure the road, same rules, and do it again. Same result, 15 minutes. You do this again and again and again and always get 15 minutes.

          Do you need to test the car on a 20 mile road of the same configuration to know that it goes 20mph?

          JSON is a text-based, uncompressed format. It has very strict rules and a limited number of data types and structures. Further, it cannot contain computational logic on it’s own. The contents can interpreted after being read to extract logic, but the JSON itself cannot change it’s own computational complexity. As such, it’s simple to express every possible form and complexity a JSON object can take within just 0.6 MB of data. And once they know they can process that file in however-the-fuck-many microseconds, they can extrapolate to Gbps from there

          • @[email protected]
            link
            fedilink
            English
            21 month ago

            Based on your analogue they drive the car for 7.5 inches (614.4 Kb by 63360 inches by 20 divided by 103179878.4 Kb) and promise based on that that car travels 20mph which might be true, yes, but the scale disproportion is too considerable to not require tests. This is not maths, this is a real physical device - how would it would behave on larger real data remains to be seen.

            • @[email protected]
              link
              fedilink
              1
              edit-2
              1 month ago

              Except we know what the lifecycle of physical storage is, it’s rate of performance decay (virtually none for solid state until failure), and that the computers performing the operations have consistent performance for the same operations over time. And again, while for a car such a small amount can’t be reasonably extrapolated, for a computer processing an extremely simple format like JSON, when it is designed to handle FAR more difficult tasks on the GPU involving billions of floating point operations, it is absolutely, without a doubt enough.

              You don’t have to believe me if you don’t want but I’m very confident in my understanding of JSON’s complexity relative to typical GPU workloads, computational analysis, computer hardware durability lifecycles, and software testing principles and best practices. 🤷

        • @[email protected]
          link
          fedilink
          11 month ago

          But to write such a file you need a few quantum computers map reducing the data in alternative universes

    • nickwitha_k (he/him)
      link
      fedilink
      11 month ago

      Personally, now that I have a machine capable of running the toolchains, I want to explore hardware accelerated compilation. Not all steps can be done in parallel but I bet a lot before linking can.

    • @refalo
      link
      331 month ago

      yea we need multithreaded json parsers

    • @[email protected]
      link
      fedilink
      English
      41 month ago

      Doesn’t a 3070 have less than 7k cores? A UHD 750 (relatively recent iGPU) only has 256.

      And I don’t know the structure of JSON that well, but can’t tokens be made of multiple chars?

  • Xyloph
    link
    fedilink
    761 month ago

    That is sometime the issue when your code editor is a disguised web browser 😅

  • voxel
    link
    fedilink
    57
    edit-2
    1 month ago

    there are simd accelerated json decoders

  • @[email protected]
    link
    fedilink
    431 month ago

    Would you rather have 100,000 kg of tasty supreme pizza, or 200 kg of steaming manure?

    Choose wisely.

      • @[email protected]
        link
        fedilink
        131 month ago

        Not sure if I’m just missing a reference here, but if you choose the pizza you can have both.

        • @[email protected]
          link
          fedilink
          61 month ago

          Yeah but then that would be human manure which is a little (although not much) more dangerous to use on stuff you’re going to eat

        • @[email protected]
          link
          fedilink
          11 month ago

          you gotta rot it down a bit anyway. stick it in the corner of the garden and leave it until the following year.

          Otherwise you’ll get weeds all up in your veggies

    • @[email protected]
      link
      fedilink
      31 month ago

      The pizza can be used to feed some people but you really have to go fast and find hungry people

      Manure can be sold easily

    • @Buttons
      link
      English
      31 month ago

      Careful, the 100,000 kg of pizza will turn into manure.

      • @[email protected]
        link
        fedilink
        21 month ago

        I figure I can probably convert about 10 kg into manure before it autoconverts into compost. Which is maybe even a worse problem.

  • @[email protected]
    link
    fedilink
    English
    371 month ago

    I have the same problem with XML too. Notepad++ has a plugin that can format a 50MB XML file in a few seconds. But my current client won’t allow plugins installed. So I have to use VS Code, which chokes on anything bigger than what I could do myself manually if I was determined.

    • @[email protected]
      link
      fedilink
      121 month ago

      Except if it’s a single line file, only god can help you then. (Or running prettier -w on it before opening it or whatever.)

      • @[email protected]
        link
        fedilink
        71 month ago

        4.2 megs on one line? Vim probably can handle it fine, although syntax won’t be highlighted past a certain point

        • @[email protected]
          link
          fedilink
          51 month ago

          I’ve accidentally opened enormous single line json files more than once. Could be lsp config or treesitter or any number of things but trying to do any operations after opening such a file is not a good time.

          • @[email protected]
            link
            fedilink
            11 month ago

            Yeah, very well may be. LSPs always slow down opening big files, so I usually inspect those with an empty/different config

        • BaldProphet
          link
          fedilink
          31 month ago

          Technically every JSON file is a single line, with line break characters here and there

      • @expr
        link
        31 month ago

        :syntax off and it works just fine.

      • Andrew
        link
        fedilink
        0
        edit-2
        1 month ago

        If you’re not aware, it was called MB because of JEDEC when IEC units weren’t invented. IEC units were introduced because they remove the double meaning of JEDEC units — decimal and binary. IEC units only carry the binary meaning, hence why they’re superior. If you convert 1000 kB to 1 MB then use MB, but in case of 1024 KiB to 1 MiB you should be using MiB. It’s all about getting the point across, and JEDEC units aren’t good at it.

        • @[email protected]
          link
          fedilink
          31 month ago

          I’m failing to understand why we would need decimal units at all. Whats the point of them? And why do the original units havr to change name to something as ridiculous as “Gibibyte” while the unnecessary decimal units get the binary’s old name?

          • Andrew
            link
            fedilink
            0
            edit-2
            1 month ago

            You poor innocent soul… I can try to explain why decimal is even mentioned, but it would probably take a lot of time, and I’m not sure if I will be able to clarify things up.

            I can at least say this: 2 TB HDD drive is indeed 2*10^12 B, but suddenly shindow$ in its File Explorer will show you that in fact the drive is only 1.82 TB. But WHY? Everyone asks, feeling scammed. Because HDD spec uses decimal units (SI; MB) and Window$ uses binary units (JEDEC; MB), i.e., 1.82 TiB (IEC; MiB). And macOS also uses JEDEC units, AFAIK.

            More and more FOSS software uses IEC units and KDE Plasma is a good example: file manager, package manager etc. uses IEC units. Simply put, JEDEC added the binary meaning to decimal units, so at first MB (and now) only carried decimal meaning (until JEDEC shit out their standard). And the only reason why “gibibyte” is ridiculous, is because we all grew up with JEDEC interpretation of SI units. So it will take many generations for everyone to adapt xxbityte words into daily conversations. I’m (already) doing my part. It’s just the legacy that we have to deal with.

            All international bodies (BIPM, NIST, EU) agree that the SI prefixes “refer strictly to powers of 10” and that the binary definitions “should not be used” for them.

            https://en.wikipedia.org/wiki/Binary_prefix#IEC_1999_Standard

            https://en.wikipedia.org/wiki/Binary_prefix#Other_standards_bodies_and_organizations

            https://en.wikipedia.org/wiki/JEDEC_memory_standards#JEDEC_Standard_100B.01

            • @[email protected]
              link
              fedilink
              21 month ago

              Well, thank you for taking the time to write this detailed explanation!

              Windows and MacOS use the abbriviation “MB” referring to the binary units, correct? How come that these big OS’s use another unit than these large international bodies recognize?

              On a side note, I’ve always found it weird why HDDs or SSDs are/were sold with 128GB, 265GB, 512GB etc. when they are referring to decimal units.

              • Andrew
                link
                fedilink
                01 month ago

                Windows and MacOS use the abbriviation “MB” referring to the binary units, correct?

                Yez. I’m only sure about the first one, but didn’t test myself whether the macOS is using power of 2 or 10 under the hood (of MB). You can open properties of something big and try converting raw number of bytes with /1024^n and /1000^n and compare the end results.

                How come that these big OS’s use another unit than these large international bodies recognize?

                Legacy, legacy everywhere (IMO). And of course they don’t want to confuse their precious users that don’t know any better. And this also would break some scripts that rely on that specific output. GNU C library also uses JEDEC units by default, hence flatpak and other software.

                On a side note, I’ve always found it weird why HDDs or SSDs are/were sold with 128GB, 265GB, 512GB etc. when they are referring to decimal units.

                It is weird for everyone, because we mainly only count with multiples of 2 when it comes to digital size of information. I didn’t investigate why they use power of 10, but I’ve seen that some other hardware also uses decimal units (I think at least in RAM, but JEDEC is used intentionally or not for CPU cache memory). I had a link where the RAM thingy is lightly addressed, but I couldn’t find it.

                spoiler

                P.S. it’s “OSes” and “macOS” BTW.

              • Andrew
                link
                fedilink
                01 month ago

                Ok, show me what I did wrong and what should I do instead to not be a prick, please.

  • @[email protected]
    link
    fedilink
    111 month ago

    You jest, but i asked for a similar (but much simpler) vector / polygon model, and it generated it.

  • @[email protected]
    link
    fedilink
    111 month ago

    Rockstar making GTA online be like: “Computer, here is a 512mb json file please download it from the server and then do nothing with it”