I have a JSON object with a huge array of nested objects. Let us assume it consists of records of license plates for vehicles. It would contain necessary fields like licenseID, issuingState, dateOfIssue, driverID etc.
What I am having problem with is how I should store data that is only used for exceptional cases, like a field for representing if the license plate is for foreign embassies (isEmbassyOwned) or if it is owned by a government entity (isGovernmentOwned) or if it is a learner license (isLearner) etc alongside fields with data types other than Boolean which would be empty or 0 and likewise when there is no information on that field. Let it be known that these exceptional scenarios would occur in less than 10% of total object instances.
I am facing confusion as to what format would be best for storing such type of data keeping balance between minimizing storage consumption and being human readable. Should I declare the fields for all objects regardless or only include them when they are not empty? Should I store them in a dedicated array instead, or maybe just introduce some code value to be used by a switch case operator in the interpreter? Or is there some other implementation I am not aware of?
Depending on your needs you can also break it into a columnar format with some standard compression on top. This allows you to search individual fields without looking at the rest.
It also compress exceptionally well, and “rare” fields will be null in most records, so run length encoding will compress them to near zero
See fx parquet