nd-db.convert
->ndnippyclj
(->ndnippy in-db out-filename)Converts .ndjson and .ndedn files to .ndnippy.
.ndnippy is MUCH faster (~10x) and requires less memory to process.
Also the resulting database size CAN BE much smaller: ~27GB .ndedn becomes ~12GB .ndnippy (~65% reduction!) in a sample with large (>10KB) documents.
For databases with small documents, the resulting size will be bigger than the original.
Note that this function doesn't create an index, if you want that immediately, use ->ndnippy-db instead (which is faster than first generating the file and creating the index afterwards)
But if you only need the index, this function is much faster than ->ndnippy-db, since it runs in parallel on available cores.
Converts .ndjson and .ndedn files to .ndnippy. .ndnippy is MUCH faster (~10x) and requires less memory to process. Also the resulting database size _CAN BE_ much smaller: ~27GB .ndedn becomes ~12GB .ndnippy (~65% reduction!) in a sample with large (>10KB) documents. For databases with small documents, the resulting size will be bigger than the original. Note that this function doesn't create an index, if you want that immediately, use ->ndnippy-db instead (which is faster than first generating the file and creating the index afterwards) But if you only need the index, this function is much faster than ->ndnippy-db, since it runs in parallel on available cores.
->ndnippy-dbclj
(->ndnippy-db in-db {:keys [filename id-path id-fn]})Converts .ndjson and .ndedn files to .ndnippy, and returns the corresponding database.
Use this function to convert to .ndnippy, and return a database, if you need to use the database immediately (or just want to pre- create the persistent index for it).
Since the documents from .ndjson or .ndedn will be parsed anyway, you simply supply a path in the document to the ID value via param :id-path.
Alternatively you can use :id-fn for more complex scenarios.
Converts .ndjson and .ndedn files to .ndnippy, and returns the corresponding database. Use this function to convert to .ndnippy, _and_ return a database, if you need to use the database immediately (or just want to pre- create the persistent index for it). Since the documents from .ndjson or .ndedn will be parsed anyway, you simply supply a path in the document to the ID value via param :id-path. Alternatively you can use :id-fn for more complex scenarios.
cljdoc is a website building & hosting documentation for Clojure/Script libraries
× close