An index kept as files in a store: a manifest in EDN, and a file for each segment in the Common Index File Format, CIFF, which other search engines can read.
CIFF is that of https://github.com/osirrc/ciff, its README and its CommonIndexFileFormat.proto of 2020-03. What CIFF lacks has fields of its own, which a reader of CIFF skips, as the encoding of protobuf has it, https://protobuf.dev/programming-guides/encoding/
An index kept as files in a store: a manifest in EDN, and a file for each segment in the Common Index File Format, CIFF, which other search engines can read. CIFF is that of https://github.com/osirrc/ciff, its README and its CommonIndexFileFormat.proto of 2020-03. What CIFF lacks has fields of its own, which a reader of CIFF skips, as the encoding of protobuf has it, https://protobuf.dev/programming-guides/encoding/
(plan index)(plan index names)The files to put into a store that holds the files of the names, and
the names to delete from it, so that it holds index.
It's a map of :put, files as maps of :name and :body to put in this
order, and :delete, the names of the segments that index no longer
has. Without names, it's every file. The store needs a place of its
own, e.g. a folder, since the segments of any other index there are
deleted.
The :body of the manifest is EDN, and that of a segment is CIFF, as a byte array on the JVM and a Uint8Array in JavaScript. A segment is named after what it holds, so it's put once. Put the files before you delete, so that a store that a crash stops in between still holds an index:
(let [{:keys [put delete]} (plan idx (map :name (list-files)))]
(doseq [{:keys [name body]} put]
(put-file! name body))
(run! delete-file! delete))
The files to put into a store that holds the files of the `names`, and
the names to delete from it, so that it holds `index`.
It's a map of :put, files as maps of :name and :body to put in this
order, and :delete, the names of the segments that `index` no longer
has. Without `names`, it's every file. The store needs a place of its
own, e.g. a folder, since the segments of any other index there are
deleted.
The :body of the manifest is EDN, and that of a segment is CIFF, as a
byte array on the JVM and a Uint8Array in JavaScript. A segment is
named after what it holds, so it's put once. Put the files before you
delete, so that a store that a crash stops in between still holds an
index:
(let [{:keys [put delete]} (plan idx (map :name (list-files)))]
(doseq [{:keys [name body]} put]
(put-file! name body))
(run! delete-file! delete))(read-files fetched)The index in the files fetched, maps of :name and :body: the manifest
index.edn, as text or UTF-8 bytes, and the segments it lists.
It throws ::malformed when a file is missing or doesn't read, e.g. for an app to build the index again, and ::newer-format for files of a newer version of the format.
The index in the files `fetched`, maps of :name and :body: the manifest index.edn, as text or UTF-8 bytes, and the segments it lists. It throws ::malformed when a file is missing or doesn't read, e.g. for an app to build the index again, and ::newer-format for files of a newer version of the format.
cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |