Monkey Projects OCI Object Storage

This is a Clojure library to access the Oracle OCI Object Storage API. You could of course use the OCI provided Java lib instead, but I've found that it's pretty cumbersome to use. Also, since it automatically marshalls everything into Java POJOs, it's not very efficient. The Clojure way is to simply process the incoming objects as a data structure.

Another reason why I wrote this is because I want to use it in a GraalVM native image, and the Java lib simply has too many dependencies which make it difficult to build a native image and it would be bloated anyway.

Structure

The lib provides two layers to access the API. One is the low-level access, that is just a thin layer on top of the REST calls. On top of that there is a layer that provides convenience functions for often-used scenarios.

Under the hood it uses Martian to send HTTP requests to the OCI API. We're using the Aleph plugin, because that http client is able to download large objects (files) without consuming much memory.

Usage

Include the library in your project:

{:deps {com.monkeyprojects/oci-os {:mvn/version ..latest..}}}

Then include the namespace and create a context:

(require '[monkey.oci.os.core :as os])

;; Configuration, must contain the necessary properties to connect,
;; see oci-sign for that
(def config {:user-ocid ... }
(def ctx (os/make-client config))

;; Now you can make requests
(def bucket-ns @(os/get-namespace ctx))  ; Returns the bucket namespace
@(os/list-objects ctx {:ns bucket-ns :bucket-name "my-bucket"}) ; Lists bucket objects

In order to gain access, you must provide the necessary configuration. This is then used to sign the request. See the oci-sign library for details, but you will need your tenancy OCID, user OCID, private key, key fingerprint and the region you're targeting.

The functions in the core namespace will unwrap the response by default, returning the response body on success. If there is a failure, an exception will be thrown, with the full response in the ex-data. See below on how to send requests and gain access to the raw response map.

Uploading Files

Creating or updating files is done with the put-object function. The options map should contain the :ns (namespace), :bucket-name, :object-name and :contents values. The contents is a string with the file contents. Request signature require calculating an SHA256 hash for the body, so streaming is not supported. For larger files, you should use multipart requests (see below).

The upload and download requests don't produce JSON so the calls return the underlying Aleph response, which contains also a :body value.

@(os/put-object ctx {:ns "..." :bucket-name "test-bucket" :object-name "test.txt" :contents "this is a test file"})
;; This will return an empty string on success

;; Now you can download the file as well
@(os/get-object {:ns "..." :bucket-name "test-bucket" :object-name "test.txt"})
;; Returns the file contents from the :body.  Depending on the content type,
;; this can be a string or an input stream.

By default the Content-Type is application/octet-stream. But you can override this by specifying the raw header in the request options:

@(os/put-object {:ns "..."
                 ...
		 :contents "File contents"
		 :martian.core/request {:headers {"content-type" "text.plain"}}})

This will explicitly pass in the Content-Type header to the backend, which will also be returned when you download the file.

Multipart Uploads

Often you will want to upload very large files, or maybe even streams of which you don't know beforehand how large they are (e.g. logfiles). For this you can use multipart uploads. With these you can upload a large object in chunks. This library provides a wrapper around this, found in the monkey.oci.os.stream namespace.

There are two main functions here: stream->multipart and input-stream->multipart. The first takes a Manifold stream and uploads each incoming message as a new multipart object. This is useful for real-time streaming uploads.

The second takes a regular Java InputStream and uploads it until EOF has been reached, or the stream is closed. After that, it commits the multipart and the object is created in the bucket. Be sure to close the stream yourself. This is most useful for large files. An example:

(require '[monkey.oci.os.stream :as oss])
(require '[manifold.deferred :as md])
(require '[clojure.java.io :as io])

;; Open the stream
(def is (io/input-stream "/path/to/very-large-file"))
;; Upload it
(md/chain
 (oss/input-stream->multipart ctx
  {:ns "my-bucket-ns"
   :object-name "/destination/path"
   :bucket-name "my-bucket"
   :input-stream "is"})
 (fn [r]
   ;; Close when EOF reached
   (.close is)
   r))
;; This will return the result of the commit operation, after closing the file.

You can also use a finally handler, to ensure the file is closed even in the case of errors. You can also pass in :close? true in the options to do this. The options map accepts the following values:

Key	Required?	Default value	Description
:ns	Yes		The namespace where the bucket resides
:bucket-nane	Yes		The name of the bucket to upload to
:object-name	Yes		The name of the destination object
:input-stream	Yes		Input stream to read from
:content-type	No	application/binary	The content type to add as metadata
:close?	No	false	Should the stream be closed after upload?
:buf-size	No	0x10000	Max size of each part that is being uploaded
:progress	No	nil	A function that will be invoked after each part upload

A progress fn can be passed if you want to be notified of upload progress. It receives a structure with the input arguments as well as the upload id (as assigned by OCI) and the total number of bytes already uploaded up to that point.

Low-level Calls

Should you need access to the full response, for example to read certain headers like ETag, you can send requests using the lower-level monkey.oci.os.martian namespace. These contain about the same functions (one for every defined route), but they won't interpret the response, and instead return the full response map.

(require '[monkey.oci.os.martian :as m])

@(m/head-object {:ns ...})
;; This will return the full response, with :headers to inspect, etc...

This allows you to have more control over how requests are handled. This can also be useful should you want to handle 'expected' 4xx responses, instead of catching exceptions (which is bad form if you're actually expecting it to happen, right?)

Metadata

Object storage allows storing additional custom information with objects in the form of metadata. These are passed on as headers when you use the put-object call. If you use multipart uploads, you should pass a :metadata property in the body. The input-stream->multipart also allows :metadata in the options map. Note that metadata keys must start with :opc-meta-, otherwise they will be ignored.

;; Explicitly pass headers using the :martian.core/headers property
(os/put-object ctx {:object-name "..." :martian.core/headers {:opc-meta-test-key "test value"}})

;; Or when using multipart
(oss/input-stream->multipart ctx {... :metadata {:opc-meta-test-key "test value"}})

Since the core functions automatically unwrap the response and only return the body, you can't get the values of the metadata like this. Instead, you will have to resort to the low level calls instead.

;; Fetch object details
(def r @(m/get-object ctx {:object-name "..."}))

;; Headers will contain the opc-meta-... values
(:headers r)
;; => {:opc-meta-test-key "test value"}, among others

TODO

• Add something that automagically generates the Martian routes from the OCI provided Java libs. (Or find the OpenAPI specs.)

Copyright

Copyright (c) 2023-2024 by Monkey Projects BV.

MIT License