File storage abstraction with local filesystem, AWS S3 and Google Cloud Storage backends. Includes file validation, image processing, and signed URL generation.
| Namespace | Purpose |
|---|---|
| Pure: file validation (size, type, extension), filename sanitization |
| Protocols: |
| Malli schemas: |
| Service layer orchestrating validation + storage |
| Local filesystem adapter (content-addressed with SHA-256) |
| AWS S3 / S3-compatible adapter (MinIO, DigitalOcean Spaces) |
| Google Cloud Storage adapter |
| Java AWT image processing (no external deps) |
| Ring handlers for upload/download/delete |
(require '[wagoe.storage.ports :as ports])
(ports/store-file storage
{:bytes (.getBytes "content") :content-type "text/plain"}
{:filename "test.txt"})
;=> {:key "ab/1234-uuid-hash.txt" :url "..." :size 7 :content-type "text/plain"}
(ports/retrieve-file storage "ab/1234-uuid-hash.txt")
(ports/file-exists? storage "ab/1234-uuid-hash.txt")
(ports/delete-file storage "ab/1234-uuid-hash.txt")
(ports/generate-signed-url storage key 3600) ; local signs only with :signing-secret(require '[wagoe.storage.core.validation :as validation])
(validation/validate-file
{:bytes file-bytes :content-type "image/png"}
{:filename "photo.png"}
{:max-size (* 5 1024 1024)
:allowed-types #{"image/jpeg" "image/png" "image/webp"}
:allowed-extensions #{"jpg" "jpeg" "png" "webp"}})(ports/resize-image processor file-bytes {:width 800 :height 600}) ; nil keeps the aspect ratio
(ports/create-thumbnail processor file-bytes 150) ; longest side, in pixelsLocal: {shard}/{timestamp}-{uuid}-{hash16}.{ext} (shard = first 2 chars of SHA-256)
S3: {prefix}/{timestamp}-{uuid8}.{ext}
:wagoe/storage in :active, chosen by :provider (default :local).
;; Local (development) — :root or :base-path
:wagoe/storage {:provider :local :root "/var/uploads"}
;; S3 (production); :endpoint for S3-compatible services
:wagoe/storage {:provider :s3 :bucket #env AWS_S3_BUCKET :region #env AWS_REGION
:access-key #env AWS_ACCESS_KEY_ID :secret-key #env AWS_SECRET_ACCESS_KEY}
;; Google Cloud Storage
:wagoe/storage {:provider :gcs :bucket "b" :project-id "p"}Every upload through the storage service is recorded in the storage_uploads
table (IUploadIndex): key, owner user and tenant, size, content type and when
it was stored. The upload routes take the owner from the request; other callers
pass :owner {:user-id … :tenant-id …} in the metadata. An upload the index
cannot record fails, and its file is deleted. Files stored before the index
existed, or straight through IFileStorage, are not in it.
The index feeds two privacy contributions:
a :storage section in a user’s or tenant’s data export, listing their
uploads; a zip export includes the files.
the :storage/uploads retention policy, which deletes uploads stored before
its period, file and record:
:wagoe/privacy {:retention {:storage/uploads {:retain "P1Y"}}}With the jobs module enabled, a signed-in user can have their export built in the background rather than during the request, for a subject whose zip takes too long to stream:
POST /api/v1/me/export-jobs queues it (202). While one is open, asking
again returns the same job.
GET /api/v1/me/export-jobs/:id reports pending, running, done or
failed, and when done a download-path.
GET /api/v1/me/export-jobs/:id/download serves the zip for 24 hours. On S3
and GCS it redirects to a signed URL that lives five minutes; on local
storage the route sends the file.
Platform admins ask for someone else’s export the same way, at
POST /api/v1/users/:id/export-jobs and POST /api/v1/tenants/:id/export-jobs,
and follow it at the Location they get back, under /api/v1/me/export-jobs.
The export is written to the audit log (the user’s, or tenant_audit_log) when
it is asked for, because that is where the admin and their request are known.
The admin can follow the job and download it only while they are a platform admin.
Only the user who asked sees a job; to anyone else it does not exist. A user
has at most one open job per subject, even when requests race. A user may
open five an hour
(:export-requests-per-hour in :wagoe/storage, 0 for no limit); more
answer 429. The archive is streamed to a temporary file under the storage
root’s .spool/ and stored private under data-exports/; it is not an upload.
A spool left by a crash is deleted at the next boot. Jobs run on the
:default queue, or :export-queue in :wagoe/storage.
Nothing deletes old archives until you configure the
:storage/export-archives retention policy. There is no default because
retention in Wagoe is always opt-in: an upgrade must not start deleting data.
After 24 hours an archive cannot be downloaded, but it still holds personal
data, so configure the policy when you enable export jobs:
:wagoe/privacy {:retention {:storage/export-archives {:retain "P2D"}}}clojure -M:test :storageCan you improve this documentation? These fine people already did:
thijscreemers & Thijs CreemersEdit on GitHub
cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |