Turns raw Yahoo Finance snapshots, as clojure-backtesting.yahoo saves
them, into datasets the backtester loads. Nothing here touches the
network, so a dataset can be rebuilt with other options without
downloading again.
Prices become a main dataset in the CRSP layout. Yahoo quotes every price, volume and dividend in today's share basis, i.e. divided by all later splits; the rows here are as traded on the day (PRC, OPENPRC, BIDLO, ASKHI, VOL, DIVAMT), with CFACPR = CFACSHR = the product of the later splits, RET the total return (CRSP's definition, dividends at the ex-date) and RETX the return without dividends. The adjusted price columns add-aprc would add (INIT-PRICE, APRC, CUM-RET) are computed here as well, so loading the dataset needs no rewrite.
Fundamentals become a point-in-time store: every statement value keeps its period end and the date it became public, taken from Yahoo's earnings report dates where they exist and from a reporting lag otherwise (recorded as such). Values are kept in the latest snapshot's share basis and currency as reported.
Turns raw Yahoo Finance snapshots, as `clojure-backtesting.yahoo` saves them, into datasets the backtester loads. Nothing here touches the network, so a dataset can be rebuilt with other options without downloading again. Prices become a main dataset in the CRSP layout. Yahoo quotes every price, volume and dividend in today's share basis, i.e. divided by all later splits; the rows here are as traded on the day (PRC, OPENPRC, BIDLO, ASKHI, VOL, DIVAMT), with CFACPR = CFACSHR = the product of the later splits, RET the total return (CRSP's definition, dividends at the ex-date) and RETX the return without dividends. The adjusted price columns add-aprc would add (INIT-PRICE, APRC, CUM-RET) are computed here as well, so loading the dataset needs no rewrite. Fundamentals become a point-in-time store: every statement value keeps its period end and the date it became public, taken from Yahoo's earnings report dates where they exist and from a reporting lag otherwise (recorded as such). Values are kept in the latest snapshot's share basis and currency as reported.
(build-dataset! raw-dir
out-dir
&
{:keys [start end allow-mixed-currencies?] :as opts})Builds a dataset under out-dir from the snapshots under raw-dir:
main/ (prices, CRSP layout, furnished with APRC), fundamentals.edn (the
point-in-time store and FX rates) and dataset.edn (provenance, coverage
and warnings). Options:
:start, :end keep price rows within these yyyy-MM-dd dates
:report-lag-days {:quarterly 45 :annual 90 :trailing 45} by default
:after-close-hour local hour from which an announcement counts from
the next day (by default the exchange's close
hour, 16 for most)
:vintage :first (default) or :latest, for values Yahoo
restated between snapshots
:allow-mixed-currencies? build a main dataset whose securities are
quoted in different currencies, pence and pounds
included (the engine values them all as one
currency, so only for strategies that never hold
more than one of them)
Returns the contents of dataset.edn.
Builds a dataset under `out-dir` from the snapshots under `raw-dir`:
main/ (prices, CRSP layout, furnished with APRC), fundamentals.edn (the
point-in-time store and FX rates) and dataset.edn (provenance, coverage
and warnings). Options:
:start, :end keep price rows within these yyyy-MM-dd dates
:report-lag-days {:quarterly 45 :annual 90 :trailing 45} by default
:after-close-hour local hour from which an announcement counts from
the next day (by default the exchange's close
hour, 16 for most)
:vintage :first (default) or :latest, for values Yahoo
restated between snapshots
:allow-mixed-currencies? build a main dataset whose securities are
quoted in different currencies, pence and pounds
included (the engine values them all as one
currency, so only for strategies that never hold
more than one of them)
Returns the contents of dataset.edn.(close-hour zone override)The hour from which an announcement in zone counts from the next day:
override when given, else the exchange's close, 16:00 by default.
The hour from which an announcement in `zone` counts from the next day: `override` when given, else the exchange's close, 16:00 by default.
(daily-bars bars zone before-date)Yahoo's daily bars as maps with a :date in the exchange's time zone, one
per date (the last wins), sorted, without bars lacking a positive close
and without bars dated on or after before-date (the download day,
whose session may not have finished).
Yahoo's daily bars as maps with a :date in the exchange's time zone, one per date (the last wins), sorted, without bars lacking a positive close and without bars dated on or after `before-date` (the download day, whose session may not have finished).
(days-between from to)Days from from to to, both yyyy-MM-dd.
Days from `from` to `to`, both yyyy-MM-dd.
Days after the period end assumed for a report whose date Yahoo does not give, roughly the SEC deadlines for 10-Q and 10-K filings. A quarter or half-year that ends a fiscal year takes the annual lag.
Days after the period end assumed for a report whose date Yahoo does not give, roughly the SEC deadlines for 10-Q and 10-K filings. A quarter or half-year that ends a fiscal year takes the annual lag.
(dividend-events events zone)Cash dividends as [{:date :amount}], sorted by ex-date, amounts in today's share basis as Yahoo gives them.
Cash dividends as [{:date :amount}], sorted by ex-date, amounts in
today's share basis as Yahoo gives them.(earnings-events {:keys [earnings-history earnings-dates]}
zone
after-close-hour)Earnings announcements with their EPS, as [{:report-date :public-date :period-end :eps-actual :eps-estimate :surprise-pct}] sorted by public date, from both Yahoo sources; :period-end is nil when only the announcement date is known. Yahoo's EPS here is the analysts' adjusted figure and its share basis is not consistent (some values are restated for later splits, some not), so compare :surprise-pct, not the EPS.
Earnings announcements with their EPS, as [{:report-date :public-date
:period-end :eps-actual :eps-estimate :surprise-pct}] sorted by public
date, from both Yahoo sources; :period-end is nil when only the
announcement date is known. Yahoo's EPS here is the analysts' adjusted
figure and its share basis is not consistent (some values are restated
for later splits, some not), so compare :surprise-pct, not the EPS.(encode-ticker ticker)A file-system-safe directory name for a ticker.
A file-system-safe directory name for a ticker.
(epoch->date epoch zone)The yyyy-MM-dd date in zone of an epoch-seconds timestamp.
The yyyy-MM-dd date in `zone` of an epoch-seconds timestamp.
(exchange-zone meta)The exchange's time zone from a chart meta map: its IANA name, else its UTC offset, else UTC.
The exchange's time zone from a chart meta map: its IANA name, else its UTC offset, else UTC.
(field-kind yahoo-field)How a Yahoo statement field scales with the share basis: :per-share (EPS and the like, divided by a split), :shares (share counts, multiplied by it), :ratio (a rate, neither scaled nor converted between currencies) or :amount (currency amounts).
How a Yahoo statement field scales with the share basis: :per-share (EPS and the like, divided by a split), :shares (share counts, multiplied by it), :ratio (a rate, neither scaled nor converted between currencies) or :amount (currency amounts).
(fundamentals-entry snapshots
{:keys [report-lag-days after-close-hour vintage]
:or {report-lag-days default-report-lag-days
vintage :first}})The point-in-time store entry for one security from its snapshots (oldest first). Options: :report-lag-days, :after-close-hour, :vintage. Returns {:currency :trading-currency :reports :fields :earnings :dividends :splits :summary}: :currency is the main reporting currency, :trading-currency the quote currency, :reports maps [frequency period-end] to {:report-date :public-date :source :currency}, and :fields maps each field to {:statement :kind :yahoo-field :values {frequency [[period-end value] ...]}} with values oldest first.
The point-in-time store entry for one security from its snapshots
(oldest first). Options: :report-lag-days, :after-close-hour, :vintage.
Returns {:currency :trading-currency :reports :fields :earnings
:dividends :splits :summary}: :currency is the main reporting currency,
:trading-currency the quote currency, :reports maps [frequency
period-end] to {:report-date :public-date :source :currency}, and
:fields maps each field to {:statement :kind :yahoo-field :values
{frequency [[period-end value] ...]}} with values oldest first.(fx-series {:keys [meta bars]})[[date rate] ...] from an FX snapshot's chart, sorted by date.
[[date rate] ...] from an FX snapshot's chart, sorted by date.
(fx-ticker from to)Yahoo's ticker for the rate that converts from into to: "EURUSD=X"
quotes US dollars per euro.
Yahoo's ticker for the rate that converts `from` into `to`: "EURUSD=X" quotes US dollars per euro.
(instant->date s)The UTC yyyy-MM-dd date of an ISO-8601 instant string such as a snapshot's :fetched-at.
The UTC yyyy-MM-dd date of an ISO-8601 instant string such as a snapshot's :fetched-at.
(major-currency currency)[currency factor] such that a price quoted in currency times factor is
in the major unit: GBp (pence) is ["GBP" 0.01].
[currency factor] such that a price quoted in `currency` times factor is in the major unit: GBp (pence) is ["GBP" 0.01].
(price-rows id ticker {:keys [meta bars events]} {:keys [before-date]})The main-dataset rows (in main-columns order) for one security from
its snapshot's :chart, oldest first. id goes in PERMNO and ticker in
TICKER. Options: :before-date drops bars on or after it.
The main-dataset rows (in `main-columns` order) for one security from its snapshot's :chart, oldest first. `id` goes in PERMNO and `ticker` in TICKER. Options: :before-date drops bars on or after it.
(read-snapshots raw-dir){ticker [snapshot ...]} for every security under raw-dir, oldest
snapshot first, plus {:fx {[from to] [snapshot ...]}}.
{ticker [snapshot ...]} for every security under `raw-dir`, oldest
snapshot first, plus {:fx {[from to] [snapshot ...]}}.(rebase kind value factor)Converts value of kind from the share basis of a snapshot to one
factor times as many shares per share.
Converts `value` of `kind` from the share basis of a snapshot to one `factor` times as many shares per share.
(report-candidates {:keys [earnings-history earnings-dates]}
zone
after-close-hour)The report dates known for a security, from Yahoo's earnings history (period end paired with its report date) and its earnings announcement dates (announcements only). Returns {:history [{:period-end :date :public-date}] :announcements [{:date :public-date}]}, each sorted.
The report dates known for a security, from Yahoo's earnings history
(period end paired with its report date) and its earnings announcement
dates (announcements only). Returns {:history [{:period-end :date
:public-date}] :announcements [{:date :public-date}]}, each sorted.(report-date frequency
period-end
{:keys [history announcements]}
{:keys [lag-days as-of annual-ends]})When the report for the period of frequency ending period-end was
made public: {:report-date :public-date :source}. :source is
:earnings-history when Yahoo pairs the period with its report date,
:announcement when the first earnings announcement within a few months
after the period end is taken as its report, and :lag when neither is
known and lag-days after the period end is assumed (the annual lag for
a period ending within ten days of one of annual-ends). Announcements after
as-of (the download day) are ignored: they are only scheduled.
When the report for the period of `frequency` ending `period-end` was
made public: {:report-date :public-date :source}. :source is
:earnings-history when Yahoo pairs the period with its report date,
:announcement when the first earnings announcement within a few months
after the period end is taken as its report, and :lag when neither is
known and `lag-days` after the period end is assumed (the annual lag for
a period ending within ten days of one of `annual-ends`). Announcements after
`as-of` (the download day) are ignored: they are only scheduled.(snapshot-file-name fetched-at)20261031T100000Z.edn for a snapshot fetched at 2026-10-31T10:00:00Z.
20261031T100000Z.edn for a snapshot fetched at 2026-10-31T10:00:00Z.
(split-events events zone)Splits as [{:date :ratio}], sorted, where :ratio is the number of new shares per old share (4.0 for a 4-for-1 split, 0.1 for 1-for-10).
Splits as [{:date :ratio}], sorted, where :ratio is the number of new
shares per old share (4.0 for a 4-for-1 split, 0.1 for 1-for-10).(split-factor-after splits date)Product of the ratios of the splits dated after date: how many of
today's shares one share held on date has become. CRSP calls this
factor CFACPR.
Product of the ratios of the `splits` dated after `date`: how many of today's shares one share held on `date` has become. CRSP calls this factor CFACPR.
(statement-values snapshots splits vintage)Statement values merged over a security's snapshots (oldest first), in
the share basis of the last one. Returns {[field frequency period-end]
{:value :currency :statement :yahoo-field :kind :revised?}}. With
vintage :first a value restated in a later snapshot keeps its earliest
version, the one closest to what was first reported; with :latest the
newest wins. Yahoo's trailing per-share values and share counts for
periods before a split are left out: unlike its quarterly and annual
ones they are not consistently restated (NVDA's trailing EPS is quoted
per pre-split share up to January 2024 and per post-split share from
April 2024, both before its June 2024 split), so for those periods
twelve-month per-share figures come from the quarters or the year.
Statement values merged over a security's snapshots (oldest first), in
the share basis of the last one. Returns {[field frequency period-end]
{:value :currency :statement :yahoo-field :kind :revised?}}. With
`vintage` :first a value restated in a later snapshot keeps its earliest
version, the one closest to what was first reported; with :latest the
newest wins. Yahoo's trailing per-share values and share counts for
periods before a split are left out: unlike its quarterly and annual
ones they are not consistently restated (NVDA's trailing EPS is quoted
per pre-split share up to January 2024 and per post-split share from
April 2024, both before its June 2024 split), so for those periods
twelve-month per-share figures come from the quarters or the year.cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |