Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Cross-base requests for pegged quotes are now anchored through the peg's base. For example, `?base=EUR&quotes=AED` previously returned the blended `EUR/AED` rate from providers; it now returns `blended(EUR/USD) × peg(USD/AED)`. The numerical difference is small (basis points) but removes spurious provider-disagreement noise on quantities that are mathematically pinned by the issuing authority. (#323)
- Pegs are now treated as a source of rate data alongside providers. When `?providers=` filters the source set, pegs are excluded along with all other unlisted sources. Requests like `?base=BMD&providers=ecb` (where ECB does not publish BMD) now return empty rather than synthesizing rates from the peg. Default behavior (no `providers=`) is unchanged. (#323)
- IMF Special Drawing Rights (XDR) is no longer filtered out of provider backfills. Several providers (NB, SBI, BCRA, etc.) publish XDR rates that were silently dropped; they now flow through like any other ISO 4217 quote. Re-backfill from `coverage_start` to ingest previously-dropped rows. (#333)
- `/v2/rates` responses now stamp every row with its actual observation date rather than a flattened reporting date. On the latest and single-date paths, a pair carried forward from an earlier date now reports that earlier date. Range queries no longer carry forward at all: each pair appears on the days it actually published, with its own date. Pairs that didn't publish during the range are absent rather than projected. Mixed-cadence providers (e.g. weekly publishers) appear at their actual publication frequency. (#338)
- Deutsche Bundesbank (BBK) as historical provider — daily pre-euro Frankfurt fixings for 18 currencies, 1948-06-21 through 1998-12-30
- Bank of Russia (CBR) precious-metal reference prices — daily XAU, XAG, XPT and XPD against RUB, available from 2008-07-01. CBR is the first source for platinum and palladium beyond the National Bank of Ukraine.
- National Bank of Moldova (NBM) precious-metal reference prices — daily XAU and XAG against MDL, available from 2012-01-02. Brings XAG to four providers (CBA, NBU, CBR, NBM), clearing the consensus threshold for silver.
Expand Down
75 changes: 35 additions & 40 deletions lib/carry_forward.rb
Original file line number Diff line number Diff line change
@@ -1,51 +1,46 @@
# frozen_string_literal: true

# Carries forward each provider's most recent rate within a lookback window. Used for single-date
# queries (latest) and range query enrichment.
module CarryForward
LATEST_LOOKBACK_DAYS = 14
RANGE_LOOKBACK_DAYS = 5
# Produces a snapshot of rates as of a target date by carrying forward each provider's most recent
# rate within a lookback window. Used for single-date and latest queries; range queries do not
# carry forward.
class CarryForward
LOOKBACK_DAYS = 14

class << self
# Returns the most recent rate per (provider, base, quote) on or before the target date, within
# the lookback window.
def latest(rows, date:, lookback: LATEST_LOOKBACK_DAYS)
cutoff = date - lookback
best = {}
def apply(rows, date:, lookback: LOOKBACK_DAYS)
new(rows, date:, lookback:).apply
end
end

rows.each do |row|
d = row[:date]
next unless d&.between?(cutoff, date)
attr_reader :rows, :date, :lookback

key = [row[:provider], row[:base], row[:quote]]
best[key] = row if !best[key] || d > best[key][:date]
end
def initialize(rows, date:, lookback:)
@rows = rows
@date = date
@lookback = lookback
end

best.values
def apply
best = {}
eligible_rows.each do |row|
key = key_for(row)
best[key] = row if !best[key] || row[:date] > best[key][:date]
end

# Enriches each date in the target range with carried-forward rates. Returns { date => [rows] }
# where each date's rows include both same-day rates and each provider's most recent rate within
# the lookback window. Carried-forward rows keep their original dates so WeightedAverage can
# discount them by staleness.
def enrich(rows, range:, lookback: RANGE_LOOKBACK_DAYS)
by_date = rows.group_by { |r| r[:date] }
target_dates = by_date.keys.select { |d| range.cover?(d) }.sort

index = {}
rows.each do |row|
key = [row[:provider], row[:base], row[:quote]]
(index[key] ||= []) << row
end
index.each_value { |v| v.sort_by! { |r| r[:date] }.reverse! }

target_dates.to_h do |date|
cutoff = date - lookback
group = index.filter_map do |_, dated_rows|
dated_rows.find { |r| r[:date].between?(cutoff, date) }
end
[date, group]
end
end
best.values
end
Comment on lines +23 to +31
Copy link

Copilot AI Apr 28, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

eligible_rows allocates an intermediate array via select, which is avoidable since apply already iterates. You can iterate rows once inside apply and next rows outside the window; this reduces memory churn and speeds up processing for larger row sets.

Copilot uses AI. Check for mistakes.
Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Deferring — the helper-method structure (with cutoff, eligible_rows, key_for) is intentional for readability. Row counts here are bounded by LOOKBACK_DAYS × providers (low thousands at most), so the extra allocation is negligible.


private

def cutoff
@cutoff ||= date - lookback
end

def eligible_rows
rows.select { |r| r[:date].between?(cutoff, date) }
end
Comment on lines +39 to +41
Copy link

Copilot AI Apr 28, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

eligible_rows allocates an intermediate array via select, which is avoidable since apply already iterates. You can iterate rows once inside apply and next rows outside the window; this reduces memory churn and speeds up processing for larger row sets.

Copilot uses AI. Check for mistakes.
Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as above — keeping eligible_rows as a named helper.


def key_for(row)
[row[:provider], row[:base], row[:quote]]
end
end
2 changes: 1 addition & 1 deletion lib/versions/v1/currency_names.rb
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ def find_currencies

today = Date.today
rows = Rate.where(provider: "ECB").where(date: (today - 14)..today).naked.all
CarryForward.latest(rows, date: today)
CarryForward.apply(rows, date: today)
end
end
end
Expand Down
2 changes: 1 addition & 1 deletion lib/versions/v1/quote/end_of_day.rb
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ def fetch_data
scope = Rate.where(provider: "ECB").where(date: (date_val - 14)..date_val)
scope = scope.only(*(symbols + [base])) if symbols

CarryForward.latest(scope.naked.all, date: date_val)
CarryForward.apply(scope.naked.all, date: date_val)
end
end
end
Expand Down
33 changes: 11 additions & 22 deletions lib/versions/v2/rate_query.rb
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,8 @@ class RateQuery

class ValidationError < StandardError; end

ALLOWED_EXPANSIONS = ["providers"].freeze
ALLOWED_PARAMS = ["base", "quotes", "providers", "date", "from", "to", "group", "expand"].freeze
CHUNK_MONTHS = { "week" => 21, "month" => 84 }.freeze
DEFAULT_CHUNK_MONTHS = 3

Expand All @@ -41,23 +43,15 @@ def each(&block)
ds = range_dataset
date_col = ds.model.date_column

if rollup?
rows = ds.between(chunk_range).all
normalize_dates!(rows, date_col) if date_col != :date
rows.group_by { |r| r[:date] }.each do |_, group_rows|
emit_blended(group_rows, &block)
end
else
expanded = (chunk_range.begin - CarryForward::RANGE_LOOKBACK_DAYS)..chunk_range.end
rows = ds.between(expanded).naked.all
CarryForward.enrich(rows, range: chunk_range).each do |target_date, group_rows|
emit_blended(group_rows, target_date:, &block)
end
rows = ds.between(chunk_range).all
normalize_dates!(rows, date_col) if date_col != :date
rows.group_by { |r| r[:date] }.each do |_, group_rows|
emit_blended(group_rows, &block)
end
end
else
window = raw_dataset.where(date: (date_scope - CarryForward::LATEST_LOOKBACK_DAYS)..date_scope)
rows = CarryForward.latest(window.naked.all, date: date_scope)
window = raw_dataset.where(date: (date_scope - CarryForward::LOOKBACK_DAYS)..date_scope)
rows = CarryForward.apply(window.naked.all, date: date_scope)
emit_blended(rows, &block)
end
end
Expand All @@ -81,7 +75,7 @@ def max_date
if date_scope.is_a?(Range)
ds.where(date: date_scope).max(:date)
else
ds.where(date: (date_scope - CarryForward::LATEST_LOOKBACK_DAYS)..date_scope).max(:date)
ds.where(date: (date_scope - CarryForward::LOOKBACK_DAYS)..date_scope).max(:date)
end
end

Expand Down Expand Up @@ -165,9 +159,6 @@ def parse_date(value)
nil
end

ALLOWED_PARAMS = ["base", "quotes", "providers", "date", "from", "to", "group", "expand"].freeze
ALLOWED_EXPANSIONS = ["providers"].freeze

def validate!
validate_params!
validate_dates!
Expand Down Expand Up @@ -218,17 +209,15 @@ def date_scope
end
end

def emit_blended(rows, target_date: nil, &block)
def emit_blended(rows, &block)
blended = Blender.new(rows, base: providers ? base : effective_base).blend
blended = PegAnchor.apply(blended, base: base, base_peg: base_peg) unless providers
return if blended.empty?

output_date = (target_date || blended.map { |r| r[:date] }.max)&.to_s

records = blended.filter_map do |r|
next if quotes && !quotes.include?(r[:quote])

record = { date: output_date, base: r[:base], quote: r[:quote], rate: round(r[:rate]) }
record = { date: r[:date].to_s, base: r[:base], quote: r[:quote], rate: round(r[:rate]) }
record[:providers] = r[:providers] if expand_providers? && r[:providers]
record
end
Expand Down
109 changes: 7 additions & 102 deletions spec/carry_forward_spec.rb
Original file line number Diff line number Diff line change
Expand Up @@ -4,15 +4,15 @@
require "carry_forward"

describe CarryForward do
describe ".latest" do
describe ".apply" do
it "returns the most recent rate per provider/base/quote" do
rows = [
{ date: Date.new(2024, 1, 5), provider: "ECB", base: "EUR", quote: "USD", rate: 1.08 },
{ date: Date.new(2024, 1, 4), provider: "ECB", base: "EUR", quote: "USD", rate: 1.07 },
{ date: Date.new(2024, 1, 5), provider: "BOC", base: "CAD", quote: "USD", rate: 0.74 },
]

result = CarryForward.latest(rows, date: Date.new(2024, 1, 6))
result = CarryForward.apply(rows, date: Date.new(2024, 1, 6))

_(result.size).must_equal(2)
ecb = result.find { |r| r[:provider] == "ECB" }
Expand All @@ -26,7 +26,7 @@
{ date: Date.new(2024, 1, 1), provider: "ECB", base: "EUR", quote: "USD", rate: 1.08 },
]

result = CarryForward.latest(rows, date: Date.new(2024, 1, 20), lookback: 14)
result = CarryForward.apply(rows, date: Date.new(2024, 1, 20), lookback: 14)

_(result).must_be_empty
end
Expand All @@ -36,7 +36,7 @@
{ date: Date.new(2024, 1, 1), provider: "ECB", base: "EUR", quote: "USD", rate: 1.08 },
]

result = CarryForward.latest(rows, date: Date.new(2024, 1, 15), lookback: 14)
result = CarryForward.apply(rows, date: Date.new(2024, 1, 15), lookback: 14)

_(result.size).must_equal(1)
end
Expand All @@ -46,7 +46,7 @@
{ date: Date.new(2024, 1, 10), provider: "ECB", base: "EUR", quote: "USD", rate: 1.08 },
]

result = CarryForward.latest(rows, date: Date.new(2024, 1, 9))
result = CarryForward.apply(rows, date: Date.new(2024, 1, 9))

_(result).must_be_empty
end
Expand All @@ -57,109 +57,14 @@
{ date: Date.new(2024, 1, 3), provider: "ECB", base: "EUR", quote: "GBP", rate: 0.86 },
]

result = CarryForward.latest(rows, date: Date.new(2024, 1, 6))
result = CarryForward.apply(rows, date: Date.new(2024, 1, 6))

_(result.size).must_equal(2)
_(result.map { |r| r[:quote] }.sort).must_equal(["GBP", "USD"])
end

it "returns empty array for empty input" do
_(CarryForward.latest([], date: Date.new(2024, 1, 6))).must_be_empty
end
end

describe ".enrich" do
it "carries forward rates from prior days into dates within the range" do
friday = Date.new(2024, 1, 5)
saturday = Date.new(2024, 1, 6)

rows = [
{ date: friday, provider: "ECB", base: "EUR", quote: "USD", rate: 1.08 },
{ date: friday, provider: "BOC", base: "CAD", quote: "USD", rate: 0.74 },
{ date: saturday, provider: "HNB", base: "EUR", quote: "USD", rate: 1.09 },
]

result = CarryForward.enrich(rows, range: saturday..saturday)

_(result.keys).must_equal([saturday])
providers = result[saturday].map { |r| r[:provider] }.sort

_(providers).must_equal(["BOC", "ECB", "HNB"])
end

it "preserves original dates on carried-forward rows" do
friday = Date.new(2024, 1, 5)
saturday = Date.new(2024, 1, 6)

rows = [
{ date: friday, provider: "ECB", base: "EUR", quote: "USD", rate: 1.08 },
{ date: saturday, provider: "HNB", base: "EUR", quote: "USD", rate: 1.09 },
]

result = CarryForward.enrich(rows, range: saturday..saturday)
ecb = result[saturday].find { |r| r[:provider] == "ECB" }

_(ecb[:date]).must_equal(friday)
end

it "excludes carry-forward beyond the lookback window" do
old = Date.new(2024, 1, 1)
target = Date.new(2024, 1, 8)

rows = [
{ date: old, provider: "ECB", base: "EUR", quote: "USD", rate: 1.08 },
{ date: target, provider: "HNB", base: "EUR", quote: "USD", rate: 1.09 },
]

result = CarryForward.enrich(rows, range: target..target, lookback: 5)
providers = result[target].map { |r| r[:provider] }

_(providers).must_include("HNB")
_(providers).wont_include("ECB")
end

it "only returns dates within the target range" do
friday = Date.new(2024, 1, 5)
saturday = Date.new(2024, 1, 6)
sunday = Date.new(2024, 1, 7)

rows = [
{ date: friday, provider: "ECB", base: "EUR", quote: "USD", rate: 1.08 },
{ date: saturday, provider: "HNB", base: "EUR", quote: "USD", rate: 1.09 },
]

result = CarryForward.enrich(rows, range: saturday..sunday)

_(result.keys).must_equal([saturday])
_(result).wont_include(friday)
end

it "picks the most recent rate per provider within the lookback" do
wed = Date.new(2024, 1, 3)
fri = Date.new(2024, 1, 5)
sat = Date.new(2024, 1, 6)

rows = [
{ date: wed, provider: "ECB", base: "EUR", quote: "USD", rate: 1.07 },
{ date: fri, provider: "ECB", base: "EUR", quote: "USD", rate: 1.08 },
{ date: sat, provider: "HNB", base: "EUR", quote: "USD", rate: 1.09 },
]

result = CarryForward.enrich(rows, range: sat..sat)
ecb = result[sat].find { |r| r[:provider] == "ECB" }

_(ecb[:rate]).must_equal(1.08)
_(ecb[:date]).must_equal(fri)
end

it "returns empty hash when no dates have data in range" do
rows = [
{ date: Date.new(2024, 1, 1), provider: "ECB", base: "EUR", quote: "USD", rate: 1.08 },
]

result = CarryForward.enrich(rows, range: Date.new(2024, 2, 1)..Date.new(2024, 2, 2))

_(result).must_be_empty
_(CarryForward.apply([], date: Date.new(2024, 1, 6))).must_be_empty
end
end
end
Loading
Loading