Proposal: support multiple feeds/sources

[tstojecki]

Background

Frankfurter currently sources all exchange rate data from the European Central Bank (ECB).
While ECB provides EUR-based rates, there is significant community demand for additional data sources to support:

1. More currency pairs - Many central banks publish rates not covered by ECB
2. Regional accuracy - Local central banks often provide more accurate rates for their currencies
3. Data redundancy - Multiple sources provide backup when one source is unavailable

Current State

Looking at the roadmap in README.md, "Multiple Data Sources" is listed as a TODO item. Is there any existing work on multiple data sources currently in progress?

Proposed Implementation (scratch if already established)

1. DB

Add a source column to the currencies table to track data origin:

ALTER TABLE currencies ADD COLUMN source VARCHAR DEFAULT 'ecb';
UPDATE currencies SET source = 'ecb' WHERE source IS NULL;
CREATE UNIQUE INDEX idx_currencies_date_iso_source ON currencies(date, iso_code, source);

2. Backend

# lib/feeds/ecb.rb - Extract existing logic
module Feeds
  class ECB
    include Enumerable
    def self.current; end
    def self.historical; end
    def self.ninety_days; end
  end
end

# lib/feeds/boc.rb
module Feeds  
  class BOC
    include Enumerable
    def self.current; end
    def self.historical; end
    def self.ninety_days; end
  end
end

3. API

/v1/latest              # ECB (default, backward compatible)
/v1/ecb/latest
/v1/boc/latest 
/v1/ecb/2024-01-01
/v1/boc/2024-01-01

Goals

• Backward Compatibility: Existing API consumers continue working unchanged
• Extensible: Easy to add new sources following the same interface
• Testable: Each provider can be tested independently

[hakanensari]

Thanks for starting this discussion! This has been on my back burner for a couple of years. Your approach is solid and simple.

Some considerations off the top of my head:

• We'd need a base column since different sources use different base currencies
• The current Bank::Feed hardcodes ECB. We could extract a base Feed class, then create source-specific subclasses.
• Your path-based /v1/[:source]/latest approach is clean. Each endpoint would return rates in its native base currency.

And deferring to v2: Blending multiple sources, a provider model, schema refinements.

@claude, what do you think about this implementation approach? Could you scan past issues to identify which sources people have actually requested? Then, sketch out the MVP implementation, a rough PR structure to ship this without overthinking.

[hakanensari]

Claude's not here, pasting their response by hand:

---

Good approach. The base currency point is crucial - ECB uses EUR, BOC uses CAD, etc.

Looking at past issues, here are the most requested sources with actual user suggestions:

Candidate Sources (from issue tracker)

1. Turkish Central Bank (TCMB) - #84

   • XML: https://www.tcmb.gov.tr/kurlar/YYYYMM/DDMMYYYY.xml
   • Covers: RUB, SAR, AED, PKR, KWD, AZN, QAR, IRR
   • Daily updates, USD base

2. National Bank of Ukraine (NBU) - #78

   • https://bank.gov.ua/NBUStatService/v1/statdirectory/exchange?json
   • Covers: UAH (and others)
   • Daily updates, UAH base

3. Bank of Canada - #47

   • https://www.bankofcanada.ca/rates/exchange/daily-exchange-rates/
   • Wide currency coverage
   • Daily updates, CAD base, JSON/CSV/XML

4. Colombian Government (Banco de la República) - #44

   • https://dev.socrata.com/foundry/www.datos.gov.co/mcec-87by
   • Covers: COP/USD
   • Daily updates

The UN Treasury source (#56) was also suggested but only updates twice monthly, so probably not suitable.

MVP Implementation

1. Database

ALTER TABLE currencies ADD COLUMN source VARCHAR DEFAULT 'ecb';
ALTER TABLE currencies ADD COLUMN base VARCHAR DEFAULT 'EUR';
CREATE UNIQUE INDEX idx_currencies_date_iso_source ON currencies(date, iso_code, source);

2. Extract base Feed class

# lib/bank/feed.rb
module Bank
  class Feed
    include Enumerable
    
    def self.source_name
      raise NotImplementedError
    end
    
    def self.base_currency
      raise NotImplementedError
    end
    
    def self.current_url
      raise NotImplementedError
    end
    
    # Common parsing/storage logic here
  end
end

# lib/bank/ecb_feed.rb  
module Bank
  class ECBFeed < Feed
    def self.source_name; 'ecb'; end
    def self.base_currency; 'EUR'; end
    def self.current_url; ECB_CURRENT_URL; end
  end
end

3. Minimal API changes

# Keep backward compatibility
get '/latest' do
  Quote.from_bank(Bank::ECBFeed, params)
end

# New source-specific endpoints
get '/:source/latest' do
  feed_class = Bank.const_get("#{params[:source].upcase}Feed")
  Quote.from_bank(feed_class, params)
end

PR Structure

1. PR 1: Database migration + Feed base class extraction (no functionality change)
2. PR 2: Add a second source (proves the abstraction works, covers most requested currencies)
3. PR 3: API routing for source-specific endpoints
4. PR 4+: Add remaining sources one by one

[tstojecki]

Good point on the base currency. Come to think of it, for my use cases, going with this approach is a bit of problem.
The accuracy difference between the synthetic cross-rates and native rates (as provided by the central bank of that currency) is significant enough to matter.

I'm curious, do you want this API to be doing these sorts of cross-rate calculations at all? Or provide rates only when you have the underlying's central bank data?
I suppouse you can have both, that is if the caller specifies they want EUR base for CAD -> USD, it's their choice. If later they specify BOC, they get a slightly different rate.

This feels like a bit of a strict mode and could very well introduce a breaking change for many currencies so versioning is critical. I believe this solution is popular enough to where you do have to weigh that in, congrats :)

Thoughts on this?

[hakanensari]

Yes, v1 gets backwards-compatible additions only. Breaking changes belong in v2.

[tstojecki]

Agreed.

v2:

Option A: Flexible (synthetic + native)

GET /v2/latest?from=CAD&to=USD&source=ecb    # Synthetic via EUR base
GET /v2/latest?from=CAD&to=USD&source=boc    # Native CAD base
GET /v2/latest?from=CAD&to=USD               # Auto-select BOC (native)

Option B: Strict mode (native only)

GET /v2/latest?from=CAD&to=USD    # Uses BOC (CAD base) or 400 if unavailable
GET /v2/latest?from=GBP&to=USD    # 400: GBP base not supported

[tstojecki]

#151
this defaults to strict source mode with cross-source rebasing available via explicit &source parameter

[hakanensari]

Hi @tstojecki! Apologies for the long delay in getting back to you. I haven’t been able to spend much time here recently.

I really appreciate the work that went into this PR. That said, after revisiting our discussion in #141, it looks like we’re not quite aligned on the direction:

1. For v1, the idea was to add new sources incrementally and keep things backward compatible, i.e. v1/latest continues returning ECB rates, while we introduce new endpoints like v1/ecb/latest, v1/foo/latest, etc.
2. For v2, my intention was to have blended/synthetic data by default, with access to individual feeds for those who need them (your strict mode).
3. On routing, I don’t think v2 should necessarily follow v1. This is a chance to introduce breaking changes, so better to take time and get it right rather than rush into it.

I’d suggest instead to start with a smaller step, say, adding one new source (e.g. Bank of Canada) to v1 using source-specific endpoints. Once that’s running and stable, it's a good point to revisit v2 and design the blending strategy properly.

I’d be happy to reopen the discussion and figure out next steps. Thanks again for putting serious effort into this!

[tstojecki]

Hi @hakanensari , all good - yeah, I figured we didn't align and so moved on. Happy to reopen the PR (and probably target v2 rather main), but let's make sure my use case makes sense for this project in the first place.

My use case:

I can't use synthetic/blended rates - I need the actual central bank rates backing each currency. This is why I jumped straight to
v2.

What the PR already does:

The current implementation defaults to native source mode:

• /v2/latest?from=EUR&to=USD → Uses ECB (native EUR source) ✅
• /v2/latest?from=CAD&to=USD → Returns 400 error (no BOC data yet) ❌
• /v2/latest?from=CAD&to=USD&source=ECB → Rebases from ECB (explicit synthetic) ✅

So it's strict by default, synthetic only when explicitly requested via &source param. This is opposite of your v2 vision
(synthetic default, native optional).

Next steps if this aligns:

I can add the second source to demonstrate how multi-source would work in practice. We'd need to figure out the feed
fetching side as each bank has a different set of files/apis to work with. But let me know if strict-by-default v2 makes sense for the project. If not, totally understand.

Skill:

I've since built a simple Claude skill that works this way - different workflow
(upload to Claude Code, rates cached locally, no network calls), but meets my needs. I am still interested in an mcp though as skills are claude only.

[hakanensari]

I was thinking blended by default. Let me clarify the thinking.

To speculate a little, I think there are two distinct use cases:

1. Coverage users (online stores, general apps) just want "the rate" and don't care about sub-basis-point precision.
2. Compliance users (accounting, legal requirements) need specific official rates from documented sources and require reproducibility.

Probably most users, including my original use case, are in the first camp. And I understand you fall in the latter, with the added layer of wanting to access multiple sources.

A blended by default v2 would be a two-tier API contract.

The blended top-level endpoints (/latest, /2024-01-15) use available data from all sources, evolve as new sources are added, and provide maximum coverage.

Then, single-source endpoints (/ecb/latest, /boc/2024-01-15) provide data that's authoritative and hence deterministic and reproducible.

Would the single-source endpoints not work for your use case? It requires hardcoding providers or minimal app-level routing to determine which provider to use based on base currency, but gives you full control and reproducibility.

Does this sound reasonable?

[tstojecki]

there are two distinct use cases:

• Coverage users (online stores, general apps) just want "the rate" and don't care about sub-basis-point precision.
• Compliance users (accounting, legal requirements) need specific official rates from documented sources and require reproducibility.

Exactly.

The single-source endpoint works. That's what the PR does using a query &source but I like your route better and that should be a quick change
/v2/latest?from=CAD&to=USD&source=boc

use available data from all sources... provide maximum coverage

With multiple sources that can provide the synthentic rate, how will you determine which bank's rates to use when it's not explictly stated?

[lineoffligbot]

Multi-source support shipped in v2. The API now blends rates from 20+ providers (central banks, IMF, Federal Reserve). See the v2 docs at /v2/openapi.json for details.

Closing as completed.