feat: handle on-chain TLC settlement for force-closed channels

[jjyr]

Summary

• Supersedes feat: handle on-chain TLC settlement for force-closed channels #1254 and closes [CCH] CchOrder status is not updated when outgoing TLC is settled on-chain via fiber (e.g. force close / watchtower) #1222.
• Separate on-chain TLC settlement without preimage from on-chain fulfillment where the preimage is discovered by the watchtower.
• Reconcile fulfilled on-chain TLCs into channel, payment, and invoice state.
• Persist discovered on-chain preimages into the regular preimage store so CCH receives the same PutPreimage success signal as off-chain fulfillment.

Testing

• cargo nextest run -p fnn --features rocksdb test_payer_payment_success_from_onchain_preimage
• cargo nextest run -p fnn --features rocksdb onchain_preimage
• cargo nextest run -p fnn --features rocksdb cch::tests
• cargo fmt --all
• git diff --check

Notes

This PR starts from the implementation in #1254 and adds the CCH-facing preimage bridge needed for CCH order reconciliation.

[gpBlockchain]

Integration-test finding: already-fulfilled offered TLC leaves payer payment Inflight

I ran an end-to-end force-close regression against this PR at 027c6aca7c07139ea6f2072a33df342651471068.

When a one-hop hold invoice is settled after the channel has been force-closed, the payee invoice and channel TLC state are reconciled, but the original payer's payment session is not finalized.

Observed result

Payment	Payer payment	Payee invoice	TLC state
CKB two-hop	Success	Paid	removed
UDT two-hop	Success	Paid	removed
CKB one-hop	Inflight	Paid	payer: Outbound.RemoteRemoved; payee: Inbound.LocalRemoved
UDT one-hop	Inflight	Paid	payer: Outbound.RemoteRemoved; payee: Inbound.LocalRemoved

The test stops at the first one-hop assertion:

self.wait_payment_state(self.fiber1, ckb_two_hop, "Success", timeout=360)
self.wait_invoice_state(fiber3, ckb_two_hop, "Paid", timeout=360)
self.wait_payment_state(self.fiber1, udt_two_hop, "Success", timeout=360)
self.wait_invoice_state(fiber3, udt_two_hop, "Paid", timeout=360)

> self.wait_payment_state(self.fiber1, ckb_one_hop, "Success", timeout=360)

test_force_close_fulfill.py:218

The payer RPC still returns:

{
  "payment_hash": "0xfd2cbd562ff6b4b2c402d4cbfdd3b5de13ffe99d4ac0b8d5665ef30466457936",
  "status": "Inflight",
  "created_at": "0x19f5a59a292",
  "last_updated_at": "0x19f5a59a292",
  "failed_error": null,
  "fee": "0x0"
}

This is not only a slow watchtower update: after the on-chain unlock completes, the invoice is Paid and the corresponding TLC is already recorded as removed, while get_payment remains permanently Inflight.

Root cause

The normal committed removal path calls apply_remove_tlc_operation. For an original payer's offered TLC, it emits NetworkActorEvent::TlcRemoveReceived, which is forwarded to the PaymentActor as OnRemoveTlcEvent. RemoveTlcFulfill then marks the attempt and payment session successful.

The fresh on-chain fulfillment path in settle_onchain_fulfilled_tlcs has equivalent behavior:

self.store.insert_preimage(tlc.payment_hash, tlc.preimage);
self.network
    .send_message(NetworkActorMessage::new_event(
        NetworkActorEvent::TlcRemoveReceived(
            tlc.payment_hash,
            tlc.attempt_id,
            fulfill.clone(),
        ),
    ))
    .expect(ASSUME_NETWORK_ACTOR_ALIVE);
state.tlc_state.set_offered_tlc_removed(id, fulfill);

However, can_reconcile_onchain_fulfillment excludes a TLC as soon as it already has a removal reason:

if tlc.removed_reason.is_some() || tlc.removed_confirmed_at.is_some() {
    return false;
}

Such TLCs instead reach sync_already_fulfilled_onchain_tlcs. Its offered-TLC branch only changes the TLC status:

for (id, reason) in offered_updates {
    state.tlc_state.set_offered_tlc_removed(id, reason);
}

It does not emit TlcRemoveReceived for an origin payer or RelayOnChainTlcRemove for a forwarding node. Therefore the channel RPC shows RemoteRemoved, but the payer PaymentActor never receives the successful fulfillment event.

Expected behavior

Already-fulfilled offered-TLC reconciliation should perform the same role-aware side effects as fresh on-chain reconciliation:

• original payer: persist the preimage and emit TlcRemoveReceived, resulting in payment Success;
• forwarding node: relay the fulfillment upstream;
• update and persist the TLC state;
• remain idempotent across repeated watchtower checks and restarts.

Full regression test

Run with:

pytest -s \
  test_cases/fiber/devnet/watch_tower_wit_tlc/test_force_close_fulfill.py::TestForceCloseFulfill::test_force_close_payee_settle_invoice_ckb_and_udt

test_force_close_fulfill.py

import time

from framework.basic_fiber import FiberTest
from framework.util import ckb_hash


class TestForceCloseFulfill(FiberTest):
    """
    Regression tests for nervosnetwork/fiber PR #1254.

    When a channel is force-closed and the channel actor is gone, a later
    RemoveTlc(Fulfill) from the peer should still be handled through persisted
    channel state, so the payer payment does not stay Inflight forever.
    """
    start_fiber_config = {"fiber_watchtower_check_interval_seconds": 3}

    def _wait_force_close_unlock(self, timeout=600):
        if len(self.get_commit_cells()) == 0:
            raise Exception("No commit cells found")
        self.node.getClient().generate_epochs("0x1", wait_time=0)
        for _ in range(timeout // 10):
            if len(self.get_commit_cells()) == 0:
                return
            time.sleep(10)
        assert len(self.get_commit_cells()) == 0

    def _get_channel_id(self, local, remote, udt=None, timeout=120):
        for _ in range(timeout):
            channels = local.get_client().list_channels(
                {"pubkey": remote.get_pubkey()}
            )["channels"]
            for channel in channels:
                if channel.get("funding_udt_type_script") != udt:
                    continue
                if channel["state"]["state_name"] == "ChannelReady":
                    return channel["channel_id"]
            time.sleep(1)
        raise TimeoutError(f"ChannelReady not found, udt={udt}")

    def _get_tlc_status(self, fiber, remote_pubkey, channel_id, payment_hash):
        channels = fiber.get_client().list_channels(
            {
                "pubkey": remote_pubkey,
                "include_closed": True,
            }
        )["channels"]
        for channel in channels:
            if channel["channel_id"] != channel_id:
                continue
            for tlc in channel.get("pending_tlcs", []):
                if tlc["payment_hash"] == payment_hash:
                    return tlc["status"]
        raise AssertionError(
            f"TLC {payment_hash} not found in channel {channel_id} on {fiber.get_pubkey()}"
        )

    def _assert_sender_remote_removed(self, sender, receiver, channel_id, payment_hash):
        status = self._get_tlc_status(
            sender,
            receiver.get_pubkey(),
            channel_id,
            payment_hash,
        )
        assert status == {"Outbound": "RemoteRemoved"}

    def _assert_receiver_local_removed(self, receiver, sender, channel_id, payment_hash):
        status = self._get_tlc_status(
            receiver,
            sender.get_pubkey(),
            channel_id,
            payment_hash,
        )
        assert status == {"Inbound": "LocalRemoved"}

    def _new_hold_invoice(self, payee, payment_hash, description, udt=None):
        params = {
            "amount": hex(1 * 100000000),
            "currency": "Fibd",
            "description": description,
            "payment_hash": payment_hash,
            # "allow_mpp": True,
            "allow_trampoline_routing": True,
        }
        if udt is not None:
            params["udt_type_script"] = udt
        return payee.get_client().new_invoice(params)

    def _send_two_hop(self, payee, description, udt=None):
        preimage = self.generate_random_preimage()
        payment_hash = ckb_hash(preimage)
        invoice = self._new_hold_invoice(payee, payment_hash, description, udt)
        payment = self.fiber1.get_client().send_payment(
            {
                "invoice": invoice["invoice_address"],
                "trampoline_hops": [
                    self.fiber2.get_client().node_info()["pubkey"],
                ],
            }
        )
        assert payment["payment_hash"] == payment_hash
        self.wait_payment_state(self.fiber1, payment_hash, "Inflight")
        self.wait_invoice_state(payee, payment_hash, "Received")
        return payment_hash, preimage

    def _send_one_hop(self, payee, description, udt=None):
        preimage = self.generate_random_preimage()
        payment_hash = ckb_hash(preimage)
        invoice = self._new_hold_invoice(payee, payment_hash, description, udt)
        payment = self.fiber1.get_client().send_payment(
            {
                "invoice": invoice["invoice_address"],
                "max_fee_rate": hex(1000000000000000),
            }
        )
        assert payment["payment_hash"] == payment_hash
        self.wait_payment_state(self.fiber1, payment_hash, "Inflight")
        self.wait_invoice_state(payee, payment_hash, "Received")
        return payment_hash, preimage

    def test_force_close_payee_settle_invoice_ckb_and_udt(self):
        """
        Open CKB and UDT A-B/B-C channels first.

        Then send both two-hop payments, send both one-hop payments, force-close
        all affected channels, settle all invoices, and assert final states after
        one shared on-chain unlock wait.
        """
        udt_script = self.get_account_udt_script(self.fiber1.account_private)
        fiber3 = self.start_new_fiber(
            self.generate_account(
                10000,
                self.fiber1.account_private,
                10000 * 100000000,
            )
        )
        self.faucet(
            self.fiber2.account_private,
            0,
            self.fiber1.account_private,
            10000 * 100000000,
        )
        self.faucet(
            self.fiber1.account_private,
            0,
            self.fiber1.account_private,
            10000 * 100000000,
        )

        self.open_channel(self.fiber1, self.fiber2, 1000 * 100000000, 0)
        self.open_channel(self.fiber2, fiber3, 1000 * 100000000, 0)
        self.open_channel(
            self.fiber1, self.fiber2, 1000 * 100000000, 0, udt=udt_script
        )
        self.open_channel(
            self.fiber2, fiber3, 1000 * 100000000, 0, udt=udt_script
        )

        ckb_bc = self._get_channel_id(self.fiber2, fiber3)
        ckb_ab = self._get_channel_id(self.fiber1, self.fiber2)
        udt_bc = self._get_channel_id(self.fiber2, fiber3, udt_script)
        udt_ab = self._get_channel_id(self.fiber1, self.fiber2, udt_script)

        time.sleep(10)
        ckb_two_hop, ckb_two_hop_preimage = self._send_two_hop(
            fiber3,
            "pr1254 CKB two-hop hold invoice",
        )
        udt_two_hop, udt_two_hop_preimage = self._send_two_hop(
            fiber3,
            "pr1254 UDT two-hop hold invoice",
            udt_script,
        )

        self.fiber2.get_client().shutdown_channel(
            {"channel_id": ckb_bc, "force": True}
        )
        self.fiber2.get_client().shutdown_channel(
            {"channel_id": udt_bc, "force": True}
        )

        ckb_one_hop, ckb_one_hop_preimage = self._send_one_hop(
            self.fiber2,
            "pr1254 CKB one-hop hold invoice",
        )
        udt_one_hop, udt_one_hop_preimage = self._send_one_hop(
            self.fiber2,
            "pr1254 UDT one-hop hold invoice",
            udt_script,
        )

        self.fiber1.get_client().shutdown_channel(
            {"channel_id": ckb_ab, "force": True}
        )
        self.fiber1.get_client().shutdown_channel(
            {"channel_id": udt_ab, "force": True}
        )

        time.sleep(10)
        fiber3.get_client().settle_invoice(
            {"payment_hash": ckb_two_hop, "payment_preimage": ckb_two_hop_preimage}
        )
        fiber3.get_client().settle_invoice(
            {"payment_hash": udt_two_hop, "payment_preimage": udt_two_hop_preimage}
        )
        self.fiber2.get_client().settle_invoice(
            {"payment_hash": ckb_one_hop, "payment_preimage": ckb_one_hop_preimage}
        )
        self.fiber2.get_client().settle_invoice(
            {"payment_hash": udt_one_hop, "payment_preimage": udt_one_hop_preimage}
        )
        self._wait_force_close_unlock()

        self.wait_payment_state(self.fiber1, ckb_two_hop, "Success", timeout=360)
        self.wait_invoice_state(fiber3, ckb_two_hop, "Paid", timeout=360)
        self.wait_payment_state(self.fiber1, udt_two_hop, "Success", timeout=360)
        self.wait_invoice_state(fiber3, udt_two_hop, "Paid", timeout=360)
        self.wait_payment_state(self.fiber1, ckb_one_hop, "Success", timeout=360)
        self.wait_invoice_state(self.fiber2, ckb_one_hop, "Paid", timeout=360)
        self.wait_payment_state(self.fiber1, udt_one_hop, "Success", timeout=360)
        self.wait_invoice_state(self.fiber2, udt_one_hop, "Paid", timeout=360)

        self._assert_sender_remote_removed(self.fiber2, fiber3, ckb_bc, ckb_two_hop)
        self._assert_receiver_local_removed(fiber3, self.fiber2, ckb_bc, ckb_two_hop)
        self._assert_sender_remote_removed(self.fiber2, fiber3, udt_bc, udt_two_hop)
        self._assert_receiver_local_removed(fiber3, self.fiber2, udt_bc, udt_two_hop)
        self._assert_sender_remote_removed(
            self.fiber1,
            self.fiber2,
            ckb_ab,
            ckb_one_hop,
        )
        self._assert_receiver_local_removed(
            self.fiber2,
            self.fiber1,
            ckb_ab,
            ckb_one_hop,
        )
        self._assert_sender_remote_removed(
            self.fiber1,
            self.fiber2,
            udt_ab,
            udt_one_hop,
        )
        self._assert_receiver_local_removed(
            self.fiber2,
            self.fiber1,
            udt_ab,
            udt_one_hop,
        )

[gpBlockchain]

Integration-test finding: MPP force-close with one consumed split leaves payer payment Inflight

I reran the MPP force-close regression against this PR head 027c6aca7c07:

pytest -vv -s \
  test_cases/fiber/devnet/watch_tower_wit_tlc/test_force_close_mpp.py::TestForceCloseMpp::test_mpp_force_close_one_channel_only_one_tlc_consumed

Scenario

• A opens two A-B channels; B opens one B-C channel.
• C creates one hold invoice with allow_mpp=True; A pays with max_parts=2.
• B force-closes only one A-B channel.
• C settles the invoice.
• The force-close unlock completes.

Observed result

The payee side reaches the terminal state, but the payer session does not:

self._wait_force_close_unlock()
self.wait_invoice_state(fiber3, payment_hash, "Paid", timeout=360)
> self.wait_payment_state(self.fiber1, payment_hash, "Success", timeout=360)

TimeoutError: payment:0x8b5e6a12cb1e717ada215ce41655c955c8404fb6545dce04bb3e562be5956482 status did not reach state: Inflight, expected:Success, within timeout period.

A local rerun reproduced the same shape:

1 failed in 468.67s
payment_hash=0xfbbc99a864765f99d47a93454d49652c9e7c18d8f357a0246691d88d36ba48bc
force_closed_channel=0xccbb1653bd4f8ff72a8db029b14b618e94d988cb0d6678b7a0237071ee1fa4b8
offchain_channel=0x6f4c65a1d5007c428ffd326ec6468f47f4018f91c22945ec427b677a62901594

The final state before teardown was:

• C invoice: Paid
• off-chain A-B split: no pending TLC left
• force-closed A-B split on A: Closed(UNCOOPERATIVE_REMOTE | WAITING_ONCHAIN_SETTLEMENT), pending TLC Outbound.RemoteRemoved
• force-closed A-B split on B: matching pending TLC Inbound.LocalRemoved
• A payment: still Inflight

This does not look like a timeout-only issue. In the rerun, the shutdown scan fired, A detected the remote close, and watchtower observed the on-chain preimage, but the payment remained Inflight after the 360s wait.

Root cause candidate

The payer receives a RemoveTlcFulfill for the offered TLC before the removal is commitment-confirmed:

• handle_remove_tlc_peer_message stores the preimage and calls set_offered_tlc_removed, which sets the offered TLC to Outbound.RemoteRemoved and records removed_reason=Some(RemoveTlcFulfill(...)).
• The normal payer completion event is emitted later from apply_remove_tlc_operation via NetworkActorEvent::TlcRemoveReceived. Because this channel is force-closed, that normal apply-remove path never finishes for this split.
• On-chain reconciliation also misses it. can_reconcile_onchain_fulfillment returns false as soon as tlc.removed_reason.is_some(), and sync_already_fulfilled_onchain_tlcs only considers offered TLCs whose status is still Committed. This force-closed split is already RemoteRemoved, so it is excluded from both paths.

The consequence is that collect_onchain_fulfilled_tlcs never returns this TLC and no later path emits TlcRemoveReceived for the original payer. The PaymentActor therefore keeps one MPP attempt Inflight, even though the invoice is Paid and the on-chain preimage was confirmed.

Relevant code paths:

• crates/fiber-lib/src/fiber/channel.rs: handle_remove_tlc_peer_message, apply_remove_tlc_operation, settle_onchain_fulfilled_tlcs, sync_already_fulfilled_onchain_tlcs
• crates/fiber-lib/src/fiber/onchain_tlc_reconcile.rs: can_reconcile_onchain_fulfillment
• crates/fiber-types/src/channel.rs: set_offered_tlc_removed

Expected behavior

Already-fulfilled offered TLCs whose removal is known but not commitment-confirmed should still be reconciled once the on-chain preimage is verified.

For this case, reconciliation should be role-aware and idempotent:

• original payer: persist the preimage and emit TlcRemoveReceived, so the payment becomes Success;
• forwarding node: relay the fulfill upstream;
• state update should not call set_offered_tlc_removed again on a TLC that is already RemoteRemoved, because that helper asserts the current status is Committed;
• repeated watchtower checks or restarts should not duplicate the payer/relay side effect.

So I think the integration assertion is valid: after invoice == Paid and force-close unlock completes, the payer payment should eventually reach Success, not remain Inflight.