Use latest CmdStan

[WardBrian]

Resolves a build failure on new XCode versions

[tcuongd]

Thank you!

[jagfirerwalker]

Databrick is calling: 'Prophet' object has no attribute 'stan_backend'

Even with: import cmdstanpy; cmdstanpy.install_cmdstan(overwrite=True)

Is there a bug?

[WardBrian]

Hi @jagfirerwalker -

Can you enable debug logging and see what error is being raised by the cmdstanpy backend? There very well could be a bug, but I doubt it was related to this PR

[jagfirerwalker]

Looks like there is an issue here:

prophet/python/prophet/forecaster.py

Line 164 in 39b3ba2

logger.debug("Trying to load backend: %s", i.name)

Should it be more like this?
try:
logger.debug("Trying to load backend: %s", i.name)
self.stan_backend = StanBackendEnum.get_backend_class(i.name)()
logger.debug("Loaded stan backend: %s", self.stan_backend.get_type())
return # Success, exit
except Exception as e:
logger.debug("Unable to load backend %s (%s), trying the next one", i.name, e)

Got it working with this workaround for Databrick

def setup_prophet_for_databricks():
"""Setup Prophet to work in Databricks environment"""

# Set cmdstan path
cmdstan_path = os.path.expanduser("~/.cmdstan/cmdstan-2.37.0")
os.environ['CMDSTAN'] = cmdstan_path
cmdstanpy.set_cmdstan_path(cmdstan_path)

# Get and compile the Stan model
stan_model_code = importlib_resources.files("prophet") / "stan_model" / "prophet.stan"

with stan_model_code.open('r') as f:
    model_code = f.read()

with tempfile.NamedTemporaryFile(mode='w', suffix='.stan', delete=False) as f:
    f.write(model_code)
    stan_file = f.name

# Compile the model
stan_model = cmdstanpy.CmdStanModel(stan_file=stan_file)

# Patch CmdStanPyBackend to use our compiled model
original_init = CmdStanPyBackend.__init__

def patched_init(self):
    self.model = stan_model
    self.stan_fit = None
    self.newton_fallback = True

CmdStanPyBackend.__init__ = patched_init

return True

Run the setup once

setup_prophet_for_databricks()

[WardBrian]

I don't believe there is an error in that code, and it hasn't changed in nearly 5 years. If you enable logging we can probably help you find the issue

[jagfirerwalker]

Thanks for that. It is most likely some weird Databrick issue. However the following is my work around:

def setup_prophet_for_any_environment():
"""Setup Prophet to work in any environment (Databricks, Airflow, etc.)"""

# Check possible cmdstan locations
possible_paths = [
    os.path.expanduser("~/.cmdstan/cmdstan-2.37.0"),  # User home
    "/root/.cmdstan/cmdstan-2.37.0",  # Root user (Airflow)
    "/home/spark-7701f3de-f23f-4b15-83c2-4e/.cmdstan/cmdstan-2.37.0",  # Databricks
]

cmdstan_path = None
for path in possible_paths:
    if os.path.exists(path):
        cmdstan_path = path
        print(f"Found cmdstan at: {path}")
        break

# If not found, install it
if cmdstan_path is None:
    print("CmdStan not found, installing...")
    cmdstanpy.install_cmdstan(overwrite=True, verbose=True)
    cmdstan_path = os.path.expanduser("~/.cmdstan/cmdstan-2.37.0")
    print(f"Installed cmdstan at: {cmdstan_path}")

# Set the path
os.environ['CMDSTAN'] = cmdstan_path
cmdstanpy.set_cmdstan_path(cmdstan_path)

# Get and compile the Stan model
stan_model_code = importlib_resources.files("prophet") / "stan_model" / "prophet.stan"

with stan_model_code.open('r') as f:
    model_code = f.read()

with tempfile.NamedTemporaryFile(mode='w', suffix='.stan', delete=False) as f:
    f.write(model_code)
    stan_file = f.name

# Compile the model
stan_model = cmdstanpy.CmdStanModel(stan_file=stan_file)

# Patch CmdStanPyBackend to use our compiled model
original_init = CmdStanPyBackend.__init__

def patched_init(self):
    self.model = stan_model
    self.stan_fit = None
    self.newton_fallback = True

CmdStanPyBackend.__init__ = patched_init

print("Prophet setup complete!")
return True