Couldn't store trigger exception

[AnKa0]

Description

We are unable to publish any workflow with a cron trigger in it, in case the cron expression is filled, because the following exception is thrown (stack trace attached below):
Quartz.JobPersistenceException: Couldn't store trigger 'Default.cf146dda767734a9' for 'Default.RunWorkflowJob' job: The job (Default.RunWorkflowJob) referenced by the trigger does not exist.

Adding a Timer activity and running the workflow results in the same exception.

Steps to Reproduce

1. Create a new workflow with a cron trigger.
2. Fill in the cron expression.
3. Press save and publish.

Attachments:

• Workflow JSON: If your issue is reproducible with a specific workflow, please attach the workflow's JSON file. This will help us understand the exact process that leads to the problem.

cron trigger

timer

Reproduction Rate: Indicate how often the bug occurs when following these steps (e.g., "every time", "intermittently: about 50% of the time").
It had not been present before, but it suddenly became present in both the test environment and on development machines. It had been tested on a different, fully new, empty database as well - the issue is present that way too.

Video/Screenshots

Expected Behavior

The workflow is published.

Actual Behavior

The workflow is not published, because an exception stops the process.

Environment

• Elsa Package Version: 3.5.1
• Operating System: Windows 11, Linux

Log Output

Quartz.JobPersistenceException: Couldn't store trigger 'Default.cf146dda767734a9' for 'Default.RunWorkflowJob' job: The job (Default.RunWorkflowJob) referenced by the trigger does not exist.
---> Quartz.JobPersistenceException: The job (Default.RunWorkflowJob) referenced by the trigger does not exist.
at Quartz.Impl.AdoJobStore.JobStoreSupport.StoreTrigger(ConnectionAndTransactionHolder conn, IOperableTrigger newTrigger, IJobDetail job, Boolean replaceExisting, String state, Boolean forceState, Boolean recovering, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at Quartz.Impl.AdoJobStore.JobStoreSupport.StoreTrigger(ConnectionAndTransactionHolder conn, IOperableTrigger newTrigger, IJobDetail job, Boolean replaceExisting, String state, Boolean forceState, Boolean recovering, CancellationToken cancellationToken)
at Quartz.Impl.AdoJobStore.JobStoreSupport.<>c__DisplayClass268_0.<b__0>d.MoveNext()
--- End of stack trace from previous location ---
at Quartz.Impl.AdoJobStore.JobStoreSupport.ExecuteInNonManagedTXLock[T](String lockName, Func2 txCallback, Func3 txValidator, Nullable1 requestorId, CancellationToken cancellationToken) at Quartz.Impl.AdoJobStore.JobStoreSupport.ExecuteInNonManagedTXLock[T](String lockName, Func2 txCallback, Func3 txValidator, Nullable1 requestorId, CancellationToken cancellationToken)
at Quartz.Core.QuartzScheduler.ScheduleJob(ITrigger trigger, CancellationToken cancellationToken)
at Elsa.Quartz.Services.QuartzWorkflowScheduler.ScheduleJobAsync(IScheduler scheduler, ITrigger trigger, CancellationToken cancellationToken)
at Elsa.Quartz.Services.QuartzWorkflowScheduler.ScheduleCronAsync(String taskName, ScheduleNewWorkflowInstanceRequest request, String cronExpression, CancellationToken cancellationToken)
at Elsa.Scheduling.Services.DefaultTriggerScheduler.ScheduleAsync(IEnumerable1 triggers, CancellationToken cancellationToken) at Elsa.Scheduling.Handlers.ScheduleWorkflows.HandleAsync(WorkflowTriggersIndexed notification, CancellationToken cancellationToken) at Elsa.Mediator.PublishingStrategies.SequentialProcessingStrategy.PublishAsync(NotificationStrategyContext context) at Elsa.Mediator.Middleware.Notification.Components.NotificationHandlerInvokerMiddleware.InvokeAsync(NotificationContext context) at Elsa.Mediator.Middleware.Notification.NotificationPipeline.ExecuteAsync(NotificationContext context) at Elsa.Mediator.Services.DefaultMediator.SendAsync(INotification notification, IEventPublishingStrategy strategy, CancellationToken cancellationToken) at Elsa.Mediator.Services.DefaultMediator.SendAsync(INotification notification, CancellationToken cancellationToken) at Elsa.Workflows.Runtime.TriggerIndexer.IndexTriggersAsync(Workflow workflow, CancellationToken cancellationToken) at Elsa.Workflows.Runtime.TriggerIndexer.IndexTriggersAsync(WorkflowDefinition definition, CancellationToken cancellationToken) at Elsa.Workflows.Runtime.Handlers.IndexTriggers.HandleAsync(WorkflowDefinitionPublished notification, CancellationToken cancellationToken) at Elsa.Mediator.PublishingStrategies.SequentialProcessingStrategy.PublishAsync(NotificationStrategyContext context) at Elsa.Mediator.Middleware.Notification.Components.NotificationHandlerInvokerMiddleware.InvokeAsync(NotificationContext context) at Elsa.Mediator.Middleware.Notification.NotificationPipeline.ExecuteAsync(NotificationContext context) at Elsa.Mediator.Services.DefaultMediator.SendAsync(INotification notification, IEventPublishingStrategy strategy, CancellationToken cancellationToken) at Elsa.Mediator.Services.DefaultMediator.SendAsync(INotification notification, CancellationToken cancellationToken) at Elsa.Workflows.Management.Services.WorkflowDefinitionPublisher.PublishAsync(WorkflowDefinition definition, CancellationToken cancellationToken) at Elsa.Workflows.Api.Endpoints.WorkflowDefinitions.Post.Post.HandleAsync(SaveWorkflowDefinitionRequest request, CancellationToken cancellationToken) at Elsa.Workflows.Api.Endpoints.WorkflowDefinitions.Post.Post.HandleAsync(SaveWorkflowDefinitionRequest request, CancellationToken cancellationToken) at FastEndpoints.Endpoint2.ExecAsync(CancellationToken ct)
at FastEndpoints.Endpoint`2.ExecAsync(CancellationToken ct)
at Microsoft.AspNetCore.Routing.EndpointMiddleware.g__AwaitRequestTask|7_0(Endpoint endpoint, Task requestTask, ILogger logger)
at NSwag.AspNetCore.Middlewares.SwaggerUiIndexMiddleware.Invoke(HttpContext context)
at NSwag.AspNetCore.Middlewares.RedirectToIndexMiddleware.Invoke(HttpContext context)
at NSwag.AspNetCore.Middlewares.SwaggerUiIndexMiddleware.Invoke(HttpContext context)
at NSwag.AspNetCore.Middlewares.RedirectToIndexMiddleware.Invoke(HttpContext context)
at NSwag.AspNetCore.Middlewares.OpenApiDocumentMiddleware.Invoke(HttpContext context)
at Elsa.Http.Middleware.HttpWorkflowsMiddleware.InvokeAsync(HttpContext httpContext, IServiceProvider serviceProvider)
at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddlewareImpl.Invoke(HttpContext context) [See nested exception: Quartz.JobPersistenceException: The job (Default.RunWorkflowJob) referenced by the trigger does not exist.
at Quartz.Impl.AdoJobStore.JobStoreSupport.StoreTrigger(ConnectionAndTransactionHolder conn, IOperableTrigger newTrigger, IJobDetail job, Boolean replaceExisting, String state, Boolean forceState, Boolean recovering, CancellationToken cancellationToken)]

Troubleshooting Attempts

Tried the same steps with an HTTP trigger, which works.
Set up a completely new database and tried the same steps - the issue was still present.

[lecramr]

I stumbled across the same issue.

[lecramr]

Looks like the issue is the CRON Expression so "0 0 * ? * *" as proposed by @AnKa0 is not working. While "0/60 0 0 ? * * *" seems to work just fine.

@sfmskywalker maybe simple Expression Validation would be enough?

[lecramr]

@sfmskywalker created a simple PR: #7133

[ascendedguard]

We've run into the same issue in our production environment, the odd thing being the CRON expressions were working last week, but as of today they stopped working. We have probably 14 workflows with CRON expressions, attempting to publish any workflow with a CRON expression gives the same 500 error shown above, and the quartz.QRTZ_CRON_TRIGGERS table is completely empty, resulting in none of our scheduled jobs running.

Has anyone found a workaround?

[ascendedguard]

I found what was causing the issue in my case. We had an activity that was outputing a complex object, and the Javascript serialization was throwing a StackOverflowException when trying to set Items on a ForEach node. When this happened, the quartz.QRTZ_CRON_TRIGGERS would consistently get deleted, no real idea why. There were no issues with the CRON expressions in this case.

As a workaround, I could restart the Elsa server, which would repopulate the table on startup.

I was able to reproduce in a dev environment using a custom ExecuteSqlQuery activity that outputs the DataSet object. Executing the workflow would wipe the Quartz table, after which attempting to publish changes to any workflows that contained a CRON expression would save, but would throw a 500 error and show the same symptoms as the original post.