Background Jobs & Messaging

Cargonerds runs deferred, recurring and event-driven work through several complementary mechanisms, all built on ABP Framework abstractions:

Mechanism	Used for	Backing technology
ABP Background Jobs	One-off / retryable queued jobs	Hangfire (SQL Server storage)
ABP Background Workers	Periodic polling / scheduling	AsyncPeriodicBackgroundWorkerBase (timer)
ABP Distributed Event Bus	Cross-module integration events	RabbitMQ
Azure Service Bus consumer	Inbound notification messages from "Spark"	Azure.Messaging.ServiceBus (BackgroundService)
Distributed locking	Coordinating work across instances	Redis (Medallion.Threading)

This page focuses on the background jobs (Hangfire) and background workers that schedule and run deferred work. Messaging (RabbitMQ event bus, Azure Service Bus) is summarised here and covered in more depth alongside Caching; distributed locking is covered on the Caching page.

ABP concepts in one paragraph

ABP separates queued work from recurring work. Background jobs are units of work pushed onto a persistent queue and executed later (with automatic retry); you enqueue them with IBackgroundJobManager and implement them as AsyncBackgroundJob<TArgs>. Background workers are long-lived components that wake up on a timer (or run continuously) to do periodic work; you derive from AsyncPeriodicBackgroundWorkerBase and register them with AddBackgroundWorkerAsync<T>(). Cargonerds uses both, and they cooperate: a worker scans for due schedules and enqueues jobs.

Where it is wired up

Almost the entire background-work stack lives in the Hub module, in modules/hub/src/Hub.Application/HubApplicationModule.cs. That module declares the ABP dependencies, configures Hangfire, registers the one ABP-style job, adds the periodic worker, and registers the Service Bus hosted service:

modules/hub/src/Hub.Application/HubApplicationModule.cs

[DependsOn(
    typeof(HubDomainModule),
    typeof(HubApplicationContractsModule),
    typeof(AbpDddApplicationModule),
    typeof(AbpAutoMapperModule),
    typeof(AbpBackgroundWorkersModule),
    typeof(AbpBackgroundJobsHangfireModule)
)]
public class HubApplicationModule : AbpModule
{
    public override void ConfigureServices(ServiceConfigurationContext context)
    {
        // ...
        Configure<AbpBackgroundJobOptions>(options =>
        {
            options.AddJob<ExcelExportDispatcherJob>();
        });
        ConfigureHangfire(context, hangfireConfiguration);

        context.Services.AddHostedService<ServiceBusNotificationConsumer>();
    }

    public override async void OnApplicationInitialization(ApplicationInitializationContext context)
    {
        await context.AddBackgroundWorkerAsync<ExcelExportSchedulerWorker>();
    }
    // ...
}

The two ABP module dependencies are the key wiring:

• AbpBackgroundJobsHangfireModule replaces ABP's default background-job manager with the Hangfire-backed implementation, so IBackgroundJobManager.EnqueueAsync(...) pushes jobs into Hangfire's SQL Server storage.
• AbpBackgroundWorkersModule enables the periodic-worker host so AddBackgroundWorkerAsync<ExcelExportSchedulerWorker>() starts the timer.

Background jobs (Hangfire)

Hangfire configuration

Hangfire is configured in HubApplicationModule.ConfigureHangfire:

modules/hub/src/Hub.Application/HubApplicationModule.cs

private void ConfigureHangfire(ServiceConfigurationContext context, IConfiguration configuration)
{
    context.Services.AddHangfire(config =>
    {
        config.UseSqlServerStorage(configuration.GetConnectionString(ConnectionStringNames.SparkDb));
        config.UseSerializerSettings(
            new JsonSerializerSettings()
            {
                ReferenceLoopHandling = ReferenceLoopHandling.Ignore,
                PreserveReferencesHandling = PreserveReferencesHandling.Objects,
            }
        );
    });
    Configure<AbpHangfireOptions>(options =>
    {
        options.ServerOptions = new BackgroundJobServerOptions { Queues = HangfireQueues.AllHangfireQueues };
    });
}

Aspect	Value	Note
Storage	SQL Server	config.UseSqlServerStorage(...)
Connection string	ConnectionStringNames.SparkDb = "Default"	The Spark / Cargonerds DB, not the Hub DB
Serializer	Newtonsoft.Json with reference-preservation	ReferenceLoopHandling.Ignore, PreserveReferencesHandling.Objects
Server queues	HangfireQueues.AllHangfireQueues	Restricts which queues this server processes

Hangfire storage lives in the Default (Spark) database, not Hub

ConnectionStringNames.SparkDb resolves to the Default connection string. Hangfire's bookkeeping tables (job state, servers, locks) are therefore created in the Cargonerds/Spark database, even though every background job class lives in the Hub module. Connection-string names are defined in src/Cargonerds.Domain.Shared/Configuration/ConnectionStringNames.cs.

ABP background-job EF Core tables exist but are not the active store

The ABP background-jobs EF Core tables are part of CargonerdsDbContext (via ConfigureBackgroundJobs()), but because AbpBackgroundJobsHangfireModule is depended on, the active job manager is Hangfire. The IBackgroundJobStore that ExcelExportSchedulerWorker queries (see below) is therefore Hangfire's implementation.

Queues

Two named queues are declared in modules/hub/src/Hub.Domain.Shared/Consts/HangfireQueues.cs:

modules/hub/src/Hub.Domain.Shared/Consts/HangfireQueues.cs

public class HangfireQueues
{
    public const string EmailNotificationQueue = "email_notification_queue";
    public const string QuoteGenerationQueue = "quote_generation_queue";

    public static readonly string[] AllHangfireQueues = [EmailNotificationQueue, QuoteGenerationQueue];
}

A job is routed to a queue with Hangfire's [Queue(...)] attribute on the job class. The Hangfire server only consumes the queues passed in AbpHangfireOptions.ServerOptions.Queues (AllHangfireQueues). Jobs that carry no [Queue] attribute run on Hangfire's implicit default queue — note that default is not part of AllHangfireQueues, so an unattributed job is only picked up if it is enqueued onto one of the two named queues.

The jobs in this codebase

All jobs derive from ABP's AsyncBackgroundJob<TArgs> (their argument type is the queue payload). The concrete jobs are:

Job class	File	Queue	Purpose
ExcelExportDispatcherJob	modules/hub/src/Hub.Application/Services/Excel/ExcelExportDispatcherJob.cs	(none → routed by enqueuer / default)	Dispatches an export to the matching IExcelExportJobHandler by ExportType
EmailNotificationJobBase<TDto>	modules/hub/src/Hub.Application/Services/Notifications/NotificationEmailJob/EmailNotificationJobBase.cs	email_notification_queue	Abstract base: renders an ABP text template and queues the email
ShipmentEmailJob	.../NotificationEmailJob/ShipmentEmailJob.cs	email_notification_queue (inherited)	Shipment change notification email
ShipmentCommentEmailJob	.../NotificationEmailJob/ShipmentCommentEmailJob.cs	email_notification_queue (inherited)	Shipment comment notification email
ShipmentBookingEmailJob	.../NotificationEmailJob/ShipmentBookingEmailJob.cs	email_notification_queue (inherited)	Shipment booking notification email
QuoteRequestEmailJob	.../NotificationEmailJob/QuoteRequestEmailJob.cs	email_notification_queue (inherited)	Quote-request notification email
QuotationEmailJob	.../NotificationEmailJob/QuotationEmailJob.cs	email_notification_queue (inherited)	Quotation notification email
QuoteGenerationJob	modules/pricing/src/Pricing.Application/Quotation/QuoteGenerationJob.cs	quote_generation_queue	Generates a quote, invalidates caches, chains a notification

Only ExcelExportDispatcherJob is registered with AddJob<>()

ABP requires the job type registered via Configure<AbpBackgroundJobOptions>(o => o.AddJob<...>()) only for the dispatcher job. The email and quote jobs are discovered through ABP's conventional registration (they implement ITransientDependency and inherit AsyncBackgroundJob<TArgs>), so an explicit AddJob<> call is not present for them.

The email job base

The email base is annotated with the queue once and shared by every concrete email job:

modules/hub/src/Hub.Application/Services/Notifications/NotificationEmailJob/EmailNotificationJobBase.cs

[Queue(HangfireQueues.EmailNotificationQueue)]
public abstract class EmailNotificationJobBase<TNotificationDto>(
    IOptions<WhiteLabelingOptions> whiteLabelOptions,
    ITemplateRenderer renderer,
    IEmailSender emailSender,
    ILogger<EmailNotificationJobBase<TNotificationDto>> logger
) : AsyncBackgroundJob<TNotificationDto>, ITransientDependency
    where TNotificationDto : EmailNotificationDto
{
    protected async Task SendEmail(List<string> recipients, HubEmailTemplate template, object model, string subject)
    {
        // ...
        var body = await renderer.RenderAsync(template.Name, model);
        await Task.WhenAll(recipients.Select(r => emailSender.QueueAsync(r, subject, body, isBodyHtml: true)));
    }
}

It uses ABP TextTemplating (ITemplateRenderer) to render the body and ABP Emailing (IEmailSender.QueueAsync) to hand the message off — so the email itself is queued again by ABP's own outbound email mechanism after the job renders it.

The dispatcher job

ExcelExportDispatcherJob is a thin router: it selects an IExcelExportJobHandler by ExportType and lets any exception bubble up so Hangfire/ABP retries it. It runs inside a unit of work:

modules/hub/src/Hub.Application/Services/Excel/ExcelExportDispatcherJob.cs

[ExposeServices(typeof(ExcelExportDispatcherJob), IncludeSelf = true)]
public class ExcelExportDispatcherJob(
    IEnumerable<IExcelExportJobHandler> handlers,
    ILogger<ExcelExportDispatcherJob> logger
) : AsyncBackgroundJob<ExcelExportJobArgs>, ITransientDependency
{
    [UnitOfWork]
    public override async Task ExecuteAsync(ExcelExportJobArgs args)
    {
        IExcelExportJobHandler handler =
            handlers.FirstOrDefault(h => h.ExportType == args.ExportType)
            ?? throw new InvalidOperationException($"No export handler registered for '{args.ExportType}'");

        // Let exceptions bubble up to ABP's retry mechanism
        await handler.ExecuteAsync(args);
    }
}

The quote-generation job

QuoteGenerationJob shows two background-work patterns at once. It runs under a synthetic principal (so domain code sees the triggering user via ICurrentPrincipalAccessor.Change(...)), runs the generation in a fresh unit of work, and then chains a follow-up using the raw Hangfire API rather than ABP's manager:

modules/pricing/src/Pricing.Application/Quotation/QuoteGenerationJob.cs

[Queue(HangfireQueues.QuoteGenerationQueue)]
public class QuoteGenerationJob(/* ... */) : AsyncBackgroundJob<QuoteGenerationJobArgs>, ITransientDependency
{
    public override async Task ExecuteAsync(QuoteGenerationJobArgs args)
    {
        using var _ = args.TriggeredByUserId.HasValue
            ? currentPrincipalAccessor.Change(/* ClaimsPrincipal with AbpClaimTypes.UserId */)
            : null;

        try
        {
            using (var uow = unitOfWorkManager.Begin(requiresNew: true))
            {
                quotation = await legacyService.CreateQuoteAsync(args.QuotationId);
                await uow.CompleteAsync();
            }
        }
        finally
        {
            await offerAppService.InvalidateOfferCacheAsync();
            await quoteAppService.InvalidateQuoteCacheAsync();
        }

        if (args.SendNotification && quotation != null && args.TriggeredByUserId != null)
        {
            // NOTE: raw Hangfire enqueue, not IBackgroundJobManager
            BackgroundJob.Enqueue<NotificationSenderService>(s =>
                s.SendQuotationCreatedNotificationAsync(quotation, args.TriggeredByUserId.Value)
            );
        }
    }
}

Mixed enqueue APIs: ABP IBackgroundJobManager vs raw Hangfire.BackgroundJob

Most code enqueues through ABP's IBackgroundJobManager.EnqueueAsync(...), which wraps a strongly-typed TArgs payload and applies ABP's job conventions. QuoteGenerationJob instead calls Hangfire.BackgroundJob.Enqueue<NotificationSenderService>(...) directly to enqueue a method call on a service. Both end up in the same Hangfire storage, but the raw call bypasses ABP's job abstraction (and ABP's retry/UoW conventions for that step).

Where jobs are enqueued

Call site	File (line)	What it enqueues
QuotationAppService	modules/pricing/src/Pricing.Application/Quotation/QuotationAppService.cs:196	QuoteGenerationJobArgs (after setting status to Calculating)
ExcelExportAppService	modules/hub/src/Hub.Application/Services/Excel/ExcelExportAppService.cs:255	ExcelExportJobArgs (one-off export; returns the Hangfire JobId)
ExcelExportSchedulerWorker	modules/hub/src/Hub.Application/Services/Excel/ExcelExportSchedulerWorker.cs:158	ExcelExportJobArgs (recurring exports that are due)
NotificationEmailService	modules/hub/src/Hub.Application/Services/Notifications/NotificationEmailService.cs:71	An EmailNotificationDto<...> payload, with a delay
QuoteGenerationJob	modules/pricing/src/Pricing.Application/Quotation/QuoteGenerationJob.cs:63	A NotificationSenderService method call (raw Hangfire)

NotificationEmailService demonstrates ABP's delayed enqueue — the email is scheduled to run after notificationMessage.EmailDelay:

modules/hub/src/Hub.Application/Services/Notifications/NotificationEmailService.cs

await backgroundJobManager.EnqueueAsync(jobArgs, delay: notificationMessage.EmailDelay);

Retry behaviour

ExcelExportDispatcherJob and ExcelExportSchedulerWorker document the retry policy in their XML comments: ABP's default policy retries on failure starting at 1 minute, then doubling (2 min, 4 min, …) up to a 2-day timeout. Jobs simply throw to trigger a retry; there is no custom retry configuration in the module. The scheduler worker then layers its own "give up" logic on top (see below).

Hosts that run vs disable job execution

A Hangfire server is only started where job execution is enabled. The default is enabled; three hosts explicitly disable it via AbpBackgroundJobOptions.IsJobExecutionEnabled = false:

Host / context	File (line)	Executes jobs?
Cargonerds.HttpApi.Host	(no override)	Yes — the API host is the worker
CargonerdsAuthServerModule	src/Cargonerds.AuthServer/CargonerdsAuthServerModule.cs:237	No
CargonerdsWebPublicModule	src/Cargonerds.Web.Public/CargonerdsWebPublicModule.cs:150	No
CargonerdsTestBase	test/Cargonerds.TestBase/CargonerdsTestBase.cs:121	No (also uses Hangfire in-memory storage)

src/Cargonerds.AuthServer/CargonerdsAuthServerModule.cs

Configure<AbpBackgroundJobOptions>(options =>
{
    options.IsJobExecutionEnabled = false;
});

The API host is the single job processor

Because only the API host leaves IsJobExecutionEnabled at its default, that host runs the Hangfire server and processes the email_notification_queue and quote_generation_queue. The auth server and public web host can still enqueue jobs (they share the same SQL storage) but do not consume them. The integration-test base disables execution and swaps storage for UseInMemoryStorage(). See Testing.

Background workers (periodic)

Recurring scheduling is handled by ABP background workers derived from AsyncPeriodicBackgroundWorkerBase. The single worker today is Hub.Services.Excel.ExcelExportSchedulerWorker, registered in HubApplicationModule.OnApplicationInitialization via context.AddBackgroundWorkerAsync<ExcelExportSchedulerWorker>().

modules/hub/src/Hub.Application/Services/Excel/ExcelExportSchedulerWorker.cs

public class ExcelExportSchedulerWorker : AsyncPeriodicBackgroundWorkerBase, ITransientDependency
{
    private const int MaxTryCount = 5;
    private const int MaxJobsToFetch = 1000;

    public ExcelExportSchedulerWorker(
        AbpAsyncTimer timer,
        IServiceScopeFactory serviceScopeFactory,
        IExcelScheduleCalculator scheduleCalculator,
        ILogger<ExcelExportSchedulerWorker> logger
    ) : base(timer, serviceScopeFactory)
    {
        // Check scheduled work every 10 minutes
        timer.Period = (int)TimeSpan.FromMinutes(10).TotalMilliseconds;
    }

    protected override async Task DoWorkAsync(PeriodicBackgroundWorkerContext workerContext) { /* ... */ }
}

What DoWorkAsync does on each 10-minute tick (inside a non-transactional unit of work, uowManager.Begin(isTransactional: false)):

1. Loads SparkExcelExportMetadata rows whose schedule is due — the filter is a raw substring match FilterJson.Contains("\"IsRecurring\":true") combined with NextRunUtc <= now.
2. Fetches up to MaxJobsToFetch = 1000 waiting jobs from IBackgroundJobStore.GetWaitingJobsAsync and builds a lookup from each job's deserialized ExcelExportJobArgs.ExcelExportMetadataId to the job, so it can correlate metadata with in-flight jobs.
3. For each due metadata row:
   • If a waiting job for that metadata has TryCount >= MaxTryCount (5), the schedule is treated as failed and invalidated.
   • If a waiting job already exists, it is skipped (no duplicate enqueue).
   • Otherwise it enqueues a new ExcelExportJobArgs (defaulting CurrencyCode to "EUR" and ExportType to Shipment), then computes the next NextRunUtc via IExcelScheduleCalculator.
   • If the next occurrence can't be computed (unsupported frequency), the schedule is invalidated.

How a schedule is invalidated

InvalidateRecurringScheduleAsync rewrites the metadata's FilterJson with Schedule.IsRecurring = false and clears NextRunUtc, so the row stops matching the due-filter on subsequent ticks. This is the worker's own "stop retrying forever" safety net, independent of ABP's per-job retry policy.

flowchart LR
    W[ExcelExportSchedulerWorker<br/>AsyncPeriodicBackgroundWorker<br/>timer.Period = 10 min] -->|reads due schedules| M[(SparkExcelExportMetadata)]
    W -->|GetWaitingJobsAsync| Q[(Hangfire<br/>SQL Server / Default DB)]
    W -->|EnqueueAsync ExcelExportJobArgs| Q
    Q -->|server consumes| J[ExcelExportDispatcherJob]
    J -->|by ExportType| H[IExcelExportJobHandler]
    J -. on failure: ABP retry<br/>1,2,4 min … 2-day timeout .-> Q
    W -. TryCount >= 5 → IsRecurring=false .-> M

Job vs worker — when to use which

Reach for a background job when you have a discrete unit of work to run once (possibly later, possibly retried) — an export, an email, a quote calculation. Reach for a background worker when you need something to happen on a schedule or to poll — here, deciding which recurring exports are due. The worker does not do the heavy lifting itself; it enqueues jobs.

Messaging (summary)

Distributed event bus (RabbitMQ)

Cross-module integration events use ABP's distributed event bus over RabbitMQ. The API host depends on AbpEventBusRabbitMqModule and configures the connection in CargonerdsHttpApiHostModule.ConfigureRabbitMQ from RabbitMQ:Connections:Default:HostName (set to {messaging} under Aspire, parsed as a URI only when RabbitMQ:Connections:Default:Override=true).

Publishers inject IDistributedEventBus (for example CommentPublicAppService publishes CommentExtraPropertiesUpdatedEto); handlers implement IDistributedEventHandler<T> (for example Pricing.Domain.EventHandler.CommentStatusChangedEventHandler recomputes a Quotation.Status). There is also an in-process local event bus handler, UserRegisteredEventHandler : ILocalEventHandler<EntityCreatedEventData<IdentityUser>>.

Azure Service Bus consumer

Hub.Services.Notifications.ServiceBusNotificationConsumer is a raw Azure.Messaging.ServiceBus BackgroundService (registered with AddHostedService in HubApplicationModule) — it is not the ABP event bus and not RabbitMQ. It subscribes to the fixed abp-notifications topic using the hubServiceBus connection string, forwards inbound messages to the notification sender, and silently no-ops (logs a warning) when hubServiceBus is empty (the local default) or when no environment name is resolvable. Both the RabbitMQ event bus and the Service Bus consumer are described in more detail on Caching.

Gotchas

Things that surprise newcomers

• Hangfire storage is the Default (Spark) SQL database, selected via ConnectionStringNames.SparkDb. It is not the Hub database, even though all job classes live in the Hub module.
• Only the API host executes jobs. AuthServer, Web.Public and the test base set IsJobExecutionEnabled = false. Other hosts can enqueue but never run jobs.
• The server consumes only two named queues (AllHangfireQueues). Hangfire's implicit default queue is not in that list, so a job without a [Queue] attribute is only processed if it is enqueued onto one of the named queues.
• Two different enqueue APIs are in use. Prefer ABP's IBackgroundJobManager.EnqueueAsync; QuoteGenerationJob deliberately uses raw Hangfire.BackgroundJob.Enqueue to enqueue a method call, which bypasses ABP's job/UoW conventions for that step.
• The scheduler worker's due-filter is a JSON substring match — FilterJson.Contains("\"IsRecurring\":true"). It depends on the exact serialized property name/casing of the ExportScheduleDto; changing how that DTO serialises would break the query.
• Two layers of "give up" exist for exports. ABP retries a failing job (1→2→4 min, 2-day timeout); separately, the worker invalidates the recurring schedule once a waiting job reaches TryCount >= 5. They are independent.
• The Service Bus consumer is not ABP messaging. Despite the topic name abp-notifications and DTO name ServiceBusAbpNotificationEto, it is hand-rolled and does nothing locally.

Related pages

• Caching — Redis distributed cache, the Hub second-level EF cache, RabbitMQ event bus, Azure Service Bus consumer, and Redis distributed locking.
• Entity Framework Core — HubDbContext pooling and the second-level cache.
• .NET Aspire Integration — Redis + RabbitMQ Aspire resources and connection-string injection.
• ABP Patterns — AsyncBackgroundJob, the event bus, keyed services and DI conventions.
• Pricing module — QuoteGenerationJob and the cross-module comment event.
• Testing — how the test base disables job execution and swaps Hangfire storage.
• Configuration reference — connection strings (Default, Hub), RabbitMQ:*, hubServiceBus.