Close of Business is taking hours. Here is why, and what to do.

In the production guide we said the Close of Business mechanics deserved their own post. This is it.

Close of Business is the job people complain about first when Fineract gets slow. It is the nightly batch every Fineract institution depends on, and at scale it can run for hours, long enough to eat into the morning if it starts late or stalls. We run COB for institutions from a few thousand loans up to portfolios in the millions, so here is the honest diagnosis: why it gets slow, and the fixes that actually matter, in the order we reach for them.

What COB actually does

The Loan COB job walks every active loan in the portfolio once a night. For each loan it recalculates arrears, applies overdue charges, and posts accrual journal entries, and then the run advances the business date for the whole tenant. (The business-level "what and why" is in the docs; this is the operations view.)

Two things follow from that, and they explain almost everything about COB's performance:

1. It is a per-loan workload. The work is roughly proportional to the number of active loans. Double the portfolio, roughly double the work.
2. The whole institution's "next day" waits on it. Until COB finishes and the business date advances, the new day has not really started. That is why a COB that creeps from one hour to three is not a cosmetic problem, it is the thing standing between your staff and Monday morning.

How it runs: a batch manager node slices the portfolio into partitions and hands them out, batch worker nodes process those partitions, and the manager waits for all of them before it advances the date. On a default single-node deployment the same process is both manager and worker, so there is no parallelism at all, one process grinds through the entire book.

Why it is slow

In rough order of how often it is the real cause:

The database, not the workers. Fineract is database-heavy, and COB is the most database-heavy thing it does: it reads and writes across the entire loan book inside one window. More often than not, a slow COB is a database that is too small or starved of connections, not a shortage of worker threads. This is the same lesson as the production guide, size the database first, and watch the connection pool, because COB is exactly when you will exhaust it.

The defaults are tuned for small. Out of the box, COB processes loans in chunks and partitions of 10. On a small portfolio that is fine. On a large one it means an enormous number of tiny partitions, and the per-partition overhead (scheduling, polling, bookkeeping) starts to dominate the actual work. Raising the partition and chunk size is the cheapest win there is for a large portfolio, and it is a config change, not an architecture change.

You are still single-node. If manager and worker are the same process, COB has no way to do work in parallel. One JVM is processing the whole portfolio in series. For tens of thousands of loans that is usually fine; past that, you are leaving the obvious lever, parallelism, untouched.

Not every job is built to scale. Fineract has two kinds of batch job, and the difference matters when you are optimising. Partitioned jobs (COB is one) are built for scale: the manager partitions the work and worker nodes process the partitions, so adding workers adds throughput. Non-partitioned jobs are simpler and run on the manager itself; they do not scale out, and a heavy one can hold up your window no matter how many workers you add. Throwing workers at a non-partitioned job does nothing, so know which of your jobs is which before you spend money on more workers.

What actually moves the needle

A ladder, cheapest first. Most institutions never need to climb past the second rung.

1. Right-size the database. Cheapest, highest-leverage, and the most common real fix. Give COB enough database to read and write the whole book without contention, and put a connection pooler in front of it.
2. Tune chunk and partition size. Raise the partition and chunk size for a large portfolio so you are not drowning in tiny partitions. Config only, no new infrastructure. The COB guide lists the exact variables and sensible starting points.
3. Go multi-node. Separate the batch manager from the workers, put a message broker between them (ActiveMQ/JMS or Kafka), and scale the workers horizontally. Each worker chews through partitions in parallel, which is where a multi-hour window collapses. In single-node mode Fineract coordinates partitions with in-process events; the moment you split across machines those events cannot cross the JVM boundary, so a broker is not optional. It is what makes multi-node COB work at all.
4. For very heavy portfolios (think ten million loan accounts and up), run many workers behind the broker and give the manager nothing to do but schedule. This is the configuration that turns an impossible overnight window into a manageable one.

One hard constraint carries over from the production guide: there is only ever one batch manager. Two managers against the same database compete over scheduling, and you get duplicated or colliding jobs. So when you scale COB, you scale workers, never the manager.

Where you land on the ladder depends mostly on portfolio size:

• Up to a few tens of thousands of loans: single node is fine. Tune the database before you touch anything else.
• Hundreds of thousands: separate the manager and workers, add a broker, run a small number of workers.
• Millions and up: many workers behind the broker, a manager that does nothing but schedule, and a database sized for the read-and-write storm.

To put rough numbers on it: a portfolio of around a million active loans took the better part of a full day to finish COB on a single node, far past any overnight window you would want to run in. Split across multiple workers behind a broker, the same run came down to about an hour. That is what parallelism buys you once the database is no longer the bottleneck, a job that spilled into the next afternoon turned into one that is done before anyone arrives.

The part nobody plans for: when COB fails

COB does not just need to be fast, it needs to finish. When it fails mid-run, whether from a data integrity error, the broker dropping, or the database running out of connections, recovery is more involved than restarting a service:

• Loans that were mid-processing stay locked. Fineract clears stale locks automatically on the next COB startup, but you want to know that is the mechanism rather than panicking about stuck loans at 6am.
• The business date has not advanced, so until you rerun COB the institution is still operating on yesterday.
• If you missed several days, say an outage that ran long, you cannot jump the business date to today. COB has to run once per missed day, advancing the date one day at a time, until it catches up.
• The events COB generated (accruals, overdue charges) queue up in the external events backlog and drain once the system is healthy again, so downstream consumers are not lost. But you need to watch that backlog while you recover.

None of this is hard once you have done it a few times. The first time, at dawn with the business day about to open, it is exactly what you do not want to be learning live.

So what should you do?

If your portfolio is small, COB is a non-event: leave it single-node, size the database sensibly, and move on. If you are growing into hundreds of thousands or millions of loans, the path is well-trodden, tune the defaults, split the manager from the workers, put a broker between them, and scale the workers. None of it is mysterious; all of it is operational work you own, plus the on-call reality of the nights it does not go to plan.

That is the part we take off your plate. Finecko runs COB for you on every plan, scheduling, worker scaling, stuck-job recovery, and alerting, sized to your portfolio, so the nightly batch is our problem and not your 6am. If you would rather run it yourself, start with the database and the partition sizes, and keep the batch-manager rule from the production guide in mind. Either way, COB does not have to be the thing that wakes you up.