[Bug]: "No transaction available to participate" Error After Rollback In Transactional Services

[dsplayerX]

Description

If a transaction were to rollback by an error in a service with a transactional endpoint, the following error message is observed in the participant service approximately ~60 seconds later:

error: No transaction is available to participate
        at ballerinai.transaction.0:createTrxContextFromGlobalID(transaction_block.bal:150)
           ballerinai.transaction.0.$anonType$_1:$post$^$prepare(service_participant_2pc.bal:36)

This error causes the participant service to crash and it does not appear when a transaction commits successfully.

Participant's Full Log

time=2024-02-28T15:17:56.459+05:30 level=INFO module=dumindusameendra/service message="DB initialized"
time=2024-02-28T15:23:53.871+05:30 level=DEBUG module=ballerina/http message="Created HTTP caching client"
time=2024-02-28T15:23:53.878+05:30 level=DEBUG module=ballerinai/transaction message="Registering for transaction: 94cc56bb-4699-4d90-a7fc-194f422643d9:1 with coordinator: http://10.100.8.135:59818/balcoordinator/initiator/1/register"
time=2024-02-28T15:23:54.051+05:30 level=DEBUG module=ballerinai/transaction message="Registered with coordinator for transaction: 94cc56bb-4699-4d90-a7fc-194f422643d9"        
time=2024-02-28T15:23:54.054+05:30 level=DEBUG module=ballerinai/transaction message="participant registered: 94cc56bb-4699-4d90-a7fc-194f422643d9"
time=2024-02-28T15:23:54.098+05:30 level=ERROR module=dumindusameendra/service message="Update failed: No rows updated"
error: No transaction is available to participate
        at ballerinai.transaction.0:createTrxContextFromGlobalID(transaction_block.bal:150)
           ballerinai.transaction.0.$anonType$_1:$post$^$prepare(service_participant_2pc.bal:36)

Coordinator's Full Log

time=2024-02-28T15:23:53.489+05:30 level=DEBUG module=ballerina/http message="Created HTTP caching client"
time=2024-02-28T15:23:53.537+05:30 level=DEBUG module=ballerina/http message="Cached response not found for: \'GET /doTransactionalWork?id=65\'"
time=2024-02-28T15:23:53.540+05:30 level=DEBUG module=ballerina/http message="Sending new request to: /doTransactionalWork?id=65"
time=2024-02-28T15:23:54.033+05:30 level=DEBUG module=ballerinai/transaction message="Registered remote participant: 834b3aa8-8fe7-40df-aecf-2ae5a71433f3:1 for transaction: 94cc56bb-4699-4d90-a7fc-194f422643d9"
time=2024-02-28T15:23:54.128+05:30 level=INFO module=dumindusameendra/bug_no_trans message="500"
time=2024-02-28T15:23:54.131+05:30 level=DEBUG module=dumindusameendra/bug_no_trans message="Rollback Handler: Decision Rollback!"
time=2024-02-28T15:25:54.305+05:30 level=DEBUG module=ballerinai/transaction message="Running 2-phase commit for transaction: 94cc56bb-4699-4d90-a7fc-194f422643d9:1"
time=2024-02-28T15:25:54.329+05:30 level=DEBUG module=ballerina/http message="Created HTTP caching client"
time=2024-02-28T15:25:54.347+05:30 level=DEBUG module=ballerinai/transaction message="Preparing remote participant: 834b3aa8-8fe7-40df-aecf-2ae5a71433f3:1"
time=2024-02-28T15:25:54.379+05:30 level=ERROR module=ballerinai/transaction message="Remote participant: 834b3aa8-8fe7-40df-aecf-2ae5a71433f3:1 failed" error={"causes":[],"message":"Prepare failed. Transaction: 94cc56bb-4699-4d90-a7fc-194f422643d9, Participant: http://10.100.8.135:59819/balcoordinator/participant/2pc/1","detail":{},"stackTrace":[{"callableName":"prepare","moduleName":"ballerinai.transaction.0.Participant2pcClientEP","fileName":"connector_participant_2pc.bal","lineNumber":63},{"callableName":"prepareMe","moduleName":"ballerinai.transaction.0.RemoteParticipant","fileName":"2pc_remote_participant.bal","lineNumber":83},{"callableName":"prepare","moduleName":"ballerinai.transaction.0.RemoteParticipant","fileName":"2pc_remote_participant.bal","lineNumber":41}]}
time=2024-02-28T15:25:54.406+05:30 level=DEBUG module=ballerinai/transaction message="Auto-committed initiated transaction: 94cc56bb-4699-4d90-a7fc-194f422643d9. Result: aborted"

Steps to Reproduce

1. Start two Ballerina services:
   • Service A (Coordinator Service) (listener on port 9710) containing the startTransaction function that initiates a transaction.
   • Service B (Participant Service) (listener on port 9712) containing the doTransactionalWork transactional resource function with a database connection.
2. Initiate a transaction from Service A that calls doTransactionWork in Service B.
3. Force a rollback of the transaction in the coordinator under any condition. Specifically, in the code sample provided, attempt to update a row that does not exist, resulting in 0 affectedRows. This failure will send an error back to the coordinator, causing it to initiate a rollback.
4. Observe the logs in both services.
5. After approximately 60 seconds, expect to see the "No transaction available to participate" error in Service B's logs causing Service B to crash.

Minimal Code:

Service A (Coordinator)

import ballerina/http;
import ballerina/log;

service / on new http:Listener(9710) {
    resource function get startTransaction(int id) returns error? {
        http:Client chain1Client = check new ("http://localhost:9712");
        transaction {
            transaction:onCommit(commitHanlder);
            transaction:onRollback(rollbackHandler);
            http:Response response = check chain1Client->get(string`/doTransactionalWork?id=${id}`);
            log:printInfo(response.statusCode.toString());

            if (response.statusCode != 200) { // force rollback transaction
                rollback;
            } else {
                check commit;
            }
        }
    }
}

isolated function commitHanlder('transaction:Info info) {
    log:printDebug("Commit Handler: Decision Commit!");
}

isolated function rollbackHandler(transaction:Info info, error? cause, boolean willRetry = true) {
    log:printDebug("Rollback Handler: Decision Rollback!");
}

Service B (Participant)

import ballerina/http;
import ballerina/log;
import ballerina/sql;
import ballerinax/mysql;
import ballerinax/mysql.driver as _;

service / on new http:Listener(9712) {
    final mysql:Client db1;

    function init() returns error? {
        self.db1 = check new (host = "localhost",
            user = "root", password = "root",
            port = 3306, database = "test",
            options = {
                useXADatasource: true
            }
        );
        log:printInfo("DB initialized");
    }

    transactional resource function get doTransactionalWork(int id) returns http:Response {
        transaction:onCommit(commitHanlder);
        transaction:onRollback(rollbackHandler);
        http:Response res = new;

        sql:ParameterizedQuery updateQuery = `UPDATE test.test SET hello = 'Updated' WHERE id = ${id};`;
        sql:ExecutionResult|sql:Error updateResult = self.db1->execute(updateQuery);

        if updateResult is sql:Error {
            log:printError("Update failed: SQL Error");
            res.statusCode = 500;
            res.setPayload("Update failed: SQL Error");
            return res;
        } else {
            if (updateResult.affectedRowCount == 0) {
                log:printError("Update failed: No rows updated");
                res.statusCode = 500;
                res.setPayload("Update failed: No rows updated");
                return res;
            }
            log:printInfo("Updated successfully");
            res.statusCode = 200;
            res.setPayload("Updated successfully");
            return res;
        }
    }
}

isolated function commitHanlder('transaction:Info info) {
    log:printDebug("Commit Handler: Decision Commit!");
}

isolated function rollbackHandler(transaction:Info info, error? cause, boolean willRetry = true) {
    log:printDebug("Rollback Handler: Decision Rollback!");
}

Affected Version(s)

Tested on 2201.8.4 with this fix. (same without the fix)

OS, DB, other environment details and versions

No response

Related area

-> Compilation

Related issue(s) (optional)

No response

Suggested label(s) (optional)

No response

Suggested assignee(s) (optional)

No response