Database Queue unreliable when Jobs use binary data

[leewillis77]

I've not yet managed to work out the precise data that causes the failure, however if you have a Job that takes a string parameter, and that string contains binary data (E.g. raw image data), then the queue item logged to the database ends up with a payload of "0" instead of the real payload.

I think the issue probably relates to the use of json_encode() in Queue.php, createPayload() - https://github.com/laravel/framework/blob/5.3/src/Illuminate/Queue/Queue.php#L78

If so, then this probably affects all queue types, not just database. I'll see if I can identify a minimal test case.

[leewillis77]

Here's how you can reproduce this. Use the following simple test Job

<?php

namespace App\Jobs;

use Illuminate\Bus\Queueable;
use Illuminate\Queue\SerializesModels;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Contracts\Queue\ShouldQueue;

class TestJob implements ShouldQueue
{
    use InteractsWithQueue, Queueable, SerializesModels;

    protected $binaryData;

    /**
     * Create a new job instance.
     *
     * @return void
     */
    public function __construct($binaryData)
    {
        $this->binaryData = $binaryData;
    }

    /**
     * Execute the job.
     *
     * @return void
     */
    public function handle()
    {
        return;
    }
}

This creates a valid job record in the queue when called with simple text, e.g.

dispatch(new App\Jobs\TestJob('string data'));

results in

> select * from jobs where id = 1\G
*************************** 1. row ***************************
          id: 1
       queue: default
     payload: {"job":"Illuminate\\Queue\\CallQueuedHandler@call","data":{"commandName":"App\\Jobs\\TestJob","command":"O:16:\"App\\Jobs\\TestJob\":5:{s:13:\"\u0000*\u0000binaryData\";s:11:\"string data\";s:6:\"\u0000*\u0000job\";N;s:10:\"connection\";N;s:5:\"queue\";N;s:5:\"delay\";N;}"}}
    attempts: 0
 reserved_at: NULL
available_at: 1472831249
  created_at: 1472831249
1 row in set (0.00 sec)

However, passing in arbitrary binary data, e.g.

$data = file_get_contents('https://github.com/laravel/art/raw/master/laravel-text-logo.png');
dispatch(new App\Jobs\TestJob($data));

results in a broken queue item - notice how payload is the string "0":

> select * from jobs where id = 2\G
*************************** 1. row ***************************
          id: 2
       queue: default
     payload: 0
    attempts: 0
 reserved_at: NULL
available_at: 1472831420
  created_at: 1472831420
1 row in set (0.00 sec)

[themsaid]

As of this PR #15284 failing to create a payload will throw an exception.

I'm not sure about passing binary data like this though.

[leewillis77]

That would certainly help people understand what's going on, although I still think it's a bug that binary data can't be handled. Either a functional bug that gets fixed, or at the very least a documentation issue that it should be documented that queues do not support binary data in the jobs without suitable encoding.

[srmklive]

I agree with @themsaid passing binary data directly to the constructor isn't right. I would rather do it like this:

$data = file_get_contents('https://github.com/laravel/art/raw/master/laravel-text-logo.png');
$job = new App\Jobs\TestJob;
$job->setBinaryData($data);

dispatch($job);

[leewillis77]

I think @themsaid was arguing against a job holding binary data at all.

Whether you pass it via the constructor, or input it via a setter, the end result is the same - when that hits a queue - it will fail (payload of "0" in current release, Exception in future release of Laravel)

[VinceG]

@leewillis77 i thought json_encode does not support binary data. in order to encode binary data you first have to escape it to be able to pass it in.

[danmatthews]

Interestingly, i ran a small test script to test json encoding binary data, and it errors. (PHP 7.0.9) - strange to see that @leewillis77 would end up with a payload of '0'.

class TestClass {
    protected $binaryData;

    public function __construct($data)
    {
        $this->binaryData = $data;
    }
}


$c = new TestClass( file_get_contents('image.png') );

$data = json_encode([
    'class' => get_class($c),
    'data' => serialize($c),
]);

dump(json_last_error() == JSON_ERROR_UTF8); // True

[VinceG]

@danmatthews Yeah. if you base64 encode the data first, it'll work.

$this->binaryData = base64_encode($data);

[danmatthews]

Yep, and i believe that's how Lee was working around it too. I think it potentially warrants a note in the docs, so i've added one for consideration.

laravel/docs#2629

[themsaid]

Thank you @danmatthews for the docs PR, and indeed encoding to base64 would solve the issue.

I'm not sure if this can/should be fixed in the core, it's tricky to make such change, so at least for 5.3 it'll stay the same IMO.

Closing this issue then since most likely it'll be documented as a limitation for the queue system, please ping me if you still think this MUST be fixed in the core.

[VinceG]

@danmatthews i like the addition to the docs, I think this should be considered a JSON doc rather than Laravel doc. Since this issue applies to all Framework with queues and similar functionality. Instead of ALL of them documenting that in their framework doc file this is stated in the JSON specification since that applies to everything. I found a discussion they had about it
http://discuss.jsonapi.org/t/handling-binary-data/315/4

[danmatthews]

I think the problem lies in the fact that you're not aware that the information you're passing will be JSON-encoded - therefore you can't know ahead of time without the addition to the docs or reading the underlying code.

@taylorotwell has said that we should 'come up with a solution first so such a caveat isn't necessary' which i agree would be fantastic - if Laravel deals with this eventuality for you, then you don't need to know anything.

There's a few options i looked into - looping over class properties to detect string values and binary data (if that can be done reliably?). Or potentially encoding every string value of a class to something that is JSON-able. But there's the potential for nested classes and things too - and all of this would have to be reversed upon unserializing the class.

OR - could we potentially base64_encode the serialized class (since it's just text), then base64_decode it when it's parsed on wake?

[themsaid]

It'll be stated as a limitation in the docs. Closing this issue then.

[DerMika]

I ran into this issue trying to queue an e-mail for sending containing a binary PDF as raw attachment. If anyone was looking for usecases when queueing binary data might be useful.

Got around it by base64 encoding the raw attachments, and decoding them in the Mailable::build() method (which is called when the queued mail is being processed, before actually sending the message).

[zachbornheimer]

I'm new to Laravel, so please take that into consideration...this is also quite an old thread.

I found that by converting the binary data in the __constructor using base64_encode then decoding it during the attachment phase, I was able to attach PDF raw binary data without issue.

Here's some example code that might help illustrate the solution I used:

protected $pdf_data;
public function __construct($itinerary) {
$pdf = PDF::loadView('view.name', compact('args_to_view'));
$this->pdf_data = base64_encode($pdf->output());
}
public function handle() {
Mail::to(...)->send(new \App\Mail\handler(base64_decode($this->pdf_data), ...));
...
}

Thought I'd leave that here for fellow searchers! :-)