Flag Whatsapp Forwarded messages

[gfd2020]

Suggestion
Include in whatsapp chat messages the indication if the message has been forwarded.

How
The flag of this behavior are indicated as below (unless I'm mistaken. In my tests, it seems correct.)

IPHONE
ZWAMESSAGE.ZFLAGS bit 7
Example and test: select (1 << 7 & ZFLAGS) as forwarded from ZWAMESSAGE

ANDROID NEW
message.origination_flags bit 1
Example and test: select (origination_flags & 1) as forwarded FROM message

ANDROID OLD
messages.forwarded bit 1
Example and test: select (forwarded & 1) as forwarded FROM messages

On the ReportGenerator.java, add something like this ...

if (message.getForwarded() > 0) {
out.println("<img src="" + RSRC_PATH + "img/forward.png" valign="top"/><i style="color:gray">Forwarded
");
}

[lfcnassif]

Thanks @gfd2020, this can be very useful.

[gfd2020]

Hi @tc-wleite. Did it work as expected in your tests?

[wladimirleite]

Hi @tc-wleite. Did it work as expected in your tests?

Hi @gfd2020!
Yes, it worked fine.
Thanks for sharing this information.
Android "old" is tricky because there are several scenarios (different columns/tables used), but that doesn't seem to affect the "forwarded" flag (at least I couldn't find any sample in which the column you pointed out was absent or use a different name).

[filipesimoes]

Hello.
This is important. The absense of the Forwared flag may cause a lot of confusion during the analysis of Whatsapp.

[lfcnassif]

I started a regression test yesterday on hundreds of whatsapp databases using PR #1651 and if everything looks good, I'll merge it. I noticed rendering lags when viewing large WhatsApp chats with that PR, I still need to check on master to confirm if it is a performance regression or not. Have you noticed something similar @tc-wleite?

[wladimirleite]

I started a regression test yesterday on hundreds of whatsapp databases using PR #1651 and if everything looks good, I'll merge it. I noticed rendering lags when viewing large WhatsApp chats with that PR, I still need to check on master to confirm if it is a performance regression or not. Have you noticed something similar @tc-wleite?

Large chats (the ones that are broken into parts) do cause rendering lags. But it doesn't seem to be something critical, at least in the cases that I tested here. It seems to be the same kind of lag present when opening larger HTMLs in the internal viewer.
I didn't observe any noticeable difference in that sense, comparing to master.

Not sure if it is the same type of delay you are observing. You can send me a sample database if you see larger lags than we had before, so I can investigate in more detail.

[gfd2020]

Is the htmlviewer using javafx?
May be this could help ...
https://docs.oracle.com/javafx/2/ui_controls/pagination.htm
I make a small demo and works. We just need to split the chat in pages ...

[lfcnassif]

We just need to split the chat in pages ...

This is already done explicitly for large chats, users can see many chat items referring to the same conversation. Maybe the max chat chunk size could be decreased, currently it is ~5MiB, but splitting too much can be annoying to users. Other approach could be splitting based on the number of messages, not on chat html size, actually it was done in the past and I don't know what is heavier to be rendered (5 MiB of text messages or 5MiB of multimedia messages).

[wladimirleite]

Is the htmlviewer using javafx? May be this could help ... https://docs.oracle.com/javafx/2/ui_controls/pagination.htm I make a small demo and works. We just need to split the chat in pages ...

Large chats are already split into parts (different items). It doesn't use a pagination like this one, but I am not sure if this is really necessary. Simply reducing a bit the chat split threshold would help, but it is probably better to understand first if it is a regression or something else happening in particular chats, as I didn't observe any critical delay.

[lfcnassif]

Hi @tc-wleite, I got more parsing exceptions with #1651 than master. After I confirm them, I'll send the triggering samples privately to you.

[lfcnassif]

PS: Thanks for helping with this!

[lfcnassif]

Hi @tc-wleite, I got more parsing exceptions with #1651 than master. After I confirm them, I'll send the triggering samples privately to you.

I confirmed 4 NPEs, 3 of them with this stacktrace for sure:

org.apache.tika.exception.TikaException: WAExtractorException Exception
	at iped.parsers.whatsapp.WhatsAppParser.parseWhatsappMessages(WhatsAppParser.java:377)
	at iped.parsers.whatsapp.WhatsAppParser.parse(WhatsAppParser.java:252)
	at org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:298)
	at iped.parsers.standard.StandardParser.parse(StandardParser.java:245)
	at iped.engine.io.ParsingReader$BackgroundParsing.run(ParsingReader.java:247)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
	at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.lang.NullPointerException
	at iped.parsers.whatsapp.ExtractorIOS.decodeReceiptInfo(ExtractorIOS.java:484)
	at iped.parsers.whatsapp.ExtractorIOS.createMessageFromDB(ExtractorIOS.java:377)
	at iped.parsers.whatsapp.ExtractorIOS.extractMessages(ExtractorIOS.java:269)
	at iped.parsers.whatsapp.ExtractorIOS.extractChatList(ExtractorIOS.java:198)
	at iped.parsers.whatsapp.Extractor.getChatList(Extractor.java:34)
	at iped.parsers.whatsapp.WhatsAppParser.parseWhatsappMessages(WhatsAppParser.java:369)

I just sent the triggering samples to you by Teams.

[wladimirleite]

Thanks!
I will look into this tomorrow.

[patrickdalla]

We could improve some of this 'lag' implementing some lazy loading logic via 'javascript'. This could be done transparently via in HTMLViewer, so if it is a good idea, we can create new specific issue.

[wladimirleite]

We could improve some of this 'lag' implementing some lazy loading logic via 'javascript'. This could be done transparently via in HTMLViewer, so if it is a good idea, we can create new specific issue.

I tested this again this morning, and didn't find any critical lag (obviously the "critical" threshold is kind of personal), neither observed any difference comparing to previous (master and 4.1.x) versions.
So unless @lfcnassif has any specific case that triggers a regression or a problematic delay, I wouldn't change the current behavior.

[lfcnassif]

I tested this again this morning, and didn't find any critical lag (obviously the "critical" threshold is kind of personal), neither observed any difference comparing to previous (master and 4.1.x) versions.
So unless @lfcnassif has any specific case that triggers a regression or a problematic delay, I wouldn't change the current behavior.

Hi @tc-wleite, I took a closer look, there is no regression, sorry. The lag issue is with a specific database with a specific chat with a corrupted text message which is a few MBs of size, that message causes the slow rendering both on #1651 and on master, so that's an unrelated issue.

By the way, I'm getting a different number of messages within different runs caused by another unrelated issue, that is making comparisons harder. I'll describe it in another ticket.

[wladimirleite]

Hi @tc-wleite, I got more parsing exceptions with #1651 than master. After I confirm them, I'll send the triggering samples privately to you.

I confirmed 4 NPEs, 3 of them with this stacktrace for sure:

I pushed a fix this morning that seems to fix this issue.

[lfcnassif]

I pushed a fix this morning that seems to fix this issue.

I saw it, thank you!

[gfd2020]

Group chats tend to be large and have a lot of incoming messages. A real case with a chat size of 5 MB, it was quite slow. The slowdown is caused by the chat bubble arrow. If you remove the 'position: relative' from the '#conversation .incoming' class and the 'position: absolute' from the '#conversation .incoming:before' class, performance is fine. However, the arrow will be drawn wrong. It would be necessary to find another way of drawing.