Skip empty messages. #1261

TheHolyRoger · 2020-10-17T14:34:40Z

Currently empty messages are still attempted to send via every enabled bridge. This adds a lot of unnecessary overhead when using tengo scripts to skip certain messages by blanking msgText and msgUsername.

This little change cuts out empty messages earlier.

It intentionally doesn't check msgUsername, because I've seen a few tengo scripts blanking msgUsername in order to send chat commands over the bridge (to achieve getting the first character of the message be an ! or $ for example).

#942

42wim · 2020-10-20T18:55:27Z

Sorry, we can't block empty messages there, we have status messages which can be empty.
Empty messages should be blocked by the bridge sending the messages.
What bridge is sending those messages?

TheHolyRoger · 2020-10-20T20:49:56Z

@42wim my bridges are performing much better with this change. No disconnects since.

What about if both username and msgtext are empty instead?

The problem is the added CPU load of sending empty messages to 8+ bridged networks only to be refused in the end by each one because it's empty.

42wim · 2020-10-20T21:07:16Z

yes, i understand the issue that's why I'm asking which bridges do still send empty messages.

Also curious about the extra cpu load how many messages per second are you sending?

TheHolyRoger · 2020-10-20T23:36:33Z

It varies depending on time of day, sometimes it can be a lot less than 1 per second, sometimes 5-10+ across all the different gateways.

Blank messages aren't actually sent by any of the bridges, but they seem to make it all the way through the chain which is why I'd rather cut them out earlier...

If i recall correctly, I think with debug mode on I've seen messages from the keybase client refusing to send blank messages.

My bridge sits in some quite busy channels, so I've got tengo scripts set up to block a lot of matching unnecessary noise.

If you really think this method will cause issues then perhaps we need to add an editable boolean to the procoutmessage tengo logic that flags messages to be discarded @42wim ?

codeclimate · 2020-10-20T23:43:52Z

Code Climate has analyzed commit 671ed3c and detected 0 issues on this pull request.

View more on Code Climate.

42wim · 2020-10-21T14:15:25Z

Blank messages aren't actually sent by any of the bridges, but they seem to make it all the way through the chain which is why I'd rather cut them out earlier...

I want to fix the cause instead of the symptoms, so I need to know who is sending those empty messages. Could you find that in your debug log?

TheHolyRoger · 2020-10-21T14:31:35Z

Blank messages aren't actually sent by any of the bridges, but they seem to make it all the way through the chain which is why I'd rather cut them out earlier...

I want to fix the cause instead of the symptoms, so I need to know who is sending those empty messages. Could you find that in your debug log?

I'll switch it back on for a bit and report back.

I still don't see what the benefit is of not skipping unwanted messages before they're passed on to each bridge though... It is such a massive performance increase on my end that I'll end up continuing to build myself and patch this in... 🤷

TheHolyRoger · 2020-10-21T14:48:03Z

Just switched debug on and realised that I obviously don't see these errors anymore with my custom build... I'll play with it at a quieter time of day

42wim · 2020-10-21T14:53:30Z

maybe tengo is generating those empty messages, if it's not coming from the bridges?

TheHolyRoger · 2020-10-21T15:27:03Z

@42wim yes, I'm using tengo to blank messages, as that's the only method of actually skipping messages using tengo rules :)

Here's the log without this patch, that's an awful lot of work for an empty message, using this patch the only output I get in the log is the Receiving PRIVMSG line

time="2020-10-21T15:18:57Z" level=debug msg="== Receiving PRIVMSG: TheHoliestRoger &skipthis  girc.Event{Source:(*girc.Source)(0xc000a43200), Tags:girc.Tags(nil), Timestamp:time.Time{wall:0xbfdc32384b68933d, ext:861396864088, loc:(*time.Location)(0x31a9360)}, Command:"PRIVMSG", Params:[]string{"#TheHolyRoger", "&skipthis "}, Sensitive:false, Echo:false}" func=handlePrivMsg file="bridge/irc/handlers.go:174" prefix=irc
time="2020-10-21T15:18:57Z" level=debug msg="detected ISO-8859-1 confidence 30" func=handlePrivMsg file="bridge/irc/handlers.go:194" prefix=irc
time="2020-10-21T15:18:57Z" level=debug msg="<= Sending message from #TheHolyRoger on irc.roger to gateway" func=handlePrivMsg file="bridge/irc/handlers.go:214" prefix=irc
time="2020-10-21T15:18:57Z" level=debug msg="=> Sending config.Message{Text:"&skipthis ", Channel:"#theholyroger", Username:"TheHoliestRoger", UserID:"~TheHolies@unaffiliated/theholiestroger", Avatar:"", Account:"irc.roger", Event:"", Protocol:"irc", Gateway:"roger", ParentID:"", Timestamp:time.Time{wall:0xbfdc32384b7ea793, ext:861398311063, loc:(*time.Location)(0x31a9360)}, ID:"", Extra:map[string][]interface {}(nil)} from irc.roger (#theholyroger) to telegram.roger (-1001300391341)" func=SendMessage file="gateway/gateway.go:421" prefix=gateway
time="2020-10-21T15:18:57Z" level=debug msg="=> Receiving config.Message{Text:"", Channel:"-1001300391341", Username:"", UserID:"~TheHolies@unaffiliated/theholiestroger", Avatar:"", Account:"irc.roger", Event:"", Protocol:"irc", Gateway:"roger", ParentID:"", Timestamp:time.Time{wall:0xbfdc32384b7ea793, ext:861398311063, loc:(*time.Location)(0x31a9360)}, ID:"", Extra:map[string][]interface {}(nil)}" func=Send file="bridge/telegram/telegram.go:73" prefix=telegram
time="2020-10-21T15:18:57Z" level=debug msg="=> Sending config.Message{Text:"&skipthis ", Channel:"#theholyroger", Username:"TheHoliestRoger", UserID:"~TheHolies@unaffiliated/theholiestroger", Avatar:"", Account:"irc.roger", Event:"", Protocol:"irc", Gateway:"roger", ParentID:"", Timestamp:time.Time{wall:0xbfdc32384b7ea793, ext:861398311063, loc:(*time.Location)(0x31a9360)}, ID:"", Extra:map[string][]interface {}(nil)} from irc.roger (#theholyroger) to matrix.roger (#TheHolyRoger:matrix.theholyroger.com)" func=SendMessage file="gateway/gateway.go:421" prefix=gateway
time="2020-10-21T15:18:57Z" level=debug msg="=> Receiving config.Message{Text:"", Channel:"#TheHolyRoger:matrix.theholyroger.com", Username:"", UserID:"~TheHolies@unaffiliated/theholiestroger", Avatar:"", Account:"irc.roger", Event:"", Protocol:"irc", Gateway:"roger", ParentID:"", Timestamp:time.Time{wall:0xbfdc32384b7ea793, ext:861398311063, loc:(*time.Location)(0x31a9360)}, ID:"", Extra:map[string][]interface {}(nil)}" func=Send file="bridge/matrix/matrix.go:94" prefix=matrix
time="2020-10-21T15:18:57Z" level=debug msg="Channel #TheHolyRoger:matrix.theholyroger.com maps to channel id !wAjlYiluyhksKYlJMk:matrix.theholyroger.com" func=Send file="bridge/matrix/matrix.go:97" prefix=matrix
time="2020-10-21T15:18:57Z" level=debug msg="mID matrix.roger: $lZlAZoJgY6ZwELfJpRUUTZ0uoDZpzHmIaggEsCvUWTk" func=SendMessage file="gateway/gateway.go:464" prefix=gateway
time="2020-10-21T15:18:57Z" level=debug msg="=> Sending config.Message{Text:"&skipthis ", Channel:"#theholyroger", Username:"TheHoliestRoger", UserID:"~TheHolies@unaffiliated/theholiestroger", Avatar:"", Account:"irc.roger", Event:"", Protocol:"irc", Gateway:"roger", ParentID:"", Timestamp:time.Time{wall:0xbfdc32384b7ea793, ext:861398311063, loc:(*time.Location)(0x31a9360)}, ID:"", Extra:map[string][]interface {}(nil)} from irc.roger (#theholyroger) to slack.roger (general)" func=SendMessage file="gateway/gateway.go:421" prefix=gateway
time="2020-10-21T15:18:57Z" level=debug msg="=> Receiving config.Message{Text:"", Channel:"general", Username:"", UserID:"~TheHolies@unaffiliated/theholiestroger", Avatar:"https://theholyroger.com/roger-avatars/TheHoliestRoger?1", Account:"irc.roger", Event:"", Protocol:"irc", Gateway:"roger", ParentID:"", Timestamp:time.Time{wall:0xbfdc32384b7ea793, ext:861398311063, loc:(*time.Location)(0x31a9360)}, ID:"", Extra:map[string][]interface {}(nil)}" func=Send file="bridge/slack/slack.go:195" prefix=slack
time="2020-10-21T15:18:57Z" level=debug msg="=> Sending config.Message{Text:"&skipthis ", Channel:"#theholyroger", Username:"TheHoliestRoger", UserID:"~TheHolies@unaffiliated/theholiestroger", Avatar:"https://theholyroger.com/roger-avatars/TheHoliestRoger?1", Account:"irc.roger", Event:"", Protocol:"irc", Gateway:"roger", ParentID:"", Timestamp:time.Time{wall:0xbfdc32384b7ea793, ext:861398311063, loc:(*time.Location)(0x31a9360)}, ID:"", Extra:map[string][]interface {}(nil)} from irc.roger (#theholyroger) to keybase.roger (general)" func=SendMessage file="gateway/gateway.go:421" prefix=gateway
time="2020-10-21T15:18:57Z" level=debug msg="=> Receiving config.Message{Text:"", Channel:"general", Username:"", UserID:"~TheHolies@unaffiliated/theholiestroger", Avatar:"https://theholyroger.com/roger-avatars/TheHoliestRoger?1", Account:"irc.roger", Event:"", Protocol:"irc", Gateway:"roger", ParentID:"", Timestamp:time.Time{wall:0xbfdc32384b7ea793, ext:861398311063, loc:(*time.Location)(0x31a9360)}, ID:"", Extra:map[string][]interface {}(nil)}" func=Send file="bridge/keybase/keybase.go:62" prefix=keybase
time="2020-10-21T15:18:57Z" level=error msg="SendMessage failed: invalid send v1 options: invalid message, body cannot be empty" func=handleMessage file="gateway/handlers.go:228" prefix=gateway
time="2020-10-21T15:18:57Z" level=debug msg="=> Sending config.Message{Text:"&skipthis ", Channel:"#theholyroger", Username:"TheHoliestRoger", UserID:"~TheHolies@unaffiliated/theholiestroger", Avatar:"https://theholyroger.com/roger-avatars/TheHoliestRoger?1", Account:"irc.roger", Event:"", Protocol:"irc", Gateway:"roger", ParentID:"", Timestamp:time.Time{wall:0xbfdc32384b7ea793, ext:861398311063, loc:(*time.Location)(0x31a9360)}, ID:"", Extra:map[string][]interface {}(nil)} from irc.roger (#theholyroger) to discord.roger (ID:465924436407615488)" func=SendMessage file="gateway/gateway.go:421" prefix=gateway
time="2020-10-21T15:18:57Z" level=debug msg="=> Receiving config.Message{Text:"", Channel:"ID:465924436407615488", Username:"", UserID:"~TheHolies@unaffiliated/theholiestroger", Avatar:"https://theholyroger.com/roger-avatars/TheHoliestRoger?1", Account:"irc.roger", Event:"", Protocol:"irc", Gateway:"roger", ParentID:"", Timestamp:time.Time{wall:0xbfdc32384b7ea793, ext:861398311063, loc:(*time.Location)(0x31a9360)}, ID:"", Extra:map[string][]interface {}(nil)}" func=Send file="bridge/discord/discord.go:192" prefix=discord
time="2020-10-21T15:18:57Z" level=debug msg="Broadcasting using Webhook" func=Send file="bridge/discord/discord.go:246" prefix=discord
time="2020-10-21T15:18:57Z" level=debug msg="Skipping empty message config.Message{Text:"", Channel:"ID:465924436407615488", Username:"", UserID:"~TheHolies@unaffiliated/theholiestroger", Avatar:"https://theholyroger.com/roger-avatars/TheHoliestRoger?1", Account:"irc.roger", Event:"", Protocol:"irc", Gateway:"roger", ParentID:"", Timestamp:time.Time{wall:0xbfdc32384b7ea793, ext:861398311063, loc:(*time.Location)(0x31a9360)}, ID:"", Extra:map[string][]interface {}(nil)}" func=Send file="bridge/discord/discord.go:250" prefix=discord
time="2020-10-21T15:18:57Z" level=debug msg="== Receiving event: &gomatrix.Event{StateKey:(*string)(nil), Sender:"@rogerbridge:matrix.theholyroger.com", Type:"m.room.message", Timestamp:1603293537202, ID:"$lZlAZoJgY6ZwELfJpRUUTZ0uoDZpzHmIaggEsCvUWTk", RoomID:"!wAjlYiluyhksKYlJMk:matrix.theholyroger.com", Redacts:"", Unsigned:map[string]interface {}{"age":617, "transaction_id":"go1603293537195348345"}, Content:map[string]interface {}{"body":"", "format":"org.matrix.custom.html", "formatted_body":"", "msgtype":"m.text"}, PrevContent:map[string]interface {}(nil)}" func=handleEvent file="bridge/matrix/matrix.go:202" prefix=matrix

42wim · 2020-10-21T17:49:21Z

Ok, now I understand you're creating those messages yourself with tengo.

Could you paste your tengo script?

42wim · 2020-10-21T19:23:54Z

@TheHolyRoger see PR #1272
This gives tengo the control to drop (empty) messages.

TheHolyRoger · 2020-10-21T20:37:57Z

@42wim brilliant thank you!!

tengo script:

text := import("text")

if inProtocol == "irc" && outProtocol != "irc" {
	re := text.re_compile('\x03(?:\d{1,2}(?:,\d{1,2})?)?|[[:cntrl:]]')
	msgText=re.replace(msgText,"")
	if text.contains(msgText, "▄▄") {
		msgUsername=""
		msgText=""
		msgDrop=true
	} else if (text.contains(msgUsername, "Wallet") && (!text.contains(msgText, "soak") && !text.contains(msgText, "Vol") && !text.contains(msgText, "~"))) {
		msgUsername=""
		msgText=""
		msgDrop=true
	} else if (text.contains(msgText, ".faucet") || text.contains(msgText, "&faucet") || text.contains(msgText, "&skipthis")) || (text.contains(msgUsername, "TheHolyRoger") && text.contains(msgText, "@Pot =")) || (text.contains(msgUsername, "Wallet") && text.contains(msgText, "@BonusPot: ")) {
		msgUsername=""
		msgText=""
		msgDrop=true
	}


// strip custom emoji
} else if inProtocol == "discord" {
	re := text.re_compile('<a?(:.*?:)[0-9]+>')
	msgText=re.replace(msgText,"$1")
	if text.contains(msgText, ":rogercoin:") {
		msgText=text.replace(msgText, ":rogercoin:", "®", 5)
	}


// tele
} else if inProtocol == "telegram" {
	if outChannel == "ID:646764479983648791" && text.contains(msgText, ".gif") {
		msgText = ""
		msgDrop=true
	}


} else if inProtocol == "matrix" {
	re := text.re_compile('<[\/]?strong>')
	if outProtocol == "discord" || outProtocol == "keybase" {
		msgText=re.replace(msgText,"**")
	} else if outProtocol == "irc" {
		msgText=re.replace(msgText,"")
	} else if outProtocol == "slack" {
		msgText=re.replace(msgText,"*")
	}
	msgText=text.replace(msgText, "  ", " ", 8)



} else if inProtocol == "slack" && outProtocol == "irc" {
	if text.has_prefix(msgText, ".") {
		oldUsername := text.replace(msgUsername, ":", "", 1)
		msgUsername=""
		if msgText != ".ltc" {
			msgText=msgText+" Sent by "+oldUsername
		} else {
			msgText="&price ltc"
		}
	}
}

if outProtocol != "telegram" && outProtocol != "discord" {
	if len(msgUsername) > 24 {
		msgUsername = string(msgUsername[0]) + string(msgUsername[1]) + string(msgUsername[2:5]) + string("") + string(msgUsername[5:])
	} else if len(msgUsername) > 6 {
		msgUsername = string(msgUsername[0]) + string(msgUsername[1]) + string("") + string(msgUsername[2:5]) + string("") + string(msgUsername[5:])
	} else if len(msgUsername) > 2 {
		msgUsername = string(msgUsername[0]) + string("") + string(msgUsername[1]) + string("") + string(msgUsername[2:])
	} else if len(msgUsername) > 1 {
		msgUsername = string(msgUsername[0]) + string("") + string(msgUsername[1:])
	}

	if outProtocol == "keybase" {
		msgText=text.replace(msgText, "%", "_", 5)
	}
}

TheHolyRoger mentioned this pull request Oct 20, 2020

Frequent IRC disconnects #1255

Closed

TheHolyRoger force-pushed the skip-empty branch from 9c6eb77 to 6476915 Compare October 20, 2020 01:48

TheHolyRoger force-pushed the skip-empty branch from 6476915 to e20e90d Compare October 20, 2020 20:57

TheHolyRoger added 2 commits October 21, 2020 00:42

Skip empty messages.

49e51d0

Add conditionals for API messages and empty usernames

671ed3c

TheHolyRoger force-pushed the skip-empty branch from e20e90d to 671ed3c Compare October 20, 2020 23:43

TheHolyRoger closed this Oct 21, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Skip empty messages. #1261

Skip empty messages. #1261

TheHolyRoger commented Oct 17, 2020

42wim commented Oct 20, 2020

TheHolyRoger commented Oct 20, 2020

42wim commented Oct 20, 2020 •

edited

Loading

TheHolyRoger commented Oct 20, 2020

codeclimate bot commented Oct 20, 2020

42wim commented Oct 21, 2020

TheHolyRoger commented Oct 21, 2020

TheHolyRoger commented Oct 21, 2020

42wim commented Oct 21, 2020

TheHolyRoger commented Oct 21, 2020 •

edited

Loading

42wim commented Oct 21, 2020 •

edited

Loading

42wim commented Oct 21, 2020

TheHolyRoger commented Oct 21, 2020 •

edited

Loading

Skip empty messages. #1261

Skip empty messages. #1261

Conversation

TheHolyRoger commented Oct 17, 2020

42wim commented Oct 20, 2020

TheHolyRoger commented Oct 20, 2020

42wim commented Oct 20, 2020 • edited Loading

TheHolyRoger commented Oct 20, 2020

codeclimate bot commented Oct 20, 2020

42wim commented Oct 21, 2020

TheHolyRoger commented Oct 21, 2020

TheHolyRoger commented Oct 21, 2020

42wim commented Oct 21, 2020

TheHolyRoger commented Oct 21, 2020 • edited Loading

42wim commented Oct 21, 2020 • edited Loading

42wim commented Oct 21, 2020

TheHolyRoger commented Oct 21, 2020 • edited Loading

42wim commented Oct 20, 2020 •

edited

Loading

TheHolyRoger commented Oct 21, 2020 •

edited

Loading

42wim commented Oct 21, 2020 •

edited

Loading

TheHolyRoger commented Oct 21, 2020 •

edited

Loading