Add `.bas` extension to VB6 and heuristics for VB6/VBA #6355

DecimalTurn · 2023-04-02T23:12:02Z

This PR:

Adds the extension .bas as an extension for VB6
Adds samples for VB6 and VBA to help with disambiguation
Implements heuristics for VBA and VB6 with respect to the .bas extension

Description

This is the 3rd instalment of a 3 part series related to VB6 (and VBA).
For more context:
Part 1: Add Visual Basic 6.0 as its own language
Part 2: Add .frm and .cls extensions to VB6

The reason why I've waited a bit to add the .bas extension was because it is more tricky than for the extensions in part 2. Indeed, .bas is used for a module file and the attributes at the top of those files are identical between VBA and VB6.
ie.:
Attribute VB_Name = "ClassOrModuleName"
So, we can't use it to separate the two, but we can at least use it to distinguish VBA/VB6 from other languages using the .bas file extension (see new heuristic with named pattern vb-module).

Hence, the general approach that I'm proposing is the same idea as what was used for the .h extension as suggested by @lildude:

This whole VB-family of languages classification issue sounds like it's going to be a lot like the .h extension used by the C-family of languages which was a long battle until we finally resolved on the strategy in use and detailed in the troublshooting doc. If a similar approach can be found for these VB languages [and] their shared extensions, then that would be great. If not, the only option is manual overrides.

From #5824 (comment)

This comment refers to part of the troubleshooting.md file reproduced here with the appropriate substitution for the case of .bas :

Correctly detecting the language for the [Visual Basic .bas module] files is tough because Linguist detects the languages of files in isolation when analysing repositories and these [module] files, especially the smaller ones, can be used across [the 2] languages without using any language-specific content. To try and reduce the number of false positives and keep some degree of predicatability (sic) for users, Linguist will assume all [.bas module files with a VB_NAME Attribute] are [VB6] by default and will only identify a file as [VBA] if the content matches the specific heuristic for that language. This will mean you will need to implement an override for some of your [module] files if you wish for them to be classified as [VBA] if they do no contain language-specific content.

Difference between VBA and VB6
Alright, now let's get to the meat of it: what are the identifiable differences between VBA and VB6?

VBA7 has new 64-bit features that were not implemented in VB6
VBA has some unique top module declarations
a. Option Private Module
b. Option Compare Database
VBA has preloaded libraries with objects that are common in VBA code

These 3 sections appear in the vba named pattern at the bottom of the heuristic file. I don't think there is a need for debate on (1) and (2) as those differences between VBA and VB6 are directly based on syntax and are well documented. However, I could see some disagreement regarding section (3) as there is a bit more of a grey area there.

Regarding the choices made for (3), I've included the invocation of a VBA library. This means that just having "Word" in the code is not enough to trigger the regex, you need to invoke a method/property/constant on it like (e.g. Word.Application). I've also included multiple common objects that are part of the DLLs VBA has natively access to.

Potential issues with section 3
The issue with this is that some VB6 code can reference the DLLs in question and then be "wrongly" labelled as VBA. I'm using quotes around "wrongly" because a case could be made that if you are using VB to automate an office application, you are basically writing VBA code at this point 😅.

Anyhow, I'm open to tweaking the list of objects included in case there is too many false positives, but since this approach gives VB6 as the default outcome for vb-like modules, it seems fair to give some extra chances to VBA.

@fafalone, @XusinboyBekchanov, @Gagniuc and @zspitz, I would be interested to have your opinion on this since you've been involved in discussions regarding this topic in the past. Cheers!

Checklist:

I am adding a new extension to a language. (VB6)
- The new extension is used in hundreds of repositories on GitHub.com
  - Search results for each extension:
    - https://github.com/search?utf8=%E2%9C%93&type=Code&ref=searchresults&q=extension%3Abas+public%20sub+NOT+nothack
- I have included a real-world usage sample for all extensions added in this PR:
  - Sample source(s):
    - https://github.com/raw/wqweto/VbAsyncSocket/master/test/Basic/Module1.bas
      Licence: MIT
    - https://github.com/raw/dragokas/hijackthis/7560e67b5e0d14f53ea25bbfb88587cbfadca1c8/src/tools/RegTLib/Module1.bas (renamed to Module 2)
      License: GPL2
    - https://github.com/ImortisInglorian/fbrtLib/blob/ecfda75a82ad928220dea28f7d477787f89e7456/array_clearobj.bas
      License: GNU Lesser General Public
    - freebasic/fbc@e43074e/src/compiler/ir.bas
      License: GNU GPLv2
    - https://github.com/freebasic/fbc/blob/e43074e5aa97a904a31c00d9917bc952308bddc0/contrib/nsis-installer/makescript.bas
      License: GNU GPLv2
- I have included a change to the heuristics to distinguish my language from others using the same extension.
I am fixing a misclassified language (VBA)
- I have included a new sample for the misclassified language:
  - Sample source(s):
    - https://github.com/raw/hilkoc/vbaDeveloper/master/src/vbaDeveloper.xlam/ErrorHandling.bas
      License: MIT
- I have included a change to the heuristics to distinguish my language from others using the same extension.

DecimalTurn · 2023-04-02T23:28:50Z

Looks like with the new samples, the FreeBasic samples are now detected as VB6:

FreeBasic/Plasma Generation.bas BAD (Visual Basic 6.0)
FreeBasic/try_catch_throw.bas BAD (Visual Basic 6.0)

@XusinboyBekchanov, are there any good samples you'd suggest to add for FreeBasic?

zspitz · 2023-04-03T00:26:34Z

I would argue that the situation around VBA/VB6 is more akin to JavaScript. The language is the same language regardless of runtime environment (browser, Node.js) or loaded libraries and global objects.

My original intention was to use VBA/VB6 as the language name, but Linguist doesn't like slashes in language names so I went with VBA.

AFAICT the only argument for VB6 as a separate language is that there are some file extensions and file formats unique to VB6.

Alhadis · 2023-04-03T00:33:09Z

My original intention was to use VBA/VB6 as the language name

That strikes me as a combination of two equally-valid names for the same technology. In which case, either VBA or VB6 should be added as an alias instead. Otherwise, if you really must include a slash, then this is something that the fs_name field should (ideally) accommodate.

I would argue that the situation around VBA/VB6 is more akin to JavaScript.

Interesting analogy. 😉 Early versions of Internet Explorer actually did support VBA (or something with "Visual Basic" in its name), but it was discontinued due to lack of adoption and the growing popularity of JavaScript.

zspitz · 2023-04-03T00:51:30Z

Early versions of Internet Explorer actually did support VBA (or something with "Visual Basic" in its name)

That would be VBScript, which (I would argue) is indeed a separate language. Most notably, because -- like JavaScript -- it has no notion of compile-time types.

Alhadis · 2023-04-03T00:58:09Z

like JavaScript -- it has no notion of compile-time types.

Static typing is overrated anyway.

DecimalTurn · 2023-04-03T01:58:04Z

Thanks for the feedback @zspitz!
However, VBA and VB6 do have a different syntax when it comes to handling 64-bit stuff and there is also module declarations that you won't find in VB6.

This means that unlike JavaScript, you can't just copy-paste code across IDEs, add some librairies and expect the code to work. It won't compile in VB6.

For this reason, I would argue that we should keep them as seperate languages.

From the PR description:

Difference between VBA and VB6
Alright, now let's get to the meat of it: what are the identifiable differences between VBA and VB6?

VBA7 has new 64-bit features that were not implemented in VB6
VBA has some unique top module declarations
a. Option Private Module
b. Option Compare Database

XusinboyBekchanov · 2023-04-03T02:14:38Z

Looks like with the new samples, the FreeBasic samples are now detected as VB6:

FreeBasic/Plasma Generation.bas BAD (Visual Basic 6.0)
FreeBasic/try_catch_throw.bas BAD (Visual Basic 6.0)

@XusinboyBekchanov, are there any good samples you'd suggest to add for FreeBasic?

Plasma Generation.bas uses a unique keyword for FreeBasic Shared: https://freebasic.net/wiki/KeyPgShared

The try_catch_throw.bas uses the #include keyword as specified in the heuristic: https://github.com/github/linguist/blob/master/lib/linguist/heuristics.yml#L116

Alhadis

I had a brief flashback to high school accounting classes (my only experience with Microsoft Excel) and recall formulae being case-insensitive. A quick Google search suggests Visual Basic—or one of its confuseables¹—is case-insensitive. If so, these regexes should begin with (?i) to mark them as case-insensitive matches. I'll leave this up to you to decide; I'm out-of-my-depth when it comes to Microsoft technologies named /Visual\s*(Studio|Basic)/.

These technologies wouldn't be Microsoft if they weren't given confusing, forgettable and/or easily-rebranded names. Or in the case of Excel's Turing-complete formula functions, no programming language name whatsoever. ↩

lib/linguist/heuristics.yml

XusinboyBekchanov · 2023-04-03T02:45:11Z

I had a brief flashback to high school accounting classes (my only experience with Microsoft Excel) and recall formulae being case-insensitive. A quick Google search suggests Visual Basic—or one of its confuseables—is case-insensitive. If so, these regexes should begin with (?i) to mark them as case-insensitive matches. I'll leave this up to you to decide; I'm out-of-my-depth when it comes to Microsoft technologies named /Visual\s*(Studio|Basic)/.

This needs to be added for FreeBasic.

XusinboyBekchanov · 2023-04-03T02:49:54Z

For VBA, you also need to add: Attribute VB_Name =

DecimalTurn · 2023-04-03T02:52:45Z

Plasma Generation.bas uses a unique keyword for FreeBasic Shared: freebasic.net/wiki/KeyPgShared

The try_catch_throw.bas uses the #include keyword as specified in the heuristic: master/lib/linguist/heuristics.yml#L116

Thanks for the clarification @XusinboyBekchanov, so my undestanding is that the supplied sample Plasma Generation.bas is not really code that can be found "in the wild" but more of an example to illustrate a feature of the language.

What if we added this sample below instead which is longer and more in line with a good sample to illustrate the use of the "shared" keyword?
https://github.com/freebasic/fbc/blob/e43074e5aa97a904a31c00d9917bc952308bddc0/src/compiler/ir.bas

XusinboyBekchanov · 2023-04-03T02:55:49Z

Thanks for the clarification @XusinboyBekchanov, so my understanding is that the supplied sample Plasma Generation.bas is not really code that can be found "in the wild" but more of an example to illustrate a feature of the language.

What if we added this sample below instead which is longer and more in line with a good sample to illustrate the use of the "shared" keyword? https://github.com/freebasic/fbc/blob/e43074e5aa97a904a31c00d9917bc952308bddc0/src/compiler/ir.bas

The one provided in your example has the same #include. In this case, it is determined by this feature.

Alhadis · 2023-04-03T03:01:18Z

Folks, let's try to avoid nested quote-blocks (preferably by quoting only the relevant parts of a comment). They make this discussion much harder to follow, not to mention encumbering CTRL+F searches to find a specific part of a user's original reply.

DecimalTurn · 2023-04-03T03:04:56Z

@XusinboyBekchanov, from what I understand, the main goal of the samples are to help the classifer make a guess when the heuristics aren't conclusive. So, it's important that the samples represent FreeBasic code "in the wild" to help classify real code and not contrived examples.

I'm going to change the sample Plasma Generation.bas for this sample to see if it helps with the classifier: freebasic/fbc@e43074e/src/compiler/ir.bas (License GNU GPLv2).

lib/linguist/heuristics.yml

DecimalTurn · 2023-04-03T03:39:07Z

For VBA, you also need to add: Attribute VB_Name =

@XusinboyBekchanov I've added this heuristic in the form of the named pattern "vb-module" as it is both used by VBA and VB6

DecimalTurn · 2023-04-03T03:54:53Z

[...] If so, these regexes should begin with (?i) to mark them as case-insensitive matches.

@Alhadis, thanks for letting me know this is possible. While it is true that Excel formulas are case-insensitive, it is not always true for VBA. For instance, the reserved keywords and compiler constants will have a specific capitalization that the VBA IDE will enforce as soon as you exit the line in the editor.

For this reason, I don't think we should worry about the case insensitivity of VBA. The only place it can play a role is with variable names (sometimes), but that shouldn't apply to the heuristics used here.

Alhadis · 2023-04-03T04:07:42Z

specific capitalization that the VBA IDE will enforce as soon as you exit the line in the editor.

Seriously? (An editor that does that would shit me right up the wall…)

For this reason, I don't think we should worry about the case insensitivity of VBA

Agreed, provided there's no substantial risk of these files being authored in other editing environments. However, @XusinboyBekchanov mentioned earlier that FreeBASIC's heuristics should be marked case-insensitive (where it makes sense to do so), although that might warrant a separate PR.

lildude

Please find smaller samples. Hint: if the PR is not displaying the full file content in the diff, it's probably unnecessarily large.

I think we're also going to need another FreeBasic sample or two (depends on how things behave after you replace the large samples) as the classifier is now detecting one of the current FreeBasic samples as Visual Basic 6.0:

FreeBasic/try_catch_throw.bas BAD (Visual Basic 6.0)

lib/linguist/languages.yml

DecimalTurn · 2023-04-05T03:35:35Z

Please find smaller samples. Hint: if the PR is not displaying the full file content in the diff, it's probably unnecessarily large.

@lildude, thanks for your patience on this one. At least I didn't go with a 8000+ lines sample this time, so there's progress 😅.
I'm working on some more adjustements to the heuristics and I'll make the changes for the samples this week.

DecimalTurn · 2023-04-09T18:26:01Z

Classifier cross-validation is still failing for these:

BASIC/mandelbrot.bas BAD (Visual Basic 6.0)
FreeBasic/try_catch_throw.bas BAD (Visual Basic 6.0)

DecimalTurn · 2023-04-09T19:01:00Z

Adding the following sample for BASIC fixed the issue with BASIC/mandelbrot.bas
https://github.com/raw/DualBrain/gw-basic/032cbf70bcddd5a971a81d7e56e795e61fe028f0/spacesc.bas
License: MIT

However, the issue with FreeBasic didn't improve by adding Picture.bas as a sample 😔

DecimalTurn · 2023-04-11T00:08:04Z

@lildude, I'd be interested to know if you have any more suggestions regarding this issue with samples for FreeBasic (FB) and the failing classifier tests. Just to recap :

What was the original impact of the PR?:

Both samples already present in Linguist for FB started to classify as VB6 (try_catch_throw.bas ❌ and Plasma Generation.bas ❌). Note that when you made your previous comment, it was right after I had temporarily removed the sample Plasma Generation.bas to see if that would help.

Which samples I've tried to add:

ir.bas => is correctly classifed as FB ✅
Picture.bas => is classified as VB6 ❌

So, after adding 2 more samples, we only have 1/4 passing the test. What should be done in this case?

XusinboyBekchanov · 2023-04-11T02:16:56Z

You have provided several different heuristics. They have Attribute VB_Name = "ClassOrModuleName" inside them too.
But each provided heuristic must contain this check.

DecimalTurn · 2023-04-11T03:15:12Z

@XusinboyBekchanov, I'm not sure I understand your comment. With the current heuristics, a file will need to match the vb-module pattern AND any of the heuristics listed inside the vba pattern to be labelled as VBA. Hence, the presence of Attribute VB_Name = is made mandatory by the use of the vb-module pattern for VBA (and VB6).

  - language: VBA
    and:
    - named_pattern: vb-module
    - named_pattern: vba
  - language: Visual Basic 6.0
    named_pattern: vb-module

vb-module: '^\s*Attribute VB_Name = '

XusinboyBekchanov · 2023-04-11T03:46:55Z

Maybe OR?
In your case, the FreeBasic examples should not have been marked as VB6.
Then you can combine all heuristics into one heuristic and check.

Tweak some more heuristics Simplify Range heuristic to account for cases like Range(B & row_number) Edit heuristics

Fix issue with `\b` and make sure we match with `VBAObject`

…ttern)

DecimalTurn · 2023-05-10T02:05:18Z

@Alhadis, I believe that the suggestions you've made were addressed. Can you please have a look and let me know if there is something I missed or something else that could be improved?

Alhadis · 2023-05-12T13:51:24Z

Hey @DecimalTurn, sorry about the wait. 👋 You've been very patient.

Yup, this LGTM. @lildude, any thoughts?

lildude

This all looks good to me. Thanks for your effort and patience.

Note: this PR will not be merged until close to when the next release is made. See here for more details.

DecimalTurn requested a review from a team as a code owner April 2, 2023 23:12

Alhadis changed the title ~~Add .bas extension to VB6 and heuristics for VB6/VBA~~ Add .bas extension to VB6 and heuristics for VB6/VBA Apr 3, 2023

Alhadis requested changes Apr 3, 2023

View reviewed changes

lib/linguist/heuristics.yml Outdated Show resolved Hide resolved

lildude requested changes Apr 3, 2023

View reviewed changes

lib/linguist/languages.yml Show resolved Hide resolved

DecimalTurn force-pushed the bas branch 2 times, most recently from a9e03cf to f348586 Compare April 9, 2023 19:33

DecimalTurn added 15 commits April 27, 2023 21:14

Remove URL

ef1e18a

Put all comments on seperate line

b6329c2

Using regular space (U+0020) where applicable

4e15c6d

Tweak Excel object rules

ed3826d

Tweak some more heuristics Simplify Range heuristic to account for cases like Range(B & row_number) Edit heuristics

Replace VB6 samples

11db47d

Change VBA sample

9dc9407

Modify heuristic to include "vba" prefix (PascalCase or camelCase)

9a00ee7

Add BASIC sample

6090761

Add FreeBasic sample

f9d0526

Change new FreeBasic sample

bee1d01

Restore Plasma Generation.bas

2d16003

Update samples

2571a7a

Combine rules

4af46cd

Heuristics adjustments

544108e

Update heuristics.yml

4b2e5d6

Fix issue with `\b` and make sure we match with `VBAObject`

DecimalTurn force-pushed the bas branch from a0fb388 to 4b2e5d6 Compare April 28, 2023 01:14

Change characters ordering and remove VBA7 (already matching first pa…

1c00759

…ttern)

Merge branch 'master' into bas

46d69e0

Alhadis approved these changes May 12, 2023

View reviewed changes

lildude approved these changes May 17, 2023

View reviewed changes

Merge branch 'master' into bas

c73ace8

lildude requested a review from a team as a code owner May 30, 2023 09:14

lildude added this pull request to the merge queue May 30, 2023

lildude approved these changes May 30, 2023

View reviewed changes

Merged via the queue into github-linguist:master with commit e9bd67d May 30, 2023

DecimalTurn mentioned this pull request Jun 6, 2023

Add VB6 highlighting based on the VBA syntax serkonda7/vscode-vba#74

Open

github-linguist locked as resolved and limited conversation to collaborators Jun 17, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add `.bas` extension to VB6 and heuristics for VB6/VBA #6355

Add `.bas` extension to VB6 and heuristics for VB6/VBA #6355

DecimalTurn commented Apr 2, 2023 •

edited by lildude

Loading

DecimalTurn commented Apr 2, 2023 •

edited

Loading

zspitz commented Apr 3, 2023

Alhadis commented Apr 3, 2023 •

edited

Loading

zspitz commented Apr 3, 2023

Alhadis commented Apr 3, 2023

DecimalTurn commented Apr 3, 2023 •

edited

Loading

XusinboyBekchanov commented Apr 3, 2023

Alhadis left a comment •

edited

Loading

XusinboyBekchanov commented Apr 3, 2023 •

edited by Alhadis

Loading

XusinboyBekchanov commented Apr 3, 2023

DecimalTurn commented Apr 3, 2023 •

edited by Alhadis

Loading

XusinboyBekchanov commented Apr 3, 2023 •

edited by Alhadis

Loading

Alhadis commented Apr 3, 2023 •

edited

Loading

DecimalTurn commented Apr 3, 2023 •

edited

Loading

DecimalTurn commented Apr 3, 2023 •

edited

Loading

DecimalTurn commented Apr 3, 2023 •

edited

Loading

Alhadis commented Apr 3, 2023

lildude left a comment

DecimalTurn commented Apr 5, 2023

DecimalTurn commented Apr 9, 2023 •

edited

Loading

DecimalTurn commented Apr 9, 2023 •

edited

Loading

DecimalTurn commented Apr 11, 2023 •

edited

Loading

XusinboyBekchanov commented Apr 11, 2023

DecimalTurn commented Apr 11, 2023 •

edited

Loading

XusinboyBekchanov commented Apr 11, 2023

DecimalTurn commented May 10, 2023

Alhadis commented May 12, 2023

lildude left a comment •

edited by Alhadis

Loading

Add .bas extension to VB6 and heuristics for VB6/VBA #6355

Add .bas extension to VB6 and heuristics for VB6/VBA #6355

Conversation

DecimalTurn commented Apr 2, 2023 • edited by lildude Loading

Description

Checklist:

DecimalTurn commented Apr 2, 2023 • edited Loading

zspitz commented Apr 3, 2023

Alhadis commented Apr 3, 2023 • edited Loading

zspitz commented Apr 3, 2023

Alhadis commented Apr 3, 2023

DecimalTurn commented Apr 3, 2023 • edited Loading

XusinboyBekchanov commented Apr 3, 2023

Alhadis left a comment • edited Loading

Choose a reason for hiding this comment

Footnotes

XusinboyBekchanov commented Apr 3, 2023 • edited by Alhadis Loading

XusinboyBekchanov commented Apr 3, 2023

DecimalTurn commented Apr 3, 2023 • edited by Alhadis Loading

XusinboyBekchanov commented Apr 3, 2023 • edited by Alhadis Loading

Alhadis commented Apr 3, 2023 • edited Loading

DecimalTurn commented Apr 3, 2023 • edited Loading

DecimalTurn commented Apr 3, 2023 • edited Loading

DecimalTurn commented Apr 3, 2023 • edited Loading

Alhadis commented Apr 3, 2023

lildude left a comment

Choose a reason for hiding this comment

DecimalTurn commented Apr 5, 2023

DecimalTurn commented Apr 9, 2023 • edited Loading

DecimalTurn commented Apr 9, 2023 • edited Loading

DecimalTurn commented Apr 11, 2023 • edited Loading

XusinboyBekchanov commented Apr 11, 2023

DecimalTurn commented Apr 11, 2023 • edited Loading

XusinboyBekchanov commented Apr 11, 2023

DecimalTurn commented May 10, 2023

Alhadis commented May 12, 2023

lildude left a comment • edited by Alhadis Loading

Choose a reason for hiding this comment

Add `.bas` extension to VB6 and heuristics for VB6/VBA #6355

Add `.bas` extension to VB6 and heuristics for VB6/VBA #6355

DecimalTurn commented Apr 2, 2023 •

edited by lildude

Loading

DecimalTurn commented Apr 2, 2023 •

edited

Loading

Alhadis commented Apr 3, 2023 •

edited

Loading

DecimalTurn commented Apr 3, 2023 •

edited

Loading

Alhadis left a comment •

edited

Loading

XusinboyBekchanov commented Apr 3, 2023 •

edited by Alhadis

Loading

DecimalTurn commented Apr 3, 2023 •

edited by Alhadis

Loading

XusinboyBekchanov commented Apr 3, 2023 •

edited by Alhadis

Loading

Alhadis commented Apr 3, 2023 •

edited

Loading

DecimalTurn commented Apr 3, 2023 •

edited

Loading

DecimalTurn commented Apr 3, 2023 •

edited

Loading

DecimalTurn commented Apr 3, 2023 •

edited

Loading

DecimalTurn commented Apr 9, 2023 •

edited

Loading

DecimalTurn commented Apr 9, 2023 •

edited

Loading

DecimalTurn commented Apr 11, 2023 •

edited

Loading

DecimalTurn commented Apr 11, 2023 •

edited

Loading

lildude left a comment •

edited by Alhadis

Loading