New language: CIL #1593

sbrl · 2018-10-23T18:19:29Z

I discovered that Prism.js doesn't have CIL (Common Intermediate Language, from .NET) support when I wrote a blog post that contained some - so I decided to create my own grammar, since the process looked fairly easy (it was!).

mAAdhaTTah · 2018-10-23T20:12:58Z

Couple of quick notes:

Filename should be prism-cil.js, to match the rest of the languages.
Run gulp to generate the minfied version.
Update the components.js file with the language information.

sbrl · 2018-10-23T20:26:14Z

Done - thanks for the comment, @mAAdhaTTah! I was wondering how that worked.

Looks like you've got some vulnerabilities in your required npm packages though:

npm audit fix should do the trick to fix that.

mAAdhaTTah · 2018-10-23T20:31:56Z

components/prism-cil.js

@@ -0,0 +1,26 @@
+Prism.languages.cil = {
+	'displayTitle': "CIL",


displayTitle doesn't go here, you already have it in the components.json.

mAAdhaTTah · 2018-10-23T20:36:43Z

Thanks for the contribution. One minor comment, but I'll need someone with stronger regex skills to give this a full review. Also, if this is a .NET extension, should this extend from the csharp language?

RunDevelopment · 2018-10-23T20:45:05Z

CIL is a language on its own and C# is to CIL basically what C is to Assembly; Namely, C# compiles to CIL.
So, I don't think that extending C# is going to help very much.

RunDevelopment

First of all, thank you for this amazing PR.

But there are a few things that have to be done before we can merge this:

Please remove the package-lock.json.
Please remove the small changes made to prism.js.

I left you a few comments and questions. Also, it'd be great if you were to add some tests.

RunDevelopment · 2018-10-24T08:47:37Z

components/prism-cil.js

+Prism.languages.cil = {
+	'displayTitle': "CIL",
+
+	'comment': /\/\/[^\n]*(?=\r?\n)/,


Because [^\n]* is greedy it will also consume the \r at the end of a line.
The pattern /\/\/.*/ will do what you want and is really simple.

Also, does CIL support multi-line comments?

components/prism-cil.js

RunDevelopment · 2018-10-24T08:51:06Z

components/prism-cil.js

+
+	'class-name': {
+		lookbehind: true,
+		pattern: /([^a-zA-Z0-9])\.[a-z]+\b/


This is really nitpicky but we usually have the pattern before everything else (namely: lookbehind).

RunDevelopment · 2018-10-24T09:03:30Z

components/prism-cil.js

+	'number': /\b(0x)?[0-9a-fA-F]+\b/i,
+	'punctuation': /[{}[\];(),:]|IL_[0-9A-Za-z]+/,
+
+	'keyword': /\b(iant|currency|syschar|void|bool|int8|int16|int32|int64|float32|float64|error|unsigned int8|unsigned int16|unsigned int32|unsigned int64|nativeType|decimal|date|bstr|lpstr|lpwstr|lptstr|objectref|iunknown|idispatch|struct|interface|int|unsigned int|nested struct|byvalstr|ansi bstr|tbstr|variant bool|lpstruct|static|public|private|family|final|specialname|virtual|abstract|assembly|default|cil managed|famandassem|famorassem|privatescope|hidebysig|newslot|rtspecialname|unmanagedexp|reqsecobj|pinvokeimpl|value|enum|interface|sealed|abstract|auto|sequential|explicit|ansi|unicode|autochar|import|serializable|nested public|nested private|nested family|nested assembly|nested famandassem|nested famorassem|beforefieldinit|specialname|rtspecialname|class|instance|string|extern|extends)\b/,


A few things about this keyword list:

initonly is missing.

Please split things like unsigned int32 into unsigned and int32. This will reduce the number of keywords.

Please sort the keywords alphabetically. This makes the list easier to maintain and you will notice the keywords which are included twice.

RunDevelopment · 2018-10-24T09:07:29Z

components/prism-cil.js

+
+	'boolean': /\b(?:true|false)\b/,
+	'number': /\b(0x)?[0-9a-fA-F]+\b/i,
+	'punctuation': /[{}[\];(),:]|IL_[0-9A-Za-z]+/,


. and = should also be punctuation.

RunDevelopment · 2018-10-24T09:07:51Z

components/prism-cil.js

+	'variable': /\[\w+\]/,
+
+	'boolean': /\b(?:true|false)\b/,
+	'number': /\b(0x)?[0-9a-fA-F]+\b/i,


What about floating point numbers?

Good point! I'll update now.

RunDevelopment · 2018-10-24T09:22:38Z

components/prism-cil.js

+
+	'keyword': /\b(iant|currency|syschar|void|bool|int8|int16|int32|int64|float32|float64|error|unsigned int8|unsigned int16|unsigned int32|unsigned int64|nativeType|decimal|date|bstr|lpstr|lpwstr|lptstr|objectref|iunknown|idispatch|struct|interface|int|unsigned int|nested struct|byvalstr|ansi bstr|tbstr|variant bool|lpstruct|static|public|private|family|final|specialname|virtual|abstract|assembly|default|cil managed|famandassem|famorassem|privatescope|hidebysig|newslot|rtspecialname|unmanagedexp|reqsecobj|pinvokeimpl|value|enum|interface|sealed|abstract|auto|sequential|explicit|ansi|unicode|autochar|import|serializable|nested public|nested private|nested family|nested assembly|nested famandassem|nested famorassem|beforefieldinit|specialname|rtspecialname|class|instance|string|extern|extends)\b/,
+
+	'function': /\b(add|add\.ovf|add\.ovf\.un|and|arglist|beq|beq\.s|bge|bge\.s|bge\.un|bge\.un\.s|bgt|bgt\.s|bgt\.un|bgt\.un\.s|ble|ble\.s|ble\.un|ble\.un\.s|blt|blt\.s|blt\.un|blt\.un\.s|bne\.un|bne\.un\.s|box|br|br\.s|break|brfalse|brfalse\.s|brinst|brinst\.s|brnull|brnull\.s|brtrue|brtrue\.s|brzero|brzero\.s|call|calli|callvirt|castclass|ceq|cgt|cgt\.un|ckfinite|clt|clt\.un|constrained\.|conv\.i|conv\.i1|conv\.i2|conv\.i4|conv\.i8|conv\.ovf\.i|conv\.ovf\.i\.un|conv\.ovf\.i1|conv\.ovf\.i1\.un|conv\.ovf\.i2|conv\.ovf\.i2\.un|conv\.ovf\.i4|conv\.ovf\.i4\.un|conv\.ovf\.i8|conv\.ovf\.i8\.un|conv\.ovf\.u|conv\.ovf\.u\.un|conv\.ovf\.u1|conv\.ovf\.u1\.un|conv\.ovf\.u2|conv\.ovf\.u2\.un|conv\.ovf\.u4|conv\.ovf\.u4\.un|conv\.ovf\.u8|conv\.ovf\.u8\.un|conv\.r\.un|conv\.r4|conv\.r8|conv\.u|conv\.u1|conv\.u2|conv\.u4|conv\.u8|cpblk|cpobj|div|div\.un|dup|endfault|endfilter|endfinally|initblk|initobj|isinst|jmp|ldarg|ldarg\.0|ldarg\.1|ldarg\.2|ldarg\.3|ldarg\.s|ldarga|ldarga\.s|ldc\.i4|ldc\.i4\.[0-9]+|ldc\.i4\.m1|ldc\.i4\.M1|ldc\.i4\.s|ldc\.i8|ldc\.r4|ldc\.r8|ldelem|ldelem\.i|ldelem\.i1|ldelem\.i2|ldelem\.i4|ldelem\.i8|ldelem\.r4|ldelem\.r8|ldelem\.ref|ldelem\.u1|ldelem\.u2|ldelem\.u4|ldelem\.u8|ldelema|ldfld|ldflda|ldftn|ldind\.i|ldind\.i1|ldind\.i2|ldind\.i4|ldind\.i8|ldind\.r4|ldind\.r8|ldind\.ref|ldind\.u1|ldind\.u2|ldind\.u4|ldind\.u8|ldlen|ldloc|ldloc\.[0-9]+|ldloc\.s|ldloca|ldloca\.s|ldnull|ldobj|ldsfld|ldsflda|ldstr|ldtoken|ldvirtftn|leave|leave\.s|localloc|mkrefany|mul|mul\.ovf|mul\.ovf\.un|neg|newarr|newobj|nop|not|or|pop|readonly\.|refanytype|refanyval|rem|rem\.un|ret|rethrow|shl|shr|shr\.un|sizeof|starg|starg\.s|stelem|stelem\.i|stelem\.i1|stelem\.i2|stelem\.i4|stelem\.i8|stelem\.r4|stelem\.r8|stelem\.ref|stfld|stind\.i|stind\.i1|stind\.i2|stind\.i4|stind\.i8|stind\.r4|stind\.r8|stind\.ref|stloc|stloc\.0|stloc\.1|stloc\.2|stloc\.3|stloc\.s|stobj|stsfld|sub|sub\.ovf|sub\.ovf\.un|switch|tail\.|throw|unaligned\.|alignment|unbox|unbox\.any|volatile\.|xor)\b/


Why do some end with .?

Please sort this list by descending length.
Because of the way alternations work and the fact that . is not included in \w (which is why \b won't help), some function names are not going to be matched correctly.
E.g.: add.ovf will be matched as [add].ovf.
This is because alternations will use the first one that matches (in this case add).

Please compress the list a little.
There is a lot of redundancy which can be used to make the list smaller.
E.g.: All functions which start with conv can be matched with the following pattern: conv\.(?:[iu][1248]?|ovf\.[iu][1248]?(?:\.un)?|r\.un|r4|r8).

Done. I originally pulled out of the Wikipedia article with a bit of JS and a regex or two to process it.

RunDevelopment · 2018-10-24T09:24:45Z

components.json

@@ -57,6 +57,10 @@
 			},
 			"option": "default"
 		},
+		"cil": {


You have to place this further down below (under C++, I think).
These here are special languages for Prism.

sbrl · 2018-10-24T12:42:12Z

All done! Is that better?

RunDevelopment · 2018-10-24T15:23:21Z

Looks good!

There are still a few things I noticed:

Please use non-capturing groups, if the group is not used as a lookbehind.
Again, please remove the package-lock.json and revert the changes made to prism.js. They simply do not belong to this PR.
Could you please consider adding some tests? It'd be great to have a test to cover all functions and keywords to verify that everything is matched correctly.

sbrl · 2018-10-24T19:28:33Z

I've made the changes requested (even though it's recommended that you commit package-lock.json).

How do I add tests? Is there any documentation on this?

RunDevelopment · 2018-10-24T19:41:16Z

Here's the documentation on how to add tests.

sbrl · 2018-10-26T19:44:33Z

Ah, thanks!

Looking at it, I think test cases are going to be challenging as I don't know the language well enough to create a valid cut-down program that would do the job as a test case. In addition, there's a lot of boilerplate code that's appears to be common across multiple IL files, which further complicates the process of writing tests.

Thoughts?

RunDevelopment · 2018-10-26T19:58:30Z

Prism test cases usually test specific features (e.g. numbers, comments, small aspects of a language in general). So you don't have to come up with a minimal program which includes all features. Just write small tests which test one feature each.

For examples on how to do tests right, I suggest looking at the JavaScript and C-like tests.

components/prism-cil.js

RunDevelopment · 2018-11-09T12:21:22Z

While playing around I also found a few minor things:

object, method (for overwrites), native (IntPtr), literal (consts and enums), valuetype (enums), runtime (delegates), and implements (interfaces) should be keywords.
try-catch/try-finally blocks don't have catch/finally highlighted.

sbrl · 2018-11-09T16:54:12Z

Thanks for the comments! PR updated. This is a much longer process than I thought it would be :P

RunDevelopment · 2018-11-09T18:45:37Z

@sbrl Looks good!

I found another missing keyword :) char. Also, all of the unsigned integers are missing, so uint and all uint<N>.
(And please update the keyword test to include the newly added ones.)

Don't worry, this is going to be the last nit.

sbrl · 2018-11-13T13:15:17Z

Done, @RunDevelopment

RunDevelopment · 2018-11-13T13:58:13Z

Looks good! Tests pass. @mAAdhaTTah I think we are ready to merge this.

sbrl · 2018-11-26T22:46:17Z

What's the status on this please, @RunDevelopment? I thought I'd done everything required, but it's been a while and this hasn't been merged yet? Did I miss something?

RunDevelopment · 2018-11-27T00:00:16Z

@sbrl You've done everything we asked of and you definitely have my OK.
It's just that @mAAdhaTTah seems to be quite busy at the moment and hasn't had the time to look at changes himself.

So don't worry! This has not been forgotten.

mAAdhaTTah · 2018-11-27T00:40:48Z

@RunDevelopment @sbrl I haven't forgotten! Holidays are busy :). I want to take a pass through Prism's PRs sometime this week(end) so I'll be taking a closer look at this soon.

mAAdhaTTah · 2018-11-28T13:30:10Z

@sbrl If you can resolve the conflict (just merge up from master & rerun gulp), I can merge this.

sbrl · 2018-11-28T17:07:23Z

Done, @mAAdhaTTah :D

mAAdhaTTah · 2018-11-28T17:45:39Z

@sbrl Thank you for the contribution!

Add support for the Common Intermediate Language from .NET.

sbrl added 2 commits October 23, 2018 19:10

cil: add language

9234e4f

Add display title. I think this is how it works?

ba405d3

sbrl added 3 commits October 23, 2018 21:15

Rename prism.cil.js to prism-cil.js

5a7a997

Add CIL to components.json

094c252

Run gulp

05da1c6

mAAdhaTTah reviewed Oct 23, 2018

View reviewed changes

RunDevelopment requested changes Oct 24, 2018

View reviewed changes

sbrl added 10 commits October 24, 2018 13:16

[CIL] Update comments regex

4e850ea

CIL: Remove displayTitle from language definition

c4267e0

CIL: Reorder lines

595d917

CIL: Move definition in components.json

6f77c0b

CIL: Update keywords list

a3ade2a

CIL: Improve number recognition regex

7f19702

CIL: Add additional punctuation

c9f9c33

CIL: Change function list

44206f0

CIL: Fix parse error

35616a1

Rerun gulp

a76a28d

sbrl added 3 commits October 24, 2018 20:20

Revert package-lock.json & prism.js

77d8318

CIL: Use non-capturing groups

6676327

Re-rerun gulp

f07575c

RunDevelopment reviewed Nov 9, 2018

View reviewed changes

components/prism-cil.js Outdated Show resolved Hide resolved

RunDevelopment reviewed Nov 9, 2018

View reviewed changes

components/prism-cil.js Outdated Show resolved Hide resolved

sbrl added 6 commits November 9, 2018 16:43

[CIL] Bugfix strings

48f7f86

[CIL] Add string test

0366685

[CIL] Fix directives by using a positive lookahead

706b0a8

[CIL] Bugfix assembly references

03b4258

[CIL] Add additional keywords

882f5c9

Re-run gulp

2758ec0

sbrl added 2 commits November 13, 2018 13:10

[CIL] Add char keyword

3eb0ed6

[CIL] Add tests for new keywords

89f49eb

RunDevelopment approved these changes Nov 15, 2018

View reviewed changes

Merge branch 'master' into patch-1

57c1fbe

mAAdhaTTah merged commit 38def33 into PrismJS:master Nov 28, 2018

sbrl deleted the patch-1 branch November 28, 2018 19:48

ggrossetie pushed a commit to ggrossetie/prism that referenced this pull request Mar 11, 2019

Add new language: CIL (PrismJS#1593)

e6494f1

Add support for the Common Intermediate Language from .NET.

This was referenced Oct 26, 2019

[Snyk] Upgrade prismjs from 1.8.1 to 1.17.1 ajesse11x/css-houdini.rocks#2

Open

[Snyk] Upgrade prismjs from 1.16.0 to 1.17.1 ajesse11x/svelte#2

Open

snyk-bot mentioned this pull request Mar 8, 2020

[Snyk] Upgrade prismjs from 1.10.0 to 1.19.0 tjenkinson/lit-html#5

Open

snyk-bot mentioned this pull request Mar 21, 2020

[Snyk] Upgrade prismjs from 1.14.0 to 1.19.0 bangbang93/freyja#143

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New language: CIL #1593

New language: CIL #1593

sbrl commented Oct 23, 2018

mAAdhaTTah commented Oct 23, 2018

sbrl commented Oct 23, 2018

mAAdhaTTah Oct 23, 2018

mAAdhaTTah commented Oct 23, 2018

RunDevelopment commented Oct 23, 2018

RunDevelopment left a comment

RunDevelopment Oct 24, 2018

RunDevelopment Oct 24, 2018

RunDevelopment Oct 24, 2018

RunDevelopment Oct 24, 2018

RunDevelopment Oct 24, 2018

RunDevelopment Oct 24, 2018

sbrl Oct 24, 2018

RunDevelopment Oct 24, 2018

sbrl Oct 24, 2018

RunDevelopment Oct 24, 2018

sbrl commented Oct 24, 2018

RunDevelopment commented Oct 24, 2018

sbrl commented Oct 24, 2018

RunDevelopment commented Oct 24, 2018

sbrl commented Oct 26, 2018

RunDevelopment commented Oct 26, 2018 •

edited

Loading

RunDevelopment commented Nov 9, 2018

sbrl commented Nov 9, 2018

RunDevelopment commented Nov 9, 2018

sbrl commented Nov 13, 2018

RunDevelopment commented Nov 13, 2018

sbrl commented Nov 26, 2018

RunDevelopment commented Nov 27, 2018 •

edited

Loading

mAAdhaTTah commented Nov 27, 2018

mAAdhaTTah commented Nov 28, 2018

sbrl commented Nov 28, 2018

mAAdhaTTah commented Nov 28, 2018

		@@ -0,0 +1,26 @@
		Prism.languages.cil = {
		'displayTitle': "CIL",


		'keyword': /\b(iant\|currency\|syschar\|void\|bool\|int8\|int16\|int32\|int64\|float32\|float64\|error\|unsigned int8\|unsigned int16\|unsigned int32\|unsigned int64\|nativeType\|decimal\|date\|bstr\|lpstr\|lpwstr\|lptstr\|objectref\|iunknown\|idispatch\|struct\|interface\|int\|unsigned int\|nested struct\|byvalstr\|ansi bstr\|tbstr\|variant bool\|lpstruct\|static\|public\|private\|family\|final\|specialname\|virtual\|abstract\|assembly\|default\|cil managed\|famandassem\|famorassem\|privatescope\|hidebysig\|newslot\|rtspecialname\|unmanagedexp\|reqsecobj\|pinvokeimpl\|value\|enum\|interface\|sealed\|abstract\|auto\|sequential\|explicit\|ansi\|unicode\|autochar\|import\|serializable\|nested public\|nested private\|nested family\|nested assembly\|nested famandassem\|nested famorassem\|beforefieldinit\|specialname\|rtspecialname\|class\|instance\|string\|extern\|extends)\b/,

		'function': /\b(add\|add\.ovf\|add\.ovf\.un\|and\|arglist\|beq\|beq\.s\|bge\|bge\.s\|bge\.un\|bge\.un\.s\|bgt\|bgt\.s\|bgt\.un\|bgt\.un\.s\|ble\|ble\.s\|ble\.un\|ble\.un\.s\|blt\|blt\.s\|blt\.un\|blt\.un\.s\|bne\.un\|bne\.un\.s\|box\|br\|br\.s\|break\|brfalse\|brfalse\.s\|brinst\|brinst\.s\|brnull\|brnull\.s\|brtrue\|brtrue\.s\|brzero\|brzero\.s\|call\|calli\|callvirt\|castclass\|ceq\|cgt\|cgt\.un\|ckfinite\|clt\|clt\.un\|constrained\.\|conv\.i\|conv\.i1\|conv\.i2\|conv\.i4\|conv\.i8\|conv\.ovf\.i\|conv\.ovf\.i\.un\|conv\.ovf\.i1\|conv\.ovf\.i1\.un\|conv\.ovf\.i2\|conv\.ovf\.i2\.un\|conv\.ovf\.i4\|conv\.ovf\.i4\.un\|conv\.ovf\.i8\|conv\.ovf\.i8\.un\|conv\.ovf\.u\|conv\.ovf\.u\.un\|conv\.ovf\.u1\|conv\.ovf\.u1\.un\|conv\.ovf\.u2\|conv\.ovf\.u2\.un\|conv\.ovf\.u4\|conv\.ovf\.u4\.un\|conv\.ovf\.u8\|conv\.ovf\.u8\.un\|conv\.r\.un\|conv\.r4\|conv\.r8\|conv\.u\|conv\.u1\|conv\.u2\|conv\.u4\|conv\.u8\|cpblk\|cpobj\|div\|div\.un\|dup\|endfault\|endfilter\|endfinally\|initblk\|initobj\|isinst\|jmp\|ldarg\|ldarg\.0\|ldarg\.1\|ldarg\.2\|ldarg\.3\|ldarg\.s\|ldarga\|ldarga\.s\|ldc\.i4\|ldc\.i4\.[0-9]+\|ldc\.i4\.m1\|ldc\.i4\.M1\|ldc\.i4\.s\|ldc\.i8\|ldc\.r4\|ldc\.r8\|ldelem\|ldelem\.i\|ldelem\.i1\|ldelem\.i2\|ldelem\.i4\|ldelem\.i8\|ldelem\.r4\|ldelem\.r8\|ldelem\.ref\|ldelem\.u1\|ldelem\.u2\|ldelem\.u4\|ldelem\.u8\|ldelema\|ldfld\|ldflda\|ldftn\|ldind\.i\|ldind\.i1\|ldind\.i2\|ldind\.i4\|ldind\.i8\|ldind\.r4\|ldind\.r8\|ldind\.ref\|ldind\.u1\|ldind\.u2\|ldind\.u4\|ldind\.u8\|ldlen\|ldloc\|ldloc\.[0-9]+\|ldloc\.s\|ldloca\|ldloca\.s\|ldnull\|ldobj\|ldsfld\|ldsflda\|ldstr\|ldtoken\|ldvirtftn\|leave\|leave\.s\|localloc\|mkrefany\|mul\|mul\.ovf\|mul\.ovf\.un\|neg\|newarr\|newobj\|nop\|not\|or\|pop\|readonly\.\|refanytype\|refanyval\|rem\|rem\.un\|ret\|rethrow\|shl\|shr\|shr\.un\|sizeof\|starg\|starg\.s\|stelem\|stelem\.i\|stelem\.i1\|stelem\.i2\|stelem\.i4\|stelem\.i8\|stelem\.r4\|stelem\.r8\|stelem\.ref\|stfld\|stind\.i\|stind\.i1\|stind\.i2\|stind\.i4\|stind\.i8\|stind\.r4\|stind\.r8\|stind\.ref\|stloc\|stloc\.0\|stloc\.1\|stloc\.2\|stloc\.3\|stloc\.s\|stobj\|stsfld\|sub\|sub\.ovf\|sub\.ovf\.un\|switch\|tail\.\|throw\|unaligned\.\|alignment\|unbox\|unbox\.any\|volatile\.\|xor)\b/

New language: CIL #1593

New language: CIL #1593

Conversation

sbrl commented Oct 23, 2018

mAAdhaTTah commented Oct 23, 2018

sbrl commented Oct 23, 2018

Choose a reason for hiding this comment

mAAdhaTTah commented Oct 23, 2018

RunDevelopment commented Oct 23, 2018

RunDevelopment left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sbrl commented Oct 24, 2018

RunDevelopment commented Oct 24, 2018

sbrl commented Oct 24, 2018

RunDevelopment commented Oct 24, 2018

sbrl commented Oct 26, 2018

RunDevelopment commented Oct 26, 2018 • edited Loading

RunDevelopment commented Nov 9, 2018

sbrl commented Nov 9, 2018

RunDevelopment commented Nov 9, 2018

sbrl commented Nov 13, 2018

RunDevelopment commented Nov 13, 2018

sbrl commented Nov 26, 2018

RunDevelopment commented Nov 27, 2018 • edited Loading

mAAdhaTTah commented Nov 27, 2018

mAAdhaTTah commented Nov 28, 2018

sbrl commented Nov 28, 2018

mAAdhaTTah commented Nov 28, 2018

RunDevelopment commented Oct 26, 2018 •

edited

Loading

RunDevelopment commented Nov 27, 2018 •

edited

Loading