Request for tips on contributing a new language definition

[GrahamHannington]

I've started developing a Prism language definition for IBM z/OS MVS JCL.

I'm already pretty happy with the results. Thank you!

Here's the work-in-progress definition:

Prism.languages.jcl = {
  "operation": {
    pattern: /^(\/\/[^\*\s]\S*\s+)\S+/mg,
    lookbehind: true
  },
  "name": {
    pattern: /^(\/\/)[^\*\s]\S*/mg,
    lookbehind: true
  },
  "comment": [ /^\/\/\*.*$/mg, /^\*.*$/mg ],
  "attr-name": /[^\s,]+(?==)/g,
  "punctuation": [ /[,'"=]/, /^\/\//mg, /^\/\*/mg ]
}

CSS:

.token.name {
  color: #DD4A68;
}
.token.operation {
  color: #07a;
}

I'd gratefully accept tips or advice on the above. I've been referring to the MDN documentation on JavaScript regular expressions.

I'm looking ahead to the possibility of submitting a pull request to add this language definition to the public Prism repo. I've done some research on how to do this:

• I've read the tips in the readme under the heading "Contribute to Prism!"
• I've read the topic "Extending Prism"
• I've browsed (some of) the source

I get that I would need to:

• Add two new files, /components/prism-jcl.js and /components/prism-jcl.min.js
• Update components.js with a new entry for this language

However, I'm unclear what else I need to do.

In particular, while I know I need to add the CSS rules for the new operation and name tokens I've coined for this language—to match the operation and name fields described in the JCL documentation (see the earlier link)—does that mean I need to add CSS rules to each of the theme-specific CSS files?

That question makes me wonder whether I should rethink coining those, or any, new tokens.

Should I coin new tokens?

Perhaps I shouldn't get hung up on using token names that match the terminology of the language; perhaps I should instead use existing token names defined by Prism, even if those names don't match JCL terminology.

Advice, thoughts, recommendations gratefully accepted. (Including: "This is the wrong place to ask such questions. Try asking a specific question on Stack Overflow and tagging it prism.js.")

[Golmote]

Hi!

I'd gratefully accept tips or advice on the above.

This looks pretty good to me. Note that you can remove all the g flags as Prism will not need them.
Also, the character * does not need to be escaped when it appears inside a character class. [^\*\s] can be written [^*\s].
I wonder if the $ is useful in .*$. All chars should consumed until the end of the line anyway, because of the default greediness of the * quantifier.
You might want to consider avoiding arrays in your definition unless you can't do without them. [ /^\/\/\*.*$/mg, /^\*.*$/mg ] can probably be written as /^(?:\/\/)?\*.*$/mg. Same thing for the punctuation feature.

However, I'm unclear what else I need to do.

You've listed the essential steps already. However, it's better if you can provide a PR that includes two more things:

• an example file (you can take a look at all the other ones to see the usual format)
• tests for every token you've defined (you can read the Test suite page for information, and look at the existing ones)

Should I coin new tokens?

Now that's a tough one. My answer should be "Yes, your classes must be semantically appropriate and you do need to update all themes accordingly indeed". Unfortunately, this is hardly maintainable (even more considering the non-official themes). We do have an open issue to address this problem (#1055) but haven't come with an appropriate solution yet.
For the time being, I would suggest you use the alias key to add additional classes to your tokens. That way, your tokens will still have one semantically appropriated class, but also another less-semantic class that will be used for the styling.
From the alias section in Extending Prism:

This can be useful, to combine the styling of a well known token, which is already supported by most of the themes, with a semantically correct token name.

[GrahamHannington]

Thanks very much for the informative reply.

I've edited the regexes as you suggest, except, so far, for the arrays: I'm going to cogitate over that. I'm torn between the readability (as far as regexes can be considered to be readable) of the arrays (matching distinctly different forms of syntax) versus performance.

Regarding coining new tokens: I've shown the styled results to a few experienced JCL users, and, while they like the idea of the highlighted syntax, they're not thrilled with the colors used for the "less-semantic" classes in the default Prism theme; in particular, red or reddish colors. Many JCL users will only ever have seen JCL syntax highlighted in the native z/OS ISPF text editor, as displayed in a terminal emulator such as IBM Personal Communications ("PCOMM"); to such users, other color schemes might look odd. For now, partly because I don't have much time to spend on this, for my local repo, I will continue to use bespoke, semantically appropriate class names for styling. Another reason: I want more variety. For example, JCL supplied with a product often includes "placeholder" values (to be replaced with actual values) that, by convention, are <enclosed in angle brackets> (similar to an XML start tag):

JS:

"placeholder": /<.+>/m,

CSS:

.token.placeholder {
  color: slategray;
  background-color: #FFFF90;
}

Finally, for now: I will admit this is entirely my problem, but I thought I'd share this "noob" experience with you...

I am not, primarily, a JavaScript developer. I have dipped into JavaScript every few years since the early web browsers (with increasing frequency, especially since node.js arrived). Web workers are new to me. This caused me some head-scratching when I added the JavaScript definition of the JCL language via a <script> element following the <script> element for prism.js. I am embarrassed to admit that it took me nearly an hour to realize that, since I had chosen to perform highlighting asynchronously, that meant using web workers, which in turn meant (if I understand this correctly; I might not) that my language definition needed to be included in prism.js. Before that realization, I spent some time questioning my own understanding of script execution order.

Why am I bothering telling you this? Perhaps in case other noobs report similar issues; perhaps, they, too, might be trying to put their own language definitions in a custom script, heeding—without fully understanding—the instructions about not editing prism.js, and then attempting to apply async highlighting. (Also: perhaps in case my explanation is entirely wrong!)

[Golmote]

since I had chosen to perform highlighting asynchronously, that meant [...] that my language definition needed to be included in prism.js.

You're right, the async highlighting uses the main prism.js file in a worker, and all languages must be included within it.

We usually consider most people will use the Download page to craft their own bundle of Prism, including all the languages they want to highlight. Doing this, they'll end up with a single file, indeed.

We might want to update the documentation somewhere to help new users address the issue you faced.