refactor: Move JS parsing logic into JS language #18448

nzakas · 2024-05-13T21:45:09Z

Prerequisites checklist

I have read the contributing guidelines.

What is the purpose of this pull request? (put an "X" next to an item)

[ ] Documentation update
[ ] Bug fix (template)
[ ] New rule (template)
[ ] Changes an existing rule (template)
[ ] Add autofix to a rule
[ ] Add a CLI option
[x] Add something to the core
[ ] Other, please explain:

What changes did you make? (Give an overview)

This PR moves JS parsing and config handling logic into a new JS language object (/lib/languages/js/index.js), and includes:

Creation of the VFile class (implements the File interface from the RFC) and tests.
Creation of the JS language object (implements the Language interface). This deviates slightly from the RFC in that I've renamed "options" in all places to "languageOptions" to avoid confusion. Otherwise, I was constantly assigning back and forth between these two names.
Added language: "@/js" to the default config.
Moved languageOptions validation out of the default schema and into the JS language object.
Updated flat config array tests to verify all the relocated behavior works.
Created a generic languageOptionsSchema that does not validate the keys (that is now done in the JS language object) but preserves the merge behavior.

Refs #16999

Is there anything you'd like reviewers to focus on?

I think these changes are backwards-compatible, but would like another set of eyes.

languageOptionsSchema is the part I'm most concerned about. The previous merging behavior was very JS-specific, so I tried to come up with a generic version that would make sense for other languages. Please double-check that.

Refs eslint/rfcs#99

netlify · 2024-05-13T21:45:24Z

✅ Deploy Preview for docs-eslint canceled.

Name	Link
🔨 Latest commit	`ff2233c`
🔍 Latest deploy log	https://app.netlify.com/sites/docs-eslint/deploys/6656125e0f64380008253c46

mdjermanovic · 2024-05-20T09:54:40Z

Sorry for the delay, I plan to review this this week.

aladdin-add · 2024-05-22T04:53:34Z

lib/languages/js/index.js

+//-----------------------------------------------------------------------------
+
+const debug = createDebug("eslint:languages:js");
+const DEFAULT_ECMA_VERSION = 5;


to confirm: DEFAULT_ECMA_VERSION set to 5, for eslintrc-compat?

That's correct. The eslintrc branch of logic will still use this file, so want to cover our bases.

lib/languages/js/index.js

lib/languages/js/validate-language-options.js

lib/languages/js/source-code/source-code.js

mdjermanovic · 2024-05-22T12:31:50Z

lib/languages/js/source-code/source-code.js

+        // We only need to do this for ESTree-compatible ASTs
+        if (!this.isESTree) {
+            return;
+        }


This might be a breaking change because this code was executed for non-estree ASTs.

Interesting. Without this, several tests failed, so I thought we were skipping logic. I'll take another look.

Okay, I think I understand what's happening.

Those tests use fixture parsers that return undefined or an invalid AST, and then new SourceCode() crashes while trying to process it.

They were passing before these changes because new SourceCode() was inside the same try-catch as parser.parse(), so when this happens, it ends up being treated as a parsing error - it produces a fatal lint error for the file, but doesn't crash linting.

But now, new SourceCode() isn't in a try-catch, so when parser returns something we can't process, it crashes the whole linting. So I think the first question is: should we wrap language.createSourceCode() in a try-catch to not crash the whole linting when parser returns an invalid result for a single file? That would retain the current behavior, though it can be argued that the linting should crash when the parser is malfunctioning (throwing an error when the passed code isn't parsable is expected behavior; returning undefined or an invalid AST is not expected behavior).

Ah good call. I think it makes sense to crash in this instance because it's not actually a parsing error. The parsing succeeded but for some reason creating the SourceCode instance failed. I'd expect this to be caught by plugin developers before they release and most end-users would not end up seeing thing. What do you think?

lib/linter/linter.js

lib/languages/js/index.js

lib/linter/linter.js

lib/languages/js/validate-language-options.js

lib/config/flat-config-schema.js

lib/languages/js/validate-language-options.js

Co-authored-by: Milos Djermanovic <milos.djermanovic@gmail.com>

mdjermanovic · 2024-05-25T19:41:44Z

@christian-bromann looks like there's a problem in WebdriverIO:

[0-0] RUNNING in chrome - file:///tests/lib/linter/linter.js
[0-0]  Error:  Test failed due to following error(s):
  - downloadFile.js?v=74a71965: Uncaught SyntaxError: The requested module '/node_modules/jszip/dist/jszip.min.js?v=74a71965' does not provide an export named 'default'

christian-bromann · 2024-05-27T04:00:26Z

@mdjermanovic I will take a look tomorrow.

christian-bromann · 2024-05-28T06:12:56Z

@mdjermanovic please re-run the tests, it should pass now.

mdjermanovic · 2024-05-28T06:51:10Z

@christian-bromann now the test fails with:

Execution of 1 workers started at 2024-05-28T06:17:14.648Z

[0-0] RUNNING in chrome - file:///tests/lib/linter/linter.js
[0-0]  Error:  Test failed due to following error(s):
  - linter.js: Timed out after 300s waiting for test results

[0-0] FAILED in chrome - file:///tests/lib/linter/linter.js

 "concise" Reporter:
========= Your concise report ==========
chrome-headless-shell (v125.0.6422.60) on linux
❌ Failed to setup tests, no tests found


Spec Files:	 0 passed, 1 failed, 1 total (100% completed) in 00:05:03

christian-bromann · 2024-05-28T16:07:58Z

@mdjermanovic all test started to run extremely slow, I will investigate as to why.

mdjermanovic · 2024-05-30T11:43:24Z

lib/linter/linter.js

+        // augment global scope with declared global variables for ESTree only
+        if (sourceCode.isESTree) {
+            addDeclaredGlobals(
+                sourceCode.scopeManager.scopes[0],
+                configuredGlobals,
+                { exportedVariables: commentDirectives.exportedVariables, enabledGlobals: commentDirectives.enabledGlobals }
+            );
+        }


Similar to #18448 (comment).

I think we should rather update the tests that are failing without this change to return a valid AST.

mdjermanovic · 2024-05-30T11:44:40Z

lib/languages/js/source-code/source-code.js

-        addDeclaredGlobals(globalScope, configGlobals, inlineGlobals);
+        /*
+         * For languages other than regular JavaScript, it's possible that there
+         * is no scope analysis done, and therefore, `globalScope` might be undefined.
+         */
+        if (globalScope) {
+            addDeclaredGlobals(globalScope, configGlobals, inlineGlobals);
+        }


Also similar to #18448 (comment), and I also think we should rather update the tests to return a valid AST.

mdjermanovic · 2024-05-30T12:08:45Z

tests/lib/linter/linter.js

+            const ast = {
+                tokens: [],
+                comments: [],
+                loc: {},
+                range: []
+            };
+            const parseSpy = { parse: sinon.fake.returns(ast) };


Suggested change

const ast = {

tokens: [],

comments: [],

loc: {},

range: []

};

const parseSpy = { parse: sinon.fake.returns(ast) };

const parseSpy = { parse: sinon.spy(espree.parse) };

We could use espree here to return a valid AST.

mdjermanovic · 2024-05-30T12:13:49Z

tests/lib/linter/linter.js


                    it("should have file path passed to it", () => {
                        const code = "/* this is code */";
-                        const parseSpy = { parse: sinon.spy() };
+                        const parseSpy = { parse: sinon.fake.returns(ast) };


Suggested change

const parseSpy = { parse: sinon.fake.returns(ast) };

const parseSpy = { parse: sinon.spy(espree.parse) };

Same here.

mdjermanovic · 2024-05-30T12:14:25Z

tests/lib/linter/linter.js

+                    const ast = {
+                        tokens: [],
+                        comments: [],
+                        loc: {},
+                        range: []
+                    };


Suggested change

const ast = {

tokens: [],

comments: [],

loc: {},

range: []

};

Then, this wouldn't be needed.

mdjermanovic · 2024-05-30T12:15:39Z

tests/lib/eslint/eslint.js

+                                return {
+                                    comments: [],
+                                    tokens: [],
+                                    type: "Root",
+                                    loc: {},
+                                    range: []
+                                };


Suggested change

return {

comments: [],

tokens: [],

type: "Root",

loc: {},

range: []

};

return espree.parse(text, parserOptions);

We could use espree to return a valid AST here as well (just need to add const espree = require("espree"); too).

nzakas · 2024-05-31T20:51:00Z

@christian-bromann any update on these wdio failures? We've not had a test pass in several days and it's becoming an issue when reviewing PRs.

nzakas added 8 commits May 13, 2024 17:33

feat: Implement language plugins

dc55410

Refs eslint/rfcs#99

Add JS language index file stub

bec614d

Refactor to use language parser

18d9372

Add language to flat config

0f55912

Fix up linter errors

eab2edd

Add VFile class

4bcfe7a

Hook up validation and esquery

af3b4db

Update flat config schema to account for new languageOptions

f41d72e

nzakas requested a review from a team as a code owner May 13, 2024 21:45

eslint-github-bot bot added the chore This change is not user-facing label May 13, 2024

github-actions bot added the core Relates to ESLint's core APIs and features label May 13, 2024

nzakas added 5 commits May 14, 2024 10:51

Fix linting errors

ab1c1ae

Add visitorKeys property per RFC

8ca8ef2

Remove unused eslint-utils

394cf46

Fix test failures

5307af5

Ensure columnStart and lineStart are honored

1d97ea8

aladdin-add added the accepted There is consensus among the team that this change meets the criteria for inclusion label May 16, 2024

aladdin-add reviewed May 22, 2024

View reviewed changes

mdjermanovic reviewed May 22, 2024

View reviewed changes

lib/linter/linter.js Outdated Show resolved Hide resolved

mdjermanovic reviewed May 22, 2024

View reviewed changes

lib/linter/linter.js Show resolved Hide resolved

mdjermanovic reviewed May 22, 2024

View reviewed changes

lib/linter/linter.js Outdated Show resolved Hide resolved