[llvm-lit] Enable lit internal shell by default

[connieyzhu]

This is a meta-issue that addresses the problem referenced in [RFC] Enabling the Lit Internal Shell by Default.

Brief problem summary:
Many RUN line commands used in LLVM tests are written in a way that only works on POSIX-compliant shells, leading to issues on platforms like Windows or non-POSIX systems. To overcome these discrepancies, lit has an internal shell implementation designed to ensure uniform execution of these commands, regardless of the platform. Although the internal shell supports most common commands, it still lacks several capabilities that are necessary for it to become the default shell when running LLVM tests.

This meta-issue tracks the following issues:
Subprojects:

• Enable internal shell for llvm test suite

Missing Capabilities Blocking Enablement:

• [llvm-lit] TypeError in lit’s built-in cat -v implementation #102374
  [llvm-lit] The lit internal shell doesn’t support env command without any arguments #102383
  [llvm-lit] Environment variable not found in LLVM subproject with lit internal shell  #102395
  [llvm-lit] The env command without args doesn't support redirection with lit internal shell #106627

• Enable internal shell for clang test suite

Missing Capabilities Blocking Enablement:

• [llvm-lit] Environment variable not found in clang subproject with lit internal shell #106601
  [llvm-lit] lit internal shell failing to parse and execute command substitution syntax #102384
  [llvm-lit] Unhashable TypeError 'GlobItem' with lit’s internal shell #102389
  [llvm-lit] Lit's built-in diff command causing clang test to fail when run with internal shell #106249

• Enable internal shell for lld test suite

Missing Capabilities Blocking Enablement:

• [llvm-lit] Environment variable not found in LLD subproject with lit internal shell #106594
  [llvm-lit] lit internal shell failing to parse and execute curly brace syntax #102382

• Enable internal shell for lldb test suite

Missing Capabilities Blocking Enablement:

• [llvm-lit] lit internal shell does not support -e option for cat commands #102377
  [llvm-lit] No such file or directory error in lldb tests with lit internal shell #102696

• Enable internal shell for BOLT test suite

Missing Capabilities Blocking Enablement:

• [llvm-lit] Unsupported |& redirection syntax in lit internal shell #102388
  [llvm-lit] lit internal shell fail to execute parentheses syntax #102401
  [llvm-lit] TypeError: string argument expected in BOLT when testing with lit internal shell #102693
  [llvm-lit] Attribute Error in lit internal shell #102399

• Enable internal shell for compiler-rt test suite

Missing Capabilities Blocking Enablement:

• [llvm-lit] Environment variable not found in compiler-rt subproject with lit internal shell  #106598
  [llvm-lit] Ulimit command not found in lit internal shell #102398
  [llvm-lit] Unset command not found in lit internal shell #102397
  [llvm-lit] Expected string not found in input error in compiler-rt when testing with lit internal shell #102695
  [llvm-lit] InternalShellError in compiler-rt test using export command #102386
  [llvm-lit] Attribute Error in lit internal shell #102399
  [llvm-lit] Unhashable TypeError 'GlobItem' with lit’s internal shell #102389
  [llvm-lit] lit internal shell failing to parse and execute for-loop syntax #102380
  [llvm-lit] Unsupported redirect in compiler-rt test using lit internal shell #102385
  [llvm-lit] lit internal shell failing to parse and execute command substitution syntax #102384
  [llvm-lit] lit internal shell fail to execute parentheses syntax #102401
  [llvm-lit] lit internal shell failing to parse and execute curly brace syntax #102382
  [llvm-lit][compiler-rt] Test using 'export' failing to execute with lit internal shell due to too many arguments #106142
  [llvm-lit] Expand and Rewrite Tests for FileNotFoundError for compiler-rt tests in lit internal shell #106111

• Enable internal shell for clang-tools-extra test suite

Missing Capabilities Blocking Enablement:

• [llvm-lit] Lit's built-in cat command not handling carriage returns correctly #106302

• Enable internal shell for polly test suite

• [mlir][polly][llvm-lit] External shell unable to be turned on for lit tests #106459

• Enable internal shell for mlir test suite (already the default)

• [mlir][polly][llvm-lit] External shell unable to be turned on for lit tests #106459

• Enable internal shell for flang test suite
• Enable internal shell for libc test suite
• Enable internal shell for libcxx test suite (already the default)
• Enable internal shell for libunwind test suite (already the default)
• Enable internal shell for libcxxabi test suite (already the default)

This is a comprehensive list of all the features to be implemented.
Feature List:

• [llvm-lit] TypeError in lit’s built-in cat -v implementation #102374
  [llvm-lit] The lit internal shell doesn’t support env command without any arguments #102383
  [llvm-lit] Environment variable not found in LLVM subproject with lit internal shell  #102395
  [llvm-lit] Environment variable not found in LLD subproject with lit internal shell #106594
  [llvm-lit] Environment variable not found in compiler-rt subproject with lit internal shell  #106598
  [llvm-lit] Environment variable not found in clang subproject with lit internal shell #106601
  [llvm-lit] lit internal shell failing to parse and execute command substitution syntax #102384
  [llvm-lit] Unhashable TypeError 'GlobItem' with lit’s internal shell #102389
  [llvm-lit] Expected string not found in input error in compiler-rt when testing with lit internal shell #102695
  [llvm-lit] lit internal shell failing to parse and execute curly brace syntax #102382
  [llvm-lit] lit internal shell does not support -e option for cat commands #102377
  [llvm-lit] No such file or directory error in lldb tests with lit internal shell #102696
  [llvm-lit] Ulimit command not found in lit internal shell #102398
  [llvm-lit] Unset command not found in lit internal shell #102397
  [llvm-lit] Unsupported |& redirection syntax in lit internal shell #102388
  [llvm-lit] lit internal shell fail to execute parentheses syntax #102401
  [llvm-lit] TypeError: string argument expected in BOLT when testing with lit internal shell #102693
  [llvm-lit] Attribute Error in lit internal shell #102399
  [llvm-lit] InternalShellError in compiler-rt test using export command #102386
  [llvm-lit] lit internal shell failing to parse and execute for-loop syntax #102380
  [llvm-lit] Unsupported redirect in compiler-rt test using lit internal shell #102385
  [llvm-lit][compiler-rt] Test using 'export' failing to execute with lit internal shell due to too many arguments #106142
  [llvm-lit] Lit's built-in diff command causing clang test to fail when run with internal shell #106249
  [llvm-lit] Lit's built-in cat command not handling carriage returns correctly #106302
  [mlir][polly][llvm-lit] External shell unable to be turned on for lit tests #106459
  [llvm-lit] The env command without args doesn't support redirection with lit internal shell #106627

This issue will help track the progress of enabling the lit internal shell across all LLVM subprojects.

[connieyzhu]

@ldionne Hi there, it seems like libcxx in LLVM uses lit's internal shell as the default shell for running lit tests. Can you confirm if this is accurate?

[ldionne]

@ldionne Hi there, it seems like libcxx in LLVM uses lit's internal shell as the default shell for running lit tests. Can you confirm if this is accurate?

Yes, that's right. We made the transition some time ago (maybe 1-3 years) and haven't looked back.