Command line parser testcase

[cameel]

Abstract

Define a new TestCase to replace the boost::test-based CommandLineParserTest. The test case should for the most part work like the one defined in #17013, just exercising a single component rather than a full compilation pipeline.

An additional output for inspecting the parsed options should be available (.cliOptions).

Motivation

boost::test test cases are flexible but cumbersome. They're somewhat verbose and don't have an expectation update mechanism like isoltest. They're also more tedious to review. We are generally replacing them with custom isoltest TestCases.

Specification

The new test case should be distinct from the one in #17013 but share most of the underlying functionality through a common base class. This includes:

• Defining commands
• Multiple commands in one test
• Parametrization
• Stdin support
• Comment support
• .exit, .stderr, .stdout outputs
• Filters

Expected differences:

• The testcase defined here does not run the full pipeline. It runs only CommandLineParser and stops once it obtains CommandLineOptions struct. We want to test a single component rather than the whole compiler end-to-end.
  • The existence or non-existence of paths used in commands should not impact success.
  • Even in Standard JSON mode we stop before running StandardCompiler. This means that we cannot access its InputsAndSettings. We might provide a separate test case for this in the future.
  • The .exit, .stderr, .stdout outputs are available but may be incomplete compared to a full run. Especially errors that would only happen at later stages will not be present.
• CommandLineOptions is exposed as an additional output: .cliOptions.
• We do not need any of the settings from CLI/Standard JSON testcase #17013 (standardJSON, implicitOptions, stripPrereleaseWarning, stripLicenseWarning, stripVersionWarning).
• New setting: showUnchangedCliOptions (bool). false by default. Unless enabled, fields that are equal to their defaults are not included in .cliOptions.

Examples

This shows what standard_json_mode_options would look like with such a test case.

Without unchanged options

solc input.json
    --standard-json
    --base-path=/home/user/
    --include-path=/usr/lib/include/
    --include-path=/home/user/include
    --allow-paths=/tmp,/home,project,../contracts
    --ignore-missing
    --output-dir=/tmp/out             # Accepted but has no effect in Standard JSON mode
    --overwrite                       # Accepted but has no effect in Standard JSON mode
    --evm-version=spuriousDragon      # Ignored in Standard JSON mode
    --revert-strings=strip            # Accepted but has no effect in Standard JSON mode
    --pretty-json
    --json-indent=1
    --no-color                        # Accepted but has no effect in Standard JSON mode
    --error-codes                     # Accepted but has no effect in Standard JSON mode
    --libraries=dir1/file1.sol:L=0x1234567890123456789012345678901234567890,dir2/file2.sol:L=0x1111122222333334444455555666667777788888 # Ignored in Standard JSON mode
    --gas                             # Accepted but has no effect in Standard JSON mode
    --combined-json=abi,bin           # Accepted but has no effect in Standard JSON mode
// ----
// .cliOptions: {
//     "input": {
//         "mode": "standard-json",
//         "paths": ["input.json"],
//         "basePath": "/home/user/",
//         "includePaths": ["/usr/lib/include/", "/home/user/include"],
//         "allowedDirectories": ["/tmp", "/home", "project", "../contracts"],
//         "ignoreMissingFiles": true
//     },
//     "output": {
//         "dir": "/tmp/out",
//         "overwriteFiles": true,
//         "revertStrings": "strip"
//     },
//     "formatting": {
//         "json": {"pretty": true, "indent": 1},
//         "coloredOutput": false,
//         "withErrorIds": false
//     },
//     "compiler": {
//         "estimateGas": true,
//         "combinedJsonRequests": ["abi", "bin"]
//     }
// }
// .cliOptions.assembly.targetMachine: <PATH NOT FOUND>

With unchanged options

solc input.json
    --standard-json
    --base-path=/home/user/
    --include-path=/usr/lib/include/
    --include-path=/home/user/include
    --allow-paths=/tmp,/home,project,../contracts
    --ignore-missing
    --output-dir=/tmp/out             # Accepted but has no effect in Standard JSON mode
    --overwrite                       # Accepted but has no effect in Standard JSON mode
    --evm-version=spuriousDragon      # Ignored in Standard JSON mode
    --revert-strings=strip            # Accepted but has no effect in Standard JSON mode
    --pretty-json
    --json-indent=1
    --no-color                        # Accepted but has no effect in Standard JSON mode
    --error-codes                     # Accepted but has no effect in Standard JSON mode
    --libraries=dir1/file1.sol:L=0x1234567890123456789012345678901234567890,dir2/file2.sol:L=0x1111122222333334444455555666667777788888 # Ignored in Standard JSON mode
    --gas                             # Accepted but has no effect in Standard JSON mode
    --combined-json=abi,bin           # Accepted but has no effect in Standard JSON mode

// ====
// showUnchangedCliOptions: true
// ----
// .cliOptions: {
//     "input": {
//         "mode": "standard-json",
//         "paths": ["input.json"],
//         "remappings": [],
//         "addStdin": false,
//         "basePath": "/home/user/",
//         "includePaths": ["/usr/lib/include/", "/home/user/include"],
//         "allowedDirectories": ["/tmp", "/home", "project", "../contracts"],
//         "ignoreMissingFiles": true,
//         "noImportCallback": false
//     },
//     "output": {
//         "dir": "/tmp/out",
//         "overwriteFiles": true,
//         "evmVersion": "osaka",
//         "viaIR": false,
//         "viaSSACFG": false,
//         "revertStrings": "strip",
//         "debugInfoSelection": null,
//         "stopAfter": "compilation-successful"
//     },
//     "assembly": {
//         "targetMachine": "evm"
//     },
//     "linker": {
//         "libraries": {}
//     },
//     "formatting": {
//         "json": {"pretty": true, "indent": 1},
//         "coloredOutput": false,
//         "withErrorIds": false
//     },
//     "compiler": {
//         "outputs": [],
//         "estimateGas": true,
//         "combinedJsonRequests": ["abi", "bin"]
//     },
//     "metadata": {
//         "format": "with-release-version-tag",
//         "hash": "ipfs",
//         "literalSources": false
//     },
//     "optimizer": {
//         "optimizeEvmasm": false,
//         "optimizeYul": false;
//         "expectedExecutionsPerDeployment": null,
//         "yulSteps": null
//     },
//     "modelChecker": {
//         "initialize": false,
//         "settings": {
//              "bmcLoopIterations": null;
//              "contracts": {},
//              "divModNoSlacks": false;
//              "engine": [],
//              "externalCalls": "untrusted",
//              "invariants": [],
//              "printQuery": false,
//              "showProvedSafe": false,
//              "showUnproved": false,
//              "showUnsupported": false,
//              "solvers": ["z3"],
//              "targets": ["constantCondition", "divByZero", "balance", "assert", "popEmptyArray", "outOfBounds"],
//              "timeout": null
//         }
//     },
//     "experimental": false
// }

[Oguidan]

Hi @cameel , I would like to learn how you are willing to resolve this with you