Update 5.0.3

vs-code-extension/CHANGELOG.md

@@ -1,9 +1,34 @@
 
+## 5.0.3
+Fixed:
+
+  - **Extension failed to start** in certain scenarios.
+  - **First start without a configured post executable** blocked VS Code's extension host for minutes. It now returns at once.
+  - **A folder on a network share or another disk that cannot be reached** (Custom folder or a folder added with **Add folder**) can no longer block VS Code.
+
+Added:
+
+  - **Post Properties: machine defaults**: when you select a machine that stores post property defaults, the extension offers to apply them. These properties show a machine icon and "machine default", and **Reset** returns to the machine default. New title bar button **Apply property defaults from the selected machine**.
+
+Added QA test:
+
+  - **QA test view** in the Explorer: run your posts with a set of test cases (CNC inputs, properties and machine) and compare the output with the expected output. Turn it on with **Enable QA testing** or the `AutodeskPostUtility.qaTestEnabled` setting.
+  - **Guided test case editing**: add, duplicate and remove test cases, edit properties, input CNC files, machine, description, output suffix and additional post engine flags, ignore a test case or skip the default properties from `common.json`.
+  - **Save Post Properties view settings as new test case**: turns the properties you changed in the Post Properties view into a test case.
+  - **Run QA tests** for the current post, selected posts, a folder or all posts with test cases, also from the Explorer context menu on `.cps` files and folders. Tests run in parallel.
+  - **Results** per post, test case and output file with status icons: success, output differs, failure, missing input, new (no expected file), ignored. **Compare with expected** (built-in diff or the external diff tool), **Accept as expected**, **Accept all new and differing outputs**, **Run failed and changed tests again** and **Show only problems**.
+  - **Machine post properties in QA tests**: properties stored in the `.mch` file of a test case are used when the post defines them with machine scope. Order: `defaultProperties` in `common.json`, then the machine, then the test case. Machine properties that the post does not define, or defines with another scope, are listed as notes in the QA test log.
+  - **QA test log** with coloured results.
+  - **Command line runner** `out/qaRun.js`: runs the QA tests without a VS Code window and writes `output/qa-report.md` and `output/qa-report.json`.
+
+---
 ## 5.0.2
 Fixed:
+
   - **Click to jump**: clicking in the `.cps` file no longer makes the cursor jump around.
   - **Machine Selector**: Machine names now always display the `.mch` filename instead of vendor + model, preventing duplicate entries.
   - **Post Properties**: Changing a property in the `.cps` file no longer resets other user-modified property values back to their defaults.
+
 ---
 ## 5.0.0 Major Update
 - **Recently used** files for CNC and Machines
@@ -24,112 +49,142 @@ Fixed:
 - **Automatic refresh** of CNC/Machine Selector list, no need anymore to refresh the content manually.
 - **Post Processor engine version** is now always shown in the status bar.
 - **Show Properties by group** setting displays properties in the Properties selector organized according to their respective groups.
+
 ---
 ## 4.1.7
 Fixed:
+
   - The 'Merge post processor' feature does now utilize the include path if specified
   - Fixed an issue where ESLint did not recognize the embedded rules file.
 Updated:
+
   - Updated embedded ESLint rules file.
 ## 4.1.6
 Changed:
+
   - Option 'Download CNC exporting post processor' now redirects to the Fusion post library website
 ## 4.1.5
 Added:
+
   - Added option to use Autodesk's ESLint rules for editing postprocessors
 ## 4.1.4
 Updated:
+
   - Added new CNC files
   - Added a turnmill machine
 ## 4.1.3
 Fixed:
+
   - Fixed broken link to help file
 Changed:
+
   - Updated readme
 
 ## 4.1.2
 Fixed:
+
   - Fixed a bug that caused the extension to stop working
 
 ## 4.1.1
 Added:
+
   - Added option to show warning messages generated by the postprocessor
 
 ## 4.0.0
 Added:
+
   - Major refactoring
   - Support for .mch (machine V2)
   - Pre-release of new G-code debugger
 
 
 ## 3.5.8
 Fixed:
+
   - Issues with missing function
 
 ## 3.5.6
 Added:
+
   - Merge post feature
   - Option to show the current post processor engine version in use
   - Option to post process automatically when a machine file is selected
 Changed:
+
   - Updated generic milling machine configuration files
 
 ##3.5.5
 Added: 
+
   - Post encryption
 Fixed: 
+
   - Settings issue causing the extension to become unresponsive
 
 ##3.5.4
 Fixed:
+
   - Fixed issue in the 'Update post processor properties' function
 
 ##3.5.3
 Added:
+
   - Added option to import custom machine configuration files
 	- Added option to open/edit machine configuration files directly from the menu
 
 Fixed:
+
   - Fixed potential issue in the 'Update post processor properties' function
 
 ##3.5.2
 Fixed:
+
   - Issue where the post executable was not found
 
 ##3.5.1
 Fixed:
+
   - Minor bug fixes
 
 ##3.5.0
 Added:
+
   - Added the ability to update old style post properties to the new post property format
 
 ##3.4.0
 Fixed:
+
   - Errors with properties
   - Auto finding post.exe on mac not working
 Removed:
+
   - Post API list as the web API manual now has search
 
 ##3.3.3
 Fixed:
+
   - Error log not showing
 Added:
+
   - Ability to deselect a machine (right click the machine in the left panel)
 
 ##3.3.2
 Fixed:
+
   - Recursive folder creation bug
 
 ##3.3.1
 Added:
+
   - Error handling
 
 ##3.3.0
 Added:
+
   - Support for new property definitions
   - Add folder option to custom CNC files
 Fixed:
+
   - Bug fixes
   - Clear output folder prior to posting
   - Remove debug lines from all files (including subprograms)
@@ -143,66 +198,82 @@ Fixed:
 
 ##3.2.0
 Added:
+
   - Machine selection
   - Additive test files
   - Tool descriptions to CNC files
 
 ##3.1.0
 Added:
+
   - GCode highlighting
 Fixed:
+
   - Multiple issues with auto line selection
 
 ##3.0.0
 Added:
+
   - Support for MAC
 Fixed:
+
   - Output window not reloading
 
 ##2.3.3
 Fixed:
+
   - Line jumping no longer works in the latest version of VSCode
 
 ##2.3.1
 Fixed:
+
   - Unable to post process with the latest version of VSCode
 
 ##2.3.0
 Added:
+
   - Option to enable auto complete for the active document (F1 > Enable auto complete for current document)
 
 ## 2.2.1
 Added:
+
   - Import option for CNC files
 
 ## 2.2.0
 Updated:
+
   - Extension naming
 
 ## 2.1.0
 Added:
+
   - A new explorer sidebar containing post properties
 
 ## 2.0.6
 Fixed:
+
   - line selection only working on the first instance of the stack trace
 
 ## 2.0.5
 Added:
+
   - Added some logic to determine the Fusion install location and automatically select the post.exe location
 
 ## 2.0.4
 Updated:
+
   - Modified export CPS file to support new VSCode update
 
 ## 2.0.0
 Added:
+
   - Option for two click line jumping
   - Open folder right click option to CNC tree
   - Auto focus on a line which causes an error (if possible)
   - Auto clear notifications
 
 Fixed:
+
   - Peck drilling CNC file
   - Coolant CNC files
 

vs-code-extension/README.md

@@ -32,6 +32,9 @@ http://cam.autodesk.com/posts
 The documentation of the postprocessor API can be found here:
 https://cam.autodesk.com/posts/reference/index.html
 
+# What's New in Autodesk Post Processor Utility 5.0.3
+- **QA test view**: run your posts with a set of test cases and compare the output with the expected output. See [QA test usage guide](#qa-test-usage-guide).
+- **Post Properties: machine defaults**: when you select a machine that stores post property defaults, the extension offers to apply them. These properties show a machine icon and "machine default" in the Post Properties view, and **Reset** returns to the machine default. Use the machine button in the view title to apply them again.
 # What's New in Autodesk Post Processor Utility 5.0.0
 
 ### Machine Selector
@@ -75,6 +78,97 @@ https://cam.autodesk.com/posts/reference/index.html
 ![propertyGroups](https://github.com/Autodesk/cam-posteditor/raw/master/vs-code-extension/res/help/propertiesByGroup.gif)
 
 
+# QA test usage guide
+QA tests run your post with a set of test cases (CNC inputs, properties and machine) and compare the output with the expected output. Use them to check that a change in the post only changes what you intended.
+
+### Turn on the QA test view
+Run **Enable QA testing** from the Command Palette, or turn on the setting `AutodeskPostUtility.qaTestEnabled`. The **QA test** view appears in the Explorer.
+
+### Create a QA project
+A QA project is a folder with this layout:
+
+| Folder | Content |
+|---|---|
+| `test_cases` | Test case files (JSON), for example `test_cases/milling/<post name>.json`, and `common.json` with the defaults for all test cases |
+| `cnc` | Input CNC files, in subfolders such as `cnc/milling` |
+| `machine` | Machine files used by test cases |
+| `expected` | Expected outputs |
+| `output` | Outputs of the last run (not meant for version control) |
+
+- **Create QA project...** creates this layout next to your posts (`tests` folder) or in a folder you choose.
+- **Use existing QA project folder...** uses a folder you already have, for example a post-library `tests` folder.
+- Without a setting, the QA project is found automatically: the extension looks for a `test_cases` folder, or a `tests` folder with `test_cases`, next to or above the open post. `AutodeskPostUtility.qaProjectFolder` sets it explicitly.
+
+### Add test cases
+Open a post and click **Add test case** in the QA test view. The guide asks for:
+1. A short description.
+2. The input folder inside `cnc`.
+3. The input CNC files. Files from outside the project are copied into it.
+4. A machine (optional). Machines from outside the project are copied into it.
+5. An output suffix, so test cases of the same post do not overwrite each other. Leave it empty for the main test case.
+
+Then choose the properties: the post defaults, the properties you changed in the Post Properties view, or pick them one by one. **Save Post Properties view settings as new test case** does the same in one step.
+
+Right-click a test case to edit its properties, inputs, machine, description, suffix or additional post engine flags, to ignore it, to skip the default properties from `common.json`, or to duplicate, remove or open it as JSON.
+
+A test case file looks like this:
+```json
+{
+  "posts": [
+    {
+      "name": "my post",
+      "testCases": [
+        {
+          "description": "Standard operations",
+          "inputFolder": "milling",
+          "input": ["2D Milling", "Drilling Cycles"],
+          "suffix": "",
+          "properties": { "useSmoothing": true },
+          "machine": "tests/machine/my machine.mch"
+        }
+      ]
+    }
+  ]
+}
+```
+- `input` names the CNC files in `cnc/<inputFolder>` without the `.cnc` extension.
+- `machine` is relative to the folder that contains the QA project.
+- Each input is run in inch and metric (setting `AutodeskPostUtility.qaUnits`), unless the test case sets the `unit` property. The output is `output/<inputFolder>/<post>/<post>_<input><suffix>_<IN|MM>.output`, and the expected file has the same path in `expected`.
+
+### Properties
+Properties are applied in this order, later ones win:
+1. `defaultProperties` in `common.json` (unless the test case skips them).
+2. Post properties stored in the test case's `.mch` machine file, but only those the post defines with machine scope. Machine properties that the post does not define, or defines with another scope, are listed as notes in the QA test log.
+3. The test case `properties`.
+
+### Run tests and review results
+- **Run QA tests** runs the tests of the current post. The **...** menu of the view runs selected posts, a folder or all posts with test cases. You can also right-click a `.cps` file or a folder in the Explorer.
+- Each post, test case and output file shows its result: success, output differs, failure, missing input, new (no expected file) or ignored. Results show **changed - run again** when the post, a CNC file, the machine or the test case changed after the run.
+- Click a differing output to **Compare with expected**. The external diff tool setting of the regression test is used here as well.
+- **Accept as expected** copies the output to `expected`. **Accept all new and differing outputs** does this for the whole run. Only accept outputs you have checked.
+- When the inch and metric outputs of a test changed in the same way, the second one shows **same change as ...**, so you only need to review one of them.
+- **Run failed and changed tests again** and **Show only problems** help with large runs. **Show QA test log** opens the full log.
+
+Test case options that need other tools, such as checking simulation results, are skipped and shown as such.
+
+### Settings
+| Setting | Meaning |
+|---|---|
+| `AutodeskPostUtility.qaTestEnabled` | Show the QA test view and commands |
+| `AutodeskPostUtility.qaProjectFolder` | QA project folder. Leave empty to find it automatically |
+| `AutodeskPostUtility.qaPostEngine` | Post engine for the tests. Leave empty to use the Fusion post engine |
+| `AutodeskPostUtility.qaUnits` | Units to test: both, MM or IN |
+| `AutodeskPostUtility.qaParallel` | Tests run at the same time. 0 chooses automatically |
+| `AutodeskPostUtility.qaTimeout` | Time limit in seconds for one post processing run |
+| `AutodeskPostUtility.qaStopOnFailure` | Stop the run at the first failing test |
+
+### Run QA tests from a terminal
+`out/qaRun.js` in the extension folder runs the QA tests without a VS Code window, for example in scripts:
+```
+node qaRun.js [<QA project folder | folder with posts | post.cps>] [--units both|MM|IN] [--engine <post engine>] [--parallel N] [--timeout seconds] [--settings <VS Code settings.json>]
+```
+It reads the QA settings from your VS Code user settings, writes `output/qa-report.md` and `output/qa-report.json` and never accepts outputs. Exit code: 0 all passed, 1 problems or new outputs to review, 2 setup error.
+
 # About
 Autodesk Fusion Post Processor Utility for Visual Studio Code © 2017 Autodesk, Inc.  All rights reserved.