Skip to content

[DRAFT] In house JsonSchema to TS generator - #1382

Open
JuanGalilea wants to merge 21 commits into
masterfrom
json-schema-to-ts-in-house
Open

JuanGalilea wants to merge 21 commits into
masterfrom
json-schema-to-ts-in-house

Conversation

@JuanGalilea

@JuanGalilea JuanGalilea commented Sep 2, 2026 •

Copy link
Copy Markdown
Contributor

base generator and tons of testing added.

  • jsonschema to ts compiler
  • tests
  • review for unexpected regressions
  • add to actor generate-schema-types
  • improve tests to check generation better
  • add perspective or some user | actor flag
  • improve diagnostics collection
  • fix kvstore generation
  • run preprocessors before compilation
  • add --check flag

Install size change (node scripts/report-install-size.mjs)

Metric This release Latest (1.10.0) Change
Tarball download 0.36 MB 0.35 MB +4.1%
Unpacked package 1.40 MB 1.36 MB +3.1%
Full install (with dependencies) 46.28 MB 58.84 MB -21.3%

AI /show-me below

Replaces the json-schema-to-typescript dependency with an in-house schema→TypeScript compiler (src/lib/schema-to-ts/, zero runtime deps — only node:crypto), and reframes the CLI flag surface around a single --perspective.

src/lib/
├── schema-transforms.ts          # DELETED (-225): makePropertiesRequired, clearAllRequired, stripTitles…
└── schema-to-ts/                 # NEW: ~950 lines, no runtime deps
    ├── index.ts                  # narrow public surface (compile, check, normalize*, readHeader)
    ├── preprocess/               # the only Apify-aware layer
    │   ├── input.ts              #   nullable rewrite only
    │   ├── dataset.ts            #   unwrap `fields`, drop views/actorSpecification
    │   ├── kvstore.ts            #   collections → one closed lookup object
    │   └── nullable.ts           #   {type:'x', nullable:true} → {type:['x','null']}
    ├── parser.ts                 # JSON Schema → IR  + diagnostics
    ├── ir.ts                     # 9 structural node kinds, IR_VERSION = 1
    ├── emit.ts                   # IR → TS text, `type` never `interface`
    ├── canonical.ts              # sorted serialization (order-insensitive)
    ├── hash.ts                   # sha256, 16 hex chars
    ├── check.ts                  # header regex → drift detection
    ├── compile.ts                # facade: compile() / check()
    └── diagnostics.ts            # error | warning | notice codes

The pipeline

compile(schema, { types: [{name, variant}], unknownRoot? })
  jsonSchemaToIR(schema)                    parser.ts — only place IR is built
    readTypes()                             `type` is MANDATORY (no inference)
    enumToNode()                            enum ⨯ type cross-narrowing
    fromObject() → readRequired/readProps/readAdditional
    → { ir, diagnostics, notices }
  emit(ir, opts)
    header(irHash(ir, opts))                canonical(ir) → sha256 → 16 hex
    renderNode() per requested type name
  → { source, diagnostics, notices }

Two orthogonal signals come out: diagnostics = fidelity lost (something became unknown), notices = schema lint with no type impact. Neither feeds the hash.

The supplied / received split is the conceptual core — same IR, two renderings:

 renderObject(node, variant)
-  optional = !required                       // supplied: writer's obligation
+  optional = !required && !hasDefault        // received: platform materialized defaults
-  if (open) → `{...} & Record<string, unknown>`   // supplied accepts extras
+  // received is closed: unknown key = typo

Drift detection

// @generated schema-ts v1-d1bb2b62dfb769b2 — do not edit
                        │  └── sha256(canonical(IR) + options)[0..16]
                        └── IR_VERSION, compared before the hash

canonical.ts sorts props/enum members while emit.ts reproduces authored order — so reordering keys in a schema never reports drift. Header is matched anywhere in the file (gm), so a user's prepended banner is free; two headers → duplicate-header error.

CLI surface change

 apify actor generate-schema-types [path]
   -o, --output=<dir>
-  --strict            (default true)
-  --all-optional      (default false)
+  --perspective=actor|user   (default actor)

Behavior changes worth a reviewer's attention

Before After
export interface Input export type input — lowercase, from const name = 'input'
Banner with biome-ignore-all, eslint-disable, prettier-ignore-start header comment only — no lint/format suppression in generated files
Empty dataset fields → skip with warning writes export type dataset = Record<string, unknown>;
KVS: one interface per collection, concatenated one closed object, collection name → record type
prettier formatting via the library raw emitter output (4-space indent, Array<T>, parenthesized intersections)

@JuanGalilea JuanGalilea self-assigned this Sep 2, 2026
@JuanGalilea
JuanGalilea force-pushed the json-schema-to-ts-in-house branch 2 times, most recently from 9c2467e to 1406406 Compare September 9, 2026 11:58
@JuanGalilea
JuanGalilea marked this pull request as ready for review September 9, 2026 15:14
@JuanGalilea
JuanGalilea force-pushed the json-schema-to-ts-in-house branch from 024469a to fd4d2f0 Compare October 20, 2026 13:51
@l2ysho
l2ysho requested a review from apify-factory October 22, 2026 20:27

@apify-factory apify-factory left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good work, one thing to fix before merge.

🏭 Factory review — ℹ️ not auto-approved

This change was reviewed but cannot be auto-approved: a gate failed, so the review cannot end in an approval.

check result
Static gates ❌ 8/16 — author-allowed, max-files, max-lines, file-extensions, deny-globs, patch-present, no-risky-content, pr-title
Review findings ❌ not clean
Reviewer escalation ❌ Replaces a dependency with a new in-house compiler and breaks the public CLI and generated-file contract (removes --strict/--all-optional, renames exported types from Input interfaces to lowercase input types, changes the KVS output shape).
Juror 1 — claude-sonnet-5-5, adversarial ⏭️ skipped — a gate failed
Why it was not auto-approved

Failed gates:

  • author-allowed: JuanGalilea: not a member of apify/product-engineering
  • max-files: 54 changed files, limit 10
  • max-lines: 22661 changed lines (+21733/-928), limit 200
  • file-extensions: disallowed extension: .vscode/settings.json, package.json, pnpm-lock.yaml, test/setup/output-schemas/valid.json, test/local/fixtures/lib/schema-to-ts/dataset/comment-scraper.json, test/local/fixtures/lib/schema-to-ts/dataset/google-places.json, test/local/fixtures/lib/schema-to-ts/dataset/tiktok-followers-scraper.json, test/local/fixtures/lib/schema-to-ts/input/api-scraper.json, test/local/fixtures/lib/schema-to-ts/input/comment-scraper.json, test/local/fixtures/lib/schema-to-ts/input/free-amazon-product-scraper.json, test/local/fixtures/lib/schema-to-ts/input/google-places.json, test/local/fixtures/lib/schema-to-ts/kitchen-sink.json, test/local/fixtures/lib/schema-to-ts/kvstore/maps-to-polygon.json, test/tsconfig.json
  • deny-globs: package.json matches **/package.json; pnpm-lock.yaml matches **/pnpm-lock.yaml
  • patch-present: no text diff for test/local/fixtures/lib/schema-to-ts/expected/input/google-places.ts, test/local/fixtures/lib/schema-to-ts/input/google-places.json
  • no-risky-content: test/local/lib/schema-to-ts/hash.test.ts: credential-like assignment (credential-assignment); test/local/lib/schema-to-ts/parser.test.ts: credential-like assignment (credential-assignment); test/local/lib/schema-to-ts/preprocess/input.test.ts: credential-like assignment (credential-assignment)
  • pr-title: title "[DRAFT] In house JsonSchema to TS generator" must match type(scope): message

The comments above are the review itself; this section only explains the approval decision. Push a change and request @apify-factory again to re-run it, or ask a human to review.

Reviewed 99faebbca · workflow run

label: 'output',
outputDir,
fileName: 'output.ts',
schema: outputSchema,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Important: the output schema now goes to compile raw, and the parser treats a missing type as an error that becomes unknown, so an output schema without a root "type": "object" (like the valid.json fixture before this PR had to add it) now generates export type output = unknown; and still exits 0. The deleted prepareOutputSchemaForCompilation (and the dataset/KVS preparers) defaulted type to 'object', so this is a regression for Actors that worked before; the same thing happens to dataset fields and KVS jsonSchema that leave out type.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants