Skip to content

NeuralOpsAI_Report.md

Latest commit

 

History

History
175 lines (140 loc) · 8.57 KB

NeuralOpsAI_Report.md

File metadata and controls

175 lines (140 loc) · 8.57 KB
Raw

NeuralOpsAI Project Report (May 15, 2026)

1) Purpose and scope

NeuralOpsAI is an open-source MLOps platform that aims to cover the full ML lifecycle: data ingestion and versioning, pipeline orchestration, model training and evaluation, model registry, AutoML, and operational monitoring. It is implemented as a microservices system with a Go-based backend and a React dashboard. The platform targets teams that want a unified, cloud-native stack for reproducible ML workflows with strong lineage and operational visibility.

Primary references used for this report:

2) What the project is for (functional goals)

  • Natural language to pipeline creation and orchestration, using a central orchestrator that compiles workflows for execution.
  • Data lake management with Iceberg and Nessie for versioned datasets and lineage.
  • Model lifecycle management with versioning and promotion workflows.
  • AutoML for automated model selection and hyperparameter tuning.
  • Observability via dashboard pages for workflows, models, pipelines, and data catalog.

3) Architecture overview

3.1 High-level system diagram

flowchart LR
  user[User]
  ui[Web Dashboard]
  gateway[API Gateway]

  orchestrator[Orchestrator]
  ai[AI Engine]
  iceberg[Iceberg Manager]
  registry[Model Registry]
  automl[AutoML Service]
  responder[Incident Responder]
  optimizer[Resource Optimizer]
  runner[Workflow Runner]

  pg[(PostgreSQL)]
  minio[(MinIO)]
  nessie[(Nessie)]
  argo[(Argo Workflows)]

  user --> ui --> gateway
  gateway --> orchestrator
  gateway --> ai
  gateway --> iceberg
  gateway --> registry
  gateway --> automl
  gateway --> responder
  gateway --> optimizer
  gateway --> runner

  orchestrator --> argo
  ai --> minio
  ai --> pg
  iceberg --> minio
  iceberg --> nessie
  registry --> pg
  registry --> minio
  automl --> pg
Loading

3.2 Pipeline and data flow (natural language to production)

sequenceDiagram
  participant User
  participant UI as Dashboard
  participant GW as API Gateway
  participant Orc as Orchestrator
  participant Argo as Argo Workflows
  participant AI as AI Engine
  participant Ice as Iceberg Manager
  participant Reg as Model Registry
  participant Store as Storage

  User->>UI: Describe pipeline in natural language
  UI->>GW: POST /v1/pipelines:nl
  GW->>Orc: gRPC request
  Orc->>Argo: Submit workflow
  Argo->>AI: Execute training tasks
  AI->>Ice: Read/write dataset versions
  AI->>Reg: Register model and metadata
  Reg->>Store: Persist artifacts and metadata
Loading

4) Current implementation status (from code and docs)

  • Documentation presents a complete v2.0 stack with dashboard, AutoML, model registry, data versioning, and workflow visualization.
  • The codebase contains the dashboard UI and service entry points, but several critical runtime behaviors are placeholders.
  • A referenced improvements document is not present in the workspace; this creates a doc gap for contributors and users.

5) Open-source impact

Positive impact so far:

  • Provides a clear reference architecture for a modern, cloud-native MLOps system built around open standards (Iceberg, Argo, gRPC).
  • Lowers the barrier for teams to experiment with end-to-end MLOps features in a single repo.
  • Encourages modular contribution because services are separated, with APIs defined via protobuf and OpenAPI.
  • Demonstrates how to combine data lineage, orchestration, and model registry into one workflow.

Limits on impact today:

  • Several critical services are scaffolds without persistence and production-grade behavior, which limits real-world adoption.
  • Missing documentation for improvements and feature verification can slow onboarding for contributors.

6) TODO tasks found in the codebase

These are the explicit TODO markers present today and should be treated as prioritized implementation gaps.

Area TODO Location Suggested implementation direction
Iceberg Manager Implement optimize logic (compaction) cmd/iceberg-manager/iceberg_grpc_server.go Call Iceberg table maintenance actions; update Nessie metadata; add metrics and backpressure.
Data Versioning Create Iceberg snapshots internal/services/data_version_service.go Use Nessie APIs to commit Iceberg snapshots; store snapshot metadata.
Data Versioning Retrieve a specific version internal/services/data_version_service.go Query Nessie for the dataset ref and snapshot metadata.
Data Versioning List dataset versions internal/services/data_version_service.go List commits/tags from Nessie and map to versions.
Data Versioning Store lineage metadata internal/services/data_version_service.go Persist lineage in a metadata store (PostgreSQL or object storage).
Data Versioning Retrieve lineage metadata internal/services/data_version_service.go Query lineage store and return typed lineage graph.
Data Versioning Compare versions internal/services/data_version_service.go Compute schema and data diffs from Iceberg snapshots.
AutoML Service Initialize tuning engine cmd/automl-service/main.go Integrate Ray Tune or Optuna with a pluggable backend.
AutoML Service Register gRPC service cmd/automl-service/main.go Wire AutoML server implementation and DI.
Model Registry Initialize repository cmd/model-registry/main.go Add PostgreSQL repository, migrations, and connection config.
Model Registry Register gRPC service cmd/model-registry/main.go Wire registry server implementation and DI.
Dashboard Call API for NL pipeline generation web/dashboard/src/pages/PipelineBuilder.tsx Call gateway endpoint and map response to nodes/edges.

7) Recommended improvements and further implementation

7.1 Platform completeness (highest priority)

  • Implement missing service logic and persistence for AutoML, Model Registry, and Data Versioning.
  • Add database migrations and repository layers; unify config loading for service connections.
  • Ensure gRPC services are registered and wired to the API Gateway.

7.2 Reliability and observability

  • Standardize tracing, metrics, and structured logs across services (OpenTelemetry).
  • Add health and readiness probes for all services; align with Kubernetes manifests.
  • Add circuit breakers and retries for service-to-service communication.

7.3 Security and governance

  • Add authentication and authorization at the gateway and service level.
  • Add RBAC for dataset/model access and audit logging.
  • Add secure secrets handling for storage and database credentials.

7.4 Developer experience

  • Fill the documentation gap for improvements and contribution workflows.
  • Provide example datasets and sample pipelines for quick validation.
  • Add unit, integration, and end-to-end tests for critical flows.

7.5 Product enhancements

  • Add model artifact validation and automatic promotion rules.
  • Add workflow cost tracking and resource budgets.
  • Add lineage visualization in the dashboard using dataset-version graph data.

8) Suggested phased roadmap

Phase 1 (stabilize core services):

  • Implement the TODOs listed in Section 6.
  • Add PostgreSQL migrations for Model Registry and AutoML metadata.
  • Connect dashboard pipeline builder to the NL pipeline API.

Phase 2 (operational hardening):

  • Add auth and RBAC policies.
  • Add tracing and metrics dashboards.
  • Add automated end-to-end tests for pipeline creation and execution.

Phase 3 (ecosystem and community):

  • Improve documentation and provide upgrade guides.
  • Add community contribution templates and sample integrations.
  • Provide a sandbox mode for contributors (single-node/minikube).

9) Key risks and gaps

  • Several services are scaffolds without persistence or business logic, which limits production readiness.
  • Documentation references a missing improvements document, which can confuse contributors.
  • API Gateway and service wiring may be incomplete relative to the documented feature set.