diff --git a/en/docs/install-and-setup/install/adding-observability-for-icp.md b/en/docs/install-and-setup/install/adding-observability-for-icp.md index b0464bdb4..ad8141d63 100644 --- a/en/docs/install-and-setup/install/adding-observability-for-icp.md +++ b/en/docs/install-and-setup/install/adding-observability-for-icp.md @@ -1,553 +1,21 @@ # Add Centralized Observability in the Integration Control Plane -ICP provides centralized observability for MI runtimes. Application logs and per-request analytics are collected via Fluent Bit, stored in OpenSearch, and displayed in the ICP Console. +ICP supports Moesif and OpenSearch for viewing logs and metrics from connected MI runtimes. Choose an observability provider and follow its setup guide. -## Architecture - -ICP observability architecture diagram showing log and metrics flow from MI through Fluent Bit to OpenSearch and ICP Console - -1. MI writes application logs to `wso2carbon.log` with an `[icp.runtimeId=]` suffix on each line. -2. MI writes per-request analytics to `synapse-analytics.log` as JSON lines prefixed with `SYNAPSE_ANALYTICS_DATA`. -3. Fluent Bit tails both files and ships each to its own OpenSearch index. -4. ICP Server queries OpenSearch (filtering by runtimeId) when a user opens Logs or Metrics in the Console. - -## Prerequisites - - - - - - - - - - - - - - - - - - - - - - - - - - -
ComponentPurpose
OpenSearch 2.xLog and metrics storage
Fluent Bit 3.xLog collection and forwarding
ICP Server 2.0.0+Observability API layer
MI serverRuntime with ICP heartbeat and analytics support
- -## Step 1: Deploy OpenSearch - -Any single-node or clustered OpenSearch deployment works. ICP needs HTTP(S) access to the OpenSearch REST API. - -For detailed OpenSearch setup and deployment instructions, refer to the [OpenSearch Installation Guide](https://opensearch.org/docs/latest/install-and-configure/install-opensearch/index/). - - -Note the host, port, and credentials — you will configure them in ICP Server and Fluent Bit. - -## Step 1.1: Deploy Fluent Bit - -ICP requires Fluent Bit **3.x** for log collection and forwarding to OpenSearch. - -For detailed Fluent Bit setup and deployment instructions, refer to the [Fluent Bit Installation Guide](https://docs.fluentbit.io/manual/installation/getting-started-with-fluent-bit). - -Ensure your Fluent Bit instance has: -- Network connectivity to both the MI runtime (to tail log files) and OpenSearch (to forward logs) -- Appropriate permissions to read MI log files -- Sufficient disk space and memory for buffering - -After deployment, you will configure Fluent Bit to tail MI's log files in Step 5. - -## Step 2: Create Index Templates - -Apply index templates before any data arrives to ensure correct field mappings. - -### Application logs template - -The `time` field must accept the timestamp format MI produces (`2026-04-29 12:01:22,874`). Add a custom date format alongside the defaults: - -```bash -curl -X PUT ':9200/_index_template/wso2_mi_application_log_template' \ - -H 'Content-Type: application/json' \ - -d '{ - "index_patterns": ["mi-application-logs-*"], - "template": { - "mappings": { - "properties": { - "time": { - "type": "date", - "format": "yyyy-MM-dd HH:mm:ss,SSS||strict_date_optional_time||epoch_millis" - }, - "message": { "type": "text" }, - "icp_runtimeId": { "type": "keyword" } - } - } - } - }' -``` - -!!! note "Testing only" - The flags below are suitable for local testing. For production, use a dedicated OpenSearch user with minimal permissions, provide a trusted CA certificate (`--cacert `), and avoid disabling SSL verification. - - - If your OpenSearch requires authentication, add `-u admin:`. - - For HTTPS with self-signed certs, add `-k` (skips SSL verification — not recommended for production). - -### Metrics index — no template needed - -**Do not** create an explicit template for `mi-metrics-logs-*`. OpenSearch's dynamic mapping will automatically map `icp_runtimeId` as `text` with a `.keyword` subfield, which is required by ICP Server metrics queries. Applying an explicit template will interfere with this automatic mapping. - -!!! note - The app logs template must map `icp_runtimeId` as `keyword` (ICP queries it with a bare `terms` filter). The metrics index must keep the dynamic mapping (`text` + `.keyword` subfield) because ICP's metrics aggregations use `icp_runtimeId.keyword`. Applying an explicit `keyword` mapping to the metrics index will break the Metrics page. - -## Step 3: Configure ICP Server - -Add the OpenSearch connection to `/conf/deployment.toml`. - -```toml -opensearchUrl = "https://localhost:9200" -opensearchUsername = "admin" -opensearchPassword = "" -``` +| Provider | Metrics collection | Application log collection | Setup | +|----------|--------------------|----------------------------|-------| +| **Moesif** | Fluent Bit transforms MI analytics records and publishes them to Moesif as actions. | A separate Fluent Bit sidecar forwards `wso2carbon.log` to Moesif. | [Set up Moesif](adding-observability-for-icp/moesif.md) | +| **OpenSearch** | Fluent Bit forwards MI analytics records to OpenSearch. | Fluent Bit forwards structured application logs to OpenSearch. | [Set up OpenSearch](adding-observability-for-icp/opensearch.md) | !!! note - Place the above config at the top of the file **before any `[section]` header** (lines starting with `[`). - -!!! warning "Testing and Development Only" - If OpenSearch is running without TLS (e.g. with the security plugin disabled), you can use `http://` **only for testing and development**: - - ```toml - opensearchUrl = "http://localhost:9200" - ``` - - For production deployments, always use HTTPS with proper TLS certificates and strong authentication credentials. Refer to [OpenSearch Security Best Practices](https://opensearch.org/docs/latest/security/index/) for configuring authentication, authorization, and encryption in production environments. - -Restart ICP Server after saving. Look for this log line to confirm: - -```text -level=INFO module=wso2/icp_server message="OpenSearch client initialized successfully" -``` - -## Step 4: Configure MI - -Three changes are needed in the MI runtime. - -### 1. Enable analytics in `deployment.toml` - -```toml -[mediation] -flow.statistics.enable=true -flow.statistics.capture_all=true - -[analytics] -enabled = true -``` - -### 2. Connect to ICP - -See [Connect MI Integration to ICP](connecting-an-integration-to-icp.md) for the full procedure. The minimum config: - -```toml -[icp_config] -enabled = true -icp_url = "https://:9445" -environment = "dev" -project = "my-project" -integration = "my-integration" -secret = "." -``` - -### 3. Route analytics to a separate log file - -Add a dedicated log4j2 appender so Fluent Bit can tail `synapse-analytics.log` independently of `wso2carbon.log`. - -In `/conf/log4j2.properties`: - -**Register the appender** — add `SYNAPSE_ANALYTICS_LOGFILE` to the `appenders` line: - -```properties -appenders = SYNAPSE_ANALYTICS_LOGFILE, CARBON_CONSOLE, CARBON_LOGFILE, ... -``` - -**Register the logger** — add `ElasticStatisticsPublisher` to the `loggers` line: - -```properties -loggers = ElasticStatisticsPublisher, AUDIT_LOG, SERVICE_LOGGER, ... -``` - -**Append the icp.runtimeId suffix** to `CARBON_LOGFILE.layout.pattern` so the Fluent Bit parser can extract it: - -{% raw %} -```properties -appender.CARBON_LOGFILE.layout.pattern = [%d] %5p {%c} - %m %ex${sys:icp.runtime.log.suffix:-}%n -``` -{% endraw %} - -**Add the appender and logger definitions** at the end of the file: - -{% raw %} -```properties -# Synapse analytics → separate file for Fluent Bit -appender.SYNAPSE_ANALYTICS_LOGFILE.type = RollingFile -appender.SYNAPSE_ANALYTICS_LOGFILE.name = SYNAPSE_ANALYTICS_LOGFILE -appender.SYNAPSE_ANALYTICS_LOGFILE.fileName = ${sys:logfiles.home}/synapse-analytics.log -appender.SYNAPSE_ANALYTICS_LOGFILE.filePattern = ${sys:logfiles.home}/synapse-analytics-%d{MM-dd-yyyy}.log -appender.SYNAPSE_ANALYTICS_LOGFILE.layout.type = PatternLayout -appender.SYNAPSE_ANALYTICS_LOGFILE.layout.pattern = [%d] %5p {%c} - %m %ex${sys:icp.runtime.log.suffix:-}%n -appender.SYNAPSE_ANALYTICS_LOGFILE.policies.type = Policies -appender.SYNAPSE_ANALYTICS_LOGFILE.policies.size.type = SizeBasedTriggeringPolicy -appender.SYNAPSE_ANALYTICS_LOGFILE.policies.size.size = 10MB -appender.SYNAPSE_ANALYTICS_LOGFILE.strategy.type = DefaultRolloverStrategy -appender.SYNAPSE_ANALYTICS_LOGFILE.strategy.max = 5 - -logger.ElasticStatisticsPublisher.name = org.wso2.micro.integrator.analytics.messageflow.data.publisher.publish.elasticsearch.ElasticStatisticsPublisher -logger.ElasticStatisticsPublisher.level = INFO -logger.ElasticStatisticsPublisher.appenderRef.SYNAPSE_ANALYTICS_LOGFILE.ref = SYNAPSE_ANALYTICS_LOGFILE -logger.ElasticStatisticsPublisher.additivity = false -``` -{% endraw %} + ICP stores one Moesif Management API Key per environment. Linking either the metrics or logs canvas configures both canvases for every integration in that environment. - - - - - - - - - - - - - - - - - -
SettingPurpose
${sys:icp.runtime.log.suffix:-}Appends [icp.runtimeId=<uuid>] to each log line. The :- default produces nothing before the ID is initialized during startup.
additivity = falsePrevents analytics lines from also appearing in wso2carbon.log.
- -This produces two log files: - - - - - - - - - - - - - - - - - - - - - -
FileContentOpenSearch index
repository/logs/wso2carbon.logApplication logs (startup, errors, mediations)mi-application-logs-*
repository/logs/synapse-analytics.logPer-request analytics (latency, entity type, status)mi-metrics-logs-*
- -## Step 5: Configure Fluent Bit - -Fluent Bit tails both MI log files and ships them to OpenSearch. - -### Parsers - -MI's default log4j2 `[%d]` pattern produces timestamps like `2026-04-29 12:01:22,874` (space-separated date/time, comma before milliseconds). The parser `Time_Format` must match this exactly. - -```ini -# parsers.conf - -[PARSER] - Name mi_log_parser - Format regex - Regex ^(?:TID:\s*)?\[(?