diff --git a/README.md b/README.md index 4c84c6ea..02ce1a59 100644 --- a/README.md +++ b/README.md @@ -78,12 +78,12 @@ Documentation is versioned for the **docs** and **graphql** sections. Versioned - `docs_versioned_docs/` and `docs_versioned_sidebars/` (and `docs_versions.json`) - `graphql_versioned_docs/` and `graphql_versioned_sidebars/` (and `graphql_versions.json`) -The folders `docs/` and `docs-graphql/` are the **“next”** (unreleased) content. They appear in the version dropdown as **“next”** only when running the site locally with `includeCurrentVersion: true` in `docusaurus.config.ts`. **When deployed, the site serves only versioned documentation** (e.g. v25.2, v25.1, v24.1); the “next” content is not shown on the live site. +The folders `docs/` and `docs-graphql/` are the **“next”** (unreleased) content. They appear in the version dropdown as **“next”** only when running the site locally with `includeCurrentVersion: true` in `docusaurus.config.ts`. **When deployed, the site serves only versioned documentation** (e.g. v25.4, v25.3, v25.2); the “next” content is not shown on the live site. **Where to edit:** - **For the next release:** Edit the page in `docs/` or `docs-graphql/`. Those are the only places that represent the upcoming release. -- **For the current release (what’s live):** Apply the same change in the corresponding versioned folder (e.g. `docs_versioned_docs/version-v25.2/...` or `graphql_versioned_docs/version-v25.2/...` for the latest). If a fix or clarification should be in the current release, duplicate the change there. +- **For the current release (what’s live):** Apply the same change in the corresponding versioned folder (e.g. `docs_versioned_docs/version-v25.4/...` or `graphql_versioned_docs/version-v25.4/...` for the latest). If a fix or clarification should be in the current release, duplicate the change there. **Note for reviewers:** The “Edit this page” link always points to the “next” page (`docs/` or `docs-graphql/`). When reviewing PRs, check whether the change should also be ported back to the current version’s versioned folder. diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/admin/admin-endpoints.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/admin/admin-endpoints.md new file mode 100644 index 00000000..a0c13410 --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/admin/admin-endpoints.md @@ -0,0 +1,87 @@ +--- +title: Admin Endpoints +--- + +The [Admin Tasks](./admin-tasks/index.md) section provides detailed guides for common administrative operations, but this page provides an exhaustive list of all administrative endpoints available in Dgraph. + +Dgraph provides administrative endpoints on both Alpha and Zero nodes. + +## Alpha HTTP Endpoints (port 8080) + +Dgraph Alpha exposes the following HTTP endpoints on port `8080` (plus optional port offset): + +- **`/admin/config/cache_mb`** - Configure cache size +- **`/admin/draining`** - Drain connections from a node +- **`/admin/shutdown`** - Shutdown a single Alpha node +- **`/alter`** - Apply schema updates and drop predicates +- **`/login`** - Authenticate ACL users +- **`/health`** - Health status +- **`/health?all`** - Health status of all servers in the cluster +- **`/state`** - Returns information about the nodes that are part of the cluster. This includes information about the size of predicates and which groups they belong to. + +## Zero HTTP Endpoints (port 6080) + +Dgraph Zero exposes the following HTTP endpoints on port `6080` (plus optional port offset): + +### GET Endpoints + +- **`/assign?what=uids&num=100`** - Allocates a range of UIDs specified by the `num` argument, and returns a JSON map containing the `startId` and `endId` that defines the range of UIDs (inclusive). This UID range can be safely assigned externally to new nodes during data ingestion. +- **`/assign?what=timestamps&num=100`** - Requests timestamps from Zero. This is useful to "fast forward" the state of the Zero node when starting from a postings directory that already has commits higher than Zero's leased timestamp. +- **`/removeNode?id=3&group=2`** - Removes a dead Zero or Alpha node. When a replica node goes offline and can't be recovered, you can remove it and add a new node to the quorum. To remove dead Zero nodes, pass `group=0` and the id of the Zero node to this endpoint. + +:::note +Before using the `/removeNode` endpoint, ensure that the node is down and ensure that it doesn't come back up ever again. Do not use the same `idx` of a node that was removed earlier. +::: + +- **`/moveTablet?tablet=name&group=2`** - Moves a tablet to a group. Zero already rebalances shards every 8 mins, but this endpoint can be used to force move a tablet. + +### POST Endpoints + +- **`/enterpriseLicense`** - Applies an enterprise license to the cluster by supplying it as part of the body. + +## Alpha GraphQL Admin Endpoints (port 8080) + +**`/admin`** - GraphQL endpoint for cluster management operations + +The GraphQL Admin API provides an alternative way to perform the same administrative tasks available through HTTP endpoints. Many operations that can be done via HTTP endpoints (such as export, backup, shutdown, draining, etc.) can also be performed using GraphQL queries and mutations. The GraphQL interface offers a more structured and type-safe approach to administrative operations. + +#### Queries + +- **`getGQLSchema`** - Get the current GraphQL schema +- **`health`** - Get health status +- **`state`** - Get cluster state +- **`config`** - Get node configuration +- **`task`** - Get task information +- **`getUser`** - Get a user by name +- **`getGroup`** - Get a group by name +- **`getCurrentUser`** - Get the currently logged in user +- **`queryUser`** - Query users with filters +- **`queryGroup`** - Query groups with filters +- **`listBackups`** - Get information about backups at a given location + +#### Mutations + +- **`updateGQLSchema`** - Update the Dgraph cluster to serve the input schema. This may change the GraphQL schema, the types and predicates in the Dgraph schema, and cause indexes to be recomputed. +- **`export`** - Start an export of all data in the cluster. Export format should be 'rdf' (the default if no format is given), or 'json'. +- **`draining`** - Set (or unset) the cluster draining mode. In draining mode no further requests are served. +- **`shutdown`** - Shutdown this node. +- **`config`** - Alter the node's config. +- **`removeNode`** - Remove a node from the cluster. +- **`moveTablet`** - Move a predicate from one group to another. +- **`assign`** - Lease UIDs, Timestamps or Namespace IDs in advance. +- **`backup`** - Start a binary backup. +- **`restore`** - Start restoring a binary backup. +- **`restoreTenant`** - Restore given tenant into namespace 0 of the cluster. +- **`login`** - Login to Dgraph. Successful login results in a JWT that can be used in future requests. If login is not successful an error is returned. +- **`addUser`** - Add a user. When linking to groups: if the group doesn't exist it is created; if the group exists, the new user is linked to the existing group. It's possible to both create new groups and link to existing groups in the one mutation. Dgraph ensures that usernames are unique, hence attempting to add an existing user results in an error. +- **`addGroup`** - Add a new group and (optionally) set the rules for the group. +- **`updateUser`** - Update users, their passwords and groups. As with AddUser, when linking to groups: if the group doesn't exist it is created; if the group exists, the new user is linked to the existing group. If the filter doesn't match any users, the mutation has no effect. +- **`updateGroup`** - Add or remove rules for groups. If the filter doesn't match any groups, the mutation has no effect. +- **`deleteGroup`** - Delete a group. +- **`deleteUser`** - Delete a user. +- **`addNamespace`** - Add a new namespace. +- **`deleteNamespace`** - Delete a namespace. +- **`resetPassword`** - Reset password can only be used by the Guardians of the galaxy to reset password of any user in any namespace. + + +For security configuration including authentication, IP whitelisting, and token-based access control, see [Admin Endpoint Security](./security/admin-endpoint-security). diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/admin/admin-tasks/binary-backups.mdx b/docusaurus-docs/docs_versioned_docs/version-v25.4/admin/admin-tasks/binary-backups.mdx new file mode 100644 index 00000000..cee10de9 --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/admin/admin-tasks/binary-backups.mdx @@ -0,0 +1,626 @@ +--- +title: Binary Backups +--- + +:::warning +Binary backups require a valid enterprise license. +::: + +Binary backups are full backups of Dgraph data that are written directly to cloud storage (such as Amazon S3 or MinIO) or to an on-premises network file system shared by all Alpha servers. Binary backups enable you to restore a Dgraph cluster to a previous state. Unlike [exports](../admin-tasks/export-database), binary backups are Dgraph-specific and provide faster restore operations. + +## Create a Backup + +To create a backup, send a GraphQL mutation to the `/admin` endpoint. + +The `BackupInput` type defines the available options for backup operations: + +```graphql +input BackupInput { + + """ + Destination for the backup: e.g. Minio or S3 bucket. + """ + destination: String! + + """ + Access key credential for the destination. + """ + accessKey: String + + """ + Secret key credential for the destination. + """ + secretKey: String + + """ + AWS session token, if required. + """ + sessionToken: String + + """ + Set to true to allow backing up to S3 or Minio bucket that requires no credentials. + """ + anonymous: Boolean + + """ + Force a full backup instead of an incremental backup. + """ + forceFull: Boolean + } +``` + +Execute the following mutation on the `/admin` endpoint using any GraphQL-compatible client (such as Insomnia, GraphQL Playground, or GraphiQL). + +### Backup to NFS + +```graphql +mutation { + backup(input: {destination: "/path/to/local/directory"}) { + response { + message + code + } + taskId + } +} +``` + +A local filesystem works only if all Alpha servers have access to it (for example, when all Alpha servers run as normal processes on the same filesystem, not in Docker containers). Use an NFS mount to ensure backups work seamlessly across multiple machines and containers. + +### Backup to Amazon S3 + +```graphql +mutation { + backup(input: {destination: "s3://s3.us-west-2.amazonaws.com/"}) { + response { + message + code + } + taskId + } +} +``` +#### Configure Amazon S3 Credentials + +To back up to Amazon S3, configure the Alpha server with the following AWS credentials using environment variables: + +| Environment Variable | Description | +|---------------------|-------------| +| `AWS_ACCESS_KEY_ID` or `AWS_ACCESS_KEY` | AWS access key with write permissions to the destination bucket | +| `AWS_SECRET_ACCESS_KEY` or `AWS_SECRET_KEY` | AWS secret access key with write permissions to the destination bucket | +| `AWS_SESSION_TOKEN` | AWS session token (required for temporary credentials) | + + +To configure IAM-based authentication: + +1. Create an [IAM Role](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create.html) with an IAM policy that grants access to the S3 bucket. +2. Attach the IAM role to your infrastructure: + - For EC2 instances: Use an [Instance Profile](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2_instance-profiles.html) or [IAM Roles for Amazon EC2](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html) + - For EKS pods: Use [IAM roles for service accounts](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html) to associate the IAM role with a [Kubernetes Service Account](https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/) + + + +### Backup to Minio + +```graphql +mutation { + backup(input: {destination: "minio://127.0.0.1:9000/"}) { + response { + message + code + } + taskId + } +} +``` +#### Configure MinIO Credentials + +To back up to MinIO, configure the Alpha server with the following MinIO credentials using environment variables: + +| Environment Variable | Description | +|---------------------|-------------| +| `MINIO_ACCESS_KEY` | MinIO access key with write permissions to the destination bucket | +| `MINIO_SECRET_KEY` | MinIO secret key with write permissions to the destination bucket | + + +### Directory Structures + +A binary backup directory has the following structure: + +```sh +backup +├── dgraph.20210102.204757.509 +│ └── r9-g1.backup +├── dgraph.20210104.224757.707 +│ └── r9-g1.backup +└── manifest.json +``` + +## Backup using a MinIO Gateway + +#### Azure Blob Storage + +You can use [Azure Blob Storage](https://azure.microsoft.com/services/storage/blobs/) through the [MinIO Azure Gateway](https://docs.min.io/docs/minio-gateway-for-azure.html). Configure a [storage account](https://docs.microsoft.com/azure/storage/common/storage-account-overview) and a [container](https://docs.microsoft.com/azure/storage/blobs/storage-blobs-introduction#containers) to organize the blobs. + +For MinIO configuration, [retrieve the storage account keys](https://docs.microsoft.com/azure/storage/common/storage-account-keys-manage). The [MinIO Azure Gateway](https://docs.min.io/docs/minio-gateway-for-azure.html) maps `MINIO_ACCESS_KEY` to the Azure Storage Account `AccountName` and `MINIO_SECRET_KEY` to the `AccountKey`. + +Once you have the `AccountName` and `AccountKey`, you can access Azure Blob Storage locally using one of these methods: + +* Run [MinIO Azure Gateway](https://docs.min.io/docs/minio-gateway-for-azure.html) using Docker + ```bash + docker run --publish 9000:9000 --name gateway \ + --env "MINIO_ACCESS_KEY=" \ + --env "MINIO_SECRET_KEY=" \ + minio/minio gateway azure + ``` +* Run [MinIO Azure Gateway](https://docs.min.io/docs/minio-gateway-for-azure.html) using the MinIO Binary + ```bash + export MINIO_ACCESS_KEY="" + export MINIO_SECRET_KEY="" + minio gateway azure + ``` + +#### Google Cloud Storage + +You can use [Google Cloud Storage](https://cloud.google.com/storage) through the [MinIO GCS Gateway](https://docs.min.io/docs/minio-gateway-for-gcs.html). [Create storage buckets](https://cloud.google.com/storage/docs/creating-buckets), create a Service Account key for GCS, and obtain a credentials file. See [Create a Service Account key](https://github.com/minio/minio/blob/master/docs/gateway/gcs.md#11-create-a-service-account-ey-for-gcs-and-get-the-credentials-file) for detailed instructions. + +Once you have a `credentials.json`, you can access GCS locally using one of these methods: + +* Run [MinIO GCS Gateway](https://docs.min.io/docs/minio-gateway-for-gcs.html) using Docker + ```bash + docker run --publish 9000:9000 --name gateway \ + --volume /path/to/credentials.json:/credentials.json \ + --env "GOOGLE_APPLICATION_CREDENTIALS=/credentials.json" \ + --env "MINIO_ACCESS_KEY=minioaccountname" \ + --env "MINIO_SECRET_KEY=minioaccountkey" \ + minio/minio gateway gcs + ``` +* Run [MinIO GCS Gateway](https://docs.min.io/docs/minio-gateway-for-gcs.html) using the MinIO Binary + ```bash + export GOOGLE_APPLICATION_CREDENTIALS=/path/to/credentials.json + export MINIO_ACCESS_KEY=minioaccesskey + export MINIO_SECRET_KEY=miniosecretkey + minio gateway gcs + ``` + +#### Verify MinIO Gateway + +MinIO Gateway includes an embedded web-based object browser. After starting the MinIO Gateway using one of the methods above, verify it is running by opening a web browser and navigating to `http://127.0.0.1:9000`. Confirm that the object browser is displayed and can access the remote object storage. + +### Disable HTTPS for S3 and MinIO Backups + +By default, Dgraph assumes the destination bucket uses HTTPS. If the bucket uses HTTP, the backup operation fails. To back up to a bucket using HTTP (insecure), set the query parameter `secure=false` in the `destination` field: + +```graphql +mutation { + backup(input: {destination: "minio://127.0.0.1:9000/?secure=false"}) { + response { + message + code + } + taskId + } +} +``` + + +### Override Credentials + +The `accessKey`, `secretKey`, and `sessionToken` parameters override the default credentials. + +**Note:** Unless HTTPS is used, credentials are transmitted in plain text. Use these parameters with caution. Prefer environment variables for credential management; these parameters provide additional flexibility when needed. + +Set the `anonymous` parameter to `true` to back up to an S3 or MinIO bucket that requires no credentials (a public bucket). + + + +### Force a Full Backup + +By default, Dgraph creates an incremental backup if a full backup exists in the specified location. To create a full backup, set the `forceFull` field to `true` in the mutation. Each backup series is identified by a unique ID, and each backup in the series is assigned a monotonically increasing number. See the restore section for details on restoring a backup series. + +```graphql +mutation { + backup(input: {destination: "/path/to/local/directory", forceFull: true}) { + response { + message + code + } + taskId + } +} +``` + +## List Backups + +The GraphQL admin interface provides the `listBackups` query that lists backups in a specified location along with information from the `manifest.json` file. The following example lists backups in the `/data/backup` location: + +``` +query backup() { + listBackups(input: {location: "/data/backup"}) { + backupId + backupNum + encrypted + groups { + groupId + predicates + } + path + since + type + } +} +``` + +The `ListBackupsInput` type supports the following fields. Only the `location` field is required. + +``` +input ListBackupsInput { + """ + Destination for the backup: e.g. Minio or S3 bucket. + """ + location: String! + + """ + Access key credential for the destination. + """ + accessKey: String + + """ + Secret key credential for the destination. + """ + secretKey: String + + """ + AWS session token, if required. + """ + sessionToken: String + + """ + Whether the destination doesn't require credentials (e.g. S3 public bucket). + """ + anonymous: Boolean +} +``` + +The query returns an array of `Manifest` objects. The fields in the `Manifest` type correspond to the fields in the `manifest.json` file. + +``` +type Manifest { + """ + Unique ID for the backup series. + """ + backupId: String + + """ + Number of this backup within the backup series. The full backup always has a value of one. + """ + backupNum: Int + + """ + Whether this backup was encrypted. + """ + encrypted: Boolean + + """ + List of groups and the predicates they store in this backup. + """ + groups: [BackupGroup] + + """ + Path to the manifest file. + """ + path: String + + """ + The timestamp at which this backup was taken. The next incremental backup will + start from this timestamp. + """ + since: Int + + """ + The type of backup, either full or incremental. + """ + type: String +} + +type BackupGroup { + """ + The ID of the cluster group. + """ + groupId: Int + + """ + List of predicates assigned to the group. + """ + predicates: [String] +} +``` + +## Convert Binary Backup to RDF export format + +The `export_backup` tool converts a binary backup into an exported folder format. + +Use this tool when upgrading between major Dgraph versions with incompatible changes. The tool enables you to apply changes to the exported `.rdf` file or schema file, then import the dataset into the new Dgraph version. + +Ensure you have created a binary backup. A typical binary backup directory structure looks like this: + +```sh +backup +├── dgraph.20210104.224757.709 +│ └── r9-g1.backup +└── manifest.json +``` + +Then run the following command: + +```sh +dgraph export_backup --location "" --destination "" +``` + +After completion, the export folder (in this example, `dgraph.r9.u0108.1621`) has the following structure: + +```sh +dgraph.r9.u0108.1621 +├── g01.gql_schema.gz +├── g01.rdf.gz +└── g01.schema.gz +``` + +## Encrypted Backups + +For encrypted backups, configure the Dgraph Alpha server with the `--encryption key-file=value` flag. You can alternatively configure the Alpha server to interface with a [HashiCorp Vault](https://www.vaultproject.io/) server to obtain encryption keys. + +:::note +The `encryption key-file=value` flag and `vault` superflag are used for both encryption-at-rest and encrypted backups. +::: + + +**Important:** All backups in a series (full and incremental) must use the same encryption setting. You cannot mix encrypted and unencrypted backups within the same backup series. The `Encrypted` flag enforces this restriction. + + +The key size (16, 24, or 32 bytes) determines the AES cipher: AES-128, AES-192, or AES-256. Dgraph uses AES in CTR mode. Binary backups are already compressed with gzip; encryption is applied to the gzipped data. + +During backup, a 16-byte initialization vector (IV) is prepended to the ciphertext after encryption. + + +## Online Restore + +To restore from a backup to a live cluster, execute a mutation on the `/admin` endpoint: + +```graphql +mutation{ + restore(input:{ + location: "/path/to/backup/directory", + backupId: "id_of_backup_to_restore" + }){ + message + code + } +} +``` + +Online restore operations return immediately after the request is sent. The restore process updates UID and timestamp leases automatically. The backup being restored must contain the same number of groups in its `manifest.json` file as the target cluster. + +:::note +When using backups made from a Dgraph cluster that uses encryption (so backups are encrypted), +you need to use the same key from that original cluster when doing a restore process. +Dgraph's [Encryption at Rest](../../installation/configuration/encryption-at-rest) uses a symmetric-key +algorithm where the same key is used for both encryption and decryption, so the encryption key from that +cluster is needed for the restore process. +::: + +Online restore can be performed from Amazon S3, MinIO, or a local directory. The `RestoreInput` type defines the available options: + +```graphql +input RestoreInput { + + """ + Destination for the backup: e.g. Minio or S3 bucket. + """ + location: String! + + """ + Backup ID of the backup series to restore. This ID is included in the manifest.json file. + If missing, it defaults to the latest series. + """ + backupId: String + + """ + Number of the backup within the backup series to be restored. Backups with a greater value + will be ignored. If the value is zero or is missing, the entire series will be restored. + """ + backupNum: Int + + """ + Path to the key file needed to decrypt the backup. This file should be accessible + by all Alpha servers in the group. The backup will be written using the encryption key + with which the cluster was started, which might be different than this key. + """ + encryptionKeyFile: String + + """ + Vault server address where the key is stored. This server must be accessible + by all Alpha servers in the group. Default "http://localhost:8200". + """ + vaultAddr: String + + """ + Path to the Vault RoleID file. + """ + vaultRoleIDFile: String + + """ + Path to the Vault SecretID file. + """ + vaultSecretIDFile: String + + """ + Vault kv store path where the key lives. Default "secret/data/dgraph". + """ + vaultPath: String + + """ + Vault kv store field whose value is the key. Default "enc_key". + """ + vaultField: String + + """ + Vault kv store field's format. Must be "base64" or "raw". Default "base64". + """ + vaultFormat: String + + """ + Access key credential for the destination. + """ + accessKey: String + + """ + Secret key credential for the destination. + """ + secretKey: String + + """ + AWS session token, if required. + """ + sessionToken: String + + """ + Set to true to allow backing up to S3 or Minio bucket that requires no credentials. + """ + anonymous: Boolean + + """ + All the backups with num >= incrementalFrom will be restored. + """ + incrementalFrom: Int + + """ + If `isPartial` is set to true then the cluster is kept in draining mode after + restore to ensure that the database is not corrupted by any mutations or tablet moves in + between two restores. + """ + isPartial: Boolean + +} +``` + +Restore requests return immediately without waiting for the operation to complete. + +## Incremental Restore + +Use incremental restore to restore a set of incremental backups on a cluster that has already been partially restored. The cluster enters draining mode during this process, which prevents mutations. Only admin requests to return the cluster to normal mode are accepted while in draining mode. + +:::important +Before starting an incremental restore, ensure you set `isPartial` to `true` in your initial restore operation. +::: + +To perform an incremental restore, execute a mutation on the `/admin` endpoint: + +```graphql +mutation{ + restore(input:{ + incrementalFrom:"incremental_backup_from", + location: "/path/to/backup/directory", + backupId: "id_of_backup_to_restore"' + }){ + message + code + } +} +``` + +## Namespace-Aware Restore + +Use namespace-aware restore to restore a single namespace from a backup that contains multiple namespaces. The restored data is available in the default namespace. For example, if you restore namespace 2 using the `restoreTenant` API, after the restore operation completes, the cluster contains only the default namespace with data from namespace 2. Namespace-aware restore supports incremental restore. + +To perform a namespace-aware restore, execute a mutation on the `/admin` endpoint: + +```graphql +mutation { + restoreTenant( + input: { + restoreInput: { + incrementalFrom: "incremental_backup_from" + location: "/path/to/backup/directory" + backupId: "id_of_backup_to_restore" + } + fromNamespace: namespaceToBeRestored + } + ) { + message + code + } +} +``` + +The `RestoreTenantInput` type defines the input parameters: + +``` +input RestoreTenantInput { + """ + restoreInput contains fields that are required for the restore operation, + i.e., location, backupId, and backupNum + """ + restoreInput: RestoreInput + + """ + fromNamespace is the namespace of the tenant that needs to be restored into namespace 0 of the new cluster. + """ + fromNamespace: Int! +} +``` + +## Offline Restore (Deprecated) + +:::warning +Offline restore is deprecated. Use online restore instead. +::: + +The `dgraph restore` command is a standalone tool that restores the postings directory from a previously created backup to a directory in the local filesystem. This command restores a backup to a new Dgraph cluster and is not designed to restore to a currently running cluster. During a restore operation, a temporary Dgraph Zero server may run to fully restore the backup state. + +Use the `--encryption key-file=value` flag to decrypt encrypted backups. The specified file must contain the same key used for encryption during backup. Starting with v20.07.0, you can use the `vault` superflag to restore encrypted backups. + +### Command Options + +- `--location` (`-l`): Specifies the source URI containing Dgraph backup objects. Supports all backup storage schemes. + +- `--postings` (`-p`): Sets the directory where restored posting directories are saved. This directory contains a posting directory for each group in the restored backup. + +- `--zero` (`-z`): Specifies a Dgraph Zero server address to update the start timestamp and UID lease using the restored version. If not specified, the command requires `--force_zero=false`, and you must manually update the timestamp and UID lease using the Dgraph Zero server's HTTP `assign` endpoint. Use the values printed at the end of the command output. + +- `--backup_id`: Specifies the ID of the backup series to restore. A backup series consists of a full backup and all incremental backups built on top of it. Each new full backup starts a new backup series with a different ID. The backup series ID is stored in each `manifest.json` file in each backup folder. + +- `--encryption key-file=value`: Required when restoring a backup from an encrypted cluster. The file path must point to the same key file used to run the original cluster. + +- `--vault` [superflag](../../cli/superflags): Specifies the [HashiCorp Vault](https://www.vaultproject.io/) server address (`addr`), role ID file (`role-id-file`), secret ID file (`secret-id-file`), and the field containing the encryption key (`enc-field`) used to encrypt the backup. + +The restore operation creates a cluster structure with as many groups as the original cluster had at the time of the last backup. For each group, `dgraph restore` creates a posting directory (`p`) that corresponds to the backup group ID. For example, a backup for Dgraph Alpha group 2 (named `.../r32-g2.backup`) is loaded to posting directory `p2`. + +After running the restore command, manually copy the directories from the `postings` directory to the machines or containers running the Dgraph Alpha servers before starting `dgraph alpha`. For example, in a cluster with two Dgraph Alpha groups and one replica each, copy `p1` to the first Alpha node and `p2` to the second Alpha node. + +By default, Dgraph looks for a posting directory named `p`. Rename the directories after moving them, or use the `-p` option of the `dgraph alpha` command to specify a different path. + +### Restore from Amazon S3 + +```sh +dgraph restore --postings "/var/db/dgraph" --location "s3://s3..amazonaws.com/" +``` + +### Restore from MinIO + +```sh +dgraph restore --postings "/var/db/dgraph" --location "minio://127.0.0.1:9000/" +``` + +### Restore from Local Directory or NFS + +```sh +dgraph restore --postings "/var/db/dgraph" --location "/var/backups/dgraph" +``` + +### Restore and Update Timestamp + +Specify the Zero server address and port for the new cluster with `--zero`/`-z` to update the timestamp. +```sh +dgraph restore --postings "/var/db/dgraph" --location "/var/backups/dgraph" --zero "localhost:5080" +``` diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/admin/admin-tasks/check-cluster-health.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/admin/admin-tasks/check-cluster-health.md new file mode 100644 index 00000000..80fea5e0 --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/admin/admin-tasks/check-cluster-health.md @@ -0,0 +1,82 @@ +--- +title: Check Cluster Health +--- + +You can check the health of your Dgraph cluster using HTTP endpoints or the GraphQL Admin API. + +## HTTP Endpoints + +Alpha exposes HTTP endpoints on port `8080` for monitoring and administration: + +**`/health?all`** - Returns information about the health of all servers in the cluster. + +The `/health` endpoint provides basic cluster health status. Use `/health?all` to get detailed information about all nodes. + +## GraphQL Admin API + +You can query the `/admin` GraphQL endpoint to get detailed health information about all servers in the cluster: + +```graphql +query { + health { + instance + address + version + status + lastEcho + group + uptime + ongoing + indexing + } +} +``` + +**Response Example:** + +```json +{ + "data": { + "health": [ + { + "instance": "zero", + "address": "localhost:5080", + "version": "v2.0.0-rc1", + "status": "healthy", + "lastEcho": 1582827418, + "group": "0", + "uptime": 1504 + }, + { + "instance": "alpha", + "address": "localhost:7080", + "version": "v2.0.0-rc1", + "status": "healthy", + "lastEcho": 1582827418, + "group": "1", + "uptime": 1505, + "ongoing": ["opIndexing"], + "indexing": ["name", "age"] + } + ] + } +} +``` + +**Response Fields:** + +- **`instance`**: Name of the instance. Either `alpha` or `zero`. +- **`status`**: Health status of the instance. Either `healthy` or `unhealthy`. +- **`version`**: Version of Dgraph running the Alpha or Zero server. +- **`uptime`**: Time in nanoseconds since the Alpha or Zero server is up and running. +- **`address`**: IP_ADDRESS:PORT of the instance. +- **`group`**: Group assigned based on the replication factor. +- **`lastEcho`**: Last time, in Unix epoch, when the instance was contacted by another Alpha or Zero server. +- **`ongoing`**: List of ongoing operations in the background. +- **`indexing`**: List of predicates for which indexes are built in the background. + +:::note +The same information (except `ongoing` and `indexing`) is available from the `/health` and `/health?all` HTTP endpoints. +::: + + diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/admin/admin-tasks/delete-database.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/admin/admin-tasks/delete-database.md new file mode 100644 index 00000000..a04fd484 --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/admin/admin-tasks/delete-database.md @@ -0,0 +1,16 @@ +--- +title: Delete Database +--- + +To drop all data, you could send a `DropAll` request via `/alter` endpoint. + +Alternatively, you could: + +* [Shutdown Dgraph](shut-down-database) and wait for all writes to complete, +* Delete (maybe do an export first) the `p` and `w` directories, then +* Restart Dgraph. + +:::warning +Always [export your data](export-database) before deleting the database to ensure you have a backup. +::: + diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/admin/admin-tasks/export-database.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/admin/admin-tasks/export-database.md new file mode 100644 index 00000000..b8d434ab --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/admin/admin-tasks/export-database.md @@ -0,0 +1,11 @@ +--- +title: Export Database +--- + +As an `Administrator` you might want to export data from Dgraph to: + +* backup your data +* move the data to another Dgraph instance +* share your data + +For more information about exporting your database, see [Export data](../../migration/export-data) diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/admin/admin-tasks/index.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/admin/admin-tasks/index.md new file mode 100644 index 00000000..fc3d8c6c --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/admin/admin-tasks/index.md @@ -0,0 +1,40 @@ +--- +title: Admin Tasks +--- + +Dgraph Alpha exposes various administrative endpoints over HTTP and GraphQL for operations like data export, cluster shutdown, and database management. + +For security configuration including authentication, IP whitelisting, and token-based access control, see [Admin Endpoint Security](../security/admin-endpoint-security). + +## Admin Endpoints + +Dgraph Alpha provides the following administrative endpoints: + +### HTTP Admin Endpoints + +- **`/admin/config/cache_mb`** - Configure cache size +- **`/admin/draining`** - Drain connections from a node +- **`/admin/shutdown`** - Shutdown a single Alpha node +- **`/alter`** - Apply schema updates and drop predicates +- **`/login`** - Authenticate ACL users +- **`/health`** - health status +- **`/health?all`** - health status of all servers in the cluster + +### GraphQL ADmin Endpoints +- **`/admin`** - GraphQL endpoint for cluster management operations + By default, the `/admin` endpoint is only accessible from the same machine as the Alpha server. For detailed information about endpoint security and authentication, see [Admin Endpoint Security](../security/admin-endpoint-security). + +## Admin Tasks + +The following administrative tasks are available: + +### Data Management + +- **[Export Database](export-database)** - Export data from Dgraph for backup, migration, or sharing +- **[Delete Database](delete-database)** - Drop all data from the database + +### Cluster Management + +- **[Check Cluster Health](check-cluster-health)** - Monitor cluster health using HTTP endpoints or GraphQL Admin API +- **[Shut Down Database](shut-down-database)** - Perform a clean shutdown of a Dgraph node +- **[Upgrade Database](upgrade-database)** - Safely upgrade Dgraph version and migrate data diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/admin/admin-tasks/multitenancy.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/admin/admin-tasks/multitenancy.md new file mode 100644 index 00000000..3847302e --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/admin/admin-tasks/multitenancy.md @@ -0,0 +1,179 @@ +--- +title: Multi-Tenancy +description: Enable multiple tenants to share a Dgraph cluster using logically separated namespaces +--- + +Multi-tenancy enables multiple tenants to coexist in the same Dgraph cluster using `uint64` namespaces. Each tenant operates in its own namespace with logical data separation—data is stored in the same `p` directory but is not accessible across namespaces. + +:::note +**Enterprise Feature**: Multi-tenancy requires [Access Control Lists](../../installation/configuration/enable-acl) (ACL) to be enabled. +::: + +Multi-tenancy builds upon ACL and scopes ACL policies to individual tenants. Access controls are applied per tenant to specific predicates or all predicates within that tenant. Tenants are logically separated; each client must authenticate within a tenant and can only access data as allowed by the tenant's ACL rules. + +The default namespace (`0x00`) is called a `galaxy`. [Guardians of the Galaxy](#guardians-of-the-galaxy) are super-admins with special privileges to create or delete namespaces and reset passwords across namespaces. Each namespace has a guardian group with root access to that namespace. Users belong to a single namespace; to access multiple namespaces, create separate user accounts for each. + +:::tip +For multi-tenant environments, consider setting a query timeout using `--limit query-limit=500ms` when starting Dgraph Alpha. +::: + +## Access Control Roles + +### Guardians of the Galaxy +Super Admins of namespace `0x00` + +- Create and delete namespaces +- Reset passwords across namespaces +- Query and mutate the default namespace (`0x00`) +- Trigger cluster-wide backups and exports +- Export all namespaces or specific namespaces + +### Guardians of a Namespace +- Create users and groups within the namespace +- Assign users to groups and predicates to groups +- Export the namespace +- Drop data within the namespace +- Query and mutate within the namespace + +**Normal Users**: +- Login into a namespace +- Query and mutate within the namespace as permitted by ACL rules + +:::note +Guardians of the Galaxy cannot read across tenants. They are used only for database administration operations such as exporting data of all tenants. +::: + +## Namespace Operations + +### Create a Namespace + +Only [Guardians of the Galaxy](#guardians-of-the-galaxy) can create namespaces. Send the JWT access token in the `X-Dgraph-AccessToken` header: + +```graphql +mutation { + addNamespace(input: {password: "mypass"}) { + namespaceId + message + } +} +``` + +This creates a namespace, automatically creates a guardian group for that namespace, and creates a `groot` user with the specified password (default is `password`) in the guardian group. Use these credentials to login and perform[`user management opertions`](user-management-access-control). + +### List Namespaces + +Only [Guardians of the Galaxy](#guardians-of-the-galaxy) can list active namespaces using the GraphQL `state` query: + +```graphql +query { + state { + namespaces + } +} +``` + +Response: + +```json +{ + "data": { + "state": { + "namespaces": [2, 1, 0] + } + } +} +``` + +### Delete a Namespace + +Only [Guardians of the Galaxy](#guardians-of-the-galaxy) can delete namespaces. Send the JWT access token in the `X-Dgraph-AccessToken` header: + +```graphql +mutation { + deleteNamespace(input: {namespaceId: 123}) { + namespaceId + message + } +} +``` + +### Reset Passwords + +Only [Guardians of the Galaxy](#guardians-of-the-galaxy) can reset passwords across namespaces: + +```graphql +mutation { + resetPassword(input: {userId: "groot", password: "newpassword", namespace: 100}) { + userId + message + } +} +``` + +## Drop Operations + +The `drop all` operation can only be triggered by a [Guardian of the Galaxy](#guardians-of-the-galaxy) and deletes data and schema across all namespaces. All other drop operations run at namespace level. Guardians of a namespace can trigger `drop data` within their namespace, which deletes all data but retains the schema. + +For example, to drop data within a namespace: + +```bash +curl 'http://localhost:8080/alter' \ + -H 'X-Dgraph-AccessToken: ' \ + --data-raw '{"drop_op":"DATA"}' +``` + +For information about other drop operations, see [Alter the database](../../clients/raw-http#alter-the-dql-schema). + +## Backups and Exports + +Backups are cluster-wide only and can only be triggered by a [Guardian of the Galaxy](#guardians-of-the-galaxy). Exports can be generated cluster-wide or at namespace level. + +[Initial import](../../migration/bulk-loader) and [Live import](../../migration/live-loader) tools support multi-tenancy. + + +### Exports + +Exports generate `.rdf` or `.json` files and schemas that include namespace information. If a Guardian of the Galaxy exports the whole cluster, a single folder contains export data of all namespaces in a single file with a single schema. + +Namespace-specific exports contain the namespace value in the generated `.rdf` file: + +```rdf +<0x01> "name" "ibrahim" <0x12> . -> belongs to namespace 0x12 +<0x01> "name" "ibrahim" <0x0> . -> belongs to namespace 0x00 +``` + +**Export a specific namespace** (Guardian of the Galaxy): + +```graphql +mutation { + export(input: {format: "rdf", namespace: 1234}) { + response { + message + } + } +} +``` + +**Export current namespace** (Guardian of a Namespace - no namespace parameter needed): + +```graphql +mutation { + export(input: {format: "rdf"}) { + response { + message + } + } +} +``` + +**Export all namespaces** (Guardian of the Galaxy only): + +```graphql +mutation { + export(input: {format: "rdf", namespace: -1}) { + response { + message + } + } +} +``` diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/admin/admin-tasks/shut-down-database.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/admin/admin-tasks/shut-down-database.md new file mode 100644 index 00000000..c8a0f8af --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/admin/admin-tasks/shut-down-database.md @@ -0,0 +1,24 @@ +--- +title: Shut Down Database +--- + +A clean exit of a single Dgraph node is initiated by running the following GraphQL mutation on /admin endpoint. + +:::warningThis won't work if called from outside the server where Dgraph is running. +You can specify a list or range of whitelisted IP addresses from which shutdown or other admin operations +can be initiated using the `--security` superflag's `whitelist` option on `dgraph alpha`. +::: + +```graphql +mutation { + shutdown { + response { + message + code + } + } +} +``` + +This stops the Alpha on which the command is executed and not the entire cluster. + diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/admin/admin-tasks/update-dgraph-types.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/admin/admin-tasks/update-dgraph-types.md new file mode 100644 index 00000000..5fda6bf7 --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/admin/admin-tasks/update-dgraph-types.md @@ -0,0 +1,78 @@ +--- +title: Update Dgraph types +--- + +You modify Dgraph types (node types and predicates types) by +- issuing a request to the ``/alter`` endpoint using the [HTTP Client](../../clients/raw-http#alter-the-dql-schema) +- using an ``alter`` operation of any [DQL client library](../../clients). +- using [Ratel UI](/ratel/schema) + + + +### Notes about predicate type change + +If data is already stored, existing values are not checked to conform to the updated predicate type. + +On query, Dgraph tries to convert existing values to the new predicate type and ignores any that fail conversion. + +If data exists and new indexes are specified, any old index not in the updated schema is dropped. New indexes are created. + + + + +## Indexes in Background + +Indexes may take long time to compute depending upon the size of the data. + +Indexes can be computed in the background and thus indexing may still be running after an Alter operation returns. + +To run index computation in the background set the flag `runInBackground` to `true` . + +```sh +curl localhost:8080/alter?runInBackground=true -XPOST -d $' + name: string @index(fulltext, term) . + age: int @index(int) @upsert . + friend: [uid] @count @reverse . +' | python -m json.tool | less +``` + +```go +op := &api.Operation{} +op.Schema = ` + name: string @index(fulltext, term) . + age: int @index(int) @upsert . + friend: [uid] @count @reverse . +` +op.RunInBackground = true +err = dg.Alter(context.Background(), op) +``` + +### Notes + +If executed before the indexing finishes, queries that require the new indices will fail with an error +notifying that a given predicate is not indexed or doesn't have reverse edges. + +In a multi-node cluster, it is possible that the alphas will finish computing indexes at different times. Alphas may return different schema in such a case until all the indexes are done computing on all the Alphas. + +You can check the background indexing status using the [Health](check-cluster-health) query on the `/admin` endpoint. + + +An alter operation will fail if one is already in progress with an error +`schema is already being modified. Please retry`. + + +Dgraph will report the indexes in the schema only when the indexes are done computing. + + +## Deleting a node type + +Type definitions can be deleted using the Alter endpoint. + +Below is an example deleting the type `Person` using the Go client: +```go +err := c.Alter(context.Background(), &api.Operation{ + DropOp: api.Operation_TYPE, + DropValue: "Person"}) +``` + + diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/admin/admin-tasks/upgrade-database.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/admin/admin-tasks/upgrade-database.md new file mode 100644 index 00000000..96f2d1ae --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/admin/admin-tasks/upgrade-database.md @@ -0,0 +1,40 @@ +--- +title: Upgrade Database +--- + +Doing periodic exports is always a good idea. This is particularly useful if you wish to upgrade Dgraph or reconfigure the sharding of a cluster. The following are the right steps to safely export and restart. + +1. Start an [export](export-database) +2. Ensure it is successful +3. [Shutdown Dgraph](shut-down-database) and wait for all writes to complete +4. Start a new Dgraph cluster using new data directories (this can be done by passing empty directories to the options `-p` and `-w` for Alphas and `-w` for Zeros) +5. Reload the data via [bulk loader](../../migration/bulk-loader) +6. Verify the correctness of the new Dgraph cluster. If all looks good, you can delete the old directories (export serves as an insurance) + +These steps are necessary because Dgraph's underlying data format could have changed, and reloading the export avoids encoding incompatibilities. + +## Blue-Green Deployment + +Blue-green deployment is a common approach to minimize downtime during the upgrade process. +This approach involves switching your application to read-only mode. To make sure that no mutations are executed during the maintenance window you can +do a rolling restart of all your Alpha using the option `--mutations disallow` when you restart the Alpha nodes. This will ensure the cluster is in read-only mode. + +At this point your application can still read from the old cluster and you can perform the steps 4. and 5. described above. +When the new cluster (that uses the upgraded version of Dgraph) is up and running, you can point your application to it, and shutdown the old cluster. + +## Enterprise Upgrade Notes + +For enterprise customers, specific upgrade procedures may be required depending on your Dgraph version. The general upgrade process uses [binary backups](binary-backups) for data migration: + +1. Use binary backup to export data from the old cluster +2. Ensure the backup is successful +3. [Shutdown Dgraph](shut-down-database) and wait for all writes to complete +4. Upgrade the `dgraph` binary to the target version +5. Restore from the backups using the upgraded `dgraph` binary +6. Start a new Dgraph cluster using the restored data directories +7. Run any required upgrade commands using `dgraph upgrade` if needed + +:::note +For specific version-to-version upgrade instructions, consult the release notes for your target Dgraph version. Always test upgrades in a non-production environment first. +::: + diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/admin/admin-tasks/user-management-access-control.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/admin/admin-tasks/user-management-access-control.md new file mode 100644 index 00000000..10b2ae90 --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/admin/admin-tasks/user-management-access-control.md @@ -0,0 +1,546 @@ +--- +title: User Management and Access Control +--- + +After enabling ACL, you can manage users, groups, and configure access control rules to protect your data. + +## Accessing Secured Dgraph + +Before managing users and groups and configuring ACL rules, you will need to login in order to get a token that is needed to access Dgraph. You will use this token with the `X-Dgraph-AccessToken` header field. + +### Logging In + +To login, send a POST request to `/admin` with the GraphQL mutation. For example, to log in as the root user `groot`: + +```graphql +mutation { + login(userId: "groot", password: "password") { + response { + accessJWT + refreshJWT + } + } +} +``` + +**Response:** + +```json +{ + "data": { + "accessJWT": "", + "refreshJWT": "" + } +} +``` + +#### Access Token + +The response includes the access and refresh JWTs which are used for the authentication itself and refreshing the authentication token, respectively. Save the JWTs from the response for later HTTP requests. + +You can run authenticated requests by passing the access JWT to a request via the `X-Dgraph-AccessToken` header. Add the header `X-Dgraph-AccessToken` with the `accessJWT` value which you got in the login response in the GraphQL tool which you're using to make the request. + +For example, if you were using the GraphQL Playground, you would add this in the headers section: + +```json +{ "X-Dgraph-AccessToken" : "" } +``` + +And in the main code section, you can add a mutation, such as: + +```graphql +mutation { + addUser(input: [{ name: "alice", password: "whiterabbit" }]) { + user { + name + } + } +} +``` + +#### Refresh Token + +The refresh token can be used in the `/admin` POST GraphQL mutation to receive new access and refresh JWTs, which is useful to renew the authenticated session once the ACL access TTL expires (controlled by Dgraph Alpha's flag `--acl_access_ttl` which is set to 6h0m0s by default). + +```graphql +mutation { + login( + userId: "groot" + password: "password" + refreshToken: "" + ) { + response { + accessJWT + refreshJWT + } + } +} +``` + +### Login using a Client + +With ACL configured, you need to log in as a user to access data protected by ACL rules. You can do this using the client's `.login(USER_ID, USER_PASSWORD)` method. + +Here are some code samples using a client: + +* **Go** ([dgo client](https://github.com/dgraph-io/dgo)): example `acl_over_tls_test.go` ([here](https://github.com/dgraph-io/dgraph/blob/main/tlstest/acl/acl_over_tls_test.go)) +* **Java** ([dgraph4j](https://github.com/dgraph-io/dgraph4j)): example `AclTest.java` ([here](https://github.com/dgraph-io/dgraph4j/blob/master/src/test/java/io/dgraph/AclTest.java)) + +### Login using curl + +If you are using `curl` from the command line, you can use the following with the above [login mutation](#logging-in) saved to `login.graphql`: + +```bash +## Login and save results +JSON_RESULT=$(curl http://localhost:8080/admin --silent --request POST \ + --header "Content-Type: application/graphql" \ + --upload-file login.graphql +) + +## Extracting a token using GNU grep, perl, the silver searcher, or jq +TOKEN=$(grep -oP '(?<=accessJWT":")[^"]*' <<< $JSON_RESULT) +TOKEN=$(perl -wln -e '/(?<=accessJWT":")[^"]*/ and print $&;' <<< $JSON_RESULT) +TOKEN=$(ag -o '(?<=accessJWT":")[^"]*' <<< $JSON_RESULT) +TOKEN=$(jq -r '.data.login.response.accessJWT' <<< $JSON_RESULT) + +## Run a GraphQL query using the token +curl http://localhost:8080/admin --silent --request POST \ + --header "Content-Type: application/graphql" \ + --header "X-Dgraph-AccessToken: $TOKEN" \ + --upload-file some_other_query.graphql +``` + +:::tip +Parsing JSON results on the command line can be challenging, so you will find some alternatives to extract the desired data using popular tools, such as [the silver searcher](https://github.com/ggreer/the_silver_searcher) or the json query tool [jq](https://stedolan.github.io/jq), embedded in this snippet. +::: + +## User and Group Administration + +The default configuration comes with a user `groot`, with a password of `password`. The `groot` user is part of administrative group called `guardians` that have access to everything. You can add more users to the `guardians` group as needed. + +### Reset the Root Password + +You can reset the root password like this example: + +```graphql +mutation { + updateUser( + input: { + filter: { name: { eq: "groot" } } + set: { password: "$up3r$3cr3t1337p@$$w0rd" } + } + ) { + user { + name + } + } +} +``` + +### Create a Regular User + +To create a user `alice`, with password `whiterabbit`, you should execute the following GraphQL mutation: + +```graphql +mutation { + addUser(input: [{name: "alice", password: "whiterabbit"}]) { + user { + name + } + } +} +``` + +### Create a Group + +To create a group `dev`, you should execute: + +```graphql +mutation { + addGroup(input: [{name: "dev"}]) { + group { + name + users { + name + } + } + } +} +``` + +### Assign a User to a Group + +To assign the user `alice` to both the group `dev` and the group `sre`, the mutation should be: + +```graphql +mutation { + updateUser( + input: { + filter: { name: { eq: "alice" } } + set: { groups: [{ name: "dev" }, { name: "sre" }] } + } + ) { + user { + name + groups { + name + } + } + } +} +``` + +### Remove a User from a Group + +To remove `alice` from the `dev` group, the mutation should be: + +```graphql +mutation { + updateUser( + input: { + filter: { name: { eq: "alice" } } + remove: { groups: [{ name: "dev" }] } + } + ) { + user { + name + groups { + name + } + } + } +} +``` + +### Delete a User + +To delete the user `alice`, you should execute: + +```graphql +mutation { + deleteUser(filter: { name: { eq: "alice" } }) { + msg + numUids + } +} +``` + +### Delete a Group + +To delete the group `sre`, the mutation should be: + +```graphql +mutation { + deleteGroup(filter: { name: { eq: "sre" } }) { + msg + numUids + } +} +``` + +## ACL Rules Configuration + +You can set up ACL rules using the Dgraph Ratel UI or by using a GraphQL tool, such as [Insomnia](https://insomnia.rest/), [GraphQL Playground](https://github.com/prisma/graphql-playground), [GraphiQL](https://github.com/skevy/graphiql-app), etc. You can set the permissions on a predicate for the group using a pattern similar to the UNIX file permission conventions shown below: + +| Permission | Value | Binary | +|-----------------------------|-------|--------| +| `READ` | `4` | `100` | +| `WRITE` | `2` | `010` | +| `MODIFY` | `1` | `001` | +| `READ` + `WRITE` | `6` | `110` | +| `READ` + `WRITE` + `MODIFY` | `7` | `111` | + +These permissions represent the following: + +* `READ` - group has permission to read the predicate +* `WRITE` - group has permission to write or update the predicate +* `MODIFY` - group has permission to change the predicate's schema + +The following examples will grant full permissions to predicates to the group `dev`. If there are no rules for a predicate, the default behavior is to block all (`READ`, `WRITE` and `MODIFY`) operations. + +### Assign Predicate Permissions to a Group + +Here we assign a permission rule for the `friend` predicate to the group: + +```graphql +mutation { + updateGroup( + input: { + filter: { name: { eq: "dev" } } + set: { rules: [{ predicate: "friend", permission: 7 }] } + } + ) { + group { + name + rules { + permission + predicate + } + } + } +} +``` + +In case you have [reverse edges](../../dql/dql-schema#reverse-predicates), they have to be given the permission to the group as well: + +```graphql +mutation { + updateGroup( + input: { + filter: { name: { eq: "dev" } } + set: { rules: [{ predicate: "~friend", permission: 7 }] } + } + ) { + group { + name + rules { + permission + predicate + } + } + } +} +``` + +In some cases, it may be desirable to manage permissions for all the predicates together rather than individual ones. This can be achieved using the `dgraph.all` keyword. + +The following example provides `read+write` access to the `dev` group over all the predicates of a given namespace using the `dgraph.all` keyword. + +```graphql +mutation { + updateGroup( + input: { + filter: { name: { eq: "dev" } } + set: { rules: [{ predicate: "dgraph.all", permission: 6 }] } + } + ) { + group { + name + rules { + permission + predicate + } + } + } +} +``` + +:::note +The permissions assigned to a group `dev` is the union of permissions from `dgraph.all` and permissions for a specific predicate `name`. So if the group is assigned `READ` permission for `dgraph.all` and `WRITE` permission for predicate `name` it will have both, `READ` and `WRITE` permissions for the `name` predicate, as a result of the union. +::: + +### Remove a Rule from a Group + +To remove a rule or rules from the group `dev`, the mutation should be: + +```graphql +mutation { + updateGroup( + input: { + filter: { name: { eq: "dev" } } + remove: { rules: [ "friend", "~friend" ] } + } + ) { + group { + name + rules { + predicate + permission + } + } + } +} +``` + +## Querying Users and Groups + +You can query and get information for users and groups. These sections show output that will show the user `alice` and the `dev` group along with rules for `friend` and `~friend` predicates. + +### Query for Users + +Let's query for the user `alice`: + +```graphql +query { + queryUser(filter: { name: { eq: "alice" } }) { + name + groups { + name + } + } +} +``` + +The output should show the groups that the user has been added to, e.g. + +```json +{ + "data": { + "queryUser": [ + { + "name": "alice", + "groups": [ + { + "name": "dev" + } + ] + } + ] + } +} +``` + +### Get User Information + +We can obtain information about a user with the following query: + +```graphql +query { + getUser(name: "alice") { + name + groups { + name + } + } +} +``` + +The output should show the groups that the user has been added to, e.g. + +```json +{ + "data": { + "getUser": { + "name": "alice", + "groups": [ + { + "name": "dev" + } + ] + } + } +} +``` + +### Query for Groups + +Let's query for the `dev` group: + +```graphql +query { + queryGroup(filter: { name: { eq: "dev" } }) { + name + users { + name + } + rules { + permission + predicate + } + } +} +``` + +The output should include the users in the group as well as the permissions, the group's ACL rules, e.g. + +```json +{ + "data": { + "queryGroup": [ + { + "name": "dev", + "users": [ + { + "name": "alice" + } + ], + "rules": [ + { + "permission": 7, + "predicate": "friend" + }, + { + "permission": 7, + "predicate": "~friend" + } + ] + } + ] + } +} +``` + +### Get Group Information + +To check the `dev` group information: + +```graphql +query { + getGroup(name: "dev") { + name + users { + name + } + rules { + permission + predicate + } + } +} +``` + +The output should include the users in the group as well as the permissions, the group's ACL rules, e.g. + +```json +{ + "data": { + "getGroup": { + "name": "dev", + "users": [ + { + "name": "alice" + } + ], + "rules": [ + { + "permission": 7, + "predicate": "friend" + }, + { + "permission": 7, + "predicate": "~friend" + } + ] + } + } +} +``` + +## Reset Groot Password + +If you have forgotten the password to the `groot` user, then you may reset the `groot` password (or the password for any user) by following these steps. + +1. Stop Dgraph Alpha. +2. Turn off ACLs by removing the `--acl_hmac_secret` config flag in the Alpha config. This leaves the Alpha open with no ACL rules, so be sure to restrict access, including stopping request traffic to this Alpha. +3. Start Dgraph Alpha. +4. Connect to Dgraph Alpha using Ratel and run the following upsert mutation to update the `groot` password to `newpassword` (choose your own secure password): + ```graphql + upsert { + query { + groot as var(func: eq(dgraph.xid, "groot")) + } + mutation { + set { + uid(groot) "newpassword" . + } + } + } + ``` +5. Restart Dgraph Alpha with ACLs turned on by setting the `--acl_hmac_secret` config flag. +6. Login as groot with your new password. + +## Related Topics + +- [Enable ACL](../../installation/configuration/enable-acl) - Configure and enable ACL feature +- [Admin Endpoints](../admin-endpoints) - GraphQL Admin API reference + diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/admin/admin-tasks/view-cluster-state.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/admin/admin-tasks/view-cluster-state.md new file mode 100644 index 00000000..839f5d3b --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/admin/admin-tasks/view-cluster-state.md @@ -0,0 +1,247 @@ +--- +title: View Cluster State +--- + +The cluster state provides detailed information about your cluster's current state, including group membership, predicate distribution (sharding), and cluster metadata. You can query cluster state using either the HTTP endpoint or the GraphQL Admin API on Alpha. + +## Query Cluster State + +### Using HTTP Endpoint + +Query the `/state` endpoint: + +```sh +curl http://localhost:8080/state | jq +``` + +### Using GraphQL Admin API + +You can also query cluster state using the GraphQL `state` query on the `/admin` endpoint (port 8080): + +```graphql +query { + state { + groups { + id + members { + id + groupId + addr + leader + amDead + } + tablets { + predicate + groupId + space + readOnly + } + } + zeros { + id + groupId + addr + leader + } + maxUID + maxTxnTs + maxRaftId + cid + license { + enabled + maxNodes + expiryTs + } + } +} +``` + +## Response Information + +The `/state` endpoint returns a JSON document containing: + +- **Cluster membership**: Instances that are part of the cluster +- **Group information**: Number of instances in Zero group and each Alpha group +- **Leadership**: Current leader of each group +- **Predicate distribution**: Which predicates (tablets) belong to which groups +- **Predicate sizes**: Estimated size in bytes of each predicate +- **Enterprise license**: License information and status +- **Transaction metadata**: Max leased transaction ID +- **UID metadata**: Max leased UID +- **Cluster ID**: Unique cluster identifier (CID) + +## Example Response + +Here's an example JSON response for a cluster with three Alpha nodes and three Zero nodes: + +```json +{ + "counter": "22", + "groups": { + "1": { + "members": { + "1": { + "id": "1", + "groupId": 1, + "addr": "alpha2:7082", + "leader": true, + "amDead": false, + "lastUpdate": "1603350485", + "clusterInfoOnly": false, + "forceGroupId": false + }, + "2": { + "id": "2", + "groupId": 1, + "addr": "alpha1:7080", + "leader": false, + "amDead": false, + "lastUpdate": "0", + "clusterInfoOnly": false, + "forceGroupId": false + }, + "3": { + "id": "3", + "groupId": 1, + "addr": "alpha3:7083", + "leader": false, + "amDead": false, + "lastUpdate": "0", + "clusterInfoOnly": false, + "forceGroupId": false + } + }, + "tablets": { + "dgraph.cors": { + "groupId": 1, + "predicate": "dgraph.cors", + "force": false, + "space": "0", + "remove": false, + "readOnly": false, + "moveTs": "0" + }, + "dgraph.graphql.schema": { + "groupId": 1, + "predicate": "dgraph.graphql.schema", + "force": false, + "space": "0", + "remove": false, + "readOnly": false, + "moveTs": "0" + }, + "dgraph.type": { + "groupId": 1, + "predicate": "dgraph.type", + "force": false, + "space": "0", + "remove": false, + "readOnly": false, + "moveTs": "0" + } + }, + "snapshotTs": "22", + "checksum": "18099480229465877561" + } + }, + "zeros": { + "1": { + "id": "1", + "groupId": 0, + "addr": "zero1:5080", + "leader": true, + "amDead": false, + "lastUpdate": "0", + "clusterInfoOnly": false, + "forceGroupId": false + }, + "2": { + "id": "2", + "groupId": 0, + "addr": "zero2:5082", + "leader": false, + "amDead": false, + "lastUpdate": "0", + "clusterInfoOnly": false, + "forceGroupId": false + }, + "3": { + "id": "3", + "groupId": 0, + "addr": "zero3:5083", + "leader": false, + "amDead": false, + "lastUpdate": "0", + "clusterInfoOnly": false, + "forceGroupId": false + } + }, + "maxUID": "10000", + "maxTxnTs": "10000", + "maxRaftId": "3", + "removed": [], + "cid": "2571d268-b574-41fa-ae5e-a6f8da175d6d", + "license": { + "user": "", + "maxNodes": "18446744073709551615", + "expiryTs": "1605942487", + "enabled": true + } +} +``` + +## Understanding the Response + +### Group Members + +The response shows node members with their addresses and HTTP port numbers: + +- **Group 1 members** (Alpha nodes): + - alpha2:7082, id: 1, leader + - alpha1:7080, id: 2 + - alpha3:7083, id: 3 +- **Group 0 members** (Dgraph Zero nodes): + - zero1:5080, id: 1, leader + - zero2:5082, id: 2 + - zero3:5083, id: 3 + +### maxUID + +The current maximum lease of UIDs used for blank node UID assignment. This increments in batches of 10,000 IDs. Once the maximum lease is reached, another 10,000 IDs are leased. In the event that the Zero leader is lost, the new leader starts a new lease from `maxUID`+1. Any UIDs lost between these leases will never be used for blank-node UID assignment. + +An admin can use the Zero endpoint HTTP GET `/assign?what=uids&num=1000` to reserve a range of UIDs (in this case, 1000) to use externally. Zero will **never** use these UIDs for blank node UID assignment, so the user can use the range to assign UIDs manually to their own data sets. + +### maxTxnTs + +The current maximum lease of transaction timestamps used to hand out start timestamps and commit timestamps. This increments in batches of 10,000 IDs. After the max lease is reached, another 10,000 IDs are leased. If the Zero leader is lost, then the new leader starts a new lease from `maxTxnTs`+1. Any lost transaction IDs between these leases will never be used. + +An admin can use the Zero endpoint HTTP GET `/assign?what=timestamps&num=1000` to increase the current transaction timestamp (in this case, by 1000). This is mainly useful in special-case scenarios; for example, using an existing `-p directory` to create a fresh cluster to be able to query the latest data in the DB. + +### maxRaftId + +The number of Zeros available to serve as a leader node. Used by the [RAFT](../../design-concepts/raft) consensus algorithm. + +### CID + +This is a unique UUID representing the *cluster-ID* for this cluster. It is generated during the initial DB startup and is retained across restarts. + +### Enterprise License + +License information including: +- Enabled status +- `maxNodes`: Maximum number of nodes allowed (unlimited if not restricted) +- License expiration, shown in seconds since the Unix epoch + +### Tablets (Predicates) + +The `tablets` section shows which predicates are assigned to which groups. Each tablet entry includes: +- `groupId`: The group that owns this predicate +- `predicate`: The predicate name +- `space`: Estimated size +- `readOnly`: Whether the predicate is read-only +- `moveTs`: Timestamp of last move operation + +:::note +The terms "tablet", "predicate", and "edge" are currently synonymous. In future, Dgraph might improve data scalability to shard a predicate into separate tablets that can be assigned to different groups. +::: + diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/admin/data-compression.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/admin/data-compression.md new file mode 100644 index 00000000..0ad5967e --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/admin/data-compression.md @@ -0,0 +1,36 @@ +--- +title: Data compression on Disk +--- + +Dgraph Alpha lets you configure the compression of data on disk using the `--badger` +superflag's `compression` option. You can choose between the +[Snappy](https://github.com/golang/snappy) and +[Zstandard](https://github.com/facebook/zstd) compression algorithms, or choose +not to compress data on disk. + +The following disk compression settings are available: + +| Setting | Notes | +|------------|----------------------------------------------------------------------| +|`none` | Data on disk will not be compressed. | +|`zstd:level`| Use Zstandard compression, with a compression level specified (1-3). | +|`snappy` | Use Snappy compression (this is the default value). | + +For example, you could choose to use Zstandard compression with the highest +compression level using the following command: + +```sh +dgraph alpha --badger compression=zstd:3 +``` + +This compression setting (Zstandard, level 3) is more CPU-intensive than other +options, but offers the highest compression ratio. To change back to the default +compression setting, use the following command: + + +```sh +dgraph alpha --badger compression=snappy +``` + +Using this compression setting (Snappy) provides a good compromise between the +need for a high compression ratio and efficient CPU usage. diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/admin/dgraph-administration.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/admin/dgraph-administration.md new file mode 100644 index 00000000..29455de6 --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/admin/dgraph-administration.md @@ -0,0 +1,7 @@ +--- +title: Admin Tasks +--- + +This page has been moved to [Admin Tasks](admin-tasks/). + +For administrative operations including data export, cluster shutdown, database deletion, and upgrade procedures, see the [Admin Tasks](admin-tasks/) section. diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/admin/enterprise-features/index.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/admin/enterprise-features/index.md new file mode 100644 index 00000000..1e3377f5 --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/admin/enterprise-features/index.md @@ -0,0 +1,4 @@ +--- +title: Advanced Features +--- + diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/admin/enterprise-features/learner-nodes.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/admin/enterprise-features/learner-nodes.md new file mode 100644 index 00000000..b1db74da --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/admin/enterprise-features/learner-nodes.md @@ -0,0 +1,91 @@ +--- +title: Learner Nodes +description: Learner nodes let you spin-up read-only replica instance to serve best-effort queries faster +--- + +A Learner node is an enterprise-only feature that allows a user to spin-up a read-only replica instance across the world without paying a latency cost. +When enabled, a Dgraph cluster using learner nodes can serve best-effort queries faster. + +A "learner node" can still accept write operations. The node forwards them over to the Alpha group leader and does the writing just like a typical Alpha node. It will just be slower, depending on the latency between the Alpha node and the learner node. + +:::note +A learner node instance can forward `/admin` operations and perform both read and write operations, +but writing will incur in network call latency to the main cluster. +::: + + +## Set up a Learner node + +The learner node feature works at the Dgraph Alpha group level. +To use it, first you need to set up an Alpha instance as a learner node. +Once the learner instance is up, this replica can be used to run best-effort queries +with zero latency overhead. Because it's an Enterprise feature, a learner node +won't be able to connect to a Dgraph Zero node until the Zero node has a valid +license. + +To spin up a learner node, first make sure that you start all the nodes, including the Dgraph Zero +leader and the Dgraph Alpha leader, with the `--my` flag so that these nodes will +be accessible to the learner node. Then, start an Alpha instance as follows: + +```sh +dgraph alpha --raft="learner=true; group=N" --my :5080 +``` + +This allows the new Alpha instance to get all the updates from the group "N" leader without participating in the Raft elections. + +:::note +You must specify the `--my` flag to set the IP address and port of Dgraph Zero, +the Dgraph Alpha leader node, and the learner node. If you don't, you will get +an error similar to the following: `Error during SubscribeForUpdates` +::: + +## Best-effort Queries + +Regular queries use the strict consistency model, and any write operation to the cluster anywhere would be read immediately. + +Best-effort queries apply the eventual consistency model. A write to the cluster will be seen eventually to the node. +In regular conditions, the eventual consistency is usually achieved quickly. + +A best-effort query to a learner node returns any data that is already available in that learner node. +The response is still a valid data snapshot, but at a timestamp which is not the latest one. + +:::note +Best-effort queries won't be forwarded to a Zero node to get the latest timestamp. +::: + +You can still send typical read queries (strict consistency) to a learner node. +They would just incur an extra latency cost due to having to reach out the Zero leader. + +:::note +If the learner node needs to serve normal queries, at least one Alpha leader must be available. +::: + +## Use-case examples + +### Geographic distribution + +Consider this scenario: + +*You want to achieve low latency for clients in a remote geographical region, +distant from your Dgraph cluster.* + +You can address this need by using a learner node to run best-effort queries. +This read-only replica instance can be across distant geographies and you can +use best-effort queries to get instant responses. + +Because learner nodes support read and write operations, users in the remote +location can do everything with this learner node, as if they were working with +the full cluster. + + diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/admin/index.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/admin/index.md new file mode 100644 index 00000000..9e606661 --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/admin/index.md @@ -0,0 +1,28 @@ +--- +title: Administration +--- + +Dgraph administration covers the operational tasks needed to manage, monitor, and maintain your Dgraph cluster. + +## Core Administration Tasks + +**[Admin Tasks](admin-tasks/)** - Administrative operations including data export, cluster shutdown, database deletion, and upgrade procedures. Also covers basic security configuration like IP whitelisting and authentication tokens. + + +**[View Cluster State](admin-tasks/view-cluster-state)** - View cluster state, group membership, and predicate distribution (sharding) information. + +## Security & Access Control + +**[Security Configuration](security/)** - TLS configuration, port usage, and network security settings. + +**[User Management and Access Control](admin-tasks/user-management-access-control.md)** - manage users, groups, and configure access control rules to protect your data. + +## Monitoring & Observability + +**[Observability](observability/)** - Monitoring with Prometheus/Grafana, metrics collection, distributed tracing, and log format documentation. + +**[Troubleshooting](troubleshooting)** - Common issues, OOM handling, file descriptor limits, and cluster setup verification. + +## Configuration + +**[Data Compression](data-compression)** - Configure disk compression algorithms (Snappy, Zstandard) for Alpha data storage. \ No newline at end of file diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/admin/observability/audit-logs.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/admin/observability/audit-logs.md new file mode 100644 index 00000000..1a73bf80 --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/admin/observability/audit-logs.md @@ -0,0 +1,145 @@ +--- +title: Audit Logging +description: Track and audit all requests (queries and mutations) with Dgraph audit logging. +--- + +:::note +**Enterprise Feature**: Audit logging requires a Dgraph Enterprise license. See [License](../../installation/configuration/license) for details. +::: + +Audit logging tracks all requests (queries and mutations) sent to your Dgraph cluster. When enabled, audit logs record the following information for each request: + +* **Endpoint** - The API endpoint used for the request +* **User Name** - The logged-in user (if ACL is enabled) +* **Server address** - The Dgraph server host address +* **Client address** - The client host address +* **Request Body** - The request payload (truncated at 4KB) +* **Timestamp** - When the request was received +* **Namespace** - The namespace ID (for multi-tenant clusters) +* **Query Parameters** - Any query parameters provided +* **Response status** - The HTTP/gRPC response status + +## Audit Log Scope + +Audit logging captures most queries and mutations sent to Dgraph Alpha and Dgraph Zero nodes. + +### Logged Requests + +* HTTP requests sent to Dgraph Zero's port 6080 and Dgraph Alpha's port 8080 (except health/monitoring endpoints) +* gRPC requests sent to Dgraph Zero's port 5080 and Dgraph Alpha's port 9080 (except internal cluster endpoints) + +### Excluded Requests + +The following requests are not logged: + +* Response payloads (only requests are logged) +* HTTP requests to `/health`, `/state`, and `/jemalloc` endpoints +* gRPC requests to Raft endpoints (internal cluster consensus) +* gRPC requests to health check endpoints (`Check` and `Watch`) +* gRPC requests to Dgraph Zero stream endpoints (`StreamMembership`, `UpdateMembership`, `Oracle`, `Timestamps`, `ShouldServe`, `Connect`) + +## Audit Log Files + +Audit logs are written in JSON format. Dgraph uses a rolling-file policy: + +* The current log file is used until it reaches a configurable size (default: 100MB) +* When the size limit is reached, Dgraph creates a new current log file +* Older audit log files are retained for a configurable number of days (default: 10 days) + + +### Example Audit Log Entry + +For this GraphQL query: + +```graphql +{ + q(func: has(actor.film)){ + count(uid) + } +} +``` + +The corresponding audit log entry is: + +```json +{ + "ts":"2021-03-22T15:03:19.165Z", + "endpoint":"/query", + "level":"AUDIT", + "user":"", + "namespace":0, + "server":"localhost:7080", + "client":"[::1]:60118", + "req_type":"Http", + "req_body":"{\"query\":\"{\\n q(func: has(actor.film)){\\n count(uid)\\n }\\n}\",\"variables\":{}}", + "query_param":{ + "timeout":[ + "20s" + ] + }, + "status":"OK" +} +``` + +## Configuration + +Enable audit logging on Dgraph Alpha or Dgraph Zero nodes using the `--audit` flag with semicolon-separated options. + +### Configuration Options + +| Option | Description | Default | +|--------|-------------|---------| +| `output=` | Directory path for storing audit logs | Required | +| `size=` | Maximum size per log file in MB | 100 | +| `days=` | Number of days to retain log files | 10 | +| `compress=true` | Enable compression for older log files | false | +| `encrypt-file=` | Path to encryption key file for log encryption | disabled | + +### Enable Audit Logging + +The simplest configuration specifies only the output directory: + +```bash +dgraph alpha --audit output=audit-log-dir +``` + +### Customize Log File Size and Retention + +Configure larger log files and extended retention: + +```bash +dgraph alpha --audit "output=audit-log-dir;size=200;days=15" +``` + +This sets log files to 200 MB and retains them for 15 days. + +### Enable Compression + +Compress older audit logs to reduce storage space: + +```bash +dgraph alpha --audit "output=audit-log-dir;compress=true" +``` + +### Enable Encryption + +Encrypt audit logs to protect sensitive information in logged requests: + +```bash +dgraph alpha --audit "output=audit-log-dir;compress=true;encrypt-file=/path/to/encrypt/key/file" +``` + +### Decrypt Audit Logs + +Decrypt encrypted audit logs using the `dgraph audit decrypt` command: + +```bash +dgraph audit decrypt \ + --encryption_key_file=/path/encrypt/key/file \ + --in /path/to/encrypted/log/file \ + --out /path/to/output/file +``` + +## Related Documentation + +For general logging and log format information, see [Log Format](log-format). diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/admin/observability/index.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/admin/observability/index.md new file mode 100644 index 00000000..6371a628 --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/admin/observability/index.md @@ -0,0 +1,3 @@ +--- +title: Observability +--- \ No newline at end of file diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/admin/observability/log-format.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/admin/observability/log-format.md new file mode 100644 index 00000000..191d68db --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/admin/observability/log-format.md @@ -0,0 +1,115 @@ +--- +title: Logging +description: Dgraph logs requests for queries and mutations, and also provides audit logging capabilities with a Dgraph Enterprise license. +--- + +Dgraph logs requests for queries and mutations, and also provides audit logging +capabilities with a Dgraph [enterprise license](../../installation/configuration/license). + +Dgraph's log format comes from the glog library and is [formatted](https://github.com/golang/glog/blob/23def4e6c14b4da8ac2ed8007337bc5eb5007998/glog.go#L523-L533) as follows: + +``` +Lmmdd hh:mm:ss.uuuuuu threadid file:line] msg... +``` + +The fields shown above are defined as follows: + + + +| Field | Definition | +|-------|------------| +| `L` | A single character, representing the log level (eg 'I' for INFO) | +| `mm` | Month (zero padded; ie May is '05') | +| `dd` | Day (zero padded) | +| `hh:mm:ss.uuuuuu` | Time in hours, minutes and fractional seconds | +| `threadid` | Space-padded thread ID as returned by GetTID() | +| `file` | Filename | +| `line` | Line number | +| `msg` | User-supplied message | + +## Log verbosity + +To increase log verbosity, set the flag `-v=3` (or `-v=2`) which will enable verbose logging for everything. You can set this flag on both Zero and Alpha nodes. +:::note +Changing log verbosity requires a restart of the node. +::: + +## Request logging + +Request logging, sometimes called *query logging*, lets you log queries and mutations. +You can dynamically turn request logging on or off. To toggle request logging on, send the following GraphQL mutation to the `/admin` endpoint of an Alpha node (e.g. `localhost:8080/admin`): + +```graphql +mutation { + config(input: {logDQLRequest: true}) { + response { + code + message + } + } +} +``` +Note this input flag was named logRequest until Dgraph version v23. + +The response should look like the following: + +```json +{ + "data": { + "config": { + "response": { + "code": "Success", + "message": "Config updated successfully" + } + } + }, + "extensions": { + "tracing": { + "version": 1, + "startTime": "2020-12-07T14:53:28.240420495Z", + "endTime": "2020-12-07T14:53:28.240569604Z", + "duration": 149114 + } + } +} +``` +Also, the Alpha node will print the following INFO message to confirm that the mutation has been applied: +``` +I1207 14:53:28.240516 20143 config.go:39] Got config update through GraphQL admin API +``` + +When enabling request logging this prints the requests that Dgraph Alpha receives from Ratel or other clients. In this case, the Alpha log will print something similar to: + +``` +I1201 13:06:26.686466 10905 server.go:908] Got a query: query:"{\n query(func: allofterms(name@en, \"Marc Caro\")) {\n uid\n name@en\n director.film\n }\n}" +``` +As you can see, we got the query that Alpha received. To read it in the original DQL format just replace every `\n` with a new line, any `\t` with a tab character and `\"` with `"`: + +``` +{ + query(func: allofterms(name@en, "Marc Caro")) { + uid + name@en + director.film + } +} +``` + +Similarly, you can turn off request logging by setting `logRequest` to `false` in the `/admin` mutation. + +```graphql +mutation { + config(input: {logRequest: false}) { + response { + code + message + } + } +} +``` + +## Audit logging (enterprise feature) + +With a Dgraph enterprise license, you can enable audit logging so that all +requests are tracked and available for use in security audits. To learn more, see +[Audit Logging](audit-logs). diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/admin/observability/metrics.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/admin/observability/metrics.md new file mode 100644 index 00000000..fa13ed94 --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/admin/observability/metrics.md @@ -0,0 +1,122 @@ +--- +title: Metrics +description: Dgraph database helps administrators by providing metrics on Dgraph instance activity, disk activity, server node health, memory, and Raft leadership. +--- + + +Dgraph database provides metrics on Dgraph instance activity, disk activity, +server node health, memory, and Raft leadership. It also provides built-in +metrics provided by Go. Dgraph metrics follow the +[metric and label conventions for the Prometheus](https://prometheus.io/docs/practices/naming/) +monitoring and alerting toolkit. + +## Activity Metrics + +Activity metrics let you track the mutations, queries, and proposals of a Dgraph +instance. + + Metric | Description + ------- | ----------- + `go_goroutines` | Total number of Goroutines currently running in Dgraph. + `dgraph_active_mutations_total` | Total number of mutations currently running. + `dgraph_pending_proposals_total` | Total pending Raft proposals. + `dgraph_pending_queries_total` | Total number of queries in progress. + `dgraph_num_queries_total{method="Server.Mutate"}` | Total number of mutations run in Dgraph. + `dgraph_num_queries_total{method="Server.Query"}` | Total number of queries run in Dgraph. + +## Disk metrics + +Disk metrics let you track the disk activity of the Dgraph process. Dgraph does +not interact directly with the filesystem. Instead it relies on +[Badger](https://github.com/dgraph-io/badger) to read from and write to disk. + + Metric | Description + ------- | ----------- + `badger_read_num_vlog` | Total count of reads by badger in vlog, + `badger_write_num_vlog` | Total count of writes by Badger in vlog, + `badger_read_bytes_vlog` | Total bytes read by Badger, + `badger_write_bytes_vlog` | Total bytes written by Badger, + `badger_read_bytes_lsm` | Total bytes read by Badger, + `badger_write_bytes_l0` | Total bytes written by Badger, + `badger_write_bytes_compaction` | Total bytes written by Badger, + `badger_get_num_lsm` | Total count of LSM gets, + `badger_get_num_memtable` | Total count of LSM gets from memtable, + `badger_hit_num_lsm_bloom_filter` | Total count of LSM bloom hits, + `badger_get_num_user` | Total count of calls to Badger's `get`, + `badger_put_num_user` | Total count of calls to Badger's `put`, + `badger_write_bytes_user` | Total bytes written by user, + `badger_get_with_result_num_user` | Total count of calls to Badger's `get` that returned value, + `badger_iterator_num_user` | Total count of iterators made in badger, + `badger_size_bytes_lsm` | Size of the LSM in bytes, + `badger_size_bytes_vlog` | Size of the value log in bytes, + `badger_write_pending_num_memtable` | Total count of pending writes, + `badger_compaction_current_num_lsm` | Number of tables being actively compacted, + + Old Metrics (Pre 23.1.0) + + `badger_disk_reads_total` | Total count of disk reads in Badger. + `badger_disk_writes_total` | Total count of disk writes in Badger. + `badger_gets_total` | Total count of calls to Badger's `get`. + `badger_memtable_gets_total` | Total count of memtable accesses to Badger's `get`. + `badger_puts_total` | Total count of calls to Badger's `put`. + `badger_read_bytes` | Total bytes read from Badger. + `badger_lsm_bloom_hits_total` | Total number of LSM tree bloom hits. + `badger_written_bytes` | Total bytes written to Badger. + `badger_lsm_size_bytes` | Total size in bytes of the LSM tree. + `badger_vlog_size_bytes` | Total size in bytes of the value log. + +## Go Metrics + +Go's built-in metrics may also be useful to measure memory usage and garbage +collection time. + +Metric | Description +------- | ----------- +`go_memstats_gc_cpu_fraction` | The fraction of this program's available CPU time used by the GC since the program started. +`go_memstats_heap_idle_bytes` | Number of heap bytes waiting to be used. +`go_memstats_heap_inuse_bytes` | Number of heap bytes that are in use. + +## Health Metrics + +Health metrics let you check the health of a server node. + +:::note +Health metrics are only available for Dgraph Alpha server nodes. +::: + + Metric | Description + ------- | ----------- + `dgraph_alpha_health_status` | Value is 1 when the Alpha node is ready to accept requests; otherwise 0. + `dgraph_max_assigned_ts` | This shows the latest max assigned timestamp. All Alpha nodes within the same Alpha group should show the same timestamp if they are in sync. + `dgraph_txn_aborts_total` | Shows the total number of server-initiated transaction aborts that have occurred on the Alpha node. + `dgraph_txn_commits_total` | Shows the total number of successful commits that have occurred on the Alpha node. + `dgraph_txn_discards_total` | Shows the total number of client-initiated transaction discards that have occurred on the Alpha node. This is incremented when the client calls for a transaction discard, such as using the Dgraph Go client's `txn.Discard` function. + +## Memory metrics + +Memory metrics let you track the memory usage of the Dgraph process. The `idle` +and `inuse` metrics give you a better sense of the active memory usage of the +Dgraph process. The process memory metric shows the memory usage as measured by +the operating system. + +By looking at all three metrics you can see how much memory a Dgraph process is +holding from the operating system and how much is actively in use. + + Metric | Description + ------- | ----------- + `dgraph_memory_idle_bytes` | Estimated amount of memory that is being held idle that could be reclaimed by the OS. + `dgraph_memory_inuse_bytes` | Total memory usage in bytes (sum of heap usage and stack usage). + `dgraph_memory_proc_bytes` | Total memory usage in bytes of the Dgraph process. This metric is equivalent to resident set size on Linux. + +## Raft leadership metrics + +Raft leadership metrics let you track changes in Raft leadership for Dgraph +Alpha and Dgraph Zero nodes in your Cluster. These metrics include a group label +along with the node name, so that you can determine which metrics apply to which +Raft groups. + +Metric | Description +------- | ----------- +`dgraph_raft_has_leader` | Value is 1 when the node has a leader; otherwise 0. +`dgraph_raft_is_leader` | Value is 1 when the node is the leader of its group; otherwise 0. +`dgraph_raft_leader_changes_total` | The total number of leader changes seen by this node. diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/admin/observability/monitoring.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/admin/observability/monitoring.md new file mode 100644 index 00000000..9b330795 --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/admin/observability/monitoring.md @@ -0,0 +1,112 @@ +--- +title: Monitoring +--- + +Dgraph exposes metrics via the `/debug/vars` endpoint in json format and the `/debug/prometheus_metrics` endpoint in Prometheus's text-based format. Dgraph doesn't store the metrics and only exposes the value of the metrics at that instant. You can either poll this endpoint to get the data in your monitoring systems or install **[Prometheus](https://prometheus.io/docs/introduction/install/)**. Replace targets in the below config file with the ip of your Dgraph instances and run prometheus using the command `prometheus --config.file my_config.yaml`. + +```sh +scrape_configs: + - job_name: "dgraph" + metrics_path: "/debug/prometheus_metrics" + scrape_interval: "2s" + static_configs: + - targets: + - 172.31.9.133:6080 # For Dgraph zero, 6080 is the http endpoint exposing metrics. + - 172.31.15.230:8080 # For Dgraph alpha, 8080 is the http endpoint exposing metrics. + - 172.31.0.170:8080 + - 172.31.8.118:8080 +``` + +:::note +Raw data exported by Prometheus is available via `/debug/prometheus_metrics` endpoint on Dgraph alphas. +::: + +Install **[Grafana](http://docs.grafana.org/installation/)** to plot the metrics. Grafana runs at port 3000 in default settings. Create a prometheus datasource by following these **[steps](https://prometheus.io/docs/visualization/grafana/#creating-a-prometheus-data-source)**. Import **[grafana_dashboard.json](https://github.com/dgraph-io/benchmarks/blob/master/scripts/grafana_dashboard.json)** by following this **[link](http://docs.grafana.org/reference/export_import/#importing-a-dashboard)**. + + +## CloudWatch + +Route53's health checks can be leveraged to create standard CloudWatch alarms to notify on change in the status of the `/health` endpoints of Alpha and Zero. + +Considering that the endpoints to monitor are publicly accessible and you have the AWS credentials and [awscli](https://aws.amazon.com/cli/) setup, we’ll go through an example of setting up a simple CloudWatch alarm configured to alert via email for the Alpha endpoint `alpha.acme.org:8080/health`. Dgraph Zero's `/health` endpoint can also be monitored in a similar way. + + + +### Create the Route53 Health Check +```sh +aws route53 create-health-check \ + --caller-reference $(date "+%Y%m%d%H%M%S") \ + --health-check-config file:///tmp/create-healthcheck.json \ + --query 'HealthCheck.Id' +``` +The file `/tmp/create-healthcheck.json` would need to have the values for the parameters required to create the health check as such: +```sh +{ + "Type": "HTTPS", + "ResourcePath": "/health", + "FullyQualifiedDomainName": "alpha.acme.org", + "Port": 8080, + "RequestInterval": 30, + "FailureThreshold": 3 +} +``` +The reference for the values one can specify while creating or updating a health check can be found on the AWS [documentation](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/health-checks-creating-values.html). + +The response to the above command would be the ID of the created health check. +```sh +"29bdeaaa-f5b5-417e-a5ce-7dba1k5f131b" +``` +Make a note of the health check ID. This will be used to integrate CloudWatch alarms with the health check. + +:::note +Currently, Route53 metrics are only (available)[https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/monitoring-health-checks.html] in the **US East (N. Virginia)** region. The Cloudwatch Alarm (and the SNS Topic) should therefore be created in `us-east-1`. +::: + +### [Optional] Creating an SNS Topic +SNS topics are used to create message delivery channels. If you do not have any SNS topics configured, one can be created by running the following command: + +```sh +aws sns create-topic --region=us-east-1 --name ops --query 'TopicArn' +``` + +The response to the above command would be as follows: +```sh +"arn:aws:sns:us-east-1:123456789012:ops" +``` +Be sure to make a note of the topic ARN. This would be used to configure the CloudWatch alarm's action parameter. + +Run the following command to subscribe your email to the SNS topic: +```sh +aws sns subscribe \ + --topic-arn arn:aws:sns:us-east-1:123456789012:ops \ + --protocol email \ + --notification-endpoint ops@acme.org +``` +The subscription will need to be confirmed through *AWS Notification - Subscription Confirmation* sent through email. Once the subscription is confirmed, CloudWatch can be configured to use the SNS topic to trigger the alarm notification. + + + +### Creating a CloudWatch Alarm +The following command creates a CloudWatch alarm with `--alarm-actions` set to the ARN of the SNS topic and the `--dimensions` of the alarm set to the health check ID. +```sh +aws cloudwatch put-metric-alarm \ + --region=us-east-1 \ + --alarm-name dgraph-alpha \ + --alarm-description "Alarm for when Alpha is down" \ + --metric-name HealthCheckStatus \ + --dimensions "Name=HealthCheckId,Value=29bdeaaa-f5b5-417e-a5ce-7dba1k5f131b" \ + --namespace AWS/Route53 \ + --statistic Minimum \ + --period 60 \ + --threshold 1 \ + --comparison-operator LessThanThreshold \ + --evaluation-periods 1 \ + --treat-missing-data breaching \ + --alarm-actions arn:aws:sns:us-east-1:123456789012:ops +``` + +One can verify the alarm status from the CloudWatch or Route53 consoles. + +#### Internal Endpoints +If the Alpha endpoint is internal to the VPC network - one would need to create a Lambda function that would periodically (triggered using CloudWatch Event Rules) request the `/health` path and create CloudWatch metrics which could then be used to create the required CloudWatch alarms. +The architecture and the CloudFormation template to achieve the same can be found [here](https://aws.amazon.com/blogs/networking-and-content-delivery/performing-route-53-health-checks-on-private-resources-in-a-vpc-with-aws-lambda-and-amazon-cloudwatch/). diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/admin/observability/tracing.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/admin/observability/tracing.md new file mode 100644 index 00000000..0ff56dea --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/admin/observability/tracing.md @@ -0,0 +1,95 @@ +--- +title: Tracing +--- + +Dgraph uses [OpenTelemetry](https://opentelemetry.io/) for distributed tracing across the Dgraph cluster. + +Trace data is always collected within Dgraph. You can adjust the trace sampling rate for Dgraph queries using the `--trace` [superflag's](../../cli/superflags) `ratio` option when running Dgraph Alpha and Zero nodes. By default, `--trace ratio` is set to 0.01 to trace 1% of queries. + +## Trace Superflag Options + +The `--trace` superflag supports the following options: + +| Option | Default | Description | +|--------|---------|-------------| +| `ratio` | `0.01` | The ratio of queries to trace (0.0 to 1.0). Set to `1.0` to trace all queries. | +| `jaeger` | (empty) | URL of Jaeger or OpenTelemetry Collector OTLP HTTP endpoint (e.g., `http://localhost:4318`). | +| `datadog` | (empty) | URL of Datadog agent to send traces. | +| `service` | (empty) | Custom service name for tracing. If set, overrides the default (`dgraph.alpha` or `dgraph.zero`). | + +Example usage: + +```bash +dgraph alpha --trace "ratio=1.0; jaeger=http://localhost:4318; service=alpha1;" +dgraph zero --trace "ratio=1.0; jaeger=http://localhost:4318; service=zero1;" +``` + +## Examining Traces with zPages + +The most basic way to view traces is with the integrated trace pages. + +OpenTelemetry's [zPages](https://opentelemetry.io/docs/languages/go/instrumentation/#zpages) are accessible via the Zero or Alpha HTTP port at `/debug/z`. + +## Examining Traces with Jaeger + +Jaeger collects distributed traces and provides a UI to view and query traces across different services. This provides the necessary observability to figure out what is happening in the system. + +Dgraph exports traces to Jaeger using the [OTLP HTTP protocol](https://opentelemetry.io/docs/specs/otlp/). Configure the `--trace jaeger` option to point to the Jaeger OTLP HTTP endpoint (port 4318 by default): + +```bash +dgraph alpha --trace "jaeger=http://localhost:4318;" +dgraph zero --trace "jaeger=http://localhost:4318;" +``` + +See [Jaeger's Getting Started docs](https://www.jaegertracing.io/docs/getting-started/) to get up and running with Jaeger. + +### Setting up multiple Dgraph clusters with Jaeger + +Jaeger allows you to examine traces from multiple Dgraph clusters. Use the `--trace service` option to give each Dgraph instance a unique service name: + +```bash +# QA cluster +dgraph alpha --trace "jaeger=http://jaeger:4318; service=alpha-qa;" +dgraph zero --trace "jaeger=http://jaeger:4318; service=zero-qa;" + +# Dev cluster +dgraph alpha --trace "jaeger=http://jaeger:4318; service=alpha-dev;" +dgraph zero --trace "jaeger=http://jaeger:4318; service=zero-dev;" +``` + +Dgraph also sets a `service.namespace` attribute on all spans, which preserves the original service type (`dgraph.alpha` or `dgraph.zero`) even when using custom service names. This allows filtering by namespace in the Jaeger UI. + +Once you have this configured, you can filter by service name in the Jaeger UI: + +![Jaeger UI](/images/jaeger-ui.png) + +Every trace shows the service name under the "Process" section of each span: + +![Jaeger Query](/images/jaeger-server-query.png) + +![Jaeger JSON](/images/jaeger-json.png) + +![Jaeger Query Result](/images/jaeger-server-query-2.png) + +To learn more about Jaeger, see [Jaeger's Deployment Guide](https://www.jaegertracing.io/docs/deployment/). + +## Using an OpenTelemetry Collector + +Instead of sending traces directly to Jaeger, you can route them through an [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/) for additional processing, filtering, or forwarding to multiple backends: + +```bash +dgraph alpha --trace "jaeger=http://otel-collector:4318;" +``` + +The Collector can then be configured to export traces to Jaeger, Zipkin, Datadog, or any other supported backend. + +## Using Grafana Tempo + +[Grafana Tempo](https://grafana.com/oss/tempo/) is a high-scale distributed tracing backend that natively supports OTLP. Configure Dgraph to send traces directly to Tempo's OTLP HTTP endpoint: + +```bash +dgraph alpha --trace "jaeger=http://tempo:4318;" +dgraph zero --trace "jaeger=http://tempo:4318;" +``` + +Traces can then be visualized in Grafana using the Tempo data source. diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/admin/security/admin-endpoint-security.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/admin/security/admin-endpoint-security.md new file mode 100644 index 00000000..7ff5c017 --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/admin/security/admin-endpoint-security.md @@ -0,0 +1,133 @@ +--- +title: Admin Endpoint Security +--- + +Dgraph Alpha exposes various administrative endpoints over HTTP and GraphQL for operations like data export and cluster shutdown. All admin endpoints are protected by three layers of authentication. + +## Authentication Layers + +Admin endpoints require authentication through three layers: + +1. **IP Whitelisting** - Use the `--security` superflag's `whitelist` option on Dgraph Alpha to whitelist IP addresses other than localhost. +2. **Token Authentication** - If Dgraph Alpha is started with the `--security` superflag's `token` option, you must pass the token as an `X-Dgraph-AuthToken` header when making HTTP requests. +3. **ACL Guardian Access** - If ACL is enabled, you must pass the ACL-JWT of a Guardian user using the `X-Dgraph-AccessToken` header when making HTTP requests. + +## Admin Endpoints + +An admin endpoint is any HTTP endpoint which provides admin functionality. Admin endpoints usually start with the `/admin` path. The current list of admin endpoints includes: + +* `/admin` +* `/admin/config/cache_mb` +* `/admin/draining` +* `/admin/shutdown` +* `/admin/schema` +* `/admin/schema/validate` +* `/alter` +* `/login` + +### Special Endpoints + +There are exceptions to the general authentication rule: + +* **`/login`**: This endpoint logs-in an ACL user and provides them with a JWT. Only IP Whitelisting and token authentication checks are performed for this endpoint. +* **`/admin`**: This GraphQL endpoint provides queries/mutations corresponding to the HTTP admin endpoints. All queries/mutations on `/admin` have all three layers of authentication, except for `login (mutation)`, which has the same behavior as the HTTP `/login` endpoint. + +## IP Whitelisting + +By default, admin operations can only be initiated from the machine on which the Dgraph Alpha runs. + +You can use the `--security` superflag's `whitelist` option to specify a comma-separated whitelist of IP addresses, IP ranges, CIDR ranges, or hostnames for hosts from which admin operations can be initiated. + +**Examples:** + +```sh +# Allow localhost only +dgraph alpha --security whitelist=127.0.0.1 ... + +# Allow IP range and specific IP +dgraph alpha --security whitelist=172.17.0.0:172.20.0.0,192.168.1.1 ... + +# Allow CIDR ranges +dgraph alpha --security whitelist=172.17.0.0/16,192.168.1.1/32 ... + +# Allow hostnames +dgraph alpha --security whitelist=admin-bastion,host.docker.internal ... + +# Allow all IPs (not recommended for production) +dgraph alpha --security whitelist=0.0.0.0/0 ... +``` + +For detailed network security configuration including TLS and port usage, see [Ports Usage](ports-usage) and [TLS Configuration](tls-configuration). + +## Token Authentication + +Token authentication provides a simple way to secure admin endpoints without full ACL. This is sometimes called "poor-man's auth" and is useful for basic protection. + +### Setting Up Token Authentication + +Specify the auth token with the `--security` superflag's `token` option for each Dgraph Alpha in the cluster: + +```sh +dgraph alpha --security token= +``` + +### Using Token Authentication + +Clients must include the same auth token in the `X-Dgraph-AuthToken` header when making admin requests: + +```sh +# Without token - will be denied +curl -s localhost:8080/alter -d '{ "drop_all": true }' +# Permission denied. No token provided. + +# With wrong token - will be denied +curl -s -H 'X-Dgraph-AuthToken: ' localhost:8080/alter -d '{ "drop_all": true }' +# Permission denied. Incorrect token. + +# With correct token - will succeed +curl -H 'X-Dgraph-AuthToken: ' localhost:8080/alter -d '{ "drop_all": true }' +# Success. Token matches. +``` + +:::note +To fully secure admin operations in the cluster, the authentication token must be set for every Alpha node. +::: + +## Securing Alter Operations + +Alter operations allow clients to apply schema updates and drop predicates from the database. By default, all clients are allowed to perform alter operations, which can be a security risk. + +You can configure Dgraph to only allow alter operations when the client provides a specific token. This prevents clients from making unintended or accidental schema updates or predicate drops. + +See the [Token Authentication](#token-authentication) section above for setup instructions. Once configured, all alter operations require the `X-Dgraph-AuthToken` header. + +For enterprise-grade access control, see [Enable ACL](../../installation/configuration/enable-acl) and [User Management and Access Control](../admin-tasks/user-management-access-control). + +## Zero admin endpoints + +Dgraph Zero exposes its own administrative endpoints over its HTTP port (default `6080`): + +* `/state` - cluster topology and tablet placement +* `/assign` - allocate UIDs, transaction timestamps, and namespace IDs +* `/removeNode` - remove a node from a Raft group +* `/moveTablet` - move a predicate (tablet) between groups + +These endpoints drive cluster membership and coordination. Zero's HTTP port is an internal control-plane port and should not be reachable from untrusted networks. Restrict access to it with firewall rules or network policies, alongside the authentication described below. + +Zero authenticates callers with the same `--security` superflag `token` and `whitelist` options used by Alpha: + +* **Token authentication** - Set `--security "token="` on Zero. Callers must then pass the token in the `X-Dgraph-AuthToken` header. +* **IP whitelisting** - Set `--security "whitelist=..."` on Zero to allow specific source IPs, IP ranges, CIDR blocks, or hostnames. Loopback is always allowed. + +Protection applies in two tiers: + +* The destructive endpoints (`/removeNode`, `/moveTablet`) are always guarded. With neither a token nor a whitelist configured, only loopback callers are allowed, so a remote caller cannot disrupt the control plane by default. +* The informational and allocation endpoints (`/state`, `/assign`) are enforced only once a `token` or `whitelist` is configured, so existing tooling that reads them over HTTP is unaffected until you opt in. + +```sh +# Require a token, and allow an internal subnet to reach all admin endpoints +dgraph zero --security "whitelist=10.0.0.0/8;token=" +``` + +To disable the Zero admin HTTP endpoints entirely, set `--limit "disable-admin-http=true"`. The `/health` endpoint stays available. + diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/admin/security/index.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/admin/security/index.md new file mode 100644 index 00000000..6a589ff9 --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/admin/security/index.md @@ -0,0 +1,22 @@ +--- +title: Security +--- + +Dgraph security configuration covers authentication, network security, and access control for your cluster. + +## Security Configuration + +**[Admin Endpoint Security](admin-endpoint-security)** - Authentication layers for admin endpoints, IP whitelisting, and token-based authentication. + +**[Ports Usage](ports-usage)** - Understanding Dgraph's port configuration and network security requirements. + +**[TLS Configuration](tls-configuration)** - Encrypting communications between Dgraph nodes and clients using TLS/mTLS. + + +**[Enable ACL](../../installation/configuration/enable-acl)** - Configure and enable Access Control Lists + +**[User Management and Access Control](../admin-tasks/user-management-access-control)** - Manage users, groups, and ACL rules + +**[Audit Logging](../observability/audit-logs)** - Track and audit all requests + +**[Encryption at Rest](../../installation/configuration/encryption-at-rest)** - Encrypt data on disk \ No newline at end of file diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/admin/security/ports-usage.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/admin/security/ports-usage.md new file mode 100644 index 00000000..bbe22709 --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/admin/security/ports-usage.md @@ -0,0 +1,65 @@ +--- +title: Ports Usage +--- + +Dgraph cluster nodes use a range of ports to communicate over gRPC and HTTP. +Choose these ports carefully based on your topology and mode of deployment, as +this will impact the access security rules or firewall configurations required +for each port. + +## Types of ports + +Dgraph Alpha and Dgraph Zero nodes use a variety of gRPC and HTTP ports, as +follows: + +- **gRPC-internal-private**: Used between the cluster nodes for internal + communication and message exchange. Communication using these ports is + TLS-encrypted. +- **gRPC-external-private**: Used by Dgraph Live Loader and Dgraph Bulk loader + to access APIs over gRPC. +- **gRPC-external-public**: Used by Dgraph clients to access APIs in a session + that can persist after a query. +- **HTTP-external-private**: Used for monitoring and administrative tasks. +- **HTTP-external-public:** Used by clients to access APIs over HTTP. + +## Default ports used by different nodes + + Dgraph Node Type | gRPC-internal-private | gRPC-external-private | gRPC-external-public | HTTP-external-private | HTTP-external-public +------------------|------------------------|-----------------------|----------------------|-----------------------|--------------------- + zero | 50801 | 50801 | | 60802 | + alpha | 7080 | | 9080 | | 8080 + ratel | | | | | 8000 + + +1: Dgraph Zero uses port 5080 for internal communication within the + cluster, and to support the [data import](../../migration/import-data) + tools: Dgraph Live Loader and Dgraph Bulk Loader. + +2: Dgraph Zero uses port 6080 for administrative operations. +Dgraph clients cannot access this port. + +Users must modify security rules or open firewall ports depending upon their +underlying network to allow communication between cluster nodes, between the +Dgraph instances, and between Dgraph clients. In general, you should configure +the gRPC and HTTP `external-public` ports for open access by Dgraph clients, +and configure the gRPC-internal ports for open access by the cluster nodes. + +**Ratel UI** accesses Dgraph Alpha on the `HTTP-external-public port` (which defaults to localhost:8080) and can be configured to talk to a remote Dgraph cluster. This +way you can run Ratel on your local machine and point to a remote cluster. But, +if you are deploying Ratel along with Dgraph cluster, then you may have to +expose port 8000 to the public. + +**Port Offset** To make it easier for users to set up a cluster, Dgraph has +default values for the ports used by Dgraph nodes. To support multiple nodes +running on a single machine or VM, you can set a node to use different ports +using an offset (using the command option `--port_offset`). This command +increments the actual ports used by the node by the offset value provided. You +can also use port offsets when starting multiple Dgraph Zero nodes in a +development environment. + +For example, when a user runs Dgraph Alpha with the `--port_offset 2` setting, +then the Alpha node binds to port 7082 (`gRPC-internal-private`), 8082 +(`HTTP-external-public`) and 9082 (`gRPC-external-public`), respectively. + +**Ratel UI** by default listens on port 8000. You can use the `-port` flag to +configure it to listen on any other port. diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/admin/security/tls-configuration.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/admin/security/tls-configuration.md new file mode 100644 index 00000000..c2339768 --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/admin/security/tls-configuration.md @@ -0,0 +1,395 @@ +--- +title: TLS Configuration +--- + +Connections between Dgraph database and its clients can be secured using TLS. In +addition, Dgraph can now secure gRPC communications between Dgraph Alpha and +Dgraph Zero server nodes using mutual TLS (mTLS). Dgraph can now also secure +communications over the Dgraph Zero `gRPC-external-private` port used by +Dgraph's Live Loader and Bulk Loader clients. To learn more about the HTTP and +gRPC ports used by Dgraph Alpha and Dgraph Zero, see [Ports Usage](ports-usage). +Password-protected private keys are **not supported**. + +To further improve TLS security, only TLS v1.2 cypher suites that use 128-bit or +greater RSA or AES encryption are supported. + +:::tip +If you're generating encrypted private keys with `openssl`, +be sure to specify the encryption algorithm explicitly (like `-aes256`). This will +force `openssl` to include `DEK-Info` header in private key, which is required +to decrypt the key by Dgraph. When default encryption is used, `openssl` doesn't +write that header and key can't be decrypted. +::: + +## Dgraph Certificate Management Tool + +:::note +This section refers to the `dgraph cert` command which was introduced in v1.0.9. +For previous releases, see the previous [TLS configuration documentation](https://github.com/dgraph-io/dgraph/blob/release/v1.0.7/wiki/content/deploy/index.md#tls-configuration). +::: + +The `dgraph cert` program creates and manages CA-signed certificates and private +keys using a generated Dgraph Root CA. There are three types of certificate/key +pairs: +1. Root CA certificate/key pair: This is used to sign and verify node and client + certificates. If the root CA certificate is changed then you must regenerate + all certificates, and this certificate must be accessible to the Alpha nodes. +2. Node certificate/key pair: This is shared by the Dgraph Alpha nodes and used + for accepting TLS connections. +3. Client certificate/key pair: This is used by the clients (like live loader + and Ratel) to communicate with Dgraph Alpha server nodes where client + authentication with mTLS is required. + +```sh +# To see the available flags. +$ dgraph cert --help + +# Create Dgraph Root CA, used to sign all other certificates. +$ dgraph cert + +# Create node certificate and private key +$ dgraph cert -n localhost + +# Create client certificate and private key for mTLS (mutual TLS) +$ dgraph cert -c dgraphuser + +# Combine all in one command +$ dgraph cert -n localhost -c dgraphuser + +# List all your certificates and keys +$ dgraph cert ls +``` + +The default location where the _cert_ command stores certificates (and keys) is +`tls` under the Dgraph working directory. The default directory path can be overridden +using the `--dir` option. For example: + +```sh +$ dgraph cert --dir ~/mycerts +``` + +### File naming conventions + +The following file naming conventions are used by Dgraph for proper TLS setup. + +| File name | Description | Use | +|-----------|-------------|-------| +| ca.crt | Dgraph Root CA certificate | Verify all certificates | +| ca.key | Dgraph CA private key | Validate CA certificate | +| node.crt | Dgraph node certificate | Shared by all nodes for accepting TLS connections | +| node.key | Dgraph node private key | Validate node certificate | +| client._name_.crt | Dgraph client certificate | Authenticate a client _name_ | +| client._name_.key | Dgraph client private key | Validate _name_ client certificate | + +For client authentication, each client must have their own certificate and key. +These are then used to connect to the Dgraph server nodes. + +The node certificate `node.crt` can support multiple node names using multiple +host names and/or IP address. Just separate the names with commas when +generating the certificate. + +```sh +$ dgraph cert -n localhost,104.25.165.23,dgraph.io,2400:cb00:2048:1::6819:a417 +``` + +:::tip +You must delete the old node cert and key before you can generate a new pair. +::: + +:::note +When using host names for node certificates, including _localhost_, your clients must connect to the matching host name -- such as _localhost_ not 127.0.0.1. If you need to use IP addresses, then add them to the node certificate. +::: + +### Certificate inspection + +The command `dgraph cert ls` lists all certificates and keys in the `--dir` +directory (default `dgraph-tls`), along with details to inspect and validate +cert/key pairs. + +Example of command output: + +```sh +-rw-r--r-- ca.crt - Dgraph Root CA certificate + Issuer: Dgraph Labs, Inc. + S/N: 043c4d8fdd347f06 + Expiration: 02 Apr 29 16:56 UTC +SHA-256 Digest: 4A2B0F0F 716BF5B6 C603E01A 6229D681 0B2AFDC5 CADF5A0D 17D59299 116119E5 + +-r-------- ca.key - Dgraph Root CA key +SHA-256 Digest: 4A2B0F0F 716BF5B6 C603E01A 6229D681 0B2AFDC5 CADF5A0D 17D59299 116119E5 + +-rw-r--r-- client.admin.crt - Dgraph client certificate: admin + Issuer: Dgraph Labs, Inc. + CA Verify: PASSED + S/N: 297e4cb4f97c71f9 + Expiration: 03 Apr 24 17:29 UTC +SHA-256 Digest: D23EFB61 DE03C735 EB07B318 DB70D471 D3FE8556 B15D084C 62675857 788DF26C + +-rw------- client.admin.key - Dgraph Client key +SHA-256 Digest: D23EFB61 DE03C735 EB07B318 DB70D471 D3FE8556 B15D084C 62675857 788DF26C + +-rw-r--r-- node.crt - Dgraph Node certificate + Issuer: Dgraph Labs, Inc. + CA Verify: PASSED + S/N: 795ff0e0146fdb2d + Expiration: 03 Apr 24 17:00 UTC + Hosts: 104.25.165.23, 2400:cb00:2048:1::6819:a417, localhost, dgraph.io +SHA-256 Digest: 7E243ED5 3286AE71 B9B4E26C 5B2293DA D3E7F336 1B1AFFA7 885E8767 B1A84D28 + +-rw------- node.key - Dgraph Node key +SHA-256 Digest: 7E243ED5 3286AE71 B9B4E26C 5B2293DA D3E7F336 1B1AFFA7 885E8767 B1A84D28 +``` + +Important points: + +* The cert/key pairs should always have matching SHA-256 digests. Otherwise, the cert(s) must be + regenerated. If the Root CA pair differ, all cert/key must be regenerated; the flag `--force` + can help. +* All certificates must pass Dgraph CA verification. +* All key files should have the least access permissions, especially the `ca.key`, but be readable. +* Key files won't be overwritten if they have limited access, even with `--force`. +* Node certificates are only valid for the hosts listed. +* Client certificates are only valid for the named client/user. + +## TLS options + +Starting in release v21.03, pre-existing TLS configuration options have been +replaced by the `--tls` [superflag](../../cli/superflags) +and its options. The following `--tls` configuration options are available for +Dgraph Alpha and Dgraph Zero nodes: + +* `ca-cert ` - Path and filename of the Dgraph Root CA (for + example, `ca.crt`) +* `server-cert ` - Path and filename of the node certificate (for + example, `node.crt`) +* `server-key ` - Path and filename of the node certificate private + key (for example, `node.key`) +* `use-system-ca` - Include System CA with Dgraph Root CA. +* `client-auth-type ` - TLS client authentication used to validate client + connections from external ports. To learn more, see + [Client Authentication Options](#client-authentication-options). + +:::note +Dgraph now allows you to specify the path and filename of the CA root +certificate, the node certificate, and the node certificate private key. So, +these files do not need to have specific filenames or exist in the same +directory, as in previous Dgraph versions that used the `--tls_dir` flag. +::: + +You can configure Dgraph Live Loader with the following `--tls` options: + +* `ca-cert ` - Dgraph root CA, such as `./tls/ca.crt` +* `use-system-ca` - Include System CA with Dgraph Root CA. +* `client-cert` - User cert file provided by the client to Alpha +* `client-key` - User private key file provided by the client to Alpha +* `server-name ` - Server name, used for validating the server's TLS host name. + + +### Using TLS with only external ports encrypted + +To encrypt communication between Dgraph server nodes and clients over external +ports, you can configure certificates and run Dgraph Alpha and Dgraph Zero using +the following commands: + +Dgraph Alpha: + +```sh +# First, create the root CA, Alpha node certificate and private keys, if not already created. +# Note that you must specify in node.crt the host name or IP addresses that clients use connect: +$ dgraph cert -n localhost,104.25.165.23,104.25.165.25,104.25.165.27 +# Set up Dgraph Alpha nodes using the following default command (after generating certificates and private keys) +$ dgraph alpha --tls "ca-cert=/dgraph-tls/ca.crt; server-cert=/dgraph-tls/node.crt; server-key=/dgraph-tls/node.key" +``` + +Dgraph Zero: + +```sh +# First, copy the root CA, node certificates and private keys used to set up Dgraph Alpha (above) to the Dgraph Zero node. +# Optionally, you can generate and use a separate Zero node certificate, where you specify the host name or IP addresses used by Live Loader and Bulk Loader to connect to Dgraph Zero. +# Next, set up Dgraph Zero nodes using the following default command: +$ dgraph zero --tls "ca-cert=/dgraph-tls/ca.crt; server-cert=/dgraph-tls/node.crt; server-key=/dgraph-tls/node.key" +``` + +You can then run Dgraph Live Loader on a Dgraph Alpha node using the following command: + +```sh +# Now, connect to server using TLS +$ dgraph live --tls "ca-cert=./dgraph-tls/ca.crt; server-name=localhost" -s 21million.schema -f 21million.rdf.gz +``` + +### Using TLS with internal and external ports encrypted + +If you require client authentication (mutual TLS, or mTLS), you can configure +certificates and run Dgraph Alpha and Dgraph Zero with settings that encrypt +both internal ports (those used within the cluster) as well as external ports +(those used by clients that connect to the cluster, including Bulk Loader and +Live Loader). + +The following example shows how to encrypt both internal and external ports: + +Dgraph Alpha: + +```sh +# First create the root CA, node certificates and private keys, if not already created. +# Note that you must specify the host name or IP address for other nodes that will share node.crt. +$ dgraph cert -n localhost,104.25.165.23,104.25.165.25,104.25.165.27 +# Set up Dgraph Alpha nodes using the following default command (after generating certificates and private keys) +$ dgraph alpha + --tls "ca-cert=/dgraph-tls/ca.crt; server-cert=/dgraph-tls/node.crt; server-key=/dgraph-tls/node.key; +internal-port=true; client-cert=/dgraph-tls/client.alpha1.crt; client-key=/dgraph-tls/client.alpha1.key" +``` + +Dgraph Zero: + +```sh +# First, copy the certificates and private keys used to set up Dgraph Alpha (above) to the Dgraph Zero node. +# Next, set up Dgraph Zero nodes using the following default command: +$ dgraph zero + --tls "ca-cert=/dgraph-tls/ca.crt; server-cert=/dgraph-tls/node.crt; server-key=/dgraph-tls/node.key; internal-port=true; client-cert=/dgraph-tls/client.zero1.crt; client-key=/dgraph-tls/client.zero1.key" +``` + +You can then run Dgraph Live Loader using the following: + +```sh +# Now, connect to server using mTLS (mutual TLS) +$ dgraph live + --tls "ca-cert=./tls/ca.crt; client-cert=./tls/client.dgraphuser.crt; client-key=./tls/client.dgraphuser.key; server-name=localhost; internal-port=true" \ + -s 21million.schema \ + -f 21million.rdf.gz +``` + +### Client Authentication Options + +The server will always **request** client authentication. There are four +different values for the `client-auth-type` option that change the security +policy of the client certificate. + +| Value | Client Cert/Key | Client Certificate Verified | +|--------------------|-----------------|--------------------| +| `REQUEST` | optional | Client certificate is not VERIFIED if provided. (least secure) | +| `REQUIREANY` | required | Client certificate is never VERIFIED | +| `VERIFYIFGIVEN` | optional | Client certificate is VERIFIED if provided (default) | +| `REQUIREANDVERIFY` | required | Client certificate is always VERIFIED (most secure) | + +`REQUIREANDVERIFY` is the most secure but also the most difficult to configure +for clients. When using this value, the value of `server-name` is matched +against the certificate SANs values and the connection host. + +:::note +If mTLS is enabled using `internal-port=true`, +internal ports (by default, 5080 and 7080) use the `REQUIREANDVERIFY` setting. +Unless otherwise configured, external ports (by default, 9080, 8080 and 6080) +use the `VERIFYIFGIVEN` setting. Changing the `client-auth-type` option to +another setting only affects client authentication on external ports. +::: + +## Using Ratel UI with Client authentication + +Ratel UI (and any other JavaScript clients built on top of `dgraph-js-http`) +connect to Dgraph servers via HTTP, when TLS is enabled servers begin to expect +HTTPS requests only. + +If you haven't already created the CA certificate and the node certificate for alpha servers from the earlier instructions (see [Dgraph Certificate Management Tool](#dgraph-certificate-management-tool)), the first step would be to generate these certificates, it can be done by the following command: +```sh +# Create rootCA and node certificates/keys +$ dgraph cert -n localhost +``` + +If Dgraph Alpha's `client-auth-type` option is set to `REQUEST` or `VERIFYIFGIVEN` +(default), then client certificate is not mandatory. The steps after generating +CA/node certificate are as follows: + +### Step 1. Install Dgraph Root CA into System CA +##### Linux (Debian/Ubuntu) +```sh +# Copy the generated CA to the ca-certificates directory +$ cp /path/to/ca.crt /usr/local/share/ca-certificates/ca.crt +# Update the CA store +$ sudo update-ca-certificates` +``` + +### Step 2. Install Dgraph Root CA into Web Browsers Trusted CA List + +##### Firefox + +* Choose Preferences -> Privacy & Security -> View Certificates -> Authorities +* Click on Import and import the `ca.crt` + +##### Chrome + +* Choose Settings -> Privacy and Security -> Security -> Manage Certificates -> Authorities +* Click on Import and import the `ca.crt` + +### Step 3. Point Ratel to the `https://` endpoint of alpha server. + +* Change the Dgraph Alpha server address to `https://` instead of `http://`, for example `https://localhost:8080`. + +For `REQUIREANY` and `REQUIREANDVERIFY` as `client-auth-type` option, you need to follow the steps above and you +also need to install client certificate on your browser: + +1. Generate a client certificate: `dgraph cert -c laptopuser`. +2. Convert it to a `.p12` file: + ```sh + openssl pkcs12 -export \ + -out laptopuser.p12 \ + -in tls/client.laptopuser.crt \ + -inkey tls/client.laptopuser.key + ``` + Use any password you like for export, it is used to encrypt the p12 file. + +3. Import the client certificate to your browser. It can be done in Chrome as follows: + * Choose Settings -> Privacy and Security -> Security -> Manage Certificates -> Your Certificates + * Click on Import and import the `laptopuser.p12`. + +:::note +Mutual TLS may not work in Firefox because Firefox is unable to send privately-signed client certificates, this issue is filed [here](https://bugzilla.mozilla.org/show_bug.cgi?id=1662607). +::: + + +Next time you use Ratel to connect to an alpha with Client authentication +enabled the browser will prompt you for a client certificate to use. Select the client's +certificate you've imported in the step above and queries/mutations will +succeed. + +## Using Curl with Client authentication + +When TLS is enabled, `curl` requests to Dgraph will need some specific options to work. +For instance (for changing draining mode): + +``` +curl --silent https://localhost:8080/admin/draining +``` + +If you are using `curl` with [Client Authentication](#client-authentication-options) set to `REQUIREANY` or `REQUIREANDVERIFY`, you will need to provide the client certificate and private key. For instance (for an export request): + +``` +curl --silent --cacert ./tls/ca.crt --cert ./tls/client.dgraphuser.crt --key ./tls/client.dgraphuser.key https://localhost:8080/admin/draining +``` + +Refer to the `curl` documentation for further information on its TLS options. + +## Access Data Using a Client + +Some examples of connecting via a [Client](../../clients) when TLS is in use can be found below: + +- [dgraph4j](https://github.com/dgraph-io/dgraph4j#creating-a-secure-client-using-tls) +- [dgraph-js](https://github.com/dgraph-io/dgraph-js/tree/master/examples/tls) +- [dgo](https://github.com/dgraph-io/dgraph/blob/main/tlstest/acl/acl_over_tls_test.go) +- [pydgraph](https://github.com/dgraph-io/pydgraph/tree/master/examples/tls) + +## Troubleshooting Ratel's Client authentication + +If you are getting errors in Ratel when TLS is enabled, try opening your Dgraph +Alpha URL as a web page. + +Assuming you are running Dgraph on your local machine, opening +`https://localhost:8080/` in the browser should produce a message `Dgraph browser is available for running separately using the dgraph-ratel binary`. + +In case you are getting a connection error, try not passing the +`client-auth-type` flag when starting an alpha. If you are still getting an +error, check that your hostname is correct and the port is open; then make sure +that "Dgraph Root CA" certificate is installed and trusted correctly. + +After that, if things work without passing `client-auth-type` but stop working when +`REQUIREANY` and `REQUIREANDVERIFY` are set, make sure the `.p12` file is +installed correctly. diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/admin/troubleshooting.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/admin/troubleshooting.md new file mode 100644 index 00000000..4ff61166 --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/admin/troubleshooting.md @@ -0,0 +1,70 @@ +--- +title: Troubleshooting +--- + +This page provides tips on how to troubleshoot issues with running Dgraph. + +## Cluster Setup Checklist + +When setting up a cluster, verify the following requirements are met: + +* Is at least one Dgraph Zero node running? +* Is each Dgraph Alpha instance in the cluster set up correctly? +* Will each Dgraph Alpha instance be accessible to all peers on 7080 (+ any port offset)? +* Does each instance have a unique ID on startup? +* Has `--bindall=true` been set for networked communication? + +See the [Production Checklist](../installation/production-checklist) for comprehensive deployment requirements. + +## Running out of memory (OOM) + +When you [bulk load](../migration/bulk-loader) +or [backup](admin-tasks/binary-backups) your data, +Dgraph can consume more memory than usual due to a high volume of writes. This +can cause OOM crashes. + +You can take the following steps to help avoid OOM crashes: + +* **Increase the amount of memory available**: If you run Dgraph with insufficient +memory, that can result in OOM crashes. The recommended minimum RAM to run Dgraph +on desktops and laptops (single-host deployment) is 16GB. For servers in a +cluster deployment, the recommended minimum is 8GB per server. This applies to +EC2 and GCE instances, as well as on-premises servers. +* **Reduce the number of Go routines**: You can troubleshoot OOM issues by reducing +the number of Go routines (`goroutines`) used by Dgraph from the default value +of eight. For example, you can reduce the `goroutines` that Dgraph uses to four +by calling the `dgraph alpha` command with the following option: + + `--badger "goroutines=4"` + +## "Too many open files" errors + +If Dgraph logs "too many open files" errors, you should increase the per-process +open file descriptor limit to permit more open files. During normal operations, +Dgraph must be able to open many files. Your operating system may have an open +file descriptor limit with a low default value that isn't adequate for a database +like Dgraph. If so, you might need to increase this limit. + +On Linux and Mac, you can get file descriptor limit settings with the `ulimit` +command, as follows: + +* Get hard limit: `ulimit -n -H` +* Get soft limit: `ulimit -n -S` + +A soft limit of `1048576` open files is the recommended minimum to use Dgraph in +production, but you can try increasing this soft limit if you continue to see +this error. To learn more, see the `ulimit` documentation for your operating +system. + +:::note +Depending on your OS, your shell session limits might not be the same as the Dgraph process limits. +::: + +For example, to properly set up the `ulimit` values on Ubuntu 20.04 systems: + +```sh +sudo sed -i 's/#DefaultLimitNOFILE=/DefaultLimitNOFILE=1048576/' /etc/systemd/system.conf +sudo sed -i 's/#DefaultLimitNOFILE=/DefaultLimitNOFILE=1048576/' /etc/systemd/user.conf +``` + +This affects the base limits for all processes. After a reboot, your OS will pick up the new values. diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/cli/acl.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/cli/acl.md new file mode 100644 index 00000000..c8a614be --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/cli/acl.md @@ -0,0 +1,40 @@ +--- +title: dgraph acl +--- + +#### `dgraph acl` + +This command runs the Dgraph Enterprise Edition ACL tool. The following replicates +the help listing shown when you run `dgraph acl --help`: + +```shell +Run the Dgraph Enterprise Edition ACL tool +Usage: + dgraph acl [command] + +Available Commands: + add Run Dgraph acl tool to add a user or group + del Run Dgraph acl tool to delete a user or group + info Show info about a user or group + mod Run Dgraph acl tool to modify a user's password, a user's group list, or agroup's predicate permissions + +Flags: + -a, --alpha string Dgraph Alpha gRPC server address (default "127.0.0.1:9080") + --guardian-creds string Login credentials for the guardian + user defines the username to login. + password defines the password of the user. + namespace defines the namespace to log into. + Sample flag could look like --guardian-creds user=username;password=mypass;namespace=2 + -h, --help help for acl + --tls string TLS Client options + ca-cert=; The CA cert file used to verify server certificates. Required for enabling TLS. + client-cert=; (Optional) The Cert file provided by the client to the server. + client-key=; (Optional) The private Key file provided by the clients to the server. + internal-port=false; (Optional) Enable inter-node TLS encryption between cluster nodes. + server-name=; Used to verify the server hostname. + use-system-ca=true; Includes System CA into CA Certs. + (default "use-system-ca=true; internal-port=false;") + +Use "dgraph acl [command] --help" for more information about a command. +``` + diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/cli/alpha.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/cli/alpha.md new file mode 100644 index 00000000..b2cd90e6 --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/cli/alpha.md @@ -0,0 +1,177 @@ +--- +title: dgraph alpha +--- + +The `dgraph alpha` command runs Dgraph Alpha database nodes, which store data and serve queries in your deployment. + +## Overview + +A Dgraph Alpha instance stores the data. Each Dgraph Alpha is responsible for storing and serving one data group. If multiple Alphas serve the same group, they form a Raft group and provide synchronous replication. + +## Usage + +```bash +dgraph alpha [flags] +``` + +## Key Flags + +| Flag | Description | Default | +|------|-------------|---------| +| `-p, --postings` | Directory to store posting lists | `"p"` | +| `-w, --wal` | Directory to store raft write-ahead logs | `"w"` | +| `--tmp` | Directory to store temporary buffers | `"t"` | +| `-z, --zero` | Comma separated list of Dgraph Zero addresses | `"localhost:5080"` | +| `--my` | Address:port of this server for cluster communication | | +| `-o, --port_offset` | Value added to all listening port numbers [Internal=7080, HTTP=8080, Grpc=9080] | `0` | +| `--export` | Folder in which to store exports | `"export"` | +| `--custom_tokenizers` | Comma separated list of tokenizer plugins for custom indices | | + +## Superflags + +Alpha uses several [superflags](superflags) for advanced configuration: + +- `--acl` - Access Control List settings (Enterprise) +- `--audit` - Audit logging configuration +- `--badger` - Badger database options +- `--cache` - Cache configuration +- `--cdc` - Change Data Capture options +- `--encryption` - Encryption at rest (Enterprise) +- `--graphql` - GraphQL settings +- `--limit` - Query and mutation limits +- `--raft` - Raft consensus options +- `--security` - Security settings (token, whitelist) +- `--telemetry` - Telemetry and crash reporting +- `--tls` - TLS configuration +- `--trace` - Distributed tracing +- `--vault` - HashiCorp Vault integration + +## Full Reference + +```shell +A Dgraph Alpha instance stores the data. Each Dgraph Alpha is responsible for +storing and serving one data group. If multiple Alphas serve the same group, +they form a Raft group and provide synchronous replication. + +Usage: + dgraph alpha [flags] + +Flags: + --acl string [Enterprise Feature] ACL options + access-ttl=6h; The TTL for the access JWT. + refresh-ttl=30d; The TTL for the refresh JWT. + secret-file=; The file that stores the HMAC secret, which is used for signing the JWT and should have at least 32 ASCII characters. Required to enable ACLs. + (default "access-ttl=6h; refresh-ttl=30d; secret-file=;") + --audit string Audit options + compress=false; Enables the compression of old audit logs. + days=10; The number of days audit logs will be preserved. + encrypt-file=; The path to the key file to be used for audit log encryption. + output=; [stdout, /path/to/dir] This specifies where audit logs should be output to. + "stdout" is for standard output. You can also specify the directory where audit logs + will be saved. When stdout is specified as output other fields will be ignored. + size=100; The audit log max size in MB after which it will be rolled over. + (default "compress=false; days=10; size=100; dir=; output=; encrypt-file=;") + --badger string Badger options + compression=snappy; [none, zstd:level, snappy] Specifies the compression algorithm and + compression level (if applicable) for the postings directory."none" would disable + compression, while "zstd:1" would set zstd compression at level 1. + numgoroutines=8; The number of goroutines to use in badger.Stream. + max-retries=-1; Commits to disk will give up after these number of retries to prevent locking the worker in a failed state. Use -1 to retry infinitely. + (default "compression=snappy; numgoroutines=8; max-retries=-1;") + --cache string Cache options + percentage=0,65,35; Cache percentages summing up to 100 for various caches (FORMAT: PostingListCache,PstoreBlockCache,PstoreIndexCache) + size-mb=1024; Total size of cache (in MB) to be used in Dgraph. + (default "size-mb=1024; percentage=0,65,35;") + --cdc string Change Data Capture options + ca-cert=; The path to CA cert file for TLS encryption. + client-cert=; The path to client cert file for TLS encryption. + client-key=; The path to client key file for TLS encryption. + file=; The path where audit logs will be stored. + kafka=; A comma separated list of Kafka hosts. + sasl-password=; The SASL password for Kafka. + sasl-user=; The SASL username for Kafka. + (default "file=; kafka=; sasl_user=; sasl_password=; ca_cert=; client_cert=; client_key=;") + --custom_tokenizers string Comma separated list of tokenizer plugins for custom indices. + --encryption string [Enterprise Feature] Encryption At Rest options + key-file=; The file that stores the symmetric key of length 16, 24, or 32 bytes. The key size determines the chosen AES cipher (AES-128, AES-192, and AES-256 respectively). + (default "key-file=;") + --export string Folder in which to store exports. (default "export") + --feature-flags string Feature flags to enable various experimental features + enable-detailed-metrics=false; Enable metrics about disk reads and cache per predicate + log-slow-query-threshold=0; Queries that take longer than this threshold will be logged with structured fields including trace ID for correlation with distributed traces. Disabled by default (0). Note: enabling this logs query text which may contain sensitive data; do not enable in deployments with strict data privacy requirements. + --graphql string GraphQL options + debug=false; Enables debug mode in GraphQL. This returns auth errors to clients, and we do not recommend turning it on for production. + extensions=true; Enables extensions in GraphQL response body. + introspection=true; Enables GraphQL schema introspection. + lambda-url=; The URL of a lambda server that implements custom GraphQL Javascript resolvers. + poll-interval=1s; The polling interval for GraphQL subscription. + (default "introspection=true; debug=false; extensions=true; poll-interval=1s; lambda-url=;") + -h, --help help for alpha + --limit string Limit options + disallow-drop=false; Set disallow-drop to true to block drop-all and drop-data operation. It still allows dropping attributes and types. + mutations-nquad=1000000; The maximum number of nquads that can be inserted in a mutation request. + mutations=allow; [allow, disallow, strict] The mutations mode to use. + normalize-node=10000; The maximum number of nodes that can be returned in a query that uses the normalize directive. + query-edge=1000000; The maximum number of edges that can be returned in a query. This applies to shortest path and recursive queries. + query-timeout=0ms; Maximum time after which a query execution will fail. If set to 0, the timeout is infinite. + txn-abort-after=5m; Abort any pending transactions older than this duration. The liveness of a transaction is determined by its last mutation. + max-pending-queries=10000; Number of maximum pending queries before we reject them as too many requests. + (default "mutations=allow; query-edge=1000000; normalize-node=10000; mutations-nquad=1000000; disallow-drop=false; query-timeout=0ms; txn-abort-after=5m; max-pending-queries=10000") + --my string addr:port of this server, so other Dgraph servers can talk to this. + -o, --port_offset int Value added to all listening port numbers. [Internal=7080, HTTP=8080, Grpc=9080] + -p, --postings string Directory to store posting lists. (default "p") + --raft string Raft options + group=; Provides an optional Raft Group ID that this Alpha would indicate to Zero to join. + idx=; Provides an optional Raft ID that this Alpha would use to join Raft groups. + learner=false; Make this Alpha a "learner" node. In learner mode, this Alpha will not participate in Raft elections. This can be used to achieve a read-only replica. + pending-proposals=256; Number of pending mutation proposals. Useful for rate limiting. + snapshot-after-duration=30m; Frequency at which we should create a new raft snapshots. Set to 0 to disable duration based snapshot. + snapshot-after-entries=10000; Create a new Raft snapshot after N number of Raft entries. The lower this number, the more frequent snapshot creation will be. Snapshots are created only if both snapshot-after-duration and snapshot-after-entries threshold are crossed. + (default "learner=false; snapshot-after-entries=10000; snapshot-after-duration=30m; pending-proposals=256; idx=; group=;") + --security string Security options + token=; If set, all Admin requests to Dgraph will need to have this token. The token can be passed as follows: for HTTP requests, in the X-Dgraph-AuthToken header. For Grpc, in auth-token key in the context. + whitelist=; A comma separated list of IP addresses, IP ranges, CIDR blocks, or hostnames you wish to whitelist for performing admin actions (i.e., --security "whitelist=144.142.126.254,127.0.0.1:127.0.0.3,192.168.0.0/16,host.docker.internal"). + (default "token=; whitelist=;") + --survive string Choose between "process" or "filesystem". + If set to "process", there would be no data loss in case of process crash, but the behavior would be nondeterministic in case of filesystem crash. + If set to "filesystem", blocking sync would be called after every write, hence guaranteeing no data loss in case of hard reboot. + Most users should be OK with choosing "process". (default "process") + --telemetry string Telemetry (diagnostic) options + reports=true; Send anonymous telemetry data to Dgraph devs. + sentry=true; Send crash events to Sentry. + (default "reports=true; sentry=true;") + --tls string TLS Server options + ca-cert=; The CA cert file used to verify server certificates. Required for enabling TLS. + client-auth-type=VERIFYIFGIVEN; The TLS client authentication method. + client-cert=; (Optional) The client Cert file which is needed to connect as a client with the other nodes in the cluster. + client-key=; (Optional) The private client Key file which is needed to connect as a client with the other nodes in the cluster. + internal-port=false; (Optional) Enable inter-node TLS encryption between cluster nodes. + server-cert=; The server Cert file which is needed to initiate the server in the cluster. + server-key=; The server Key file which is needed to initiate the server in the cluster. + use-system-ca=true; Includes System CA into CA Certs. + (default "use-system-ca=true; client-auth-type=VERIFYIFGIVEN; internal-port=false;") + --tmp string Directory to store temporary buffers. (default "t") + --trace string Trace options + datadog=; URL of Datadog to send OpenTelemetry traces. As of now, the trace exporter does not support annotation logs and discards them. + jaeger=; URL of Jaeger or other OpenTelemetry-compatible trace collector to send OpenTelemetry traces. + ratio=0.01; The ratio of queries to trace. + service=; Custom service name for tracing. If set, overrides the default (dgraph.alpha/dgraph.zero). + (default "ratio=0.01; jaeger=; datadog=;") + --vault string Vault options + acl-field=; Vault field containing ACL key. + acl-format=base64; ACL key format, can be 'raw' or 'base64'. + addr=http://localhost:8200; Vault server address (format: http://ip:port). + enc-field=; Vault field containing encryption key. + enc-format=base64; Encryption key format, can be 'raw' or 'base64'. + path=secret/data/dgraph; Vault KV store path (e.g. 'secret/data/dgraph' for KV V2, 'kv/dgraph' for KV V1). + role-id-file=; Vault RoleID file, used for AppRole authentication. + secret-id-file=; Vault SecretID file, used for AppRole authentication. + (default "addr=http://localhost:8200; role-id-file=; secret-id-file=; path=secret/data/dgraph; acl-field=; acl-format=base64; enc-field=; enc-format=base64") + -w, --wal string Directory to store raft write-ahead logs. (default "w") + -z, --zero string Comma separated list of Dgraph Zero addresses of the form IP_ADDRESS:PORT. (default "localhost:5080") + +Use "dgraph alpha [command] --help" for more information about a command. +``` + + + diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/cli/audit.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/cli/audit.md new file mode 100644 index 00000000..914ba7ef --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/cli/audit.md @@ -0,0 +1,53 @@ +--- +title: Dgraph CLI Reference +--- + +#### `dgraph audit` + +This command decrypts audit files. These files are created using the `--audit` +when you run the `dgraph alpha` command. The following replicates the help listing +shown when you run `dgraph audit --help`: + +```shell +Dgraph audit tool +Usage: + dgraph audit [command] + +Available Commands: + decrypt Run Dgraph Audit tool to decrypt audit files + +Flags: + -h, --help help for audit + +Use "dgraph audit [command] --help" for more information about a command. +``` + +#### `dgraph cert` + +This command lets you manage [TLS certificates](../admin/security/tls-configuration). +The following replicates the help listing shown when you run `dgraph cert --help`: + +```shell +Dgraph TLS certificate management +Usage: + dgraph cert [flags] + dgraph cert [command] + +Available Commands: + ls lists certificates and keys + +Flags: + -k, --ca-key string path to the CA private key (default "ca.key") + -c, --client string create cert/key pair for a client name + -d, --dir string directory containing TLS certs and keys (default "tls") + --duration int duration of cert validity in days (default 365) + -e, --elliptic-curve string ECDSA curve for private key. Values are: "P224", "P256", "P384", "P521". + --force overwrite any existing key and cert + -h, --help help for cert + -r, --keysize int RSA key bit size for creating new keys (default 2048) + -n, --nodes strings creates cert/key pair for nodes + --verify verify certs against root CA when creating (default true) + +Use "dgraph cert [command] --help" for more information about a command. +``` + diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/cli/bulk.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/cli/bulk.md new file mode 100644 index 00000000..7bf138a2 --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/cli/bulk.md @@ -0,0 +1,280 @@ +--- +title: dgraph bulk +--- + +The `dgraph bulk` command runs the Dgraph Bulk Loader, which efficiently imports large datasets into Dgraph by bypassing the Alpha server and directly creating posting list files. + +## Overview + +The Bulk Loader is designed for initial data import of large datasets (millions or billions of triples). It's significantly faster than the Live Loader because it: +- Processes data in parallel using MapReduce-like operations +- Creates posting list files directly without going through a running Alpha +- Shards data across multiple output directories for distributed deployment + +:::note +The Bulk Loader should be used for initial import only. For incremental updates on a running cluster, use the [Live Loader](live). +::: + +## Usage + +```bash +dgraph bulk [flags] +``` + +## Key Flags + +| Flag | Description | Default | +|------|-------------|---------| +| `-f, --files` | Location of *.rdf(.gz) or *.json(.gz) file(s) to load | | +| `-s, --schema` | Location of schema file | | +| `-g, --graphql_schema` | Location of the GraphQL schema file | | +| `--out` | Location to write the final dgraph data directories | `"./out"` | +| `--reduce_shards` | Number of reduce shards (determines number of Alpha nodes) | `1` | +| `--map_shards` | Number of map output shards | `1` | +| `-j, --num_go_routines` | Number of worker threads to use | `1` | +| `--tmp` | Temp directory for on-disk scratch space | `"tmp"` | +| `-z, --zero` | gRPC address for Dgraph Zero | `"localhost:5080"` | +| `--format` | Specify file format (rdf or json) | | +| `--replace_out` | Replace out directory if it exists | `false` | + +## Superflags + +Bulk uses several [superflags](superflags): + +- `--badger` - Badger database options (compression, numgoroutines) +- `--encryption` - Encryption at rest +- `--tls` - TLS configuration +- `--vault` - HashiCorp Vault integration + +## Examples + +### Basic RDF Import + +```bash +dgraph bulk --files data.rdf.gz --schema schema.txt --out ./out +``` + +### Import Multiple Files + +```bash +dgraph bulk --files "data1.rdf.gz,data2.rdf.gz,data3.rdf.gz" \ + --schema schema.txt \ + --out ./out +``` + +### Import with Multiple Shards + +For a 3-node Alpha cluster with replication factor of 3: + +```bash +dgraph bulk --files data.rdf.gz \ + --schema schema.txt \ + --reduce_shards 1 \ + --out ./out +``` + +For a 6-node Alpha cluster (2 groups with 3 replicas each): + +```bash +dgraph bulk --files data.rdf.gz \ + --schema schema.txt \ + --reduce_shards 2 \ + --out ./out +``` + +### Improve Performance + +Increase parallelism for faster loading: + +```bash +dgraph bulk --files data.rdf.gz \ + --schema schema.txt \ + --num_go_routines 8 \ + --map_shards 4 \ + --reduce_shards 2 \ + --out ./out +``` + +### JSON Format + +```bash +dgraph bulk --files data.json.gz \ + --schema schema.txt \ + --format json \ + --out ./out +``` + +### With GraphQL Schema + +```bash +dgraph bulk --files data.rdf.gz \ + --schema schema.txt \ + --graphql_schema graphql_schema.graphql \ + --out ./out +``` + +### Encrypted Output + +```bash +dgraph bulk --files data.rdf.gz \ + --schema schema.txt \ + --encryption "key-file=./enc-key" \ + --encrypted_out \ + --out ./out +``` + +## Full Reference + +```shell + Run Dgraph Bulk Loader +Usage: + dgraph bulk [flags] + +Flags: + --badger string Badger options (Refer to badger documentation for all possible options) + compression=snappy; Specifies the compression algorithm and compression level (if applicable) for the postings directory. "none" would disable compression, while "zstd:1" would set zstd compression at level 1. + numgoroutines=8; The number of goroutines to use in badger.Stream. + (default "compression=snappy; numgoroutines=8;") + --cleanup_tmp Clean up the tmp directory after the loader finishes. Setting this to false allows the bulk loader can be re-run while skipping the map phase. (default true) + --custom_tokenizers string Comma separated list of tokenizer plugins + --encrypted Flag to indicate whether schema and data files are encrypted. Must be specified with --encryption or vault option(s). + --encrypted_out Flag to indicate whether to encrypt the output. Must be specified with --encryption or vault option(s). + --encryption string [Enterprise Feature] Encryption At Rest options + key-file=; The file that stores the symmetric key of length 16, 24, or 32 bytes. The key size determines the chosen AES cipher (AES-128, AES-192, and AES-256 respectively). + (default "key-file=;") + --error_log string path to error log file when --log_errors is set (default "bulk_errors.log") + -f, --files string Location of *.rdf(.gz) or *.json(.gz) file(s) to load. + --force-namespace uint Namespace onto which to load the data. If not set, will preserve the namespace. (default 18446744073709551615) + --format string Specify file format (rdf or json) instead of getting it from filename. + -g, --graphql_schema string Location of the GraphQL schema file. + -h, --help help for bulk + --http string Address to serve http (pprof). (default "localhost:8080") + --ignore_errors ignore line parsing errors in rdf files + --log_errors log parsing errors to a file (requires --ignore_errors) + --map_shards int Number of map output shards. Must be greater than or equal to the number of reduce shards. Increasing allows more evenly sized reduce shards, at the expense of increased memory usage. (default 1) + --mapoutput_mb int The estimated size of each map file output. Increasing this increases memory usage. (default 2048) + --new_uids Ignore UIDs in load files and assign new ones. + -j, --num_go_routines int Number of worker threads to use. MORE THREADS LEAD TO HIGHER RAM USAGE. (default 1) + --out string Location to write the final dgraph data directories. (default "./out") + --partition_mb int Pick a partition key every N megabytes of data. (default 4) + --reduce_shards int Number of reduce shards. This determines the number of dgraph instances in the final cluster. Increasing this potentially decreases the reduce stage runtime by using more parallelism, but increases memory usage. (default 1) + --reducers int Number of reducers to run concurrently. Increasing this can improve performance, and must be less than or equal to the number of reduce shards. (default 1) + --replace_out Replace out directory and its contents if it exists. + -s, --schema string Location of schema file. + --skip_map_phase Skip the map phase (assumes that map output files already exist). + --skip_reduce_phase Skip the reduce phase (stops after map phase completion). + --store_xids Generate an xid edge for each node. + --tls string TLS Client options + ca-cert=; The CA cert file used to verify server certificates. Required for enabling TLS. + client-cert=; (Optional) The Cert file provided by the client to the server. + client-key=; (Optional) The private Key file provided by the clients to the server. + internal-port=false; (Optional) Enable inter-node TLS encryption between cluster nodes. + server-name=; Used to verify the server hostname. + use-system-ca=true; Includes System CA into CA Certs. + (default "use-system-ca=true; internal-port=false;") + --tmp string Temp directory used to use for on-disk scratch space. Requires free space proportional to the size of the RDF file and the amount of indexing used. (default "tmp") + --vault string Vault options + acl-field=; Vault field containing ACL key. + acl-format=base64; ACL key format, can be 'raw' or 'base64'. + addr=http://localhost:8200; Vault server address (format: http://ip:port). + enc-field=; Vault field containing encryption key. + enc-format=base64; Encryption key format, can be 'raw' or 'base64'. + path=secret/data/dgraph; Vault KV store path (e.g. 'secret/data/dgraph' for KV V2, 'kv/dgraph' for KV V1). + role-id-file=; Vault RoleID file, used for AppRole authentication. + secret-id-file=; Vault SecretID file, used for AppRole authentication. + (default "addr=http://localhost:8200; role-id-file=; secret-id-file=; path=secret/data/dgraph; acl-field=; acl-format=base64; enc-field=; enc-format=base64") + --version Prints the version of Dgraph Bulk Loader. + --xidmap string Directory to store xid to uid mapping + -z, --zero string gRPC address for Dgraph zero (default "localhost:5080") + +Use "dgraph bulk [command] --help" for more information about a command. +``` + +## Output Structure + +After bulk loading, the `--out` directory will contain subdirectories for each group: + +``` +out/ +├── 0/ +│ └── p/ # Posting lists for group 1 +│ ├── 000000.sst +│ ├── 000001.sst +│ └── MANIFEST +└── 1/ + └── p/ # Posting lists for group 2 (if reduce_shards > 1) + ├── 000000.sst + ├── 000001.sst + └── MANIFEST +``` + +Each subdirectory corresponds to an Alpha group and should be copied to the appropriate Alpha node's `-p` directory. + +## Performance Tuning + +### Memory Considerations + +The Bulk Loader is memory-intensive. Key parameters affecting memory: + +- `--num_go_routines`: More threads = faster but more RAM +- `--map_shards`: More shards = better distribution but more RAM +- `--mapoutput_mb`: Larger values = more RAM per map task + +**Rule of thumb**: For N GB of input data, allocate at least N GB of RAM. + +### Optimizing for Large Datasets + +For datasets > 100 million triples: + +```bash +dgraph bulk --files data.rdf.gz \ + --schema schema.txt \ + --num_go_routines 16 \ + --map_shards 8 \ + --reduce_shards 3 \ + --mapoutput_mb 4096 \ + --out ./out +``` + +### Disk Space Requirements + +Ensure adequate disk space: +- Input data size +- 2-3x input size for temporary files (can be controlled with `--tmp`) +- Output size (varies based on indexing, typically 1-2x input size) + +## Workflow + +1. **Prepare Data**: RDF or JSON format, optionally compressed (.gz) +2. **Prepare Schema**: Define types, indexes, and constraints +3. **Run Bulk Loader**: Process and shard data +4. **Deploy Output**: Copy each group's directory to corresponding Alpha nodes +5. **Start Cluster**: Launch Zero and Alpha nodes + +## Common Issues + +### Out of Memory + +- Reduce `--num_go_routines` +- Reduce `--map_shards` +- Reduce `--mapoutput_mb` +- Add more RAM to the system + +### Slow Performance + +- Increase `--num_go_routines` (if RAM allows) +- Increase `--map_shards` for better parallelism +- Use faster storage for `--tmp` directory + +### Invalid Data + +- Use `--ignore_errors` to skip malformed lines +- Validate RDF/JSON format before bulk loading + +## See Also + +- [Live Loader](live) - For incremental updates +- [Data Migration](../migration/import-data) - Migration strategies +- [Schema](../dql/dql-schema) - Schema definition +- [Bulk Loader Guide](../migration/bulk-loader) - Detailed bulk loading guide + diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/cli/cert.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/cli/cert.md new file mode 100644 index 00000000..5dbd00ab --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/cli/cert.md @@ -0,0 +1,30 @@ +--- +title: dgraph cert +--- + +The `dgraph cert` command manages TLS certificates for securing Dgraph cluster communication and client connections. + + +```shell +Dgraph TLS certificate management +Usage: + dgraph cert [flags] + dgraph cert [command] + +Available Commands: + ls lists certificates and keys + +Flags: + -k, --ca-key string path to the CA private key (default "ca.key") + -c, --client string create cert/key pair for a client name + -d, --dir string directory containing TLS certs and keys (default "tls") + --duration int duration of cert validity in days (default 365) + -e, --elliptic-curve string ECDSA curve for private key. Values are: "P224", "P256", "P384", "P521". + --force overwrite any existing key and cert + -h, --help help for cert + -r, --keysize int RSA key bit size for creating new keys (default 2048) + -n, --nodes strings creates cert/key pair for nodes + --verify verify certs against root CA when creating (default true) + +Use "dgraph cert [command] --help" for more information about a command. +``` diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/cli/completion.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/cli/completion.md new file mode 100644 index 00000000..f87b39b9 --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/cli/completion.md @@ -0,0 +1,109 @@ +--- +title: dgraph completion +--- + +The `dgraph completion` command generates shell completion scripts for `bash` and `zsh`, making it easier to work with the Dgraph CLI by enabling tab completion for commands, subcommands, and flags. + +## Installation + +### Bash + +To enable bash completion for the current session: + +```bash +source <(dgraph completion bash) +``` + +To install bash completion permanently: + +#### Linux + +```bash +# Generate and save the completion script +dgraph completion bash > /etc/bash_completion.d/dgraph + +# Reload your shell +source ~/.bashrc +``` + +#### macOS + +```bash +# Install bash-completion if not already installed +brew install bash-completion + +# Generate and save the completion script +dgraph completion bash > $(brew --prefix)/etc/bash_completion.d/dgraph + +# Reload your shell +source ~/.bash_profile +``` + +### Zsh + +To enable zsh completion for the current session: + +```bash +source <(dgraph completion zsh) +``` + +To install zsh completion permanently: + +```bash +# Add completion script to fpath +dgraph completion zsh > "${fpath[1]}/_dgraph" + +# Reload your shell +exec $SHELL +``` + +Or add to your `~/.zshrc`: + +```bash +autoload -U compinit +compinit +source <(dgraph completion zsh) +``` + +## Command Reference + +```shell +Generates shell completion scripts for bash or zsh +Usage: + dgraph completion [command] + +Available Commands: + bash bash shell completion + zsh zsh shell completion + +Flags: + -h, --help help for completion + +Use "dgraph completion [command] --help" for more information about a command. +``` + +## Usage + +Once installed, you can use tab completion to: + +- Complete command names: `dgraph al` → `dgraph alpha` +- Complete subcommands: `dgraph acl ` → shows `add`, `del`, `info`, `mod` +- Complete flag names: `dgraph alpha --re` → `dgraph alpha --replicas` + +## Troubleshooting + +If completion isn't working: + +1. **Verify installation**: Make sure the completion script is in the correct directory +2. **Check permissions**: Ensure the completion script is readable +3. **Reload shell**: Try opening a new terminal or running `exec $SHELL` +4. **Check version**: Ensure you're using a compatible shell version + +For bash, you can verify completion is loaded: + +```bash +complete -p dgraph +``` + +This should show output indicating the completion function is registered. + diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/cli/config.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/cli/config.md new file mode 100644 index 00000000..6b1e2873 --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/cli/config.md @@ -0,0 +1,170 @@ +--- +title: Flag configuration +--- + +You can see the list of available subcommands with `dgraph --help`. You can view the full set of configuration options for a given subcommand with `dgraph --help` (for example, `dgraph zero --help`). + +You can configure options in multiple ways, which are listed below from highest precedence to lowest precedence: + +- Using command line flags (as described in the help output). +- Using environment variables. +- Using a configuration file. + +If no configuration for an option is used, then the default value as described +in the `--help` output applies. + +You can use multiple configuration methods at the same time, so a core +set of options could be set in a config file, and instance specific options +could be set using environment vars or flags. + +## Command Line Flags + +Dgraph has *global flags* that apply to all subcommands and flags specific to a subcommand. + +Several commands (`alpha`, `backup`, `bulk`,`debug`, `live`, and `zero`) use [superflags](superflags). Superflags are compound flags that contain +one or more options that let you define multiple settings in a semicolon-delimited +list. The general syntax for superflags is: `-- option-a=value-a; option-b=value-b`. + +The following example shows how to use superflags when running the `dgraph alpha` command. + +```bash +dgraph alpha --my=alpha.example.com:7080 --zero=zero.example.com:5080 \ + --badger "compression=zstd:1" \ + --block_rate "10" \ + --trace "jaeger=http://jaeger:4318" \ + --tls "ca-cert=/dgraph/tls/ca.crt;client-auth-type=REQUIREANDVERIFY;server-cert=/dgraph/tls/node.crt;server-key=/dgraph/tls/node.key;use-system-ca=true;internal-port=true;client-cert=/dgraph/tls/client.dgraphuser.crt;client-key=/dgraph/tls/client.dgraphuser.key" + --security "whitelist=10.0.0.0/8,172.0.0.0/8,192.168.0.0/16" +``` + +## Environment Variables + +The environment variable names for Dgraph mirror the flag names shown in the Dgraph CLI `--help` output. These environment variable names are formed the concatenation of `DGRAPH`, the subcommand invoked (`ALPHA`, `ZERO`, `LIVE`, or `BULK`), and then the name of the flag (in uppercase). For example, instead running a command like `dgraph alpha --block_rate 10`, you could set the following environment variable: `DGRAPH_ALPHA_BLOCK_RATE=10 dgraph alpha`. + +So, the environment variable syntax for a superflag (`-- option-a=value; option-b=value`) is `="option-a=value;option-b=value"`. + +The following is an example of environment variables for `dgraph alpha`: + +```bash +DGRAPH_ALPHA_BADGER="compression=zstd:1" +DGRAPH_ALPHA_BLOCK_RATE="10" +DGRAPH_ALPHA_TRACE="jaeger=http://jaeger:4318" +DGRAPH_ALPHA_TLS="ca-cert=/dgraph/tls/ca.crt;client-auth-type=REQUIREANDVERIFY;server-cert=/dgraph/tls/node.crt;server-key=/dgraph/tls/node.key;use-system-ca=true;internal-port=true;client-cert=/dgraph/tls/client.dgraphuser.crt;client-key=/dgraph/tls/client.dgraphuser.key" +DGRAPH_ALPHA_SECURITY="whitelist=10.0.0.0/8,172.0.0.0/8,192.168.0.0/16" +``` + +## Configuration File + +You can specify a configuration file using the Dgraph CLI with the `--config` flag (for example, +`dgraph alpha --config my_config.json`), or using an environment variable, (for example, `DGRAPH_ALPHA_CONFIG=my_config.json dgraph alpha`). + +Dgraph supports configuration file formats that it detects based on file extensions ([`.json`](https://www.json.org/json-en.html), [`.yml`](https://yaml.org/) or [`.yaml`](https://yaml.org/)). In these files, the name of the superflag is used as a key that points to a hash. The hash consists of `key: value` pairs that correspond to the superflag's list of `option=value` pairs. + +:::tip +When representing the superflag options in the hash, you can use either *kebab-case* or *snake_case* for names of the keys. +::: + +### JSON Config File + +In JSON, you can represent a superflag and its options (`-- +option-a=value;option-b=value`) as follows: + +```json +{ + "": { + "option-a": "value", + "option-b": "value" + } +} +``` + +The following example JSON config file (`config.json`) using *kebab-case*: + +```json +{ + "badger": { "compression": "zstd:1" }, + "trace": { "jaeger": "http://jaeger:4318" }, + "security": { "whitelist": "10.0.0.0/8,172.0.0.0/8,192.168.0.0/16" }, + "tls": { + "ca-cert": "/dgraph/tls/ca.crt", + "client-auth-type": "REQUIREANDVERIFY", + "server-cert": "/dgraph/tls/node.crt", + "server-key": "/dgraph/tls/node.key", + "use-system-ca": true, + "internal-port": true, + "client-cert": "/dgraph/tls/client.dgraphuser.crt", + "client-key": "/dgraph/tls/client.dgraphuser.key" + } +} +``` + +The following example JSON config file (`config.json`) using *snake_case*: + +```json +{ + "badger": { "compression": "zstd:1" }, + "trace": { "jaeger": "http://jaeger:4318" }, + "security": { "whitelist": "10.0.0.0/8,172.0.0.0/8,192.168.0.0/16" }, + "tls": { + "ca_cert": "/dgraph/tls/ca.crt", + "client_auth_type": "REQUIREANDVERIFY", + "server_cert": "/dgraph/tls/node.crt", + "server_key": "/dgraph/tls/node.key", + "use_system_ca": true, + "internal_port": true, + "client_cert": "/dgraph/tls/client.dgraphuser.crt", + "client_key": "/dgraph/tls/client.dgraphuser.key" + } +} +``` + + +### YAML Config File + +In YAML, you can represent a superflag and its options (`-- +option-a=value;option-b=value`) as follows: + +```yaml +: + option-a: value + option-b: value +``` + +The following example YAML config file (`config.yml`) uses *kebab-case*: + +```yaml +badger: + compression: zstd:1 +trace: + jaeger: http://jaeger:4318 +security: + whitelist: 10.0.0.0/8,172.0.0.0/8,192.168.0.0/16 +tls: + ca-cert: /dgraph/tls/ca.crt + client-auth-type: REQUIREANDVERIFY + server-cert: /dgraph/tls/node.crt + server-key: /dgraph/tls/node.key + use-system-ca: true + internal-port: true + client-cert: /dgraph/tls/client.dgraphuser.crt + client-key: /dgraph/tls/client.dgraphuser.key +``` + +The following example YAML config file (`config.yml`) uses *snake_case*: + +```yaml +badger: + compression: zstd:1 +trace: + jaeger: http://jaeger:4318 +security: + whitelist: 10.0.0.0/8,172.0.0.0/8,192.168.0.0/16 +tls: + ca_cert: /dgraph/tls/ca.crt + client_auth_type: REQUIREANDVERIFY + server_cert: /dgraph/tls/node.crt + server_key: /dgraph/tls/node.key + use_system_ca: true + internal_port: true + client_cert: /dgraph/tls/client.dgraphuser.crt + client_key: /dgraph/tls/client.dgraphuser.key +``` diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/cli/conv.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/cli/conv.md new file mode 100644 index 00000000..ac893046 --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/cli/conv.md @@ -0,0 +1,25 @@ +--- +title: dgraph conv +--- + +#### `dgraph conv` + +This command runs the Dgraph geographic file converter, which converts geographic +files into RDF so that they can be consumed by Dgraph. The following replicates +the help listing shown when you run `dgraph conv --help`: + +```shell +Dgraph Geo file converter +Usage: + dgraph conv [flags] + +Flags: + --geo string Location of geo file to convert + --geopred string Predicate to use to store geometries (default "loc") + -h, --help help for conv + --out string Location of output rdf.gz file (default "output.rdf.gz") + +Use "dgraph conv [command] --help" for more information about a command. +``` + + diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/cli/debuginfo.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/cli/debuginfo.md new file mode 100644 index 00000000..88f56192 --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/cli/debuginfo.md @@ -0,0 +1,22 @@ +--- +title: dgraph debuginfo +--- + +The `dgraph debuginfo` command generates comprehensive debug information about the current Dgraph node, useful for troubleshooting cluster issues. + +```shell +Generate debug information on the current node +Usage: + dgraph debuginfo [flags] + +Flags: + -a, --alpha string Address of running dgraph alpha. (default "localhost:8080") + -x, --archive Whether to archive the generated report (default true) + -d, --directory string Directory to write the debug info into. + -h, --help help for debuginfo + -p, --profiles strings List of pprof profiles to dump in the report. (default [goroutine,heap,threadcreate,block,mutex,profile,trace]) + -s, --seconds uint32 Duration for time-based profile collection. (default 15) + -z, --zero string Address of running dgraph zero. + +Use "dgraph debuginfo [command] --help" for more information about a command. +``` \ No newline at end of file diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/cli/decrypt.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/cli/decrypt.md new file mode 100644 index 00000000..1a05047c --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/cli/decrypt.md @@ -0,0 +1,71 @@ +--- +title: dgraph decrypt +--- + +You might need to decrypt data from an encrypted Dgraph cluster for a variety of reasons, including: + +* Migration of data from an encrypted cluster to a non-encrypted cluster +* Changing your data or schema by directly editing an RDF file or schema file + +To support these scenarios, Dgraph includes a `decrypt` command that decrypts encrypted RDF and schema files. To learn how to export RDF and schema files from Dgraph, see: +[Dgraph Administration: Export database](../migration/export-data). + +The `decrypt` command supports a variety of symmetric key lengths, which +determine the AES cypher used for encryption and decryption, as follows: + + +| Symmetric key length | AES encryption cypher | +|----------------------|-----------------------| +| 128 bits (16-bytes) | AES-128 | +| 192 bits (24-bytes) | AES-192 | +| 256 bits (32-bytes) | AES-256 | + + +The `decrypt` command also supports the use of [Hashicorp Vault](https://www.vaultproject.io/) to store secrets, including support for Vault's +[AppRole authentication](https://www.vaultproject.io/docs/auth/approle.html). + +## Decryption options + +The following decryption options (or *flags*) are available for the `decrypt` command: + + +| Flag or Superflag | Superflag Option | Notes | +|-------------------------|------------------|-----------------------------------------------------------------------------------------------| +| `--encryption` | `key-file` | Encryption key filename | +| `-f`, `--file` | | Path to file for the encrypted RDF or schema **.gz** file | +| `-h`, `--help` | | Help for the decrypt command | +| `-o`, `--out` | | Path to file for the decrypted **.gz** file that decrypt creates | +| `--vault` | `addr` | Vault server address, (default: `http://localhost:8200` ) | +| | `enc-field` | Name of the Vault server's key/value store field that holds the Base64 encryption key | +| | `enc-format` | Vault server field format; can be `raw` or `base64` (default: `base64`) | +| | `path` | Vault server key/value store path (default: `secret/data/dgraph`) | +| | `role-id-file` | File containing the [Vault](https://www.vaultproject.io/) `role_id` used for AppRole authentication | +| | `secret-id-file` | File containing the [Vault](https://www.vaultproject.io/) `secret_id` used for AppRole authentication | + + + +## Data decryption examples + +For example, you could use the following command with an encrypted RDF file +(**encrypted.rdf.gz**) and an encryption key file (**enc_key_file**), to +create a decrypted RDF file: + +```bash +# Encryption Key from the file path +dgraph decrypt --file "encrypted.rdf.gz" --out "decrypted_rdf.gz" --encryption key-file="enc-key-file" + +# Encryption Key from HashiCorp Vault +dgraph decrypt --file "encrypted.rdf.gz" --out "decrypted_rdf.gz" \ + --vault addr="http://localhost:8200";enc-field="enc_key";enc-format="raw";path="secret/data/dgraph/alpha";role-id-file="./role_id";secret-id-file="./secret_id" +``` + +You can use similar syntax to create a decrypted schema file: + +```bash +# Encryption Key from the file path +dgraph decrypt --file "encrypted.schema.gz" --out "decrypted_schema.gz" --encryption key-file="enc-key-file" + +# Encryption Key from HashiCorp Vault +dgraph decrypt --file "encrypted.schema.gz" --out "decrypted_schema.gz" \ + --vault addr="http://localhost:8200";enc-field="enc_key";enc-format="raw";path="secret/data/dgraph/alpha";role-id-file="./role_id";secret-id-file="./secret_id" +``` diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/cli/export_backup.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/cli/export_backup.md new file mode 100644 index 00000000..9bee5fa2 --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/cli/export_backup.md @@ -0,0 +1,42 @@ +--- +title: dgraph export_backup +--- + +#### `dgraph export_backup` + +This command is used to convert a [binary backup](../admin/admin-tasks/binary-backups) +created using Dgraph Enterprise Edition into an exported folder. The following +replicates key information from the help listing shown when you run `dgraph export_backup --help`: + +```shell +Export data inside single full or incremental backup +Usage: + dgraph export_backup [flags] + +Flags: + -d, --destination string The folder to which export the backups. + --encryption string [Enterprise Feature] Encryption At Rest options + key-file=; The file that stores the symmetric key of length 16, 24, or 32 bytes. The key size determines the chosen AES cipher (AES-128, AES-192, and AES-256 respectively). + (default "key-file=;") + -f, --format string The format of the export output. Accepts a value of either rdf or json (default "rdf") + -h, --help help for export_backup + -l, --location string Sets the location of the backup. Both file URIs and s3 are supported. + This command will take care of all the full + incremental backups present in the location. + --upgrade If true, retrieve the CORS from DB and append at the end of GraphQL schema. + It also deletes the deprecated types and predicates. + Use this option when exporting a backup of 20.11 for loading onto 21.03. + --vault string Vault options + acl-field=; Vault field containing ACL key. + acl-format=base64; ACL key format, can be 'raw' or 'base64'. + addr=http://localhost:8200; Vault server address (format: http://ip:port). + enc-field=; Vault field containing encryption key. + enc-format=base64; Encryption key format, can be 'raw' or 'base64'. + path=secret/data/dgraph; Vault KV store path (e.g. 'secret/data/dgraph' for KV V2, 'kv/dgraph' for KV V1). + role-id-file=; Vault RoleID file, used for AppRole authentication. + secret-id-file=; Vault SecretID file, used for AppRole authentication. + (default "addr=http://localhost:8200; role-id-file=; secret-id-file=; path=secret/data/dgraph; acl-field=; acl-format=base64; enc-field=; enc-format=base64") + +Use "dgraph export_backup [command] --help" for more information about a command. +``` + + diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/cli/increment.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/cli/increment.md new file mode 100644 index 00000000..702004bc --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/cli/increment.md @@ -0,0 +1,40 @@ +--- +title: dgraph increment +--- + +This command increments a counter transactionally, so that you can confirm that +an Alpha node is able to handle both query and mutation requests. To learn more, +see [Using the Increment Tool](/learn/howto/using-increment-tool). +The following replicates the help listing shown when you run `dgraph increment --help`: + +```shell +Increment a counter transactionally +Usage: + dgraph increment [flags] + +Flags: + --alpha string Address of Dgraph Alpha. (default "localhost:9080") + --be Best-effort. Read counter value without retrieving timestamp from Zero. + --creds string Various login credentials if login is required. + user defines the username to login. + password defines the password of the user. + namespace defines the namespace to log into. + Sample flag could look like --creds user=username;password=mypass;namespace=2 + -h, --help help for increment + --jaeger string Send opencensus traces to Jaeger. + --num int How many times to run. (default 1) + --pred string Predicate to use for storing the counter. (default "counter.val") + --retries int How many times to retry setting up the connection. (default 10) + --ro Read-only. Read the counter value without updating it. + --tls string TLS Client options + ca-cert=; The CA cert file used to verify server certificates. Required for enabling TLS. + client-cert=; (Optional) The Cert file provided by the client to the server. + client-key=; (Optional) The private Key file provided by the clients to the server. + internal-port=false; (Optional) Enable inter-node TLS encryption between cluster nodes. + server-name=; Used to verify the server hostname. + use-system-ca=true; Includes System CA into CA Certs. + (default "use-system-ca=true; internal-port=false;") + --wait duration How long to wait. + +Use "dgraph increment [command] --help" for more information about a command. +``` \ No newline at end of file diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/cli/index.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/cli/index.md new file mode 100644 index 00000000..f10b8024 --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/cli/index.md @@ -0,0 +1,60 @@ +--- +title: Dgraph CLI +--- + +The Dgraph command-line interface (CLI) provides comprehensive tools for deploying and managing Dgraph in self-managed environments. Whether you're running Dgraph on on-premises infrastructure or cloud platforms (AWS, GCP, Azure), the CLI gives you complete control over your deployment. + +## CLI Structure + +The Dgraph CLI is built around the root `dgraph` command and its subcommands. Many commands support their own subcommands, creating a hierarchical structure. For example, `dgraph acl` requires you to specify a subcommand like `add`, `del`, `info`, or `mod`. + +## Available Commands + +The Dgraph CLI includes the following command groups: + +### Core Commands +- [**`dgraph alpha`**](alpha) - Run Dgraph Alpha database nodes +- [**`dgraph zero`**](zero) - Run Dgraph Zero management nodes + +### Data Loading Commands +- [**`dgraph bulk`**](bulk) - Bulk load data with the Bulk Loader +- [**`dgraph live`**](live) - Load data with the Live Loader +- [**`dgraph restore`**](restore) - Restore backups from Enterprise Edition + +### Security Commands +- [**`dgraph acl`**](acl) - Manage Access Control Lists (ACL) +- [**`dgraph audit`**](audit) - Decrypt audit files +- [**`dgraph cert`**](cert) - Manage TLS certificates + +### Debug Commands +- [**`dgraph debug`**](../dql/query/debug) - Debug Dgraph instances +- [**`dgraph debuginfo`**](debuginfo) - Generate debug information + +### Utility Commands +- [**`dgraph completion`**](completion) - Generate shell completion scripts +- [**`dgraph conv`**](conv) - Convert geographic files to RDF +- [**`dgraph decrypt`**](decrypt) - Decrypt exported files +- [**`dgraph export_backup`**](export_backup) - Export binary backups +- [**`dgraph increment`**](increment) - Test with transactional counter +- [**`dgraph lsbackup`**](lsbackup) - List backup information +- [**`dgraph migrate`**](migrate) - Migrate from MySQL to Dgraph +- [**`dgraph upgrade`**](upgrade) - Upgrade Dgraph versions + +## Configuration + +Dgraph provides flexible configuration options: + +- **[Superflags](superflags)** - Learn about compound flags for complex commands +- **[Configuration Guide](config)** - Configure using flags, environment variables, or config files + +## Getting Help + +You can view help for any command using the `--help` flag: + +```bash +dgraph --help # Show all available commands +dgraph alpha --help # Show alpha-specific options +dgraph acl add --help # Show help for acl add subcommand +``` + + diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/cli/live.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/cli/live.md new file mode 100644 index 00000000..110baa4d --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/cli/live.md @@ -0,0 +1,65 @@ +--- +title: dgraph live +--- + +#### `dgraph live` + +This command is used to load live data with the Dgraph [Live Loader](../migration/live-loader) tool. +The following replicates the help listing shown when you run `dgraph live --help`: + +```shell + Run Dgraph Live Loader +Usage: + dgraph live [flags] + +Flags: + -a, --alpha string Comma-separated list of Dgraph alpha gRPC server addresses (default "127.0.0.1:9080") + -t, --auth_token string The auth token passed to the server for Alter operation of the schema file. If used with --slash_grpc_endpoint, then this should be set to the API token issuedby Slash GraphQL + -b, --batch int Number of N-Quads to send as part of a mutation. (default 1000) + -m, --bufferSize string Buffer for each thread (default "100") + -c, --conc int Number of concurrent requests to make to Dgraph (default 10) + --creds string Various login credentials if login is required. + user defines the username to login. + password defines the password of the user. + namespace defines the namespace to log into. + Sample flag could look like --creds user=username;password=mypass;namespace=2 + --encryption string [Enterprise Feature] Encryption At Rest options + key-file=; The file that stores the symmetric key of length 16, 24, or 32 bytes. The key size determines the chosen AES cipher (AES-128, AES-192, and AES-256 respectively). + (default "key-file=;") + -f, --files string Location of *.rdf(.gz) or *.json(.gz) file(s) to load + --force-namespace int Namespace onto which to load the data.Only guardian of galaxy should use this for loading data into multiple namespaces or somespecific namespace. Setting it to negative value will preserve the namespace. + --format string Specify file format (rdf or json) instead of getting it from filename + -h, --help help for live + --http string Address to serve http (pprof). (default "localhost:6060") + --new_uids Ignore UIDs in load files and assign new ones. + -s, --schema string Location of schema file + --slash_grpc_endpoint string Path to Slash GraphQL GRPC endpoint. If --slash_grpc_endpoint is set, all other TLS options and connection options will beignored + --tls string TLS Client options + ca-cert=; The CA cert file used to verify server certificates. Required for enabling TLS. + client-cert=; (Optional) The Cert file provided by the client to the server. + client-key=; (Optional) The private Key file provided by the clients to the server. + internal-port=false; (Optional) Enable inter-node TLS encryption between cluster nodes. + server-name=; Used to verify the server hostname. + use-system-ca=true; Includes System CA into CA Certs. + (default "use-system-ca=true; internal-port=false;") + --tmp string Directory to store temporary buffers. (default "t") + -U, --upsertPredicate string run in upsertPredicate mode. the value would be used to store blank nodes as an xid + -C, --use_compression Enable compression on connection to alpha server + --vault string Vault options + acl-field=; Vault field containing ACL key. + acl-format=base64; ACL key format, can be 'raw' or 'base64'. + addr=http://localhost:8200; Vault server address (format: http://ip:port). + enc-field=; Vault field containing encryption key. + enc-format=base64; Encryption key format, can be 'raw' or 'base64'. + path=secret/data/dgraph; Vault KV store path (e.g. 'secret/data/dgraph' for KV V2, 'kv/dgraph' for KV V1). + role-id-file=; Vault RoleID file, used for AppRole authentication. + secret-id-file=; Vault SecretID file, used for AppRole authentication. + (default "addr=http://localhost:8200; role-id-file=; secret-id-file=; path=secret/data/dgraph; acl-field=; acl-format=base64; enc-field=; enc-format=base64") + --verbose Run the live loader in verbose mode + -x, --xidmap string Directory to store xid to uid mapping + -z, --zero string Dgraph zero gRPC server address (default "127.0.0.1:5080") + +Use "dgraph live [command] --help" for more information about a command. +``` + + diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/cli/lsbackup.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/cli/lsbackup.md new file mode 100644 index 00000000..c92dfeab --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/cli/lsbackup.md @@ -0,0 +1,162 @@ +--- +title: Backup List Tool +--- + +The `lsbackup` command-line tool prints information about the stored backups in a user-defined location. + +## Parameters + +The `lsbackup` command has two flags: + +```txt +Flags: + -h, --help help for lsbackup + -l, --location string Sets the source location URI (required). + --verbose Outputs additional info in backup list. +``` + +- `--location`: indicates a [source URI](#source-uri) with Dgraph backup objects. This URI supports all the schemes used for backup. +- `--verbose`: if enabled will print additional information about the selected backup. + +For example, you can execute the `lsbackup` command as follows: + +```sh +dgraph lsbackup -l +``` + +### Source URI + +Source URI formats: + +- `[scheme]://[host]/[path]?[args]` +- `[scheme]:///[path]?[args]` +- `/[path]?[args]` (only for local or NFS) + +Source URI parts: + +- `scheme`: service handler, one of: `s3`, `minio`, `file` +- `host`: remote address; e.g.: `dgraph.s3.amazonaws.com` +- `path`: directory, bucket or container at target; e.g.: `/dgraph/backups/` +- `args`: specific arguments that are ok to appear in logs + +## Output + +The following snippet is an example output of `lsbackup`: + +```json +[ + { + "path": "/home/user/Dgraph/20.11/backup/manifest.json", + "since": 30005, + "backup_id": "reverent_vaughan0", + "backup_num": 1, + "encrypted": false, + "type": "full" + }, +] +``` + +If the `--verbose` flag was enabled, the output would look like this: + +```json +[ + { + "path": "/home/user/Dgraph/20.11/backup/manifest.json", + "since": 30005, + "backup_id": "reverent_vaughan0", + "backup_num": 1, + "encrypted": false, + "type": "full", + "groups": { + "1": [ + "dgraph.graphql.schema_created_at", + "dgraph.graphql.xid", + "dgraph.drop.op", + "dgraph.type", + "dgraph.cors", + "dgraph.graphql.schema_history", + "score", + "dgraph.graphql.p_query", + "dgraph.graphql.schema", + "dgraph.graphql.p_sha256hash", + "series" + ] + } + }, +] +``` + +### Return values + +- `path`: Name of the backup + +- `since`: is the timestamp at which this backup was taken. It's called Since because it will become the timestamp from which to backup in the next incremental backup. + +- `groups`: is the map of valid groups to predicates at the time the backup was created. This is printed only if `--verbose` flag is enabled + +- `encrypted`: Indicates whether this backup is encrypted or not + +- `type`: Indicates whether this backup is a full or incremental one + +- `drop_operation`: lists the various DROP operations that took place since the last backup. These are used during restore to redo those operations before applying the backup. (This is printed only if `--verbose` flag is enabled) + +- `backup_num`: is a monotonically increasing number assigned to each backup in a series. The full backup as BackupNum equal to one and each incremental backup gets assigned the next available number. This can be used to verify the integrity of the data during a restore. + +- `backup_id`: is a unique ID assigned to all the backups in the same series. + + +## Examples + +### S3 + +Checking information about backups stored in an AWS S3 bucket: + +```sh +dgraph lsbackup -l s3:///s3.us-west-2.amazonaws.com/dgraph_backup +``` + +You might need to set up access and secret key environment variables in the shell (or session) you are going to run the `lsbackup` command. For example: +``` +AWS_SECRET_ACCESS_KEY= +AWS_ACCESS_ID= +``` + +### MinIO + +Checking information about backups stored in a MinIO bucket: + +```sh +dgraph lsbackup -l minio://localhost:9000/dgraph_backup +``` + +In case the MinIO server is started without `tls`, you must specify that `secure=false` as it set to `true` by default. You also need to set the environment variables for the access key and secret key. + +In order to get the `lsbackup` running, you should following these steps: + +- Set `MINIO_ACCESS_KEY` as an environment variable for the running shell this can be done with the following command: + (`minioadmin` is the default access key, unless is changed by the user) + + ``` + export MINIO_ACCESS_KEY=minioadmin + ``` + +- Set MINIO_SECRET_KEY as an environment variable for the running shell this can be done with the following command: + (`minioadmin` is the default secret key, unless is changed by the user) + + ``` + export MINIO_SECRET_KEY=minioadmin + ``` + +- Add the argument `secure=false` to the `lsbackup command`, that means the command will look like: (the double quotes `"` are required) + + ```sh + dgraph lsbackup -l "minio://localhost:9000/?secure=false" + ``` + +### Local + +Checking information about backups stored locally (on disk): + +```sh +dgraph lsbackup -l ~/dgraph_backup +``` diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/cli/migrate.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/cli/migrate.md new file mode 100644 index 00000000..7d0924e1 --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/cli/migrate.md @@ -0,0 +1,55 @@ +--- +title: dgraph migrate +--- + + +#### `dgraph migrate` + +This command runs the Dgraph [migration tool](../migration/migrate-tool) +to move data from a MySQL database to Dgraph. The following replicates the help +listing shown when you run `dgraph migrate --help`: + +```shell +Run the Dgraph migration tool from a MySQL database to Dgraph +Usage: + dgraph migrate [flags] + +Flags: + --db string The database to import + -h, --help help for migrate + --host string The hostname or IP address of the database server. (default "localhost") + -o, --output_data string The data output file (default "sql.rdf") + -s, --output_schema string The schema output file (default "schema.txt") + --password string The password used for logging in + --port string The port of the database server. (default "3306") + -q, --quiet Enable quiet mode to suppress the warning logs + -p, --separator string The separator for constructing predicate names (default ".") + --tables string The comma separated list of tables to import, an empty string means importing all tables in the database + --user string The user for logging in + +Use "dgraph migrate [command] --help" for more information about a command. +``` + +#### `dgraph upgrade` + +This command helps you to upgrade from an earlier Dgraph release to a newer release. +The following replicates the help listing shown when you run `dgraph upgrade --help`: + +```shell +This tool is supported only for the mainstream release versions of Dgraph, not for the beta releases. +Usage: + dgraph upgrade [flags] + +Flags: + --acl upgrade ACL from v1.2.2 to >=v20.03.0 + -a, --alpha string Dgraph Alpha gRPC server address (default "127.0.0.1:9080") + -d, --deleteOld Delete the older ACL types/predicates (default true) + --dry-run dry-run the upgrade + -f, --from string The version string from which to upgrade, e.g.: v1.2.2 + -h, --help help for upgrade + -p, --password string Password of ACL user + -t, --to string The version string till which to upgrade, e.g.: v20.03.0 + -u, --user string Username of ACL user + +Use "dgraph upgrade [command] --help" for more information about a command. +``` diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/cli/restore.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/cli/restore.md new file mode 100644 index 00000000..6d317f85 --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/cli/restore.md @@ -0,0 +1,90 @@ +--- +title: dgraph restore +--- + +#### `dgraph restore` + +This command loads objects from available backups. The following replicates the +help listing shown when you run `dgraph restore --help`: + +```shell +Restore loads objects created with the backup feature in Dgraph Enterprise Edition (EE). + +Backups taken using the GraphQL API can be restored using CLI restore +command. Restore is intended to be used with new Dgraph clusters in offline state. + +The --location flag indicates a source URI with Dgraph backup objects. This URI supports all +the schemes used for backup. + +Source URI formats: + [scheme]://[host]/[path]?[args] + [scheme]:///[path]?[args] + /[path]?[args] (only for local or NFS) + +Source URI parts: + scheme - service handler, one of: "s3", "minio", "file" + host - remote address. ex: "dgraph.s3.amazonaws.com" + path - directory, bucket or container at target. ex: "/dgraph/backups/" + args - specific arguments that are ok to appear in logs. + +The --posting flag sets the posting list parent dir to store the loaded backup files. + +Using the --zero flag will use a Dgraph Zero address to update the start timestamp using +the restored version. Otherwise, the timestamp must be manually updated through Zero's HTTP +'assign' command. + +Dgraph backup creates a unique backup object for each node group, and restore will create +a posting directory 'p' matching the backup group ID. Such that a backup file +named '.../r32-g2.backup' will be loaded to posting dir 'p2'. + +Usage examples: + +# Restore from local dir or NFS mount: +$ dgraph restore -p . -l /var/backups/dgraph + +# Restore from S3: +$ dgraph restore -p /var/db/dgraph -l s3://s3.us-west-2.amazonaws.com/srfrog/dgraph + +# Restore from dir and update Ts: +$ dgraph restore -p . -l /var/backups/dgraph -z localhost:5080 + + +Usage: + dgraph restore [flags] + +Flags: + --backup_id string The ID of the backup series to restore. If empty, it will restore the latest series. + -b, --badger string Badger options + compression=snappy; Specifies the compression algorithm and compression level (if applicable) for the postings directory. "none" would disable compression, while "zstd:1" would set zstd compression at level 1. + goroutines=; The number of goroutines to use in badger.Stream. + (default "compression=snappy; numgoroutines=8;") + --encryption string [Enterprise Feature] Encryption At Rest options + key-file=; The file that stores the symmetric key of length 16, 24, or 32 bytes. The key size determines the chosen AES cipher (AES-128, AES-192, and AES-256 respectively). + (default "key-file=;") + --force_zero If false, no connection to a zero in the cluster will be required. Keep in mind this requires you to manually update the timestamp and max uid when you start the cluster. The correct values are printed near the end of this command's output. (default true) + -h, --help help for restore + -l, --location string Sets the source location URI (required). + -p, --postings string Directory where posting lists are stored (required). + --tls string TLS Client options + ca-cert=; The CA cert file used to verify server certificates. Required for enabling TLS. + client-cert=; (Optional) The Cert file provided by the client to the server. + client-key=; (Optional) The private Key file provided by the clients to the server. + internal-port=false; (Optional) Enable inter-node TLS encryption between cluster nodes. + server-name=; Used to verify the server hostname. + use-system-ca=true; Includes System CA into CA Certs. + (default "use-system-ca=true; internal-port=false;") + --vault string Vault options + acl-field=; Vault field containing ACL key. + acl-format=base64; ACL key format, can be 'raw' or 'base64'. + addr=http://localhost:8200; Vault server address (format: http://ip:port). + enc-field=; Vault field containing encryption key. + enc-format=base64; Encryption key format, can be 'raw' or 'base64'. + path=secret/data/dgraph; Vault KV store path (e.g. 'secret/data/dgraph' for KV V2, 'kv/dgraph' for KV V1). + role-id-file=; Vault RoleID file, used for AppRole authentication. + secret-id-file=; Vault SecretID file, used for AppRole authentication. + (default "addr=http://localhost:8200; role-id-file=; secret-id-file=; path=secret/data/dgraph; acl-field=; acl-format=base64; enc-field=; enc-format=base64") + -z, --zero string gRPC address for Dgraph zero. ex: localhost:5080 + +Use "dgraph restore [command] --help" for more information about a command. +``` + diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/cli/superflags.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/cli/superflags.md new file mode 100644 index 00000000..1a9d679f --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/cli/superflags.md @@ -0,0 +1,186 @@ +--- +title: Superflags +--- + +Dgraph uses *superflags* for complex commands: `alpha`, `backup`, `bulk`, `debug`, `live` and `zero`. Superflags are compound flags that contain one or more options, allowing you to define multiple related settings in a single, semicolon-delimited list. + +## Syntax + +The general syntax for superflags is: + +```bash +-- option-a=value; option-b=value +``` + +Semicolons are required between superflag options, but a semicolon after the last option is optional. + +:::note +You should encapsulate the options for a superflag in double-quotes (`"`) if any of those option values include spaces. You can also use quotes to improve readability: +`-- "option-a=value; option-b=value"` +::: + +## Available Superflags + +* `--acl` +* `--badger` +* `--cache` +* `--encryption` +* `--graphql` +* `--limit` +* `--raft` +* `--security` +* `--telemetry` +* `--tls` +* `--trace` +* `--vault` + +## ACL Superflag + +The `--acl` superflag configures **Access Control List** settings: + +| Option | Type | Applies to | Description | +|--------|------|------------|-------------| +| `secret-file` | string | `alpha` | File that stores the HMAC secret used for signing the JWT | +| `access-ttl` | [duration](https://github.com/dgraph-io/ristretto/blob/master/z/flags.go#L80-L98) | `alpha` | TTL for the access JWT | +| `refresh-ttl` | [duration](https://github.com/dgraph-io/ristretto/blob/master/z/flags.go#L80-L98) | `alpha` | TTL for the refresh JWT | + +## Badger Superflag + +The `--badger` superflag configures [Badger](https://dgraph.io/docs/badger) database options: + +| Option | Type | Applies to | Description | +|--------|------|------------|-------------| +| `compression` | string | `alpha`, `bulk`, `backup` | Specifies the compression level and algorithm | +| `numgoroutines` | int | `alpha`, `bulk`, `backup` | Number of Go routines used by Dgraph | + +:::note +The `--badger` superflag allows you to set many advanced [Badger options](https://pkg.go.dev/github.com/dgraph-io/badger/v3#Options), including: +`dir`, `valuedir`, `syncwrites`, `numversionstokeep`, `readonly`, `inmemory`, `metricsenabled`, `memtablesize`, +`basetablesize`, `baselevelsize`, `levelsizemultiplier`, `tablesizemultiplier`, `maxlevels`, `vlogpercentile`, +`valuethreshold`, `nummemtables`, `blocksize`, `bloomfalsepositive`, `blockcachesize`, `indexcachesize`, `numlevelzerotables`, +`numlevelzerotablesstall`, `valuelogfilesize`, `valuelogmaxentries`, `numcompactors`, `compactl0onclose`, `lmaxcompaction`, +`zstdcompressionlevel`, `verifyvaluechecksum`, `encryptionkeyrotationduration`, `bypasslockguard`, `checksumverificationmode`, +`detectconflicts`, `namespaceoffset`. +::: + +## Cache Superflag + +The `--cache` superflag configures cache settings: + +| Option | Type | Applies to | Description | +|--------|------|------------|-------------| +| `size-mb` | string | `alpha` | Total size of cache (in MB) per shard in the reducer | +| `percentage` | string | `alpha` | Cache percentages for block cache and index cache | + +## Encryption Superflag + +The `--encryption` superflag configures encryption settings: + +| Option | Type | Applies to | Description | +|--------|------|------------|-------------| +| `key-file` | string | `alpha`, `bulk`, `live`, `restore`, `debug`, `decrypt`, `export_backup` | The file that stores the symmetric key | + +## GraphQL Superflag + +The `--graphql` superflag configures GraphQL settings: + +| Option | Type | Applies to | Description | +|--------|------|------------|-------------| +| `introspection` | bool | `alpha` | Enables GraphQL schema introspection | +| `debug` | bool | `alpha` | Enables debug mode in GraphQL | +| `extensions` | bool | `alpha` | Enables extensions in GraphQL response body | +| `poll-interval` | [duration](https://github.com/dgraph-io/ristretto/blob/master/z/flags.go#L80-L98) | `alpha` | The polling interval for GraphQL subscriptions | +| `lambda-url` | string | `alpha` | The URL of a lambda server that implements custom GraphQL JavaScript resolvers | + +## Limit Superflag + +The `--limit` superflag configures limit settings for Dgraph Alpha: + +| Option | Type | Applies to | Description | +|--------|------|------------|-------------| +| `txn-abort-after` | string | `alpha` | Abort any pending transactions older than this duration | +| `disable-admin-http` | string | `zero` | Turn on/off the administrative endpoints | +| `max-retries` | int | `alpha` | Maximum number of retries | +| `mutations` | string | `alpha` | Mutation mode: `allow`, `disallow`, or `strict` | +| `query-edge` | uint64 | `alpha` | Maximum number of edges that can be returned in a query | +| `normalize-node` | int | `alpha` | Maximum number of nodes that can be returned in a query that uses the normalize directive | +| `mutations-nquad` | int | `alpha` | Maximum number of nquads that can be inserted in a mutation request | +| `max-pending-queries` | int | `alpha` | Maximum number of concurrently processing requests allowed before requests are rejected with 429 Too Many Requests | + +## Raft Superflag + +The `--raft` superflag configures [Raft](../design-concepts/raft) consensus settings: + +| Option | Type | Applies to | Description | +|--------|------|------------|-------------| +| `pending-proposals` | int | `alpha` | Maximum number of pending mutation proposals; useful for rate limiting | +| `idx` | int | `alpha`, `zero` | Provides an optional Raft ID that an Alpha node can use to join Raft groups | +| `group` | int | `alpha` | Provides an optional Raft group ID that an Alpha node can use to request group membership from a Zero node | +| `learner` | bool | `alpha`, `zero` | Make this Alpha a learner node (used for read-only replicas) | +| `snapshot-after-duration` | duration | `alpha` | Frequency at which Raft snapshots are created | +| `snapshot-after-entries` | int | `alpha` | Create a new Raft snapshot after the specified number of Raft entries | + +## Security Superflag + +The `--security` superflag configures security settings: + +| Option | Type | Applies to | Description | +|--------|------|------------|-------------| +| `token` | string | `alpha`, `zero` | Authentication token. When set, admin requests must present it in the `X-Dgraph-AuthToken` header | +| `whitelist` | string | `alpha`, `zero` | A comma separated list of IP addresses, IP ranges, CIDR blocks, or hostnames for administration | + +On Zero, `--security` protects the administrative endpoints exposed over the HTTP port (`/state`, `/assign`, `/removeNode`, `/moveTablet`). See [Admin Endpoint Security](../admin/security/admin-endpoint-security#zero-admin-endpoints). + +## Telemetry Superflag + +The `--telemetry` superflag configures telemetry settings: + +| Option | Type | Applies to | Description | +|--------|------|------------|-------------| +| `reports` | bool | `alpha`, `zero` | Sends anonymous telemetry data to Dgraph | +| `sentry` | bool | `alpha`, `zero` | Enable sending crash events to Sentry | + +## TLS Superflag + +The `--tls` superflag configures [TLS](../admin/security/tls-configuration) settings: + +| Option | Type | Applies to | Description | +|--------|------|------------|-------------| +| `ca-cert` | string | `alpha`, `zero`, `bulk`, `backup`, `live` | The CA cert file used to verify server certificates | +| `use-system-ca` | bool | `alpha`, `zero`, `bulk`, `backup`, `live` | Include System CA with Dgraph Root CA | +| `server-name` | string | `alpha`, `zero`, `bulk`, `backup`, `live` | Server name, used for validating the server's TLS host name | +| `client-auth-type` | string | `alpha`, `zero` | TLS client authentication used to validate client connections from external ports | +| `server-cert` | string | `alpha`, `zero` | Path and filename of the node certificate (for example, `node.crt`) | +| `server-key` | string | `alpha`, `zero` | Path and filename of the node certificate private key (for example, `node.key`) | +| `internal-port` | bool | `alpha`, `zero`, `bulk`, `backup`, `live` | Makes internal ports (by default, 5080 and 7080) use the REQUIREANDVERIFY setting | +| `client-cert` | string | `alpha`, `zero`, `bulk`, `backup`, `live` | User cert file provided by the client to the Alpha node | +| `client-key` | string | `alpha`, `zero`, `bulk`, `backup`, `live` | User private key file provided by the client to the Alpha node | + +## Trace Superflag + +The `--trace` superflag configures [tracing](/admin/observability/tracing) settings: + +| Option | Type | Applies to | Description | +|--------|------|------------|-------------| +| `ratio` | float64 | `alpha`, `zero` | The ratio of queries to trace | +| `jaeger` | string | `alpha`, `zero` | URL of Jaeger to send OpenTelemetry traces | +| `datadog` | string | `alpha`, `zero` | URL of Datadog to send OpenTelemetry traces | +| `service` | string | `alpha`, `zero` | Custom service name for tracing. If set, overrides the default (dgraph.alpha/dgraph.zero) | + +## Vault Superflag + +The `--vault` superflag configures Vault integration settings: + +| Option | Type | Applies to | Description | +|--------|------|------------|-------------| +| `addr` | string | `alpha`, `bulk`, `backup`, `live`, `debug` | Vault server address, formatted as `http://ip-address:port` | +| `role-id-file` | string | `alpha`, `bulk`, `backup`, `live`, `debug` | File containing Vault `role-id` used for AppRole authentication | +| `secret-id-file` | string | `alpha`, `bulk`, `backup`, `live`, `debug` | File containing Vault `secret-id` used for AppRole authentication | +| `path` | string | `alpha`, `bulk`, `backup`, `live`, `debug` | Vault key=value store path (example: `secret/data/dgraph` for kv-v2, `kv/dgraph` for kv-v1) | +| `field` | string | `alpha`, `bulk`, `backup`, `live`, `debug` | Vault key=value store field whose value is the base64 encoded encryption key | +| `format` | string | `alpha`, `bulk`, `backup`, `live`, `debug` | Vault field format (`raw` or `base64`) | + +## Using Superflags + +To learn more about each superflag and its options, see the `--help` output of the specific Dgraph CLI commands, or refer to the individual command documentation pages. + diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/cli/upgrade.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/cli/upgrade.md new file mode 100644 index 00000000..ee6dd10f --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/cli/upgrade.md @@ -0,0 +1,252 @@ +--- +title: dgraph upgrade +--- + +The `dgraph upgrade` command helps you upgrade from an earlier Dgraph release to a newer release by migrating ACL data and performing other version-specific migrations. + +## Overview + +This tool is designed specifically for upgrading ACL (Access Control List) data structures when moving between major Dgraph versions. It handles schema changes and data migrations required for backward compatibility. + +:::note +This tool is supported only for mainstream release versions of Dgraph, not for beta releases. +::: + +## Usage + +```bash +dgraph upgrade [flags] +``` + +## Key Flags + +| Flag | Description | Default | +|------|-------------|---------| +| `-a, --alpha` | Dgraph Alpha gRPC server address | `"127.0.0.1:9080"` | +| `--acl` | Upgrade ACL from v1.2.2 to >=v20.03.0 | `false` | +| `-f, --from` | The version string from which to upgrade (e.g., v1.2.2) | | +| `-t, --to` | The version string till which to upgrade (e.g., v20.03.0) | | +| `-u, --user` | Username of ACL user | | +| `-p, --password` | Password of ACL user | | +| `-d, --deleteOld` | Delete the older ACL types/predicates | `true` | +| `--dry-run` | Perform a dry-run of the upgrade without making changes | `false` | + +## When to Use + +Use the upgrade command when: +- Upgrading from v1.2.2 to v20.03.0 or later (ACL schema changes) +- Migrating between versions with incompatible ACL structures +- You need to validate upgrade feasibility before applying changes + +## Examples + +### Dry Run to Check Compatibility + +Before performing an actual upgrade, do a dry run to validate: + +```bash +dgraph upgrade --acl \ + --alpha localhost:9080 \ + --from v1.2.2 \ + --to v20.03.0 \ + --user groot \ + --password password \ + --dry-run +``` + +### Upgrade ACL from v1.2.2 to v20.03.0 + +```bash +dgraph upgrade --acl \ + --alpha localhost:9080 \ + --from v1.2.2 \ + --to v20.03.0 \ + --user groot \ + --password password +``` + +### Upgrade with Credentials + +```bash +dgraph upgrade --acl \ + --alpha myhost.example.com:9080 \ + --from v1.2.2 \ + --to v21.03.0 \ + --user admin \ + --password mySecurePassword +``` + +### Keep Old ACL Types + +If you want to preserve old ACL types/predicates during upgrade: + +```bash +dgraph upgrade --acl \ + --alpha localhost:9080 \ + --from v1.2.2 \ + --to v20.03.0 \ + --user groot \ + --password password \ + --deleteOld=false +``` + +## Full Reference + +```shell +This tool is supported only for the mainstream release versions of Dgraph, not for the beta releases. +Usage: + dgraph upgrade [flags] + +Flags: + --acl upgrade ACL from v1.2.2 to >=v20.03.0 + -a, --alpha string Dgraph Alpha gRPC server address (default "127.0.0.1:9080") + -d, --deleteOld Delete the older ACL types/predicates (default true) + --dry-run dry-run the upgrade + -f, --from string The version string from which to upgrade, e.g.: v1.2.2 + -h, --help help for upgrade + -p, --password string Password of ACL user + -t, --to string The version string till which to upgrade, e.g.: v20.03.0 + -u, --user string Username of ACL user + +Use "dgraph upgrade [command] --help" for more information about a command. +``` + +## Upgrade Process + +The upgrade tool performs the following steps: + +1. **Connects to Alpha**: Establishes connection to the specified Alpha node +2. **Authenticates**: Logs in with provided user credentials +3. **Validates Versions**: Checks source and target version compatibility +4. **Analyzes Schema**: Examines current ACL schema structure +5. **Migrates Data**: Transforms ACL data to new format (if not dry-run) +6. **Cleans Up**: Removes old ACL types/predicates (if `--deleteOld=true`) + +## Prerequisites + +Before running the upgrade: + +1. **Backup Your Data**: Always create a full backup before upgrading +2. **Stop Write Operations**: Ensure no ACL modifications are happening +3. **Access Credentials**: Have guardian/admin user credentials ready +4. **Network Access**: Ensure connectivity to the Alpha node +5. **Review Release Notes**: Check version-specific migration requirements + +## Best Practices + +### Planning the Upgrade + +1. **Read Release Notes**: Review breaking changes between versions +2. **Test in Staging**: Run upgrade on a staging environment first +3. **Use Dry Run**: Always perform a dry-run before actual upgrade +4. **Schedule Downtime**: Plan for maintenance window if needed + +### During the Upgrade + +```bash +# Step 1: Backup +dgraph live backup --alpha localhost:9080 + +# Step 2: Dry run +dgraph upgrade --acl \ + --alpha localhost:9080 \ + --from v1.2.2 \ + --to v20.03.0 \ + --user groot \ + --password password \ + --dry-run + +# Step 3: If dry run succeeds, perform actual upgrade +dgraph upgrade --acl \ + --alpha localhost:9080 \ + --from v1.2.2 \ + --to v20.03.0 \ + --user groot \ + --password password + +# Step 4: Verify +# Test ACL functionality after upgrade +``` + +### After the Upgrade + +1. **Verify ACL Functionality**: Test user authentication and permissions +2. **Check Logs**: Review Alpha logs for any warnings or errors +3. **Test Applications**: Ensure client applications work correctly +4. **Document Changes**: Note any configuration changes made + +## Common Upgrade Paths + +### v1.2.2 → v20.03.0+ + +Major ACL schema changes were introduced in v20.03.0: + +```bash +dgraph upgrade --acl \ + --alpha localhost:9080 \ + --from v1.2.2 \ + --to v20.03.0 \ + --user groot \ + --password password +``` + +### v20.x → v21.03+ + +If upgrading from v20.x to v21.03, check if ACL migration is needed: + +```bash +dgraph upgrade --acl \ + --alpha localhost:9080 \ + --from v20.11.0 \ + --to v21.03.0 \ + --user groot \ + --password password \ + --dry-run +``` + +## Troubleshooting + +### Authentication Failures + +If you encounter authentication errors: +- Verify user credentials are correct +- Ensure ACL is enabled on the cluster +- Check that the user has sufficient permissions + +### Connection Issues + +If unable to connect to Alpha: +- Verify Alpha is running: `curl http://localhost:8080/health` +- Check network connectivity +- Verify the gRPC port (9080) is accessible + +### Migration Errors + +If the upgrade fails: +1. Restore from backup +2. Review error messages in Alpha logs +3. Check for version compatibility issues +4. Try with `--deleteOld=false` if cleanup is causing issues + +## Version-Specific Notes + +### ACL Changes in v20.03.0 + +- New predicate structure for permissions +- Group-based permission model +- Guardian user privileges expanded + +### Changes in v21.03.0 + +- Superflag introduction +- Configuration file format changes +- Namespace support added + +## Limitations + +- Only supports mainstream releases (not beta versions) +- Primarily designed for ACL migrations +- Requires guardian/admin user credentials +- Cannot downgrade versions (one-way migration) + + diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/cli/zero.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/cli/zero.md new file mode 100644 index 00000000..30389295 --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/cli/zero.md @@ -0,0 +1,129 @@ +--- +title: dgraph zero +--- + +The `dgraph zero` command runs Dgraph Zero management nodes, which control the cluster and coordinate data distribution. + +## Overview + +A Dgraph Zero instance manages the Dgraph cluster. Typically, a single Zero instance is sufficient for the cluster; however, one can run multiple Zero instances to achieve high-availability. + +## Usage + +```bash +dgraph zero [flags] +``` + +## Key Flags + +| Flag | Description | Default | +|------|-------------|---------| +| `--my` | Address:port of this server for cluster communication | | +| `--peer` | Address of another dgraphzero server | | +| `-o, --port_offset` | Value added to all listening port numbers [Grpc=5080, HTTP=6080] | `0` | +| `-w, --wal` | Directory storing WAL | `"zw"` | +| `--replicas` | How many Dgraph Alpha replicas to run per data shard group | `1` | +| `--rebalance_interval` | Interval for trying a predicate move | `8m0s` | +| `--enterprise_license` | Path to the enterprise license file | | + +## Superflags + +Zero uses several [superflags](superflags) for advanced configuration: + +- `--audit` - Audit logging configuration +- `--limit` - UID lease and admin endpoint settings +- `--raft` - Raft consensus options +- `--security` - Authentication token and IP whitelist for the admin HTTP endpoints +- `--telemetry` - Telemetry and crash reporting +- `--tls` - TLS configuration +- `--trace` - Distributed tracing + +## Securing the admin HTTP endpoints + +Zero exposes administrative endpoints over its HTTP port (default `6080`): `/state`, `/assign`, `/removeNode`, and `/moveTablet`. These control cluster membership and coordination, so the HTTP port is an internal control-plane port and should not be reachable from untrusted networks. + +Use the `--security` superflag to authenticate callers of these endpoints: + +```bash +# Require a token in the X-Dgraph-AuthToken header +dgraph zero --security "token=" + +# Allow specific source IPs, IP ranges, CIDR blocks, or hostnames (loopback is always allowed) +dgraph zero --security "whitelist=10.0.0.0/8,192.168.1.1" +``` + +The destructive endpoints (`/removeNode`, `/moveTablet`) are restricted to loopback by default; set a `whitelist` or `token` to reach them from another host. The `/state` and `/assign` endpoints are enforced only once a `token` or `whitelist` is configured. To turn the admin HTTP endpoints off entirely, set `--limit "disable-admin-http=true"`. + +See [Admin Endpoint Security](../admin/security/admin-endpoint-security#zero-admin-endpoints) for details. + +## Full Reference + +```shell +A Dgraph Zero instance manages the Dgraph cluster. Typically, a single Zero +instance is sufficient for the cluster; however, one can run multiple Zero +instances to achieve high-availability. + +Usage: + dgraph zero [flags] + +Flags: + --audit string Audit options + compress=false; Enables the compression of old audit logs. + days=10; The number of days audit logs will be preserved. + encrypt-file=; The path to the key file to be used for audit log encryption. + output=; [stdout, /path/to/dir] This specifies where audit logs should be output to. + "stdout" is for standard output. You can also specify the directory where audit logs + will be saved. When stdout is specified as output other fields will be ignored. + size=100; The audit log max size in MB after which it will be rolled over. + (default "compress=false; days=10; size=100; dir=; output=; encrypt-file=;") + --enterprise_license string Path to the enterprise license file. + -h, --help help for zero + --limit string Limit options + disable-admin-http=false; Turn on/off the administrative endpoints exposed over Zero's HTTP port. + refill-interval=30s; The interval after which the tokens for UID lease are replenished. + uid-lease=0; The maximum number of UIDs that can be leased by namespace (except default namespace) + in an interval specified by refill-interval. Set it to 0 to remove limiting. + (default "uid-lease=0; refill-interval=30s; disable-admin-http=false;") + --my string addr:port of this server, so other Dgraph servers can talk to this. + --peer string Address of another dgraphzero server. + -o, --port_offset int Value added to all listening port numbers. [Grpc=5080, HTTP=6080] + --raft string Raft options + idx=1; Provides an optional Raft ID that this Alpha would use to join Raft groups. + learner=false; Make this Zero a "learner" node. In learner mode, this Zero will not participate in Raft elections. This can be used to achieve a read-only replica. + (default "idx=1; learner=false;") + --rebalance_interval duration Interval for trying a predicate move. (default 8m0s) + --replicas int How many Dgraph Alpha replicas to run per data shard group. The count includes the original shard. (default 1) + --security string Security options + token=; If set, all requests to Zero's administrative HTTP endpoints must present this token in the X-Dgraph-AuthToken header. + whitelist=; A comma separated list of IP addresses, IP ranges, CIDR blocks, or hostnames that are allowed to reach Zero's administrative HTTP endpoints (loopback is always allowed). e.g. --security "whitelist=127.0.0.1,192.168.0.0/16,host.docker.internal". + (default "token=; whitelist=;") + --survive string Choose between "process" or "filesystem". + If set to "process", there would be no data loss in case of process crash, but the behavior would be nondeterministic in case of filesystem crash. + If set to "filesystem", blocking sync would be called after every write, hence guaranteeing no data loss in case of hard reboot. + Most users should be OK with choosing "process". (default "process") + --telemetry string Telemetry (diagnostic) options + reports=true; Send anonymous telemetry data to Dgraph devs. + sentry=true; Send crash events to Sentry. + (default "reports=true; sentry=true;") + --tls string TLS Server options + ca-cert=; The CA cert file used to verify server certificates. Required for enabling TLS. + client-auth-type=VERIFYIFGIVEN; The TLS client authentication method. + client-cert=; (Optional) The client Cert file which is needed to connect as a client with the other nodes in the cluster. + client-key=; (Optional) The private client Key file which is needed to connect as a client with the other nodes in the cluster. + internal-port=false; (Optional) Enable inter-node TLS encryption between cluster nodes. + server-cert=; The server Cert file which is needed to initiate the server in the cluster. + server-key=; The server Key file which is needed to initiate the server in the cluster. + use-system-ca=true; Includes System CA into CA Certs. + (default "use-system-ca=true; client-auth-type=VERIFYIFGIVEN; internal-port=false;") + --trace string Trace options + datadog=; URL of Datadog to send OpenTelemetry traces. + jaeger=; URL of Jaeger to send OpenTelemetry traces. + ratio=0.01; The ratio of queries to trace. + service=; Custom service name for tracing. If set, overrides the default (dgraph.zero). + (default "ratio=0.01; jaeger=; datadog=;") + -w, --wal string Directory storing WAL. (default "zw") + +Use "dgraph zero [command] --help" for more information about a command. +``` + + diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/clients/csharp.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/clients/csharp.md new file mode 100644 index 00000000..a1b0c611 --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/clients/csharp.md @@ -0,0 +1,176 @@ +--- +title: C# +--- + +An implementation for a Dgraph client in C#, using [gRPC](https://grpc.io/). +This client follows the [Dgraph Go client](go) closely. + +:::tip +The official C# client [can be found here](https://github.com/dgraph-io/dgraph.net). +Follow the [install instructions](https://github.com/dgraph-io/dgraph.net#install) to get +it up and running. +::: + +## Supported Versions + +More details on the supported versions can be found at +[this link](https://github.com/dgraph-io/dgraph.net#supported-versions). + +## Using a Client + +### Creating a Client + +Make a new client by passing in one or more GRPC channels pointing to alphas. + +```c# +var client = new DgraphClient(new Channel("127.0.0.1:9080", ChannelCredentials.Insecure)); +``` + +### Multi-tenancy + +In multi-tenants environments, Dgraph provides a method `LoginRequest()`, +which will allow the users to login to a specific namespace. + +In order to create a Dgraph client, and make the client login into namespace `123`: + +```c# +var lr = new Api.LoginRequest() { + UserId = "userId", + Password = "password", + Namespace = 123 +} +client.Login(lr) +``` + +In the example above, the client logs into namespace `123` using username `userId` and password `password`. +Once logged in, the client can perform all the operations allowed to the `userId` user of namespace `123`. + + + +### Altering the Database + +To set the schema, pass the schema into the `DgraphClient.Alter` function, as seen below: + +```c# +var schema = "name: string @index(exact) ."; +var result = client.Alter(new Operation{ Schema = schema }); +``` + +The returned result object is based on the FluentResults library. You can check the status using `result.isSuccess` or `result.isFailed`. More information on the result object can be found [here](https://github.com/altmann/FluentResults). + + +### Creating a Transaction + +To create a transaction, call `DgraphClient.NewTransaction` method, which returns a +new `Transaction` object. This operation incurs no network overhead. + +It is good practice to call to wrap the `Transaction` in a `using` block, so that the `Transaction.Dispose` function is called after running +the transaction. + +```c# +using(var transaction = client.NewTransaction()) { + ... +} +``` + +You can also create Read-Only transactions. Read-Only transactions only allow querying, and can be created using `DgraphClient.NewReadOnlyTransaction`. + + +### Running a Mutation + +`Transaction.Mutate(RequestBuilder)` runs a mutation. It takes in a json mutation string. + +We define a person object to represent a person and serialize it to a json mutation string. In this example, we are using the [JSON.NET](https://www.newtonsoft.com/json) library, but you can use any JSON serialization library you prefer. + +```c# +using(var txn = client.NewTransaction()) { + var alice = new Person{ Name = "Alice" }; + var json = JsonConvert.SerializeObject(alice); + + var transactionResult = await txn.Mutate(new RequestBuilder().WithMutations(new MutationBuilder{ SetJson = json })); +} +``` + +You can also set mutations using RDF format, if you so prefer, as seen below: + +```c# +var mutation = "_:alice \"Alice\" ."; +var transactionResult = await txn.Mutate(new RequestBuilder().WithMutations(new MutationBuilder{ SetNquads = mutation })); +``` + +Check out the example in `source/Dgraph.tests.e2e/TransactionTest.cs`. + +### Running a Query + +You can run a query by calling `Transaction.Query(string)`. You will need to pass in a +DQL query string. If you want to pass an additional map of any variables that +you might want to set in the query, call `Transaction.QueryWithVars(string, Dictionary)` with +the variables dictionary as the second argument. + +The response would contain the response string. + +Let’s run the following query with a variable `$a`: + +```console +query all($a: string) { + all(func: eq(name, $a)) + { + name + } +} +``` + +Run the query, deserialize the result from Uint8Array (or base64) encoded JSON and +print it out: + +```c# +// Run query. +var query = @"query all($a: string) { + all(func: eq(name, $a)) + { + name + } +}"; + +var vars = new Dictionary { { $a: "Alice" } }; +var res = await dgraphClient.NewReadOnlyTransaction().QueryWithVars(query, vars); + +// Print results. +Console.Write(res.Value.Json); +``` + +### Running an Upsert: Query + Mutation + +The `Transaction.Mutate` function allows you to run upserts consisting of one query and one mutation. + + +```c# +var query = @" + query { + user as var(func: eq(email, \"wrong_email@dgraph.io\")) + }"; + +var mutation = new MutationBuilder{ SetNquads = "uid(user) \"correct_email@dgraph.io\" ." }; + +var request = new RequestBuilder{ Query = query, CommitNow = true }.withMutation(mutation); + +// Upsert: If wrong_email found, update the existing data +// or else perform a new mutation. +await txn.Mutate(request); +``` + +### Committing a Transaction + +A transaction can be committed using the `Transaction.Commit` method. If your transaction +consisted solely of calls to `Transaction.Query` or `Transaction.QueryWithVars`, and no calls to +`Transaction.Mutate`, then calling `Transaction.Commit` is not necessary. + +An error will be returned if other transactions running concurrently modify the same +data that was modified in this transaction. It is up to the user to retry +transactions when they fail. + +```c# +using(var txn = client.NewTransaction()) { + var result = txn.Commit(); +} +``` diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/clients/go.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/clients/go.md new file mode 100644 index 00000000..edefe7c2 --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/clients/go.md @@ -0,0 +1,100 @@ +--- +title: Go +--- + +[![GoDoc](https://pkg.go.dev/badge/github.com/dgraph-io/dgo)](https://pkg.go.dev/github.com/dgraph-io/dgo/v250) + +The official Dgraph Go client communicates with the server using [gRPC](https://grpc.io/). + +## Installation + +```sh +go get github.com/dgraph-io/dgo/v250 +``` + +## Supported Versions + +| Dgraph version | dgo version | Import path | +| -------------- | ----------- | ------------------------------- | +| dgraph 23.X.Y | dgo 230.X.Y | `github.com/dgraph-io/dgo/v230` | +| dgraph 24.X.Y | dgo 240.X.Y | `github.com/dgraph-io/dgo/v240` | +| dgraph 25.X.Y | dgo 250.X.Y | `github.com/dgraph-io/dgo/v250` | + +## Quick Start + +### Using Connection Strings (v25+) + +The simplest way to connect is using a connection string: + +```go +client, err := dgo.Open("dgraph://localhost:9080") +if err != nil { + log.Fatal(err) +} +defer client.Close() +``` + +With ACL authentication: + +```go +client, err := dgo.Open("dgraph://groot:password@localhost:9080") +``` + + +### Running Queries and Mutations + +```go +// Set schema +err := client.SetSchema(ctx, `name: string @index(exact) .`) + +// Run a mutation +resp, err := client.RunDQL(ctx, `{ + set { + _:alice "Alice" . + } +}`) + +// Run a query +resp, err := client.RunDQL(ctx, `{ + alice(func: eq(name, "Alice")) { + name + } +}`) +fmt.Printf("%s\n", resp.Json) +``` + +## Multi-tenancy + +In multi-tenant environments, use `LoginIntoNamespace()` to authenticate to a specific namespace: + +```go +conn, err := grpc.Dial("127.0.0.1:9080", grpc.WithInsecure()) +if err != nil { + log.Fatal(err) +} +dc := dgo.NewDgraphClient(api.NewDgraphClient(conn)) +ctx := context.Background() + +// Login to namespace 123 +if err := dc.LoginIntoNamespace(ctx, "groot", "password", 123); err != nil { + log.Fatal(err) +} +``` + +Once logged in, the client can perform all operations allowed for that user in the specified namespace. + +## Documentation + +For complete API documentation, examples, and advanced usage: + +- **[GitHub Repository](https://github.com/dgraph-io/dgo)** — Full README with all APIs and examples +- **[GoDoc Reference](https://pkg.go.dev/github.com/dgraph-io/dgo/v250)** — Complete API documentation + +The GitHub README covers: +- Connection strings and advanced client creation +- Transactions (read-only, best-effort) +- Mutations (JSON and RDF formats) +- Queries with variables +- Upserts and conditional upserts +- Namespace management +- And more diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/clients/index.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/clients/index.md new file mode 100644 index 00000000..d024e93d --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/clients/index.md @@ -0,0 +1,56 @@ +--- +title: Client Libraries +description: Dgraph client libraries in various programming languages. +--- + +Dgraph client libraries allow you to run DQL transactions, queries and mutations in various programming languages. + +If you are interested in clients for GraphQL endpoint, please refer to [GraphQL clients](/graphql/graphql-clients) section. + + +Go, python, Java, C# and JavaScript clients are using **[gRPC](http://www.grpc.io/):** protocol and [Protocol + Buffers](https://developers.google.com/protocol-buffers) (the proto file +used by Dgraph is located at +[api.proto](https://github.com/dgraph-io/dgo/blob/master/protos/api.proto)). + +A JavaScript client using **HTTP:** is also available. + + +It's possible to interface with Dgraph directly via gRPC or HTTP. However, if a +client library exists for your language, that will be an easier option. + +:::tip +For multi-node setups, predicates are assigned to the group that first sees that +predicate. Dgraph also automatically moves predicate data to different groups in +order to balance predicate distribution. This occurs automatically every 10 +minutes. It's possible for clients to aid this process by communicating with all +Dgraph instances. For the Go client, this means passing in one +`*grpc.ClientConn` per Dgraph instance, or routing traffic through a load balancer. +Mutations will be made in a round robin +fashion, resulting in a semi-random initial predicate distribution. +::: + + +### Transactions + +Dgraph clients perform mutations and queries using transactions. A +transaction bounds a sequence of queries and mutations that are committed by +Dgraph as a single unit: that is, on commit, either all the changes are accepted +by Dgraph or none are. + +A transaction always sees the database state at the moment it began, plus any +changes it makes --- changes from concurrent transactions aren't visible. + +On commit, Dgraph will abort a transaction, rather than committing changes, when +a conflicting, concurrently running transaction has already been committed. Two +transactions conflict when both transactions: + +- write values to the same scalar predicate of the same node (e.g both + attempting to set a particular node's `address` predicate); or +- write to a singular `uid` predicate of the same node (changes to `[uid]` predicates can be concurrently written); or +- write a value that conflicts on an index for a predicate with `@upsert` set in the schema (see [upserts](/dql/upserts)). + +When a transaction is aborted, all its changes are discarded. Transactions can be manually aborted. + + +### In this section diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/clients/java.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/clients/java.md new file mode 100644 index 00000000..730d8db4 --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/clients/java.md @@ -0,0 +1,101 @@ +--- +title: Java +--- + +[![Maven Central](https://img.shields.io/maven-central/v/io.dgraph/dgraph4j)](https://search.maven.org/artifact/io.dgraph/dgraph4j) + +The official Dgraph Java client communicates with the server using [gRPC](https://grpc.io/). + +## Installation + +Via Gradle: + +```groovy +implementation 'io.dgraph:dgraph4j:25.0.0' +``` + +Via Maven: + +```xml + + io.dgraph + dgraph4j + 25.0.0 + +``` + +## Supported Versions + +| Dgraph version | dgraph4j version | Java version | +| -------------- | ----------------- | ------------ | +| dgraph 24.X.Y | dgraph4j 24.X.Y | 11 | +| dgraph 25.X.Y | dgraph4j 25.X.Y | 11 | + +## Quick Start + +### Using Connection Strings (v25+) + +The simplest way to connect is using a connection string: + +```java +DgraphClient client = DgraphClient.open("dgraph://localhost:9080"); +``` + +With ACL authentication: + +```java +DgraphClient client = DgraphClient.open("dgraph://groot:password@localhost:9080"); +``` + +### Running Queries and Mutations + +```java +// Set schema +client.setSchema("name: string @index(exact) ."); + +// Run a DQL mutation +client.runDQL("{set { _:alice \"Alice\" . }}"); + +// Run a query +Response response = client.runDQL( + "{ alice(func: eq(name, \"Alice\")) { name } }"); +System.out.println(response.getJson().toStringUtf8()); + +// Clean up +client.shutdown(); +``` + +## Multi-tenancy + +In multi-tenant environments, use `loginIntoNamespace()` to authenticate to a specific namespace: + +```java +ManagedChannel channel = ManagedChannelBuilder + .forAddress("localhost", 9080) + .usePlaintext().build(); +DgraphClient client = new DgraphClient(DgraphGrpc.newStub(channel)); + +// Login to namespace 123 +client.loginIntoNamespace("groot", "password", 123); +``` + +Once logged in, the client can perform all operations allowed for that user in the specified namespace. + +## Documentation + +For complete API documentation, examples, and advanced usage: + +- **[GitHub Repository](https://github.com/dgraph-io/dgraph4j)** — Full README with all APIs and examples +- **[Maven Central](https://search.maven.org/artifact/io.dgraph/dgraph4j)** — Package information and releases + +The GitHub README covers: +- Connection strings and advanced client creation +- Transactions (read-only, best-effort) +- Mutations (JSON and RDF formats) +- Queries with variables +- Upserts and conditional upserts +- Exception handling and automatic retry +- Namespace management and ID allocation +- TLS configuration +- Async client usage +- And more diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/clients/javascript/grpc.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/clients/javascript/grpc.md new file mode 100644 index 00000000..de1f5467 --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/clients/javascript/grpc.md @@ -0,0 +1,404 @@ +--- +title: gRPC Client +--- + +The official Dgraph client implementation for JavaScript, using +[gRPC-js](https://www.npmjs.com/package/@grpc/grpc-js) (the original +[gRPC](https://grpc.io/) client for JavaScript is now deprecated). + +This client follows the [Dgraph Go client](../go) closely. + + +:::tip +You can find the official Dgraph JavaScript gRPC client at: https://github.com/dgraph-io/dgraph-js. +Follow the [installation instructions](https://github.com/dgraph-io/dgraph-js#install) to get it up and running. +::: + +## Supported Versions + +More details on the supported versions can be found at [this link](https://github.com/dgraph-io/dgraph-js#supported-versions). + +## Quickstart + +Build and run the [simple project](https://github.com/dgraph-io/dgraph-js/tree/master/examples/simple), which +contains an end-to-end example of using the Dgraph JavaScript client. Follow the +instructions in the [README](https://github.com/dgraph-io/dgraph-js/tree/master/examples/simple/README.md) of that project. + +### Examples + +- [simple](https://github.com/dgraph-io/dgraph-js/tree/master/examples/simple): Quickstart example of using dgraph-js. +- [tls](https://github.com/dgraph-io/dgraph-js/tree/master/examples/tls): Example of using dgraph-js with a Dgraph cluster secured with TLS. + +## Using a Client + +:::tip +You can find a [simple example](https://github.com/dgraph-io/dgraph-js/tree/master/examples/simple) +project, which contains an end-to-end working example of how to use the JavaScript gRPC client, +for Node.js >= v6. +::: + +### Creating a Client + +A `DgraphClient` object can be initialized by passing it a list of +`DgraphClientStub` clients as variadic arguments. Connecting to multiple Dgraph +servers in the same cluster allows for better distribution of workload. + +The following code snippet shows just one connection. + +```js +const dgraph = require("dgraph-js"); +const grpc = require("grpc"); + +const clientStub = new dgraph.DgraphClientStub( + // addr: optional, default: "localhost:9080" + "localhost:9080", + // credentials: optional, default: grpc.credentials.createInsecure() + grpc.credentials.createInsecure(), +); +const dgraphClient = new dgraph.DgraphClient(clientStub); +``` + +To facilitate debugging, [debug mode](#debug-mode) can be enabled for a client. + +### Multi-tenancy + +In [multi-tenancy](../../../admin/admin-tasks/multitenancy) environments, `dgraph-js` provides a new method `loginIntoNamespace()`, +which will allow the users to login to a specific namespace. + +In order to create a JavaScript client, and make the client login into namespace `123`: + +```js +const dgraphClientStub = new dgraph.DgraphClientStub("localhost:9080"); +await dgraphClientStub.loginIntoNamespace("groot", "password", 123); // where 123 is the namespaceId +``` + +In the example above, the client logs into namespace `123` using username `groot` and password `password`. +Once logged in, the client can perform all the operations allowed to the `groot` user of namespace `123`. + + +### Altering the Database + +To set the schema, create an `Operation` object, set the schema and pass it to +`DgraphClient#alter(Operation)` method. + +```js +const schema = "name: string @index(exact) ."; +const op = new dgraph.Operation(); +op.setSchema(schema); +await dgraphClient.alter(op); +``` + +Indexes can be computed in the background. +You can set `setRunInBackground` field of the `Operation` object to `true` +before passing it to the `DgraphClient#alter(Operation)` method. You can find more details +[here](../../admin/admin-tasks/update-dgraph-types.md#indexes-in-background). + +```js +const schema = "name: string @index(exact) ."; +const op = new dgraph.Operation(); +op.setSchema(schema); +op.setRunInBackground(true); +await dgraphClient.alter(op); +``` + +> NOTE: Many of the examples here use the `await` keyword which requires +> `async/await` support which is available on Node.js >= v7.6.0. For prior versions, +> the expressions following `await` can be used just like normal `Promise`: +> +> ```js +> dgraphClient.alter(op) +> .then(function(result) { ... }, function(err) { ... }) +> ``` + +`Operation` contains other fields as well, including drop predicate and drop all. +Drop all is useful if you wish to discard all the data, and start from a clean +slate, without bringing the instance down. + +```js +// Drop all data including schema from the Dgraph instance. This is useful +// for small examples such as this, since it puts Dgraph into a clean +// state. +const op = new dgraph.Operation(); +op.setDropAll(true); +await dgraphClient.alter(op); +``` + +### Creating a Transaction + +To create a transaction, call `DgraphClient#newTxn()` method, which returns a +new `Txn` object. This operation incurs no network overhead. + +It is good practice to call `Txn#discard()` in a `finally` block after running +the transaction. Calling `Txn#discard()` after `Txn#commit()` is a no-op +and you can call `Txn#discard()` multiple times with no additional side-effects. + +```js +const txn = dgraphClient.newTxn(); +try { + // Do something here + // ... +} finally { + await txn.discard(); + // ... +} +``` + +To create a read-only transaction, set `readOnly` boolean to `true` while calling +`DgraphClient#newTxn()` method. Read-only transactions cannot contain mutations and +trying to call `Txn#mutate()` or `Txn#commit()` will result in an error. Calling +`Txn.Discard()` will be a no-op. + +You can optionally set the `bestEffort` boolean to `true`. This may yield improved +latencies in read-bound workloads where linearizable reads are not strictly needed. + +```js +const txn = dgraphClient.newTxn({ + readOnly: true, + bestEffort: false +}); +// ... +const res = await txn.queryWithVars(query, vars); +``` + +### Running a Mutation + +`Txn#mutate(Mutation)` runs a mutation. It takes in a `Mutation` object, which +provides two main ways to set data: JSON and RDF N-Quad. You can choose whichever +way is convenient. + +We define a person object to represent a person and use it in a `Mutation` object. + +```js +// Create data. +const p = { + name: "Alice", +}; + +// Run mutation. +const mu = new dgraph.Mutation(); +mu.setSetJson(p); +await txn.mutate(mu); +``` + +For a more complete example with multiple fields and relationships, look at the +[simple] project in the `examples` folder. + +Sometimes, you only want to commit a mutation, without querying anything further. +In such cases, you can use `Mutation#setCommitNow(true)` to indicate that the +mutation must be immediately committed. + +`Mutation#setIgnoreIndexConflict(true)` can be applied on a `Mutation` object to +not run conflict detection over the index, which would decrease the number of +transaction conflicts and aborts. However, this would come at the cost of potentially +inconsistent upsert operations. + +Mutation can be run using `txn.doRequest` as well. + +```js +const mu = new dgraph.Mutation(); +mu.setSetJson(p); + +const req = new dgraph.Request(); +req.setCommitNow(true); +req.setMutationsList([mu]); + +await txn.doRequest(req); +``` + +### Running a Query + +You can run a query by calling `Txn#query(string)`. You will need to pass in a +GraphQL+- query string. If you want to pass an additional map of any variables that +you might want to set in the query, call `Txn#queryWithVars(string, object)` with +the variables object as the second argument. + +The response would contain the method `Response#getJSON()`, which returns the response +JSON. + +Let’s run the following query with a variable $a: + +```console +query all($a: string) { + all(func: eq(name, $a)) + { + name + } +} +``` + +Run the query, deserialize the result from Uint8Array (or base64) encoded JSON and +print it out: + +```js +// Run query. +const query = `query all($a: string) { + all(func: eq(name, $a)) + { + name + } +}`; +const vars = { $a: "Alice" }; +const res = await dgraphClient.newTxn().queryWithVars(query, vars); +const ppl = res.getJson(); + +// Print results. +console.log(`Number of people named "Alice": ${ppl.all.length}`); +ppl.all.forEach((person) => console.log(person.name)); +``` + +This should print: + +```console +Number of people named "Alice": 1 +Alice +``` + +You can also use `txn.doRequest` function to run the query. +```js +const req = new dgraph.Request(); +const vars = req.getVarsMap(); +vars.set("$a", "Alice"); +req.setQuery(query); + +const res = await txn.doRequest(req); +console.log(JSON.stringify(res.getJson())); +``` + +### Running an Upsert: Query + Mutation + +The `txn.doRequest` function allows you to run upserts consisting of one query and one mutation. +Query variables could be defined and can then be used in the mutation. You can also use the +`txn.doRequest` function to perform just a query or a mutation. + + +```js +const query = ` + query { + user as var(func: eq(email, "wrong_email@dgraph.io")) + }` + +const mu = new dgraph.Mutation(); +mu.setSetNquads(`uid(user) "correct_email@dgraph.io" .`); + +const req = new dgraph.Request(); +req.setQuery(query); +req.setMutationsList([mu]); +req.setCommitNow(true); + +// Upsert: If wrong_email found, update the existing data +// or else perform a new mutation. +await dgraphClient.newTxn().doRequest(req); +``` + +### Running a Conditional Upsert + +The upsert block allows specifying a conditional mutation block using an `@if` directive. The mutation is executed +only when the specified condition is true. If the condition is false, the mutation is silently ignored. + +See more about Conditional Upsert [Here](../../dql/dql-mutation#conditional-upsert). + +```js +const query = ` + query { + user as var(func: eq(email, "wrong_email@dgraph.io")) + }` + +const mu = new dgraph.Mutation(); +mu.setSetNquads(`uid(user) "correct_email@dgraph.io" .`); +mu.setCond(`@if(eq(len(user), 1))`); + +const req = new dgraph.Request(); +req.setQuery(query); +req.addMutations(mu); +req.setCommitNow(true); + +await dgraphClient.newTxn().doRequest(req); +``` + +### Committing a Transaction + +A transaction can be committed using the `Txn#commit()` method. If your transaction +consisted solely of calls to `Txn#query` or `Txn#queryWithVars`, and no calls to +`Txn#mutate`, then calling `Txn#commit()` is not necessary. + +An error will be returned if other transactions running concurrently modify the same +data that was modified in this transaction. It is up to the user to retry +transactions when they fail. + +```js +const txn = dgraphClient.newTxn(); +try { + // ... + // Perform any number of queries and mutations + // ... + // and finally... + await txn.commit(); +} catch (e) { + if (e === dgraph.ERR_ABORTED) { + // Retry or handle exception. + } else { + throw e; + } +} finally { + // Clean up. Calling this after txn.commit() is a no-op + // and hence safe. + await txn.discard(); +} +``` + +### Cleanup Resources + +To cleanup resources, you have to call `DgraphClientStub#close()` individually for +all the instances of `DgraphClientStub`. + +```js +const SERVER_ADDR = "localhost:9080"; +const SERVER_CREDENTIALS = grpc.credentials.createInsecure(); + +// Create instances of DgraphClientStub. +const stub1 = new dgraph.DgraphClientStub(SERVER_ADDR, SERVER_CREDENTIALS); +const stub2 = new dgraph.DgraphClientStub(SERVER_ADDR, SERVER_CREDENTIALS); + +// Create an instance of DgraphClient. +const dgraphClient = new dgraph.DgraphClient(stub1, stub2); + +// ... +// Use dgraphClient +// ... + +// Cleanup resources by closing all client stubs. +stub1.close(); +stub2.close(); +``` + +### Debug mode + +Debug mode can be used to print helpful debug messages while performing alters, +queries and mutations. It can be set using the`DgraphClient#setDebugMode(boolean?)` +method. + +```js +// Create a client. +const dgraphClient = new dgraph.DgraphClient(...); + +// Enable debug mode. +dgraphClient.setDebugMode(true); +// OR simply dgraphClient.setDebugMode(); + +// Disable debug mode. +dgraphClient.setDebugMode(false); +``` + +### Setting Metadata Headers + +Metadata headers such as authentication tokens can be set through the context of gRPC methods. Below is an example of how to set a header named "auth-token". + +```js +// The following piece of code shows how one can set metadata with +// auth-token, to allow Alter operation, if the server requires it. + +var meta = new grpc.Metadata(); +meta.add('auth-token', 'mySuperSecret'); + +await dgraphClient.alter(op, meta); +``` diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/clients/javascript/http.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/clients/javascript/http.md new file mode 100644 index 00000000..be2cc44e --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/clients/javascript/http.md @@ -0,0 +1,323 @@ +--- +title: HTTP Client +--- + +A Dgraph client implementation for JavaScript using HTTP. It supports both +browser and Node.js environments. +This client follows the [Dgraph JavaScript gRPC client](grpc) closely. + +:::tip +The official JavaScript HTTP client [can be found here](https://github.com/dgraph-io/dgraph-js-http). +Follow the [install instructions](https://github.com/dgraph-io/dgraph-js-http#install) to get it up and running. +::: + +## Supported Versions + +More details on the supported versions can be found at [this link](https://github.com/dgraph-io/dgraph-js-http#supported-versions). + +## Quickstart + +Build and run the [simple project](https://github.com/dgraph-io/dgraph-js-http/tree/master/examples/simple), which +contains an end-to-end example of using the Dgraph javascript HTTP client. Follow +the instructions in the [README](https://github.com/dgraph-io/dgraph-js-http/tree/master/examples/simple/README.md) of that project. + +## Using a client + +:::tip +You can find a [simple example](https://github.com/dgraph-io/dgraph-js-http/tree/master/examples/simple) +project, which contains an end-to-end working example of how to use the JavaScript HTTP client, +for Node.js >= v6. +::: + +### Create a client + +A `DgraphClient` object can be initialized by passing it a list of +`DgraphClientStub` clients as variadic arguments. Connecting to multiple Dgraph +servers in the same cluster allows for better distribution of workload. + +The following code snippet shows just one connection. + +```js +const dgraph = require("dgraph-js-http"); + +const clientStub = new dgraph.DgraphClientStub( + // addr: optional, default: "http://localhost:8080" + "http://localhost:8080", + // legacyApi: optional, default: false. Set to true when connecting to Dgraph v1.0.x + false, +); +const dgraphClient = new dgraph.DgraphClient(clientStub); +``` + +To facilitate debugging, [debug mode](#debug-mode) can be enabled for a client. + + +### Login into Dgraph + +If your Dgraph server has Access Control Lists enabled (Dgraph v1.1 or above), +the clientStub must be logged in for accessing data: + +```js +await clientStub.login("groot", "password"); +``` + +Calling `login` will obtain and remember the access and refresh JWT tokens. +All subsequent operations via the logged in `clientStub` will send along the +stored access token. + +Access tokens expire after 6 hours, so in long-lived apps (e.g. business logic servers) +you need to `login` again on a periodic basis: + +```js +// When no parameters are specified the clientStub uses existing refresh token +// to obtain a new access token. +await clientStub.login(); +``` + +### Configure access tokens + +Some Dgraph configurations require extra access tokens. + +Alpha servers can be configured with [Secure Alter Operations](../../admin/security/admin-endpoint-security). + In this case the token needs to be set on the client instance: + +```js +dgraphClient.setAlphaAuthToken("My secret token value"); +``` + + + +### Create https connection + +If your cluster is using tls/mtls you can pass a node `https.Agent` configured with you +certificates as follows: + +```js +const https = require("https"); +const fs = require("fs"); +// read your certificates +const cert = fs.readFileSync("./certs/client.crt", "utf8"); +const ca = fs.readFileSync("./certs/ca.crt", "utf8"); +const key = fs.readFileSync("./certs/client.key", "utf8"); + +// create your https.Agent +const agent = https.Agent({ + cert, + ca, + key, +}); + +const clientStub = new dgraph.DgraphClientStub( + "https://localhost:8080", + false, + { agent }, +); +const dgraphClient = new dgraph.DgraphClient(clientStub); +``` + +### Alter the database + +To set the schema, pass the schema to `DgraphClient#alter(Operation)` method. + +```js +const schema = "name: string @index(exact) ."; +await dgraphClient.alter({ schema: schema }); +``` + +> NOTE: Many of the examples here use the `await` keyword which requires +> `async/await` support which is not available in all javascript environments. +> For unsupported environments, the expressions following `await` can be used +> just like normal `Promise` instances. + +`Operation` contains other fields as well, including drop predicate and drop all. +Drop all is useful if you wish to discard all the data, and start from a clean +slate, without bringing the instance down. + +```js +// Drop all data including schema from the Dgraph instance. This is useful +// for small examples such as this, since it puts Dgraph into a clean +// state. +await dgraphClient.alter({ dropAll: true }); +``` + +### Create a transaction + +To create a transaction, call `DgraphClient#newTxn()` method, which returns a +new `Txn` object. This operation incurs no network overhead. + +It is good practice to call `Txn#discard()` in a `finally` block after running +the transaction. Calling `Txn#discard()` after `Txn#commit()` is a no-op +and you can call `Txn#discard()` multiple times with no additional side-effects. + +```js +const txn = dgraphClient.newTxn(); +try { + // Do something here + // ... +} finally { + await txn.discard(); + // ... +} +``` + +You can make queries read-only and best effort by passing `options` to `DgraphClient#newTxn`. For example: + +```js +const options = { readOnly: true, bestEffort: true }; +const res = await dgraphClient.newTxn(options).query(query); +``` + +Read-only transactions are useful to increase read speed because they can circumvent the usual consensus protocol. Best effort queries can also increase read speed in read bound system. Please note that best effort requires readonly. + +### Run a mutation + +`Txn#mutate(Mutation)` runs a mutation. It takes in a `Mutation` object, which +provides two main ways to set data: JSON and RDF N-Quad. You can choose whichever +way is convenient. + +We define a person object to represent a person and use it in a `Mutation` object. + +```js +// Create data. +const p = { + name: "Alice", +}; + +// Run mutation. +await txn.mutate({ setJson: p }); +``` + +For a more complete example with multiple fields and relationships, look at the +[simple] project in the `examples` folder. + +For setting values using N-Quads, use the `setNquads` field. For delete mutations, +use the `deleteJson` and `deleteNquads` fields for deletion using JSON and N-Quads +respectively. + +Sometimes, you only want to commit a mutation, without querying anything further. +In such cases, you can use `Mutation#commitNow = true` to indicate that the +mutation must be immediately committed. + +```js +// Run mutation. +await txn.mutate({ setJson: p, commitNow: true }); +``` + +### Run a query + +You can run a query by calling `Txn#query(string)`. You will need to pass in a +GraphQL+- query string. If you want to pass an additional map of any variables that +you might want to set in the query, call `Txn#queryWithVars(string, object)` with +the variables object as the second argument. + +The response would contain the `data` field, `Response#data`, which returns the response +JSON. + +Let’s run the following query with a variable \$a: + +```console +query all($a: string) { + all(func: eq(name, $a)) + { + name + } +} +``` + +Run the query and print out the response: + +```js +// Run query. +const query = `query all($a: string) { + all(func: eq(name, $a)) + { + name + } +}`; +const vars = { $a: "Alice" }; +const res = await dgraphClient.newTxn().queryWithVars(query, vars); +const ppl = res.data; + +// Print results. +console.log(`Number of people named "Alice": ${ppl.all.length}`); +ppl.all.forEach(person => console.log(person.name)); +``` + +This should print: + +```console +Number of people named "Alice": 1 +Alice +``` + +### Commit a transaction + +A transaction can be committed using the `Txn#commit()` method. If your transaction +consisted solely of calls to `Txn#query` or `Txn#queryWithVars`, and no calls to +`Txn#mutate`, then calling `Txn#commit()` is not necessary. + +An error will be returned if other transactions running concurrently modify the same +data that was modified in this transaction. It is up to the user to retry +transactions when they fail. + +```js +const txn = dgraphClient.newTxn(); +try { + // ... + // Perform any number of queries and mutations + // ... + // and finally... + await txn.commit(); +} catch (e) { + if (e === dgraph.ERR_ABORTED) { + // Retry or handle exception. + } else { + throw e; + } +} finally { + // Clean up. Calling this after txn.commit() is a no-op + // and hence safe. + await txn.discard(); +} +``` + +### Check request latency + +To see the server latency information for requests, check the +`extensions.server_latency` field from the Response object for queries or from +the Assigned object for mutations. These latencies show the amount of time the +Dgraph server took to process the entire request. It does not consider the time +over the network for the request to reach back to the client. + +```js +// queries +const res = await txn.queryWithVars(query, vars); +console.log(res.extensions.server_latency); +// { parsing_ns: 29478, +// processing_ns: 44540975, +// encoding_ns: 868178 } + +// mutations +const assigned = await txn.mutate({ setJson: p }); +console.log(assigned.extensions.server_latency); +// { parsing_ns: 132207, +// processing_ns: 84100996 } +``` + +### Debug mode + +Debug mode can be used to print helpful debug messages while performing alters, +queries and mutations. It can be set using the`DgraphClient#setDebugMode(boolean?)` +method. + +```js +// Create a client. +const dgraphClient = new dgraph.DgraphClient(...); + +// Enable debug mode. +dgraphClient.setDebugMode(true); +// OR simply dgraphClient.setDebugMode(); + +// Disable debug mode. +dgraphClient.setDebugMode(false); +``` diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/clients/javascript/index.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/clients/javascript/index.md new file mode 100644 index 00000000..0c6b33f5 --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/clients/javascript/index.md @@ -0,0 +1,27 @@ +--- +title: JavaScript +--- + +## gRPC JS Client + +The official JavaScript gRPC client documentation [can be found here](grpc). + +More details on the supported versions can be found at [this link](https://github.com/dgraph-io/dgraph-js#supported-versions). + +:::tip +You can find a [simple example](https://github.com/dgraph-io/dgraph-js/tree/master/examples/simple) +project, which contains an end-to-end working example of how to use the JavaScript gRPC client, +for Node.js >= v6. +::: + +## HTTP JS Client + +The official JavaScript HTTP client documentation [can be found here](http). + +More details on the supported versions can be found at [this link](https://github.com/dgraph-io/dgraph-js-http#supported-versions). + +:::tip +You can find a [simple example](https://github.com/dgraph-io/dgraph-js-http/tree/master/examples/simple) +project, which contains an end-to-end working example of how to use the JavaScript HTTP client, +for Node.js >= v6. +::: diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/clients/python.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/clients/python.md new file mode 100644 index 00000000..44b5fee9 --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/clients/python.md @@ -0,0 +1,100 @@ +--- +title: Python +--- + +[![PyPI](https://img.shields.io/pypi/v/pydgraph)](https://pypi.org/project/pydgraph/) + +The official Dgraph Python client communicates with the server using [gRPC](https://grpc.io/). + +## Installation + +```sh +pip install pydgraph +``` + +## Supported Versions + +| Dgraph version | pydgraph version | +| -------------- | ---------------- | +| dgraph 21.03.X | pydgraph 21.03.X | +| dgraph 23.0.X | pydgraph 23.0.X | +| dgraph 24.X.Y | pydgraph 24.X.Y | +| dgraph 25.X.Y | pydgraph 25.X.Y | + +## Quick Start + +### Using Connection Strings (v25+) + +The simplest way to connect is using a connection string: + +```python +import pydgraph + +client = pydgraph.open("dgraph://localhost:9080") +``` + +With ACL authentication: + +```python +client = pydgraph.open("dgraph://groot:password@localhost:9080") +``` + +### Running Queries and Mutations + +```python +import pydgraph +import json + +# Create client +client = pydgraph.open("dgraph://localhost:9080") + +# Set schema +op = pydgraph.Operation(schema='name: string @index(exact) .') +client.alter(op) + +# Run a mutation +txn = client.txn() +try: + txn.mutate(set_obj={'name': 'Alice'}, commit_now=True) +finally: + txn.discard() + +# Run a query +query = '{ alice(func: eq(name, "Alice")) { name } }' +res = client.txn(read_only=True).query(query) +print(json.loads(res.json)) + +# Clean up +client.close() +``` + +## Multi-tenancy + +In multi-tenant environments, use `login_into_namespace()` to authenticate to a specific namespace: + +```python +client_stub = pydgraph.DgraphClientStub('localhost:9080') +client = pydgraph.DgraphClient(client_stub) + +# Login to namespace 123 +client.login_into_namespace("groot", "password", "123") +``` + +Once logged in, the client can perform all operations allowed for that user in the specified namespace. + +## Documentation + +For complete API documentation, examples, and advanced usage: + +- **[GitHub Repository](https://github.com/dgraph-io/pydgraph)** — Full README with all APIs and examples +- **[PyPI Package](https://pypi.org/project/pydgraph/)** — Package information and releases + +The GitHub README covers: +- Connection strings and client creation +- Transactions (read-only, best-effort) +- Mutations (JSON and RDF formats) +- Queries with variables +- Upserts and conditional upserts +- Async/await support +- TLS configuration +- And more diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/clients/raw-http.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/clients/raw-http.md new file mode 100644 index 00000000..ecd4f392 --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/clients/raw-http.md @@ -0,0 +1,590 @@ +--- +title: Raw HTTP +--- + +Dgraph exposes HTTP endpoints for querying, mutating, and managing the database. Use these endpoints to build clients in languages without gRPC support. + +This documentation describes the HTTP API endpoints and request/response formats. Examples use `curl` and [`jq`](https://stedolan.github.io/jq/) to demonstrate the API. For gRPC-based client implementation, see the [Go client](go). + +## Alter the DQL Schema + +You need to alter the DQL schema to declare predicate types, add predicate search indexes, and declare the predicates expected in entities of specific types. + +Update the DQL schema by posting schema data to the `/alter` endpoint: + +```sh +curl "localhost:8080/alter" --silent --request POST \ + --data $' +name: string @index(term) . +balance: int . + +type User { + name + balance +} +' | jq +``` + +Alternatively, you can post the schema from a file: + +```sh +curl "localhost:8080/alter" --silent --request POST \ + --data-binary @schema.dql | jq +``` + +Where `schema.dql` contains: + +```dql +name: string @index(term) . +balance: int . + +type User { + name + balance +} +``` + +**Success response:** + +```json +{ + "data": { + "code": "Success", + "message": "Done" + } +} +``` + +**Error response:** + +In case of errors, the API will reply with an error message such as: + +```json +{ + "errors": [ + { + "extensions": { + "code": "Error" + }, + "message": "line 5 column 18: Invalid ending" + } + ] +} +``` + +:::note +The request will update or create the predicates and types present in the request. It will not modify or delete other schema information that may be present. +::: + +## Query Current DQL Schema + +Obtain the DQL schema by issuing a DQL query on the `/query` endpoint: + +```sh +curl -X POST \ + -H "Content-Type: application/dql" \ + localhost:8080/query -d $'schema {}' | jq +``` + +:::note +The schema is returned as json document. + +You may need to convert this json to the format used by the `alter` operation. + +Check [dgraph_schema_converter] (https://github.com/dgraph-io/dgraph-experimental/blob/main/dql-helper/dgraph_schema_converter.sh) in dgraph-experimental repository. +::: + +## Create Data with Mutations + +Mutations can be done over HTTP by making a `POST` request to an Alpha's `/mutate` endpoint. Set the `Content-Type` header to `application/rdf` to specify that the mutation is written in RDF format. + +### Mutation using RDF format +To create Alice and Bob with initial balances: +```sh +curl -H "Content-Type: application/rdf" -X POST "localhost:8080/mutate?commitNow=true" -d $' +{ + set { + _:alice "Alice" . + _:alice "100" . + _:alice "User" . + _:bob "Bob" . + _:bob "70" . + _:bob "User" . + } +} +' | jq +``` + +**Response:** + +```json +{ + "data": { + "code": "Success", + "message": "Done", + "uids": { + "alice": "0x1", + "bob": "0x2" + } + }, + "extensions": { + "server_latency": { + "parsing_ns": 50901, + "processing_ns": 14631082 + }, + "txn": { + "start_ts": 4, + "commit_ts": 5 + } + } +} +``` + +The response contains `uids` which maps the blank node identifiers (`_:alice`, `_:bob`) to the actual UIDs assigned by Dgraph (`0x1`, `0x2`). These UIDs can be used in subsequent queries and mutations. + +:::note +The `commitNow=true` parameter commits the transaction immediately after the mutation. For multi-step transactions, you can omit this parameter and use the `/commit` endpoint separately. +::: + +### Mutation using JSON format + +The `mutate` operation also accepts JSON input format: +```sh +curl -H "Content-Type: application/json" \ + -X POST "localhost:8080/mutate?commitNow=true" \ + -d '{ + "set": [ + { + "uid": "_:alice", + "name": "Alice", + "balance": 100, + "dgraph.type": "User" + }, + { + "uid": "_:bob", + "name": "Bob", + "balance": 70, + "dgraph.type": "User" + } + ] + }' | jq +``` +Note that +- Content-type is 'application/json' +- The payload is valid json document: the `set` keyword has quotes. + +#### UID +`uid` can be omitted in the JSON if you don't have to retrieve the generated UID in the response. + + +## Query Data + +To query the database, use the `/query` endpoint. Set the `Content-Type` header to `application/dql` to ensure that the body of the request is parsed correctly. + +To get the balances for both users: + +```sh +curl -H "Content-Type: application/dql" -X POST localhost:8080/query -d $' +{ + users(func: anyofterms(name, "Alice Bob")) { + uid + name + balance + } +}' | jq +``` + +**Response:** + +```json +{ + "data": { + "users": [ + { + "uid": "0x1", + "name": "Alice", + "balance": "100" + }, + { + "uid": "0x2", + "name": "Bob", + "balance": "70" + } + ] + }, + "extensions": { + "server_latency": { + "parsing_ns": 70494, + "processing_ns": 697140, + "encoding_ns": 1560151 + }, + "txn": { + "start_ts": 6 + } + } +} +``` + +Notice that along with the query result under the `data` field, there is additional data in the `extensions -> txn` field. This data includes the transaction start timestamp (`start_ts`), which will need to be used if you want to perform mutations as part of the same transaction. + +### Query with RDF Response Format + +You can request query results in RDF format by adding the `respFormat=RDF` parameter to the query endpoint: + +```sh +curl -H "Content-Type: application/dql" -X POST "localhost:8080/query?respFormat=rdf" -d $' +{ + users(func: anyofterms(name, "Alice Bob")) { + uid + name + balance + } +}' | jq -r '.data' | sed 's/\\n/\n/g' +``` + +**Response (RDF format):** + +``` +<0x1> "Alice" . +<0x1> "100" . +<0x1> "User" . +<0x2> "Bob" . +<0x2> "70" . +<0x2> "User" . +``` +### Running Read-Only Queries + +You can set the query parameter `ro=true` to `/query` to set it as a [read-only](../dql/index.md#read-only-transactions) query: + +```sh +curl -H "Content-Type: application/dql" -X POST "localhost:8080/query?ro=true" -d $' +{ + users(func: anyofterms(name, "Alice Bob")) { + uid + name + balance + } +}' +``` + +### Running Best-Effort Queries + +You can set the query parameter `be=true` to `/query` to set it as a [best-effort](../dql/index.md#read-only-transactions) query: + +```sh +curl -H "Content-Type: application/dql" -X POST "localhost:8080/query?be=true" -d $' +{ + users(func: anyofterms(name, "Alice Bob")) { + uid + name + balance + } +}' +``` +## Transactions + +### Transaction State + +A client built on top of the HTTP API needs to track three pieces of state for each transaction: + +1. **A start timestamp (`start_ts`)**. This uniquely identifies a transaction and doesn't change over the transaction lifecycle. + +2. **The set of keys modified by the transaction (`keys`)**. This aids in transaction conflict detection. Every mutation returns a new set of keys. The client must merge them with the existing set. + +3. **The set of predicates modified by the transaction (`preds`)**. This aids in predicate move detection. Every mutation returns a new set of predicates. The client must merge them with the existing set. + +**For both query and mutation, if the `start_ts` is provided as a path parameter, then the operation is performed as part of the ongoing transaction. Otherwise, a new transaction is initiated.** + +### Update Data in a Transaction + +To update data within a transaction, first run a query to get the current state and transaction timestamp: + +```sh +curl -H "Content-Type: application/dql" -X POST localhost:8080/query -d $' +{ + users(func: anyofterms(name, "Alice Bob")) { + uid + name + balance + } +}' | jq +``` + +The response will include a `start_ts` in the `extensions -> txn` field. Use this `start_ts` for subsequent mutations in the same transaction. + +Now, if Bob transfers $10 to Alice, update the balances: + +```sh +curl -H "Content-Type: application/rdf" -X POST "localhost:8080/mutate?startTs=6" -d $' +{ + set { + <0x1> "110" . + <0x2> "60" . + } +}' | jq +``` + +**Response:** + +```json +{ + "data": { + "code": "Success", + "message": "Done", + "uids": {} + }, + "extensions": { + "server_latency": { + "parsing_ns": 50901, + "processing_ns": 14631082 + }, + "txn": { + "start_ts": 6, + "keys": [ + "2ahy9oh4s9csc", + "3ekeez23q5149" + ], + "preds": [ + "1-balance" + ] + } + } +} +``` + +The result contains `keys` and `preds` which should be added to the transaction state. + +### Commit the Transaction + +Commit the transaction using the `/commit` endpoint. Provide the `start_ts` you've been using for the transaction along with the list of `keys` and the list of predicates. If you performed multiple mutations in the transaction, the keys and predicates provided during the commit should be the union of all keys and predicates returned in the responses from the `/mutate` endpoint. + +The `preds` field is used to abort the transaction in cases where some of the predicates are moved. This field is not required and the `/commit` endpoint also accepts the old format, which was a single array of keys. + +```sh +curl -X POST localhost:8080/commit?startTs=6 -d $' +{ + "keys": [ + "2ahy9oh4s9csc", + "3ekeez23q5149" + ], + "preds": [ + "1-balance" + ] +}' | jq +``` + +**Response:** + +```json +{ + "data": { + "code": "Success", + "message": "Done" + }, + "extensions": { + "txn": { + "start_ts": 6, + "commit_ts": 7 + } + } +} +``` + +The transaction is now complete. + +If another client were to perform another transaction concurrently affecting the same keys, then it's possible that the transaction would *not* be successful. This is indicated in the response when the commit is attempted: + +```json +{ + "errors": [ + { + "code": "Error", + "message": "Transaction has been aborted. Please retry." + } + ] +} +``` + +In this case, it should be up to the user of the client to decide if they wish to retry the transaction. + +### Abort the Transaction + +To abort a transaction, use the same `/commit` endpoint with the `abort=true` parameter while specifying the `startTs` value for the transaction: + +```sh +curl -X POST "localhost:8080/commit?startTs=6&abort=true" | jq +``` + +**Response:** + +```json +{ + "code": "Success", + "message": "Done" +} +``` + +## Delete Data + +### Delete Specific Triples + +To delete specific triples (data), use the `delete` block in a mutation: + +```sh +curl -H "Content-Type: application/rdf" -X POST localhost:8080/mutate?commitNow=true -d $' +{ + delete { + <0x1> * . + } +}' | jq +``` + +This deletes all `balance` values for the node with UID `0x1`. + +To delete a specific value: + +```sh +curl -H "Content-Type: application/rdf" -X POST localhost:8080/mutate?commitNow=true -d $' +{ + delete { + <0x1> "110" . + } +}' | jq +``` + +### Delete an Entire Node + +To delete an entire node and all its predicates, use the `*` wildcard: + +```sh +curl -H "Content-Type: application/rdf" -X POST localhost:8080/mutate?commitNow=true -d $' +{ + delete { + <0x1> * * . + } +}' | jq +``` + +This deletes the node with UID `0x1` and all its data. + +### Delete Predicate + +To delete a predicate from the schema (and all its data), use the `/alter` endpoint with the `@drop` directive: + +```sh +curl "localhost:8080/alter" --silent --request POST \ + --data $'balance: int @drop .' | python -m json.tool +``` + +**Response:** + +```json +{ + "data": { + "code": "Success", + "message": "Done" + } +} +``` + +This removes the `balance` predicate from the schema and deletes all data stored in that predicate across all nodes. + +:::warning +Deleting a predicate is a destructive operation that cannot be undone. All data stored in that predicate will be permanently deleted. +::: + +### Delete Type + +To delete a type from the schema, use the `/alter` endpoint with the `@drop` directive on the type: + +```sh +curl "localhost:8080/alter" --silent --request POST \ + --data $'type User @drop .' | python -m json.tool +``` + +**Response:** + +```json +{ + "data": { + "code": "Success", + "message": "Done" + } +} +``` + +This removes the `User` type from the schema. Note that this does not delete the data nodes themselves, only the type definition. The nodes will still exist but will no longer have the `User` type assigned. + +:::note +To delete all nodes of a specific type, you would need to query for all nodes of that type and then delete them individually using the delete mutation syntax shown above. +::: + + + +## Compression via HTTP + +Dgraph supports gzip-compressed requests to and from Dgraph Alphas for `/query`, `/mutate`, and `/alter`. + +**Compressed requests:** To send compressed requests, set the HTTP request header `Content-Encoding: gzip` along with the gzip-compressed payload. + +**Compressed responses:** To receive gzipped responses, set the HTTP request header `Accept-Encoding: gzip` and Alpha will return gzipped responses. + +**Example of a compressed request via curl:** + +```sh +curl -X POST \ + -H 'Content-Encoding: gzip' \ + -H "Content-Type: application/rdf" \ + localhost:8080/mutate?commitNow=true --data-binary @mutation.gz +``` + +**Example of a compressed response via curl:** + +```sh +curl -X POST \ + -H 'Accept-Encoding: gzip' \ + -H "Content-Type: application/dql" \ + localhost:8080/query -d $'schema {}' | gzip --decompress +``` + +**Example of a compressed request and response via curl:** + +```sh +curl -X POST \ + -H 'Content-Encoding: gzip' \ + -H 'Accept-Encoding: gzip' \ + -H "Content-Type: application/dql" \ + localhost:8080/query --data-binary @query.gz | gzip --decompress +``` + +:::note +Curl has a `--compressed` option that automatically requests for a compressed response (`Accept-Encoding` header) and decompresses the compressed response: + +```sh +curl -X POST --compressed -H "Content-Type: application/dql" localhost:8080/query -d $'schema {}' +``` +::: + +## Run a Query in JSON Format + +The HTTP API also accepts requests in JSON format. For queries you have the keys "query" and "variables". The JSON format is required to set [GraphQL Variables](../dql/query/graphql-variables) with the HTTP API. + +This query: + +```dql +{ + users(func: anyofterms(name, "Alice Bob")) { + uid + name + balance + } +} +``` + +Should be escaped to this: + +```sh +curl -H "Content-Type: application/json" localhost:8080/query -XPOST -d '{ + "query": "{\n users(func: anyofterms(name, \"Alice Bob\")) {\n uid\n name\n balance\n }\n }" +}' | python -m json.tool | jq +``` diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/clients/unofficial-clients.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/clients/unofficial-clients.md new file mode 100644 index 00000000..fa66b05b --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/clients/unofficial-clients.md @@ -0,0 +1,32 @@ +--- +title: Unofficial Dgraph Clients +--- + +:::note +These third-party clients are contributed by the community and are not officially supported by Dgraph. +::: + + +## Apache Spark Connector + +- https://github.com/G-Research/spark-dgraph-connector + +## Dart + +- https://github.com/marceloneppel/dgraph + +## Elixir + +- https://github.com/liveforeverx/dlex +- https://github.com/ospaarmann/exdgraph + +## Rust + +- https://github.com/Swoorup/dgraph-rs +- https://github.com/selmeci/dgraph-tonic + +## C# +- https://github.com/schivei/dgraph4net - DQL Client with migration management + + + diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/design-concepts/acl-concept.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/design-concepts/acl-concept.md new file mode 100644 index 00000000..f69dc3bc --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/design-concepts/acl-concept.md @@ -0,0 +1,11 @@ +--- +title: ACLs +--- + +ACLs are a typical mechanism to list who can access what, specifying either users or roles and what they can access. ACLs help determine who is "authorized" to access what. + +Dgraph Access Control Lists (ACLs) are sets of permissions for which `Relationships` a user may access. Recall that Dgraph is "predicate based" so all data is stored in and is implicit in relationships. This allows relationship-based controls to be very powerful in restricting a graph based on roles (RBAC). + +Note that the Dgraph multi-tenancy feature relies on ACLs to ensure each tenant can only see their own data in one server. + +Using ACLs requires a client to authenticate (log in) differently and specify credentials that will drive which relationships are visible in their view of the graph database. diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/design-concepts/badger-concept.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/design-concepts/badger-concept.md new file mode 100644 index 00000000..87d57a46 --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/design-concepts/badger-concept.md @@ -0,0 +1,10 @@ +--- +title: Badger +--- + + +[Badger](https://github.com/dgraph-io/badger) is a key-value store developed and maintained by Dgraph. It is also open source, and it is the backing store for Dgraph data. + +It is largely transparent to users that Dgraph uses Badger to store data internally. Badger is packaged into the Dgraph binary, and is the persistence layer. However, various configuration settings and log messages may reference Badger, such as cache sizes. + +Badger values are `Posting Lists` and indexes. Badger Keys are formed by concatenating <elationshipName>+<NodeUID>. diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/design-concepts/clients-concept.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/design-concepts/clients-concept.md new file mode 100644 index 00000000..1438de13 --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/design-concepts/clients-concept.md @@ -0,0 +1,13 @@ +--- +title: Dgraph Clients +--- + +A client is a program that calls dgraph. Broadly, there are stand alone clients such as Ratel, which is a graphical web-based application, and programmatic client libraries which are embedded in larger programs to efficiently and idomatically call Dgraph. + +GraphQL is an open standard with many clients (graphical and libraries) also, and GraphQL clients work with Dgraph. + +Dgraph provides [client libraries](../clients) for many languages. These clients send DQL queries, and perform useful functions such as logging in, in idomatic ways in each language. + +Note that Dgraph does not force or insist on any particular GraphQL client. Any GraphQL client, GUI, tool, or library will work well with Dgraph, and it is the users' choice which to choose. Dgraph only provides clients for the proprietary DQL query language. GraphQL clients are available for free from many organizations. + +However, Dgraph's cloud console does support basic GraphQL querying, so this is something of a tool. We recommend using a mature GraphQL console instead, as they are more mature. Dgraph's GraphQL GUI function is for quick start and convenience. diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/design-concepts/consistency-model.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/design-concepts/consistency-model.md new file mode 100644 index 00000000..db604b52 --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/design-concepts/consistency-model.md @@ -0,0 +1,39 @@ +--- +title: Consistency Model +--- + +### Dgraph supports MVCC, Read Snapshots and Distributed ACID transactions +Multi-version concurrency control (MVCC) is a technique where many versions of data are written (but never modified) on disk, so many versions exist. This helps control concurrency because the database is queried at a particular "timestamp" for the duration of one query to provide snapshot isolation and ensure data is consistent for that transaction. (Note that MVCC is losely related to LSM trees - in LSM parlance, data is "logged" to write-only files, which are later merged via Log Compaction.) + +Writes are faster with MVCC because data is always written by flushing a larger in-memory buffer (a memtable) to new, contiguous files (SST files), and newer data obscures or replaces older data. Consistent updates from each transaction share a logical commit timestamp (a 64 bit, increasing number loosely correlated to wall clock time), and all reads occur "at a point in time" meaning any read accesses a known, stable set of committed data using these same commit timestamps. New or in-process commits are associated with a later timestamp so they do not affect running queries at earlier timestamps. This allows pure queries (reads) to execute without any locks. + +One special set of structures are "memtables" which are also referred to as being Level 0 of the LSM tree. These are buffers for fast writes, which later are flushed to on-disk files called SSTs. + +### Dgraph transactions are cluster-wide (not key-only, or any other non-ACID version of transactions) +Dgraph uses the RAFT protocol to synchronize updates and ensure updates are durably written to a majority of alpha nodes in a cluster before the transaction is considered successful. RAFT ensures true, distributed, cluster wide transactions across multiple nodes, keys, edges, indexes and facets. Dgraph provides true ACID transactions, and does not impose limitations on what can be in a transaction: a transaction can involve multiple predicates, multiple nodes, multiple keys and even multiple shards. + +### Transactions are lockless +Dgraph transactoins do not use locks, allowing fast, distributed transactions. + +For reads, queries execute at a particular timestamp based on snapshot isolation, which isolates reads from any concurrent write activity. All reads access snapshots across the entire cluster, seeing all previously committed transactions in full, regardless of which alpha node received earlier queries. + +Writes use optimistic lock semantics, where a transaction will be aborted if another (concurrent) transaction updates exactly the same data (same edge on the same node) first. This will be reported as an "aborted" transaction to the caller. + +Dgraph ensures monotonically increasing transaction timestamps to sequence all updates in the database. This provides serializability: if any transaction Tx1 commits before Tx2 starts, then Ts_commit(Tx1) < Ts_start(Tx2), and in turn a read at any point in time can never see Tx1 changes but not Tx2 changes. + +Dgraph also ensures proper read-after-write semantics. Any commit at timestamp Tc is guaranteed to be seen by a read at timestamp Tr by any client, if Tr >= Tc. + +### Terminology + +- **Snapshot isolation:** all reads see a consistent view of the database at the point in time when the read was submitted +- **Oracle:** a logical process that tracks timestamps and which data (keys, predicates, etc.) has been committed or is being modified. The oracle hands out timestamps and aborts transactions if another transaction has modified its data. +- **RAFT:** a well-known consistency algorithm to ensure distributed processes durably store data +- **Write-Ahead Log:** Also WAL. A fast log of updates on each alpha that ensures buffered in-memory structures are persisted. +- **Proposal:** A process within the RAFT algorithm to track possible updates during the consensus process. +- **SST:** Persistent files comprising the LSM tree, together with memtables. +- **Memtable:** An in-memory version of an SST, supporting fast updates. Memtables are mutable, and SSTs are immutable. +- **Log Compaction:** The process of combining SSTs into newer SSTs while eliminating obsolte data and reclaiming disk space. +- **Timestamp:** Or point in time. A numeric counter representing the sequential order of all transactions, and indicating when a transaction became valid and query-able. +- **Optimistic Lock:** a logical process whereby all transactions execute without blocking on other transactions, and are aborted if there is a conflict. Aborted transactions should typically be retried if they occur. +- **Pessimistic Lock:** a process, not used in Dgraph, where all concurrent transactions mutating the same data except one block and wait for each other to complete. +- **ACID** An acronym representing attributes of true transactions: Atomic, Consistent, Isolated, and Durable diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/design-concepts/discovery-concept.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/design-concepts/discovery-concept.md new file mode 100644 index 00000000..fef545c9 --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/design-concepts/discovery-concept.md @@ -0,0 +1,7 @@ +--- +title: Discovery +--- + +### New Servers and Discovery +Dgraph clusters will detect new machines allocated to the [cluster](../installation/dgraph-architecture), +establish connections, and transfer data to the new server based on the group the new machine is in. diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/design-concepts/dql-concept.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/design-concepts/dql-concept.md new file mode 100644 index 00000000..2229d947 --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/design-concepts/dql-concept.md @@ -0,0 +1,5 @@ +--- +title: DQL +--- + +DQL is the "Dgraph Query Language" and is based on GraphQL. It is neither a superset nor subset of GraphQL, but is generally more powerful than GraphQL. DQL coexists nicely with GraphQL so many users perform most access using GraphQL and only "drop down" into DQL when there is a particular query mechanism needed that is not supported in the GraphQL spec. E.g. @recurse query operations are only in DQL. Other customers simply use DQL. DQL supports both queries and mutations, as well as hybrid "upsert" operations. diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/design-concepts/dql-graphql-layering-concept.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/design-concepts/dql-graphql-layering-concept.md new file mode 100644 index 00000000..343238a3 --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/design-concepts/dql-graphql-layering-concept.md @@ -0,0 +1,15 @@ +--- +title: DQL and GraphQL +--- + +## Dgraph Schemas +Dgraph natively supports GraphQL, including `GraphQL Schema`s. GraphQL schemas "sit on top of" DQL schemas, in the sense that when a GraphQL schema is added to Dgraph, a corresponding `DQL Schema` is automatically created. + +Refer to [GraphQL-DQL interoperability](/graphql/graphql-dql) section for details. + +## Dgraph Queries, Mutations and Upserts +Similarly, GraphQL mutations are implemented on top of DQL in the sense that a GraphQL query is converted internally into a DQL query, which is then executed. This translation is not particularly complex, since DQL is based on GraphQL, with some syntax changes and some extensions. + +This is generally transaparent to all callers, however users should be aware that +1) Anything done in GraphQL can also be done in DQL if needed. Some small exceptions include the enforcement of non-null constraints and other checks done before Dgraph transpiles GraphQL to DQL and executes it. +2) Some logging including Request Logging and OpenTrace (Jaeger) tracing may show DQL converted from the GraphQL. diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/design-concepts/facets-concept.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/design-concepts/facets-concept.md new file mode 100644 index 00000000..46432b57 --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/design-concepts/facets-concept.md @@ -0,0 +1,9 @@ +--- +title: Facets +--- + +Dgraph allows a set of properties to be associated with any `Relationship`. E.g. if there is a "worksFor" relationships between Node "Bob" and Node "Google", this relationship may have facet values of "since": 2002-05-05 and "position": "Engineer". + +Facets can always be replaced by adding a new Node representing the relationship and storing the facet data as attriubutes of the new Node. + +The term "facet" is also common in database and search engine technology, and indicates a dimension or classification of data. One way to use facets it to indicate a relationship type. diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/design-concepts/graphql-concept.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/design-concepts/graphql-concept.md new file mode 100644 index 00000000..225c2086 --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/design-concepts/graphql-concept.md @@ -0,0 +1,10 @@ + +--- +title: GraphQL +--- + + `GraphQL` is a query and update standard defined at [GraphQL.org](https://graphql.org/). `GraphQL` is natively supported by Dgraph, without requiring additional servers, data mappings or resolvers. Typically, "resolving" a data field in GraphQL simply corresponds to walking that relationship in Dgraph. + + Dgraph also auto-generates access functions for any `GraphQL Schema`, allowing users to get up and running in minutes with Dgraph + a GraphQL schema. The APIs are auto-generated. + +GraphQL is internally converted to the (similar-but-different) `DQL` query language before being executed. We can think of GraphQL as "sitting on top" of DQL. diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/design-concepts/group-concept.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/design-concepts/group-concept.md new file mode 100644 index 00000000..e83e3b14 --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/design-concepts/group-concept.md @@ -0,0 +1,22 @@ +--- +title: Group +--- + +A group is a set of 1 or 3 or more servers that work together and have a single `leader` in the sense defined by the RAFT protocol. +## Alpha Group +An Alpha `Group` in Dgraph is a shard of data, and may or may not be highly-available (HA). An HA group typically has three Dgraph instances (servers or K8s pods), and a non-HA group is a single instance. Every Alpha instance belongs to one group, and each group is responsible for serving a +particular set of tablets (relations). In an HA configuration, the three or more instances in a single group replicate the same data to every instance to ensure redundancy of data. + +In a sharded Dgraph cluster, tablets are automatically assigned to each group, and dynamically relocated as sizes change to keep the groups balanced. Predicates can also be moved manually if desired. + +In a future version, if a tablet gets too big, it will be split among two groups, but currently data is balanced by moving each tablet to one group only. + +To avoid confusion, remember that you may have many Dgraph alpha instances due to either sharding, or due to HA configuration. If you have both sharding and HA, you will have 3*N groups: + + config | Non-HA | HA +-------------|-------------------|-------- +Non-sharded | 1 alpha total | 3 alphas total +Sharded | 1 alpha per group | 3*N alphas for N groups + +## Zero Group +Group Zero is a lightweight server or group of servers which helps control the overall cluster. It manages timestamps and UIDs, determines when data should be rebalanced among shards, and other functions. The servers in this group are generally called "Zeros." diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/design-concepts/index-tokenize-concept.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/design-concepts/index-tokenize-concept.md new file mode 100644 index 00000000..0bf2e9c8 --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/design-concepts/index-tokenize-concept.md @@ -0,0 +1,13 @@ +--- +title: Index and Tokenizer +--- +### Indexing +An index is an optimized data structure, stored on disk and loaded into memory, that speeds or optimizes query processing. It is created and stored in addition to the primary data. E.g. a "hasName" property or relation is the primary storage structure for a graph in Dgraph, but may also have an additional index structure configured. + +Typically, Dgraph query access is optimized for forward access. When other access is needed, an index may speed up queries. Indexes are large structures that hold all values for some Relation (vs `Posting Lists`, which are typically smaller, per-Node structures). + +### Tokenizers + +Tokenizers are simply small algorithms that create indexed values from some Node property. E.g. if a Book Node has a Title attribute, and you add a "term" index, each word (term) in the text will be indexed. The word "Tokenizer" derives its name from tokenizing operations to create this index type. + +Similary if the Book has a publicationDateTime you can add a day or year index. The "tokenizer" here extracts the value to be indexed, which may be the day or hour of the dateTime, or only the year. diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/design-concepts/index.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/design-concepts/index.md new file mode 100644 index 00000000..4c39b468 --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/design-concepts/index.md @@ -0,0 +1,10 @@ +--- +title: Design Concepts +--- + + + +This section of the documentation covers various concepts that are relevant to the Dgraph system. + + +### In this section \ No newline at end of file diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/design-concepts/lambda-concept.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/design-concepts/lambda-concept.md new file mode 100644 index 00000000..4f86473b --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/design-concepts/lambda-concept.md @@ -0,0 +1,5 @@ +--- +title: Lambdas +--- + +Dgraph Lambdas are JavaScript functions that can be used during query or mutation processing to extend GraphQL or DQL queries and mutations. Lambdas are not related at all to AWS Lambdas. They are functions that run in an (optional) node.js server that is included in the Dgraph Cloud offering. diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/design-concepts/minimizing-network-calls.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/design-concepts/minimizing-network-calls.md new file mode 100644 index 00000000..efcdb7ae --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/design-concepts/minimizing-network-calls.md @@ -0,0 +1,73 @@ +--- +title: Minimal Network Calls +--- + +### Predicate-based storage and sharding + +Dgraph is unique in its use of predicate-based sharding, which allows complex and deep distributed queries to run without incurring high network overhead and associated delays. + +Rather than store and shard by putting different _nodes_ (aka entities*) on different servers, Dgraph stores predicates or triples of the form <node1> <predicateRelation> <node2>. The nodes are therefore implicit in the predicate storage, rather than vice versa. + +This makes querying much different and particularly allows network optimizations in a distributed database. + +### Example +To explain how this works, let's use an example query: + + `Find all posts liked by friends of friends of mine over the last year, written by a popular author A.` + +### SQL/NoSQL +In a distributed SQL database or (non-graph) NoSQL database, this query requires retrieval of a lot of data. Consider two approaches: + +Approach 1: + +* Find all the friends (~ 338 [friends](https://www.pewresearch.org/fact-tank/2014/02/03/what-people-like-dislike-about-facebook/)). +* Find all their friends (~ 338 * 338 = 40,000 people). +* Find all the posts liked by these people over the last year (resulting set in the millions). +* Intersect these posts with posts authored by person A. + +Approach 2: + +* Find all posts written by popular author A over the last year (possibly thousands). +* Find all people who liked those posts (easily millions) (call this `result set 1`). +* Find all your friends. +* Find all their friends (call this `result set 2`). +* Intersect `result set 1` with `result set 2`. + +Both approaches wouild result in a lot of data moving back and forth between database and +application; would be slow to execute, and may require running an offline job. + +### Dgraph Approach +This is how it would run in Dgraph: + +Sharding assumptions (which predicates live where): +* Assume Server X contains the predicate `friends` representing all friend relations. +* Assume Server Y contains the predicate `posts_liked` representing who likes each post. +* Assume Server Z contains the predicate `author` representing all who authored each post. +* Assume Server W contains the predicate `title` representing the uid->string title property of posts. + +Algorithm: +* Server X + * If the request was not sent to Server X, route it to Server X where the friends predicate lives. **(1 RPC)**. + * Seek to my uid within predicate (tablet) `friends` and retrieve a list of my friends as a list of uids. + * Still on Server X, use the friends predicate again to get friends for all of those uids, generating a list of my friends of friends. Call this `result set myFOF`. +* Server Y + * Send result set myFOF to Server Y, which holds the posts_liked predicate **(1 RPC)**. + * Retrieve all posts liked by my friends-of-friends. Call this `result set postsMyFOFLiked`. +* Server Z + * Send postsMyFOFLiked result set to Server Z **(1 RPC)**. + * Retrieve all posts authored by A. Call this `result set authoredByA`. + * Still on Server Z, intersect the two sorted lists to get posts that are both liked and authored by A: `result set postsMyFOFLiked` intersect `result set authoredByA`. Call this `result set postsMyFOFLikedByA` + * at this point we have done the hard work, but have the uids of the posts, instead of the post titles. +* Server W + * Send `result set postsMyFOFLikedByA` to Server W which holds the title predicate **(1 RPC)**. + * Convert uids to names by looking up the title for each uid. `result set postUidsAndTitles` +* Respond to caller with `result set postUidsAndTitles`. + +## Net Result - predictable distributed graph scaling +In at most 4 RPCs, we have figured out all the posts liked by friends of friends, written by popular author X, with titles. Typically, all four predicates will not live on four different Servers, so this is a worst-case scenario. Dgraph network activity is limited to the level of query join depth, rather than increasing arbitrarily according to the number of nodes in the graph, and how they are broken up across servers. + +There is no way we are aware of that a node-based sharding database can avoid high network RPC counts during arbitrary queries because "node-hopping" does not mix well with a graph that is segmented across servers. + + +---- +* _Throughout this note, we call entities in a graph "nodes" which is a standard terminology when talking about nodes and predicates. These may be confused with RAFT or Kubernetes nodes in some contexts, but generally we mean nodes in a graph_. diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/design-concepts/namespace-tenant-concept.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/design-concepts/namespace-tenant-concept.md new file mode 100644 index 00000000..224dbb0b --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/design-concepts/namespace-tenant-concept.md @@ -0,0 +1,7 @@ +--- +title: Namespace and Tenant +--- + +A Dgraph `Namespace` (aka Tenant) is a logically separate database within a Dgraph cluster. A Dgraph cluster can host many Namespaces (and this is how the Dgraph "shared" cloud offering works). Each user must then into their own namespace using namespace-specific own credentials, and sees only their own data. Note that this usually requires an extra or specific login. + +There is no mechanism to query in a way that combines data from two namespaces, which simplifies and enforces security in use cases where this is the requirement. An API layer or client would have to pull data from multiple namespaces using different authenticated queries if data needed to be combined. diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/design-concepts/network-call-minimization-concept.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/design-concepts/network-call-minimization-concept.md new file mode 100644 index 00000000..1b0e2341 --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/design-concepts/network-call-minimization-concept.md @@ -0,0 +1,7 @@ +--- +title: Network Call Minimization +--- + +Compared to RAM or SSD access, network calls are slow, so Dgraph is built from the ground up to minimize them. For graph databases which store sub-graphs on different shards, this is difficult or impossible, but predicate-based (relationship-based) sharding allows fast distributed query with Dgraph. + +See [How Dgraph Minmizes Network Calls](minimizing-network-calls) for more details. diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/design-concepts/posting-list-concept.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/design-concepts/posting-list-concept.md new file mode 100644 index 00000000..00da61b9 --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/design-concepts/posting-list-concept.md @@ -0,0 +1,99 @@ +--- +title: Posting List and Tablet +--- + +Posting lists and tablets are internal storage mechanisms and are generally hidden from users or developers, but logs, core product code, blog posts and discussions about Dgraph may use the terms "posting list" and "tablet." + +Posting lists are a form of inverted index. Posting lists correspond closely to the RDF concept of a graph, where the entire graph is a collection of triples, `` ``. In this view, a posting list is a list of all triples that share a ``+`` pair. + +(Note that in Dgraph docs, we typically use the term "relationship" rather than predicate, but here we will refer to predicates explicitly.) + +The posting lists are grouped by predicate into `tablets`. A tablet therefore has all data for a predicate, for all subject UIDs. + +Tablets are the basis for data shards in Dgraph. In the near future, Dgraph may split a single tablet into two shards, but currently every data shard is a single predicate. Every server then hosts and stores a set of tablets. Dgraph will move or allocate different tablets to different servers to achieve balance across a sharded cluster. + + +### Example +If we're storing friendship relationships among four people, we may have four posting lists represented by the four tables below: + +Node | Attribute| Value +------- |----------|-------- +person1 | friend | person2 +person1 | friend | person4 + +  + +Node | Attribute| Value +------- |----------|-------- +person2 | friend | person1 + +  + +Node | Attribute| Value +------- |----------|-------- +person3 | friend | person2 +person3 | friend | person4 + +  + +Node | Attribute| Value +------- |----------|-------- +person4 | friend | person2 +person4 | friend | person1 +person4 | friend | person3 + +  + +The corrsponding posting lists would be something like: + +``` +person1UID+friend->[person2UID, person4UID] +person2UID+friend->[person1UID] +person3UID+friend->[person2UID, person4UID] +person4UID+friend->[person1UID, person2UID, person3UID] +``` +  + +Similarly, a posting list will also hold all literal value properties for every node. E.g. consider the names of people in these three tables: + +Node | Attribute| Value +------- |----------|-------- +person1 | name | "James" +person1 | name | "Jimmy" +person1 | name | "Jim" + +  + +Node | Attribute| Value +------- |----------|-------- +person2 | name | "Rajiv" + +  + +Node | Attribute| Value +------- |----------|-------- +person3 | name | "Rachel" + +  +The posting lists would look like: +``` +person1UID+name->["James", "Jimmy", "Jim"] +person2UID+friend->["Rajiv"] +person3UID+friend->["Rachel"] +``` +  + +Note that person4 has no name attribute specified, so that posting list would not exist. + +In these examples, two predicates (relations) are defined, and therefore two tablets will exist. + +The tablet for the `friend` predicate will hold all posting lists for all "friend" relationships in the entire graph. The tablet for the `name` property will hold all posting lists for `name` in the graph. + +If other types such as Pets or Cities also have a name property, their data will be in the same tablet as the Person names. + +### Performance implications + +A key advantage of grouping data into predicate-based shards is that we have all the data to do one join in one `tablet` on one server/shard. This means, one RPC to +the machine serving that `tablet` will be adequate, as documented in [How Dgraph Minmizes Network Calls](minimizing-network-calls). + +Posting lists are the unit of data access and caching in Dgraph. The underlying key-value store stores and retrieves posting lists as a unit. Queries that access larger posting lists will use more cache and may incur more disk access for un-cached posting lists. diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/design-concepts/protocol-buffers-concept.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/design-concepts/protocol-buffers-concept.md new file mode 100644 index 00000000..4c0dc048 --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/design-concepts/protocol-buffers-concept.md @@ -0,0 +1,5 @@ +--- +title: Protocol Buffers +--- + +All data in Dgraph that is stored or transmitted among the Dgraph instances (servers) is converted into space-optimized byte arrays using [Protocol Buffers](https://developers.google.com/protocol-buffers/). Protocol Buffers are a standard, optimized technology to speed up network communications. diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/design-concepts/queries-process.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/design-concepts/queries-process.md new file mode 100644 index 00000000..951d2f91 --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/design-concepts/queries-process.md @@ -0,0 +1,42 @@ +--- +title: Query Process +--- + + +To understand how query execution works, look at an example. + +``` +{ + me(func: uid(0x1)) { + rel_A + rel_B { + rel_B1 + rel_B2 + } + rel_C { + rel_C1 + rel_C2 { + rel_C2_1 + } + } + } +} + +``` + +Let's assume we have 3 Alpha instances, and instance id=2 receives this query. These are the steps: + +* This query specifies the exact UID list (one UID) to start with, so there is no root query clause. +* Retreive posting lists using keys = `0x1::rel_A`, `0x1::rel_B`, and `0x1::rel_C`. + * At worst, these predicates could belong to 3 different groups if the DB is sharded, so this would incur at most 3 network calls. +* The above posting lists would include three lists of UIDs or values. + * The UID results (id1, id2, ..., idn) for `rel_B` are converted into queries for `id1::rel_B1` `id2::rel_B1`, etc., and for `id1::rel_B2` `id2::rel_B2`, etc. + * Similarly, results for rel_C will be used to get the next set of UIDs from posting list keys like `id::rel_C1` and `id::rel_C2`. +* This process continues recursively for `rel_C2_1` as well, and as deep as any query requires. + +More complex queries may do filtering operations, or intersections and unions of UIDs, but this recursive walk to execute a number of (often parallel) `Tasks` to retrieve UIDs characterizes Dgraph querying. + +If the query was run via HTTP interface `/query`, the resulting subgraph then gets converted into JSON for +replying back to the client. If the query was run via [gRPC](https://www.grpc.io/) interface using +the language [clients](../clients), the subgraph gets converted to +[protocol buffer](https://developers.google.com/protocol-buffers/) format and similarly returned to the client. diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/design-concepts/raft.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/design-concepts/raft.md new file mode 100644 index 00000000..9428ed31 --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/design-concepts/raft.md @@ -0,0 +1,159 @@ +--- +title: RAFT +--- + +Dgraph uses RAFT whenever consensus among a distribued set of servers is required, such as ensuring that a transaction has been properly committed, or determining the proper timestamp for a read or write. Each zero or alpha `group` uses raft to elect leaders. + +This section aims to explain the RAFT consensus algorithm in simple terms. The idea is to give you +just enough to make you understand the basic concepts, without going into explanations about why it +works accurately. For a detailed explanation of RAFT, please read the original thesis paper by +[Diego Ongaro](https://github.com/ongardie/dissertation). + +## Term +Each election cycle is considered a **term**, during which there is a single leader +*(just like in a democracy)*. When a new election starts, the term number is increased. This is +straightforward and obvious but is a critical factor for the accuracy of the algorithm. + +In rare cases, if no leader could be elected within an `ElectionTimeout`, that term can end without +a leader. + +## Server States +Each server in cluster can be in one of the following three states: + +* Leader +* Follower +* Candidate + +Generally, the servers are in leader or follower state. When the leader crashes or the communication +breaks down, the followers will wait for election timeout before converting to candidates. The +election timeout is randomized. This would allow one of them to declare candidacy before others. +The candidate would vote for itself and wait for the majority of the cluster to vote for it as well. +If a follower hears from a candidate with a higher term than the current (*dead in this case*) leader, +it would vote for it. The candidate who gets majority votes wins the election and becomes the leader. + +The leader then tells the rest of the cluster about the result (Heartbeat +[Communication](#communication)) and the other candidates then become followers. +Again, the cluster goes back into leader-follower model. + +A leader could revert to being a follower without an election, if it finds another leader in the +cluster with a higher [Term](#term)). This might happen in rare cases (network partitions). + +## Communication +There is unidirectional RPC communication, from the leader to all/any followers. The followers never ping the +leader. The leader sends `AppendEntries` messages to the followers with logs containing state +updates. When the leader sends `AppendEntries` with zero logs (updates), that's considered a +Heartbeat. The leader sends all followers Heartbeats at regular intervals. + +If a follower doesn't receive a Heartbeat for `ElectionTimeout` duration (generally between +150ms to 300ms), the leader may be down, so it converts it's state to candidate (as mentioned in [Server States](#server-states)). +It then requests for votes by sending a `RequestVote` call to other servers. If it gets votes from the majority, the candidate becomes the leader. On becoming leader, it sends Heartbeats +to all other servers to establish its authority. + +Every communication request contains a term number. If a server receives a request with a stale term +number, it rejects the request. + +## Log Entries +Dgraph uses LSM Trees, so we call commits or updates "Log Entries." Log Entries are numbered sequentially and contain a term number. An Entry is considered **committed** if it has been replicated (and stored) by a majority of the servers. + +On being notified of the results of a client request (which is often processed on other servers), the leader does four things to coordinate RAFT consensus (this is also called Log Replication): + +* Appends and persists to its log. +* Issue `AppendEntries` in parallel to other servers. +* Monitors for the majority to report it is replicated, after which it considers the entry committed and applies it to the leader's state machine. +* Notifies followers that the entry is committed so that they can apply it to their state machines. + +A leader never overwrites or deletes its entries. RAFT guarantees that if an entry is committed, +all future leaders will have it. A leader can, however, force overwrite the followers' logs, so they +match leader's logs if necessary. + +## Voting +Each server persists its current term and vote, so it doesn't end up voting twice in the same term. +On receiving a `RequestVote` RPC, the server denies its vote if its log is more up-to-date than the +candidate. It would also deny a vote, if a minimum `ElectionTimeout` hasn't passed since the last +Heartbeat from the leader. Otherwise, it gives a vote and resets its `ElectionTimeout` timer. + +Up-to-date property of logs is determined as follows: + +* Term number comparison +* Index number or log length comparison + +:::tipTo understand the above sections better, you can see this +[interactive visualization](http://thesecretlivesofdata.com/raft).::: + +## Cluster membership +Raft only allows single-server changes, i.e. only one server can be added or deleted at a time. +This is achieved by cluster configuration changes. Cluster configurations are communicated using +special entries in `AppendEntries`. + +The significant difference in how cluster configuration changes are applied compared to how typical +[Log Entries](#log-entries) are applied is that the followers don't wait for a +commitment confirmation from the leader before enabling it. + +A server can respond to both `AppendEntries` and `RequestVote`, without checking current +configuration. This mechanism allows new servers to participate without officially being part of +the cluster. Without this feature, things won't work. + +When a new server joins, it won't have any logs, and they need to be streamed. To ensure cluster +availability, Raft allows this server to join the cluster as a non-voting member. Once it's caught +up, voting can be enabled. This also allows the cluster to remove this server in case it's too slow +to catch up, before giving voting rights *(sort of like getting a green card to allow assimilation +before citizenship is awarded providing voting rights)*. + + +:::tipIf you want to add a few servers and remove a few servers, do the addition +before the removal. To bootstrap a cluster, start with one server to allow it to become the leader, +and then add servers to the cluster one-by-one.::: + +## Snapshots +One of the ways to do this is snapshotting. As soon as the state machine is synced to disk, the +logs can be discarded. + +Snapshots are taken by default after 10000 Raft entries, with a frequency of 30 minutes. The frequency indicates the time between two subsequent snapshots. These numbers can be adjusted using the `--raft` [superflag](../cli/superflags)'s `snapshot-after-entries` and `snapshot-after-duration` options respectively. Snapshots are created only when conditions set by both of these options have been met. + +## Clients +Clients must locate the cluster to interact with it. Various approaches can be used for discovery. + +A client can randomly pick up any server in the cluster. If the server isn't a leader, the request +should be rejected, and the leader information passed along. The client can then re-route it's query +to the leader. Alternatively, the server can proxy the client's request to the leader. + +When a client first starts up, it can register itself with the cluster using `RegisterClient` RPC. +This creates a new client id, which is used for all subsequent RPCs. + +## Linearizable Semantics + +Servers must filter out duplicate requests. They can do this via session tracking where they use +the client id and another request UID set by the client to avoid reprocessing duplicate requests. +RAFT also suggests storing responses along with the request UIDs to reply back in case it receives +a duplicate request. + +Linearizability requires the results of a read to reflect the latest committed write. +Serializability, on the other hand, allows stale reads. + +## Read-only queries + +To ensure linearizability of read-only queries run via leader, leader must take these steps: + +* Leader must have at least one committed entry in its term. This would allow for up-to-dated-ness. +*(C'mon! Now that you're in power do something at least!)* +* Leader stores it's latest commit index. +* Leader sends Heartbeats to the cluster and waits for ACK from majority. Now it knows +that it's the leader. *(No successful coup. Yup, still the democratically elected dictator I was before!)* +* Leader waits for its state machine to advance to readIndex. +* Leader can now run the queries against state machine and reply to clients. + +Read-only queries can also be serviced by followers to reduce the load on the leader. But this +could lead to stale results unless the follower confirms that its leader is the real leader(network partition). +To do so, it would have to send a query to the leader, and the leader would have to do steps 1-3. +Then the follower can do 4-5. + +Read-only queries would have to be batched up, and then RPCs would have to go to the leader for each +batch, who in turn would have to send further RPCs to the whole cluster. *(This is not scalable +without considerable optimizations to deal with latency.)* + +**An alternative approach** would be to have the servers return the index corresponding to their +state machine. The client can then keep track of the maximum index it has received from replies so far. +And pass it along to the server for the next request. If a server's state machine hasn't reached the +index provided by the client, it will not service the request. This approach avoids inter-server +communication and is a lot more scalable. *(This approach does not guarantee linearizability, but +should converge quickly to the latest write.)* diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/design-concepts/relationships-concept.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/design-concepts/relationships-concept.md new file mode 100644 index 00000000..78b5f3d6 --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/design-concepts/relationships-concept.md @@ -0,0 +1,24 @@ + +--- +title: Relationships +--- + + +Dgraph stores `relationships` among `nodes` to represent graph structures, and also stores literal properties of `nodes`. + +This makes it easy for Dgraph to ingest the RDF [N-Quad](https://www.w3.org/TR/n-quads/) format, where each line represents + +* `Node, RelationName, Node, Label` or +* `Node, RelationName, ValueLiteral, Label` + +The first represents relations among entities (nodes in graph terminology) and the second represents the relationship of a Node to all it's named attributes. + +Often, the optional `Label` is omitted, and therefore the N-Quad data is also referred to as "triples." When it is included, it represents which `Tenant` or `Namespace` the data lives in within Dgraph. + +:::tipDgraph can automatically generate a reverse relation. If the user wants to run +queries in that direction, they would need to define the [reverse relationship](../dql/dql-schema#reverse-predicates) +::: + +For `Relationships`, the subject and object are represented as 64-bit numeric UIDs and the relationship name itself links them: <subjectUID> <relationshipName> <bjectUID>. + +For literal attributes of a `Node`, the subject must still (and always) be a numeric UID, but the Object will be a primitive value. These can be thought of as <subjectUID> <elationshipName> <value>, where value is not a 64-bit UID, and is instead a: string, float, int, dateTime, geopoint, or boolean. diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/design-concepts/replication-concept.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/design-concepts/replication-concept.md new file mode 100644 index 00000000..915fce4a --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/design-concepts/replication-concept.md @@ -0,0 +1,8 @@ +--- +title: High Availability Replication +--- + +Each Highly-Available (HA) group will be served by at least 3 instances (or two if one is temporarily unavailable). In the case of an alpha instance +failure, other alpha instances in the same group still handle the load for data in that group. In case of a zero instance failure, the remaining two zeros in the zero group will continue to hand out timestamps and perform other zero functions. + +In addition, Dgraph `Learner Nodes` are alpha instances that hold replicas of data, but this replication is to support read replicas, often in a different geography from the master cluster. This replication is implemented the same way as HA replication, but the learner nodes do not participate in quorum, and do not take over from failed nodes to provide high availability. diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/design-concepts/transaction-mutation-concept.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/design-concepts/transaction-mutation-concept.md new file mode 100644 index 00000000..8ca782cb --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/design-concepts/transaction-mutation-concept.md @@ -0,0 +1,8 @@ +--- +title: Transaction and Mutation +--- + + +Borrowing from GraphQL, Dgraph calls writes to the database `Mutations`. As noted elsewhere (MVCC, LSM Trees and Write Ahead Log sections) writes are written persistently to the Write Ahead Log, and ephemerally to a memtable. + +Data is queried from the combination of persistent SST files and ephemeral memtable data structures. The mutations therefore always go into the memtables first (though are also written durably to the WAL). The memtables are the "Level 0" in the LSM Tree, and conceptually sit on top of the immutable SST files. diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/design-concepts/transactions-concept.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/design-concepts/transactions-concept.md new file mode 100644 index 00000000..7d1e2e0b --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/design-concepts/transactions-concept.md @@ -0,0 +1,19 @@ +--- +title: ACID Transactions +--- + +ACID is an acronym for +* Atomic +* Consistent +* Isolated +* Durable + +If these properties are maintained, there is a guarantee that data updates will not be lost, corrupted or unpredictable. Broadly, an ACID database safely and reliably stores data, but other databases have failure modes where data can be lost or corrupted. + +### ACID in Dgraph +Dgraph supports distributed ACID transactions through snapshot isolation and the RAFT consensus protocol. Dgraph is fully transactional, and is tested via Jepsen tests, which is a gold standard to verify transactional consistency. + +Dgraph ensure snapshot isolation plus realtime safety: if transaction T1 commits before T2 begins, than the commit timestamp of T1 is strictly less than the start timestamp of T2. This ensures that the sequence of writes on shared data by many processes is reflected in database state. + +Snapshot isolation is ensured by maintaining a consistent view of the database at any (relatively recent) point in time. Every read (query) takes place at the point-in-time it was submitted, accesses a consistent snapshot that does not change or include any partial updates due to concurrent writes that are processing or committing. + diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/design-concepts/wal-memtable-concept.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/design-concepts/wal-memtable-concept.md new file mode 100644 index 00000000..cb3ea9a2 --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/design-concepts/wal-memtable-concept.md @@ -0,0 +1,8 @@ +--- +title: WAL and Memtable +--- + + +Per the RAFT (and MVCC) approach, transactions write data to a `Write-Ahead Log` (WAL) to ensure it is durably stored. Soon after commit, data is also updated in the `memtables` which are memory buffers holding recently-updated data. The `memtables` are mutable, unlike the SST files written to disk which hold most data. Once full, memtables are flushed to disk and become SST files. See Log Compaction for more details on this process. + +In the event of a system crash, the persistent data in the Write Ahead Logs is replayed to rebuild the memtables and restore the full system state from before the crash. diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/design-concepts/workers-concept.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/design-concepts/workers-concept.md new file mode 100644 index 00000000..b84c673f --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/design-concepts/workers-concept.md @@ -0,0 +1,6 @@ +--- +title: Workers +--- + +### Workers and Worker Pools +Dgraph maintains a fixed set of worker processes (much like threads or goroutines) that retrieve and execute queries in parallel as they are sent over HTTP or gRPC. Dgraph also parallelizes Tasks within a single query execution, to maximize parallelism and more fully utilize system resources. Dgraph is written in the go language, which supports high numbers of parallel goroutines, enabling this approach without creating large numbers of OS threads which would be slower. diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/dgraph-glossary.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/dgraph-glossary.md new file mode 100644 index 00000000..4a6d26ad --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/dgraph-glossary.md @@ -0,0 +1,80 @@ +--- +title: Dgraph Glossary +description: Dgraph terms +--- + +:::note +*This is a glossary of Dgraph terms* + +### Alpha ### +A Dgraph cluster consists of [Zero](#zero) and Alpha nodes. Alpha nodes host relationships (also known as predicates) and indexes. Dgraph scales horizontally by adding more Alphas. + +### Badger ### +Badger is a fast, open-source key-value database written in pure Go that provides the storage layer for Dgraph. +More at [Badger documentation](https://dgraph.io/docs/badger) + +### DQL ### +Dgraph Query Language is Dgraph's proprietary language to insert, update, delete and query data. It is based on GraphQL, but is more expressive. (See also: [GraphQL](#graphql)) + +### Edge ### +In the mental picture of a graph: bubbles connected by lines ; the bubbles are nodes, the lines are edges. +In Dgraph terminology edges are [relationships](#relationship) i.e an information about the relation between two nodes. + +### Facet ### +A facet represents a property of a [relationship](#relationship). + +### Graph ### +A graph is a simple structure that maps relations between objects. In Dgraph terminology, the objects are [nodes](#node) and the connections between them are [relationships](#relationship). + +### GraphQL ### +[GraphQL](https://graphql.org/) is a declarative language for querying data used by application developers to get the data they need using GraphQL APIs. GraphQL is an open standard with a robust ecosystem. Dgraph supports the deployment of a GraphQL data model (GraphQL schema) and automatically exposes a GraphQL API endpoint accepting GraphQL queries. + +### gRPC ### +[gRPC](https://grpc.io/) is a high performance Remote Procedure Call (RPC) framework used by Dgraph to interface with clients. Dgraph has official gRPC clients for go, C#, Java, JavaScript and Python. Applications written in those language can perform mutations and queries inside transactions using Dgraph clients. + +### Lambda ### +A Lambda Resolver (Lambda for short) is a GraphQL resolver supported within Dgraph. A Lambda is a user-defined JavaScript function that performs custom actions over the GraphQL types, interfaces, queries, and mutations. Dgraph Lambdas are unrelated to AWS Lambdas. + +### Mutation ### +A mutation is a request to modify the database. Mutations include insert, update, or delete operations. A Mutation can be combined with a query to form an [Upsert](#upsert). + +### Node ### +Conceptually, a node is "a thing" or an object of the business domain. For every node, Dgraph stores and maintains a universal identifier [UID](#uid), a list of properties, and the [relationships](#relationship) the node has with other nodes. + +The term "node" is also used in software architecture to reference a physical computer or a virtual machine running a module of Dgraph in a cluster. See [Aplha node](#alpha) and [Zero node](#zero). + +### Predicate ### +In [RDF](#rdf) terminology, a predicate is the smallest piece of information about an object. A predicate can hold a literal value or can describe a relation to another entity : +- when we store that an entity name is "Alice". The predicate is ``name`` and predicate value is the string "Alice". It becomes a node property. +- when we store that Alice knows Bob, we may use a predicate ``knows`` with the node representing Alice. The value of this predicate would be the [uid](#uid) of the node representing Bob. In that case, ``knows`` is a [relationship](#relationship). + +### RATEL ### +Ratel is an open source GUI tool for data visualization and cluster management that’s designed to work with Dgraph and DQL. See also: [Ratel Overview](/ratel). + +### RDF ### +RDF 1.1 is a Semantic Web Standard for data interchange. It allows us to make statements about resources. The format of these statements is simple and in the form of ` `. +Dgraph supports the RDF format to create, import and export data. Note that Dgraph also supports the JSON format. + + +### Relationship ### +A relationship is a named, directed link relating one [node](#node) to another. It is the Dgraph term similar to [edge](#edge) and [predicate](#predicate). In Dgraph a relationship may itself have properties representing information about the relation, such as weight, cost, timeframe, or type. In Dgraph the properties of a relationship are called [facets](#facet). + +### Sharding ### +Sharding is a database architecture pattern to achieve horizontal scale by distributing data among many servers. Dgraph shards data per relationship, so all data for one relationship form a single shard, and are stored on one (group of) servers, an approach referred to as 'predicate-based sharding'. + +### Triple ### +Because RDF statements consist of three elements: ` `, they are called triples. A triple represents a single atomic statement about a node. The object in an RDF triple can be a literal value or can point to another node. See [DQL RDF Syntax](dql/dql-rdf) for more details. +- when we store that a node name is "Alice". The predicate is ``name`` and predicate value is the string "Alice". The string becomes a node property. +- when we store that Alice knows Bob, we may use a predicate ``knows`` with the node representing Alice. The value of this predicate would be the [uid](#uid) of the node representing Bob. In that case, ``knows`` is a [relationship](#relationship). + + +### UID ### +A UID is the Universal Identifier of a node. `uid` is a reserved property holding the UID value for every node. UIDs can either be generated by Dgraph when creating nodes, or can be set explicitly. + + +### Upsert ### +An upsert operation combines a Query with a [Mutation](#mutation). Typically, a node is searched for, and then depending on if it is found or not, a new node is created with associated predicates or the exixting node relationships are updated. Upsert operations are important to implement uniqueness of predicates. + +### Zero ### +Dgraph consists of Zero and [Alpha](#alpha) nodes. Zero nodes control the Dgraph database cluster. It assigns Alpha nodes to groups, re-balances data between groups, handles transaction timestamp and UID assignment. +::: diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/dgraph-overview.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/dgraph-overview.md new file mode 100644 index 00000000..4dc4fa16 --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/dgraph-overview.md @@ -0,0 +1,119 @@ +--- +title: Overview +description: Introduction to Dgraph Database. Dgraph is a horizontally scalable and distributed graph database that supports GraphQL natively. You can run Dgraph on-premise, in your cloud infrastructure, or as a service fully-managed by Dgraph. +--- + +Dgraph is a distributed graph database designed for modern applications that need to work with highly connected data. It provides a scalable foundation for storing and querying complex relationships between entities. + +## Understanding the Graph Model + +At its core, Dgraph stores data as a graph composed of **nodes** and **relationships**. Nodes represent entities in your data (like users, products, or locations), while relationships connect these nodes to show how they relate to each other (like "follows", "purchased", or "located_in"). + +Each node is identified by a unique identifier (UID) and can have multiple **attributes** that describe its properties. For example, a person node might have attributes like name, age, and email. Attributes can store various data types including strings, integers, floats, dates, and geographic coordinates. + +## Data Formats + +Dgraph is flexible in how you provide data to it. You can save data in two formats: + +**RDF (Resource Description Format)** uses a triple-based structure with subject-predicate-object statements: +``` +<0x1> "Alice" . +<0x1> "30" . +<0x1> <0x2> . +``` + +**JSON** provides a more familiar structure for developers: +```json +{ + "uid": "0x1", + "name": "Alice", + "age": 30, + "friend": { + "uid": "0x2" + } +} +``` + +Both formats are stored internally as graph structures, allowing you to choose the format that best fits your workflow. + +## Schema and Types + +While Dgraph can operate in a schema-less manner (you can add any predicate to any node at any time), defining a schema provides important benefits. The schema tells Dgraph about your predicates—their data types and which indexes to use. +Indexes are required to use certain query functions. + +## Distributed Architecture + +Dgraph is built from the ground up as a distributed system. Data is automatically sharded across multiple nodes in a cluster, allowing you to scale horizontally as your graph grows. The distributed architecture enables Dgraph to handle graphs with billions of nodes and triples while maintaining low-latency query performance. + +Each Dgraph cluster consists of multiple server groups (shards) that work together to store and query your data. Queries are automatically distributed across the relevant shards and results are aggregated, making the distributed nature transparent to your application. This architecture provides both horizontal scalability and high availability. + +For detailed information about Dgraph's distributed architecture, clustering, and replication, see the [Architecture documentation](/installation/dgraph-architecture). + +## Enterprise-Grade Features + +Dgraph includes production-ready features for running mission-critical applications: + +**High Availability**: Configure multiple replicas within each server group to ensure your database remains available even when individual nodes fail. Automatic failover maintains service continuity without manual intervention. + +**Backup and Restore**: Create full and incremental backups of your graph data. Backups can be stored locally or in cloud storage, and point-in-time recovery allows you to restore your database to any previous state. + +**Monitoring and Observability**: Built-in metrics and integration with monitoring tools like Prometheus and Grafana provide visibility into cluster health, query performance, and resource utilization. + +**Access Control**: Fine-grained access control lists (ACLs) allow you to manage user permissions at the predicate level, ensuring data security in multi-tenant environments. + +**Encryption**: Support for encryption at rest and in transit protects your data throughout its lifecycle. + +These features make Dgraph suitable for production deployments requiring reliability, security, and operational excellence. + +## Querying Dgraph Query Language (DQL) + +Dgraph uses **DQL**, a query language inspired by GraphQL but extended with graph-specific capabilities. Queries in Dgraph allow you to traverse the graph, following relationships from node to node to retrieve connected data in a single request. + +A typical query starts at one or more nodes and traverses relationships to gather related information: +```graphql +{ + person(func: eq(name, "Alice")) { + name + age + friend { + name + friend { + name + } + } + } +} +``` + +This traverses from Alice to her friends, and then to her friends' friends, returning the nested structure in one query. + +## Graph Traversals and Filtering + +Dgraph excels at traversing complex relationships. You can filter at any level of traversal, aggregate data, sort results, and paginate through large result sets. The query language supports recursive queries for exploring paths of variable length, filtering by regular expressions, geographic proximity, and full-text search. + +Variables and value aggregation allow you to build sophisticated queries that analyze patterns across your graph, such as finding the most connected nodes or calculating metrics across relationships. + +## Mutations + +Data modifications in Dgraph are called **mutations**. You can add new nodes, update existing attributes, create or remove relationships, and delete nodes. Mutations can be submitted in either RDF or JSON format, and multiple operations can be batched together in a single transaction for consistency. + +## Transactions and Consistency + +Dgraph provides ACID transactions, ensuring that your data remains consistent even under concurrent access. Transactions can span multiple queries and mutations, and Dgraph handles conflicts automatically to maintain data integrity across your distributed cluster. + +## Getting Started + +Working with Dgraph typically involves: +1. Defining your schema (optional but recommended) +2. Loading your data through mutations +3. Querying the graph to retrieve and analyze connected information +4. Iterating on your schema and queries as your application evolves + +The graph model naturally represents connected data, making it straightforward to model domains like social networks, recommendation systems, knowledge graphs, access control systems, and any application where relationships between entities matter as much as the entities themselves. + + +## What's Next + +- Get familiar with some terms in our Glossary +- Go through some tutorials + diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/dql-endpoints.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/dql-endpoints.md new file mode 100644 index 00000000..f6f1e847 --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/dql-endpoints.md @@ -0,0 +1,88 @@ +--- +title: Endpoints +--- + +Dgraph Alpha exposes endpoints for querying, mutating, and managing the database. You can interact with Dgraph using HTTP on port 8080 (plus optional port offset) or gRPC on port 9080 (plus optional port offset). Port offsets can be configured when starting Dgraph Alpha. If you're using a load balancer or reverse proxy, check your [architecture configuration](../installation/dgraph-architecture) to determine the actual endpoint addresses. + +## HTTP Endpoints + +Dgraph Alpha exposes the following HTTP endpoints on port `8080`: + +### `/query` + +Execute DQL queries. + +- **Method**: `POST` +- **Content-Type**: `application/dql` or `application/json` +- **Query Parameters**: + - `ro=true`: Execute as read-only transaction + - `be=true`: Execute as best-effort query + - `respFormat=rdf`: Return results in RDF format (default is JSON) + +### `/mutate` + +Execute DQL mutations (add, modify, or delete data). + +- **Method**: `POST` +- **Content-Type**: `application/rdf` or `application/json` +- **Query Parameters**: + - `startTs=`: Execute as part of an existing transaction + - `commitNow=true`: Commit the transaction immediately after mutation + +### `/commit` + +Commit or abort a transaction. + +- **Method**: `POST` +- **Query Parameters**: + - `startTs=`: Transaction start timestamp + - `abort=true`: Abort the transaction instead of committing + +### `/alter` + +Modify the DQL schema (add predicates, types, indexes, or drop schema elements). + +- **Method**: `POST` +- **Content-Type**: `text/plain` (DQL schema format) + +For detailed information on request/response formats, authentication, and examples, see [Raw HTTP](../clients/raw-http). + +## gRPC Methods + +Dgraph Alpha exposes the following gRPC methods on port `9080`: + +### `Query` + +Execute DQL queries. By default, returns results in JSON format. To get RDF format results, set `resp_format` to `RespFormat.RDF` in the request. + +### `Mutate` + +Execute DQL mutations (add, modify, or delete data). + +### `Commit` + +Commit or abort a transaction. + +### `Alter` + +Modify the DQL schema (add predicates, types, indexes, or drop schema elements). + +### Protocol Buffer Definitions + +The gRPC service definitions and message types are defined in the [api.proto](https://github.com/dgraph-io/dgo/blob/master/protos/api.proto) file. Refer to this file for complete method signatures, request/response message structures, and field definitions. + +For detailed information on using gRPC methods, see the [Go client](../clients/go) documentation, which provides comprehensive examples of gRPC usage. + +## Payload Format + +The rest of the DQL documentation describes the payload format in DQL syntax. The payload content (queries, mutations, schema definitions) is the same whether you use HTTP or gRPC—only the transport protocol differs. + +- **HTTP**: Send DQL payloads as request bodies with appropriate `Content-Type` headers +- **gRPC**: Send DQL payloads as protocol buffer messages + +## Additional Services + +In addition to DQL endpoints, Dgraph Alpha also exposes administrative services on the same ports: + +For details on administrative endpoints and operations, see the [Administration](../admin/index.md) section. + diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/dql-mutation.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/dql-mutation.md new file mode 100644 index 00000000..7e8308bc --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/dql-mutation.md @@ -0,0 +1,272 @@ +--- +title: Mutation +--- +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + + + +Dgraph Query Language (DQL) is Dgraph's proprietary language to add, modify, delete and fetch data. + +Fetching data is done through [DQL Queries](query/dql-query). Adding, modifying or deleting data is done through ***DQL Mutations***. + +This overview explains the structure of DQL Mutations and provides links to the appropriate DQL reference documentation. + + +DQL mutations support JSON or [RDF](dql-rdf) format. + +## set block +In DQL, you add data using a set mutation, identified by the `set` keyword. + + +```dql + { + "set": [ + { + "name":"Star Wars: Episode IV - A New Hope", + "release_date": "1977-05-25", + "director": { + "name": "George Lucas", + "dgraph.type": "Person" + }, + "starring" : [ + { + "name": "Luke Skywalker" + }, + { + "name": "Princess Leia" + }, + { + "name": "Han Solo" + } + ] + }, + { + "name":"Star Trek: The Motion Picture", + "release_date": "1979-12-07" + } + ] + } +``` + + +triples are in [RDF](dql-rdf) format. +```sh +{ + set { + # triples in here + _:n1 "Star Wars: Episode IV - A New Hope" . + _:n1 "1977-05-25" . + _:n1 _:n2 . + _:n2 "George Lucas" . + + } +} +``` + + + +### Node reference +A mutation can include a blank nodes as an identifier for the subject or object, or a known UID. +``` +{ + set { + # triples in here + <0x632ea2> "1977-05-25" . + } +} +``` +will add the `release_date` information to the node identified by UID `0x632ea2`. + +### language support +``` +{ + set { + # triples in here + <0x632ea2> "Star Wars, épisode IV : Un nouvel espoir"@fr . + } +} +``` + + + + +## delete block +A delete mutation, identified by the `delete` keyword, removes +[triples](dql-rdf) from the store. + +For example, if the store contained the following: +```RDF +<0xf11168064b01135b> "Lewis Carrol" +<0xf11168064b01135b> "1998" +<0xf11168064b01135b> "Person" . +``` + +Then, the following delete mutation deletes the specified erroneous data, and +removes it from any indexes: + +```sh +{ + delete { + <0xf11168064b01135b> "1998" . + } +} +``` + +### Wildcard delete + +In many cases you will need to delete multiple types of data for a predicate. +For a particular node `N`, all data for predicate `P` (and all corresponding +indexing) is removed with the pattern `S P *`. + +```sh +{ + delete { + <0xf11168064b01135b> * . + } +} +``` + +The pattern `S * *` deletes all predicates from a node `S`, along with any reverse edges +corresponding to the removed edges, and any indexing for the removed data. + +```sh +{ + delete { + <0xf11168064b01135b> * * . + } +} +``` + +:::note +When using the `S * *` pattern, Dgraph only deletes predicates that are defined in the types associated with the node via `dgraph.type`. + +For example, if a node has `dgraph.type: ["Person", "Author"]`, and the `Person` type defines predicates `name`, `age`, and `email`, while the `Author` type defines predicates `name` and `author.of`, then `S * *` will only delete triples for predicates that appear in at least one of these types (`name`, `age`, `email`, `author.of`). + +Any predicates on the node that are not defined in any of the node's types will remain after the `S * *` delete mutation. This means: +- If a node has untyped predicates (predicates not in any `dgraph.type`), those predicates will not be deleted. +- If a node has no `dgraph.type` assigned, `S * *` will have no effect. +::: + +**Example:** + +Consider a node with UID `0x123` that has: +- `dgraph.type: "Person"` (where `Person` type defines `name`, `age`, `email`) +- Predicate `name: "Alice"` (typed - will be deleted) +- Predicate `age: "30"` (typed - will be deleted) +- Predicate `custom_field: "value"` (untyped - will NOT be deleted) + +After executing `delete { <0x123> * * . }`, the node will still exist with only `custom_field: "value"` remaining. + +:::note +The patterns `* P O` and `* * O` are not supported. +::: + +### Deletion of non-list predicates + +Deleting the value of a non-list predicate (i.e a 1-to-1 relationship) can be +done in two ways. + +* Using the [wildcard delete](#wildcard-delete) (star notation) + mentioned in the last section. +* Setting the object to a specific value. If the value passed is not the +current value, the mutation will succeed but will have no effect. If the value +passed is the current value, the mutation will succeed and will delete the +non-list predicate. + +For language-tagged values, the following special syntax is supported: + +``` +{ + delete { + <0x12345> * . + } +} +``` + +In this example, the value of the `name` field that is tagged with the language +tag `es` is deleted. Other tagged values are left untouched. + +## upsert block +Upsert is an operation where: + +1. A node is searched for, and then +2. Depending on if it is found or not, either: + - Updating some of its attributes, or + - Creating a new node with those attributes. + +The upsert block allows performing queries and mutations in a single request. The upsert +block contains one query block and mutation blocks. + +The structure of the upsert block is as follows: + +``` +upsert { + query + mutation + [mutation ] + ... +} +``` + +Execution of an upsert block also returns the response of the query executed on the state +of the database *before mutation was executed*. +To get the latest result, you have to execute another query after the transaction is committed. + +Variables defined in the query block can be used in the mutation blocks using the [uid](upserts#val-function) and [val](upserts#val-function) functions. + +## conditional upsert +The upsert block also allows specifying conditional mutation blocks using an `@if` +directive. The mutation is executed only when the specified condition is true. If the +condition is false, the mutation is silently ignored. The general structure of +Conditional Upsert looks like as follows: + +``` +upsert { + query + [fragment ] + mutation [@if()] + [mutation [@if()] ] + ... +} +``` +The `@if` directive accepts a condition on variables defined in the query block and can be +connected using `AND`, `OR` and `NOT`. + +## Example of Conditional Upsert + +Let's say in our previous example, we know the `company1` has less than 100 employees. +For safety, we want the mutation to execute only when the variable `v` stores less than +100 but greater than 50 UIDs in it. This can be achieved as follows: + +```sh +curl -H "Content-Type: application/rdf" -X POST localhost:8080/mutate?commitNow=true -d $' +upsert { + query { + v as var(func: regexp(email, /.*@company1.io$/)) + } + + mutation @if(lt(len(v), 100) AND gt(len(v), 50)) { + delete { + uid(v) * . + uid(v) * . + uid(v) * . + } + } +}' | jq +``` + +We can achieve the same result using `json` dataset as follows: + +```sh +curl -H "Content-Type: application/json" -X POST localhost:8080/mutate?commitNow=true -d '{ + "query": "{ v as var(func: regexp(email, /.*@company1.io$/)) }", + "cond": "@if(lt(len(v), 100) AND gt(len(v), 50))", + "delete": { + "uid": "uid(v)", + "name": null, + "email": null, + "age": null + } +}' | jq +``` diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/dql-rdf.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/dql-rdf.md new file mode 100644 index 00000000..0593a10e --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/dql-rdf.md @@ -0,0 +1,138 @@ +--- +title: RDF Data Format +--- +Dgraph natively supports Resource Description Framework (RDF) when creating, importing and exporting data. Dgraph Client libraries can be used to query RDF as well. + +[RDF 1.1](https://www.w3.org/RDF/) is a Semantic Web Standard for data interchange defined by the W3C. It expresses statements about resources. The format of these statements is simple and in the form of triples. + + +A triple has the form +``` + . +``` + +In RDF terminology, each triple represents one fact about a node. + +In Dgraph, the `` of a triple is always a node, and must be a numeric UID. The `` of a triple may be another node or a literal value: +``` +<0x01> "Alice" . +<0x01> <0x02> . +``` +The first triple specifies that a node has a name property of “Alice”. The subject is the UID of the first node, the predicate is `name`, and the object is the literal value string: `"Alice"`. +The second triple specifies that Alice knows Bob. The subject is again the UID of a node (the "alice" node), the predicate is `knows`, and the object of this triple is the uid of the other node (the "bob" node). When the object is a UID, the triple represents a relationship in Dgraph. + +Each triple representation in RDF ends with a period. + +### Blank nodes in mutations +When creating nodes in Dgraph, you often let Dgraph assign the node [UID](../dgraph-glossary#uid) by specifing a blank node starting with "_:". All references to the same blank node, such as `_:identifier123`, will identify the same node within a mutation. Dgraph creates a UID identifying each blank node. +### Language for string values +Languages are written using `@lang`. For example +``` +<0x01> "Adelaide"@en . +<0x01> "Аделаида"@ru . +<0x01> "Adélaïde"@fr . +<0x01> "Person" . +``` +See also [how language strings are handled in queries](query/language-support). + +### Types +Dgraph understands standard RDF types specified in RDF using the `^^` separator. For example +``` +<0x01> "32"^^ . +<0x01> "1985-06-08"^^ . +``` + +The supported [RDF datatypes](https://www.w3.org/TR/rdf11-concepts/#section-Datatypes) and the corresponding internal Dgraph type are as follows. + +| Storage Type | Dgraph type | +| ------------- | :------------: | +| <xs:string> | `string` | +| <xs:dateTime> | `dateTime` | +| <xs:date> | `datetime` | +| <xs:int> | `int` | +| <xs:integer> | `int` | +| <xs:boolean> | `bool` | +| <xs:double> | `float` | +| <xs:float> | `float` | +| <geo:geojson> | `geo` | +| <xs:password> | `password` | +| <http://www.w3.org/2001/XMLSchema#string> | `string` | +| <http://www.w3.org/2001/XMLSchema#dateTime> | `dateTime` | +| <http://www.w3.org/2001/XMLSchema#date> | `dateTime` | +| <http://www.w3.org/2001/XMLSchema#int> | `int` | +| <http://www.w3.org/2001/XMLSchema#positiveInteger> | `int` | +| <http://www.w3.org/2001/XMLSchema#integer> | `int` | +| <http://www.w3.org/2001/XMLSchema#boolean> | `bool` | +| <http://www.w3.org/2001/XMLSchema#double> | `float` | +| <http://www.w3.org/2001/XMLSchema#float> | `float` | + + +### Facets + +Dgraph is more expressive than RDF in that it allows properties to be stored on every relation. These properties are called Facets in Dgraph, and dgraph allows an extension to RDF where facet values are incuded in any triple. +#### Creating a list with facets + +The following set operation uses a sequence of RDF statements with additional facet information: +```sh +{ + set { + _:Julian "Julian" . + _:Julian "Jay-Jay" (kind="first") . + _:Julian "Jules" (kind="official") . + _:Julian "JB" (kind="CS-GO") . + } +} +``` + +```graphql +{ + q(func: eq(name,"Julian")){ + name + nickname @facets + } +} +``` +Result: +```JSON +{ + "data": { + "q": [ + { + "name": "Julian", + "nickname|kind": { + "0": "first", + "1": "official", + "2": "CS-GO" + }, + "nickname": [ + "Jay-Jay", + "Jules", + "JB" + ] + } + ] + } +} +``` +:::tip +Dgraph can automatically generate a reverse relation. If the user wants to run +queries in that direction, they would define the [reverse relationship](dql-schema#reverse-predicates). +::: + +## N-quads format +While most RDF data uses only triples (with three parts) an optional fourth part is allowed. This fourth component in RDF is called a graph label, and in Dgraph it must be the UID of the namespace that the data should go into. + +## Processing RDF to comply with Dgraph syntax for subjects + +While it is valid RDF to specify subjects that are IRI strings, Dgraph requires a numeric UID or a blank node as the subject. If a string IRI is required, Dgraph support them via [xid properties](upserts#external-ids). When importing RDF from another source that does not use numeric UID subjects, it will be required to replace arbitrary subject IRIs with blank node IRIs. + +Typically this is done simply by prepending "_:" to the start of the original IRI. So a triple such as: + +``` "somevalue"^^xs:string``` + +may be rewritten as + +```<_:http://abc.org/schema/foo#item1> "somevalue"^^xs:string``` + +Dgraph will create a consistent UID for all references to the uniquely-named blank node. To maintain this uniqueness over multiple data loads, use the [dgraph live](../dgraph-glossary#uid) utility with the xid option, or use specific UIDs such as the hash of the IRI in the source RDF directly. + diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/dql-schema.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/dql-schema.md new file mode 100644 index 00000000..3804cc93 --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/dql-schema.md @@ -0,0 +1,367 @@ +--- +title: Schema +--- + +The Dgraph schema defines [predicate types](#predicate-types) and [node types](#node-types). + +**Example schema:** + +```dql +name: string @index(term) . +release_date: datetime @index(year) . +revenue: float . +running_time: int . +starring: [uid] . +director: [uid] . +description: string . +description_vector: float32vector @index(hnsw(metric:"cosine")) . + +type Person { + name +} + +type Film { + name + release_date + revenue + running_time + starring + director + description + description_vector +} +``` + +## Predicate Types + +Predicates are declared in the Dgraph schema with their type, cardinality, indexes, and language support. + +A predicate is created either: +- By altering the schema (see [Update Dgraph types](../admin/admin-tasks/update-dgraph-types/)) +- During a mutation, if the cluster's **schema mode** is `flexible` and the predicate doesn't exist + +When a predicate type isn't declared: +- The type is inferred from the first mutation +- [RDF type annotations](dql-rdf) are used if present +- Otherwise, the type defaults to `default` + +A predicate holds either a literal value ([scalar type](#scalar-types)) or a relationship ([UID type](#uid-type)). + +### Scalar Types + +| Dgraph Type | Go Type | Notes | +|-------------|---------|-------| +| `default` | string | Default when type cannot be inferred | +| `int` | int64 | | +| `float` | float | | +| `bigfloat` | big.Float| from math/big | +| `string` | string | | +| `bool` | bool | | +| `dateTime` | time.Time | RFC3339 format (e.g., `2006-01-02T15:04:05.999999999+10:00`) | +| `geo` | [go-geom](https://github.com/twpayne/go-geom) | Geographic data | +| `password` | string | Encrypted with bcrypt | + +:::note +Dgraph requires RFC 3339 format for `dateTime`, which differs from ISO 8601. Convert values before sending to Dgraph. +::: + +### Vector Type + +The `float32vector` type stores an ordered array of 32-bit floats, typically used for ML embeddings. + +When [indexed](predicate-indexing), vectors enable similarity search using the [similar_to](query/functions#vector-similarity-search) function. + +### UID Type + +The `uid` type represents a relationship to another node. Internally, each node is identified by a `uint64` UID. + +### Predicate Naming + +Predicate names can be any alphanumeric combination. Dgraph also supports [Internationalized Resource Identifiers](https://en.wikipedia.org/wiki/Internationalized_Resource_Identifier) (IRIs) — see [Predicates i18n](#predicates-i18n). + +:::note +Names starting with `dgraph.` are reserved for internal use. +::: + +**Allowed special characters** (when prefixed/suffixed with alphanumerics): +``` +][&*()_-+=!#$% +``` + +**Not allowed:** +``` +^}|{`\~ +``` + +:::tip +The `@` suffix is allowed but ignored. +::: + +### Predicates i18n + +For predicate names with language-specific characters or URIs, enclose them in angle brackets `<>`: + +```dql +<职业>: string @index(exact) . +<年龄>: int @index(int) . +<地点>: geo @index(geo) . +<公司>: string @index(fulltext) @lang . +``` + +Use the `@lang` directive for proper full-text tokenization: + +**Mutation:** +```dql +{ + set { + _:a <公司> "Dgraph Labs Inc"@en . + _:b <公司> "夏新科技有限责任公司"@zh . + _:a "Company" . + } +} +``` + +**Query:** +```dql +{ + q(func: alloftext(<公司>@zh, "夏新科技有限责任公司")) { + uid + <公司>@. + } +} +``` + +### Schema Directives + +#### `@unique` + +Ensures all values of a predicate are distinct. Requires an index. + +```dql +email: string @unique @index(exact) . +``` + +| Data Type | Required Index | +|-----------|----------------| +| `string` | `hash` or `exact` | +| `int` | `int` | + +Dgraph automatically adds `@upsert` when `@unique` is specified. + +#### `@upsert` + +Enables [upsert operations](upserts) with conflict detection on index keys: + +```dql +email: string @index(exact) @upsert . +``` + +#### `@noconflict` + +Disables conflict detection for a predicate. Use with caution. + +```dql +counter: int @noconflict . +``` + +:::warning +This is experimental and can cause data loss, especially with count indexes. +::: + +### Password Type + +Passwords are stored encrypted and can only be verified, not queried directly. + +**Schema:** +```dql +pass: password . +``` + +**Set password:** +```dql +{ + set { + <0x123> "Password Example" . + <0x123> "ThePassword" . + } +} +``` + +**Verify password:** +```dql +{ + check(func: uid(0x123)) { + name + checkpwd(pass, "ThePassword") + } +} +``` + +**Response:** +```json +{ + "data": { + "check": [ + { + "name": "Password Example", + "checkpwd(pass)": true + } + ] + } +} +``` + +Use an alias for cleaner output: + +```dql +{ + check(func: uid(0x123)) { + name + secret: checkpwd(pass, "ThePassword") + } +} +``` + +### RDF Type Inference + +When a mutation includes an RDF type that differs from the schema type, Dgraph checks convertibility and stores in the RDF type's corresponding Dgraph type. Query results return the schema type. + +**Example** (no schema defined for `age`): + +```dql +{ + set { + _:a "15"^^ . + _:b "13" . + _:c "14"^^ . + _:d "14.5"^^ . + _:e "14.5" . + } +} +``` + +Dgraph: +- Sets schema type to `int` (from first triple) +- Converts `"13"` to `int` +- Stores `"14"` as `string` (convertible to `int`) +- Throws error for `"14.5"` triples (not convertible to `int`) + +## Predicate Indexing + +Indexes enable [filtering functions](query/functions) in queries. See [Predicate Indexing](predicate-indexing/) for details. + +## Facets (Edge Attributes) + +Facets are **key-value pairs attached to predicates** rather than nodes. They add properties to attributes and relationships. + +### When to Use Facets + +Facets are ideal for relationship metadata: +- `friend` edge with `since` timestamp +- `rated` edge with `rating` score +- `member_of` edge with `role` + +:::note +Facets cannot be indexed or used in root query functions. +::: + +### Facet Types + +| Type | Description | +|------|-------------| +| `string` | Text value | +| `bool` | `true` or `false` | +| `int` | 32-bit signed integer | +| `float` | 64-bit floating point | +| `bigfloat` | big.Float from math/big | +| `dateTime` | RFC3339 timestamp | + +### Facets Are Not in Schema + +Facets are defined inline during mutations — not declared in the schema: + +```rdf +_:alice _:bob (close=true, since=2020-01-01T00:00:00) . +_:alice "MA0123" (since=2006-02-02T13:01:09, first=true) . +``` + +Dgraph infers facet types from values. + +For querying facets, see [Facets in Queries](query/facets). + +## Node Types + +Node types declare which predicates a node can have. They are optional. + +### Defining Node Types + +```dql +name: string @index(term) . +dob: datetime . +home_address: string . +friends: [uid] . + +type Student { + name + dob + home_address + friends +} +``` + +- All predicates in a type must be defined in the schema +- Different types can share predicates + +### Reverse Predicates + +Include reverse edges using the `~` prefix: + +```dql +children: [uid] @reverse . +name: string @index(term) . + +type Parent { + name + children +} + +type Child { + name + <~children> +} +``` + +:::tip +Enclose predicates with special characters in angle brackets `<>`. +::: + +### Assigning Types to Nodes + +Set the `dgraph.type` predicate (supports multiple types): + +```dql +{ + set { + _:a "Garfield" . + _:a "Pet" . + _:a "Animal" . + } +} +``` + +:::note +DQL types are declarative only — Dgraph doesn't enforce them. You can: +- Add nodes without `dgraph.type` +- Add predicates not declared in the node's type +::: + +### When Node Types Matter + +Node types are required for: +- **Delete all predicates**: `delete { * * . }` uses the type to find predicates +- **Expand all**: [expand(_all_)](query/expand-predicates) uses the type to list predicates +- **Type filtering**: The [type()](query/functions#type) function in queries + +:::warning +`delete { * * . }` only deletes predicates declared in the type. Predicates added outside the type definition remain. +::: diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/index.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/index.md new file mode 100644 index 00000000..dec064b8 --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/index.md @@ -0,0 +1,55 @@ +--- +title: Query Language +description: Dgraph Query Language (DQL) is Dgraph's proprietary language to add, modify, delete and fetch data. +--- + +Dgraph Query Language (DQL) is Dgraph's proprietary language to add, modify, delete and fetch data. It provides a powerful and expressive way to work with graph data, allowing you to traverse relationships, filter nodes, and retrieve complex graph structures. + +## Essential Concepts + +### Nodes, Predicates, and Facets +Before diving into DQL, it's important to understand how Dgraph structures data: + +**Nodes** represent entities or "things" in your domain—like a person, a movie, or a company. Each node has a unique identifier ([UID](../dgraph-glossary#uid)) that Dgraph assigns. + +**Predicates** are the smallest unit of information in Dgraph. They describe facts about nodes and come in two forms: +- **Attributes**: Store literal values (strings, numbers, dates) directly on a node. For example, a `name` predicate with value "Alice" is an attribute. +- **Relationships**: Connect one node to another node. For example, a `knows` predicate linking Alice's node to Bob's node is a relationship. + +Dgraph stores data as facts using predicates. Each fact follows the pattern: ` `, where the value can be either a literal (for attributes) or another node's UID (for relationships). + +**Facets** are metadata that can be attached to any predicate—both attributes and relationships. They provide additional context about the predicate itself, such as when a relationship was established, the confidence level of an attribute, or any other property about the fact. Learn more about [facets](../design-concepts/facets-concept). + + +### Transactions + +Dgraph supports transactions to ensure data consistency and atomicity. A transaction groups multiple operations (queries and mutations) into a single atomic unit. + +#### Transaction Lifecycle + +1. **Start a transaction**: When you execute a query or mutation, Dgraph automatically starts a new transaction and assigns it a unique transaction ID (also called `start_ts` or start timestamp). + +2. **Execute operations**: Within a transaction, you can perform multiple queries and mutations. All operations in the same transaction use the same transaction ID to ensure they see a consistent view of the data. + +3. **Commit or rollback**: + - **Commit**: Makes all changes in the transaction permanent. After committing, the changes are visible to other transactions. + - **Rollback (abort)**: Discards all changes in the transaction. The database returns to its state before the transaction started. + +#### Transaction Contents + +A transaction can contain: +- **Queries**: Read data from the database +- **Mutations**: Add, modify, or delete data + +You can perform multiple queries and mutations within a single transaction before committing. This allows you to read data, make decisions based on that data, and then apply mutations—all while maintaining consistency. + +#### Read-Only Transactions + +**Read-only** transactions are optimized for read operations and cannot contain mutations. They are useful to increase read speed because they can circumvent the usual consensus protocol. Attempting to perform a mutation in a read-only transaction will result in an error. + +Read-only queries can optionally be set as **best-effort**. Using this flag asks the Dgraph Alpha to try to get timestamps from memory on a best-effort basis to reduce the number of outbound requests to Zero. This may yield improved latencies in read-bound workloads where linearizable reads are not strictly needed. + +For more details, see [Transactions](../design-concepts/transactions-concept).. + + + diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/indexing-custom-tokenizers.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/indexing-custom-tokenizers.md new file mode 100644 index 00000000..a3b3db38 --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/indexing-custom-tokenizers.md @@ -0,0 +1,568 @@ +--- +title: Custom Tokenizers +--- + +Dgraph comes with a large toolkit of builtin indexes, but sometimes for niche +use cases they're not always enough. + +Dgraph allows you to implement custom tokenizers via a plugin system in order +to fill the gaps. + +## Caveats + +The plugin system uses Go's [`pkg/plugin`](https://golang.org/pkg/plugin/). +This brings some restrictions to how plugins can be used. + +- Plugins must be written in Go. + +- As of Go 1.9, `pkg/plugin` only works on Linux. Therefore, plugins will only + work on Dgraph instances deployed in a Linux environment. + +- The version of Go used to compile the plugin should be the same as the version + of Go used to compile Dgraph itself. Dgraph always uses the latest version of +Go (and so should you!). + +## Implementing a plugin + +:::note +You should consider Go's [plugin](https://golang.org/pkg/plugin/) documentation +to be supplementary to the documentation provided here. +::: + +Plugins are implemented as their own main package. They must export a +particular symbol that allows Dgraph to hook into the custom logic the plugin +provides. + +The plugin must export a symbol named `Tokenizer`. The type of the symbol must +be `func() interface{}`. When the function is called the result returned should +be a value that implements the following interface: + +``` +type PluginTokenizer interface { + // Name is the name of the tokenizer. It should be unique among all + // builtin tokenizers and other custom tokenizers. It identifies the + // tokenizer when an index is set in the schema and when search/filter + // is used in queries. + Name() string + + // Identifier is a byte that uniquely identifiers the tokenizer. + // Bytes in the range 0x80 to 0xff (inclusive) are reserved for + // custom tokenizers. + Identifier() byte + + // Type is a string representing the type of data that is to be + // tokenized. This must match the schema type of the predicate + // being indexed. Allowable values are shown in the table below. + Type() string + + // Tokens should implement the tokenization logic. The input is + // the value to be tokenized, and will always have a concrete type + // corresponding to Type(). The return value should be a list of + // the tokens generated. + Tokens(interface{}) ([]string, error) +} +``` + +The return value of `Type()` corresponds to the concrete input type of +`Tokens(interface{})` in the following way: + + `Type()` return value | `Tokens(interface{})` input type +-----------------------|---------------------------------- + `"int"` | `int64` + `"float"` | `float64` + `"string"` | `string` + `"bool"` | `bool` + `"datetime"` | `time.Time` + +## Building the plugin + +The plugin has to be built using the `plugin` build mode so that an `.so` file +is produced instead of a regular executable. For example: + +```sh +go build -buildmode=plugin -o myplugin.so ~/go/src/myplugin/main.go +``` + +## Running Dgraph with plugins + +When starting Dgraph, use the `--custom_tokenizers` flag to tell Dgraph which +tokenizers to load. It accepts a comma separated list of plugins. E.g. + +```sh +dgraph ...other-args... --custom_tokenizers=plugin1.so,plugin2.so +``` + +:::note +Plugin validation is performed on startup. If a problem is detected, Dgraph +will refuse to initialize. +::: + +## Adding the index to the schema + +To use a tokenization plugin, an index has to be created in the schema. + +The syntax is the same as adding any built-in index. To add an custom index +using a tokenizer plugin named `foo` to a `string` predicate named +`my_predicate`, use the following in the schema: + +```sh +my_predicate: string @index(foo) . +``` + +## Using the index in queries + +There are two functions that can use custom indexes: + + Mode | Behavior +--------|------- + `anyof` | Returns nodes that match on *any* of the tokens generated + `allof` | Returns nodes that match on *all* of the tokens generated + +The functions can be used either at the query root or in filters. + +There behavior here an analogous to `anyofterms`/`allofterms` and +`anyoftext`/`alloftext`. + +## Examples + +The following examples should make the process of writing a tokenization plugin +more concrete. + +### Unicode Characters + +This example shows the type of tokenization that is similar to term +tokenization of full-text search. Instead of being broken down into terms or +stem words, the text is instead broken down into its constituent unicode +codepoints (in Go terminology these are called *runes*). + +:::note +This tokenizer would create a very large index that would be expensive to +manage and store. That's one of the reasons that text indexing usually occurs +at a higher level; stem words for full-text search or terms for term search. +::: + +The implementation of the plugin looks like this: + +```go +package main + +import "encoding/binary" + +func Tokenizer() interface{} { return RuneTokenizer{} } + +type RuneTokenizer struct{} + +func (RuneTokenizer) Name() string { return "rune" } +func (RuneTokenizer) Type() string { return "string" } +func (RuneTokenizer) Identifier() byte { return 0xfd } + +func (t RuneTokenizer) Tokens(value interface{}) ([]string, error) { + var toks []string + for _, r := range value.(string) { + var buf [binary.MaxVarintLen32]byte + n := binary.PutVarint(buf[:], int64(r)) + tok := string(buf[:n]) + toks = append(toks, tok) + } + return toks, nil +} +``` + +**Hints and tips:** + +- Inside `Tokens`, you can assume that `value` will have concrete type + corresponding to that specified by `Type()`. It's safe to do a type +assertion. + +- Even though the return value is `[]string`, you can always store non-unicode + data inside the string. See [this blogpost](https://blog.golang.org/strings) +for some interesting background how string are implemented in Go and why they +can be used to store non-textual data. By storing arbitrary data in the string, +you can make the index more compact. In this case, varints are stored in the +return values. + +Setting up the indexing and adding data: +``` +name: string @index(rune) . +``` + + +``` +{ + set{ + _:ad "Adam" . + _:ad "Person" . + _:aa "Aaron" . + _:aa "Person" . + _:am "Amy" . + _:am "Person" . + _:ro "Ronald" . + _:ro "Person" . + } +} +``` +Now queries can be performed. + +The only person that has all of the runes `A` and `n` in their `name` is Aaron: +``` +{ + q(func: allof(name, rune, "An")) { + name + } +} +=> +{ + "data": { + "q": [ + { "name": "Aaron" } + ] + } +} +``` +But there are multiple people who have both of the runes `A` and `m`: +``` +{ + q(func: allof(name, rune, "Am")) { + name + } +} +=> +{ + "data": { + "q": [ + { "name": "Amy" }, + { "name": "Adam" } + ] + } +} +``` +Case is taken into account, so if you search for all names containing `"ron"`, +you would find `"Aaron"`, but not `"Ronald"`. But if you were to search for +`"no"`, you would match both `"Aaron"` and `"Ronald"`. The order of the runes in +the strings doesn't matter. + +It's possible to search for people that have *any* of the supplied runes in +their names (rather than *all* of the supplied runes). To do this, use `anyof` +instead of `allof`: +``` +{ + q(func: anyof(name, rune, "mr")) { + name + } +} +=> +{ + "data": { + "q": [ + { "name": "Adam" }, + { "name": "Aaron" }, + { "name": "Amy" } + ] + } +} +``` +`"Ronald"` doesn't contain `m` or `r`, so isn't found by the search. + +:::note +Understanding what's going on under the hood can help you intuitively +understand how `Tokens` method should be implemented. + +When Dgraph sees new edges that are to be indexed by your tokenizer, it +will tokenize the value. The resultant tokens are used as keys for posting +lists. The edge subject is then added to the posting list for each token. + +When a query root search occurs, the search value is tokenized. The result of +the search is all of the nodes in the union or intersection of the corresponding +posting lists (depending on whether `anyof` or `allof` was used). +::: + +### CIDR Range + +Tokenizers don't always have to be about splitting text up into its constituent +parts. This example indexes [IP addresses into their CIDR +ranges](https://en.wikipedia.org/wiki/Classless_Inter-Domain_Routing). This +allows you to search for all IP addresses that fall into a particular CIDR +range. + +The plugin code is more complicated than the rune example. The input is an IP +address stored as a string, e.g. `"100.55.22.11/32"`. The output are the CIDR +ranges that the IP address could possibly fall into. There could be up to 32 +different outputs (`"100.55.22.11/32"` does indeed have 32 possible ranges, one +for each mask size). + +```go +package main + +import "net" + +func Tokenizer() interface{} { return CIDRTokenizer{} } + +type CIDRTokenizer struct{} + +func (CIDRTokenizer) Name() string { return "cidr" } +func (CIDRTokenizer) Type() string { return "string" } +func (CIDRTokenizer) Identifier() byte { return 0xff } + +func (t CIDRTokenizer) Tokens(value interface{}) ([]string, error) { + _, ipnet, err := net.ParseCIDR(value.(string)) + if err != nil { + return nil, err + } + ones, bits := ipnet.Mask.Size() + var toks []string + for i := ones; i >= 1; i-- { + m := net.CIDRMask(i, bits) + tok := net.IPNet{ + IP: ipnet.IP.Mask(m), + Mask: m, + } + toks = append(toks, tok.String()) + } + return toks, nil +} +``` +An example of using the tokenizer: + +Setting up the indexing and adding data: +``` +ip: string @index(cidr) . + +``` + +``` +{ + set{ + _:a "100.55.22.11/32" . + _:b "100.33.81.19/32" . + _:c "100.49.21.25/32" . + _:d "101.0.0.5/32" . + _:e "100.176.2.1/32" . + } +} +``` +``` +{ + q(func: allof(ip, cidr, "100.48.0.0/12")) { + ip + } +} +=> +{ + "data": { + "q": [ + { "ip": "100.55.22.11/32" }, + { "ip": "100.49.21.25/32" } + ] + } +} +``` +The CIDR ranges of `100.55.22.11/32` and `100.49.21.25/32` are both +`100.48.0.0/12`. The other IP addresses in the database aren't included in the +search result, since they have different CIDR ranges for 12 bit masks +(`100.32.0.0/12`, `101.0.0.0/12`, `100.154.0.0/12` for `100.33.81.19/32`, +`101.0.0.5/32`, and `100.176.2.1/32` respectively). + +Note that we're using `allof` instead of `anyof`. Only `allof` will work +correctly with this index. Remember that the tokenizer generates all possible +CIDR ranges for an IP address. If we were to use `anyof` then the search result +would include all IP addresses under the 1 bit mask (in this case, `0.0.0.0/1`, +which would match all IPs in this dataset). + +### Anagram + +Tokenizers don't always have to return multiple tokens. If you just want to +index data into groups, have the tokenizer just return an identifying member of +that group. + +In this example, we want to find groups of words that are +[anagrams](https://en.wikipedia.org/wiki/Anagram) of each +other. + +A token to correspond to a group of anagrams could just be the letters in the +anagram in sorted order, as implemented below: + +```go +package main + +import "sort" + +func Tokenizer() interface{} { return AnagramTokenizer{} } + +type AnagramTokenizer struct{} + +func (AnagramTokenizer) Name() string { return "anagram" } +func (AnagramTokenizer) Type() string { return "string" } +func (AnagramTokenizer) Identifier() byte { return 0xfc } + +func (t AnagramTokenizer) Tokens(value interface{}) ([]string, error) { + b := []byte(value.(string)) + sort.Slice(b, func(i, j int) bool { return b[i] < b[j] }) + return []string{string(b)}, nil +} +``` +In action: + +Setting up the indexing and adding data: +``` +word: string @index(anagram) . +``` + +``` +{ + set{ + _:1 "airmen" . + _:2 "marine" . + _:3 "beat" . + _:4 "beta" . + _:5 "race" . + _:6 "care" . + } +} +``` +``` +{ + q(func: allof(word, anagram, "remain")) { + word + } +} +=> +{ + "data": { + "q": [ + { "word": "airmen" }, + { "word": "marine" } + ] + } +} +``` + +Since a single token is only ever generated, it doesn't matter if `anyof` or +`allof` is used. The result will always be the same. + +### Integer prime factors + +All of the custom tokenizers shown previously have worked with strings. +However, other data types can be used as well. This example is contrived, but +nonetheless shows some advanced usages of custom tokenizers. + +The tokenizer creates a token for each prime factor in the input. + +``` +package main + +import ( + "encoding/binary" + "fmt" +) + +func Tokenizer() interface{} { return FactorTokenizer{} } + +type FactorTokenizer struct{} + +func (FactorTokenizer) Name() string { return "factor" } +func (FactorTokenizer) Type() string { return "int" } +func (FactorTokenizer) Identifier() byte { return 0xfe } + +func (FactorTokenizer) Tokens(value interface{}) ([]string, error) { + x := value.(int64) + if x <= 1 { + return nil, fmt.Errorf("Cannot factor int <= 1: %d", x) + } + var toks []string + for p := int64(2); x > 1; p++ { + if x%p == 0 { + toks = append(toks, encodeInt(p)) + for x%p == 0 { + x /= p + } + } + } + return toks, nil + +} + +func encodeInt(x int64) string { + var buf [binary.MaxVarintLen64]byte + n := binary.PutVarint(buf[:], x) + return string(buf[:n]) +} +``` +:::note +Notice that the return of `Type()` is `"int"`, corresponding to the concrete +type of the input to `Tokens` (which is `int64`). +::: + +This allows you do things like search for all numbers that share prime +factors with a particular number. + +In particular, we search for numbers that contain any of the prime factors of +15, i.e. any numbers that are divisible by either 3 or 5. + +Setting up the indexing and adding data: +``` +num: int @index(factor) . +``` + +``` +{ + set{ + _:2 "2"^^ . + _:3 "3"^^ . + _:4 "4"^^ . + _:5 "5"^^ . + _:6 "6"^^ . + _:7 "7"^^ . + _:8 "8"^^ . + _:9 "9"^^ . + _:10 "10"^^ . + _:11 "11"^^ . + _:12 "12"^^ . + _:13 "13"^^ . + _:14 "14"^^ . + _:15 "15"^^ . + _:16 "16"^^ . + _:17 "17"^^ . + _:18 "18"^^ . + _:19 "19"^^ . + _:20 "20"^^ . + _:21 "21"^^ . + _:22 "22"^^ . + _:23 "23"^^ . + _:24 "24"^^ . + _:25 "25"^^ . + _:26 "26"^^ . + _:27 "27"^^ . + _:28 "28"^^ . + _:29 "29"^^ . + _:30 "30"^^ . + } +} +``` +``` +{ + q(func: anyof(num, factor, 15)) { + num + } +} +=> +{ + "data": { + "q": [ + { "num": 3 }, + { "num": 5 }, + { "num": 6 }, + { "num": 9 }, + { "num": 10 }, + { "num": 12 }, + { "num": 15 }, + { "num": 18 } + { "num": 20 }, + { "num": 21 }, + { "num": 25 }, + { "num": 24 }, + { "num": 27 }, + { "num": 30 }, + ] + } +} +``` diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/json-mutation-format.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/json-mutation-format.md new file mode 100644 index 00000000..20267e2b --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/json-mutation-format.md @@ -0,0 +1,460 @@ +--- +title: JSON Data Format +--- +import Highlight from '@site/src/components/Highlight'; + + +Dgraph supports [Mutations](dql-mutation) in JSON or [RDF](dql-rdf) format. +When using JSON format Dgraph creates nodes and relationships from the JSON structure and assigns UIDs to nodes. + +## Quick Start Example + +If you followed the [Quick Start guide](../quick-start), you added data to your graph using RDF format. The same data can also be added using JSON format. Here's an example of how to create the movie data from the quick start using JSON: + +```dql +{ + "set": [ + { + "name": "Star Wars: Episode IV - A New Hope", + "release_date": "1977-05-25", + "director": { + "name": "George Lucas", + "dgraph.type": "Person" + }, + "starring": [ + { + "name": "Luke Skywalker" + }, + { + "name": "Princess Leia" + }, + { + "name": "Han Solo" + } + ] + }, + { + "name": "Star Trek: The Motion Picture", + "release_date": "1979-12-07" + } + ] +} +``` + +The sample JSON data is an array of two movies with some attributes. These are stored as [nodes](../dgraph-glossary#node) in Dgraph. + +The "Star Wars" movie has a `director` field which is a JSON object and a `starring` field which is an array of JSON objects. Each object is also stored as a node in Dgraph. The `director` and `starring` are stored as [relationships](../dgraph-glossary#relationship). + +## Specifying node UIDs + +When you create nodes using JSON mutations, Dgraph automatically assigns a [UID](../dgraph-glossary#uid) to each new node. Dgraph also generates an internal identifier during the transaction, which is then converted to the final UID. + +For example, this mutation creates a single node: + +```dql + { + "set": [ + { + "name": "diggy", + "dgraph.type": "Mascot" + } + ] + } +``` + +Dgraph responds with: +```json +{ + "data": { + "code": "Success", + "message": "Done", + "queries": null, + "uids": { + "dg.3162278161.22055": "0xfffd8d72745f0650" + } + } +``` + +Meaning that Dgraph has created one node from the JSON. It has used the identifier `dg.3162278161.22055` during the transaction. And the final UID value for this node is `0xfffd8d72745f0650`. + +You can control the identifier name by specifying a `uid` field in your JSON data and using the notation: +``` "uid" : "_:" ``` + + +In this mutation, there are two JSON objects and because they are referring to the same identifier, Dgraph creates only one node: + +```dql + { + "set": [ + { + "uid": "_:diggy", + "name": "diggy", + "dgraph.type": "Mascot" + }, + { + "uid": "_:diggy", + "specie": "badger" + } + ] + } +``` + +When you run this mutation, you can see that Dgraph returns the UID of the node that was created with the `diggy` identifier: + + +{`{ + "data": { + "code": "Success", + "message": "Done", + "queries": null, + "uids": { + "diggy": "0xfffd8d72745f0691" + } + } +}`} + + +Note that the `specie` field is added to the node already created with `name` and `dgraph.type` information. + +### Referencing existing nodes + +You can use the `"uid"` field to reference an existing node. To do so, you must specify the UID value of the node. + + +For example: +```dql + { + "set": [ + { + "uid": "0xfffd8d72745f0650", + "specie": "badger" + } + ] + } +``` + +Adds the `specie` information to the node that was created earlier. + + +## Language support + +To set a string value for a specific lnguage, append the language tag to the field name. +In case, `specie` predicate has the @lang directive, the JSON mutation +```dql + { + "set": [ + { + "uid": "_:diggy", + "name": "diggy", + "dgraph.type": "Mascot", + "specie@en" : "badger", + "specie@fr" : "blaireau" + } + ] + } +``` +Dgraph sets the `specie` string predicate in English and in French. + + +## Geolocation support + +Geo-location data must be specified using keys `type` and `coordinates` in the JSON document. +The supported types are `Point`, `Polygon`, or `MultiPolygon` . + +```dql + { + "set": [ + { + "name": "diggy", + "dgraph.type": "Mascot", + "home" : { + "type": "Point", + "coordinates": [-122.475537, 37.769229 ] + } + } + ] + } +``` + + + + + +## Relationships + +Relationships are simply created from the nested structure of JSON. + +For example: +```dql + { + "set": [ + { + "uid": "_:diggy", + "name": "diggy", + "dgraph.type": "Mascot", + "food" : [ + { + "uid":"_:f1", + "name": "earthworms" + }, + { + "uid":"_:f2", + "name": "apples" + }] + } + ] + } + +``` + +This result in the creation of three nodes and the `food` predicate as a relationship. + + +{`{ + "data": { + "code": "Success", + "message": "Done", + "queries": null, + "uids": { + "diggy": "0xfffd8d72745f06d7", + "f1": "0xfffd8d72745f06d8", + "f2": "0xfffd8d72745f06d9" + } + ... +}`} + + + + +You can use references to existing nodes at any level of your nested JSON. + + +## Deleting literal values + +To delete node predicates, specify the UID of the node you are changing and set +the predicates to delete to the JSON value `null`. + +For example, to remove the predicate `name` from node `0xfffd8d72745f0691` : +```dql +{ + "delete": [ + { + "uid": "0xfffd8d72745f0691", + "name": null + } + ] +} +``` + +## Deleting relationship + +A relationship can be defined with a cardinality of 1 or many (list). +Setting a relationship to `null` removes all the relationships. + +```JSON +{ + "uid": "0xfffd8d72745f06d7", + "food": null +} +``` + + +To delete a single relationship in a list, you must specify the target node of the relationship. + +```dql +{ + "delete": [ + { + "uid": "0xfffd8d72745f06d7", + "food": { + "uid": "0xfffd8d72745f06d9" + } + } + ] +} + +``` + +deletes only one `food` relationship. + + +To delete all predicates of a given node: +- make sure the node has a `dgraph.type` predicate +- the type is defined in the [Dgraph types schema](dql-schema) +- run a delete mutation specifying only the uid field + + +```JSON +{ + "delete": [ + { + "uid": "0x123" + } + ] +} +``` +## Handling arrays + +To create a predicate as a list of string: + +```JSON +{ + "set": [ + { + "testList": [ + "Grape", + "Apple", + "Strawberry", + "Banana", + "watermelon" + ] + } + ] +} +``` + +For example, if `0x06` is the UID of the node created. + +To remove one value from the list: + +```JSON +{ + "delete": { + "uid": "0x6", #UID of the list. + "testList": "Apple" + } +} +``` + +To remove multiple multiple values: +```JSON +{ + "delete": { + "uid": "0x6", + "testList": [ + "Strawberry", + "Banana", + "watermelon" + ] + } +} +``` + +To add a value: + +```JSON +{ + "uid": "0x6", #UID of the list. + "testList": "Pineapple" +} +``` + +## Adding Facets + +Facets can be created by using the `|` character to separate the predicate +and facet key in a JSON object field name. This is the same encoding schema +used to show facets in query results. E.g. +```JSON +{ + "name": "Carol", + "name|initial": "C", + "dgraph.type": "Person", + "friend": { + "name": "Daryl", + "friend|close": "yes", + "dgraph.type": "Person" + } +} +``` + +Facets do not contain type information but Dgraph will try to guess a type from +the input. If the value of a facet can be parsed to a number, it will be +converted to either a float or an int. If it can be parsed as a Boolean, it will +be stored as a Boolean. If the value is a string, it will be stored as a +datetime if the string matches one of the time formats that Dgraph recognizes +(YYYY, MM-YYYY, DD-MM-YYYY, RFC339, etc.) and as a double-quoted string +otherwise. If you do not want to risk the chance of your facet data being +misinterpreted as a time value, it is best to store numeric data as either an +int or a float. + +## Deleting Facets + +To delete a `Facet`, overwrite it. When you run a mutation for the same entity without a `Facet`, the existing `Facet` is deleted automatically. + + +## Facets in List +Schema: +```sh +: string @index(exact). +: [string] . +``` +To create a List-type predicate you need to specify all value in a single list. Facets for all +predicate values should be specified together. It is done in map format with index of predicate +values inside list being map key and their respective facets value as map values. Predicate values +which does not have facets values will be missing from facets map. E.g. +```JSON +{ + "set": [ + { + "uid": "_:Julian", + "name": "Julian", + "nickname": ["Jay-Jay", "Jules", "JB"], + "nickname|kind": { + "0": "first", + "1": "official", + "2": "CS-GO" + } + } + ] +} +``` +Above you see that we have three values ​​to enter the list with their respective facets. +You can run this query to check the list with facets: +```graphql +{ + q(func: eq(name,"Julian")) { + uid + nickname @facets + } +} +``` +Later, if you want to add more values ​​with facets, just do the same procedure, but this time instead of using Blank-node you must use the actual node's UID. +```JSON +{ + "set": [ + { + "uid": "0x3", + "nickname|kind": "Internet", + "nickname": "@JJ" + } + ] +} +``` +And the final result is: +```JSON +{ + "data": { + "q": [ + { + "uid": "0x3", + "nickname|kind": { + "0": "first", + "1": "Internet", + "2": "official", + "3": "CS-GO" + }, + "nickname": [ + "Jay-Jay", + "@JJ", + "Jules", + "JB" + ] + } + ] + } +} +``` + +## Reserved values + +The string values `uid(...)`, `val(...)` are not accepted. diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/predicate-indexing.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/predicate-indexing.md new file mode 100644 index 00000000..c2a88bfd --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/predicate-indexing.md @@ -0,0 +1,259 @@ +--- +title: Indexes +--- + +Filtering on a predicate by applying a [function](query/functions) requires an index. + +Indices are defined in the [Dgraph types schema](dql-schema) using `@index` directive. + +Here are some examples: +``` +name: string @index(term) . +release_date: datetime @index(year) . +description_vector: float32vector @index(hnsw(metric:"cosine")) . +``` + +When filtering by applying a function, Dgraph uses the index to make the search through a potentially large dataset efficient. + +All scalar types can be indexed. + +Types `int`, `float`, `bool` and `geo` have only a default index each: with tokenizers named `int`, `float`, `bool` and `geo`. + +Types `string` and `dateTime` have a number of indices. + +Type `float32vector` supports `hnsw` index. + +## String Indices +The indices available for strings are as follows. + +| Dgraph function | Required index / tokenizer | Notes | +| :----------------------- | :------------ | :--- | +| `eq` | `hash`, `exact`, `term`, or `fulltext` | The most performant index for `eq` is `hash`. Only use `term` or `fulltext` if you also require term or full-text search. If you're already using `term`, there is no need to use `hash` or `exact` as well. | +| `le`, `ge`, `lt`, `gt` | `exact` | Allows faster sorting. | +| `allofterms`, `anyofterms` | `term` | Allows searching by a term in a sentence. | +| `alloftext`, `anyoftext` | `fulltext` | Matching with language specific stemming and stopwords. | +| `regexp` | `trigram` | Regular expression matching. Can also be used for equality checking. | +| `ngram` | `ngram` | Contiguous sequence matching (shingles) with stop word removal and stemming. | + + +:::warning +Incorrect index choice can impose performance penalties and an increased +transaction conflict rate. Use only the minimum number of and simplest indexes +that your application needs. +::: + +## Vector Indices + +The indices available for `float32vector` are as follows. + +| Dgraph function | Required index / tokenizer | Notes | +| :----------------------- | :------------ | :--- | +| `similar_to` | `hnsw` | HNSW index supports parameters `metric` and `exponent`. | + + + +`hnsw` (**Hierarchical Navigable Small World**) index supports the following parameters +- metric : indicate the metric to use to compute vector similarity. One of `cosine`, `euclidean`, and `dotproduct`. Default is `euclidean`. + +- exponent : an integer, represented as a string, roughly representing the number of vectors expected in the index in power of 10. The exponent value is used to set "reasonable defaults" for HNSW internal tuning parameters. The default is "3" (10^3 vectors). This is a high-level convenience parameter that acts as a "complexity knob" to scale multiple HNSW parameters simultaneously, namely the `maxlevels`, `efconstruction` and `efsearch`. When you set `exponent: "X"`, Dgraph automatically sets default values for three parameters (only if they're not explicitly specified): + ``` + maxLevels = X - Number of hierarchical layers + efConstruction = 50 × X - Size of dynamic candidate list during index construction + efSearch = 30 × X - Size of dynamic candidate list during search + ``` +- maxLevels : an integer, represented as a string, that controls the maximum number of hierarchical layers in the HNSW. Defaults to the `exponent` parameter. More layers = faster search for large datasets but more memory overhead. Fewer layers = slower search but less memory usage. + +- efConstruction : an integer, represented as a string, that controls the size of the candidate list during index construction - higher values create better quality graphs but take longer to build. Defaults to 50 times the value of `maxLevels`. + +- efSearch : an integer, represented as a string, that controls the size of the candidate list during search queries - higher values improve recall/accuracy but make searches slower. Defaults to 30 times the value of `maxLevels`. + +Here are some examples: +``` +simple_vector: float32vector @index(hnsw) . +description_vector: float32vector @index(hnsw(metric:"cosine")) . +large_vector: float32vector @index(hnsw(metric:"euclidean", exponent:"6")) . +larger_vector: float32vector @index(hnsw(metric:"euclidean", maxLevels: "7", efConstruction: "250", efSearch: "80")) +``` + +## DateTime Indices + +The indices available for `dateTime` are as follows. + +| Index name / Tokenizer | Part of date indexed | +| :----------- | :------------------------------------------------------------------ | +| `year` | index on year (default) | +| `month` | index on year and month | +| `day` | index on year, month and day | +| `hour` | index on year, month, day and hour | + +The choices of `dateTime` index allow selecting the precision of the index. Applications, such as the movies examples in these docs, that require searching over dates but have relatively few nodes per year may prefer the `year` tokenizer; applications that are dependent on fine grained date searches, such as real-time sensor readings, may prefer the `hour` index. + + +All the `dateTime` indices are sortable. + + +## Sortable Indices + +Not all the indices establish a total order among the values that they index. Sortable indices allow inequality functions and sorting. + +* Indexes `int` and `float` are sortable. +* `string` index `exact` is sortable. +* All `dateTime` indices are sortable. + +For example, given an edge `name` of `string` type, to sort by `name` or perform inequality filtering on names, the `exact` index must have been specified. In which case a schema query would return at least the following tokenizers. + +``` +{ + "predicate": "name", + "type": "string", + "index": true, + "tokenizer": [ + "exact" + ] +} +``` + +## Count index + +For predicates with the `@count` Dgraph indexes the number of edges out of each node. This enables fast queries of the form: +``` +{ + q(func: gt(count(pred), threshold)) { + ... + } +} +``` + +## List Type + +Predicate with scalar types can also store a list of values if specified in the schema. The scalar +type needs to be enclosed within `[]` to indicate that its a list type. + +``` +occupations: [string] . +score: [int] . +``` + +* A set operation adds to the list of values. The order of the stored values is non-deterministic. +* A delete operation deletes the value from the list. +* Querying for these predicates would return the list in an array. +* Indexes can be applied on predicates which have a list type and you can use [Functions](query/functions) on them. +* Sorting is not allowed using these predicates. +* These lists are like an unordered set. For example: `["e1", "e1", "e2"]` may get stored as `["e2", "e1"]`, i.e., duplicate values will not be stored and order may not be preserved. + +## Filtering on list + +Dgraph supports filtering based on the list. +Filtering works similarly to how it works on edges and has the same available functions. + +For example, `@filter(eq(occupations, "Teacher"))` at the root of the query or the +parent edge will display all the occupations from a list of each node in an array but +will only include nodes which have `Teacher` as one of the occupations. However, filtering +on value edge is not supported. + +## Reverse Edges + +A graph edge is unidirectional. For node-node edges, sometimes modeling requires reverse edges. If only some subject-predicate-object triples have a reverse, these must be manually added. But if a predicate always has a reverse, Dgraph computes the reverse edges if `@reverse` is specified in the schema. + +The reverse edge of `anEdge` is `~anEdge`. + +For existing data, Dgraph computes all reverse edges. For data added after the schema mutation, Dgraph computes and stores the reverse edge for each added triple. + +``` +type Person { + name +} +type Car { + regnbr + owner +} +owner: uid @reverse . +regnbr: string @index(exact) . +name: string @index(exact) . +``` + +This makes it possible to query Persons and their cars by using: +``` +q(func: type(Person)) { + name + ~owner { regnbr } +} +``` +To get a different key than `~owner` in the result, the query can be written with the wanted label +(`cars` in this case): + +``` +q(func: type(Person)) { + name + cars: ~owner { regnbr } +} +``` + +This also works if there are multiple "owners" of a `car`: +``` +owner [uid] @reverse . +``` + +In both cases the `owner` edge should be set on the `Car`: +``` +_:p1 "Mary" . +_:p1 "Person" . +_:c1 "ABC123" . +_:c1 "Car" . +_:c1 _:p1 . +``` + +## Querying Schema + +A schema query queries for the whole schema: + +``` +schema {} +``` + +:::note Unlike regular queries, the schema query is not surrounded +by curly braces. Also, schema queries and regular queries cannot be combined. +::: + +You can query for particular schema fields in the query body. + +``` +schema { + type + index + reverse + tokenizer + list + count + upsert + lang +} +``` + +You can also query for particular predicates: + +``` +schema(pred: [name, friend]) { + type + index + reverse + tokenizer + list + count + upsert + lang +} +``` + +:::note If ACL is enabled, then the schema query returns only the +predicates for which the logged-in ACL user has read access. ::: + +Types can also be queried. Below are some example queries. + +``` +schema(type: Movie) {} +schema(type: [Person, Animal]) {} +``` + +Note that type queries do not contain anything between the curly braces. The +output will be the entire definition of the requested types. diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/query/aggregation.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/query/aggregation.md new file mode 100644 index 00000000..052c733c --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/query/aggregation.md @@ -0,0 +1,215 @@ +--- +title: Aggregation +--- +import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock'; + + + +Syntax Example: `AG(val(varName))` + +For `AG` replaced with + +* `min` : select the minimum value in the value variable `varName` +* `max` : select the maximum value +* `sum` : sum all values in value variable `varName` +* `avg` : calculate the average of values in `varName` + +Schema Types: + +| Aggregation | Schema Types | +|:-----------|:--------------| +| `min` / `max` | `int`, `float`, `string`, `dateTime`, `default` | +| `sum` / `avg` | `int`, `float` | + +Aggregation can only be applied to [value variables](variables#value-variables). An index is not required (the values have already been found and stored in the value variable mapping). + +An aggregation is applied at the query block enclosing the variable definition. As opposed to query variables and value variables, which are global, aggregation is computed locally. For example: +``` +A as predicateA { + ... + B as predicateB { + x as ...some value... + } + min(val(x)) +} +``` +Here, `A` and `B` are the lists of all UIDs that match these blocks. Value variable `x` is a mapping from UIDs in `B` to values. The aggregation `min(val(x))`, however, is computed for each UID in `A`. That is, it has a semantics of: for each UID in `A`, take the slice of `x` that corresponds to `A`'s outgoing `predicateB` edges and compute the aggregation for those values. + +Aggregations can themselves be assigned to value variables, making a UID to aggregation map. + + +## Min + +### Usage at Root + +Query Example: Get the min initial release date for any Harry Potter movie. + +The release date is assigned to a variable, then it is aggregated and fetched in an empty block. + + +```dql +{ + var(func: allofterms(name@en, "Harry Potter")) { + d as initial_release_date + } + me() { + min(val(d)) + } +} +``` + + + +### Usage at other levels + +Query Example: Directors called Steven and the date of release of their first movie, in ascending order of first movie. + + + +```dql +{ + stevens as var(func: allofterms(name@en, "steven")) { + director.film { + ird as initial_release_date + # ird is a value variable mapping a film UID to its release date + } + minIRD as min(val(ird)) + # minIRD is a value variable mapping a director UID to their first release date + } + + byIRD(func: uid(stevens), orderasc: val(minIRD)) { + name@en + firstRelease: val(minIRD) + } +} +``` + + + +## Max + +### Usage at Root + +Query Example: Get the max initial release date for any Harry Potter movie. + +The release date is assigned to a variable, then it is aggregated and fetched in an empty block. + + +```dql +{ + var(func: allofterms(name@en, "Harry Potter")) { + d as initial_release_date + } + me() { + max(val(d)) + } +} +``` + + + +### Usage at other levels + +Query Example: Quentin Tarantino's movies and date of release of the most recent movie. + + + +```dql +{ + director(func: allofterms(name@en, "Quentin Tarantino")) { + director.film { + name@en + x as initial_release_date + } + max(val(x)) + } +} +``` + + + +## Sum and Avg + +### Usage at Root + +Query Example: Get the sum and average of number of count of movies directed by people who have +Steven or Tom in their name. + + + +```dql +{ + var(func: anyofterms(name@en, "Steven Tom")) { + a as count(director.film) + } + + me() { + avg(val(a)) + sum(val(a)) + } +} +``` + + + +### Usage at other levels + +Query Example: Steven Spielberg's movies, with the number of recorded genres per movie, and the total number of genres and average genres per movie. + + + +```dql +{ + director(func: eq(name@en, "Steven Spielberg")) { + name@en + director.film { + name@en + numGenres : g as count(genre) + } + totalGenres : sum(val(g)) + genresPerMovie : avg(val(g)) + } +} +``` + + + + +## Aggregating Aggregates + +Aggregations can be assigned to value variables, and so these variables can in turn be aggregated. + +Query Example: For each actor in a Peter Jackson film, find the number of roles played in any movie. Sum these to find the total number of roles ever played by all actors in the movie. Then sum the lot to find the total number of roles ever played by actors who have appeared in Peter Jackson movies. Note that this demonstrates how to aggregate aggregates; the answer in this case isn't quite precise though, because actors that have appeared in multiple Peter Jackson movies are counted more than once. + + + +```dql +{ + PJ as var(func:allofterms(name@en, "Peter Jackson")) { + director.film { + starring { # starring an actor + performance.actor { + movies as count(actor.film) + # number of roles for this actor + } + perf_total as sum(val(movies)) + } + movie_total as sum(val(perf_total)) + # total roles for all actors in this movie + } + gt as sum(val(movie_total)) + } + + PJmovies(func: uid(PJ)) { + name@en + director.film (orderdesc: val(movie_total), first: 5) { + name@en + totalRoles : val(movie_total) + } + grandTotal : val(gt) + } +} +``` + + + diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/query/alias.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/query/alias.md new file mode 100644 index 00000000..16eeb898 --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/query/alias.md @@ -0,0 +1,45 @@ +--- +title: Aliases +--- +import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock'; + + + +Syntax Examples: + +* `aliasName : predicate` +* `aliasName : predicate { ... }` +* `aliasName : varName as ...` +* `aliasName : count(predicate)` +* `aliasName : max(val(varName))` + +An alias provides an alternate name in results. Predicates, variables and aggregates can be aliased by prefixing with the alias name and `:`. Aliases do not have to be different to the original predicate name, but, within a block, an alias must be distinct from predicate names and other aliases returned in the same block. Aliases can be used to return the same predicate multiple times within a block. + +Query Example: Directors with `name` matching term `Steven`, their UID, English name, average number of actors per movie, total number of films, and the name of each film in English and French. + + +```dql +{ + ID as var(func: allofterms(name@en, "Steven")) @filter(has(director.film)) { + director.film { + num_actors as count(starring) + } + average as avg(val(num_actors)) + } + + films(func: uid(ID)) { + director_id : uid + english_name : name@en + average_actors : val(average) + num_films : count(director.film) + + films : director.film { + name : name@en + english_name : name@en + french_name : name@fr + } + } +} +``` + + diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/query/count.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/query/count.md new file mode 100644 index 00000000..8dd9175f --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/query/count.md @@ -0,0 +1,73 @@ +--- +title: Count +--- +import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock'; + + + +Syntax Examples: + +* `count(predicate)` +* `count(uid)` + +The form `count(predicate)` counts how many `predicate` edges lead out of a node. + +The form `count(uid)` counts the number of UIDs matched in the enclosing block. + +Query Example: The number of films acted in by each actor with `Orlando` in their name. + + + +```dql +{ + me(func: allofterms(name@en, "Orlando")) @filter(has(actor.film)) { + name@en + count(actor.film) + } +} +``` + + + +Count can be used at root and [aliased](alias). + +Query Example: Count of directors who have directed more than five films. When used at the query root, the [count index](../predicate-indexing#count-index) is required. + + + +```dql +{ + directors(func: gt(count(director.film), 5)) { + totalDirectors : count(uid) + } +} +``` + + + + +Count can be assigned to a [value variable](variables#value-variables). + +Query Example: The actors of Ang Lee's "Eat Drink Man Woman" ordered by the number of movies acted in. + + + +```dql +{ + var(func: allofterms(name@en, "eat drink man woman")) { + starring { + actors as performance.actor { + totalRoles as count(actor.film) + } + } + } + + edmw(func: uid(actors), orderdesc: val(totalRoles)) { + name@en + name@zh + totalRoles : val(totalRoles) + } +} +``` + + diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/query/debug.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/query/debug.md new file mode 100644 index 00000000..1d5c44a7 --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/query/debug.md @@ -0,0 +1,60 @@ +--- +title: Debug +--- + +For the purposes of debugging, you can attach a query parameter `debug=true` to a query. Attaching this parameter lets you retrieve the `uid` attribute for all the entities along with the `server_latency` and `start_ts` information under the `extensions` key of the response. + +- `parsing_ns`: Latency in nanoseconds to parse the query. +- `processing_ns`: Latency in nanoseconds to process the query. +- `encoding_ns`: Latency in nanoseconds to encode the JSON response. +- `start_ts`: The logical start timestamp of the transaction. + +Query with debug as a query parameter +```sh +curl -H "Content-Type: application/dql" http://localhost:8080/query?debug=true -XPOST -d $'{ + tbl(func: allofterms(name@en, "The Big Lebowski")) { + name@en + } +}' | python -m json.tool | less +``` + +Returns `uid` and `server_latency` +``` +{ + "data": { + "tbl": [ + { + "uid": "0x41434", + "name@en": "The Big Lebowski" + }, + { + "uid": "0x145834", + "name@en": "The Big Lebowski 2" + }, + { + "uid": "0x2c8a40", + "name@en": "Jeffrey \"The Big\" Lebowski" + }, + { + "uid": "0x3454c4", + "name@en": "The Big Lebowski" + } + ], + "extensions": { + "server_latency": { + "parsing_ns": 18559, + "processing_ns": 802990982, + "encoding_ns": 1177565 + }, + "txn": { + "start_ts": 40010 + } + } + } +} +``` +:::note +GraphQL+- has been renamed to Dgraph Query Language (DQL). While `application/dql` +is the preferred value for the `Content-Type` header, we will continue to support +`Content-Type: application/graphql+-` to avoid making breaking changes. +::: diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/query/directive/cascade-directive.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/query/directive/cascade-directive.md new file mode 100644 index 00000000..2b5213db --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/query/directive/cascade-directive.md @@ -0,0 +1,280 @@ +--- +title: "@cascade" +--- +import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock'; + + + +With the `@cascade` directive, nodes that don't have all predicates specified in the query are removed. This can be useful in cases where some filter was applied or if nodes might not have all listed predicates. + + +Query Example: Harry Potter movies, with each actor and characters played. With `@cascade`, any character not played by an actor called Warwick is removed, as is any Harry Potter movie without any actors called Warwick. Without `@cascade`, every character is returned, but only those played by actors called Warwick also have the actor name. + + +```dql +{ + HP(func: allofterms(name@en, "Harry Potter")) @cascade { + name@en + starring{ + performance.character { + name@en + } + performance.actor @filter(allofterms(name@en, "Warwick")){ + name@en + } + } + } +} +``` + + + +You can apply `@cascade` on inner query blocks as well. + + +```dql +{ + HP(func: allofterms(name@en, "Harry Potter")) { + name@en + genre { + name@en + } + starring @cascade { + performance.character { + name@en + } + performance.actor @filter(allofterms(name@en, "Warwick")){ + name@en + } + } + } +} +``` + + + +## Parameterized `@cascade` + +The `@cascade` directive can optionally take a list of fields as an argument. +This changes the default behavior, considering only the supplied fields as mandatory instead of all the fields for a type. +Listed fields are automatically cascaded as a required argument to nested selection sets. +A parameterized cascade works on levels (e.g. on the root function or on lower levels), so +you need to specify `@cascade(param)` on the exact level where you want it to be applied. + +:::tip +The rule with `@cascade(predicate)` is that the predicate needs to be in the query at the same level `@cascade` is. +::: + +Take the following query as an example: + + + +```dql +{ + nodes(func: allofterms(name@en, "jones indiana")) { + name@en + genre @filter(anyofterms(name@en, "action adventure")) { + name@en + } + produced_by { + name@en + } + } +} +``` + + + +This query gets nodes that have all the terms _"jones indiana"_ and then traverses to `genre` and `produced_by`. +It also adds an additional filter for `genre`, to only get the ones that either have _"action"_ or _"adventure"_ in the name. +The results include nodes that have no `genre` and nodes that have no `genre` and no `producer`. + +If you apply a regular `@cascade` without a parameter, you'll lose the ones that had `genre` but no `producer`. + +To get the nodes that have the traversed `genre` but possibly not `produced_by`, you can parameterize the cascade: + + + +```dql +{ + nodes(func: allofterms(name@en, "jones indiana")) @cascade(genre) { + name@en + genre @filter(anyofterms(name@en, "action adventure")) { + name@en + } + produced_by { + name@en + } + written_by { + name@en + } + } +} +``` + + + +If you want to check for multiple fields, just comma separate them. For example, to cascade over `produced_by` and `written_by`: + + + +```dql +{ + nodes(func: allofterms(name@en, "jones indiana")) @cascade(produced_by,written_by) { + name@en + genre @filter(anyofterms(name@en, "action adventure")) { + name@en + } + produced_by { + name@en + } + written_by { + name@en + } + } +} +``` + + + +### Nesting and parameterized cascade + +The cascading nature of field selection is overwritten by a nested `@cascade`. + +The previous example can be cascaded down the chain as well, and be overridden on each level as needed. + +For example, if you only want the _"Indiana Jones movies that were produced by the same person who produced a Jurassic World movie"_: + + + +```dql +{ + nodes(func: allofterms(name@en, "jones indiana")) @cascade(produced_by) { + name@en + genre @filter(anyofterms(name@en, "action adventure")) { + name@en + } + produced_by @cascade(producer.film) { + name@en + producer.film @filter(allofterms(name@en, "jurassic world")) { + name@en + } + } + written_by { + name@en + } + } +} +``` + + + +Another nested example: _"Find the Indiana Jones movie that was written by the same person who wrote a Star Wars movie and was produced by the same person who produced Jurassic World"_: + + + +```dql +{ + nodes(func: allofterms(name@en, "jones indiana")) @cascade(produced_by,written_by) { + name@en + genre @filter(anyofterms(name@en, "action adventure")) { + name@en + } + produced_by @cascade(producer.film) { + name@en + producer.film @filter(allofterms(name@en, "jurassic world")) { + name@en + } + } + written_by @cascade(writer.film) { + name@en + writer.film @filter(allofterms(name@en, "star wars")) { + name@en + } + } + } +} +``` + + + +## Cascade Performance + +The `@cascade` directive processes the nodes after the query, but before Dgraph +returns query results. This means that all of the nodes that would normally be +returned if there was no `@cascade` applied are still touched in the internal +query process. If you see slower-than-expected performance when using the +`@cascade` directive, it is probably because the internal query process returns +a large set of nodes but the cascade reduces those to a small set of nodes in query +results. To improve the performance of queries that use the `@cascade` directive, +you might want to use `var` blocks or `has` filters, as described below. + +### Cascade with `var` blocks + +The performance impact of using `var` blocks is that it reduces the graph that is touched to generate the final query results. +For example, many of the previous examples could be replaced entirely using [`var` blocks](/dql/query/dql-query#var-block) instead of utilizing `@cascade`. + +The following query provides an alternative way to structure the query shown above, +_"Find the Indiana Jones movie that was written by the same person who wrote a +Star Wars movie and was produced by the same person who produced Jurassic World"_, +without using the `@cascade` directive: + + + +```dql +{ + var(func: allofterms(name@en, "jurassic world")) { + produced_by { + ProducedBy as producer.film + } + } + var(func: allofterms(name@en, "star wars")) { + written_by { + WrittenBy as writer.film + } + } + nodes(func: allofterms(name@en,"indiana jones")) @filter(uid(ProducedBy) AND uid(WrittenBy)) { + name@en + genre { + name@en + } + } +} +``` + + + +The performance impact of building queries with multiple `var` blocks versus +using `@cascade` depends on the nodes touched to reach the end results. Depending +on the size of your data set and distribution between nodes, refactoring a query +with `var` blocks instead of `@cascade` might actually decrease performance +if the query must touch more nodes as a result of the refactor. + +### Cascade with `has` filter + +In cases where only a small set of nodes have the predicates where `@cascade` is +applied, it might be beneficial to query performance to include a `has` filter +for those predicates. + +For example, you could run a query like _"Find movies that have a sequel whose name contains the term **Star Wars**"_ as follows: + + + +```dql +{ + nodes(func: has(sequel)) @filter(type(Film)) @cascade { + count(uid) + name@en + sequel @filter(allofterms(name@en,"Star Wars")) { + name@en + } + } +} +``` + + + +By using a `has` filter in the root function instead of `type(Movie)`, you can +reduce the root graph from `275,195` nodes down to `7,747` nodes. Reducing the +root graph before the post-query cascade process results in a higher-performing +query. diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/query/directive/filter.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/query/directive/filter.md new file mode 100644 index 00000000..5e91b23d --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/query/directive/filter.md @@ -0,0 +1,91 @@ +--- +title: "@filter" +--- +import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock'; + + + +The `@filter` directive allows you to apply additional filtering conditions to nodes in a query block. Filters use [functions](../functions) to test node attributes or relationships and can be applied to both root query blocks and nested blocks. + +## Using @filter + +### In Query Blocks + +A query block may have a combination of filters to apply to the root nodes. The `@filter` directive appears after the `func:` criteria and before the opening curly bracket: + +```graphql +{ + me(func: eq(name@en, "Steven Spielberg")) @filter(has(director.film)) { + name@en + director.film { + name@en + } + } +} +``` + +### In Nested Blocks + +For relationships to fetch, nested blocks may specify filters to apply on the related nodes: + +```graphql +{ + director(func: eq(name@en, "Steven Spielberg")) { + name@en + director.film @filter(allofterms(name@en, "indiana jones")) { + uid + name@en + } + } +} +``` + +Nested blocks may also specify criteria on the relationships attributes using [filtering on facets](../facets#filtering-on-facets). + +## Filter Functions + +Filters use the same [functions](../functions) that are available for root criteria. These functions can test: + +- **String attributes**: term matching, regular expressions, fuzzy matching, full-text search +- **Attribute values**: equality, inequalities, ranges +- **Node properties**: predicate existence, UID, relationships, node type +- **Relationship counts**: equality and inequality comparisons +- **Geolocation attributes**: proximity, containment, intersection + +Common functions include: + +- String matching: [allofterms](../functions#allofterms), [anyofterms](../functions#anyofterms), [regexp](../functions#regular-expressions), [match](../functions#fuzzy-matching), [alloftext](../functions#full-text-search) +- Value comparisons: [eq](../functions#equal-to), [le, lt, ge, gt](../functions#less-than-less-than-or-equal-to-greater-than-and-greater-than-or-equal-to), [between](../functions#between) +- Node tests: [has](../functions#has), [uid](../functions#uid), [uid_in](../functions#uid_in), `type()` +- Geolocation: [near](../functions#near), [within](../functions#within), [contains](../functions#contains), [intersects](../functions#intersects) + +Variables may be used as function parameters in filters. See [query variables](../variables#query-variables) and [value variables](../variables#value-variables) for more information. + +Filters can also be combined with directives like [@cascade](cascade-directive) to create pattern matching queries where only nodes matching the complete query structure are returned. + +## Connecting Filters + +Within `@filter` multiple functions can be used with boolean operators AND, OR, and NOT. + +### AND, OR and NOT + +Connectives `AND`, `OR` and `NOT` join filters and can be built into arbitrarily complex filters, such as `(NOT A OR B) AND (C AND NOT (D OR E))`. Note that, `NOT` binds more tightly than `AND` which binds more tightly than `OR`. + +Query Example: All Steven Spielberg movies that contain either both "indiana" and "jones" OR both "jurassic" and "park". + + + +```dql +{ + me(func: eq(name@en, "Steven Spielberg")) @filter(has(director.film)) { + name@en + director.film @filter(allofterms(name@en, "jones indiana") OR allofterms(name@en, "jurassic park")) { + uid + name@en + } + } +} +``` + + + diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/query/directive/groupby.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/query/directive/groupby.md new file mode 100644 index 00000000..cf6d6089 --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/query/directive/groupby.md @@ -0,0 +1,64 @@ +--- +title: "@groupby" +--- +import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock'; + + + +Syntax Examples: + +* `q(func: ...) @groupby(predicate) { min(...) }` +* `predicate @groupby(pred) { count(uid) }` + + +A `groupby` query aggregates query results given a set of properties on which to group elements. For example, a query containing the block `friend @groupby(age) { count(uid) }`, finds all nodes reachable along the friend edge, partitions these into groups based on age, then counts how many nodes are in each group. The returned result is the grouped edges and the aggregations. + +Inside a `groupby` block, only aggregations are allowed and `count` may only be applied to `uid`. + +If the `groupby` is applied to a `uid` predicate, the resulting aggregations can be saved in a variable (mapping the grouped UIDs to aggregate values) and used elsewhere in the query to extract information other than the grouped or aggregated edges. + +Query Example: For Steven Spielberg movies, count the number of movies in each genre and for each of those genres return the genre name and the count. The name can't be extracted in the `groupby` because it is not an aggregate, but `uid(a)` can be used to extract the UIDs from the UID to value map and thus organize the `byGenre` query by genre UID. + + + + +```dql +{ + var(func:allofterms(name@en, "steven spielberg")) { + director.film @groupby(genre) { + a as count(uid) + # a is a genre UID to count value variable + } + } + + byGenre(func: uid(a), orderdesc: val(a)) { + name@en + total_movies : val(a) + } +} +``` + + + +Query Example: Actors from Tim Burton movies and how many roles they have played in Tim Burton movies. + + +```dql +{ + var(func:allofterms(name@en, "Tim Burton")) { + director.film { + starring @groupby(performance.actor) { + a as count(uid) + # a is an actor UID to count value variable + } + } + } + + byActor(func: uid(a), orderdesc: val(a)) { + name@en + val(a) + } +} +``` + + diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/query/directive/ignorereflex-directive.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/query/directive/ignorereflex-directive.md new file mode 100644 index 00000000..c9ac7a83 --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/query/directive/ignorereflex-directive.md @@ -0,0 +1,30 @@ +--- +title: "@ignorereflex" +--- +import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock'; + + + +The `@ignorereflex` directive forces the removal of child nodes that are reachable from themselves as a parent, through any path in the query result + +Query Example: All the co-actors of Rutger Hauer. Without `@ignorereflex`, the result would also include Rutger Hauer for every movie. + + + +```dql +{ + coactors(func: eq(name@en, "Rutger Hauer")) @ignorereflex { + actor.film { + performance.film { + starring { + performance.actor { + name@en + } + } + } + } + } +} +``` + + \ No newline at end of file diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/query/directive/index.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/query/directive/index.md new file mode 100644 index 00000000..456e4c06 --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/query/directive/index.md @@ -0,0 +1,30 @@ +--- +title: Directives +--- + +Directives in Dgraph Query Language (DQL) are special annotations that modify how queries are executed or how results are formatted. They are prefixed with the `@` symbol and can be applied to query blocks or predicates to change their behavior. + +Directives provide powerful control over: + +- **Filtering**: Apply conditions to filter nodes in query results +- **Response structure**: Format and organize query results +- **Graph traversal**: Control how the graph is explored +- **Pattern matching**: Filter results based on complete query structure +- **Aggregation**: Group and aggregate data + +## Available Directives + +- **[@filter](/dql/query/directive/filter)**: Applies additional filtering conditions to nodes in query blocks using functions and boolean operators. + +- **[@normalize](/dql/query/directive/normalize-directive)**: Flattens the response structure by removing nesting and returning only aliased predicates. + +- **[@cascade](/dql/query/directive/cascade-directive)**: Filters out nodes that don't match all predicates specified in the query at any nested level, enabling pattern matching behavior. + +- **[@recurse](/dql/query/directive/recurse-query)**: Performs recursive graph traversal, following relationships to explore paths of variable depth. + +- **[@ignorereflex](/dql/query/directive/ignorereflex-directive)**: Ignores reflexive edges (edges that point back to the same node) during graph traversal. + +- **[@groupby](/dql/query/directive/groupby)**: Groups query results based on specified predicates and allows aggregation functions to be applied to each group. + +Directives can be combined in a single query to achieve complex querying and result formatting requirements. + diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/query/directive/normalize-directive.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/query/directive/normalize-directive.md new file mode 100644 index 00000000..9e404123 --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/query/directive/normalize-directive.md @@ -0,0 +1,64 @@ +--- +title: "@normalize" +--- +import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock'; + + + +With the `@normalize` directive, only aliased predicates are returned and the result is flattened to remove nesting. + +Query Example: Film name, country and first two actors (by UID order) of every Steven Spielberg movie, without `initial_release_date` because no alias is given and flattened by `@normalize` + + +```dql +{ + director(func:allofterms(name@en, "steven spielberg")) @normalize { + director: name@en + director.film { + film: name@en + initial_release_date + starring(first: 2) { + performance.actor { + actor: name@en + } + performance.character { + character: name@en + } + } + country { + country: name@en + } + } + } +} +``` + + + +You can also apply `@normalize` on nested query blocks. It will work similarly but only flatten the result of the nested query block where `@normalize` has been applied. `@normalize` will return a list irrespective of the type of attribute on which it is applied. + + +```dql +{ + director(func:allofterms(name@en, "steven spielberg")) { + director: name@en + director.film { + film: name@en + initial_release_date + starring(first: 2) @normalize { + performance.actor { + actor: name@en + } + performance.character { + character: name@en + } + } + country { + country: name@en + } + } + } +} +``` + + \ No newline at end of file diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/query/directive/recurse-query.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/query/directive/recurse-query.md new file mode 100644 index 00000000..7deef23a --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/query/directive/recurse-query.md @@ -0,0 +1,34 @@ +--- +title: "@recurse" +--- +import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock'; + + + +`Recurse` queries let you traverse a set of predicates (with filter, facets, etc.) until we reach all leaf nodes or we reach the maximum depth which is specified by the `depth` parameter. + +To get 10 movies from a genre that has more than 30000 films and then get two actors for those movies we'd do something as follows: + + +```dql +{ + me(func: gt(count(~genre), 30000), first: 1) @recurse(depth: 5, loop: true) { + name@en + ~genre (first:10) @filter(gt(count(starring), 2)) + starring (first: 2) + performance.actor + } +} +``` + + +Some points to keep in mind while using recurse queries are: + +- You can specify only one level of predicates after root. These would be traversed recursively. Both scalar and entity-nodes are treated similarly. +- Only one recurse block is advised per query. +- Be careful as the result size could explode quickly and an error would be returned if the result set gets too large. In such cases use more filters, limit results using pagination, or provide a depth parameter at root as shown in the example above. +- The `loop` parameter can be set to false, in which case paths which lead to a loop would be ignored + while traversing. +- If not specified, the value of the `loop` parameter defaults to false. +- If the value of the `loop` parameter is false and depth is not specified, `depth` will default to `math.MaxUint64`, which means that the entire graph might be traversed until all the leaf nodes are reached. + diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/query/dql-query.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/query/dql-query.md new file mode 100644 index 00000000..f93f8884 --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/query/dql-query.md @@ -0,0 +1,330 @@ +--- +title: Query Structure +--- +import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock'; + + + +Fetching data with Dgraph Query Language (DQL), is done through **DQL Queries**. Adding, modifying or deleting data is done through [DQL Mutations](/dql/dql-mutation). + +This overview explains the structure of DQL Queries and provides links to the appropriate DQL reference documentation. + +### DQL query structure +DQL is **declarative**, which means that queries return a response back in a similar shape to the query. It gives the client application the control of what it gets: the request return exactly what you ask for, nothing less and nothing more. In this, DQL is similar to GraphQL from which it is inspired. + +A DQL query finds nodes based on search criteria, matches patterns in the graph and returns the node attributes, relationships specified in the query. + +A DQL query has +- an optional parameterization, ie a name and a list of parameters +- an opening curly bracket +- at least one [query block](#query-block), but can contain many blocks +- optional var blocks +- a closing curly bracket + +![DQL Query with parameterization](/images/dql-syntax/query-syntax-1.png) + +### Read-Only Transactions + +Read-only transactions can be created by calling `c.NewReadOnlyTxn()`. Read-only +transactions are useful to increase read speed because they can circumvent the +usual consensus protocol. Read-only transactions cannot contain mutations and +trying to call `txn.Commit()` will result in an error. Calling `txn.Discard()` +will be a no-op. + +Read-only queries can optionally be set as best-effort. Using this flag will ask +the Dgraph Alpha to try to get timestamps from memory on a best-effort basis to +reduce the number of outbound requests to Zero. This may yield improved +latencies in read-bound workloads where linearizable reads are not strictly +needed. + +### Query parameterization +**Parameters** +* must have a name starting with a `$` symbol. +* must have a type `int`, `float`, `bool` or `string`. +* may have a default value. In the example below, `$age` has a default value of `95` +* may be mandatory by suffixing the type with a `!`. Mandatory parameters can't have a default value. + +Parameters can be used in the query where a string, float, int or bool value are needed. + +You can also use a variable holding ``uids`` by using a string variable and by providing the value as a quoted list in square brackets: +`query title($uidsParam: string = "[0x1, 0x2, 0x3]") { ... }`. + + + +**Error handling** +When submitting a query using parameters, Dgraph responds with errors if +* A parameter value is not parsable to the given type. +* The query is using a parameter that is not declared. +* A mandatory parameter is not provided + + +The query parameterization is optional. If you don't use parameters you can omit it and send only the query blocks. + +![DQL Query without parameters](/images/dql-syntax/query-syntax-2.png) +:::noteThe current documentation is usually using example of queries without parameters. ::: + +If you execute this query in our [Movies demo database](/dql/query/running-examples) you can see that Dgraph will return a JSON structure similar to the request : +![DQL response structure](/images/dql-syntax/query-syntax-3.png) + +### Query block + +A query block specifies information to retrieve from Dgraph. + +A query block +- must have name +- must have a node criteria defined by the keyword ``func:`` +- may have ordering and pagination information +- may have a combination of filters (to apply to the root nodes) +- must provide the list of attributes and relationships to fetch for each node matching the root nodes. + +Refer to [pagination](pagination), [ordering](sorting), [connecting filters](directive/filter#connecting-filters) for more information. + +For each relationships to fetch, the query is using a nested block. + +A nested block +- may specify filters to apply on the related nodes +- may specify criteria on the relationships attributes using [filtering on facets](facets#filtering-on-facets)) +- provides the list of relationship attributes ([facets](facets))) to fetch. +- provides the list of attributes and relationships to fetch for the related nodes. + +A nested block may contain another nested block, and such at any level. + +### Multiple query blocks +Inside a single query, multiple query blocks are allowed, and each block can +have a name. Multiple query blocks are executed in parallel, and they don't +need to be related in any way. + +Query Example: _"All of Angelina Jolie's films, with genres, and Peter Jackson's films since 2008"_ + + + +```dql +{ + AngelinaInfo(func:allofterms(name@en, "angelina jolie")) { + name@en + actor.film { + performance.film { + genre { + name@en + } + } + } + } + + DirectorInfo(func: eq(name@en, "Peter Jackson")) { + name@en + director.film @filter(ge(initial_release_date, "2008")) { + Release_date: initial_release_date + Name: name@en + } + } +} +``` + + + + +If queries contain some overlap in answers, the result sets are still independent. + +Query Example: _"The movies Mackenzie Crook has acted in and the movies Jack Davenport has acted in"_ + +The results sets overlap because both have acted in the _Pirates of the Caribbean_ +movies, but the results are independent and both contain the full answers sets. + + + +```dql +{ + Mackenzie(func:allofterms(name@en, "Mackenzie Crook")) { + name@en + actor.film { + performance.film { + uid + name@en + } + performance.character { + name@en + } + } + } + + Jack(func:allofterms(name@en, "Jack Davenport")) { + name@en + actor.film { + performance.film { + uid + name@en + } + performance.character { + name@en + } + } + } +} +``` + + + +### Escape characters in predicate names + If your predicate has special characters, wrap it with angular brackets `< >` in the query. + + E.g. + ` + ` + +### Formatting options +Dgraph returns the attributes and relationships that you specified in the query. You can specify an alternate name for the result by using [aliases](alias). + +You can flatten the response structure at any level using [@normalize](directive/normalize-directive) directive. + +Entering the list of all the attributes you want to fetch could be fastidious for large queries or repeating blocks : you may take advantage of [fragments](fragments) and the [expand function](expand-predicates). + +### Node criteria (used by root function or by filter) + +Root criteria and filters are using [functions](functions) applied to nodes attributes or variables. + +Dgraph offers functions for +- testing string attributes + - term matching : [allofterms](functions#allofterms), [anyofterms](functions#anyofterms) + - regular Expression : [regexp](functions#regular-expressions) + - fuzzy match : [match](functions#fuzzy-matching) + - full-text search : [alloftext](functions#full-text-search) +- testing attribute value + - equality : [eq](functions#equal-to) + - inequalities : [le,lt,ge,gt](functions#less-than-less-than-or-equal-to-greater-than-and-greater-than-or-equal-to) + - range : [between](functions#between) +- testing if a node + - has a particular predicate (an attribute or a relation) : [has](functions#has) + - has a given UID : [uid](functions#uid) + - has a relationship to a given node : [uid_in](functions#uid_in) + - is of a given type : type() +- testing the number of node relationships + - equality : [eq](functions#equal-to) + - inequalities : [le,lt,ge,gt](functions#less-than-less-than-or-equal-to-greater-than-and-greater-than-or-equal-to) +- testing geolocation attributes + - if geo location is within distance : [near](functions#near) + - if geo location lies within a given area : [within](functions#within) + - if geo area contains a given location : [contains](functions#contains) + - if geo area intersects a given are : [intersects](functions#intersects) + + +### Var block + + Variable blocks (`var` blocks) start with the keyword `var` instead of a block name. + + var blocks are not reflected in the query result. They are used to compute [query-variables](variables#query-variables) which are lists of node UIDs, or [value-variables](variables#value-variables) which are maps from node UIDs to the corresponding scalar values. + + Note that query-variables and value-variables can also be computed in query blocks. In that case, the query block is used to fetch and return data, and to define some variables which must be used in other blocks of the same query. + + Variables may be used as functions parameters in filters or root criteria in other blocks. + + Query Example: _"Angelina Jolie's movies ordered by genre"_ + + + +```dql +{ + var(func:allofterms(name@en, "angelina jolie")) { + name@en + actor.film { + A AS performance.film { + B AS genre + } + } + } + + films(func: uid(B), orderasc: name@en) { + name@en + ~genre @filter(uid(A)) { + name@en + } + } +} +``` + + + +## Multiple `var` blocks + +You can also use multiple `var` blocks within a single query operation. You can +use variables from one `var` block in any of the subsequent blocks, but not +within the same block. + +Query Example: _"Movies containing both Angelina Jolie and Morgan Freeman sorted by name"_ + + + +```dql +{ + var(func:allofterms(name@en, "angelina jolie")) { + name@en + actor.film { + A AS performance.film + } + } + var(func:allofterms(name@en, "morgan freeman")) { + name@en + actor.film { + B as performance.film @filter(uid(A)) + } + } + + films(func: uid(B), orderasc: name@en) { + name@en + } +} +``` + + + + +### Combining multiple `var` blocks + +You could get the same query results by logically combining both both `var` blocks +in the films block, as follows: +``` +{ + var(func:allofterms(name@en, "angelina jolie")) { + name@en + actor.film { + A AS performance.film + } + } + var(func:allofterms(name@en, "morgan freeman")) { + name@en + actor.film { + B as performance.film + } + } + films(func: uid(A,B), orderasc: name@en) @filter(uid(A) AND uid(B)) { + name@en + } +} +``` +The root `uid` function unions the `uid`s from `var` `A` and `B`, so you need a +filter to intersect the `uid`s from `var` `A` and `B`. + +### Summarizing functions + +When dealing with array attributes or with relationships to many node, the query may use summary functions [count](count) , [min](aggregation#min), [max](aggregation#max), [avg](aggregation#sum-and-avg) or [sum](aggregation#sum-and-avg). + +The query may also contain [mathematical functions](variables#math-on-value-variables) on value variables. + +Summary functions can be used in conjunction with [@grouby](directive/groupby) directive to create aggregated value variables. + +The query may contain **anonymous block** to return computed values. **Anonymous block** don't have a root criteria as they are not used to search for nodes but only to returned computed values. + +### Graph traversal + +When you specify nested blocks and filters you basically describe a way to traverse the graph. + +[@recurse](directive/recurse-query) and [@ignorereflex](directive/ignorereflex-directive) are directives used to optionally configure the graph traversal. + +### Pattern matching +Queries with nested blocks with filters may be turned into pattern matching using [@cascade](directive/cascade-directive) directive : nodes that don’t have all attributes and all relationships specified in the query at any sub level are not considered in the result. So only nodes "matching" the complete query structure are returned. + +### Graph algorithms +The query can ask for the shortest path between a source (from) node and destination (to) node using the [shortest](kshortest-path-queries) query block. + +### Comments +Anything on a line following a `#` is a comment diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/query/expand-predicates.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/query/expand-predicates.md new file mode 100644 index 00000000..67be6e69 --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/query/expand-predicates.md @@ -0,0 +1,87 @@ +--- +title: Expand Predicates +--- +import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock'; + + + +The `expand()` function can be used to expand the predicates out of a node. To +use `expand()`, the [type system](/dql/dql-schema) is required. +Refer to the section on the type system to check how to set the types +nodes. The rest of this section assumes familiarity with that section. + +There are two ways to use the `expand` function. + +* Types can be passed to `expand()` to expand all the predicates in the type. + +Query example: List the movies from the Harry Potter series: + + + +```dql +{ + all(func: eq(name@en, "Harry Potter")) @filter(type(Series)) { + name@en + expand(Series) { + name@en + expand(Film) + } + } +} +``` + + + +* If `_all_` is passed as an argument to `expand()`, the predicates to be +expanded will be the union of fields in the types assigned to a given node. + +The `_all_` keyword requires that the nodes have types. Dgraph will look for all +the types that have been assigned to a node, query the types to check which +attributes they have, and use those to compute the list of predicates to expand. + +For example, consider a node that has types `Animal` and `Pet`, which have +the following definitions: + +``` +type Animal { + name + species + dob +} + +type Pet { + owner + veterinarian +} +``` + +When `expand(_all_)` is called on this node, Dgraph will first check which types +the node has (`Animal` and `Pet`). Then it will get the definitions of `Animal` +and `Pet` and build a list of predicates from their type definitions. + +``` +name +species +dob +owner +veterinarian +``` + +:::note +For `string` predicates, `expand` only returns values not tagged with a language +(see [language preference](/dql/query/language-support)). So it's often +required to add `name@fr` or `name@.` as well to an expand query. +::: + +## Filtering during expand + +Expand queries support filters on the type of the outgoing edge. For example, +`expand(_all_) @filter(type(Person))` will expand on all the predicates but will +only include edges whose destination node is of type Person. Since only nodes of +type `uid` can have a type, this query will filter out any scalar values. + +Please note that other type of filters and directives are not currently supported +with the expand function. The filter needs to use the `type` function for the +filter to be allowed. Logical `AND` and `OR` operations are allowed. For +example, `expand(_all_) @filter(type(Person) OR type(Animal))` will only expand +the edges that point to nodes of either type. diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/query/facets.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/query/facets.md new file mode 100644 index 00000000..c758b21e --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/query/facets.md @@ -0,0 +1,284 @@ +--- +title: Querying Facets +--- + +This page covers how to query and filter using facets in DQL. For an introduction to what facets are and how to create them, see [Facets in Schema](../dql-schema#facets-edge-attributes). + +## Sample Data + +The examples on this page use this data: + +```rdf +# Schema +name: string @index(exact, term) . +rated: [uid] @reverse @count . + +# Data +_:alice "Alice" . +_:alice "Person" . +_:alice "040123456" (since=2006-01-02T15:04:05) . +_:alice "MA0123" (since=2006-02-02T13:01:09, first=true) . + +_:bob "Bob" . +_:bob "Person" . +_:bob "MA0134" (since=2006-02-02T13:01:09) . + +_:charlie "Charlie" . +_:charlie "Person" . + +_:alice _:bob (close=true, relative=false) . +_:alice _:charlie (close=false, relative=true) . +_:alice _:dave (close=true, relative=true) . + +_:movie1 "Movie 1" . +_:movie1 "Movie" . + +_:alice _:movie1 (rating=3) . +_:bob _:movie1 (rating=5) . +_:charlie _:movie1 (rating=2) . +``` + +## Querying Facets + +### Query Specific Facets + +Use `@facets(facet-name)` to retrieve specific facet values: + +```dql +{ + data(func: eq(name, "Alice")) { + name + mobile @facets(since) + car @facets(since) + } +} +``` + +Facets appear in the response at the same level as the edge, with keys like `edge|facet`. + +### Query All Facets + +Use `@facets` without arguments to retrieve all facets on an edge: + +```dql +{ + data(func: eq(name, "Alice")) { + name + mobile @facets + car @facets + } +} +``` + +### Facets on UID Predicates + +For relationship edges, facets appear with the child node: + +```dql +{ + data(func: eq(name, "Alice")) { + name + friend @facets(close) { + name + } + } +} +``` + +The `close` facet appears with key `friend|close` alongside each friend's data. + +### Using Aliases + +Assign custom names to facet results: + +```dql +{ + data(func: eq(name, "Alice")) { + name + car @facets(car_since: since) + friend @facets(close_friend: close) { + name + } + } +} +``` + +:::note +`orderasc` and `orderdesc` are reserved and cannot be used as aliases. +::: + +## Filtering on Facets + +Filter edges based on facet values using `@facets(condition)`: + +### Basic Filter + +```dql +{ + data(func: eq(name, "Alice")) { + friend @facets(eq(close, true)) { + name + } + } +} +``` + +### Filter and Return Facets + +Combine filtering with facet retrieval: + +```dql +{ + data(func: eq(name, "Alice")) { + friend @facets(eq(close, true)) @facets(relative) { + name + } + } +} +``` + +### Compound Filters + +Use `AND`, `OR`, and `NOT` to combine conditions: + +```dql +{ + data(func: eq(name, "Alice")) { + friend @facets(eq(close, true) AND eq(relative, true)) { + name + } + } +} +``` + +## Sorting by Facets + +Sort results by facet values on UID edges: + +```dql +{ + me(func: anyofterms(name, "Alice Bob Charlie")) { + name + rated @facets(orderdesc: rating) { + name + } + } +} +``` + +## Facets with Variables + +### Assign Facets to Variables + +Store facet values in variables for use elsewhere in the query: + +```dql +{ + var(func: eq(name, "Alice")) { + friend @facets(a as close, b as relative) + } + + close_friends(func: uid(a)) { + name + val(a) + } + + relatives(func: uid(b)) { + name + val(b) + } +} +``` + +### Variable Propagation + +Numeric facets (`int`, `float`) propagate through queries. When multiple paths reach the same node, values are summed: + +```dql +{ + var(func: anyofterms(name, "Alice Bob Charlie")) { + num_raters as math(1) + rated @facets(r as rating) { + total_rating as math(r) + average_rating as math(total_rating / num_raters) + } + } + + data(func: uid(total_rating)) { + name + val(total_rating) + val(average_rating) + } +} +``` + +## Aggregating Facets + +Facet values in variables can be aggregated: + +```dql +{ + data(func: eq(name, "Alice")) { + name + rated @facets(r as rating) { + name + } + avg(val(r)) + } +} +``` + +:::warning +When a query reaches nodes through multiple paths, facet values are summed. This affects aggregations: + +```dql +# This does NOT calculate individual averages correctly +{ + data(func: anyofterms(name, "Alice Bob")) { + name + rated @facets(r as rating) { + name + } + avg(val(r)) # Incorrect: r contains summed values + } +} +``` + +Calculate per-user averages with separate variable mappings: + +```dql +{ + var(func: has(rated)) { + num_rated as math(1) + rated @facets(r as rating) { + avg_rating as math(r / num_rated) + } + } + + data(func: uid(avg_rating)) { + name + val(avg_rating) + } +} +``` +::: + +## Internationalized Facet Keys + +Facet keys can use language-specific characters. When querying, enclose them in angle brackets: + +**Mutation:** +```rdf +_:person1 "Daniel" (वंश="स्पेनी", ancestry="Español") . +``` + +**Query:** +```dql +{ + q(func: has(name)) { + name @facets(<वंश>) + } +} +``` + +See [Predicates i18n](../dql-schema#predicates-i18n) for more on internationalization. diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/query/fragments.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/query/fragments.md new file mode 100644 index 00000000..3fc1ed80 --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/query/fragments.md @@ -0,0 +1,32 @@ +--- +title: Fragments +--- + +The `fragment` keyword lets you define new fragments that can be referenced +in a query, per the [Fragments section of the GraphQL specification](http://spec.graphql.org/June2018/#sec-Language.Fragments). +Fragments allow for the reuse of common repeated selections of fields, reducing +duplicated text in the DQL documents. Fragments can be nested inside fragments, +but no cycles are allowed in such cases. For example: + +```sh +curl -H "Content-Type: application/dql" localhost:8080/query -XPOST -d $' +query { + debug(func: uid(1)) { + name@en + ...TestFrag + } +} +fragment TestFrag { + initial_release_date + ...TestFragB +} +fragment TestFragB { + country +}' | python -m json.tool | less +``` + +:::note +GraphQL+- has been renamed to Dgraph Query Language (DQL). While `application/dql` +is the preferred value for the `Content-Type` header, we will continue to support +`Content-Type: application/graphql+-` to avoid making breaking changes. +::: diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/query/functions.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/query/functions.md new file mode 100644 index 00000000..9d6c939a --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/query/functions.md @@ -0,0 +1,876 @@ +--- +title: Functions +--- +import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock'; + + + +Functions allow filtering based on properties of nodes or [variables](variables#value-variables). Functions can be applied in the query root or in filters. + + +Comparison functions (`eq`, `ge`, `gt`, `le`, `lt`) in the query root (aka `func:`) can only +be applied on [indexed predicates](../predicate-indexing). +Comparison functions can be used on [@filter](directive/filter) directives even on predicates that have not been indexed. +Filtering on non-indexed predicates can be slow for large datasets, as they require +iterating over all of the possible values at the level where the filter is being used. + +All other functions, in the query root or in the filter can only be applied to indexed predicates. + +For functions on string valued predicates, if no language preference is given, the function is applied to all languages and strings without a language tag; if a language preference is given, the function is applied only to strings of the given language. + +## Term matching + +### allofterms + +Syntax Example: `allofterms(predicate, "space-separated term list")` + +Schema Types: `string` + +Index Required: `term` + + +Matches strings that have all specified terms in any order; case insensitive. +#### Usage at root + +Query Example: All nodes that have `name` containing terms `indiana` and `jones`, returning the English name and genre in English. + + + +```dql +{ + me(func: allofterms(name@en, "jones indiana")) { + name@en + genre { + name@en + } + } +} +``` + + +#### Usage as Filter + +Query Example: All Steven Spielberg films that contain the words `indiana` and `jones`. The `@filter(has(director.film))` removes nodes with name Steven Spielberg that aren't the director --- the data also contains a character in a film called Steven Spielberg. + + + +```dql +{ + me(func: eq(name@en, "Steven Spielberg")) @filter(has(director.film)) { + name@en + director.film @filter(allofterms(name@en, "jones indiana")) { + name@en + } + } +} +``` + + + +### anyofterms + + +Syntax Example: `anyofterms(predicate, "space-separated term list")` + +Schema Types: `string` + +Index Required: `term` + + +Matches strings that have any of the specified terms in any order; case insensitive. +#### Usage at root + +Query Example: All nodes that have a `name` containing either `poison` or `peacock`. Many of the returned nodes are movies, but people like Joan Peacock also meet the search terms because without a [cascade directive](directive/cascade-directive) the query doesn't require a genre. + + + +```dql +{ + me(func:anyofterms(name@en, "poison peacock")) { + name@en + genre { + name@en + } + } +} +``` + + + +#### Usage as filter + +Query Example: All Steven Spielberg movies that contain `war` or `spies`. The `@filter(has(director.film))` removes nodes with name Steven Spielberg that aren't the director --- the data also contains a character in a film called Steven Spielberg. + + + +```dql +{ + me(func: eq(name@en, "Steven Spielberg")) @filter(has(director.film)) { + name@en + director.film @filter(anyofterms(name@en, "war spies")) { + name@en + } + } +} +``` + + + +### ngram + +Syntax Examples: `ngram(predicate, "quick brown fox")` + +Schema Types: `string` + +Index Required: `ngram` + +The `ngram` function matches strings that contain the given sequence of terms with support for stop word removal and stemming. + +#### Usage at root + +Query example: all nodes that have a `name` predicate containing a sequence of terms `frankly` and `dear`. + + + +```dql +{ + me(func: ngram(name@en, "frankly dear")) { + name@en + } +} +``` + + +#### Usage as filter + +Query example: all nodes that have a `description` predicate containing a sequence of terms `brown` and `fox`. + + + +```dql +{ + me(func: has(description)) @filter(ngram(description, "brown fox")) { + uid + description + } +} +``` + + + +## Regular Expressions + + +Syntax Examples: `regexp(predicate, /regular-expression/)` or case insensitive `regexp(predicate, /regular-expression/i)` + +Schema Types: `string` + +Index Required: `trigram` + + +Matches strings by regular expression. The regular expression language is that of [go regular expressions](https://golang.org/pkg/regexp/syntax/). + +Query Example: At root, match nodes with `Steven Sp` at the start of `name`, followed by any characters. For each such matched uid, match the films containing `ryan`. Note the difference with `allofterms`, which would match only `ryan` but regular expression search will also match within terms, such as `bryan`. + + + +```dql +{ + directors(func: regexp(name@en, /^Steven Sp.*$/)) { + name@en + director.film @filter(regexp(name@en, /ryan/i)) { + name@en + } + } +} +``` + + + +### Technical details + +A Trigram is a substring of three continuous runes. For example, `Dgraph` has trigrams `Dgr`, `gra`, `rap`, `aph`. + +To ensure efficiency of regular expression matching, Dgraph uses [trigram indexing](https://swtch.com/~rsc/regexp/regexp4.html). That is, Dgraph converts the regular expression to a trigram query, uses the trigram index and trigram query to find possible matches and applies the full regular expression search only to the possibles. +### Writing Efficient Regular Expressions and Limitations + +Keep the following in mind when designing regular expression queries. + +- At least one trigram must be matched by the regular expression (patterns shorter than 3 runes are not supported). That is, Dgraph requires regular expressions that can be converted to a trigram query. +- The number of alternative trigrams matched by the regular expression should be as small as possible (`[a-zA-Z][a-zA-Z][0-9]` is not a good idea). Many possible matches means the full regular expression is checked against many strings; where as, if the expression enforces more trigrams to match, Dgraph can make better use of the index and check the full regular expression against a smaller set of possible matches. +- Thus, the regular expression should be as precise as possible. Matching longer strings means more required trigrams, which helps to effectively use the index. +- If repeat specifications (`*`, `+`, `?`, `{n,m}`) are used, the entire regular expression must not match the _empty_ string or _any_ string: for example, `*` may be used like `[Aa]bcd*` but not like `(abcd)*` or `(abcd)|((defg)*)` +- Repeat specifications after bracket expressions (e.g. `[fgh]{7}`, `[0-9]+` or `[a-z]{3,5}`) are often considered as matching any string because they match too many trigrams. +- If the partial result (for subset of trigrams) exceeds 1000000 uids during index scan, the query is stopped to prohibit expensive queries. + +## Fuzzy matching + + +Syntax: `match(predicate, string, distance)` + +Schema Types: `string` + +Index Required: `trigram` + +Matches predicate values by calculating the [Levenshtein distance](https://en.wikipedia.org/wiki/Levenshtein_distance) to the string, +also known as _fuzzy matching_. The distance parameter must be greater than zero (0). Using a greater distance value can yield more but less accurate results. + +Query Example: At root, fuzzy match nodes similar to `Stephen`, with a distance value of less than or equal to 8. + + + +```dql +{ + directors(func: match(name@en, Stephen, 8)) { + name@en + } +} +``` + + + +Same query with a Levenshtein distance of 3. + + + +```dql +{ + directors(func: match(name@en, Stephen, 3)) { + name@en + } +} +``` + + + +## Vector Similarity Search + +Syntax Examples: `similar_to(predicate, 3, "[0.9, 0.8, 0, 0]")` + +Alternatively the vector can be passed as a variable: `similar_to(predicate, 3, $vec)` + +This function finds the nodes that have `predicate` close to the provided vector. The search is based on the distance metric specified in the index (`cosine`, `euclidean`, or `dotproduct`). The shorter distance indicates more similarity. +The second parameter, `3` specifies that top 3 matches be returned. + +Schema Types: `float32vector` + +Index Required: `hnsw` + + + +## Full-Text Search + +Syntax Examples: `alloftext(predicate, "space-separated text")` and `anyoftext(predicate, "space-separated text")` + +Schema Types: `string` + +Index Required: `fulltext` + + +Apply full-text search with stemming and stop words to find strings matching all or any of the given text. + +The following steps are applied during index generation and to process full-text search arguments: + +1. Tokenization (according to Unicode word boundaries). +1. Conversion to lowercase. +1. Unicode-normalization (to [Normalization Form KC](http://unicode.org/reports/tr15/#Norm_Forms)). +1. Stemming using language-specific stemmer (if supported by language). +1. Stop words removal (if supported by language). + +Dgraph uses [bleve](https://github.com/blevesearch/bleve) for its full-text search indexing. See also the bleve language specific [stop word lists](https://github.com/blevesearch/bleve/tree/master/analysis/lang). + +Following table contains all supported languages, corresponding country-codes, stemming and stop words filtering support. + +| Language | Country Code | Stemming | Stop words | +| :--------: | :----------: | :------: | :--------: | +| Arabic | ar | ✓ | ✓ | +| Armenian | hy | | ✓ | +| Basque | eu | | ✓ | +| Bulgarian | bg | | ✓ | +| Catalan | ca | | ✓ | +| Chinese | zh | ✓ | ✓ | +| Czech | cs | | ✓ | +| Danish | da | ✓ | ✓ | +| Dutch | nl | ✓ | ✓ | +| English | en | ✓ | ✓ | +| Finnish | fi | ✓ | ✓ | +| French | fr | ✓ | ✓ | +| Gaelic | ga | | ✓ | +| Galician | gl | | ✓ | +| German | de | ✓ | ✓ | +| Greek | el | | ✓ | +| Hindi | hi | ✓ | ✓ | +| Hungarian | hu | ✓ | ✓ | +| Indonesian | id | | ✓ | +| Italian | it | ✓ | ✓ | +| Japanese | ja | ✓ | ✓ | +| Korean | ko | ✓ | ✓ | +| Norwegian | no | ✓ | ✓ | +| Persian | fa | | ✓ | +| Portuguese | pt | ✓ | ✓ | +| Romanian | ro | ✓ | ✓ | +| Russian | ru | ✓ | ✓ | +| Spanish | es | ✓ | ✓ | +| Swedish | sv | ✓ | ✓ | +| Turkish | tr | ✓ | ✓ | + + +Query Example: All names that have `dog`, `dogs`, `bark`, `barks`, `barking`, etc. Stop word removal eliminates `the` and `which`. + + + +```dql +{ + movie(func:alloftext(name@en, "the dog which barks")) { + name@en + } +} +``` + + + +## Inequality +### equal to + +Syntax Examples: + +* `eq(predicate, value)` +* `eq(val(varName), value)` +* `eq(predicate, val(varName))` +* `eq(count(predicate), value)` +* `eq(predicate, [val1, val2, ..., valN])` +* `eq(predicate, [$var1, "value", ..., $varN])` + +Schema Types: `int`, `float`, `bool`, `string`, `dateTime` + +Index Required: An index is required for the `eq(predicate, ...)` forms (see table below) when used at query root. For `count(predicate)` at the query root, the `@count` index is required. For variables the values have been calculated as part of the query, so no index is required. + +| Type | Index Options | +|:-----------|:--------------| +| `int` | `int` | +| `float` | `float` | +| `bool` | `bool` | +| `string` | `exact`, `hash`, `term`, `fulltext` | +| `dateTime` | `dateTime` | + +Test for equality of a predicate or variable to a value or find in a list of values. + +The boolean constants are `true` and `false`, so with `eq` this becomes, for example, `eq(boolPred, true)`. + +Query Example: Movies with exactly thirteen genres. + + + +```dql +{ + me(func: eq(count(genre), 13)) { + name@en + genre { + name@en + } + } +} +``` + + + + +Query Example: Directors called Steven who have directed 1,2 or 3 movies. + + + +```dql +{ + steve as var(func: allofterms(name@en, "Steven")) { + films as count(director.film) + } + + stevens(func: uid(steve)) @filter(eq(val(films), [1,2,3])) { + name@en + numFilms : val(films) + } +} +``` + + + +### less than, less than or equal to, greater than and greater than or equal to + +Syntax Examples: for inequality `IE` + +* `IE(predicate, value)` +* `IE(val(varName), value)` +* `IE(predicate, val(varName))` +* `IE(count(predicate), value)` + +With `IE` replaced by + +* `le` less than or equal to +* `lt` less than +* `ge` greater than or equal to +* `gt` greater than + +Schema Types: `int`, `float`, `string`, `dateTime` + +Index required: An index is required for the `IE(predicate, ...)` forms (see table below) when used at query root. For `count(predicate)` at the query root, the `@count` index is required. For variables the values have been calculated as part of the query, so no index is required. + +| Type | Index Options | +|:-----------|:--------------| +| `int` | `int` | +| `float` | `float` | +| `string` | `exact` | +| `dateTime` | `dateTime` | + + +Query Example: Ridley Scott movies released before 1980. + + + +```dql +{ + me(func: eq(name@en, "Ridley Scott")) { + name@en + director.film @filter(lt(initial_release_date, "1980-01-01")) { + initial_release_date + name@en + } + } +} +``` + + + + +Query Example: Movies with directors with `Steven` in `name` and have directed more than `100` actors. + + + +```dql +{ + ID as var(func: allofterms(name@en, "Steven")) { + director.film { + num_actors as count(starring) + } + total as sum(val(num_actors)) + } + + dirs(func: uid(ID)) @filter(gt(val(total), 100)) { + name@en + total_actors : val(total) + } +} +``` + + + + + +Query Example: A movie in each genre that has over 30000 movies. Because there is no order specified on genres, the order will be by UID. The [count index](../predicate-indexing#count-index) records the number of edges out of nodes and makes such queries more . + + + +```dql +{ + genre(func: gt(count(~genre), 30000)){ + name@en + ~genre (first:1) { + name@en + } + } +} +``` + + + +Query Example: Directors called Steven and their movies which have `initial_release_date` greater +than that of the movie Minority Report. + + + +```dql +{ + var(func: eq(name@en,"Minority Report")) { + d as initial_release_date + } + + me(func: eq(name@en, "Steven Spielberg")) { + name@en + director.film @filter(ge(initial_release_date, val(d))) { + initial_release_date + name@en + } + } +} +``` + + + +## between + +Syntax Example: `between(predicate, startDateValue, endDateValue)` + +Schema Types: Scalar types, including `dateTime`, `int`, `float` and `string` + +Index Required: `dateTime`, `int`, `float`, and `exact` on strings + +Returns nodes that match an inclusive range of indexed values. The `between` +keyword performs a range check on the index to improve query efficiency, +helping to prevent a wide-ranging query on a large set of data from running +slowly. + +A common use case for the `between` keyword is to search within a +dataset indexed by `dateTime`. The following example query demonstrates this +use case. + +Query Example: Movies initially released in 1977, listed by genre. + + + +```dql +{ + me(func: between(initial_release_date, "1977-01-01", "1977-12-31")) { + name@en + genre { + name@en + } + } +} +``` + + + +## uid + +Syntax Examples: + +* `q(func: uid()) ` +* `predicate @filter(uid(, ..., ))` +* `predicate @filter(uid(a))` for variable `a` +* `q(func: uid(a,b))` for variables `a` and `b` +* `q(func: uid($uids))` for multiple uids in DQL Variables. You have to set the value of this variable as a string (e.g`"[0x1, 0x2, 0x3]"`) in queryWithVars. + +Filters nodes at the current query level to only nodes in the given set of UIDs. + +For query variable `a`, `uid(a)` represents the set of UIDs stored in `a`. For value variable `b`, `uid(b)` represents the UIDs from the UID to value map. With two or more variables, `uid(a,b,...)` represents the union of all the variables. + +`uid()`, like an identity function, will return the requested UID even if the node does not have any edges. + +:::tip +If the UID of a node is known, values for the node can be read directly. +::: + +Query Example: The films of Priyanka Chopra by known UID. + + +```dql +{ + films(func: uid(0x2c964)) { + name@hi + actor.film { + performance.film { + name@hi + } + } + } +} +``` + + + +Query Example: The films of Taraji Henson by genre. + + +```dql +{ + var(func: allofterms(name@en, "Taraji Henson")) { + actor.film { + F as performance.film { + G as genre + } + } + } + + Taraji_films_by_genre(func: uid(G)) { + genre_name : name@en + films : ~genre @filter(uid(F)) { + film_name : name@en + } + } +} +``` + + + + + +Query Example: Taraji Henson films ordered by number of genres, with genres listed in order of how many films Taraji has made in each genre. + + +```dql +{ + var(func: allofterms(name@en, "Taraji Henson")) { + actor.film { + F as performance.film { + G as count(genre) + genre { + C as count(~genre @filter(uid(F))) + } + } + } + } + + Taraji_films_by_genre_count(func: uid(G), orderdesc: val(G)) { + film_name : name@en + genres : genre (orderdesc: val(C)) { + genre_name : name@en + } + } +} +``` + + + +## uid_in + +Syntax Examples: + +* `q(func: ...) @filter(uid_in(predicate, ))` +* `predicate1 @filter(uid_in(predicate2, ))` +* `predicate1 @filter(uid_in(predicate2, [, ..., ]))` +* `predicate1 @filter(uid_in(predicate2, uid(myVariable) ))` + +Schema Types: UID + +Index Required: none + +While the `uid` function filters nodes at the current level based on UID, function `uid_in` allows looking ahead along an edge to check that it leads to a particular UID. This can often save an extra query block and avoids returning the edge. + +`uid_in` cannot be used at root. It accepts multiple UIDs as its argument, and it accepts a UID variable (which can contain a map of UIDs). + +Query Example: The collaborations of Marc Caro and Jean-Pierre Jeunet (UID 0x99706). If the UID of Jean-Pierre Jeunet is known, querying this way removes the need to have a block extracting his UID into a variable and the extra edge traversal and filter for `~director.film`. + + + +```dql +{ + caro(func: eq(name@en, "Marc Caro")) { + name@en + director.film @filter(uid_in(~director.film, 0x99706)) { + name@en + } + } +} +``` + + + +You can also query for Jean-Pierre Jeunet if you don't know his UID and use it in a UID variable. + + + +```dql +{ + getJeunet as q(func: eq(name@fr, "Jean-Pierre Jeunet")) + + caro(func: eq(name@en, "Marc Caro")) { + name@en + director.film @filter(uid_in(~director.film, uid(getJeunet) )) { + name@en + } + } +} +``` + + +## type + + +Query Example: all nodes of type "Animal" + + + +```dql +{ + q(func: type(Animal)) { + uid + name + } +} +``` + + + +`type(Animal)` equivalent to `eq(dgraph.type,"Animal")` + +type() can also be used as a filter: + + + +```dql +{ + q(func: has(parent)) { + uid + parent @filter(type(Person)) { + uid + name + } + } +} +``` + + +## has + +Syntax Examples: `has(predicate)` + +Schema Types: all + +Determines if a node has a particular predicate. + +Query Example: First five directors and all their movies that have a release date recorded. Directors have directed at least one film --- equivalent semantics to `gt(count(director.film), 0)`. + + +```dql +{ + me(func: has(director.film), first: 5) { + name@en + director.film @filter(has(initial_release_date)) { + initial_release_date + name@en + } + } +} +``` + + +## Geolocation + +:::note As of now we only support indexing Point, Polygon and MultiPolygon [geometry types](https://github.com/twpayne/go-geom#geometry-types). However, Dgraph can store other types of gelocation data. ::: + +Note that for geo queries, any polygon with holes is replace with the outer loop, ignoring holes. Also, as for version 0.7.7 polygon containment checks are approximate. +### Mutations + +To make use of the geo functions you would need an index on your predicate. +``` +loc: geo @index(geo) . +``` + +Here is how you would add a `Point`. + +``` +{ + set { + <_:0xeb1dde9c> "{'type':'Point','coordinates':[-122.4220186,37.772318]}"^^ . + <_:0xeb1dde9c> "Hamon Tower" . + <_:0xeb1dde9c> "Location" . + } +} +``` + +Here is how you would associate a `Polygon` with a node. Adding a `MultiPolygon` is also similar. + +``` +{ + set { + <_:0xf76c276b> "{'type':'Polygon','coordinates':[[[-122.409869,37.7785442],[-122.4097444,37.7786443],[-122.4097544,37.7786521],[-122.4096334,37.7787494],[-122.4096233,37.7787416],[-122.4094004,37.7789207],[-122.4095818,37.7790617],[-122.4097883,37.7792189],[-122.4102599,37.7788413],[-122.409869,37.7785442]],[[-122.4097357,37.7787848],[-122.4098499,37.778693],[-122.4099025,37.7787339],[-122.4097882,37.7788257],[-122.4097357,37.7787848]]]}"^^ . + <_:0xf76c276b> "Best Western Americana Hotel" . + <_:0xf76c276b> "Location" . + } +} +``` + +The above examples have been picked from our [SF Tourism](https://github.com/dgraph-io/benchmarks/blob/master/data/sf.tourism.gz?raw=true) dataset. +### Query +#### near + +Syntax Example: `near(predicate, [long, lat], distance)` + +Schema Types: `geo` + +Index Required: `geo` + +Matches all entities where the location given by `predicate` is within `distance` meters of geojson coordinate `[long, lat]`. + +Query Example: Tourist destinations within 1000 meters (1 kilometer) of a point in Golden Gate Park in San Francisco. + + + +```dql +{ + tourist(func: near(loc, [-122.469829, 37.771935], 1000) ) { + name + } +} +``` + + + +#### within + +Syntax Example: `within(predicate, [[[long1, lat1], ..., [longN, latN]]])` + +Schema Types: `geo` + +Index Required: `geo` + +Matches all entities where the location given by `predicate` lies within the polygon specified by the geojson coordinate array. + +Query Example: Tourist destinations within the specified area of Golden Gate Park, San Francisco. + + + +```dql +{ + tourist(func: within(loc, [[[-122.47266769409178, 37.769018558337926 ], [ -122.47266769409178, 37.773699921075135 ], [ -122.4651575088501, 37.773699921075135 ], [ -122.4651575088501, 37.769018558337926 ], [ -122.47266769409178, 37.769018558337926]]] )) { + name + } +} +``` + + + +#### contains + +Syntax Examples: `contains(predicate, [long, lat])` or `contains(predicate, [[long1, lat1], ..., [longN, latN]])` + +Schema Types: `geo` + +Index Required: `geo` + +Matches all entities where the polygon describing the location given by `predicate` contains geojson coordinate `[long, lat]` or given geojson polygon. + +Query Example : All entities that contain a point in the flamingo enclosure of San Francisco Zoo. + + +```dql +{ + tourist(func: contains(loc, [ -122.50326097011566, 37.73353615592843 ] )) { + name + } +} +``` + + + +#### intersects + +Syntax Example: `intersects(predicate, [[[long1, lat1], ..., [longN, latN]]])` + +Schema Types: `geo` + +Index Required: `geo` + +Matches all entities where the polygon describing the location given by `predicate` intersects the given geojson polygon. + + + + +```dql +{ + tourist(func: intersects(loc, [[[-122.503325343132, 37.73345766902749 ], [ -122.503325343132, 37.733903134117966 ], [ -122.50271648168564, 37.733903134117966 ], [ -122.50271648168564, 37.73345766902749 ], [ -122.503325343132, 37.73345766902749]]] )) { + name + } +} +``` + + diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/query/graphql-variables.mdx b/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/query/graphql-variables.mdx new file mode 100644 index 00000000..983793b5 --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/query/graphql-variables.mdx @@ -0,0 +1,100 @@ +--- +title: Query parameters +--- +import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock'; + +Syntax Examples (using default values): + +* `query title($name: string = "Bauman") { ... }` +* `query title($age: int = "95") { ... }` +* `query title($uids: string = "0x1") { ... }` +* `query title($uids: string = "[0x1, 0x2, 0x3]") { ... }`. The value of the variable is a quoted array. + +`Variables` can be defined and used in queries which helps in query reuse and avoids costly string building in clients at runtime by passing a separate variable map. A variable starts with a `$` symbol. +For **HTTP requests** with Query parameters, we must use `Content-Type: application/json` header and pass data with a JSON object containing `query` and `variables`. + +```sh +curl -H "Content-Type: application/json" localhost:8080/query -XPOST -d $'{ + "query": "query test($a: string) { test(func: eq(name, $a)) { \n uid \n name \n } }", + "variables": { "$a": "Alice" } +}' | python -m json.tool | less +``` + + +```dql + +query test($a: int, $b: int, $name: string) { + me(func: allofterms(name@en, $name)) { + name@en + director.film (first: $a, offset: $b) { + name @en + genre(first: $a) { + name@en + } + } + } +} +``` + + +* Variables can have default values. In the example below, `$a` has a default value of `2`. Since the value for `$a` isn't provided in the variable map, `$a` takes on the default value. +* Variables whose type is suffixed with a `!` can't have a default value but must have a value as part of the variables map. +* The value of the variable must be parsable to the given type, if not, an error is thrown. +* The variable types that are supported as of now are: `int`, `float`, `bool` and `string`. +* Any variable that is being used must be declared in the named query clause in the beginning. + + + +```dql +{{< runnable vars="{\"$b\": \"10\", \"$name\": \"Steven Spielberg\"}" >}} +query test($a: int = 2, $b: int!, $name: string) { + me(func: allofterms(name@en, $name)) { + director.film (first: $a, offset: $b) { + genre(first: $a) { + name@en + } + } + } +} +``` + + +You can also use array with Query parameters. + + +```dql +{{< runnable vars="{\"$b\": \"10\", \"$aName\": \"Steven Spielberg\", \"$bName\": \"Quentin Tarantino\"}" >}} +query test($a: int = 2, $b: int!, $aName: string, $bName: string) { + me(func: eq(name@en, [$aName, $bName])) { + director.film (first: $a, offset: $b) { + genre(first: $a) { + name@en + } + } + } +} +``` + + +We also support variable substitution in facets. + + +```dql +{{< runnable vars="{\"$name\": \"Alice\", \"$IsClose\": \"true\"}" >}} +query test($name: string = "Alice", $IsClose: string = "true") { + data(func: eq(name, $name)) { + friend @facets(eq(close, $IsClose)) { + name + } + colleague : friend @facets(eq(close, false)) { + name + } + } +} +``` + + +:::note +If you want to input a list of uids as a GraphQL variable value, you can have the variable as string type and +have the value surrounded by square brackets like `["13", "14"]`. +::: \ No newline at end of file diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/query/index.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/query/index.md new file mode 100644 index 00000000..8e216b93 --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/query/index.md @@ -0,0 +1,9 @@ +--- +title: Query +--- + +Dgraph Query Language (DQL) is Dgraph’s proprietary language to add, modify, delete and fetch data. + +Fetching data is done through [Queries](dql-query). + +Adding, modifying or deleting data is done through [Mutations](../dql-mutation). diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/query/kshortest-path-queries.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/query/kshortest-path-queries.md new file mode 100644 index 00000000..b22fa026 --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/query/kshortest-path-queries.md @@ -0,0 +1,288 @@ +--- +title: Shortest Path Queries +--- + +The shortest path between a source (`from`) node and destination (`to`) node can be found using the keyword `shortest` for the query block name. It requires the source node UID, destination node UID and the predicates (at least one) that have to be considered for traversal. A `shortest` query block returns the shortest path under `_path_` in the query response. The path can also be stored in a variable which is used in other query blocks. + +## K-Shortest Path queries + +By default the shortest path is returned. With `numpaths: k`, and `k > 1`, the k-shortest paths are returned. Cyclical paths are pruned out from the result of k-shortest path query. With `depth: n`, the paths up to `n` depth away are returned. + +:::note +- If no predicates are specified in the `shortest` block, no path can be fetched as no edge is traversed. +- If you're seeing queries take a long time, you can set a [gRPC deadline](https://grpc.io/blog/deadlines) to stop the query after a certain amount of time. +::: + +For example: + +```sh +curl localhost:8080/alter -XPOST -d $' + name: string @index(exact) . +' | python -m json.tool | less +``` + +```graphql +{ + set { + _:a _:b (weight=0.1) . + _:b _:c (weight=0.2) . + _:c _:d (weight=0.3) . + _:a _:d (weight=1) . + _:a "Alice" . + _:a "Person" . + _:b "Bob" . + _:b "Person" . + _:c "Tom" . + _:c "Person" . + _:d "Mallory" . + _:d "Person" . + } +} +``` + +The shortest path between Alice and Mallory (assuming UIDs `0x2` and `0x5` respectively) can be found with this query: + +```graphql +{ + path as shortest(from: 0x2, to: 0x5) { + friend + } + path(func: uid(path)) { + name + } +} +``` + +Which returns the following results. + +:::note +without considering the `weight` facet, each edges' weight is considered as `1` +::: + +``` +{ + "data": { + "path": [ + { + "name": "Alice" + }, + { + "name": "Mallory" + } + ], + "_path_": [ + { + "uid": "0x2", + "friend": [ + { + "uid": "0x5" + } + ] + } + ] + } +} +``` + +We can return more paths by specifying `numpaths`. Setting `numpaths: 2` returns the shortest two paths: + +```graphql +{ + + A as var(func: eq(name, "Alice")) + M as var(func: eq(name, "Mallory")) + + path as shortest(from: uid(A), to: uid(M), numpaths: 2) { + friend + } + path(func: uid(path)) { + name + } +} +``` + +:::noteIn the query above, instead of using UID literals, we query both people using var blocks and the `uid()` function. You can also combine it with [GraphQL Variables](graphql-variables).::: + +## Edge weight + +The shortest path implementation in Dgraph relies on facets to provide weights. Using `facets` on the edges let you define the edges' weight as follows: + +:::noteOnly one facet per predicate is allowed in the shortest query block.::: + +```graphql +{ + path as shortest(from: 0x2, to: 0x5) { + friend @facets(weight) + } + + path(func: uid(path)) { + name + } +} +``` + +``` +{ + "data": { + "path": [ + { + "name": "Alice" + }, + { + "name": "Bob" + }, + { + "name": "Tom" + }, + { + "name": "Mallory" + } + ], + "_path_": [ + { + "uid": "0x2", + "friend": [ + { + "uid": "0x3", + "friend|weight": 0.1, + "friend": [ + { + "uid": "0x4", + "friend|weight": 0.2, + "friend": [ + { + "uid": "0x5", + "friend|weight": 0.3 + } + ] + } + ] + } + ] + } + ] + } +} +``` + +### Traverse example + +Here is a graph traversal example that allows you to find the shortest path between friends using a `Car` or a `Bus`. + +:::tip +Car and Bus movement for each relation is modeled as facets and specified in the shortest query +::: + +```graphql +{ + set { + _:a _:b (weightCar=10, weightBus=1 ) . + _:b _:c (weightCar=20, weightBus=1) . + _:c _:d (weightCar=11, weightBus=1.1) . + _:a _:d (weightCar=70, weightBus=2) . + _:a "Alice" . + _:a "Person" . + _:b "Bob" . + _:b "Person" . + _:c "Tom" . + _:c "Person" . + _:d "Mallory" . + _:d "Person" . + } +} +``` + +Query to find the shortest path relying on `Car` and `Bus`: + +```graphql +{ + + A as var(func: eq(name, "Alice")) + M as var(func: eq(name, "Mallory")) + + sPathBus as shortest(from: uid(A), to: uid(M)) { + friend + @facets(weightBus) + } + + sPathCar as shortest(from: uid(A), to: uid(M)) { + friend + @facets(weightCar) + } + + pathBus(func: uid(sPathBus)) { + name + } + + pathCar(func: uid(sPathCar)) { + name + } +} +``` + +The response contains the following paths conforming to the specified weights: + +``` + "pathBus": [ + { + "name": "Alice" + }, + { + "name": "Mallory" + } + ], + "pathCar": [ + { + "name": "Alice" + }, + { + "name": "Bob" + }, + { + "name": "Tom" + }, + { + "name": "Mallory" + } + ] +``` + +## Constraints + +Constraints can be applied to the intermediate nodes as follows. + +```graphql +{ + path as shortest(from: 0x2, to: 0x5) { + friend @filter(not eq(name, "Bob")) @facets(weight) + relative @facets(liking) + } + + relationship(func: uid(path)) { + name + } +} +``` + +The k-shortest path algorithm (used when `numpaths` > 1) also accepts the arguments `minweight` and `maxweight`, which take a float as their value. When they are passed, only paths within the weight range `[minweight, maxweight]` will be considered as valid paths. This can be used, for example, to query the shortest paths that traverse between 2 and 4 nodes. + +```graphql +{ + path as shortest(from: 0x2, to: 0x5, numpaths: 2, minweight: 2, maxweight: 4) { + friend + } + path(func: uid(path)) { + name + } +} +``` + +## Notes + +Some points to keep in mind for shortest path queries: + +- Weights must be non-negative. Dijkstra's algorithm is used to calculate the shortest paths. +- Only one facet per predicate in the shortest query block is allowed. +- Only one `shortest` path block is allowed per query. Only one `_path_` is returned in the result. For queries with `numpaths` > 1, `_path_` contains all the paths. +- Cyclical paths are not included in the result of k-shortest path query. +- For k-shortest paths (when `numpaths` > 1), the result of the shortest path query variable will only return a single path which will be the shortest path among the k paths. All k paths are returned in `_path_`. diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/query/language-support.mdx b/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/query/language-support.mdx new file mode 100644 index 00000000..8b5ce3c6 --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/query/language-support.mdx @@ -0,0 +1,70 @@ +--- +title: Language Support +--- +import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock'; + + + + +:::noteA `@lang` directive must be specified in the schema to query or mutate +predicates with language tags.::: + +Dgraph supports UTF-8 strings. + +In a query, for a string valued edge `edge`, the syntax +``` +edge@lang1:...:langN +``` +specifies the preference order for returned languages, with the following rules. + +* At most one result will be returned (except in the case where the language list is set to *). +* The preference list is considered left to right: if a value in given language is not found, the next language from the list is considered. +* If there are no values in any of the specified languages, no value is returned. +* A final `.` means that a value without a specified language is returned or if there is no value without language, a value in ''some'' language is returned. +* Setting the language list value to * will return all the values for that predicate along with their language. Values without a language tag are also returned. + +For example: + +- `name` => Look for an untagged string; return nothing if no untagged value exits. +- `name@.` => Look for an untagged string, then any language. +- `name@en` => Look for `en` tagged string; return nothing if no `en` tagged string exists. +- `name@en:.` => Look for `en`, then untagged, then any language. +- `name@en:pl` => Look for `en`, then `pl`, otherwise nothing. +- `name@en:pl:.` => Look for `en`, then `pl`, then untagged, then any language. +- `name@*` => Look for all the values of this predicate and return them along with their language. For example, if there are two values with languages en and hi, this query will return two keys named "name@en" and "name@hi". + + +:::note + +In functions, language lists (including the `@*` notation) are not allowed. +Untagged predicates, Single language tags, and `.` notation work as described +above. + +--- +In [full-text search functions](functions#full-text-search) +(`alloftext`, `anyoftext`), when no language is specified (untagged or `@.`), +the default (English) full-text tokenizer is used. This does not mean that +the value with the `en` tag will be searched when querying the untagged value, +but that untagged values will be treated as English text. If you don't want that +to be the case, use the appropriate tag for the desired language, both for +mutating and querying the value. +::: +Query Example: Some of Bollywood director and actor Farhan Akhtar's movies have a name stored in Russian as well as Hindi and English, others do not. + + +```dql +{ + q(func: allofterms(name@en, "Farhan Akhtar")) { + name@hi + name@en + director.film { + name@ru:hi:en + name@en + name@hi + name@ru + } + } +} +``` + + \ No newline at end of file diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/query/pagination.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/query/pagination.md new file mode 100644 index 00000000..8b6202a2 --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/query/pagination.md @@ -0,0 +1,154 @@ +--- +title: Pagination +--- +import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock'; + + + +Pagination allows returning only a portion, rather than the whole, result set. This can be useful for top-k style queries as well as to reduce the size of the result set for client side processing or to allow paged access to results. + +Pagination is often used with [sorting](/dql/query/sorting). + +:::noteWithout a sort order specified, the results are sorted by `uid`, which is assigned randomly. So the ordering, while deterministic, might not be what you expected.::: +## First + +Syntax Examples: + +* `q(func: ..., first: N)` +* `predicate (first: N) { ... }` +* `predicate @filter(...) (first: N) { ... }` + +For positive `N`, `first: N` retrieves the first `N` results, by sorted or UID order. + +For negative `N`, `first: N` retrieves the last `N` results, by sorted or UID order. Currently, negative is only supported when no order is applied. To achieve the effect of a negative with a sort, reverse the order of the sort and use a positive `N`. + + +Query Example: Last two films, by UID order, directed by Steven Spielberg and the first three genres of those movies, sorted alphabetically by English name. + + + +```dql +{ + me(func: allofterms(name@en, "Steven Spielberg")) { + director.film (first: -2) { + name@en + initial_release_date + genre (orderasc: name@en) (first: 3) { + name@en + } + } + } +} +``` + + + + + +Query Example: The three directors named Steven who have directed the most actors of all directors named Steven. + + + +```dql +{ + ID as var(func: allofterms(name@en, "Steven")) @filter(has(director.film)) { + director.film { + stars as count(starring) + } + totalActors as sum(val(stars)) + } + + mostStars(func: uid(ID), orderdesc: val(totalActors), first: 3) { + name@en + stars : val(totalActors) + + director.film { + name@en + } + } +} +``` + + +## Offset + +Syntax Examples: + +* `q(func: ..., offset: N)` +* `predicate (offset: N) { ... }` +* `predicate (first: M, offset: N) { ... }` +* `predicate @filter(...) (offset: N) { ... }` + +With `offset: N` the first `N` results are not returned. Used in combination with first, `first: M, offset: N` skips over `N` results and returns the following `M`. + +:::noteSkipping over `N` results takes time proportional to `N` (complexity `O(N)`). In other words, the larger `N`, the longer it takes to compute the result set. Prefer [after](#after) over `offset`.::: + +Query Example: Order Hark Tsui's films by English title, skip over the first 4 and return the following 6. + + + +```dql +{ + me(func: allofterms(name@en, "Hark Tsui")) { + name@zh + name@en + director.film (orderasc: name@en) (first:6, offset:4) { + genre { + name@en + } + name@zh + name@en + initial_release_date + } + } +} +``` + + +## After + +Syntax Examples: + +* `q(func: ..., after: UID)` +* `predicate (first: N, after: UID) { ... }` +* `predicate @filter(...) (first: N, after: UID) { ... }` + +Another way to get results after skipping over some results is to use the default UID ordering and skip directly past a node specified by UID. For example, a first query could be of the form `predicate (after: 0x0, first: N)`, or just `predicate (first: N)`, with subsequent queries of the form `predicate(after: , first: N)`. + +:::noteSkipping over results with `after` takes constant time (complexity `O(1)`). In other words, no matter how many results are skipped, no extra time adds to computing the result set. This should be preferred over [offset](#offset).::: + +Query Example: The first five of Baz Luhrmann's films, sorted by UID order. + + + +```dql +{ + me(func: allofterms(name@en, "Baz Luhrmann")) { + name@en + director.film (first:5) { + uid + name@en + } + } +} +``` + + + +The fifth movie is the Australian movie classic Strictly Ballroom. It has UID `0x99e44`. The results after Strictly Ballroom can now be obtained with `after`. + + + +```dql +{ + me(func: allofterms(name@en, "Baz Luhrmann")) { + name@en + director.film (first:5, after: 0x99e44) { + uid + name@en + } + } +} +``` + + diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/query/running-examples.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/query/running-examples.md new file mode 100644 index 00000000..b28b14bd --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/query/running-examples.md @@ -0,0 +1,55 @@ +--- +title: Running examples +--- + +The following pages are the language reference for DQL. + +They contain examples that you can run interactively using a database of 21 million triples about movies and actors. + +The queries are executed on an instance of Dgraph running at https://play.dgraph.io/. + +#### Example database schema + +The example movie database uses the following schema: + +``` +# Define Directives and index + +director.film: [uid] @reverse . +actor.film: [uid] @count . +genre: [uid] @reverse . +initial_release_date: dateTime @index(year) . +name: string @index(exact, term) @lang . +starring: [uid] . +performance.film: [uid] . +performance.character_note: string . +performance.character: [uid] . +performance.actor: [uid] . +performance.special_performance_type: [uid] . +type: [uid] . + +# Define Types + +type Person { + name + director.film + actor.film +} + +type Movie { + name + initial_release_date + genre + starring +} + +type Genre { + name +} + +type Performance { + performance.film + performance.character + performance.actor +} +``` diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/query/sorting.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/query/sorting.md new file mode 100644 index 00000000..6e4cfe1d --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/query/sorting.md @@ -0,0 +1,89 @@ +--- +title: Sorting +--- +import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock'; + + + +Syntax Examples: + +* `q(func: ..., orderasc: predicate)` +* `q(func: ..., orderdesc: val(varName))` +* `predicate (orderdesc: predicate) { ... }` +* `predicate @filter(...) (orderasc: N) { ... }` +* `q(func: ..., orderasc: predicate1, orderdesc: predicate2)` + +Sortable Types: `int`, `float`, `String`, `dateTime`, `default` + +Results can be sorted in ascending order (`orderasc`) or descending order (`orderdesc`) by a predicate or variable. + +For sorting on predicates with [sortable indices](/dql/predicate-indexing#sortable-indices), Dgraph sorts on the values and with the index in parallel and returns whichever result is computed first. + +:::note +Dgraph returns `null` values at the end of the results, irrespective of their sort. This behavior is consistent across indexed and non-indexed sorts. +::: + +:::tip +Sorted queries retrieve up to 1000 results by default. This can be changed with [first](/dql/query/pagination#first). +::: + + +Query Example: French director Jean-Pierre Jeunet's movies sorted by release date. + + + +```dql +{ + me(func: allofterms(name@en, "Jean-Pierre Jeunet")) { + name@fr + director.film(orderasc: initial_release_date) { + name@fr + name@en + initial_release_date + } + } +} +``` + + + +Sorting can be performed at root and on value variables. + +Query Example: All genres sorted alphabetically and the five movies in each genre with the most genres. + + + +```dql +{ + genres as var(func: has(~genre)) { + ~genre { + numGenres as count(genre) + } + } + + genres(func: uid(genres), orderasc: name@en) { + name@en + ~genre (orderdesc: val(numGenres), first: 5) { + name@en + genres : val(numGenres) + } + } +} +``` + + + +Sorting can also be performed by multiple predicates as shown below. If the values are equal for the +first predicate, then they are sorted by the second predicate and so on. + +Query Example: Find all nodes which have type Person, sort them by their first_name and among those +that have the same first_name sort them by last_name in descending order. + +``` +{ + me(func: type("Person"), orderasc: first_name, orderdesc: last_name) { + first_name + last_name + } +} +``` diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/query/variables.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/query/variables.md new file mode 100644 index 00000000..df342d23 --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/query/variables.md @@ -0,0 +1,307 @@ +--- +title: Variables +--- +import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock'; + + +## Query Variables +Syntax Examples: + +* `varName as q(func: ...) { ... }` +* `varName as var(func: ...) { ... }` +* `varName as predicate { ... }` +* `varName as predicate @filter(...) { ... }` + +Types : `uid` + +Nodes (UIDs) matched at one place in a query can be stored in a variable and used elsewhere. Query variables can be used in other query blocks or in a child node of the defining block. + +Query variables do not affect the semantics of the query at the point of definition. Query variables are evaluated to all nodes matched by the defining block. + +In general, query blocks are executed in parallel, but variables impose an evaluation order on some blocks. Cycles induced by variable dependence are not permitted. + +If a variable is defined, it must be used elsewhere in the query. + +A query variable is used by extracting the UIDs in it with `uid(var-name)`. + +The syntax `func: uid(A,B)` or `@filter(uid(A,B))` means the union of UIDs for variables `A` and `B`. + +Query Example: The movies of Angelia Jolie and Brad Pitt where both have acted on movies in the same genre. Note that `B` and `D` match all genres for all movies, not genres per movie. + + +```dql +{ + var(func:allofterms(name@en, "angelina jolie")) { + actor.film { + A AS performance.film { # All films acted in by Angelina Jolie + B As genre # Genres of all the films acted in by Angelina Jolie + } + } + } + + var(func:allofterms(name@en, "brad pitt")) { + actor.film { + C AS performance.film { # All films acted in by Brad Pitt + D as genre # Genres of all the films acted in by Brad Pitt + } + } + } + + films(func: uid(D)) @filter(uid(B)) { # Genres from both Angelina and Brad + name@en + ~genre @filter(uid(A, C)) { # Movies in either A or C. + name@en + } + } +} +``` + + + +## Value Variables +Syntax Examples: + +* `varName as scalarPredicate` +* `varName as count(predicate)` +* `varName as avg(...)` +* `varName as math(...)` + +Types : `int`, `float`, `String`, `dateTime`, `default`, `geo`, `bool` + +Value variables store scalar values. Value variables are a map from the UIDs of the enclosing block to the corresponding values. + +It therefore only makes sense to use the values from a value variable in a context that matches the same UIDs - if used in a block matching different UIDs the value variable is undefined. + +It is an error to define a value variable but not use it elsewhere in the query. + +Value variables are used by extracting the values with `val(var-name)`, or by extracting the UIDs with `uid(var-name)`. + +[Facet](/dql/query/facets) values can be stored in value variables. + +Query Example: The number of movie roles played by the actors of the 80's classic "The Princess Bride". Query variable `pbActors` matches the UIDs of all actors from the movie. Value variable `roles` is thus a map from actor UID to number of roles. Value variable `roles` can be used in the `totalRoles` query block because that query block also matches the `pbActors` UIDs, so the actor to number of roles map is available. + + + +```dql +{ + var(func:allofterms(name@en, "The Princess Bride")) { + starring { + pbActors as performance.actor { + roles as count(actor.film) + } + } + } + totalRoles(func: uid(pbActors), orderasc: val(roles)) { + name@en + numRoles : val(roles) + } +} +``` + + + + +Value variables can be used in place of UID variables by extracting the UID list from the map. + +Query Example: The same query as the previous example, but using value variable `roles` for matching UIDs in the `totalRoles` query block. + + + +```dql +{ + var(func:allofterms(name@en, "The Princess Bride")) { + starring { + performance.actor { + roles as count(actor.film) + } + } + } + totalRoles(func: uid(roles), orderasc: val(roles)) { + name@en + numRoles : val(roles) + } +} +``` + + + + +## Variable Propagation + +Like query variables, value variables can be used in other query blocks and in blocks nested within the defining block. When used in a block nested within the block that defines the variable, the value is computed as a sum of the variable for parent nodes along all paths to the point of use. This is called variable propagation. + +For example: +``` +{ + q(func: uid(0x01)) { + myscore as math(1) # A + friends { # B + friends { # C + ...myscore... + } + } + } +} +``` +At line A, a value variable `myscore` is defined as mapping node with UID `0x01` to value 1. At B, the value for each friend is still 1: there is only one path to each friend. Traversing the friend edge twice reaches the friends of friends. The variable `myscore` gets propagated such that each friend of friend will receive the sum of its parents values: if a friend of a friend is reachable from only one friend, the value is still 1, if they are reachable from two friends, the value is two and so on. That is, the value of `myscore` for each friend of friends inside the block marked C will be the number of paths to them. + +**The value that a node receives for a propagated variable is the sum of the values of all its parent nodes.** + +This propagation is useful, for example, in normalizing a sum across users, finding the number of paths between nodes and accumulating a sum through a graph. + + + +Query Example: For each Harry Potter movie, the number of roles played by actor Warwick Davis. + + +```dql +{ + num_roles(func: eq(name@en, "Warwick Davis")) @cascade @normalize { + + paths as math(1) # records number of paths to each character + + actor : name@en + + actor.film { + performance.film @filter(allofterms(name@en, "Harry Potter")) { + film_name : name@en + characters : math(paths) # how many paths (i.e. characters) reach this film + } + } + } +} +``` + + + + +Query Example: Each actor who has been in a Peter Jackson movie and the fraction of Peter Jackson movies they have appeared in. + + +```dql +{ + movie_fraction(func:eq(name@en, "Peter Jackson")) @normalize { + + paths as math(1) + total_films : num_films as count(director.film) + director : name@en + + director.film { + starring { + performance.actor { + fraction : math(paths / (num_films/paths)) + actor : name@en + } + } + } + } +} +``` + + + +More examples can be found in two Dgraph blog posts about using variable propagation for recommendation engines ([post 1](https://open.dgraph.io/post/recommendation/), [post 2](https://open.dgraph.io/post/recommendation2/)). + +## Math on value variables +Value variables can be combined using mathematical functions. For example, this could be used to associate a score which is then used to order or perform other operations, such as might be used in building news feeds, simple recommendation systems, and so on. + +Math statements must be enclosed within `math( )` and must be stored to a value variable. + +The supported operators are as follows: + +| Operators | Types accepted | What it does | +| :------------: | :--------------: | :------------------------: | +| `+` `-` `*` `/` `%` | `int`, `float` | performs the corresponding operation | +| `min` `max` | All types except `geo`, `bool` (binary functions) | selects the min/max value among the two | +| `<` `>` `<=` `>=` `==` `!=` | All types except `geo`, `bool` | Returns true or false based on the values | +| `floor` `ceil` `ln` `exp` `sqrt` | `int`, `float` (unary function) | performs the corresponding operation | +| `since` | `dateTime` | Returns the number of seconds in float from the time specified | +| `pow(a, b)` | `int`, `float` | Returns `a to the power b` | +| `logbase(a,b)` | `int`, `float` | Returns `log(a)` to the base `b` | +| `cond(a, b, c)` | first operand must be a Boolean | selects `b` if `a` is true else `c` | + + +:::note +If an integer overflow occurs, or an operand is passed to a math operation (such as `ln`, `logbase`, `sqrt`, `pow`) +which results in an illegal operation, Dgraph will return an error. +::: + +Query Example: Form a score for each of Steven Spielberg's movies as the sum of number of actors, number of genres and number of countries. List the top five such movies in order of decreasing score. + + + +```dql +{ + var(func:allofterms(name@en, "steven spielberg")) { + films as director.film { + p as count(starring) + q as count(genre) + r as count(country) + score as math(p + q + r) + } + } + + TopMovies(func: uid(films), orderdesc: val(score), first: 5){ + name@en + val(score) + } +} +``` + + + +Value variables and aggregations of them can be used in filters. + +Query Example: Calculate a score for each Steven Spielberg movie with a condition on release date to penalize movies that are more than 20 years old, filtering on the resulting score. + + + +```dql +{ + var(func:allofterms(name@en, "steven spielberg")) { + films as director.film { + p as count(starring) + q as count(genre) + date as initial_release_date + years as math(since(date)/(365*24*60*60)) + score as math(cond(years > 20, 0, ln(p)+q-ln(years))) + } + } + + TopMovies(func: uid(films), orderdesc: val(score)) @filter(gt(val(score), 2)){ + name@en + val(score) + val(date) + } +} +``` + + + + +Values calculated with math operations are stored to value variables and so can be aggregated. + +Query Example: Compute a score for each Steven Spielberg movie and then aggregate the score. + + + +```dql +{ + steven as var(func:eq(name@en, "Steven Spielberg")) @filter(has(director.film)) { + director.film { + p as count(starring) + q as count(genre) + r as count(country) + score as math(p + q + r) + } + directorScore as sum(val(score)) + } + + score(func: uid(steven)){ + name@en + val(directorScore) + } +} +``` + + diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/tips/index.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/tips/index.md new file mode 100644 index 00000000..78f682cc --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/tips/index.md @@ -0,0 +1,168 @@ +--- +title: "DQL: Tips and Tricks" +--- +import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock'; + + + +## Get Sample Data + +Use the `has` function to get some sample nodes. + + + +```dql +{ + result(func: has(director.film), first: 10) { + uid + expand(_all_) + } +} +``` + + + + +## Count number of connecting nodes + +Use `expand(_all_)` to expand the nodes' edges, then assign them to a variable. +The variable can now be used to iterate over the unique neighboring nodes. +Then use `count(uid)` to count the number of nodes in a block. + + + +```dql +{ + uids(func: has(director.film), first: 1) { + uid + expand(_all_) { u as uid } + } + + result(func: uid(u)) { + count(uid) + } +} +``` + + + +## Search on non-indexed predicates + +Use the `has` function among the value variables to search on non-indexed predicates. + + + +```dql +{ + var(func: has(festival.date_founded)) { + p as festival.date_founded + } + query(func: eq(val(p), "1961-01-01T00:00:00Z")) { + uid + name@en + name@ru + name@pl + festival.date_founded + festival.focus { name@en } + festival.individual_festivals { total : count(uid) } + } +} +``` + + + +## Sort edge by nested node values + +Dgraph [sorting](/dql/query/sorting) is based on a single +level of the subgraph. To sort a level by the values of a deeper level, use +[query variables](/dql/query/variables#query-variables) to bring +nested values up to the level of the edge to be sorted. + +Example: Get all actors from a Steven Spielberg movie sorted alphabetically. +The actor's name is not accessed from a single traversal from the `starring` edge; +the name is accessible via `performance.actor`. + + + +```dql +{ + spielbergMovies as var(func: allofterms(name@en, "steven spielberg")) { + name@en + director.film (orderasc: name@en, first: 1) { + starring { + performance.actor { + ActorName as name@en + } + # Stars is a uid-to-value map mapping + # starring edges to performance.actor names + Stars as min(val(ActorName)) + } + } + } + + movies(func: uid(spielbergMovies)) @cascade { + name@en + director.film (orderasc: name@en, first: 1) { + name@en + starring (orderasc: val(Stars)) { + performance.actor { + name@en + } + } + } + } +} +``` + + + +## Obtain unique results by using variables + +To obtain unique results, assign the node's edge to a variable. +The variable can now be used to iterate over the unique nodes. + +Example: Get all unique genres from all of the movies directed by Steven Spielberg. + + + +```dql +{ + var(func: eq(name@en, "Steven Spielberg")) { + director.film { + genres as genre + } + } + + q(func: uid(genres)) { + name@. + } +} +``` + + + +## Usage of checkpwd boolean + +Store the result of `checkpwd` in a query variable and then match it against `1` (`checkpwd` is `true`) or `0` (`checkpwd` is `false`). + + + +```dql +{ + exampleData(func: has(email)) { + uid + email + check as checkpwd(pass, "1bdfhJHb!fd") + } + userMatched(func: eq(val(check), 1)) { + uid + email + } + userIncorrect(func: eq(val(check), 0)) { + uid + email + } +} +``` + + diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/upserts.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/upserts.md new file mode 100644 index 00000000..5818916a --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/dql/upserts.md @@ -0,0 +1,427 @@ +--- +title: Upsert +--- + +Upsert-style operations are operations where: + +1. A node is searched for, and then +2. Depending on if it is found or not, either: + - Updating some of its attributes, or + - Creating a new node with those attributes. + +The upsert has to be an atomic operation such that either a new node is +created, or an existing node is modified. It's not allowed that two concurrent +upserts both create a new node. + +There are many examples where upserts are useful. Most examples involve the +creation of a 1 to 1 mapping between two different entities. E.g. associating +email addresses with user accounts. + +Upserts are common in both traditional RDBMSs and newer NoSQL databases. +Dgraph is no exception. + +## Upsert Procedure + +In Dgraph, upsert-style behavior can be implemented by users on top of +transactions. The steps are as follows: + +1. Create a new transaction. + +2. Query for the node. This will usually be as simple as `{ q(func: eq(email, + "bob@example.com")) { uid }}`. If a `uid` result is returned, then that's the +`uid` for the existing node. If no results are returned, then the user account +doesn't exist. + +3. In the case where the user account doesn't exist, then a new node has to be + created. This is done in the usual way by making a mutation (inside the +transaction), e.g. the RDF `_:newAccount "bob@example.com" .`. The +`uid` assigned can be accessed by looking up the blank node name `newAccount` +in the `Assigned` object returned from the mutation. + +4. Now that you have the `uid` of the account (either new or existing), you can + modify the account (using additional mutations) or perform queries on it in +whichever way you wish. + +## Upserts in DQL and GraphQL + +You can also use the `Upsert Block` in DQL to achieve the upsert procedure in a single + mutation. The request will contain both the query and the mutation as explained +[here](dql-mutation#upsert-block). + +In GraphQL, you can use the `upsert` input variable in an `add` mutation, as explained [here](/graphql/mutations/upsert). + +## Conflicts + +Upsert operations are intended to be run concurrently, as per the needs of the +application. As such, it's possible that two concurrently running operations +could try to add the same node at the same time. For example, both try to add a +user with the same email address. If they do, then one of the transactions will +fail with an error indicating that the transaction was aborted. + +If this happens, the transaction is rolled back and it's up to the user's +application logic to retry the whole operation. The transaction has to be +retried in its entirety, all the way from creating a new transaction. + +The choice of index placed on the predicate is important for performance. +**Hash is almost always the best choice of index for equality checking.** + +:::note +It's the _index_ that typically causes upsert conflicts to occur. The index is +stored as many key/value pairs, where each key is a combination of the +predicate name and some function of the predicate value (e.g. its hash for the +hash index). If two transactions modify the same key concurrently, then one +will fail. +::: + +The upsert block contains one query block and mutation blocks. Variables defined +in the query block can be used in the mutation blocks using the `uid` and `val` function. + +The `uid` function allows extracting UIDs from variables defined in the query block. +There are two possible outcomes based on the results of executing the query block: + +* If the variable is empty i.e. no node matched the query, the `uid` function returns a new UID in case of a `set` operation and is thus treated similar to a blank node. On the other hand, for `delete/del` operation, it returns no UID, and thus the operation becomes a no-op and is silently ignored. A blank node gets the same UID across all the mutation blocks. +* If the variable stores one or more than one UIDs, the `uid` function returns all the UIDs stored in the variable. In this case, the operation is performed on all the UIDs returned, one at a time. + + +## Example of `uid` Function + +Consider an example with the following schema: + +```sh +curl localhost:8080/alter -X POST -d $' + name: string @index(term) . + email: string @index(exact, trigram) @upsert . + age: int @index(int) .' | jq +``` + +Now, let's say we want to create a new user with `email` and `name` information. +We also want to make sure that one email has exactly one corresponding user in +the database. To achieve this, we need to first query whether a user exists +in the database with the given email. If a user exists, we use its UID +to update the `name` information. If the user doesn't exist, we create +a new user and update the `email` and `name` information. + +We can do this using the upsert block as follows: + +```sh +curl -H "Content-Type: application/rdf" -X POST localhost:8080/mutate?commitNow=true -d $' +upsert { + query { + q(func: eq(email, "user@company1.io")) { + v as uid + name + } + } + + mutation { + set { + uid(v) "first last" . + uid(v) "user@company1.io" . + } + } +}' | jq +``` + +Result: + +```json +{ + "data": { + "q": [], + "code": "Success", + "message": "Done", + "uids": { + "uid(v)": "0x1" + } + }, + "extensions": {...} +} +``` + +The query part of the upsert block stores the UID of the user with the provided email +in the variable `v`. The mutation part then extracts the UID from variable `v`, and +stores the `name` and `email` information in the database. If the user exists, +the information is updated. If the user doesn't exist, `uid(v)` is treated +as a blank node and a new user is created as explained above. + +If we run the same mutation again, the data would just be overwritten, and no new uid is +created. Note that the `uids` map is empty in the result when the mutation is executed +again and the `data` map (key `q`) contains the uid that was created in the previous upsert. + +```json +{ + "data": { + "q": [ + { + "uid": "0x1", + "name": "first last" + } + ], + "code": "Success", + "message": "Done", + "uids": {} + }, + "extensions": {...} +} +``` + +We can achieve the same result using `json` dataset as follows: + +```sh +curl -H "Content-Type: application/json" -X POST localhost:8080/mutate?commitNow=true -d ' +{ + "query": "{ q(func: eq(email, \"user@company1.io\")) {v as uid, name} }", + "set": { + "uid": "uid(v)", + "name": "first last", + "email": "user@company1.io" + } +}' | jq +``` + +Now, we want to add the `age` information for the same user having the same email +`user@company1.io`. We can use the upsert block to do the same as follows: + +```sh +curl -H "Content-Type: application/rdf" -X POST localhost:8080/mutate?commitNow=true -d $' +upsert { + query { + q(func: eq(email, "user@company1.io")) { + v as uid + } + } + + mutation { + set { + uid(v) "28" . + } + } +}' | jq +``` + +Result: + +```json +{ + "data": { + "q": [ + { + "uid": "0x1" + } + ], + "code": "Success", + "message": "Done", + "uids": {} + }, + "extensions": {...} +} +``` + +Here, the query block queries for a user with `email` as `user@company1.io`. It stores +the `uid` of the user in variable `v`. The mutation block then updates the `age` of the +user by extracting the uid from the variable `v` using `uid` function. + +We can achieve the same result using `json` dataset as follows: + +```sh +curl -H "Content-Type: application/json" -X POST localhost:8080/mutate?commitNow=true -d $' +{ + "query": "{ q(func: eq(email, \\"user@company1.io\\")) {v as uid} }", + "set":{ + "uid": "uid(v)", + "age": "28" + } +}' | jq +``` + +If we want to execute the mutation only when the user exists, we could use +[Conditional Upsert](dql-mutation#conditional-upsert). + + + +## Bulk Delete Example + +Let's say we want to delete all the users of `company1` from the database. This can be +achieved in just one query using the upsert block as follows: + +```sh +curl -H "Content-Type: application/rdf" -X POST localhost:8080/mutate?commitNow=true -d $' +upsert { + query { + v as var(func: regexp(email, /.*@company1.io$/)) + } + + mutation { + delete { + uid(v) * . + uid(v) * . + uid(v) * . + } + } +}' | jq +``` + +We can achieve the same result using `json` dataset as follows: + +```sh +curl -H "Content-Type: application/json" -X POST localhost:8080/mutate?commitNow=true -d '{ + "query": "{ v as var(func: regexp(email, /.*@company1.io$/)) }", + "delete": { + "uid": "uid(v)", + "name": null, + "email": null, + "age": null + } +}' | jq +``` +## val function +Variables defined in the query block can be used in the mutation blocks using the `uid` and `val` function. + + +The `val` function allows extracting values from value variables. Value variables store +a mapping from UIDs to their corresponding values. Hence, `val(v)` is replaced by the value +stored in the mapping for the UID (Subject) in the N-Quad. If the variable `v` has no value +for a given UID, the mutation is silently ignored. The `val` function can be used with the +result of aggregate variables as well, in which case, all the UIDs in the mutation would +be updated with the aggregate value. + + +### Example of `val` Function + +Let's say we want to migrate the predicate `age` to `other`. We can do this using the +following mutation: + +```sh +curl -H "Content-Type: application/rdf" -X POST localhost:8080/mutate?commitNow=true -d $' +upsert { + query { + v as var(func: has(age)) { + a as age + } + } + + mutation { + # we copy the values from the old predicate + set { + uid(v) val(a) . + } + + # and we delete the old predicate + delete { + uid(v) * . + } + } +}' | jq +``` + +Result: + +```json +{ + "data": { + "code": "Success", + "message": "Done", + "uids": {} + }, + "extensions": {...} +} +``` + +Here, variable `a` will store a mapping from all the UIDs to their `age`. The mutation +block then stores the corresponding value of `age` for each UID in the `other` predicate +and deletes the `age` predicate. + +We can achieve the same result using `json` dataset as follows: + +```sh +curl -H "Content-Type: application/json" -X POST localhost:8080/mutate?commitNow=true -d $'{ + "query": "{ v as var(func: regexp(email, /.*@company1.io$/)) }", + "delete": { + "uid": "uid(v)", + "age": null + }, + "set": { + "uid": "uid(v)", + "other": "val(a)" + } +}' | jq +``` +## External ids +The upsert block makes managing external IDs easy. + +Set the schema. +``` +xid: string @index(exact) . +: string @index(exact) . +: [uid] @reverse . +``` + +Set the type first of all. +``` +{ + set { + _:blank "http://schema.org/Person" . + _:blank "ExternalType" . + } +} +``` + +Now you can create a new person and attach its type using the upsert block. +``` + upsert { + query { + var(func: eq(xid, "http://schema.org/Person")) { + Type as uid + } + var(func: eq(, "Robin Wright")) { + Person as uid + } + } + mutation { + set { + uid(Person) "https://www.themoviedb.org/person/32-robin-wright" . + uid(Person) uid(Type) . + uid(Person) "Robin Wright" . + uid(Person) "Person" . + } + } + } +``` + +You can also delete a person and detach the relation between Type and Person Node. It's the same as above, but you use the keyword "delete" instead of "set". "`http://schema.org/Person`" will remain but "`Robin Wright`" will be deleted. + +``` + upsert { + query { + var(func: eq(xid, "http://schema.org/Person")) { + Type as uid + } + var(func: eq(, "Robin Wright")) { + Person as uid + } + } + mutation { + delete { + uid(Person) "https://www.themoviedb.org/person/32-robin-wright" . + uid(Person) uid(Type) . + uid(Person) "Robin Wright" . + uid(Person) "Person" . + } + } + } +``` + +Query by user. +``` +{ + q(func: eq(, "Robin Wright")) { + uid + xid + + { + uid + xid + } + } +} +``` \ No newline at end of file diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/installation/configuration/change-data-capture.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/installation/configuration/change-data-capture.md new file mode 100644 index 00000000..8df678d3 --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/installation/configuration/change-data-capture.md @@ -0,0 +1,99 @@ +--- +title: Change Data Capture +description: Stream database mutations and drop events to Kafka or local file sinks +--- + +:::note +**Enterprise Feature**: Change Data Capture requires a Dgraph Enterprise license. See [License](license) for details. +::: + +Change Data Capture (CDC) streams database mutations and drop events to external sinks (Kafka or local files). CDC tracks all `set` and `delete` mutations except those affecting password fields, along with all drop events. Live Loader events are recorded; Bulk Loader events are not. + +CDC events are based on Raft log changes. If the sink is unreachable by the Alpha leader, Raft logs expand as events accumulate until the sink becomes available. Enable CDC on all Alpha nodes to avoid interruptions in the event stream. + +## Enable CDC with Kafka + +Kafka records CDC events under the `dgraph-cdc` topic. Create the topic before events are sent to the broker. + +Start Dgraph Alpha with the `--cdc` option: + +```bash +dgraph alpha --cdc "kafka=kafka-hostname:port; sasl-user=tstark; sasl-password=m3Ta11ic" +``` + +For localhost Kafka without SASL authentication: + +```bash +dgraph alpha --cdc "localhost:9092" +``` + +For TLS-enabled Kafka clusters, the `ca-cert` option is required. The certificate can be self-signed. + +## Enable CDC with File Sink + +To stream CDC events to a local unencrypted file, start Dgraph Alpha with: + +```bash +dgraph alpha --cdc "file=local-file-path" +``` + +## Command Reference + +The `--cdc` option supports the following sub-options: + +| Sub-option | Example `dgraph alpha` command option | Notes | +|------------------|-------------------------------------------|----------------------------------------------------------------------| +| `tls` | `--tls=false` | boolean flag to enable/disable TLS while connecting to Kafka. | +| `ca-cert` | `--cdc "ca-cert=/cert-dir/ca.crt"` | Path and filename of the CA root certificate used for TLS encryption, if not specified, Dgraph uses system certs if `tls=true` | +| `client-cert` | `--cdc "client-cert=/c-certs/client.crt"` | Path and filename of the client certificate used for TLS encryption | +| `client-key` | `--cdc "client-cert=/c-certs/client.key"` | Path and filename of the client certificate private key | +| `file` | `--cdc "file=/sink-dir/cdc-file"` | Path and filename of a local file sink (alternative to Kafka sink) | +| `kafka` | `--cdc "kafka=kafka-hostname; sasl-user=tstark; sasl-password=m3Ta11ic"` | Hostname(s) of the Kafka hosts. May require authentication using the `sasl-user` and `sasl-password` sub-options | +| `sasl-user` | `--cdc "kafka=kafka-hostname; sasl-user=tstark; sasl-password=m3Ta11ic"` | SASL username for Kafka. Requires the `kafka` and `sasl-password` sub-options | +| `sasl-password` | `--cdc "kafka=kafka-hostname; sasl-user=tstark; sasl-password=m3Ta11ic"` | SASL password for Kafka. Requires the `kafka` and `sasl-username` sub-options | +| `sasl-mechanism` | `--cdc "kafka=kafka-hostname; sasl-mechanism=PLAIN"` | The SASL mechanism for Kafka (PLAIN, SCRAM-SHA-256 or SCRAM-SHA-512). The default is PLAIN | + +## Data Format + +CDC events are in JSON format. Example: + +```json +{ "key": "0", "value": {"meta":{"commit_ts":5},"type":"mutation","event":{"operation":"set","uid":2,"attr":"counter.val","value":1,"value_type":"int"}}} +``` + +The `meta.commit_ts` value increases with each CDC event. Use this value to identify duplicate events that may occur due to Raft leadership changes. + +### Mutation Events + +**Set mutation:** + +```json +{"meta":{"commit_ts":29},"type":"mutation","event":{"operation":"set","uid":3,"attr":"counter.val","value":10,"value_type":"int"}} +``` + +**Delete mutation:** + +```json +{"meta":{"commit_ts":44},"type":"mutation","event":{"operation":"del","uid":7,"attr":"Author.name","value":"_STAR_ALL","value_type":"default"}} +``` + +### Drop Events + +**Drop all:** + +```json +{"meta":{"commit_ts":13},"type":"drop","event":{"operation":"all"}} +``` + +The `operation` field specifies the drop operation: `attribute`, `type`, `data`, or `all`. + +## Multi-Tenancy + +In multi-tenants environment, CDC events streamed to Kafka are distributed across Kafka partitions by the Kafka client based on the multi-tenancy namespace. + +## Limitations + +- CDC events track only new values, not old values updated or removed by mutations or drop operations +- Schema updates are not tracked +- CDC can only be configured when starting Alpha nodes with the `dgraph alpha` command +- Node crashes or Raft leadership changes may result in duplicate events, but no data loss diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/installation/configuration/enable-acl.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/installation/configuration/enable-acl.md new file mode 100644 index 00000000..2cb72ed5 --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/installation/configuration/enable-acl.md @@ -0,0 +1,78 @@ +--- +id: enable-acl +title: Enable ACL +--- + +Access Control List (ACL) provides access protection to your data stored in Dgraph. When the ACL feature is enabled, a client must authenticate with a username and password before executing any transactions, and is only allowed to access the data permitted by the ACL rules. + +:::note +**Enterprise Feature**: ACL requires a Dgraph Enterprise license. See [License](license) for details. +::: + +## Enable Enterprise ACL Feature + +1. Generate a data encryption key that is 32 bytes long: + + ```bash + tr -dc 'a-zA-Z0-9' < /dev/urandom | dd bs=1 count=32 of=enc_key_file + ``` + :::note + On a macOS you may have to use `LC_CTYPE=C; tr -dc 'a-zA-Z0-9' < /dev/urandom | dd bs=1 count=32 of=enc_key_file`. + ::: + +2. To view the secret key value use `cat enc_key_file`. +3. Create a plain text file named `hmac_secret_file`, and store a randomly generated `` in it. The secret key is used by Dgraph Alpha nodes to sign JSON Web Tokens (JWT). + + ```bash + echo '' > hmac_secret_file + ``` + +4. Start all the Dgraph Alpha nodes in your cluster with the option `--acl secret-file="/path/to/secret"`, and make sure that they are all using the same secret key file created in Step 1. Alternatively, you can [store the secret in Hashicorp Vault](#storing-acl-secret-in-hashicorp-vault). + + ```bash + dgraph alpha --acl "secret-file=/path/to/secret" --security "whitelist=" + ``` + + +## Storing ACL Secret in Hashicorp Vault + +You can save the ACL secret on [Hashicorp Vault](https://www.vaultproject.io/) server instead of saving the secret on the local file system. + +### Configuring a Hashicorp Vault Server + +Do the following to set up on the [Hashicorp Vault](https://www.vaultproject.io/) server for use with Dgraph: + +1. Ensure that the Vault server is accessible from Dgraph Alpha and configured using URL `http://fqdn[ip]:port`. +2. Enable [AppRole Auth method](https://www.vaultproject.io/docs/auth/approle) and enable [KV Secrets Engine](https://www.vaultproject.io/docs/secrets/kv). +3. Save the 256-bits (32 ASCII characters) long ACL secret in a KV Secret path ([K/V Version 1](https://www.vaultproject.io/docs/secrets/kv/kv-v1) or [K/V Version 2](https://www.vaultproject.io/docs/secrets/kv/kv-v2)). For example, you can upload this below to KV Secrets Engine Version 2 path of `secret/data/dgraph/alpha`: + ```json + { + "options": { + "cas": 0 + }, + "data": { + "hmac_secret_file": "" + } + } + ``` +4. Create or use a role with an attached policy that grants access to the secret. For example, the following policy would grant access to `secret/data/dgraph/alpha`: + ```hcl + path "secret/data/dgraph/*" { + capabilities = [ "read", "update" ] + } + ``` +5. Using the `role_id` generated from the previous step, create a corresponding `secret_id`, and copy the `role_id` and `secret_id` over to local files, like `./dgraph/vault/role_id` and `./dgraph/vault/secret_id`, that will be used by Dgraph Alpha nodes. + +:::tip +To learn more about the above steps, see [Dgraph Vault Integration: Docker](https://github.com/dgraph-io/dgraph/blob/main/contrib/config/vault/docker/README.md). +::: + +:::note +The key format for the `acl-field` option can be defined using `acl-format` with the values `base64` (default) or `raw`. +::: + +## Related Topics + +- [User Management and Access Control](../../admin/admin-tasks/user-management-access-control) - Manage users, groups, and ACL rules after enabling ACL + + diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/installation/configuration/encryption-at-rest.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/installation/configuration/encryption-at-rest.md new file mode 100644 index 00000000..0a131baf --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/installation/configuration/encryption-at-rest.md @@ -0,0 +1,111 @@ +--- +title: Encryption at Rest +description: Encrypt data stored on disk using AES encryption +--- + +:::note +**Enterprise Feature**: Encryption at Rest requires a Dgraph Enterprise license. See [License](license) for details. +::: + +Encryption at Rest encrypts data stored on disk, ensuring sensitive data is not readable without a valid decryption key. Dgraph uses the [Advanced Encryption Standard (AES)](https://en.wikipedia.org/wiki/Advanced_Encryption_Standard) algorithm for encryption. + +Encryption keys can be stored on Hashicorp Vault servers in addition to local file systems. + +## Setup + +To enable encryption, pass a file containing the data encryption key using the `--encryption key-file=value` option. The key size must be 16, 24, or 32 bytes, determining the AES block size: AES-128, AES-192, or AES-256, respectively. + +Generate an encryption key file (set `count` to the desired key size): + +```bash +tr -dc 'a-zA-Z0-9' < /dev/urandom | dd bs=1 count=32 of=enc_key_file +``` + +:::note +On macOS, use `LC_CTYPE=C; tr -dc 'a-zA-Z0-9' < /dev/urandom | dd bs=1 count=32 of=enc_key_file`. To view the key, use `cat enc_key_file`. +::: + +Alternatively, use the `--vault` [superflag](../../cli/superflags) options to enable encryption with Hashicorp Vault, as [explained below](#hashicorp-vault-configuration). + +## Enable Encryption + +Start Zero and Alpha with encryption enabled: + +```bash +dgraph zero --my="localhost:5080" --replicas 1 --raft "idx=1" +dgraph alpha --encryption key-file="./enc_key_file" --my="localhost:7080" --zero="localhost:5080" +``` + +If multiple Alpha nodes are in the cluster, pass the `--encryption key-file` option to each Alpha. + +Once encryption is enabled on an Alpha, the encryption key must be provided to start the server. If the Alpha restarts, the `--encryption key-file` option must be set with the key to restart successfully. + +### Hashicorp Vault Configuration + +You can store the encryption key in [Hashicorp Vault](https://www.vaultproject.io/) K/V Secrets instead of a local file. + +**Prerequisites:** + +1. Ensure the Vault server is accessible from Dgraph Alpha and configured using URL `http://fqdn[ip]:port`. +2. Enable [AppRole Auth method](https://www.vaultproject.io/docs/auth/approle) and [KV Secrets Engine](https://www.vaultproject.io/docs/secrets/kv). +3. Save the encryption key (16, 24, or 32 bytes) in a KV Secret path ([K/V Version 1](https://www.vaultproject.io/docs/secrets/kv/kv-v1) or [K/V Version 2](https://www.vaultproject.io/docs/secrets/kv/kv-v2)). For example, upload to KV Secrets Engine Version 2 path `secret/data/dgraph/alpha`: + ```json + { + "options": { + "cas": 0 + }, + "data": { + "enc_key": "qIvHQBVUpzsOp74PmMJjHAOfwIA1e6zm%" + } + } + ``` +4. Create or use a role with an attached policy that grants access to the secret. For example, the following policy grants access to `secret/data/dgraph/alpha`: + ```hcl + path "secret/data/dgraph/*" { + capabilities = [ "read", "update" ] + } + ``` +5. Using the `role_id` from the previous step, create a corresponding `secret_id`, and copy both to local files (e.g., `./dgraph/vault/role_id` and `./dgraph/vault/secret_id`) for use by Dgraph Alpha nodes. + +:::note +The key format for the `enc-field` option can be defined using `enc-format` with values `base64` (default) or `raw`. +::: + +### Example: Using Hashicorp Vault + +Start Dgraph with a Vault server holding the encryption key: + +```bash +## Start Dgraph Zero in a separate terminal +dgraph zero --my=localhost:5080 --replicas 1 --raft "idx=1" + +## Start Dgraph Alpha in a separate terminal +dgraph alpha --my="localhost:7080" --zero="localhost:5080" \ + --vault addr="http://localhost:8200";enc-field="enc_key";enc-format="raw";path="secret/data/dgraph/alpha";role-id-file="./role_id";secret-id-file="./secret_id" +``` + +If multiple Alpha nodes are in the cluster, pass the `--encryption key-file` flag or the `--vault` superflag with appropriate options to each Alpha. + +After encryption is enabled on an Alpha, you must provide the encryption key to start the server. If the Alpha restarts, the `--encryption key-file` or `--vault` superflag options must be set with the key to restart successfully. + +## Disable Encryption + +Use [live loader](../../migration/live-loader) or [bulk loader](../../migration/bulk-loader) to decrypt data during import. + +## Key Rotation + +The master encryption key set by `--encryption key-file` (or stored in Vault) does not change automatically. The master key encrypts underlying data keys, which are rotated automatically (see the [encryption-at-rest blog post][encblog] for details). + +[encblog]: https://dgraph.io/blog/post/encryption-at-rest-dgraph-badger#one-key-to-rule-them-all-many-keys-to-find-them + +To rotate the master encryption key, use the `badger rotate` command on both `p` and `w` directories for each Alpha. In HA cluster configurations, rotate keys one Alpha at a time in a rolling manner to maintain availability. + +You need both the current key and the new key in separate files. Specify the directory to rotate (`p` or `w`) with `--dir`, the old key with `--old-key-path`, and the new key with `--new-key-path`: + +```bash +badger rotate --dir p --old-key-path enc_key_file --new-key-path new_enc_key_file +badger rotate --dir w --old-key-path enc_key_file --new-key-path new_enc_key_file +``` + +Then start Alpha with the `new_enc_key_file` to use the new key. + diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/installation/configuration/learner-nodes.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/installation/configuration/learner-nodes.md new file mode 100644 index 00000000..bd71ddf4 --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/installation/configuration/learner-nodes.md @@ -0,0 +1,31 @@ +--- +title: Learner Nodes +description: Deploy read-only replica instances for low-latency best-effort queries in remote geographic regions +--- +:::note +**Enterprise Feature**: Learner nodes require a Dgraph Enterprise license. See [License](license) for details. +::: + +Learner nodes are read-only replica instances that serve best-effort queries with zero latency overhead. Use learner nodes to provide low-latency access for clients in remote geographic regions distant from your main Dgraph cluster. + +A learner node receives all updates from its Alpha group leader without participating in Raft elections. It can accept read and write operations, but write operations are forwarded to the Alpha group leader and incur network latency to the main cluster. + +## Best-Effort Queries + +Best-effort queries use eventual consistency and return data available on the learner node at a timestamp that may not be the latest. They do not contact Zero nodes to get the latest timestamp, providing instant responses for geographically distributed clients. + +You can also send strict consistency queries to a learner node, but these incur additional latency as they must reach the Zero leader. At least one Alpha leader must be available for the learner node to serve normal queries. + +## Setup + +Start all nodes (Dgraph Zero leader and Dgraph Alpha leader) with the `--my` flag so they are accessible to the learner node. Then start an Alpha instance as a learner node: + +```sh +dgraph alpha --raft="learner=true; group=N" --my :5080 +``` + +This creates a replica that receives updates from group "N" leader without participating in Raft elections. + +:::note +You must specify the `--my` flag for Dgraph Zero, the Dgraph Alpha leader, and the learner node. Omitting it results in an error: `Error during SubscribeForUpdates`. +::: diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/installation/configuration/license.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/installation/configuration/license.md new file mode 100644 index 00000000..55fa9038 --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/installation/configuration/license.md @@ -0,0 +1,47 @@ +--- +title: License +description: Apply and manage Dgraph Enterprise licenses +--- + +Dgraph enterprise features are proprietary licensed under the [Dgraph Community License][dcl]. All Dgraph releases contain proprietary code for enterprise features. Enabling these features requires an enterprise contract from [contact@dgraph.io](mailto:contact@dgraph.io) or the [discuss forum](https://discuss.dgraph.io). + +**Dgraph enterprise features are enabled by default for 30 days in a new cluster.** After the 30-day trial period, the cluster must obtain a license from Dgraph to continue using enterprise features. + +:::note +At the conclusion of your 30-day trial period, if a license has not been applied to the cluster, access to enterprise features will be suspended. The cluster will continue to operate without enterprise features. +::: + +## Apply License + +Apply an enterprise license key to the cluster using one of the following methods: + +**HTTP endpoint** (POST request to any Zero server): + +```sh +curl -X POST localhost:6080/enterpriseLicense --upload-file ./licensekey.txt +``` + +**Command-line flag** (useful for automation): + +```sh +dgraph zero --enterprise_license ./licensekey.txt +``` + +## License Expiry Warnings + +Dgraph prints warning messages in the logs when your license is about to expire. If you implement log monitoring, configure alerts for these patterns. + +**Before expiry:** + +```sh +Your enterprise license will expire in 6 days from now. To continue using enterprise features after 6 days from now, apply a valid license. To get a new license, contact us at https://dgraph.io/contact. +``` + +**After expiry:** + +```sh +Your enterprise license has expired and enterprise features are disabled. To continue using enterprise features, apply a valid license. To receive a new license, contact us at https://dgraph.io/contact. +``` + +[dcl]: https://github.com/dgraph-io/dgraph/blob/main/licenses/DCL.txt + diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/installation/configuration/restrict-mutation-operations.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/installation/configuration/restrict-mutation-operations.md new file mode 100644 index 00000000..67675166 --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/installation/configuration/restrict-mutation-operations.md @@ -0,0 +1,35 @@ +--- +title: Read only and Strict mode +--- + +You can control mutation operations using the `--limit` flag with the `mutations` parameter. There are three modes available: `allow`, `disallow`, and `strict`. + +## Mutation Modes + +### Allow (Default) + +The default mode is `allow`. In this mode: +- Users can read and mutate data +- Mutations can be performed on predicates that don't exist in the DQL schema +- If a predicate in a mutation doesn't exist in the schema, it gets automatically added to the schema with an appropriate [Dgraph Type](../../dql/dql-schema) + +No configuration is needed as this is the default behavior. + +### Disallow + +The `disallow` mode disables all mutation operations. This makes the database read-only. + +```sh +dgraph alpha --limit "mutations=disallow;" +``` + +### Strict + +The `strict` mode enforces schema validation for mutations: +- Mutations are only allowed on predicates that are already declared in the schema +- Before performing a mutation on a predicate that doesn't exist in the schema, you must first perform an alter operation to add that predicate and its schema type + +```sh +dgraph alpha --limit "mutations=strict" +``` + diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/installation/deployment-patterns.mdx b/docusaurus-docs/docs_versioned_docs/version-v25.4/installation/deployment-patterns.mdx new file mode 100644 index 00000000..ae08e150 --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/installation/deployment-patterns.mdx @@ -0,0 +1,258 @@ +--- +title: Deployment Patterns +--- + +This guide covers different Dgraph deployment patterns, from simple development setups to production-grade highly available clusters. + +## Pattern Selection Guide + +| Pattern | Use Case | Nodes | HA | Sharding | +|---------|----------|-------|----|---------| +| Basic | Dev/Test environments, non critical production | 1 Zero, 1 Alpha | ❌ | ❌ | +| HA | Production, <1TB | 3 Zeros, 3 Alphas | ✅ | ❌ | +| Distributed | Dev/Test environments for large dataset| 1 Zero, 2+ Alphas | ❌ | ✅ | +| Distributed HA | Large production, >10TB | 3 Zeros, 6+ Alphas | ✅ | ✅ | + +--- +> **Getting Started?** For first-time users and local development, see the [Learning Environment](/installation/single-host-setup) guide, which covers Docker standalone and Docker Compose setups with Ratel UI. +--- + +## Basic cluster + +**Best for:** Development teams, staging environments, CI/CD + +### Architecture + +``` +┌──────────────┐ +│ Dgraph Zero │ :5080, :6080 +└──────┬───────┘ + │ +┌──────▼───────┐ +│ Dgraph Alpha │ :7080, :8080, :9080 +└──────────────┘ +``` + + + +Refer to [Basic Cluster](/installation/single-host-setup) instructions. + +--- +## HA cluster +**Best for:** Production workloads up to 10TB +### Architecture +``` +Zero Cluster (3 nodes) - Raft Group 0 + ├─ Zero 1 :5080 (Leader) + ├─ Zero 2 :5080 (Follower) + └─ Zero 3 :5080 (Follower) + │ +Alpha Group 1 (3 replicas) - Raft Group 1 + ├─ Alpha 1 :7080, :8080, :9080 (Leader) + ├─ Alpha 2 :7080, :8080, :9080 (Follower) + └─ Alpha 3 :7080, :8080, :9080 (Follower) +``` +### Setup Steps +**1. Start Zero Cluster:** +```sh +# Zero 1 (on host1) - First Zero initializes the cluster +dgraph zero --my=host1:5080 --raft "idx=1" --replicas=3 +# Zero 2 (on host2) - Uses --peer to join existing cluster +dgraph zero --my=host2:5080 --raft "idx=2" --peer=host1:5080 +# Zero 3 (on host3) - Uses --peer to join existing cluster +dgraph zero --my=host3:5080 --raft "idx=3" --peer=host1:5080 +``` +**Important Notes:** +- **Raft IDs**: Each Zero node must have a unique Raft ID set via `--raft "idx=N"`. Dgraph does not auto-assign Raft IDs to Zero nodes. +- **Cluster Initialization**: The first Zero node starts the cluster. All subsequent Zero nodes must use `--peer=` to join the existing cluster. If `--peer` is omitted, a new independent cluster will be created. +- **Replication**: The `--replicas=3` flag on Zero controls how many Alpha replicas will be in each Alpha group. +**2. Start Alpha Cluster:** +```sh +# Alpha 1 (on host1) +dgraph alpha --my=host1:7080 --zero=host1:5080,host2:5080,host3:5080 +# Alpha 2 (on host2) +dgraph alpha --my=host2:7080 --zero=host1:5080,host2:5080,host3:5080 +# Alpha 3 (on host3) +dgraph alpha --my=host3:7080 --zero=host1:5080,host2:5080,host3:5080 +``` +**Important Notes:** +- **Zero Connection**: Alphas can connect to any Zero in the cluster; list all Zeros for redundancy. +- **Group Assignment**: Zero automatically assigns Alphas to groups based on the `--replicas` setting. With `--replicas=3`, the first 3 Alphas join Group 1. +- **Alpha Raft IDs**: Unlike Zero nodes, Alpha nodes receive their Raft IDs automatically from Zero. +### Kubernetes (Helm) +```sh +helm repo add dgraph https://charts.dgraph.io +helm install my-dgraph dgraph/dgraph \ + --set zero.replicaCount=3 \ + --set alpha.replicaCount=3 +``` +**Characteristics:** +- Tolerates 1 node failure (in each group) +- All data replicated 3x +- No sharding (all predicates on all Alphas) +- Suitable for datasets up to ~1TB +**Pros:** High availability, automatic failover +**Cons:** Storage scales vertically only +--- + +## Distributed (Multi-Group) - Basic +Sharding, No HA + +**Best for:** Development with large datasets (>10TB) + +### Architecture + +``` +┌──────────────┐ +│ Dgraph Zero │ :5080 +└──────┬───────┘ + │ + ├─ Group 1: Alpha 1 :7080 + ├─ Group 2: Alpha 2 :7081 (port offset) + └─ Group 3: Alpha 3 :7082 (port offset) +``` + +### Setup (Single Host with Port Offsets) + +```sh +# Start Zero with replicas=1 (no replication) +dgraph zero --my=localhost:5080 --replicas=1 + +# Start Alpha nodes with port offsets +dgraph alpha --my=localhost:7080 --zero=localhost:5080 -p data/p1 -w data/w1 +dgraph alpha --my=localhost:7081 --zero=localhost:5080 -p data/p2 -w data/w2 -o 1 +dgraph alpha --my=localhost:7082 --zero=localhost:5080 -p data/p3 -w data/w3 -o 2 +``` + +**Characteristics:** +- 3 Alpha groups (no replication within groups) +- Data sharded by predicate across groups +- Horizontal storage scaling +- No fault tolerance + +**Pros:** Horizontal scalability, handles large datasets +**Cons:** No HA, any node failure loses data + +--- +## Distributed - HA (Production Large-Scale) +**Best for:** Production workloads >10TB, high traffic, mission-critical +### Architecture +``` +Zero Cluster (3 nodes) + └─ Replicates cluster metadata + │ + ├─ Group 1: Alpha 1,2,3 (3 replicas) + │ └─ Predicates: name, age, email + │ + ├─ Group 2: Alpha 4,5,6 (3 replicas) + │ └─ Predicates: friend, follows + │ + └─ Group 3: Alpha 7,8,9 (3 replicas) + └─ Predicates: location, company +``` +### Setup (9 Alpha Nodes across 3 Hosts) +**Zeros (3 nodes):** +```sh +# Host 1: Zero 1 +dgraph zero --my=host1:5080 --raft "idx=1" --replicas=3 +# Host 2: Zero 2 +dgraph zero --my=host2:5080 --raft "idx=2" --peer=host1:5080 +# Host 3: Zero 3 +dgraph zero --my=host3:5080 --raft "idx=3" --peer=host1:5080 +``` +**Alphas (3 groups × 3 replicas = 9 nodes):** +```sh +# Host 1: Alphas 1, 4, 7 +dgraph alpha --my=host1:7080 --zero=host1:5080,host2:5080,host3:5080 -p p1 -w w1 +dgraph alpha --my=host1:7081 --zero=host1:5080,host2:5080,host3:5080 -p p4 -w w4 -o 1 +dgraph alpha --my=host1:7082 --zero=host1:5080,host2:5080,host3:5080 -p p7 -w w7 -o 2 +# Host 2: Alphas 2, 5, 8 +dgraph alpha --my=host2:7080 --zero=host1:5080,host2:5080,host3:5080 -p p2 -w w2 +dgraph alpha --my=host2:7081 --zero=host1:5080,host2:5080,host3:5080 -p p5 -w w5 -o 1 +dgraph alpha --my=host2:7082 --zero=host1:5080,host2:5080,host3:5080 -p p8 -w w8 -o 2 +# Host 3: Alphas 3, 6, 9 +dgraph alpha --my=host3:7080 --zero=host1:5080,host2:5080,host3:5080 -p p3 -w w3 +dgraph alpha --my=host3:7081 --zero=host1:5080,host2:5080,host3:5080 -p p6 -w w6 -o 1 +dgraph alpha --my=host3:7082 --zero=host1:5080,host2:5080,host3:5080 -p p9 -w w9 -o 2 +``` +**Group Assignment:** +- Zero automatically assigns Alphas 1,2,3 → Group 1 +- Zero assigns Alphas 4,5,6 → Group 2 +- Zero assigns Alphas 7,8,9 → Group 3 +**Characteristics:** +- 3 groups with 3x replication each +- Tolerates 1 node failure per group +- Data sharded across groups +- All predicates replicated 3x within their group +**Pros:** Maximum scalability and availability +**Cons:** Higher operational complexity, more resources +--- + +## Configuration Flags Reference + +### Common Flags + +| Flag | Component | Description | Default | +|------|-----------|-------------|---------| +| `--my` | Zero/Alpha | Address:port that other nodes connect to | `localhost:5080` (Zero `localhost:7080` (Alpha) | +| `--zero` | Alpha | Address(es) of Zero node(s) to connect to | Required | +| `--peer` | Zero | Address of existing Zero to join cluster | None (creates new cluster if omitted) | +| `--raft "idx=N"` | Zero | Unique Raft ID for Zero node (required for HA) | `1` | +| `--replicas` | Zero | Number of Alpha replicas per group | `1` | +| `-w` / `--wal` | Zero/Alpha | Directory for write-ahead log entries | `zw` (Zero) `w` (Alpha) | +| `-p` / `--postings` | Alpha | Directory for data storage | `p` | +| `--bindall` | Zero/Alpha | Bind to `0.0.0.0` for network access | `true` | +| `--v=2` | Zero/Alpha | Log verbosity level (recommended: 2) | `0` | + +**Configuration Methods:** +Flags can be set via command-line arguments, environment variables, or configuration files. See [Config](/cli/config) for details. + +## Best Practices + +### Node Placement + +1. **Different Physical Hosts**: Run each replica on a separate machine +2. **Availability Zones**: Distribute across 3 AZs when possible +3. **Network Latency**: Keep inter-node latency <5ms for best performance + +### Resource Planning + +| Deployment | CPUs/Node | RAM/Node | Disk/Node | +|------------|-----------|----------|-----------| +| Development | 2 cores | 4GB | 50GB | +| Small Production | 8 cores | 16GB | 250GB SSD | +| Large Production | 16 cores | 32GB | 1TB NVMe | + +### Scaling Strategy + +**Vertical First:** +1. Start with HA single-group (3 Alphas) +2. Increase CPU/RAM per node as load grows + +**Horizontal When:** +1. Dataset >1TB +2. Query latency increases despite vertical scaling +3. Need to isolate hot predicates + +**Add 3 Alphas at a time** to maintain replication factor + +--- +## Deployment Checklist +Before production deployment: +- [ ] Set `--replicas=3` on Zero nodes +- [ ] Configure persistent storage volumes +- [ ] Enable TLS for client connections +- [ ] Set up IP whitelisting for admin endpoints +- [ ] Configure monitoring (Prometheus/Grafana) +- [ ] Set up binary backups (Enterprise) +- [ ] Test failover scenarios +- [ ] Document cluster topology +- [ ] Plan capacity for 2x growth +--- + +## Next Steps + +- [Configure Security](/admin/security/) +- [Set Up Monitoring](/admin/observability/monitoring) +- [Production Checklist](/installation/production-checklist) +- [Administration Guide](/admin/) diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/installation/dgraph-architecture.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/installation/dgraph-architecture.md new file mode 100644 index 00000000..d0b7c3a4 --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/installation/dgraph-architecture.md @@ -0,0 +1,243 @@ +--- +title: Architecture +--- + +Dgraph is a distributed graph database built for horizontal scalability, high availability, and high performance. + +## Core Components + +A Dgraph cluster consists of two types of nodes working together: + +### Dgraph Zero (Control Plane) + +Zero nodes manage cluster coordination and metadata. Each cluster requires at least one Zero node. + +**Responsibilities:** +- **Cluster Membership**: Track which Alpha nodes are part of the cluster +- **Data Distribution**: Assign predicates to Alpha groups for balanced load +- **Transaction Coordination**: Allocate transaction timestamps and UIDs +- **Rebalancing**: Automatically redistribute data as the cluster scales +- **Schema Management**: Coordinate schema changes across the cluster + +**Ports:** +- `5080` - Internal gRPC (Alpha ↔ Zero communication, Live/Bulk Loader) +- `6080` - HTTP admin endpoint (cluster state, assignments) + +### Dgraph Alpha (Data Plane) + +Alpha nodes store data and serve queries. Clusters need at least one Alpha node. + +**Responsibilities:** +- **Data Storage**: Store graph data (nodes, edges, predicates) +- **Index Management**: Maintain indexes for efficient queries +- **Query Execution**: Process DQL and GraphQL queries +- **Mutation Handling**: Execute data mutations with ACID guarantees +- **Predicate Ownership**: Each Alpha group owns specific predicates + +**Ports:** +- `7080` - Internal gRPC (Alpha ↔ Alpha, Alpha ↔ Zero) +- `8080` - External HTTP (client queries, admin) +- `9080` - External gRPC (client connections) + +See [Admin Tasks](../admin/admin-tasks) for health monitoring, backup, export, and other operations. + +## Cluster Architecture + +### Minimum Cluster (Development) + +``` +┌─────────────┐ +│ Dgraph Zero │ :5080 +└──────┬──────┘ + │ +┌──────▼───────┐ +│ Dgraph Alpha │ :7080, :8080, :9080 +└──────────────┘ +``` + +**Use case:** Local development, testing +**Configuration:** 1 Zero, 1 Alpha +**Characteristics:** No HA, no sharding + +### High Availability Cluster + +``` +┌─────────────┐ ┌─────────────┐ ┌─────────────┐ +│ Zero Node 1 │ │ Zero Node 2 │ │ Zero Node 3 │ +│ (Leader) │───│ (Follower) │───│ (Follower) │ +└──────┬──────┘ └──────┬──────┘ └──────┬──────┘ + │ │ │ + └──────────────────┴──────────────────┘ + │ + ┌──────────────────┴──────────────────┐ + │ │ │ +┌──────▼───────┐ ┌──────▼──────┐ ┌──────▼──────┐ +│ Alpha Node 1 │ │ Alpha Node 2│ │ Alpha Node 3│ +│ (Leader) │───│ (Follower) │────│ (Follower) │ +│ Group 1 │ │ Group 1 │ │ Group 1 │ +└──────────────┘ └─────────────┘ └─────────────┘ +``` + +**Use case:** Production workloads +**Configuration:** 3 Zeros, 3 Alphas (replicas=3) +**Characteristics:** +- Tolerates 1 node failure per group +- All predicates replicated 3x +- No data sharding (single group) + +### Sharded HA Cluster + +``` + Zero Cluster (3 nodes) + │ + ┌───────────────────┼───────────────────┐ + │ │ │ + Group 1 Group 2 Group 3 +┌───────────────┐ ┌───────────────┐ ┌───────────────┐ +│ Alpha 1,2,3 │ │ Alpha 4,5,6 │ │ Alpha 7,8,9 │ +│ Predicates: │ │ Predicates: │ │ Predicates: │ +│ - name │ │ - friend │ │ - location │ +│ - age │ │ - email │ │ - bio │ +└───────────────┘ └───────────────┘ └───────────────┘ +``` + +**Use case:** Large datasets (>1TB), horizontal scaling +**Configuration:** 3 Zeros, 9 Alphas (3 groups × 3 replicas) +**Characteristics:** +- Data sharded across multiple groups +- Each group has 3x replication +- Horizontal scalability + +## Data Model + +### Predicate-Based Sharding + +Unlike traditional graph databases that shard by nodes, Dgraph shards by **predicates** (relationship types): + +- Each predicate is assigned to an Alpha group +- All Alpha nodes in a group serve the same predicates +- Queries for a predicate route to its owning group +- Cross-predicate queries are distributed across groups + +**Example:** +``` +Group 1: name, age, email +Group 2: friend, follows +Group 3: location, company +``` + +**Automatic Rebalancing:** + +Zero continuously monitors disk usage across groups and automatically rebalances predicates to maintain even distribution: + +- Runs rebalancing checks every 8-10 minutes +- Moves predicates from high-usage groups to lower-usage groups +- During a predicate move: + - The predicate becomes temporarily read-only + - Queries continue to be served normally + - Mutations are rejected and should be retried after the move completes +- Each additional Alpha allows Zero to further split and redistribute predicates + +### Replication and Consistency + +Dgraph uses **Raft consensus** for replication. A Dgraph cluster is divided into Raft groups: Zero nodes form group 0, and each Alpha shard is a subsequent numbered group (group 1, group 2, etc.). Nodes of the same type (all Zeros or all Alphas in a group) form a Raft group. + +**Raft Consensus:** +- Each Raft group elects a single leader among its peers +- Non-leader nodes are followers +- If the leader becomes unavailable, the group automatically elects a new leader +- Writes require majority (quorum) acknowledgment +- Linearizable reads and writes +- Snapshot isolation for transactions + +**Replica Configuration:** + +The `--replicas` flag on Zero controls how many nodes serve each Alpha group. Use odd numbers (1, 3, 5) to maintain proper quorum: + +- `--replicas=1`: No replication (single node per group) +- `--replicas=3`: 3 nodes per group, tolerates 1 node failure (recommended) +- `--replicas=5`: 5 nodes per group, tolerates 2 node failures + +:::tip +If the number of replicas in a Raft group is **2N + 1**, up to **N** nodes can go offline without any impact on reads or writes. With 3 replicas, 1 can fail; with 5 replicas, 2 can fail. +::: + +**HA Setup:** + +**Zero nodes:** Deploy 3 Zero nodes for fault tolerance. Assign a unique integer ID to each using `--raft idx`, and pass the address of any healthy Zero instance using `--peer`. + +**Alpha nodes:** Set `--replicas=3` on Zero. You can run as many Alpha nodes as needed. Manually set `--raft idx` or leave it empty for Zero to auto-assign an ID (persists in write-ahead log). New Alpha nodes automatically detect each other via Zero. If you don't have a proxy or load balancer for Zero, provide Zero addresses with `--zero=zero1,zero2,zero3`. + +Zero first attempts to replicate existing groups by assigning new Alphas to the same group. After a group reaches the `--replicas` count, Zero creates new groups. Ensure the number of Alpha nodes is a multiple of the replication setting (e.g., 3, 6, or 9 Alphas with `--replicas=3`). + +## Scaling Strategies + +### Vertical Scaling (Per-Node) +- Add CPU cores for higher concurrency +- Add RAM for larger working sets +- Use faster SSDs for better I/O + +### Horizontal Scaling + +**Add Replicas (No Sharding):** +Start with `--replicas=3`, add 3 more Alphas → still 1 group, increased replication (6x) + +**Add Groups (Sharding):** +Start with 3 Alphas (group 1), add 3 more Alphas → Zero creates group 2 and rebalances predicates + +**Best Practice:** Keep Alpha count as a multiple of `--replicas`. With `--replicas=3`: 6 Alphas = 2 groups, 9 Alphas = 3 groups. + +## Query Flow + +1. **Client connects** to any Alpha node (HTTP/gRPC) +2. **Alpha parses query** and identifies required predicates +3. **Local predicates** are queried directly +4. **Remote predicates** are fetched from other Alphas via distributed joins +5. **Results are merged** and returned to client + +**Performance:** N-hop queries require only N network hops, regardless of data size. + +## Operational Characteristics + +### Resource Requirements + +| Component | CPU | Memory | Disk IOPS | +|-----------|-----|--------|-----------| +| Alpha (prod) | 8+ cores | 16GB+ | 3000+ | +| Zero (prod) | 2-4 cores | 4GB | 1000+ | + +### Fault Tolerance + +With `--replicas=3`: +- **1 node down**: Cluster fully operational +- **2 nodes down**: Read-only mode (no quorum for writes) +- **3 nodes down**: Group unavailable + +### Backup and Recovery + +- **Binary backups** (Enterprise): Incremental, production-ready +- **Exports**: Full RDF/JSON exports via admin API +- **Point-in-time recovery** available with binary backups + +## Monitoring + +Key metrics to monitor: +- Raft health (`/health` endpoint) +- Disk usage per Alpha +- Query latency (p50, p95, p99) +- Transaction throughput +- Pending proposals (write backpressure) + +See [Monitoring](../admin/observability/monitoring) for Prometheus/Grafana setup. + +## Security Considerations + +- **Network Isolation**: Zero nodes can run in private network +- **TLS Encryption**: Enable for client connections and inter-node communication +- **Access Control**: Use ACL (Enterprise) for fine-grained permissions +- **IP Whitelisting**: Restrict admin endpoints to trusted IPs + +## Next Steps + +- [Choose a Deployment Pattern](deployment-patterns) + diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/installation/download.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/installation/download.md new file mode 100644 index 00000000..cb2980bb --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/installation/download.md @@ -0,0 +1,68 @@ +--- +title: Download +description: Download the images and source files to build and install for a production-ready Dgraph cluster +--- +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + + + + + + +You can obtain Dgraph binary for the latest version as well as previous releases using automatic install script, manual download, through Docker images or by building the binary from the open source code. + + + + +1. Install Docker. + +1. Pull the latest Dgraph image using docker: + ```sh + docker pull dgraph/dgraph:latest + ``` + To set up a [learning environment](single-host-setup), you may pull the [dgraph standalone](https://hub.docker.com/r/dgraph/standalone) image : + + ```sh + docker pull dgraph/standalone:latest + ``` +1. Verify that the image is downloaded: + + ```sh + docker images + ``` + + + + + + +On linux system, you can get the binary using the automatic script: +1. Download the Dgraph installation script to install Dgraph automatically: + ```sh + curl https://get.dgraph.io -sSf | bash + ``` + +1. Verify that it works fine, by running: + ``` + dgraph version + ``` + For more information about the various installation scripts that you can use, see [install scripts](https://github.com/dgraph-io/Install-Dgraph). + + +On linux system, you can download a tar file and install manually. +Download the appropriate tar for your platform from **[Dgraph releases](https://github.com/dgraph-io/dgraph/releases)**. After downloading the tar for your platform from Github, extract the binary to `/usr/local/bin` like so. + +1. Download the installation file: + ``` + $ sudo tar -C /usr/local/bin -xzf dgraph-linux-amd64-VERSION.tar.gz + ``` +1. Verify that it works fine, by running: + ``` + dgraph version + ``` + + +You can also build **Dgraph** and **Ratel UI** from the source code by following the instructions from [Contributing to Dgraph](https://github.com/dgraph-io/dgraph/blob/master/CONTRIBUTING.md) or [Building and running ratel](https://github.com/dgraph-io/ratel/blob/master/INSTRUCTIONS.md). + + diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/installation/ha-cluster/ha-cluster-k8s-kind.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/installation/ha-cluster/ha-cluster-k8s-kind.md new file mode 100644 index 00000000..3569c7fe --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/installation/ha-cluster/ha-cluster-k8s-kind.md @@ -0,0 +1,783 @@ +--- +title: Kubernetes (kind) +--- + +This guide walks you through installing a highly available Dgraph cluster on Kubernetes using [kind](https://kind.sigs.k8s.io/) (Kubernetes in Docker). kind is ideal for local development and testing, as it creates multi-node Kubernetes clusters that closely simulate production environments. + +## Architecture and Key Benefits + +### Cluster Architecture + +This setup creates a **highly available Dgraph cluster** with: + +- **1 control plane node** - Manages the Kubernetes cluster +- **3 worker nodes** - Run Dgraph workloads +- **3 Dgraph Zero pods** - One per worker node (cluster coordination) +- **3 Dgraph Alpha pods** - One per worker node (data storage and queries) + +Each worker node runs exactly one Zero and one Alpha pod, ensuring high availability while maximizing resource efficiency. + +### Storage Architecture + +For local development, data is mapped from your host machine to the kind cluster: + +``` +Host Machine (Mac/Linux) kind Worker Nodes Dgraph Pods +───────────────────────── ────────────────── ──────────── +$HOME/dgraph-data/ /dgraph-data/ /dgraph/ + ├── alpha-0/ ──────────────────> ├── alpha-0/ ─────────> alpha-0 pod + ├── alpha-1/ ──────────────────> ├── alpha-1/ ─────────> alpha-1 pod + └── alpha-2/ ──────────────────> └── alpha-2/ ─────────> alpha-2 pod +``` + +This allows you to directly inspect and experiment with Dgraph's data files, including the `p` folders (posting lists). + +### Key Benefits + +- **High Availability**: Each Zero and Alpha pod runs on a different node, ensuring fault tolerance +- **Resource Efficiency**: Uses only 3 nodes instead of 6, with one Zero and one Alpha per node +- **Production-Like**: Multi-node setup closely simulates real Kubernetes environments +- **Local Development**: Perfect for testing and development without cloud costs +- **Data Persistence**: Persistent storage ensures data survives pod restarts +- **Direct Data Access**: Local disk mapping allows direct inspection and experimentation with data files + +## Prerequisites + +Before you begin, ensure you have the following tools installed: + +- **Docker** - Running and accessible +- **kubectl** - Kubernetes command-line tool +- **Helm** - Kubernetes package manager +- **kind** - Kubernetes in Docker + +### Install Prerequisites + +On macOS, you can install these tools using Homebrew: + +```bash +brew install kind kubectl helm +``` + +Verify your installations: + +```bash +docker --version +kubectl version --client +helm version +kind --version +``` + +## Step 1: Create Local Data Directories + +Create directories on your host machine for each Alpha pod's data: + +```bash +mkdir -p $HOME/dgraph-data/alpha-0 +mkdir -p $HOME/dgraph-data/alpha-1 +mkdir -p $HOME/dgraph-data/alpha-2 +``` + +**Note:** Use absolute paths (e.g., `/Users/your-username/dgraph-data/alpha-0`). Shell shortcuts like `~` do not work in Kubernetes YAML. + +## Step 2: Create kind Cluster with Volume Mounts + +Create a kind cluster configuration that mounts your local data directory into all worker nodes. + +Create a file named `kind-config.yaml`: + +```yaml +kind: Cluster +apiVersion: kind.x-k8s.io/v1alpha4 +nodes: + - role: control-plane + - role: worker + extraMounts: + - hostPath: /Users/your-username/dgraph-data # Replace with your absolute path + containerPath: /dgraph-data + - role: worker + extraMounts: + - hostPath: /Users/your-username/dgraph-data # Replace with your absolute path + containerPath: /dgraph-data + - role: worker + extraMounts: + - hostPath: /Users/your-username/dgraph-data # Replace with your absolute path + containerPath: /dgraph-data +``` + +**Important:** Replace `/Users/your-username/dgraph-data` with your actual absolute path. You can use: + +```bash +echo $HOME/dgraph-data +``` + +Then create the cluster: + +```bash +kind create cluster --config kind-config.yaml +``` + +**What happens during cluster creation:** +- Docker containers are created for each node (1 control plane + 3 workers) +- Kubernetes control plane components are set up in the control plane node +- Worker nodes join the cluster and become ready +- Your local `dgraph-data` directory is mounted into each worker node at `/dgraph-data` +- A kubectl context named `kind-kind` is automatically created and set as default + +### Verify Cluster Creation + +Verify that your cluster is running correctly: + +```bash +kubectl get nodes +``` + +Expected output: + +``` +NAME STATUS ROLES AGE VERSION +kind-control-plane Ready control-plane 2m v1.27.x +kind-worker Ready 2m v1.27.x +kind-worker2 Ready 2m v1.27.x +kind-worker3 Ready 2m v1.27.x +``` + +All nodes should show `STATUS: Ready`. + +## Step 3: Create StorageClass + +Create a StorageClass for local persistent volumes: + +```bash +kubectl apply -f - < --type json -p '[{"op": "remove", "path": "/metadata/finalizers"}]' +``` + +**Solution - Force delete PV:** + +```bash +kubectl patch pv --type json -p '[{"op": "remove", "path": "/metadata/finalizers"}]' +``` + +#### Silent Storage Failures + +**Symptom:** PVCs remain in `Pending` state without obvious errors + +**Causes:** +- Missing StorageClass +- No available PVs matching the PVC requirements +- StorageClass `volumeBindingMode` set incorrectly + +**Solutions:** + +```bash +# Check if StorageClass exists +kubectl get storageclass + +# Check PVC status and events +kubectl describe pvc datadir-dgraph-dgraph-alpha-0 + +# Verify PVs are available +kubectl get pv + +# Ensure StorageClass name matches in Helm values +helm get values dgraph | grep storageClass +``` + +#### StatefulSet Won't Update + +**Cause:** StatefulSets don't allow changes to `volumeClaimTemplates` after creation + +**Solution:** Delete and recreate the StatefulSet (data in PVCs will persist): + +```bash +# Delete StatefulSet (pods will be recreated) +kubectl delete statefulset dgraph-dgraph-alpha + +# Or use Helm upgrade with new values +helm upgrade dgraph dgraph/dgraph -f new-values.yaml +``` + +### Accessing Pods for Debugging + +**Access pod shell:** + +```bash +kubectl exec -it dgraph-dgraph-alpha-0 -- /bin/sh +``` + +**Check logs:** + +```bash +# Current logs +kubectl logs dgraph-dgraph-alpha-0 + +# Follow logs +kubectl logs -f dgraph-dgraph-alpha-0 + +# Previous container logs (if pod restarted) +kubectl logs dgraph-dgraph-alpha-0 --previous +``` + +**Check volume mounts:** + +```bash +kubectl describe pod dgraph-dgraph-alpha-0 | grep -A 10 "Mounts" +``` + +**List files in data directory:** + +```bash +kubectl exec -it dgraph-dgraph-alpha-0 -- ls -la /dgraph +kubectl exec -it dgraph-dgraph-alpha-0 -- ls -la /dgraph/p +``` + +## Best Practices + +### Storage Configuration + +- **Use absolute paths** for `hostPath` volumes in PVs (e.g., `/Users/username/dgraph-data/alpha-0`, not `~/dgraph-data/alpha-0`) +- **Pre-create StorageClass and PVs** before installing the Helm chart +- **Test volume mounts** by writing a file manually to verify the mount works +- **Use `Retain` reclaim policy** on PVs to prevent accidental data loss during cleanup + +### Resource Management + +- **Set appropriate resource requests and limits** to avoid OOM crashes: + +```yaml +alpha: + resources: + requests: + cpu: "1" + memory: "2Gi" + limits: + cpu: "2" + memory: "4Gi" +``` + +### Namespace Isolation + +- **Use namespaces** to isolate releases and simplify cleanup: + +```bash +kubectl create namespace dgraph +helm install dgraph dgraph/dgraph -n dgraph -f dgraph-ha-values.yaml +``` + +### Dependencies + +- **Ensure Zero is running** before Alpha starts. Dgraph Alpha depends on Zero for cluster coordination. + +## Useful Commands Reference + +| Task | Command | +|------|---------| +| Check pod status | `kubectl get pods` | +| Check pod status with nodes | `kubectl get pods -o wide` | +| Access pod shell | `kubectl exec -it dgraph-dgraph-alpha-0 -- /bin/sh` | +| Check pod logs | `kubectl logs dgraph-dgraph-alpha-0` | +| Follow pod logs | `kubectl logs -f dgraph-dgraph-alpha-0` | +| Check volume mounts | `kubectl describe pod dgraph-dgraph-alpha-0 \| grep -A 10 "Mounts"` | +| Check PVC/PV binding | `kubectl get pvc,pv` | +| Check StorageClass | `kubectl get storageclass` | +| Port-forward service | `kubectl port-forward svc/dgraph-alpha-public 8080:8080` | +| Delete StatefulSet | `kubectl delete statefulset dgraph-dgraph-alpha` | +| Force delete PV | `kubectl patch pv --type json -p '[{"op": "remove", "path": "/metadata/finalizers"}]'` | +| Check pod events | `kubectl describe pod dgraph-dgraph-alpha-0` | +| Check node resources | `kubectl top nodes` | +| Check pod resources | `kubectl top pod dgraph-dgraph-alpha-0` | + +## Next Steps + +- Learn about [Dgraph configuration options](/admin/admin-tasks) +- Explore [production deployment patterns](/installation/deployment-patterns) +- Set up [monitoring and observability](/admin/observability) +- Configure [security and access control](/admin/security) diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/installation/ha-cluster/helm-chart.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/installation/ha-cluster/helm-chart.md new file mode 100644 index 00000000..eb66f247 --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/installation/ha-cluster/helm-chart.md @@ -0,0 +1,257 @@ +--- +title: Helm Chart +--- +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + + + +You can run three Dgraph Alpha servers and three Dgraph Zero servers in a highly available cluster setup. For a highly available setup, start the Dgraph Zero server with `--replicas 3` flag, so that all data is replicated on three Alpha servers and forms one Alpha group. You can install a highly available cluster using Helm charts. + +#### Before you begin + +* Install [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/). +* Ensure that you have a production-ready Kubernetes cluster with atleast three worker nodes running in a cloud provider of your choice. +* Install [Helm](https://helm.sh/docs/intro/install/). +* (Optional) To run Dgraph Alpha with TLS, see [TLS Configuration](/admin/security/tls-configuration). + +#### Installing a highly available Dgraph cluster using Helm + +1. Verify that you are able to access the nodes in the Kubernetes cluster: + + ```bash + kubectl get nodes + ``` + + An output similar to this appears: + + ```bash + NAME STATUS ROLES AGE VERSION + ..compute.internal Ready 1m v1.15.11-eks-af3caf + ..compute.internal Ready 1m v1.15.11-eks-af3caf + ..compute.internal Ready 1m v1.15.11-eks-af3caf + ``` + After your Kubernetes cluster is up and running, you can use of the [Dgraph Helm chart](https://github.com/dgraph-io/charts/) to install a highly available Dgraph cluster + +1. Add the Dgraph helm repository:: + + ```bash + helm repo add dgraph https://charts.dgraph.io + ``` +1. Install the chart with ``: + + ```bash + helm install dgraph/dgraph + ``` + + You can also specify the version using: + ```bash + helm install dgraph/dgraph --set image.tag="{{< version >}}" + ``` + When configuring the Dgraph image tag, be careful not to use `latest` or `main` in a production environment. These tags may have the Dgraph version change, causing a mixed-version Dgraph cluster that can lead to an outage and potential data loss. + + An output similar to this appears: + + ```bash + NAME: + LAST DEPLOYED: Wed Feb 1 21:26:32 2023 + NAMESPACE: default + STATUS: deployed + REVISION: 1 + TEST SUITE: None + NOTES: + 1. You have just deployed Dgraph, version 'v21.12.0'. + + For further information: + * Documentation: https://dgraph.io/docs/ + * Community and Issues: https://discuss.dgraph.io/ + 2. Get the Dgraph Alpha HTTP/S endpoint by running these commands. + export ALPHA_POD_NAME=$(kubectl get pods --namespace default --selector "statefulset.kubernetes.io/pod-name=-dgraph-alpha-0,release=-dgraph" --output jsonpath="{.items[0].metadata.name}") + echo "Access Alpha HTTP/S using http://localhost:8080" + kubectl --namespace default port-forward $ALPHA_POD_NAME 8080:8080 + + NOTE: Change "http://" to "https://" if TLS was added to the Ingress, Load Balancer, or Dgraph Alpha service. + ``` +1. Get the name of the Pods in the cluster using `kubectl get pods`: + ```bash + NAME READY STATUS RESTARTS AGE + -dgraph-alpha-0 1/1 Running 0 4m48s + -dgraph-alpha-1 1/1 Running 0 4m2s + -dgraph-alpha-2 1/1 Running 0 3m31s + -dgraph-zero-0 1/1 Running 0 4m48s + -dgraph-zero-1 1/1 Running 0 4m10s + -dgraph-zero-2 1/1 Running 0 3m50s + +1. Get the Dgraph Alpha HTTP/S endpoint by running these commands: + ```bash + export ALPHA_POD_NAME=$(kubectl get pods --namespace default --selector "statefulset.kubernetes.io/pod-name=-dgraph-alpha-0,release=-dgraph" --output jsonpath="{.items[0].metadata.name}") + echo "Access Alpha HTTP/S using http://localhost:8080" + kubectl --namespace default port-forward $ALPHA_POD_NAME 8080:8080 + ``` +#### Deleting the resources from the cluster + +1. Delete the Helm deployment using: + + ```sh + helm delete my-release + ``` +2. Delete associated Persistent Volume Claims: + + ```sh + kubectl delete pvc --selector release=my-release + ``` + + +### Dgraph configuration files + +You can create a Dgraph [Config](/cli/config) files for Alpha server and Zero server with Helm chart configuration values, ``. For more information about the values, see the latest [configuration settings](https://github.com/dgraph-io/charts/blob/master/charts/dgraph/README.md#configuration). + +1. Open an editor of your choice and create a config file named `.yaml`: + +```yaml +# .yaml +alpha: + configFile: + config.yaml: | + alsologtostderr: true + badger: + compression_level: 3 + tables: mmap + vlog: mmap + postings: /dgraph/data/p + wal: /dgraph/data/w +zero: + configFile: + config.yaml: | + alsologtostderr: true + wal: /dgraph/data/zw +``` + +2. Change to the director in which you created ``.yaml and then install with Alpha and Zero configuration using: + +```sh +helm install dgraph/dgraph --values .yaml +``` + +### Exposing Alpha and Ratel Services + +By default Zero and Alpha services are exposed only within the Kubernetes cluster as +Kubernetes service type `ClusterIP`. + +In order to expose the Alpha service and Ratel service publicly you can use Kubernetes service type `LoadBalancer` or an Ingress resource. + + + + +##### Public Internet + +To use an external load balancer, set the service type to `LoadBalancer`. + +:::noteFor security purposes we recommend limiting access to any public endpoints, such as using a white list.::: + +1. To expose Alpha service to the Internet use: + +```sh +helm install dgraph/dgraph --set alpha.service.type="LoadBalancer" +``` + +2. To expose Alpha and Ratel services to the Internet use: + +```sh +helm install dgraph/dgraph --set alpha.service.type="LoadBalancer" --set ratel.service.type="LoadBalancer" +``` + +##### Private Internal Network + +An external load balancer can be configured to face internally to a private subnet rather the public Internet. This way it can be accessed securely by clients on the same network, through a VPN, or from a jump server. In Kubernetes, this is often configured through service annotations by the provider. Here's a small list of annotations from cloud providers: + +|Provider | Documentation Reference | Annotation | +|------------|---------------------------|------------| +|AWS |[Amazon EKS: Load Balancing](https://docs.aws.amazon.com/eks/latest/userguide/load-balancing.html)|`service.beta.kubernetes.io/aws-load-balancer-internal: "true"`| +|Azure |[AKS: Internal Load Balancer](https://docs.microsoft.com/azure/aks/internal-lb)|`service.beta.kubernetes.io/azure-load-balancer-internal: "true"`| +|Google Cloud|[GKE: Internal Load Balancing](https://cloud.google.com/kubernetes-engine/docs/how-to/internal-load-balancing)|`cloud.google.com/load-balancer-type: "Internal"`| + + +As an example, using Amazon [EKS](https://aws.amazon.com/eks/) as the provider. + +1. Create a Helm chart configuration values file ``.yaml file: + +```yaml +# .yaml +alpha: + service: + type: LoadBalancer + annotations: + service.beta.kubernetes.io/aws-load-balancer-internal: "true" +ratel: + service: + type: LoadBalancer + annotations: + service.beta.kubernetes.io/aws-load-balancer-internal: "true" +``` + +1. To expose Alpha and Ratel services privately, use: + +```sh +helm install dgraph/dgraph --values .yaml +``` + + + +You can expose Alpha and Ratel using an [ingress](https://kubernetes.io/docs/concepts/services-networking/ingress/) resource that can route traffic to service resources. Before using this option you may need to install an [ingress controller](https://kubernetes.io/docs/concepts/services-networking/ingress-controllers/) first, as is the case with [AKS](https://docs.microsoft.com/azure/aks/) and [EKS](https://aws.amazon.com/eks/), while in the case of [GKE](https://cloud.google.com/kubernetes-engine), this comes bundled with a default ingress controller. When routing traffic based on the `hostname`, you may want to integrate an addon like [ExternalDNS](https://github.com/kubernetes-sigs/external-dns) so that DNS records can be registered automatically when deploying Dgraph. + +As an example, you can configure a single ingress resource that uses [ingress-nginx](https://github.com/kubernetes/ingress-nginx) for Alpha and Ratel services. + +1. Create a Helm chart configuration values file, ``.yaml file: + +```yaml +# .yaml +global: + ingress: + enabled: false + annotations: + kubernetes.io/ingress.class: nginx + ratel_hostname: "ratel." + alpha_hostname: "alpha." +``` + +2. To expose Alpha and Ratel services through an ingress: + +```sh +helm install dgraph/dgraph --values .yaml +``` + +You can run `kubectl get ingress` to see the status and access these through their hostname, such as `http://alpha.` and `http://ratel.` + + +:::tipIngress controllers will likely have an option to configure access for private internal networks. Consult documentation from the ingress controller provider for further information.::: + + + +### Upgrading the Helm chart + +You can update your cluster configuration by updating the configuration of the +Helm chart. Dgraph is a stateful database that requires some attention on +upgrading the configuration carefully in order to update your cluster to your +desired configuration. + +In general, you can use [`helm upgrade`][helm-upgrade] to update the +configuration values of the cluster. Depending on your change, you may need to +upgrade the configuration in multiple steps. + +[helm-upgrade]: https://helm.sh/docs/helm/helm_upgrade/ + +To upgrade to an HA cluster setup: + +1. Ensure that the shard replication setting is more than one and `zero.shardReplicaCount`. For example, set the shard replica flag on the Zero node group to 3,`zero.shardReplicaCount=3`. +2. Run the Helm upgrade command to restart the Zero node group: + ```sh + helm upgrade dgraph/dgraph [options] + ``` +3. Set the Alpha replica count flag. For example: `alpha.replicaCount=3`. +4. Run the Helm upgrade command again: + ```sh + helm upgrade dgraph/dgraph [options] + ``` + + diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/installation/index.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/installation/index.md new file mode 100644 index 00000000..cc659874 --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/installation/index.md @@ -0,0 +1,5 @@ +--- +title: Installation +--- +This section is about installing Dgraph in dev or hobbyist environment as well as production environments with HA and horizontal scalability using multiple Alpha nodes in a cluster. + diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/installation/lambda-server.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/installation/lambda-server.md new file mode 100644 index 00000000..6bfdef84 --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/installation/lambda-server.md @@ -0,0 +1,90 @@ +--- +title: Lambda Server +description: Setup a Dgraph database with a lambda server. Dgraph Lambda is a serverless platform for running JavaScript on Dgraph and Dgraph Cloud +--- + + + +[Dgraph Lambda](https://github.com/dgraph-io/dgraph-lambda) is a component that allows you to execute custom business logic using JavaScript/TypeScript functions within your GraphQL API. It's part of Dgraph's GraphQL implementation and enables you to extend the auto-generated GraphQL API with custom resolvers. +:::note +Dgraph lambda is an optional server, only used for GraphQL API. +::: + +### Running with Docker + +To run a Dgraph Lambda server with Docker: +```bash +docker run -it --rm -p 8686:8686 -v /path/to/script.js:/app/script/script.js -e DGRAPH_URL=http://host.docker.internal:8080 dgraph/dgraph-lambda +``` + +:::note +`host.docker.internal` doesn't work on older versions of Docker on Linux. You can use `DGRAPH_URL=http://172.17.0.1:8080` instead. +::: + + +### Adding libraries + +If you would like to add libraries to Dgraph Lambda, use `webpack --target=webworker` to compile your script. + +### Working with TypeScript + +You can import `@slash-graphql/lambda-types` to get types for `addGraphQLResolver` and `addGraphQLMultiParentResolver`. + + +## Dgraph Alpha + +To set up Dgraph Alpha, you need to define the `--graphql` superflag's `lambda-url` option, which is used to set the URL of the lambda server. All the `@lambda` fields will be resolved through the lambda functions implemented on the given lambda server. + +For example: + +```bash +dgraph alpha --graphql lambda-url=http://localhost:8686/graphql-worker +``` + +Then test it out with the following `curl` command: +```bash +curl localhost:8686/graphql-worker -H "Content-Type: application/json" -d '{"resolver":"MyType.customField","parent":[{"customField":"Dgraph Labs"}]}' +``` + +### Docker settings + +If you're using Docker, you need to add the `--graphql` superflag's `lambda-url` option to your Alpha configuration. For example: + +```yml + command: /gobin/dgraph alpha --zero=zero1:5180 -o 100 --expose_trace --trace ratio=1.0 + --profile_mode block --block_rate 10 --logtostderr -v=2 + --security whitelist=10.0.0.0/8,172.16.0.0/12,192.168.0.0/16 --my=alpha1:7180 + --graphql lambda-url=http://lambda:8686/graphql-worker +``` + +Next, you need to add the Dgraph Lambda server configuration, and map the JavaScript file that contains the code for lambda functions to the `/app/script/script.js` file. Remember to set the `DGRAPH_URL` environment variable to your Alpha server. + + +Here's a complete Docker example that uses the base Dgraph image and adds Lambda server support: + +```yml +services: + dgraph: + image: dgraph/standalone:latest + environment: + DGRAPH_ALPHA_GRAPHQL: "lambda-url=http://dgraph_lambda:8686/graphql-worker" + ports: + - "8080:8080" + - "9080:9080" + - "8000:8000" + volumes: + - dgraph:/dgraph + + dgraph_lambda: + image: dgraph/dgraph-lambda:latest + + ports: + - "8686:8686" + environment: + DGRAPH_URL: http://dgraph:8080 + volumes: + - ./gql/script.js:/app/script/script.js:ro + +volumes: + dgraph: {} +``` diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/installation/production-checklist.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/installation/production-checklist.md new file mode 100644 index 00000000..abc09d5d --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/installation/production-checklist.md @@ -0,0 +1,117 @@ +--- +title: Production checklist +description: Requirements to install Dgraph in a production environment +--- + +This guide describes important setup recommendations for a production-ready Dgraph cluster, ensuring high availability with external persistent storage, automatic recovery of failed services, automatic recovery of failed systems such as virtual machines, and disaster recovery such as backup/restore or export/import with automation. + +:::note +In this guide, a node refers to a Dgraph instance unless specified otherwise. +::: + +A **Dgraph cluster** is comprised of multiple **Dgraph instances** or nodes connected together to form a single distributed database. A Dgraph instance is either a **Dgraph Zero** or **Dgraph Alpha**, each of which serves a different role in the cluster. + +Once installed you may also install or use a **Dgraph client** to communicate with the database and perform queries, mutations, alter schema operations and so on. Pure HTTP calls from curl, Postman, or another program are also possible without a specific client, but there are a range of clients that provide higher-level language bindings, and which use optimized gRPC for communications to the database. Any standards-compliant GraphQL client will work with Dgraph to run GraphQL operations. To run DQL and other Dgraph-specific operations, use a Dgraph client. + +Dgraph provides official clients for Go, Java, Python, and JavaScript, and C#, and the JavaScript client supports both gRPC and HTTP to run more easily in a browser. Community-developed Dgraph clients for other languages are also available. The full list of clients can be found in [Clients](../clients) page. One particular client, Dgraph Ratel, is a more sophisticated UI tool used to visualize queries, run mutations, and manage schemas in both GraphQL and DQL. Note that clients are not part of a database cluster, and simply connect to one or more Dgraph Alpha instances. + +### Cluster Requirements + +A minimum of one Dgraph Zero and one Dgraph Alpha is needed for a working cluster. + +There can be multiple Dgraph Zeros and Dgraph Alphas running in a single cluster. + + +### Machine Requirements + +To ensure predictable performance characteristics, Dgraph instances should **not** run on "burstable" or throttled machines that limit resources. That includes t2 class machines on AWS. + +To ensure that Dgraph is highly-available, we recommend each Dgraph instance be deployed to a different underlying host machine, and ideally that machines are in different availability zones or racks. In the event of an underlying machine failure, it is critical that only one Dgraph alpha and one Dgraph zero be offline so that 2 of the 3 instances in each group maintain a quorum. Also when using VMs or Docker/K8s, ensure machines are not over-subscribed and ideally not co-resident with other processes that will interrupt and delay Dgraph processing. + +If you'd like to run Dgraph with fewer machines, then the recommended configuration is to run a single Dgraph Zero and a single Dgraph Alpha per machine. In a high availability setup, that allows the cluster to lose a single machine (simultaneously losing a Dgraph Zero and a Dgraph Alpha) with continued availability of the database. + +Do not run multiple Dgraph Zeros or Dgraph Alpha processes on a single machine. This can affect performance due to shared resource issues and reduce availability in the event of machine failures. + +### Operating System + +Dgraph is designed to run on Linux. + +To run Dgraph on Windows and macOS, use the [standalone Docker image](/learn/). + +### CPU and Memory + + +We recommend 8 vCPUs or cores on each of three HA alpha instances for production loads, with 16 GiB+ memory per node. + +You'll want a ensure that your CPU and memory resources are sufficient for your production workload. A common configuration for Dgraph is 16 CPUs and 32 GiB of memory per machine. Dgraph is designed with concurrency in mind, so more cores means quicker processing and higher throughput of requests. + +You may find you'll need more CPU cores and memory for your specific use case. + +In addition, we highly recommend that your CPU clock rate is equal or above 3.4GHz. + +### Disk + +Dgraph instances make heavy use of disks, so storage with high IOPS is highly recommended to ensure reliable performance. Specifically SSDs, not HDDs. + +Regarding disk IOPS, the recommendation is: +* 1000 IOPS minimum +* 3000 IOPS for medium and large datasets + +Instances such as c5d.4xlarge have locally-attached NVMe SSDs with high IOPS. You can also use EBS volumes with provisioned IOPS (io1). If you are not running performance-critical workloads, you can also choose to use cheaper gp2 EBS volumes. Typically, AWS [gp3](https://aws.amazon.com/about-aws/whats-new/2020/12/introducing-new-amazon-ebs-general-purpose-volumes-gp3/?nc1=h_ls) disks are a good option and have 3000 Baseline IOPS at any disk size. + +Recommended disk sizes for Dgraph Zero and Dgraph Alpha: + +* Dgraph Zero: 200 GB to 300 GB. Dgraph Zero stores cluster metadata information and maintains a write-ahead log for cluster operations. +* Dgraph Alpha: 250 GB to 750 GB. Dgraph Alpha stores database data, including the schema, indices, and the data values. It maintains a write-ahead log of changes to the database. Your cloud provider may provide better disk performance based on the volume size. +* If you plan to store over 1.1TB per Dgraph Alpha instance, you must increase either the MaxLevels or TableSizeMultiplier. + +Additional recommendations: + +* The recommended Linux filesystem is ext4. +* Avoid using shared storage such as NFS, CIFS, and CEPH storage. + +### Firewall Rules + +Dgraph instances communicate over several ports. Firewall rules should be configured appropriately for the ports documented in [Ports Usage](../admin/security/ports-usage). + +Internal ports must be accessible by all Zero and Alpha peers for proper cluster-internal communication. Database clients must be able to connect to Dgraph Alpha external ports either directly or through a load balancer. + +Dgraph Zeros can be set up in a private network where communication is only with Dgraph Alphas, database administrators, internal services (such as Prometheus or Jaeger), and possibly developers (see note below). Dgraph Zero's 6080 external port is only necessary for database administration. + +:::note +Developers using Dgraph Live Loader or Dgraph Bulk Loader require access to both Dgraph Zero port 5080 and Dgraph Alpha port 9080. When using those tools, consider using them within your environment that has network access to both ports of the cluster. +::: + +### Operating System Tuning + +The OS should be configured with the recommended settings to ensure that Dgraph runs properly. + +#### File Descriptors Limit + +Dgraph can use a large number of open file descriptors. Most operating systems set a default limit that is lower than what is required. + +It is recommended to set the file descriptors limit to unlimited. If that is not possible, set it to at least a million (1,048,576) which is recommended to account for cluster growth over time. + +### Deployment + +A Dgraph instance is run as a single process from a single static binary. It does not require any additional dependencies or separate services in order to run (see the [Supplementary Services](#supplementary-services) section for third-party services that work alongside Dgraph). A Dgraph cluster is set up by running multiple Dgraph processes networked together. + +### Backup Policy + +A backup policy is a predefined, set schedule used to schedule backups of information from business applications. A backup policy helps to ensure data recoverability in the event of accidental data deletion, data corruption, or a system outage. + +For Dgraph, backups are created using the [backups feature](/admin/admin-tasks/binary-backups). You can also create full exports of your data and schema using [data exports](/admin/admin-tasks/export-database). + +We **strongly** recommend that you have a backup policy in place before moving your application to the production phase, and we also suggest that you have a backup policy even for pre-production apps supported by Dgraph database instances running in development, staging, QA or pre-production clusters. + +We suggest that your policy include frequent full and incremental backups. Accordingly, we suggest the following backup policy for your production apps: +* [full backup](/admin/admin-tasks/binary-backups/#force-a-full-backup) every 24hrs +* incremental backup every 2/4hrs + +### Supplementary Services + +These services are not required for a Dgraph cluster to function but are recommended for better insight when operating a Dgraph cluster. + +[Metrics]: [metrics](/admin/observability/metrics) +[Monitoring]: [monitoring](/admin/observability/monitoring) +[Distributed tracing]: [tracing](/admin/observability/tracing) diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/installation/single-host-setup.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/installation/single-host-setup.md new file mode 100644 index 00000000..6cabc51f --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/installation/single-host-setup.md @@ -0,0 +1,134 @@ +--- +title: Basic cluster setup +--- + + + + +The recommended way to get started with Dgraph for local development is by using Docker. There are two main approaches: + +## Option 1: Standalone Docker Image (Learning Environment) + +The [`dgraph/standalone`](https://hub.docker.com/r/dgraph/standalone) Docker image is the fastest way to get started. This single container runs both Dgraph Zero and Dgraph Alpha, making it ideal for quick testing and development. + +Ensure you have [Docker installed](https://www.docker.com/), then run the following command: + +```bash +docker run --name dgraph-dev -d -p 8080:8080 -p 9080:9080 \ + -v ~/dgraph:/dgraph \ + dgraph/standalone:latest +``` + +This command initiates a new Docker container +* `--name dgraph-dev` - creates a persistent container named `dgraph-dev` +* `-d` - runs the container in detached (daemon) mode +* `-p 8080:8080` - maps port 8080 for HTTP connections +* `-p 9080:9080` - maps port 9080 for gRPC connections +* `-v ~/dgraph:/dgraph` - persists data to your local `~/dgraph` directory +* `dgraph/standalone:latest` - uses the official Dgraph standalone image + +## Option 2: Docker Compose + +For a more production-like setup that separates Zero, Alpha, and Ratel into individual containers, use Docker Compose. This approach gives you better control and is easier to scale. + +Create a `docker-compose.yml` file with the following configuration: + +```yaml +version: '3.8' +name: dgraph-basic-cluster +services: + zero: + image: dgraph/dgraph:latest + ports: + - "5080:5080" + - "6080:6080" + command: dgraph zero --my=zero:5080 + restart: unless-stopped + + alpha: + image: dgraph/dgraph:latest + ports: + - "8080:8080" + - "9080:9080" + command: dgraph alpha --my=alpha:7080 --zero=zero:5080 --security whitelist=0.0.0.0/0 + depends_on: + - zero + restart: unless-stopped + + ratel: + image: dgraph/ratel:latest + ports: + - "8000:8000" + restart: unless-stopped + +``` + +Then start the cluster: + +```bash +docker-compose up -d +``` + +This starts three separate containers: +* **Zero**: Manages cluster membership and assigns UIDs (ports 5080, 6080) +* **Alpha**: Handles queries and mutations (ports 8080, 9080) +* **Ratel**: Web UI for interacting with Dgraph (port 8000) + +To stop the cluster: + +```bash +docker-compose down +``` + +To remove all data volumes: + +```bash +docker-compose down -v +``` + +### Check your Dgraph cluster health +Verify your Dgraph instance using the `/health` REST endpoint. + +```shell +curl http://localhost:8080/health | jq +``` +The command should return basic cluster information: +```json +[ + { + "instance": "alpha", + "address": "localhost:7080", + "status": "healthy", + "group": "1", + "version": "v24.1.5", + "uptime": 11, + "lastEcho": 1761430795, + "ongoing": [ + "opRollup" + ], + "ee_features": [ + "backup_restore", + "cdc" + ], + "max_assigned": 30002 + } +] +``` + +### Access Ratel UI + +Ratel is a web-based UI dashboard for interacting with Dgraph using Dgraph's query language, [DQL](/dgraph-glossary#dql). + +**If using Option 1 (Standalone)**: Launch Ratel separately: + +```bash +docker run --name ratel -d -p "8000:8000" dgraph/ratel:latest +``` + +**If using Option 2 (Docker Compose)**: Ratel is already included and will start automatically. + +Navigate to Ratel at `http://localhost:8000` and enter `http://localhost:8080` for the "Dgraph Conn String". This will allow Ratel to connect to your local Dgraph instance and execute DQL queries. + +![Setting up Ratel](/images/dgraph/quickstart/ratel-docker-connection.png) + + diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/learn/developer/sample-apps/devjokes.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/learn/developer/sample-apps/devjokes.md new file mode 100644 index 00000000..2b4fc72d --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/learn/developer/sample-apps/devjokes.md @@ -0,0 +1,30 @@ +--- +title: DevJokes +description: A sample app for Dgraph database and Dgraph Cloud that lets you find, like and share jokes. +pageType: sample-apps +image: /images/sample-apps/devjokes.svg +--- + +DevJokes is a sample app that lets users find, like, and share their favorite geeky jokes with other software developers. This app demonstrates how to use React hooks with an Apollo client to create jokes, filter jokes by tags, add user reactions to jokes, and moderate jokes. Also demonstrated is the client side to uploading images to AWS S3 and storing links to the images. The repo for this app can be found on GitHub at [dgraph.io/graphql-sample-apps/dev-jokes](https://github.com/dgraph-io/graphql-sample-apps/tree/master/dev-jokes). + +### Features +- Support two kind of jokes: Text Joke and Meme +- Moderated content to feed you the best jokes and add enable better searching. +- Logged in users can like and share the jokes. +- Community moderation enabled through flagging. + +### Front-end +- [React](https://reactjs.org/) (3.4.3)—a JavaScript library for building user interfaces. +- [Apollo Client](https://www.npmjs.com/package/apollo-client) (3.0+)—a comprehensive state management library for JavaScript that enables you to manage both local and remote data with GraphQL. +- [Material-UI](https://material-ui.com/)—a user interface framework for faster and easier web development. +- [Emoji Mart](https://github.com/missive/emoji-mart)—a Slack-like customizable emoji picker component for React. +- [React Grid Gallery](https://benhowell.github.io/react-grid-gallery/)—a justified image gallery component for React inspired by Google Photos and based upon React Images. +- [React Share](https://github.com/nygardk/react-share)—social media share buttons and share counts for React. +- [React Markdown](https://github.com/remarkjs/react-markdown)—markdown component for React using remark. + +### Back-end +- [Dgraph Cloud](https://dgraph.io/cloud)—a fully managed GraphQL backend service +- [Auth0](https://auth0.com/)—Secure access for everyone. +- [AWS-S3](https://aws.amazon.com/s3/)—an object storage service that offers industry-leading scalability, data availability, security, and performance. + + diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/learn/developer/sample-apps/pokedex.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/learn/developer/sample-apps/pokedex.md new file mode 100644 index 00000000..3d51e3c4 --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/learn/developer/sample-apps/pokedex.md @@ -0,0 +1,19 @@ +--- +title: Pokedex +pageType: sample-apps +image: /images/sample-apps/pokedex.svg +--- + +Pokedex is a sample app inspired by the [Pokedex](https://www.pokemon.com/us/pokedex/) website. This sample app lets users search for Pokemon by type, weakness, ability, and more! This app demonstrates how to filter Pokemon by tags and update a captured status. The repo for this app can be found on GitHub at [dgraph.io/graphql-sample-apps/pokedex](https://github.com/dgraph-io/graphql-sample-apps/tree/master/pokedex). + +### Features +- Filter Pokemon by type +- Mark Pokemon as Captured +- Filter Pokemon by Captured + +### Front-end +- [React](https://reactjs.org/) (3.4.3) — a JavaScript library for building user interfaces. +- [Material-UI](https://material-ui.com/) — a user interface framework for faster and easier web development. + +### Back-end +- [Dgraph Cloud](https://dgraph.io/cloud) — a fully managed GraphQL backend service diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/learn/developer/sample-apps/surveyo.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/learn/developer/sample-apps/surveyo.md new file mode 100644 index 00000000..d40878ad --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/learn/developer/sample-apps/surveyo.md @@ -0,0 +1,27 @@ +--- +title: Surveyo +description: A sample app for Dgraph database and Dgraph Cloud that lets create surveys. +pageType: sample-apps +image: /images/sample-apps/surveyo.svg +--- + +Surveyo is a sample app that provides users with a survey tool that they can use to quickly create and respond to surveys. Advanced users can use Surveyo’s GraphQL endpoint to run complex queries on survey results. This app demonstrates how to use React hooks with Apollo client to create surveys, collect responses, visualize responses with charts, export responses into CSV, and delete surveys. The repo for this app can be found on GitHub at [dgraph.io/graphql-sample-apps/surveyo](https://github.com/dgraph-io/graphql-sample-apps/tree/master/surveyo). + +### Features +- Supports Short Answer, Multiple-Choice Question, Date Query and Rating type of questions +- Visualize responses collected as Pie Chart, Word Cloud and Bar chart +- Provides inline GraphiQL IDE to make GraphQL queries to chart data directly +- Export survey responses into CSV. + +### Front-end +- [React](https://reactjs.org/) (3.4.1)—a JavaScript library for building user interfaces. +- [Apollo Client](https://www.npmjs.com/package/apollo-client) (3.1.1)—a comprehensive state management library for JavaScript that enables you to manage both local and remote data with GraphQL. +- [Auth0 React](https://github.com/auth0/auth0-react)—Auth0 SDK for React Single Page Applications (SPA). +- [AntDesign](https://ant.design/)—a design system for enterprise-level products. +- [Chart.js](https://www.chartjs.org/)—a simple yet flexible JavaScript charting for designers & developers +- [TypeScript](https://www.typescriptlang.org/)—extends JavaScript by adding types. +- [GraphiQL](https://github.com/graphql/graphiql)—a graphical interactive in-browser GraphQL IDE + +### Back-end +- [Dgraph Cloud](https://dgraph.io/cloud)—a fully managed GraphQL backend service +- [Auth0](https://auth0.com/)—Secure access for everyone. diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/learn/developer/sample-apps/todos.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/learn/developer/sample-apps/todos.md new file mode 100644 index 00000000..749088ae --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/learn/developer/sample-apps/todos.md @@ -0,0 +1,26 @@ +--- +title: Todo App +description: A sample app for Dgraph database and Dgraph Cloud that lets you manage the tasks on your personal to-do list. +pageType: sample-apps +image: /images/sample-apps/todos.svg +--- + +To-Do is a sample app that lets users manage the tasks on their personal to-do list. This app demonstrates how to use React hooks with an Apollo client to easily create, read, update, and delete to-do list items. The repo for this app can be found on GitHub at [dgraph.io/graphql-sample-apps/todo-app-react](https://github.com/dgraph-io/graphql-sample-apps/tree/master/todo-app-react) + +### Features +- Add a new task +- Update an existing task to mark tasks completed +- Delete existing tasks + +### Front-end +- [React](https://reactjs.org/) (3.4.0)—a JavaScript library for building user interfaces. +- [Mobx](https://mobx.js.org/README.html)— MobX is a battle tested library that makes state management simple and scalable by transparently applying functional reactive programming (TFRP). +- [Apollo Client](https://www.npmjs.com/package/apollo-client) (2.6.8)—a comprehensive state management library for JavaScript that enables you to manage both local and remote data with GraphQL. +- [ToDoMVC app CSS](https://github.com/tastejs/todomvc-app-css)—CSS for a ToDo App +- [React Router](https://reactrouter.com/)—a collection of navigational components +- [clipboard.js](https://clipboardjs.com/)—a modern approach to copy text to clipboard +- [history](https://github.com/ReactTraining/history)—lets you easily manage session history + +### Back-end +- [Dgraph Cloud](https://dgraph.io/cloud)—a fully managed GraphQL backend service +- [Auth0](https://auth0.com/)—Secure access for everyone. diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/migration/bulk-loader.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/migration/bulk-loader.md new file mode 100644 index 00000000..85af1dd2 --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/migration/bulk-loader.md @@ -0,0 +1,275 @@ +--- +title: Initial import +--- + +Bulk Loader performs fast initial data imports into a **new** Dgraph cluster. It's significantly faster than [Live Loader](live-loader) for large datasets and is the recommended approach for initial data ingestion. + +**Use Bulk Loader when:** +- Setting up a new Dgraph cluster +- Importing large datasets (GBs to TBs) +- Performance is critical for the initial load + +:::warning +Bulk Loader can only be used with a new cluster. For importing data into an existing cluster, use [Live Loader](live-loader). +::: + +## Prerequisites + +Before running Bulk Loader: +- Start one or more Dgraph Zeros (Alphas will be started later) +- Prepare data files in RDF (`.rdf`, `.rdf.gz`) or JSON (`.json`, `.json.gz`) format +- Prepare a schema file + +:::note +Bulk Loader only accepts [RDF N-Quad/Triple data](https://www.w3.org/TR/n-quads/) or JSON. See [data migration](import-data) for converting other formats. +::: + +## Quick Start + +```sh +dgraph bulk \ + --files data.rdf.gz \ + --schema schema.txt \ + --zero localhost:5080 \ + --map_shards 4 \ + --reduce_shards 1 +``` + +## Understanding Shards + +Before running Bulk Loader, determine your cluster topology: + +- **`--reduce_shards`** — Set to the number of Alpha **groups** in your cluster +- **`--map_shards`** — Set equal to or higher than `--reduce_shards` for even distribution + +| Cluster Setup | Alpha Groups | `--reduce_shards` | +|--------------|--------------|-------------------| +| 3 Alphas, 3 replicas/group | 1 | 1 | +| 6 Alphas, 3 replicas/group | 2 | 2 | +| 9 Alphas, 3 replicas/group | 3 | 3 | + +## Basic Usage + +```sh +dgraph bulk \ + --files ./data.rdf.gz \ + --schema ./schema.txt \ + --zero localhost:5080 \ + --map_shards 4 \ + --reduce_shards 2 +``` + +### Output Structure + +Bulk Loader generates `p` directories in the `out` folder: + +``` +./out +├── 0 +│ └── p +│ ├── 000000.vlog +│ ├── 000002.sst +│ └── MANIFEST +└── 1 + └── p + └── ... +``` + +With `--reduce_shards=2`, two directories are created (`./out/0` and `./out/1`). + +### Deploying Output to Cluster + +Copy each shard's `p` directory to the corresponding Alpha group: + +- **Group 1** (Alpha1, Alpha2, Alpha3) → copy `./out/0/p` +- **Group 2** (Alpha4, Alpha5, Alpha6) → copy `./out/1/p` + +![Bulk Loader diagram](/images/deploy/bulk-loader.png) + +:::note +Every Alpha replica in a group must have a copy of the same `p` directory. +::: + +## Loading from Cloud Storage + +### Amazon S3 + +Set credentials via environment variables or use [IAM roles](#iam-setup): + +| Environment Variable | Description | +|---------------------|-------------| +| `AWS_ACCESS_KEY_ID` | AWS access key with S3 read permissions | +| `AWS_SECRET_ACCESS_KEY` | AWS secret key | + +```sh +dgraph bulk \ + --files s3:///bucket/data \ + --schema s3:///bucket/data/schema.txt \ + --zero localhost:5080 +``` + +#### IAM Setup + +1. Create an [IAM Role](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create.html) with S3 access +2. Attach using [Instance Profile](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2_instance-profiles.html) (EC2) or [IAM roles for service accounts](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html) (EKS) + +### MinIO + +| Environment Variable | Description | +|---------------------|-------------| +| `MINIO_ACCESS_KEY` | MinIO access key | +| `MINIO_SECRET_KEY` | MinIO secret key | + +```sh +dgraph bulk \ + --files minio://server:port/bucket/data \ + --schema minio://server:port/bucket/data/schema.txt \ + --zero localhost:5080 +``` + +## Deployment Strategies + +### Small Datasets (< 10 GB) + +Let Dgraph stream snapshots between replicas: + +1. Run Bulk Loader on one server +2. Start only the first Alpha replica +3. Wait ~1 minute for snapshot creation: + ``` + Creating snapshot at index: 30. ReadTs: 4. + ``` +4. Start remaining Alpha replicas — snapshots stream automatically: + ``` + Streaming done. Sent 1093470 entries. Waiting for ACK... + ``` + +### Large Datasets (> 10 GB) + +Copy `p` directories directly for faster deployment: + +1. Run Bulk Loader on one server +2. Copy/rsync `p` directories to all Alpha servers +3. Start all Alphas simultaneously +4. Verify all Alphas create snapshots with matching index values + +## Multi-tenancy + +By default, Bulk Loader preserves namespace information from data files. Without namespace info, data loads into the default namespace. + +Force all data into a specific namespace with `--force-namespace`: + +```sh +dgraph bulk \ + --files data.rdf.gz \ + --schema schema.txt \ + --zero localhost:5080 \ + --force-namespace 123 +``` + +## Encryption + +### Loading into Encrypted Cluster + +Generate encrypted `p` directories: + +```sh +dgraph bulk \ + --files data.rdf.gz \ + --schema schema.txt \ + --zero localhost:5080 \ + --encryption key-file=./encryption.key +``` + +### Loading Encrypted Exports + +Decrypt encrypted export files during import: + +```sh +# Encrypted input → Encrypted output +dgraph bulk \ + --files encrypted-data.rdf.gz \ + --schema encrypted-schema.txt \ + --zero localhost:5080 \ + --encrypted=true \ + --encryption key-file=./encryption.key + +# Encrypted input → Unencrypted output (migration) +dgraph bulk \ + --files encrypted-data.rdf.gz \ + --schema encrypted-schema.txt \ + --zero localhost:5080 \ + --encrypted=true \ + --encrypted_out=false \ + --encryption key-file=./encryption.key +``` + +Using HashiCorp Vault: + +```sh +dgraph bulk \ + --files encrypted-data.rdf.gz \ + --schema encrypted-schema.txt \ + --zero localhost:5080 \ + --encrypted=true \ + --vault addr="http://localhost:8200";enc-field="enc_key";enc-format="raw";path="secret/data/dgraph" +``` + +### Encryption Flag Combinations + +| `--encrypted` | `--encryption key-file` | Result | +|--------------|------------------------|--------| +| true | not set | Error | +| true | set | Encrypted input → Encrypted output | +| false | not set | Unencrypted input → Unencrypted output | +| false | set | Unencrypted input → Encrypted output | + +## Performance Tuning + +:::tip +Disable swap space when running Bulk Loader. It's better to reduce memory usage via flags than let swapping slow the process. +::: + +### Map Phase + +Reduce memory usage: + +| Flag | Description | +|------|-------------| +| `--num_go_routines` | Lower = less memory | +| `--mapoutput_mb` | Lower = less memory | + +**Tip:** For large datasets, split RDF files into ~256MB chunks to parallelize gzip decoding. + +### Reduce Phase + +Increase if you have RAM to spare: + +| Flag | Description | +|------|-------------| +| `--reduce_shards` | Higher = more parallelism, more memory | +| `--map_shards` | Higher = better distribution, more memory | + +## CLI Options Reference + +| Flag | Description | +|------|-------------| +| `--files`, `-f` | Data file(s) or directory path | +| `--schema`, `-s` | Schema file path | +| `--graphql_schema`, `-g` | GraphQL schema file (optional) | +| `--zero` | Dgraph Zero address | +| `--map_shards` | Number of map shards | +| `--reduce_shards` | Number of reduce shards (= Alpha groups) | +| `--out` | Output directory (default: `out`) | +| `--tmp` | Temp directory (default: `tmp`) | +| `--new_uids` | Assign new UIDs instead of preserving | +| `--store_xids` | Store XIDs as `xid` predicate | +| `--xidmap` | Directory for XID→UID mappings | +| `--format` | Force format (`rdf` or `json`) | +| `--force-namespace` | Load into specific namespace | +| `--encryption` | Encryption key file | +| `--encrypted` | Input files are encrypted | +| `--encrypted_out` | Encrypt output (default: true if key provided) | +| `--badger compression` | Compression: `snappy`, `zstd`, or `none` | + +See [dgraph bulk CLI reference](../cli/bulk) for the complete list. diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/migration/export-data.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/migration/export-data.md new file mode 100644 index 00000000..af103cb4 --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/migration/export-data.md @@ -0,0 +1,296 @@ +--- +title: Export data +--- + +As an `Administrator` you can export data from Dgraph to an an object store, NFS, or a file path. + +When you export data, three files are generated: + +* `g01.gql_schema.gz`: The GraphQL schema file. This file can be imported using the Schema APIs +* `g01.json.gz` or `g01.rdf.gz`: the data from your instance in JSON format or RDF format. By default, Dgraph exports data in RDF format. +* `g01.schema.gz`: This file is the internal Dgraph schema. If you have set up the Dgraph Cloud instance with a GraphQL schema, then you can ignore this file. + +## Export data using the GraphQL admin endpoint + +You can export the entire data by executing a GraphQL mutation on the `/admin` endpoint of any Alpha node. + +**Before you begin**: + +* Ensure that there is sufficient space on disk to store the export. Each Dgraph Alpha leader for a group writes output as a gzipped file to the export directory specified through the `--export` flag (defaults to an **export** directory). If any of the groups fail because of insufficient space on the disk, the entire export process is considered failed and an error is returned. + +* Make a note of the export directories of the Alpha server nodes. For more information about configuring the Dgraph Alpha server, see [Config](/cli/config). + +This mutation triggers the export from each of the Alpha leader for a group. Depending on the Dgraph configuration several files are exported. It is recommended that you copy the files from the Alpha server nodes to a safe place when the export is complete. + +```graphql +mutation { + export(input: {}) { + response { + message + code + } + } +} +``` +The export data of the group: + +* in the Alpha instance is stored in the Alpha. +* in every other group is stored in the Alpha leader of that group. + +You need to retrieve the right export files from the Alpha instances in the cluster. Dgraph does not copy all files to the Alpha that initiated the export. + +When the export is complete a response similar to this appears: + +``` +{"data":{ + "export":{ + "response":{ + "message":"Export completed.", + "code":"Success" + } + } + }, + "extensions":{ + "tracing":{ + "version":1, + "startTime":"2022-12-14T07:39:51.061712416Z","endTime":"2022-12-14T07:39:51.129431494Z", + "duration":67719080 + } + } + } +``` + +## Export data format + +By default, Dgraph exports data in RDF format. Replace ``with `json` or `rdf` in this GraphQL mutation: + +```graphql +mutation { + export(input: { format: "" }) { + response { + message + code + } + } +} +``` + +## Export to NFS or a file path + +You can override the default folder path by adding the `destination` input field to the directory where you want to export data. Replace `` in this GraphQL mutation with the absolute path of the directory to export data. + +```graphql +mutation { + export(input: { + format: "" + destination: "" + }) { + response { + message + code + } + } +} +``` + +## Export to an object store +You can export to an AWS S3, Azure Blob Storage or Google Cloud Storage. + +### Example mutation to export to AWS S3 + +```graphql +mutation { + export(input: { + destination: "s3://s3..amazonaws.com/" + accessKey: "" + secretKey: "" + }) { + response { + message + code + } + } +} +``` + +:::note +The Dgraph URL used for S3 is different than the AWS CLI tools with the `aws s3` command, which uses a shortened format: `s3://`. +::: + + +### Example mutation to export to MinIO + +```graphql +mutation { + export(input: { + destination: "minio://
:9000/" + accessKey: "" + secretKey: "" + }) { + response { + message + code + } + } +} +``` + +## Export to a MinIO gateway + +You can use MinIO as a gateway to other object stores, such as [Azure Blob Storage](https://azure.microsoft.com/services/storage/blobs/) or [Google Cloud Storage](https://cloud.google.com/storage). + +### Azure Blob Storage + +You can use [Azure Blob Storage](https://azure.microsoft.com/services/storage/blobs/) through the [MinIO Azure Gateway](https://docs.min.io/docs/minio-gateway-for-azure.html). + +**Before you begin**: + +* Configure a [storage account](https://docs.microsoft.com/azure/storage/common/storage-account-overview) and a Blob [container](https://docs.microsoft.com/azure/storage/blobs/storage-blobs-introduction#containers) to organize the blobs. +* Make a note the name of the blob container. It is the `` when specifying the `destination` in the GraphQL mutation. +* [Retrieve storage accounts keys](https://docs.microsoft.com/azure/storage/common/storage-account-keys-manage) to configure MinIO. Because, [MinIO Azure Gateway](https://docs.min.io/docs/minio-gateway-for-azure.html) uses `MINIO_ACCESS_KEY` and `MINIO_SECRET_KEY` to correspond to Azure Storage Account `AccountName` and `AccountKey`. + +You can access Azure Blob Storage locally using one of these methods: + +* Using [MinIO Azure Gateway](https://docs.min.io/docs/minio-gateway-for-azure.html) with the MinIO Binary + ```bash + export MINIO_ACCESS_KEY="" + export MINIO_SECRET_KEY="" + minio gateway azure + ``` +* Using [MinIO Azure Gateway](https://docs.min.io/docs/minio-gateway-for-azure.html) with Docker + ```bash + docker run --detach --rm --name gateway \ + --publish 9000:9000 \ + --env MINIO_ACCESS_KEY="" \ + --env MINIO_SECRET_KEY="" \ + minio/minio gateway azure + ``` + * Using [MinIO Azure Gateway](https://docs.min.io/docs/minio-gateway-for-azure.html) with the [MinIO Helm chart](https://github.com/minio/charts) for Kubernetes: + ```bash + helm repo add minio https://helm.min.io/ + helm install my-gateway minio/minio \ + --set accessKey="",secretKey="" \ + --set azuregateway.enabled=true + ``` +You can use the [MinIO GraphQL mutation](export-data#example-mutation-to-export-to-minio) with MinIO configured as a gateway. + +### Google Cloud Storage + +You can use [Google Cloud Storage](https://cloud.google.com/storage) through the [MinIO GCS Gateway](https://docs.min.io/docs/minio-gateway-for-gcs.html). + +**Before you begin**: +* Create [storage buckets](https://cloud.google.com/storage/docs/creating-buckets) +* Create a Service Account key for GCS and get a credentials file. For more information, see [Create a Service Account key](https://github.com/minio/minio/blob/master/docs/gateway/gcs.md#11-create-a-service-account-ey-for-gcs-and-get-the-credentials-file). + +When you have a `credentials.json`, you can access GCS locally using one of these methods: + +* Using [MinIO GCS Gateway](https://docs.min.io/docs/minio-gateway-for-gcs.html) with the MinIO Binary + ```bash + export GOOGLE_APPLICATION_CREDENTIALS="/path/to/credentials.json" + export MINIO_ACCESS_KEY="" + export MINIO_SECRET_KEY="" + minio gateway gcs "" + ``` +* Using [MinIO GCS Gateway](https://docs.min.io/docs/minio-gateway-for-gcs.html) with Docker + ```bash + docker run --detach --rm --name gateway \ + --publish 9000:9000 \ + --volume "":/credentials.json \ + --env GOOGLE_APPLICATION_CREDENTIALS=/credentials.json \ + --env MINIO_ACCESS_KEY="" \ + --env MINIO_SECRET_KEY="" \ + minio/minio gateway gcs "" + ``` +* Using [MinIO GCS Gateway](https://docs.min.io/docs/minio-gateway-for-gcs.html) with the [MinIO Helm chart](https://github.com/minio/charts) for Kubernetes: + ```bash + ## create MinIO Helm config + cat <<-EOF > myvalues.yaml + accessKey: + secretKey: + + gcsgateway: + enabled: true + projectId: + gcsKeyJson: | + $(IFS='\n'; while read -r LINE; do printf ' %s\n' "$LINE"; done < "") + EOF + + ## deploy MinIO GCS Gateway + helm repo add minio https://helm.min.io/ + helm install my-gateway minio/minio \ + --values myvalues.yaml + ``` +You can use the [MinIO GraphQL mutation](export-data#example-mutation-to-export-to-minio) with MinIO configured as a gateway. + +## Disable HTTPS for exports to S3 and Minio + +By default, Dgraph assumes the destination bucket is using HTTPS. If that is not the case, the export fails. To export to a bucket using HTTP (insecure), set the query parameter `secure=false` with the destination endpoint in the `destination` field: + +```graphql +mutation { + export(input: { + destination: "minio://
:9000/?secure=false" + accessKey: "" + secretKey: "" + }) { + response { + message + code + } + } +} +``` + +## Use anonymous credentials + +When exporting to S3 or MinIO where credentials are not required, can set `anonymous` to true. + +```graphql +mutation { + export(input: { + destination: "s3://s3..amazonaws.com/" + anonymous: true + }) { + response { + message + code + } + } +} +``` + +## Encrypt exports + +Export is available wherever an Alpha is running. To encrypt an export, the Alpha must be configured with the `--encryption key-file=value`. + +:::note +The `--encryption key-file` was used for [Encryption at Rest](../installation/configuration/encryption-at-rest) and will now also be used for encrypted exports. +::: + +## Use `curl` to trigger an export + +This is an example of how you can use `curl` to trigger an export. + + 1. Create GraphQL file for the desired mutation: + ```bash + cat <<-EOF > export.graphql + mutation { + export(input: { + destination: "s3://s3..amazonaws.com/" + accessKey: "" + secretKey: "" + }) { + response { + message + code + } + } + } + EOF + ``` + 2. Trigger an export with `curl` + ```bash + curl http://localhost:8080/admin --silent --request POST \ + --header "Content-Type: application/graphql" \ + --upload-file export.graphql + ``` diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/migration/import-data.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/migration/import-data.md new file mode 100644 index 00000000..06f343b1 --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/migration/import-data.md @@ -0,0 +1,23 @@ +--- +title: Import Data +--- + +As an `Administrator` you can initialize a new Dgraph cluster by doing an [Initial import](bulk-loader) and you can import data into a running instance by performing a [Live import](live-loader). + + +Initial import is **considerably faster** than the live import but can only be used to load data into a new cluster (without prior data) and is executed before starting the Alpha nodes. + + +:::note +Both options accept [RDF N-Quad/Triple +data](https://www.w3.org/TR/n-quads/) or JSON format. Refers to [data migration](import-data) to see how to convert other data formats. +::: + + +To load CSV-formatted data or SQL data into Dgraph, +first convert the dataset into one of the accepted formats ([RDF N-Quad/Triple](https://www.w3.org/TR/n-quads/) or JSON) and then load the +resulting dataset into Dgraph. + +After you convert the `.csv` or `.sql` files to [RDF N-Quad/Triple](https://www.w3.org/TR/n-quads/) or JSON, +you can use [Dgraph Live Loader](live-loader) or +[Dgraph Bulk Loader](bulk-loader) to import your data. diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/migration/index.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/migration/index.md new file mode 100644 index 00000000..dc9c47bb --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/migration/index.md @@ -0,0 +1,3 @@ +--- +title: Data migration +--- diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/migration/live-loader.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/migration/live-loader.md new file mode 100644 index 00000000..57e0fab7 --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/migration/live-loader.md @@ -0,0 +1,230 @@ +--- +title: Live import +--- +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +Live Loader imports data into a running Dgraph cluster using the [dgraph live](/cli/live) command. Unlike [Bulk Loader](bulk-loader), Live Loader can import data into an existing database with prior data and supports upserts for updating existing nodes. + +**Use Live Loader when:** +- Importing data into a running cluster +- Updating or adding data to existing nodes +- Loading smaller datasets (for large initial loads, consider [Bulk Loader](bulk-loader)) + +## Prerequisites + +Before importing, ensure you have: +- A running Dgraph cluster +- Data files in RDF (`.rdf`, `.rdf.gz`) or JSON (`.json`, `.json.gz`) format +- A schema file (optional but recommended) + +:::note +Live Loader accepts [RDF N-Quad/Triple data](https://www.w3.org/TR/n-quads/) or JSON in plain or gzipped format. See [data migration](import-data) for converting other formats. +::: + +## Quick Start + +```sh +dgraph live --files ./data.rdf.gz --schema ./schema.txt --alpha localhost:9080 +``` + +## Basic Usage + + + + +```sh +dgraph live \ + --files \ + --schema \ + --alpha localhost:9080 +``` + + + + +```sh +docker run -it --rm -v :/tmp dgraph/dgraph:latest \ + dgraph live \ + --files /tmp/ \ + --schema /tmp/ \ + --alpha :9080 +``` + + + + +**Key options:** +- `--alpha` — Dgraph Alpha gRPC endpoint (default: `localhost:9080`). Specify multiple addresses (comma-separated) to distribute load. +- `--files` — Path to data file or directory. When a directory is specified, all `.rdf`, `.rdf.gz`, `.json`, and `.json.gz` files are loaded. +- `--schema` — Path to schema file (use a different extension like `.txt` or `.schema`). + +## Upserts: Update Existing Data + +Live Loader can update existing nodes using upserts. Use one of these approaches: + +### Using `--upsertPredicate` + +Specify a predicate that serves as a unique identifier: + +```sh +dgraph live \ + --files ./data.rdf.gz \ + --schema ./schema.txt \ + --upsertPredicate xid +``` + +The upsert predicate must exist in the schema and be indexed. + +If you are using `xid` as the upsert predicate name, make sure your schema contains: +``` +: string @index(exact) @upsert . +``` + +**Example:** If your data contains: +```rdf + "Alice Smith" . + +``` + +This creates or updates the node where `xid = "urn:uuid:550e8400-e29b-41d4-a716-446655440000>"` and sets its predicate `http://xmlns.com/foaf/0.1/name` to `"Alice Smith"`. + + +### Using `--xidmap` + +Store UID mappings in a local directory for consistent imports: + +```sh +dgraph live \ + --files ./data.rdf.gz \ + --schema ./schema.txt \ + --xidmap ./xid-directory +``` + +Live Loader looks up existing UIDs or stores new mappings in this directory. + +## Loading from Cloud Storage + +### Amazon S3 + +Set credentials via environment variables or use [IAM roles](#iam-setup): + +| Environment Variable | Description | +|---------------------|-------------| +| `AWS_ACCESS_KEY_ID` | AWS access key with S3 read permissions | +| `AWS_SECRET_ACCESS_KEY` | AWS secret key | + +```sh +# Short form (note triple slash) +dgraph live \ + --files s3://// \ + --schema s3://///schema.txt + +# Long form +dgraph live \ + --files s3://s3..amazonaws.com// \ + --schema s3://s3..amazonaws.com///schema.txt +``` + +#### IAM Setup + +Instead of credentials, configure IAM: + +1. Create an [IAM Role](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create.html) with S3 access +2. Attach it using: + - [Instance Profile](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2_instance-profiles.html) for EC2 + - [IAM roles for service accounts](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html) for EKS + +### MinIO + +| Environment Variable | Description | +|---------------------|-------------| +| `MINIO_ACCESS_KEY` | MinIO access key | +| `MINIO_SECRET_KEY` | MinIO secret key | + +```sh +dgraph live \ + --files minio://:// \ + --schema minio://:///schema.txt +``` + +## Multi-tenancy + +When ACL is enabled, provide credentials with `--creds`. By default, data loads into the user's namespace. + +```sh +dgraph live \ + --files ./data.rdf.gz \ + --schema ./schema.txt \ + --creds "user=groot;password=password;namespace=0" +``` + +### Loading into a Specific Namespace + +[Guardians of the Galaxy](/admin/admin-tasks/multitenancy#guardians-of-the-galaxy) can load data into any namespace using `--force-namespace`: + +```sh +# Load into namespace 123 +dgraph live \ + --files ./data.rdf.gz \ + --schema ./schema.txt \ + --creds "user=groot;password=password;namespace=0" \ + --force-namespace 123 +``` + +To preserve namespaces from export files, use `--force-namespace -1`: + +```sh +dgraph live \ + --files ./data.rdf.gz \ + --schema ./schema.txt \ + --creds "user=groot;password=password;namespace=0" \ + --force-namespace -1 +``` + +:::note +The target namespace must exist before loading data. +::: + +## Encrypted Data + +To load encrypted export files, provide the decryption key: + +```sh +# Using key file +dgraph live \ + --files ./encrypted-data.rdf.gz \ + --schema ./encrypted-schema.txt \ + --encryption key-file=./encryption.key + +# Using HashiCorp Vault +dgraph live \ + --files ./encrypted-data.rdf.gz \ + --schema ./encrypted-schema.txt \ + --vault addr="http://localhost:8200";enc-field="enc_key";enc-format="raw";path="secret/data/dgraph/alpha";role-id-file="./role_id";secret-id-file="./secret_id" +``` + +:::note +Encrypted exports can be imported into unencrypted Dgraph instances. The `p` directory will only be encrypted if the Alpha has encryption enabled. +::: + +## CLI Options Reference + +| Flag | Default | Description | +|------|---------|-------------| +| `--files`, `-f` | | Data file or directory path | +| `--schema`, `-s` | | Schema file path | +| `--alpha`, `-a` | `localhost:9080` | Dgraph Alpha gRPC address(es) | +| `--batch`, `-b` | `1000` | N-Quads per mutation batch | +| `--conc`, `-c` | `10` | Concurrent requests to Dgraph | +| `--upsertPredicate`, `-U` | | Predicate for upsert matching | +| `--xidmap`, `-x` | | Directory for UID mappings | +| `--new_uids` | `false` | Assign new UIDs instead of preserving existing | +| `--format` | | Force format (`rdf` or `json`) | +| `--use_compression`, `-C` | `false` | Enable gRPC compression | +| `--creds` | | ACL credentials (`user=;password=;namespace=`) | +| `--force-namespace` | | Load into specific namespace (Guardian only) | +| `--encryption` | | Encryption key file for decryption | +| `--vault` | | Vault configuration for encryption key | + +See [dgraph live CLI reference](/cli/live) for the complete list of options. diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/migration/loading-csv-data.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/migration/loading-csv-data.md new file mode 100644 index 00000000..5b1664f3 --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/migration/loading-csv-data.md @@ -0,0 +1,178 @@ +--- +title: CSV data +--- + +## Convert CSV to JSON + +There are many tools available to convert CSV to JSON. You can import large data sets to Dgraph using [Dgraph Live Loader](live-loader) or [Dgraph Bulk Loader](bulk-loader). In these examples, the `csv2json` tool is used, and the data is imported using the **Mutate** tab in Ratel. + +### Before you begin + +* Install [`csv2json`](https://www.npmjs.com/package/csv2json) conversion tool. +* Install `jq` a lightweight and flexible command-line JSON processor. +* Connect the Dgraph instance to Ratel for queries, mutations and visualizations. + +#### Example 1 + +1. Create a `names.csv` file with these details: + + ```csv + Name,URL + Dgraph,https://github.com/dgraph-io/dgraph + Badger,https://github.com/dgraph-io/badger + ``` + +2. Change to the directory that contains the `names.csv` file and convert it to `names.json`: + + ```sh + $ csv2json names.csv --out names.json + ``` +3. To prettify a JSON file, use the jq '.' command: + + ```sh + $ cat names.json | jq '.' + ``` + The output is similar to: + ```sh + [ + { + "Name": "Dgraph", + "URL": "https://github.com/dgraph-io/dgraph" + }, + { + "Name": "Badger", + "URL": "https://github.com/dgraph-io/badger" + } + ] + ``` + + This JSON file follows + the [JSON Mutation Format](/dql/json-mutation-format), it can be loaded into Dgraph using [Dgraph Live Loader](live-loader) , [Dgraph Bulk Loader](bulk-loader) or the programmatic clients. + +4. To load the data to Ratel and HTTP clients. The JSON data has to be stored within the `"set"` key. You can use `jq` to transform the JSON into the correct format: + + ```sh + $ cat names.json | jq '{ set: . }' + ``` + + An output similar to this appears: + ```json + { + "set": [ + { + "Name": "Dgraph", + "URL": "https://github.com/dgraph-io/dgraph" + }, + { + "Name": "Badger", + "URL": "https://github.com/dgraph-io/badger" + } + ] + } + ``` +5. Paste the output in the **Mutate** tab of **Console** in Ratel. +6. Click **Run** to import data. +7. To view the imported data paste the following in the **Query** tab and click **Run**: + + ```dql + { + names(func: has(URL)) { + Name + } + } + ``` + + +#### Example 2 + +1. Create a `connects.csv` file that's connecting nodes together. The `connects` field should be of the `uid` type. + + ```csv + uid,connects + _:a,_:b + _:a,_:c + _:c,_:d + _:d,_:a + ``` + +2. To get the correct JSON format, you can convert the CSV into JSON and use `jq` +to transform it in the correct format where the `connects` edge is a node `uid`. +This JSON file can be loaded into Dgraph using the programmatic clients. + + ```sh + $ csv2json connects.csv | jq '[ .[] | { uid: .uid, connects: { uid: .connects } } ]' + ``` + The output is similar to: + + ```json + [ + { + "uid": "_:a", + "connects": { + "uid": "_:b" + } + }, + { + "uid": "_:a", + "connects": { + "uid": "_:c" + } + }, + { + "uid": "_:c", + "connects": { + "uid": "_:d" + } + }, + { + "uid": "_:d", + "connects": { + "uid": "_:a" + } + } + ] + ``` + +3. To get an output of the mutation format accepted in Ratel UI and HTTP clients: + + ```sh + $ csv2json connects.csv | jq '{ set: [ .[] | {uid: .uid, connects: { uid: .connects } } ] }' + ``` + + The output is similar to: + + ```json + { + "set": [ + { + "uid": "_:a", + "connects": { + "uid": "_:b" + } + }, + { + "uid": "_:a", + "connects": { + "uid": "_:c" + } + }, + { + "uid": "_:c", + "connects": { + "uid": "_:d" + } + }, + { + "uid": "_:d", + "connects": { + "uid": "_:a" + } + } + ] + } + ``` +:::note +To reuse existing integer IDs from a CSV file as UIDs in Dgraph, use Dgraph Zero's [assign endpoint](/admin/admin-endpoints) before loading data to allocate a range of UIDs that can be safely assigned. +::: + +4. Paste the output in the **Mutate** tab of **Console** in Ratel, and click **Run** to import data. diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/migration/migrate-tool.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/migration/migrate-tool.md new file mode 100644 index 00000000..81ca2682 --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/migration/migrate-tool.md @@ -0,0 +1,72 @@ +--- +title: MySQL data +--- + +You can use the Dgraph migration tool to convert a MySQL database tables into a schema and RDF file, and then load the resulting dataset into Dgraph. + +## Deriving a Dgraph schema from SQL + +Before converting the data, the migration tool needs to derive the schema of each predicate. +Dgraph follows two simple rules for converting the schema: + +1. For plain attributes, there is usually a one-to-one mapping between a SQL data type and the +Dgraph datatype. For instance, a `Body` column in the `Posts` table is of type `text`, +and hence, the predicate `posts.Body` is of type `string`: `posts.Body: string .` +2. The predicates representing inter-object relationships, like `posts.OwnerUserId.`, simply have the type +`[uid]`, meaning following the predicate leads us to a set of other objects. + +### Using the Migration tool +You can run the Dgraph migrate tool using this command: + +```sh +dgraph migrate [flags] +``` +1. Create a `config.properties` file that has the following settings and values should not be in quotes: + + ```txt + user = + password = + db = + ``` + +2. Export the SQL database into `schema.txt` and `sql.rdf` file: + + ```sh + dgraph migrate --config config.properties --output_schema schema.txt --output_data sql.rdf + ``` + + An output similar to this appears: + + ```txt + Dumping table xyz + Dumping table constraints xyz + ... + ``` + +:::note +If you are connecting to a remote DB hosted on AWS, GCP, and others, you need to pass the flags `--host`, and `--port`. +For description of the various flags in the migration tool, see [command line options](import-data). +::: + +After the migration is complete, two new files are available: + +- an RDF file `sql.rdf` containing all the N-Quad entries +- a schema file `schema.txt`. + +### Importing the data + +The two files can then be imported into Dgraph using the [Dgraph Live Loader](live-loader) +or [Bulk Loader](bulk-loader). Sometimes you might want to customize your schema. +For example, you might add an index to a predicate, or change an inter-object predicate (edge) from +unidirectional to bidirectional by adding the `@reverse` directive. If you would like such customizations, you should do it by editing the schema file generated by the migration tool before feeding the files to the Live Loader or Bulk Loader. + +* To import the data into Dgraph using the Live Loader to Dgraph Zero and Alpha servers running on the default ports use: + + ```sh + dgraph live -z localhost:5080 -a localhost:9080 --files sql.rdf --format=rdf --schema schema.txt + ``` +* To import data to Dgraph Cloud use: + ```sh + dgraph live --slash_grpc_endpoint=:443 -f sql.rdf --format=rdf --schema schema.txt -t + ``` + For detailed instructions to import data to Dgraph cloud, see [import data](https://dgraph.io/docs/cloud/admin/import-export/). diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/quick-start.mdx b/docusaurus-docs/docs_versioned_docs/version-v25.4/quick-start.mdx new file mode 100644 index 00000000..c6a539aa --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/quick-start.mdx @@ -0,0 +1,157 @@ +--- +title: Quick Start +--- +import QueryTab from '/images/quickstart-querytab.png'; + + + +Welcome to Dgraph! This guide will get you up and running in minutes. You'll learn how to start Dgraph, load sample data, and run your first queries. + +## Prerequisites + +- [Docker](https://www.docker.com/) installed and running +- A terminal window +- About 5 minutes + +## Step 1: Create Quickstart Working Directory + +Create a directory to work from while following the guide, and open a terminal session in that directory. + +For example: + +```bash +mkdir ~/Desktop/dgraph-quickstart +cd ~/Desktop/dgraph-quickstart +`` +## Step 2: Start Dgraph + +From the directory you created, run Dgraph using the official Docker image: + +```bash +docker run --detach --name dgraph-play \ + -v $(pwd):/dgraph \ + -p "8080:8080" \ + -p "9080:9080" \ + --pull always dgraph/standalone:latest +``` +This command: +- Starts Dgraph in the background (`--detach`) +- Names the container `dgraph-play` for easy management +- Mounts your current directory to `/dgraph` in the container +- Exposes ports 8080 (HTTP) and 9080 (gRPC) + +**Verify it's running:** + +```bash +curl http://localhost:8080/health | jq '.[0].status' +``` + +Or, if `jq` is *not* installed on your system: + +```bash +(curl http://localhost:8080/health 2>/dev/null | grep -q '"status":"healthy"' && echo "healthy") || echo "NOT healthy" +``` +Either will output `healthy` if everything is OK. + + + +## Step 3: Download Sample Data + +Download the sample dataset (movie data) to the working directory you created: + +```bash +curl -LO https://github.com/dgraph-io/dgraph-benchmarks/raw/refs/heads/main/data/1million.rdf.gz +curl -LO https://raw.githubusercontent.com/dgraph-io/dgraph-benchmarks/refs/heads/main/data/1million.schema +``` + +## Step 4: Load Sample Data + +Load the data into Dgraph: + +```bash +docker exec -it dgraph-play dgraph live \ + -f 1million.rdf.gz \ + -s 1million.schema +``` + +This loads over 1 million movie-related facts into your database. The process takes about 15-30 seconds. + + + +## Step 5: Open Ratel UI + +[Ratel](dgraph-glossary#ratel) is Dgraph's visual query interface. Start it with: + +```bash +docker run --rm -it -p 8000:8000 dgraph/ratel:latest +``` + +And then navigate to [http://localhost:8000](http://localhost:8000) in your browser. + +Ratel should start already configured to connect to your local Dgraph instance. +You can confirm by verifying that the text underneath the 'lightning' connection button at the top of left side toolbar shows "http://localhost:8080". + +If it does not: + 1. Click on the connection icon. + 1. Enter `http://localhost:8080` as the Dgraph connection string + 2. Click **Connect** + + + +You're now connected! Click **Continue** to access the console. + +## Step 4: Run Your First Query + +In Ratel's **Query** tab, paste this [DQL](dgraph-glossary#dql) query: + +```dql +{ + film(func: has(genre), first: 3) { + name@* + genre { + name: name@. + } + starring { + performance.actor { + name: name@. + } + performance.character { + name: name@. + } + } + } +} +``` + +Ratel Console + +Click **Run** to execute the query. This finds movies with genres and displays their details, including actors and characters. + +**What this query does:** +- Finds nodes that have a `genre` predicate +- Returns the first 3 results +- Retrieves movie names, genres, and starring information + +## Step 5: Explore the Results + +View your results in two ways: + +1. **JSON tab** - See the raw data structure +2. **Graph tab** - Visualize the relationships + +Switch to the **Graph** tab to see how movies, genres, actors, and characters are connected. + +![image](/images/quickstart-dql-ratel.png) + +## What's Next? + +Congratulations! You've successfully: +- ✅ Started a Dgraph instance +- ✅ Loaded real-world data +- ✅ Run your first graph query +- ✅ Explored results visually + +**Continue learning:** +- [DQL Query Guide](dql/query/dql-query) - Master Dgraph's query language +- [Clients](clients) - Connect from your application +- [Installation Guide](installation/download) - Production deployment options diff --git a/docusaurus-docs/docs_versioned_docs/version-v25.4/releases/index.md b/docusaurus-docs/docs_versioned_docs/version-v25.4/releases/index.md new file mode 100644 index 00000000..18051b63 --- /dev/null +++ b/docusaurus-docs/docs_versioned_docs/version-v25.4/releases/index.md @@ -0,0 +1,29 @@ +--- +title: Changelog +--- + +The latest Dgraph release is the v25 series. + +Dgraph releases are following semver + +To learn about the latest releases and other important announcements, watch the +[announcements][] category on Github discussions. + +[announcements]: https://github.com/orgs/dgraph-io/discussions/categories/announcements + +## Release series + + Release | First Release Date | +-----------------------|--------------------| + [v25.3.0][] | Mar 2026 | + [v25.2.0][] | Jan 2026 | + [v25.1.0][] | Dec 2025 | + [v25.0.0][] | Oct 2025 | + [v24.1.4][] | Aug 2025 | + +[v25.3.0]: https://github.com/dgraph-io/dgraph/releases/tag/v25.3.0 +[v25.2.0]: https://github.com/dgraph-io/dgraph/releases/tag/v25.2.0 +[v25.1.0]: https://github.com/dgraph-io/dgraph/releases/tag/v25.1.0 +[v25.0.0]: https://github.com/dgraph-io/dgraph/releases/tag/v25.0.0 +[v24.1.4]: https://github.com/dgraph-io/dgraph/releases/tag/v24.1.4 + diff --git a/docusaurus-docs/docs_versioned_sidebars/version-v25.4-sidebars.json b/docusaurus-docs/docs_versioned_sidebars/version-v25.4-sidebars.json new file mode 100644 index 00000000..28d84a45 --- /dev/null +++ b/docusaurus-docs/docs_versioned_sidebars/version-v25.4-sidebars.json @@ -0,0 +1,271 @@ +{ + "docsSidebar": [ + "dgraph-overview", + "quick-start", + { + "type": "category", + "label": "Query Language", + "link": { + "type": "doc", + "id": "dql/index" + }, + "items": [ + "dql/dql-endpoints", + "dql/dql-schema", + "dql/dql-rdf", + "dql/json-mutation-format", + "dql/predicate-indexing", + "dql/indexing-custom-tokenizers", + { + "type": "category", + "label": "Query", + "link": { + "type": "doc", + "id": "dql/query/index" + }, + "items": [ + "dql/query/running-examples", + "dql/query/dql-query", + "dql/query/facets", + "dql/query/functions", + "dql/query/graphql-variables", + "dql/query/alias", + "dql/query/pagination", + "dql/query/count", + "dql/query/sorting", + "dql/query/variables", + "dql/query/aggregation", + "dql/query/expand-predicates", + "dql/query/kshortest-path-queries", + "dql/query/debug", + "dql/query/fragments", + "dql/query/language-support", + { + "type": "category", + "label": "Directives", + "link": { + "type": "doc", + "id": "dql/query/directive/index" + }, + "items": [ + "dql/query/directive/cascade-directive", + "dql/query/directive/filter", + "dql/query/directive/groupby", + "dql/query/directive/ignorereflex-directive", + "dql/query/directive/normalize-directive", + "dql/query/directive/recurse-query" + ] + } + ] + }, + "dql/dql-mutation", + "dql/upserts", + "dql/tips/index" + ] + }, + { + "type": "category", + "label": "Clients", + "link": { + "type": "doc", + "id": "clients/index" + }, + "items": [ + "clients/raw-http", + "clients/python", + "clients/go", + "clients/java", + "clients/csharp", + "clients/unofficial-clients", + { + "type": "category", + "label": "JavaScript", + "items": [ + "clients/javascript/index", + "clients/javascript/grpc", + "clients/javascript/http" + ] + } + ] + }, + { + "type": "category", + "label": "Installation", + "link": { + "type": "doc", + "id": "installation/index" + }, + "items": [ + "installation/download", + "installation/dgraph-architecture", + "installation/deployment-patterns", + "installation/single-host-setup", + { + "type": "category", + "label": "HA Cluster setup", + "items": [ + "installation/ha-cluster/helm-chart", + "installation/ha-cluster/ha-cluster-k8s-kind" + ] + }, + "installation/lambda-server", + { + "type": "category", + "label": "Configuration", + "items": [ + "installation/configuration/restrict-mutation-operations", + "installation/configuration/enable-acl", + "installation/configuration/change-data-capture", + "installation/configuration/encryption-at-rest", + "installation/configuration/learner-nodes", + "installation/configuration/license" + ] + }, + "installation/production-checklist" + ] + }, + { + "type": "category", + "label": "Administration", + "link": { + "type": "doc", + "id": "admin/index" + }, + "items": [ + "admin/admin-endpoints", + { + "type": "category", + "label": "Admin Tasks", + "link": { + "type": "doc", + "id": "admin/admin-tasks/index" + }, + "items": [ + "admin/admin-tasks/check-cluster-health", + "admin/admin-tasks/view-cluster-state", + "admin/admin-tasks/update-dgraph-types", + "admin/admin-tasks/user-management-access-control", + "admin/admin-tasks/multitenancy", + "admin/admin-tasks/export-database", + "admin/admin-tasks/shut-down-database", + "admin/admin-tasks/delete-database", + "admin/admin-tasks/upgrade-database", + "admin/admin-tasks/binary-backups" + ] + }, + { + "type": "category", + "label": "Observability", + "link": { + "type": "doc", + "id": "admin/observability/index" + }, + "items": [ + "admin/observability/monitoring", + "admin/observability/metrics", + "admin/observability/tracing", + "admin/observability/audit-logs", + "admin/observability/log-format" + ] + }, + { + "type": "category", + "label": "Security", + "link": { + "type": "doc", + "id": "admin/security/index" + }, + "items": [ + "admin/security/admin-endpoint-security", + "admin/security/tls-configuration", + "admin/security/ports-usage" + ] + }, + "admin/data-compression", + "admin/troubleshooting" + ] + }, + { + "type": "category", + "label": "Data Migration", + "link": { + "type": "doc", + "id": "migration/index" + }, + "items": [ + "migration/import-data", + "migration/bulk-loader", + "migration/live-loader", + "migration/export-data", + "migration/loading-csv-data", + "migration/migrate-tool" + ] + }, + { + "type": "category", + "label": "Design Concepts", + "link": { + "type": "doc", + "id": "design-concepts/index" + }, + "items": [ + "design-concepts/acl-concept", + "design-concepts/badger-concept", + "design-concepts/clients-concept", + "design-concepts/consistency-model", + "design-concepts/discovery-concept", + "design-concepts/dql-concept", + "design-concepts/dql-graphql-layering-concept", + "design-concepts/facets-concept", + "design-concepts/graphql-concept", + "design-concepts/group-concept", + "design-concepts/index-tokenize-concept", + "design-concepts/lambda-concept", + "design-concepts/minimizing-network-calls", + "design-concepts/namespace-tenant-concept", + "design-concepts/network-call-minimization-concept", + "design-concepts/posting-list-concept", + "design-concepts/protocol-buffers-concept", + "design-concepts/queries-process", + "design-concepts/raft", + "design-concepts/relationships-concept", + "design-concepts/replication-concept", + "design-concepts/transaction-mutation-concept", + "design-concepts/transactions-concept", + "design-concepts/wal-memtable-concept", + "design-concepts/workers-concept" + ] + }, + { + "type": "category", + "label": "CLI", + "link": { + "type": "doc", + "id": "cli/index" + }, + "items": [ + "cli/acl", + "cli/alpha", + "cli/audit", + "cli/bulk", + "cli/cert", + "cli/completion", + "cli/config", + "cli/conv", + "cli/debuginfo", + "cli/decrypt", + "cli/export_backup", + "cli/increment", + "cli/live", + "cli/lsbackup", + "cli/migrate", + "cli/restore", + "cli/superflags", + "cli/upgrade", + "cli/zero" + ] + }, + "dgraph-glossary", + "releases/index" + ] +} diff --git a/docusaurus-docs/docs_versions.json b/docusaurus-docs/docs_versions.json index 63b866f5..a790137b 100644 --- a/docusaurus-docs/docs_versions.json +++ b/docusaurus-docs/docs_versions.json @@ -1,4 +1,5 @@ [ + "v25.4", "v25.3", "v25.2", "v25.1", diff --git a/docusaurus-docs/docusaurus.config.ts b/docusaurus-docs/docusaurus.config.ts index f30a2c04..c7164152 100644 --- a/docusaurus-docs/docusaurus.config.ts +++ b/docusaurus-docs/docusaurus.config.ts @@ -62,12 +62,16 @@ const config: Config = { remarkPlugins: [remarkMath], rehypePlugins: [rehypeKatex], includeCurrentVersion: false, - lastVersion: 'v25.3', + lastVersion: 'v25.4', versions: { - 'v25.3': { - label: 'v25.3 (latest)', + 'v25.4': { + label: 'v25.4 (latest)', path: '', }, + 'v25.3': { + label: 'v25.3', + path: 'v25.3', + }, 'v25.2': { label: 'v25.2', path: 'v25.2', @@ -95,12 +99,20 @@ const config: Config = { routeBasePath: 'graphql', sidebarPath: './sidebars-graphql.ts', includeCurrentVersion: false, - lastVersion: 'v25.2', + lastVersion: 'v25.4', versions: { - 'v25.2': { - label: 'v25.2 (latest)', + 'v25.4': { + label: 'v25.4 (latest)', path: '', }, + 'v25.3': { + label: 'v25.3', + path: 'v25.3', + }, + 'v25.2': { + label: 'v25.2', + path: 'v25.2', + }, 'v25.1': { label: 'v25.1', path: 'v25.1', diff --git a/docusaurus-docs/graphql_versioned_docs/version-v25.4/admin/admin-api.md b/docusaurus-docs/graphql_versioned_docs/version-v25.4/admin/admin-api.md new file mode 100644 index 00000000..00801b4d --- /dev/null +++ b/docusaurus-docs/graphql_versioned_docs/version-v25.4/admin/admin-api.md @@ -0,0 +1,814 @@ +--- +title: "Administrative API Schema" +description: "This documentation presents the Admin API and explains how to run a Dgraph database with GraphQL." + +--- + + +Here are the important types, queries, and mutations from the `admin` schema. + +```graphql + """ + The Int64 scalar type represents a signed 64‐bit numeric non‐fractional value. + Int64 can represent values in range [-(2^63),(2^63 - 1)]. + """ + scalar Int64 + + """ + The UInt64 scalar type represents an unsigned 64‐bit numeric non‐fractional value. + UInt64 can represent values in range [0,(2^64 - 1)]. + """ + scalar UInt64 + + """ + The DateTime scalar type represents date and time as a string in RFC3339 format. + For example: "1985-04-12T23:20:50.52Z" represents 20 minutes and 50.52 seconds after the 23rd hour of April 12th, 1985 in UTC. + """ + scalar DateTime + + """ + Data about the GraphQL schema being served by Dgraph. + """ + type GQLSchema @dgraph(type: "dgraph.graphql") { + id: ID! + + """ + Input schema (GraphQL types) that was used in the latest schema update. + """ + schema: String! @dgraph(pred: "dgraph.graphql.schema") + + """ + The GraphQL schema that was generated from the 'schema' field. + This is the schema that is being served by Dgraph at /graphql. + """ + generatedSchema: String! + } + + type Cors @dgraph(type: "dgraph.cors"){ + acceptedOrigins: [String] + } + + """ + A NodeState is the state of an individual node in the Dgraph cluster. + """ + type NodeState { + + """ + Node type : either 'alpha' or 'zero'. + """ + instance: String + + """ + Address of the node. + """ + address: String + + """ + Node health status : either 'healthy' or 'unhealthy'. + """ + status: String + + """ + The group this node belongs to in the Dgraph cluster. + See : https://dgraph.io/docs/deploy/cluster-setup/. + """ + group: String + + """ + Version of the Dgraph binary. + """ + version: String + + """ + Time in nanoseconds since the node started. + """ + uptime: Int64 + + """ + Time in Unix epoch time that the node was last contacted by another Zero or Alpha node. + """ + lastEcho: Int64 + + """ + List of ongoing operations in the background. + """ + ongoing: [String] + + """ + List of predicates for which indexes are built in the background. + """ + indexing: [String] + + """ + List of Enterprise Features that are enabled. + """ + ee_features: [String] + } + + type MembershipState { + counter: UInt64 + groups: [ClusterGroup] + zeros: [Member] + maxUID: UInt64 + maxNsID: UInt64 + maxTxnTs: UInt64 + maxRaftId: UInt64 + removed: [Member] + cid: String + license: License + """ + Contains list of namespaces. Note that this is not stored in proto's MembershipState and + computed at the time of query. + """ + namespaces: [UInt64] + } + + type ClusterGroup { + id: UInt64 + members: [Member] + tablets: [Tablet] + snapshotTs: UInt64 + checksum: UInt64 + } + + type Member { + id: UInt64 + groupId: UInt64 + addr: String + leader: Boolean + amDead: Boolean + lastUpdate: UInt64 + clusterInfoOnly: Boolean + forceGroupId: Boolean + } + + type Tablet { + groupId: UInt64 + predicate: String + force: Boolean + space: Int + remove: Boolean + readOnly: Boolean + moveTs: UInt64 + } + + type License { + user: String + maxNodes: UInt64 + expiryTs: Int64 + enabled: Boolean + } + + directive @dgraph(type: String, pred: String) on OBJECT | INTERFACE | FIELD_DEFINITION + directive @id on FIELD_DEFINITION + directive @secret(field: String!, pred: String) on OBJECT | INTERFACE + + type UpdateGQLSchemaPayload { + gqlSchema: GQLSchema + } + + input UpdateGQLSchemaInput { + set: GQLSchemaPatch! + } + + input GQLSchemaPatch { + schema: String! + } + + input ExportInput { + """ + Data format for the export, e.g. "rdf" or "json" (default: "rdf") + """ + format: String + """ + Namespace for the export in multi-tenant cluster. Users from guardians of galaxy can export + all namespaces by passing a negative value or specific namespaceId to export that namespace. + """ + namespace: Int + + """ + Destination for the export: e.g. Minio or S3 bucket or /absolute/path + """ + destination: String + + """ + Access key credential for the destination. + """ + accessKey: String + + """ + Secret key credential for the destination. + """ + secretKey: String + + """ + AWS session token, if required. + """ + sessionToken: String + + """ + Set to true to allow backing up to S3 or Minio bucket that requires no credentials. + """ + anonymous: Boolean + } + + input TaskInput { + id: String! + } + type Response { + code: String + message: String + } + + type ExportPayload { + response: Response + exportedFiles: [String] + } + + type DrainingPayload { + response: Response + } + + type ShutdownPayload { + response: Response + } + + type TaskPayload { + kind: TaskKind + status: TaskStatus + lastUpdated: DateTime + } + enum TaskStatus { + Queued + Running + Failed + Success + Unknown + } + enum TaskKind { + Backup + Export + Unknown + } + input ConfigInput { + """ + Estimated memory the caches can take. Actual usage by the process would be + more than specified here. The caches will be updated according to the + cache_percentage flag. + """ + cacheMb: Float + + """ + True value of logRequest enables logging of all the requests coming to alphas. + False value of logRequest disables above. + """ + logRequest: Boolean + } + + type ConfigPayload { + response: Response + } + + type Config { + cacheMb: Float + } + input RemoveNodeInput { + """ + ID of the node to be removed. + """ + nodeId: UInt64! + """ + ID of the group from which the node is to be removed. + """ + groupId: UInt64! + } + type RemoveNodePayload { + response: Response + } + input MoveTabletInput { + """ + Namespace in which the predicate exists. + """ + namespace: UInt64 + """ + Name of the predicate to move. + """ + tablet: String! + """ + ID of the destination group where the predicate is to be moved. + """ + groupId: UInt64! + } + type MoveTabletPayload { + response: Response + } + enum AssignKind { + UID + TIMESTAMP + NAMESPACE_ID + } + input AssignInput { + """ + Choose what to assign: UID, TIMESTAMP or NAMESPACE_ID. + """ + what: AssignKind! + """ + How many to assign. + """ + num: UInt64! + } + type AssignedIds { + """ + The first UID, TIMESTAMP or NAMESPACE_ID assigned. + """ + startId: UInt64 + """ + The last UID, TIMESTAMP or NAMESPACE_ID assigned. + """ + endId: UInt64 + """ + TIMESTAMP for read-only transactions. + """ + readOnly: UInt64 + } + type AssignPayload { + response: AssignedIds + } + + input BackupInput { + """ + Destination for the backup: e.g. Minio or S3 bucket. + """ + destination: String! + """ + Access key credential for the destination. + """ + accessKey: String + """ + Secret key credential for the destination. + """ + secretKey: String + """ + AWS session token, if required. + """ + sessionToken: String + """ + Set to true to allow backing up to S3 or Minio bucket that requires no credentials. + """ + anonymous: Boolean + """ + Force a full backup instead of an incremental backup. + """ + forceFull: Boolean + } + type BackupPayload { + response: Response + taskId: String + } + input RestoreInput { + """ + Destination for the backup: e.g. Minio or S3 bucket. + """ + location: String! + """ + Backup ID of the backup series to restore. This ID is included in the manifest.json file. + If missing, it defaults to the latest series. + """ + backupId: String + """ + Number of the backup within the backup series to be restored. Backups with a greater value + will be ignored. If the value is zero or missing, the entire series will be restored. + """ + backupNum: Int + """ + Path to the key file needed to decrypt the backup. This file should be accessible + by all alphas in the group. The backup will be written using the encryption key + with which the cluster was started, which might be different than this key. + """ + encryptionKeyFile: String + """ + Vault server address where the key is stored. This server must be accessible + by all alphas in the group. Default "http://localhost:8200". + """ + vaultAddr: String + """ + Path to the Vault RoleID file. + """ + vaultRoleIDFile: String + """ + Path to the Vault SecretID file. + """ + vaultSecretIDFile: String + """ + Vault kv store path where the key lives. Default "secret/data/dgraph". + """ + vaultPath: String + """ + Vault kv store field whose value is the key. Default "enc_key". + """ + vaultField: String + """ + Vault kv store field's format. Must be "base64" or "raw". Default "base64". + """ + vaultFormat: String + """ + Access key credential for the destination. + """ + accessKey: String + """ + Secret key credential for the destination. + """ + secretKey: String + """ + AWS session token, if required. + """ + sessionToken: String + """ + Set to true to allow backing up to S3 or Minio bucket that requires no credentials. + """ + anonymous: Boolean + } + type RestorePayload { + """ + A short string indicating whether the restore operation was successfully scheduled. + """ + code: String + """ + Includes the error message if the operation failed. + """ + message: String + } + input ListBackupsInput { + """ + Destination for the backup: e.g. Minio or S3 bucket. + """ + location: String! + """ + Access key credential for the destination. + """ + accessKey: String + """ + Secret key credential for the destination. + """ + secretKey: String + """ + AWS session token, if required. + """ + sessionToken: String + """ + Whether the destination doesn't require credentials (e.g. S3 public bucket). + """ + anonymous: Boolean + } + type BackupGroup { + """ + The ID of the cluster group. + """ + groupId: UInt64 + """ + List of predicates assigned to the group. + """ + predicates: [String] + } + type Manifest { + """ + Unique ID for the backup series. + """ + backupId: String + """ + Number of this backup within the backup series. The full backup always has a value of one. + """ + backupNum: UInt64 + """ + Whether this backup was encrypted. + """ + encrypted: Boolean + """ + List of groups and the predicates they store in this backup. + """ + groups: [BackupGroup] + """ + Path to the manifest file. + """ + path: String + """ + The timestamp at which this backup was taken. The next incremental backup will + start from this timestamp. + """ + since: UInt64 + """ + The type of backup, either full or incremental. + """ + type: String + } + type LoginResponse { + """ + JWT token that should be used in future requests after this login. + """ + accessJWT: String + """ + Refresh token that can be used to re-login after accessJWT expires. + """ + refreshJWT: String + } + type LoginPayload { + response: LoginResponse + } + type User @dgraph(type: "dgraph.type.User") @secret(field: "password", pred: "dgraph.password") { + """ + Username for the user. Dgraph ensures that usernames are unique. + """ + name: String! @id @dgraph(pred: "dgraph.xid") + groups: [Group] @dgraph(pred: "dgraph.user.group") + } + type Group @dgraph(type: "dgraph.type.Group") { + """ + Name of the group. Dgraph ensures uniqueness of group names. + """ + name: String! @id @dgraph(pred: "dgraph.xid") + users: [User] @dgraph(pred: "~dgraph.user.group") + rules: [Rule] @dgraph(pred: "dgraph.acl.rule") + } + type Rule @dgraph(type: "dgraph.type.Rule") { + """ + Predicate to which the rule applies. + """ + predicate: String! @dgraph(pred: "dgraph.rule.predicate") + """ + Permissions that apply for the rule. Represented following the UNIX file permission + convention. That is, 4 (binary 100) represents READ, 2 (binary 010) represents WRITE, + and 1 (binary 001) represents MODIFY (the permission to change a predicate’s schema). + The options are: + * 1 (binary 001) : MODIFY + * 2 (010) : WRITE + * 3 (011) : WRITE+MODIFY + * 4 (100) : READ + * 5 (101) : READ+MODIFY + * 6 (110) : READ+WRITE + * 7 (111) : READ+WRITE+MODIFY + Permission 0, which is equal to no permission for a predicate, blocks all read, + write and modify operations. + """ + permission: Int! @dgraph(pred: "dgraph.rule.permission") + } + input StringHashFilter { + eq: String + } + enum UserOrderable { + name + } + enum GroupOrderable { + name + } + input AddUserInput { + name: String! + password: String! + groups: [GroupRef] + } + input AddGroupInput { + name: String! + rules: [RuleRef] + } + input UserRef { + name: String! + } + input GroupRef { + name: String! + } + input RuleRef { + """ + Predicate to which the rule applies. + """ + predicate: String! + """ + Permissions that apply for the rule. Represented following the UNIX file permission + convention. That is, 4 (binary 100) represents READ, 2 (binary 010) represents WRITE, + and 1 (binary 001) represents MODIFY (the permission to change a predicate’s schema). + The options are: + * 1 (binary 001) : MODIFY + * 2 (010) : WRITE + * 3 (011) : WRITE+MODIFY + * 4 (100) : READ + * 5 (101) : READ+MODIFY + * 6 (110) : READ+WRITE + * 7 (111) : READ+WRITE+MODIFY + Permission 0, which is equal to no permission for a predicate, blocks all read, + write and modify operations. + """ + permission: Int! + } + input UserFilter { + name: StringHashFilter + and: UserFilter + or: UserFilter + not: UserFilter + } + input UserOrder { + asc: UserOrderable + desc: UserOrderable + then: UserOrder + } + input GroupOrder { + asc: GroupOrderable + desc: GroupOrderable + then: GroupOrder + } + input UserPatch { + password: String + groups: [GroupRef] + } + input UpdateUserInput { + filter: UserFilter! + set: UserPatch + remove: UserPatch + } + input GroupFilter { + name: StringHashFilter + and: UserFilter + or: UserFilter + not: UserFilter + } + input SetGroupPatch { + rules: [RuleRef!]! + } + input RemoveGroupPatch { + rules: [String!]! + } + input UpdateGroupInput { + filter: GroupFilter! + set: SetGroupPatch + remove: RemoveGroupPatch + } + type AddUserPayload { + user: [User] + } + type AddGroupPayload { + group: [Group] + } + type DeleteUserPayload { + msg: String + numUids: Int + } + type DeleteGroupPayload { + msg: String + numUids: Int + } + input AddNamespaceInput { + password: String + } + input DeleteNamespaceInput { + namespaceId: Int! + } + type NamespacePayload { + namespaceId: UInt64 + message: String + } + input ResetPasswordInput { + userId: String! + password: String! + namespace: Int! + } + type ResetPasswordPayload { + userId: String + message: String + namespace: UInt64 + } + input EnterpriseLicenseInput { + """ + The contents of license file as a String. + """ + license: String! + } + type EnterpriseLicensePayload { + response: Response + } + + type Query { + getGQLSchema: GQLSchema + health: [NodeState] + state: MembershipState + config: Config + task(input: TaskInput!): TaskPayload + + getUser(name: String!): User + getGroup(name: String!): Group + """ + Get the currently logged in user. + """ + getCurrentUser: User + queryUser(filter: UserFilter, order: UserOrder, first: Int, offset: Int): [User] + queryGroup(filter: GroupFilter, order: GroupOrder, first: Int, offset: Int): [Group] + """ + Get the information about the backups at a given location. + """ + listBackups(input: ListBackupsInput!) : [Manifest] + } + type Mutation { + + """ + Update the Dgraph cluster to serve the input schema. This may change the GraphQL + schema, the types and predicates in the Dgraph schema, and cause indexes to be recomputed. + """ + updateGQLSchema(input: UpdateGQLSchemaInput!) : UpdateGQLSchemaPayload + + """ + Starts an export of all data in the cluster. Export format should be 'rdf' (the default + if no format is given), or 'json'. + See : https://dgraph.io/docs/deploy/dgraph-administration/#export-database + """ + export(input: ExportInput!): ExportPayload + + """ + Set (or unset) the cluster draining mode. In draining mode no further requests are served. + """ + draining(enable: Boolean): DrainingPayload + + """ + Shutdown this node. + """ + shutdown: ShutdownPayload + + """ + Alter the node's config. + """ + config(input: ConfigInput!): ConfigPayload + """ + Remove a node from the cluster. + """ + removeNode(input: RemoveNodeInput!): RemoveNodePayload + """ + Move a predicate from one group to another. + """ + moveTablet(input: MoveTabletInput!): MoveTabletPayload + """ + Lease UIDs, Timestamps or Namespace IDs in advance. + """ + assign(input: AssignInput!): AssignPayload + + """ + Start a binary backup. See : https://dgraph.io/docs/enterprise-features/binary-backups/#create-a-backup + """ + backup(input: BackupInput!) : BackupPayload + """ + Start restoring a binary backup. See : https://dgraph.io/docs/enterprise-features/binary-backups/#online-restore + """ + restore(input: RestoreInput!) : RestorePayload + """ + Login to Dgraph. Successful login results in a JWT that can be used in future requests. + If login is not successful an error is returned. + """ + login(userId: String, password: String, namespace: Int, refreshToken: String): LoginPayload + """ + Add a user. When linking to groups: if the group doesn't exist it is created; if the group + exists, the new user is linked to the existing group. It's possible to both create new + groups and link to existing groups in the one mutation. + Dgraph ensures that usernames are unique, hence attempting to add an existing user results + in an error. + """ + addUser(input: [AddUserInput!]!): AddUserPayload + """ + Add a new group and (optionally) set the rules for the group. + """ + addGroup(input: [AddGroupInput!]!): AddGroupPayload + """ + Update users, their passwords and groups. As with AddUser, when linking to groups: if the + group doesn't exist it is created; if the group exists, the new user is linked to the existing + group. If the filter doesn't match any users, the mutation has no effect. + """ + updateUser(input: UpdateUserInput!): AddUserPayload + """ + Add or remove rules for groups. If the filter doesn't match any groups, + the mutation has no effect. + """ + updateGroup(input: UpdateGroupInput!): AddGroupPayload + deleteGroup(filter: GroupFilter!): DeleteGroupPayload + deleteUser(filter: UserFilter!): DeleteUserPayload + """ + Add a new namespace. + """ + addNamespace(input: AddNamespaceInput): NamespacePayload + """ + Delete a namespace. + """ + deleteNamespace(input: DeleteNamespaceInput!): NamespacePayload + """ + Reset password can only be used by the Guardians of the galaxy to reset password of + any user in any namespace. + """ + resetPassword(input: ResetPasswordInput!): ResetPasswordPayload + """ + Apply enterprise license. + """ + enterpriseLicense(input: EnterpriseLicenseInput!): EnterpriseLicensePayload + } +``` + +You'll notice that the `/admin` schema is very much the same as the schemas generated by Dgraph GraphQL. + +* The `health` query lets you know if everything is connected and if there's a schema currently being served at `/graphql`. +* The `state` query returns the current state of the cluster and group membership information. +* The `config` query returns the configuration options of the cluster set at the time of starting it. +* The `getGQLSchema` query gets the current GraphQL schema served at `/graphql`, or returns null if there's no such schema. +* The `updateGQLSchema` mutation allows you to change the schema currently served at `/graphql`. + + diff --git a/docusaurus-docs/graphql_versioned_docs/version-v25.4/admin/index.md b/docusaurus-docs/graphql_versioned_docs/version-v25.4/admin/index.md new file mode 100644 index 00000000..8d837ea3 --- /dev/null +++ b/docusaurus-docs/graphql_versioned_docs/version-v25.4/admin/index.md @@ -0,0 +1,165 @@ +--- +title: "Administrative API" +description: "This documentation presents the Admin API and explains how to run a Dgraph database with GraphQL." + +--- + + + +## GraphQL schema introspection + +GraphQL schema introspection is enabled by default, but you can disable it by +setting the `--graphql` superflag's `introspection` option to false (`--graphql introspection=false`) when +starting the Dgraph Alpha nodes in your cluster. + +## Dgraph's schema + +Dgraph's GraphQL runs in Dgraph and presents a GraphQL schema where the queries and mutations are executed in the Dgraph cluster. So the GraphQL schema is backed by Dgraph's schema. + +:::warning +this means that if you have a Dgraph instance and change its GraphQL schema, the schema of the underlying Dgraph will also be changed! +::: + +## Endpoints + +When you start Dgraph, two GraphQL endpoints are served. + +### /graphql + +At `/graphql` you'll find the GraphQL API for the types you've added. That's what your app would access and is the GraphQL entry point to Dgraph. If you need to know more about this, see the [quick start](https://dgraph.io/docs/graphql/quick-start/) and [schema docs](https://dgraph.io/docs/graphql/schema/). + +## First start + +On first starting with a blank database: + +* There's no schema served at `/graphql`. +* Querying the `/admin` endpoint for `getGQLSchema` returns `"getGQLSchema": null`. +* Querying the `/admin` endpoint for `health` lets you know that no schema has been added. + +## Validating a schema + +You can validate a GraphQL schema before adding it to your database by sending +your schema definition in an HTTP POST request to the to the +`/admin/schema/validate` endpoint, as shown in the following example: + +Request header: + +```ssh +path: /admin/schema/validate +method: POST +``` + +Request body: + +```graphql +type Person { + name: String +} +``` + +This endpoint returns a JSON response that indicates if the schema is valid or +not, and provides an error if isn't valid. In this case, the schema is valid, +so the JSON response includes the following message: `Schema is valid`. + +## Modifying a schema + +There are two ways you can modify a GraphQL schema: +- Using `/admin/schema` +- Using the `updateGQLSchema` mutation on `/admin` + +:::tip +While modifying the GraphQL schema, if you get errors like `errIndexingInProgress`, `another operation is already running` or `server is not ready`, please wait a moment and then retry the schema update. +::: + +### Using `/admin/schema` + +The `/admin/schema` endpoint provides a simplified method to add and update schemas. + +To create a schema you only need to call the `/admin/schema` endpoint with the required schema definition. For example: + +```graphql +type Person { + name: String +} +``` + +If you have the schema definition stored in a `schema.graphql` file, you can use `curl` like this: +``` +curl -X POST localhost:8080/admin/schema --data-binary '@schema.graphql' +``` + +On successful execution, the `/admin/schema` endpoint will give you a JSON response with a success code. + +### Using `updateGQLSchema` to add or modify a schema + +Another option to add or modify a GraphQL schema is the `updateGQLSchema` mutation. + +For example, to create a schema using `updateGQLSchema`, run this mutation on the `/admin` endpoint: + +```graphql +mutation { + updateGQLSchema( + input: { set: { schema: "type Person { name: String }"}}) + { + gqlSchema { + schema + generatedSchema + } + } +} +``` + +## Initial schema + +Regardless of the method used to upload the GraphQL schema, on a black database, adding this schema + +```graphql +type Person { + name: String +} +``` + +would cause the following: + +* The `/graphql` endpoint would refresh and serve the GraphQL schema generated from type `type Person { name: String }`. +* The schema of the underlying Dgraph instance would be altered to allow for the new `Person` type and `name` predicate. +* The `/admin` endpoint for `health` would return that a schema is being served. +* The mutation would return `"schema": "type Person { name: String }"` and the generated GraphQL schema for `generatedSchema` (this is the schema served at `/graphql`). +* Querying the `/admin` endpoint for `getGQLSchema` would return the new schema. + +## Migrating a schema + +Given an instance serving the GraphQL schema from the previous section, updating the schema to the following + +```graphql +type Person { + name: String @search(by: [regexp]) + dob: DateTime +} +``` + +would change the GraphQL definition of `Person` and result in the following: + +* The `/graphql` endpoint would refresh and serve the GraphQL schema generated from the new type. +* The schema of the underlying Dgraph instance would be altered to allow for `dob` (predicate `Person.dob: datetime .` is added, and `Person.name` becomes `Person.name: string @index(regexp).`) and indexes are rebuilt to allow the regexp search. +* The `health` is unchanged. +* Querying the `/admin` endpoint for `getGQLSchema` would return the updated schema. + +## Removing indexes from a schema + +Adding a schema through GraphQL doesn't remove existing data (it only removes indexes). + +For example, starting from the schema in the previous section and modifying it with the initial schema + +```graphql +type Person { + name: String +} +``` + +would have the following effects: + +* The `/graphql` endpoint would refresh to serve the schema built from this type. +* Thus, field `dob` would no longer be accessible, and there would be no search available on `name`. +* The search index on `name` in Dgraph would be removed. +* The predicate `dob` in Dgraph would be left untouched (the predicate remains and no data is deleted). diff --git a/docusaurus-docs/graphql_versioned_docs/version-v25.4/custom/custom-dql.md b/docusaurus-docs/graphql_versioned_docs/version-v25.4/custom/custom-dql.md new file mode 100644 index 00000000..d3ece23f --- /dev/null +++ b/docusaurus-docs/graphql_versioned_docs/version-v25.4/custom/custom-dql.md @@ -0,0 +1,113 @@ +--- +title: "Custom DQL" +description: "Dgraph Query Language (DQL) includes support for custom logic. Specify the DQL query you want to execute and the Dgraph GraphQL API will execute it." + +--- + + +Dgraph Query Language ([DQL](/dql/)) lets you build custom resolvers logic that goes beyond what is possible with the current GraphQL CRUD API. + +To define a DQL custom query, use the notation: +```graphql + @custom(dql: """ + ... + """) +``` + +:::tip +Since v21.03, you can also [subscribe to custom DQL](/graphql/subscriptions/#subscriptions-to-custom-dql) queries. +::: + +For example, lets say you had following schema: +```graphql +type Tweets { + id: ID! + text: String! @search(by: [fulltext]) + author: User + timestamp: DateTime! @search +} +type User { + screen_name: String! @id + followers: Int @search + tweets: [Tweets] @hasInverse(field: author) +} +``` + +and you wanted to query tweets containing some particular text sorted by the number of followers their author has. Then, +this is not possible with the automatically generated CRUD API. Similarly, let's say you have a table sort of UI +component in your application which displays only a user's name and the number of tweets done by that user. Doing this +with the auto-generated CRUD API would require you to fetch unnecessary data at client side, and then employ client side +logic to find the count. Instead, all this could simply be achieved by specifying a DQL query for such custom use-cases. + +So, you would need to modify your schema like this: +```graphql +type Tweets { + id: ID! + text: String! @search(by: [fulltext]) + author: User + timestamp: DateTime! @search +} +type User { + screen_name: String! @id + followers: Int @search + tweets: [Tweets] @hasInverse(field: author) +} +type UserTweetCount @remote { + screen_name: String + tweetCount: Int +} + +type Query { + queryTweetsSortedByAuthorFollowers(search: String!): [Tweets] @custom(dql: """ + query q($search: string) { + var(func: type(Tweets)) @filter(anyoftext(Tweets.text, $search)) { + Tweets.author { + followers as User.followers + } + authorFollowerCount as sum(val(followers)) + } + queryTweetsSortedByAuthorFollowers(func: uid(authorFollowerCount), orderdesc: val(authorFollowerCount)) { + id: uid + text: Tweets.text + author: Tweets.author { + screen_name: User.screen_name + followers: User.followers + } + timestamp: Tweets.timestamp + } + } + """) + + queryUserTweetCounts: [UserTweetCount] @custom(dql: """ + query { + queryUserTweetCounts(func: type(User)) { + screen_name: User.screen_name + tweetCount: count(User.tweets) + } + } + """) +} + +``` + +Now, if you run following query, it would fetch you the tweets containing "GraphQL" in their text, sorted by the number +of followers their author has: +```graphql +query { + queryTweetsSortedByAuthorFollowers(search: "GraphQL") { + text + } +} +``` + +There are following points to note while specifying the DQL query for such custom resolvers: + +* The name of the DQL query that you want to map to the GraphQL response, should be same as the name of the GraphQL query. +* You must use proper aliases inside DQL queries to map them to the GraphQL response. +* If you are using variables in DQL queries, their names should be same as the name of the arguments for the GraphQL query. +* For variables, only scalar GraphQL arguments like `Boolean`, `Int`, `Float`, etc are allowed. Lists and Object types are not allowed to be used as variables with DQL queries. +* You would be able to query only those many levels with GraphQL which you have mapped with the DQL query. For instance, in the first custom query above, we haven't mapped an author's tweets to GraphQL alias, so, we won't be able to fetch author's tweets using that query. +* If the custom GraphQL query returns an interface, and you want to use `__typename` in GraphQL query, then you should add `dgraph.type` as a field in DQL query without any alias. This is not required for types, only for interfaces. +* to subscribe to a custom DQL query, use the `@withSubscription` directive. See the [Subscriptions article](/graphql/subscriptions/) for more information. + +--- diff --git a/docusaurus-docs/graphql_versioned_docs/version-v25.4/custom/custom-overview.md b/docusaurus-docs/graphql_versioned_docs/version-v25.4/custom/custom-overview.md new file mode 100644 index 00000000..4c3a17c3 --- /dev/null +++ b/docusaurus-docs/graphql_versioned_docs/version-v25.4/custom/custom-overview.md @@ -0,0 +1,54 @@ +--- +title: "Custom Resolvers Overview" +description: "Dgraph creates a GraphQL API from nothing more than GraphQL types. To customize the behavior of your schema, you can implement custom resolvers." + +--- + +Dgraph creates a GraphQL API from nothing more than GraphQL types. That's great, and gets you moving fast from an idea to a running app. However, at some point, as your app develops, you might want to customize the behavior of your schema. + +In Dgraph, you do that with code (in any language you like) that implements custom resolvers. + +Dgraph doesn't execute your custom logic itself. It makes external HTTP requests. That means, you can deploy your custom logic into the same Kubernetes cluster as your Dgraph instance, deploy and call, for example, AWS Lambda functions, or even make calls to existing HTTP and GraphQL endpoints. + +## The `@custom` directive + +There are three places you can use the `@custom` directive and thus tell Dgraph where to apply custom logic. + +1) You can add custom queries to the Query type + +```graphql +type Query { + myCustomQuery(...): QueryResultType @custom(...) +} +``` + +2) You can add custom mutations to the Mutation type + +```graphql +type Mutation { + myCustomMutation(...): MutationResult @custom(...) +} +``` + +3) You can add custom fields to your types + +```graphql +type MyType { + ... + customField: FieldType @custom(...) + ... +} +``` + +## Learn more + +Find out more about the `@custom` directive [here](/graphql/custom/directive), or check out: + +* [custom query examples](/graphql/custom/query) +* [custom mutation examples](/graphql/custom/mutation), or +* [custom field examples](/graphql/custom/field) + + + + +--- diff --git a/docusaurus-docs/graphql_versioned_docs/version-v25.4/custom/directive.md b/docusaurus-docs/graphql_versioned_docs/version-v25.4/custom/directive.md new file mode 100644 index 00000000..9c2aed06 --- /dev/null +++ b/docusaurus-docs/graphql_versioned_docs/version-v25.4/custom/directive.md @@ -0,0 +1,432 @@ +--- +title: "The @custom Directive" +description: "The @custom directive is used to define custom queries, mutations, and fields. The result types can be local or remote." + +--- + +The `@custom` directive is used to define custom queries, mutations and fields. + +In all cases, the result type (of the query, mutation or field) can be either: + +* a type that's stored in Dgraph (that's any type you've defined in your schema), or +* a type that's not stored in Dgraph and is marked with the `@remote` directive. + +Because the result types can be local or remote, you can call other HTTP endpoints, call remote GraphQL, or even call back to your Dgraph instance to add extra logic on top of Dgraph's graph search or mutations. + +Here's the GraphQL definition of the directives: + +```graphql +directive @custom(http: CustomHTTP) on FIELD_DEFINITION +directive @remote on OBJECT | INTERFACE + +input CustomHTTP { + url: String! + method: HTTPMethod! + body: String + graphql: String + mode: Mode + forwardHeaders: [String!] + secretHeaders: [String!] + introspectionHeaders: [String!] + skipIntrospection: Boolean +} + +enum HTTPMethod { GET POST PUT PATCH DELETE } +enum Mode { SINGLE BATCH } +``` + +Each definition of custom logic must include: + +* the `url` where the custom logic is called. This can include a path and parameters that depend on query/mutation arguments or other fields. +* the HTTP `method` to use in the call. For example, when calling a REST endpoint with `GET`, `POST`, etc. + +Optionally, the custom logic definition can also include: + +* a `body` definition that can be used to construct a HTTP body from from arguments or fields. +* a list of `forwardHeaders` to take from the incoming request and add to the outgoing HTTP call. +Used, for example, if the incoming request contains an auth token that must be passed to the custom logic. +* a list of `secretHeaders` to take from the `Dgraph.Secret` defined in the schema file and add to the outgoing HTTP call. +Used, for example, for a server side API key and other static value that must be passed to the custom logic. +* the `graphql` query/mutation to call if the custom logic is a GraphQL server and whether to introspect or not (`skipIntrospection`) the remote GraphQL endpoint. +* `mode` which is used for resolving fields by calling an external GraphQL query/mutation. It can either be `BATCH` or `SINGLE`. +* a list of `introspectionHeaders` to take from the `Dgraph.Secret` [object](#dgraphsecret) defined in the schema file. They're added to the +introspection requests sent to the endpoint. + + +The result type of custom queries and mutations can be any object type in your schema, including `@remote` types. For custom fields the type can be object types or scalar types. + +The `method` can be any of the HTTP methods: `GET`, `POST`, `PUT`, `PATCH`, or `DELETE`, and `forwardHeaders` is a list of headers that should be passed from the incoming request to the outgoing HTTP custom request. Let's look at each of the other `http` arguments in detail. + +## Dgraph.Secret + +Sometimes you might want to forward some static headers to your custom API which can't be exposed +to the client. This could be an API key from a payment processor or an auth token for your organization +on GitHub. These secrets can be specified as comments in the schema file and then can be used in +`secretHeaders` and `introspectionHeaders` while defining the custom directive for a field/query. + + +```graphql + type Query { + getTopUsers(id: ID!): [User] @custom(http: { + url: "http://api.github.com/topUsers", + method: "POST", + introspectionHeaders: ["Github-Api-Token"], + secretHeaders: ["Authorization:Github-Api-Token"], + graphql: "..." + }) +} + +# Dgraph.Secret Github-Api-Token "long-token" +``` + +In the above request, `Github-Api-Token` would be sent as a header with value `long-token` for +the introspection request. For the actual `/graphql` request, the `Authorization` header would be sent with +the value `long-token`. + +:::note +`Authorization:Github-Api-Token` syntax tells us to use the value for +`Github-Api-Token` from `Dgraph.Secret` and forward it to the custom API with the header key as `Authorization`. +::: + +## The URL and method + +The URL can be as simple as a fixed URL string, or include details drawn from the arguments or fields. + +A simple string might look like: + +```graphql +type Query { + myCustomQuery: MyResult @custom(http: { + url: "https://my.api.com/theQuery", + method: GET + }) +} +``` + +While, in more complex cases, the arguments of the query/mutation can be used as a pattern for the URL: + +```graphql +type Query { + myGetPerson(id: ID!): Person @custom(http: { + url: "https://my.api.com/person/$id", + method: GET + }) + + getPosts(authorID: ID!, numToFetch: Int!): [Post] @custom(http: { + url: "https://my.api.com/person/$authorID/posts?limit=$numToFetch", + method: GET + }) +} +``` + +In this case, a query like + +```graphql +query { + getPosts(authorID: "auth123", numToFetch: 10) { + title + } +} +``` + +gets transformed to an outgoing HTTP GET request to the URL `https://my.api.com/person/auth123/posts?limit=10`. + +When using custom logic on fields, the URL can draw from other fields in the type. For example: + +```graphql +type User { + username: String! @id + ... + posts: [Post] @custom(http: { + url: "https://my.api.com/person/$username/posts", + method: GET + }) +} +``` + +Note that: + +* Fields or arguments used in the path of a URL, such as `username` or `authorID` in the examples above, must be marked as non-nullable (have `!` in their type); whereas, those used in parameters, such as `numToFetch`, can be nullable. +* Currently, only scalar fields or arguments are allowed to be used in URLs or bodies; though, see body below, this doesn't restrict the objects you can construct and pass to custom logic functions. +* Currently, the body can only contain alphanumeric characters in the key and other characters like `_` are not yet supported. +* Currently, constant values are not also not allowed in the body template. This would soon be supported. + +## The body + +Many HTTP requests, such as add and update operations on REST APIs, require a JSON formatted body to supply the data. In a similar way to how `url` allows specifying a url pattern to use in resolving the custom request, Dgraph allows a `body` pattern that is used to build HTTP request bodies. + +For example, this body can be structured JSON that relates a mutation's arguments to the JSON structure required by the remote endpoint. + +```graphql +type Mutation { + newMovie(title: String!, desc: String, dir: ID, imdb: ID): Movie @custom(http: { + url: "http://myapi.com/movies", + method: "POST", + body: "{ title: $title, imdbID: $imdb, storyLine: $desc, director: { id: $dir }}", + }) +``` + +A request with `newMovie(title: "...", desc: "...", dir: "dir123", imdb: "tt0120316")` is transformed into a `POST` request to `http://myapi.com/movies` with a JSON body of: + +```json +{ + "title": "...", + "imdbID": "tt0120316", + "storyLine": "...", + "director": { + "id": "dir123" + } +} +``` + +`url` and `body` templates can be used together in a single custom definition. + +For both `url` and `body` templates, any non-null arguments or fields must be present to evaluate the custom logic. And the following rules are applied when building the request from the template for nullable arguments or fields. + +* If the value of a nullable argument is present, it's used in the template. +* If a nullable argument is present, but null, then in a body `null` is inserted, while in a url nothing is added. For example, if the `desc` argument above is null then `{ ..., storyLine: null, ...}` is constructed for the body. Whereas, in a URL pattern like `https://a.b.c/endpoint?arg=$gqlArg`, if `gqlArg` is present, but null, the generated URL is `https://a.b.c/endpoint?arg=`. +* If a nullable argument is not present, nothing is added to the URL/body. That would mean the constructed body would not contain `storyLine` if the `desc` argument is missing, and in `https://a.b.c/endpoint?arg=$gqlArg` the result would be `https://a.b.c/endpoint` if `gqlArg` were not present in the request arguments. + +## Calling GraphQL custom resolvers + +Custom queries, mutations and fields can be implemented by custom GraphQL resolvers. In this case, use the `graphql` argument to specify which query/mutation on the remote server to call. The syntax includes if the call is a query or mutation, the arguments, and what query/mutation to use on the remote endpoint. + +For example, you can pass arguments to queries onward as arguments to remote GraphQL endpoints: + +```graphql +type Query { + getPosts(authorID: ID!, numToFetch: Int!): [Post] @custom(http: { + url: "https://my.api.com/graphql", + method: POST, + graphql: "query($authorID: ID!, $numToFetch: Int!) { posts(auth: $authorID, first: $numToFetch) }" + }) +} +``` + +You can also define your own inputs and pass those to the remote GraphQL endpoint. + +```graphql +input NewMovieInput { ... } + +type Mutation { + newMovie(input: NewMovieInput!): Movie @custom(http: { + url: "http://movies.com/graphql", + method: "POST", + graphql: "mutation($input: NewMovieInput!) { addMovie(data: $input) }", + }) +``` + +When a schema is uploaded, Dgraph will try to introspect the remote GraphQL endpoints on any custom logic that uses the `graphql` argument. From the results of introspection, it tries to match up arguments, input and object types to ensure that the calls to and expected responses from the remote GraphQL make sense. + +If that introspection isn't possible, set `skipIntrospection: true` in the custom definition and Dgraph won't perform GraphQL schema introspection for this custom definition. + +## Remote types + +Any type annotated with the `@remote` directive is not stored in Dgraph. This allows your Dgraph GraphQL instance to serve an API that includes both data stored locally and data stored or generated elsewhere. You can also use custom fields, for example, to join data from disparate datasets. + +Remote types can only be returned by custom resolvers and Dgraph won't generate any search or CRUD operations for remote types. + +The schema definition used to define your Dgraph GraphQL API must include definitions of all the types used. If a custom logic call returns a type not stored in Dgraph, then that type must be added to the Dgraph schema with the `@remote` directive. + +For example, you api might use custom logic to integrate with GitHub, using either `https://api.github.com` or the GitHub GraphQL api `https://api.github.com/graphql` and calling the `user` query. Either way, your GraphQL schema will need to include the type you expect back from that remote call. That could be linking a `User` as stored in your Dgraph instance with the `Repository` data from GitHub. With `@remote` types, that's as simple as adding the type and custom call to your schema. + +```graphql +# GitHub's repository type +type Repository @remote { ... } + +# Dgraph user type +type User { + # local user name = GitHub id + username: String! @id + + # ... + # other data stored in Dgraph + # ... + + # join local data with remote + repositories: [Repository] @custom(http: { + url: "https://api.github.com/users/$username/repos", + method: GET + }) +} +``` + +Just defining the connection is all it takes and then you can ask a single GraphQL query that performs a local query and joins with (potentially many) remote data sources. + +### RemoteResponse directive + +In combination with the `@remote` directive, in a GraphQL schema you can also use the `@remoteResponse` directive. +You can define the `@remoteResponse` directive on the fields of a `@remote` type in order to map the JSON key response of a custom query to a GraphQL field. + +For example, in the given GraphQL schema there's a defined custom DQL query, whose JSON response contains the results of the `groupby` clause in the `@groupby` key. By using the `@remoteResponse` directive you'll map the `groupby` field in `GroupUserMapQ` type to the `@groupby` key in the JSON response: + +```graphql +type User { + screen_name: String! @id + followers: Int @search + tweets: [Tweets] @hasInverse(field: user) +} +type UserTweetCount @remote { + screen_name: String + tweetCount: Int +} +type UserMap @remote { + followers: Int + count: Int +} +type GroupUserMapQ @remote { + groupby: [UserMap] @remoteResponse(name: "@groupby") +} +``` + +it's possible to define the following `@custom` DQL query: + +```graphql +queryUserKeyMap: [GroupUserMapQ] @custom(dql: """ +{ + queryUserKeyMap(func: type(User)) @groupby(followers: User.followers) { + count(uid) + } +} +""") +``` + +## How Dgraph processes custom results + +Given types like + +```graphql +type Post @remote { + id: ID! + title: String! + datePublished: DateTime + author: Author +} + +type Author { ... } +``` + +and a custom query + +```graphql +type Query { + getCustomPost(id: ID!): Post @custom(http: { + url: "https://my.api.com/post/$id", + method: GET + }) + + getPosts(authorID: ID!, numToFetch: Int!): [Post] @custom(http: { + url: "https://my.api.com/person/$authorID/posts?limit=$numToFetch", + method: GET + }) +} +``` + +Dgraph turns the `getCustomPost` query into a HTTP request to `https://my.api.com/post/$id` and expects a single JSON object with fields `id`, `title`, `datePublished` and `author` as result. Any additional fields are ignored, while if non-nullable fields (like `id` and `title`) are missing, GraphQL error propagation will be triggered. + +For `getPosts`, Dgraph expects the HTTP call to `https://my.api.com/person/$authorID/posts?limit=$numToFetch` to return a JSON array of JSON objects, with each object matching the `Post` type as described above. + +If the custom resolvers are GraphQL calls, like: + +```graphql +type Query { + getCustomPost(id: ID!): Post @custom(http: { + url: "https://my.api.com/graphql", + method: POST, + graphql: "query(id: ID) { post(postID: $id) }" + }) + + getPosts(authorID: ID!, numToFetch: Int!): [Post] @custom(http: { + url: "https://my.api.com/graphql", + method: POST, + graphql: "query(id: ID) { postByAuthor(authorID: $id, first: $numToFetch) }" + }) +} +``` + +then Dgraph expects a GraphQL call to `post` to return a valid GraphQL result like `{ "data": { "post": {...} } }` and will use the JSON object that is the value of `post` as the data resolved by the request. + +Similarly, Dgraph expects `postByAuthor` to return data like `{ "data": { "postByAuthor": [ {...}, ... ] } }` and will use the array value of `postByAuthor` to build its array of posts result. + +## How errors from custom endpoints are handled + +When a query returns an error while resolving from a custom HTTP endpoint, the error is added to the `errors` array and sent back to the user in the JSON response. + +When a field returns an error while resolving a custom HTTP endpoint, the field's value becomes `null` and the error is added to the `errors` JSON array. The rest of the fields are still resolved as required by the request. + +For example, a query from a custom HTTP endpoint will return an error in the following format: + +```json +{ + "errors": [ + { + "message": "Rest API returns Error for myFavoriteMovies query", + "locations": [ + { + "line": 5, + "column": 4 + } + ], + "path": [ + "Movies", + "name" + ] + } + ] +} +``` + +## How custom fields are resolved + +When evaluating a request that includes custom fields, Dgraph might run multiple resolution stages to resolve all the fields. Dgraph must also ensure it requests enough data to forfull the custom fields. For example, given the `User` type defined as: + +```graphql +type User { + username: String! @id + ... + posts: [Post] @custom(http: { + url: "https://my.api.com/person/$username/posts", + method: GET + }) +} +``` + +a query such as: + +```graphql +query { + queryUser { + username + posts + } +} +``` + +is executed by first querying in Dgraph for `username` and then using the result to resolve the custom field `posts` (which relies on `username`). For a request like: + +```graphql +query { + queryUser { + posts + } +} +``` + +Dgraph works out that it must first get `username` so it can run the custom field `posts`, even though `username` isn't part of the original query. So Dgraph retrieves enough data to satisfy the custom request, even if that involves data that isn't asked for in the query. + +There are currently a few limitations on custom fields: + +* each custom call must include either an `ID` or `@id` field +* arguments are not allowed (soon custom field arguments will be allowed and will be used in the `@custom` directive in the same manner as for custom queries and mutations), and +* a custom field can't depend on another custom field (longer term, we intend to lift this restriction). + +## Restrictions / Roadmap + +Our custom logic is still in beta and we are improving it quickly. Here's a few points that we plan to work on soon: + +* adding arguments to custom fields +* relaxing the restrictions on custom fields using id values +* iterative evaluation of `@custom` and `@remote` - in the current version you can't have `@custom` inside an `@remote` type once we add this, you'll be able to extend remote types with custom fields, and +* allowing fine tuning of the generated API, for example removing of customizing the generated CRUD mutations. + +--- diff --git a/docusaurus-docs/graphql_versioned_docs/version-v25.4/custom/field.md b/docusaurus-docs/graphql_versioned_docs/version-v25.4/custom/field.md new file mode 100644 index 00000000..be1c2bed --- /dev/null +++ b/docusaurus-docs/graphql_versioned_docs/version-v25.4/custom/field.md @@ -0,0 +1,81 @@ +--- +title: "Custom Fields" +description: "Custom fields allow you to extend your types with custom logic as well as make joins between your local data and remote data." + +--- + +Custom fields allow you to extend your types with custom logic as well as make joins between your local data and remote data. + +Let's say we are building an app for managing projects. Users will login with their GitHub id and we want to connect some data about their work stored in Dgraph with say their GitHub profile, issues, etc. + +Our first version of our users might start out with just their GitHub username and some data about what projects they are working on. + +```graphql +type User { + username: String! @id + projects: [Project] + tickets: [Ticket] +} +``` + +We can then add their GitHub repositories by just extending the definitions with the types and custom field needed to make the remote call. + +```graphql +# GitHub's repository type +type Repository @remote { ... } + +# Dgraph user type +type User { + # local user name = GitHub id + username: String! @id + + # join local data with remote + repositories: [Repository] @custom(http: { + url: "https://api.github.com/users/$username/repos", + method: GET + }) +} +``` + +We could similarly join with say the GitHub user details, or open pull requests, to further fill out the join between GitHub and our local data. Instead of the REST API, let's use the GitHub GraphQL endpoint + + +```graphql +# GitHub's User type +type GitHubUser @remote { ... } + +# Dgraph user type +type User { + # local user name = GitHub id + username: String! @id + + # join local data with remote + gitDetails: GitHubUser @custom(http: { + url: "https://api.github.com/graphql", + method: POST, + graphql: "query(username: String!) { user(login: $username) }", + skipIntrospection: true + }) +} +``` + +Perhaps our app has some measure of their velocity that's calculated by a custom function that looks at both their GitHub commits and some other places where work is added. Soon we'll have a schema where we can render a user's home page, the projects they work on, their open tickets, their GitHub details, etc. in a single request that queries across multiple sources and can mix Dgraph filtering with external calls. + +```graphql +query { + getUser(id: "aUser") { + username + projects(order: { asc: lastUpdate }, first: 10) { + projectName + } + tickets { + connectedGitIssue { ... } + } + velocityMeasure + gitDetails { ... } + repositories { ... } + } +} +``` + +--- diff --git a/docusaurus-docs/graphql_versioned_docs/version-v25.4/custom/index.md b/docusaurus-docs/graphql_versioned_docs/version-v25.4/custom/index.md new file mode 100644 index 00000000..92d87ef7 --- /dev/null +++ b/docusaurus-docs/graphql_versioned_docs/version-v25.4/custom/index.md @@ -0,0 +1,4 @@ +--- +title: "Custom Resolvers" + +--- \ No newline at end of file diff --git a/docusaurus-docs/graphql_versioned_docs/version-v25.4/custom/mutation.md b/docusaurus-docs/graphql_versioned_docs/version-v25.4/custom/mutation.md new file mode 100644 index 00000000..3a1e70fd --- /dev/null +++ b/docusaurus-docs/graphql_versioned_docs/version-v25.4/custom/mutation.md @@ -0,0 +1,49 @@ +--- +title: "Custom Mutations" +description: "With custom mutations, you can use custom logic to define values for one or more fields in a mutation." + +--- + +With custom mutations, you can use custom logic to define values for one or more +fields in a mutation. + +Let's say we have an application about authors and posts. Logged in authors can add posts, but we want to do some input validation and add extra value when a post is added. The key types might be as follows. + +```graphql +type Author { ... } + +type Post { + id: ID! + title: String + text: String + datePublished: DateTime + author: Author + ... +} +``` + +Dgraph generates an `addPost` mutation from those types, but we want to do something extra. We don't want the `author` field to come in with the mutation, that should get filled in from the JWT of the logged in user. Also, the `datePublished` shouldn't be in the input; it should be set as the current time at point of mutation. Maybe we also have some community guidelines about what might constitute an offensive `title` or `text` in a post. Maybe users can only post if they have enough community credit. + +We'll need custom code to do all that, so we can write a custom function that takes in only the title and text of the new post. Internally, it can check that the title and text satisfy the guidelines and that this user has enough credit to make a post. If those checks pass, it then builds a full post object by adding the current time as the `datePublished` and adding the `author` from the JWT information it gets from the forward header. It can then call the `addPost` mutation constructed by Dgraph to add the post into Dgraph and returns the resulting post as its GraphQL output. + +So as well as the types above, we need a custom mutation: + +```graphql +type Mutation { + newPost(title: String!, text: String): Post @custom(http:{ + url: "https://my.api.com/addPost" + method: "POST", + body: "{ postText: $text, postTitle: $title }" + forwardHeaders: ["AuthHdr"] + }) +} +``` + +## Learn more + +Find out more about how to turn off generated mutations and protecting mutations with authorization rules at: + +* Remote Types - Turning off Generated Mutations with `@remote` [Directive](/graphql/schema/directives) +* [Securing Mutations with the `@auth` Directive](/graphql/security/mutations) + +--- diff --git a/docusaurus-docs/graphql_versioned_docs/version-v25.4/custom/query.md b/docusaurus-docs/graphql_versioned_docs/version-v25.4/custom/query.md new file mode 100644 index 00000000..c52a58e1 --- /dev/null +++ b/docusaurus-docs/graphql_versioned_docs/version-v25.4/custom/query.md @@ -0,0 +1,68 @@ +--- +title: "Custom Queries" +description: "A custom query takes any number of scalar arguments and constructs the path, parameters, and body of the request that's sent to the remote endpoint." + +--- + +Let's say we want to integrate our app with an existing external REST API. There's a few things we need to know: + +* The URL of the API, the path and any parameters required +* The shape of the resulting JSON data +* The method (GET, POST, etc.), and +* What authorization we need to pass to the external endpoint + +The custom query can take any number of scalar arguments and use those to construct the path, parameters and body (we'll see an example of that in the custom mutation section) of the request that gets sent to the remote endpoint. + +In an app, you'd deploy an endpoint that does some custom work and returns data that's used in your UI, or you'd wrap some logic or call around an existing endpoint. So that we can walk through a whole example, let's use the Twitter API. + +To integrate a call that returns the data of Twitter user with our app, all we need to do is add the expected result type `TwitterUser` and set up a custom query: + +```graphql +type TwitterUser @remote { + id: ID! + name: String + screen_name: String + location: String + description: String + followers_count: Int + ... +} + +type Query{ + getCustomTwitterUser(name: String!): TwitterUser @custom(http:{ + url: "https://api.twitter.com/1.1/users/show.json?screen_name=$name" + method: "GET", + forwardHeaders: ["Authorization"] + }) +} +``` + +Dgraph will then be able to accept a GraphQL query like + +```graphql +query { + getCustomTwitterUser(name: "dgraphlabs") { + location + description + followers_count + } +} +``` + +construct a HTTP GET request to `https://api.twitter.com/1.1/users/show.json?screen_name=dgraphlabs`, attach header `Authorization` from the incoming GraphQL request to the outgoing HTTP, and make the call and return a GraphQL result. + +The result JSON of the actual HTTP call will contain the whole object from the REST endpoint (you can see how much is in the Twitter user object [here](https://developer.twitter.com/en/docs/tweets/data-dictionary/overview/user-object)). But, the GraphQL query only asked for some of that, so Dgraph filters out any returned values that weren't asked for in the GraphQL query and builds a valid GraphQL response to the query and returns GraphQL. + +```json +{ + "data": { + "getCustomTwitterUser": { "location": ..., "description": ..., "followers_count": ... } + } +} +``` + +Your version of the remote type doesn't have to be equal to the remote type. For example, if you don't want to allow users to query the full Twitter user, you include in the type definition only the fields that can be queried. + +All the usual options for custom queries are allowed; for example, you can have multiple queries in a single GraphQL request and a mix of custom and Dgraph generated queries, you can get the result compressed by setting `Accept-Encoding` to `gzip`, etc. + +--- diff --git a/docusaurus-docs/graphql_versioned_docs/version-v25.4/federation/index.md b/docusaurus-docs/graphql_versioned_docs/version-v25.4/federation/index.md new file mode 100644 index 00000000..edf61bae --- /dev/null +++ b/docusaurus-docs/graphql_versioned_docs/version-v25.4/federation/index.md @@ -0,0 +1,167 @@ +--- +title: "Apollo Federation" +description: "Dgraph now supports Apollo federation so that you can create a gateway GraphQL service that includes the Dgraph GraphQL API and other GraphQL services." + +--- + +Dgraph supports [Apollo federation](https://www.apollographql.com/docs/federation/) starting in release version 21.03. This lets you create a gateway GraphQL service that includes the Dgraph GraphQL API and other GraphQL services. + +## Support for Apollo federation directives + +The current implementation supports the following five directives: `@key`, `@extends`, `@external`, `@provides`, and `@requires`. + +### `@key` directive +This directive takes one field argument inside it: the `@key` field. There are few limitations on how to use `@key` directives: + +- Users can define the `@key` directive only once for a type +- Support for multiple key fields is not currently available. +- Since the `@key` field acts as a foreign key to resolve entities from the service where it is extended, the field provided as an argument inside the `@key` directive should be of `ID` type or have the `@id` directive on it. + +For example - + +```graphql +type User @key(fields: "id") { + id: ID! + name: String +} +``` + +### `@extends` directive +This directive provides support for extended definitions. For example, if the above-defined `User` type is defined in some other service, you can extend it in Dgraph's GraphQL service by using the `@extends` directive, as follows: + +```graphql +type User @key(fields: "id") @extends{ + id: String! @id @external + products: [Product] +} +``` +You can also achieve this with the `extend` keyword; so you have a choice between two types of syntax to extend a type into your Dgraph GraphQL service: `extend type User ...` or `type User @extends ...`. + +### `@external` directive +You use this directive when the given field is not stored in this service. It can only be used on extended type definitions. For example, it is used in the example shown above on the `id` field of the `User` type. + +### `@provides` directive +You use this directive on a field that tells the gateway to return a specific fieldset from the base type while fetching the field. + +For example - + +```graphql +type Review @key(fields: "id") { + product: Product @provides(fields: "name price") +} + +extend type Product @key(fields: "upc") { + upc: String @external + name: String @external + price: Int @external +} +``` + +While fetching `Review.product` from the `review` service, and if the `name` or `price` is also queried, the gateway will fetch these from the `review` service itself. So, the `review` service also resolves these fields, even though both fields are `@external`. + +### `@requires` directive +You use this directive on a field to annotate the fieldset of the base type. You can use it to develop a query plan where the required fields may not be needed by the client, but the service may need additional information from other services. + +For example - + +```graphql +extend type User @key(fields: "id") { + id: ID! @external + email: String @external + reviews: [Review] @requires(fields: "email") +} +``` + +When the gateway fetches `user.reviews` from the `review` service, the gateway will get `user.email` from the `User` service and provide it as an argument to the `_entities` query. + +Using `@requires` alone on a field doesn't make much sense. In cases where you need to use `@requires`, you should also add some custom logic on that field. You can add such logic using the `@lambda` or `@custom(http: {...})` directives. + +Here's an example - + +1. Schema: +```graphql +extend type User @key(fields: "id") { + id: ID! @external + email: String @external + reviews: [Review] @requires(fields: "email") @lambda +} +``` +2. Lambda Script: +```js +// returns a list of reviews for a user +async function userReviews({parent, graphql}) { + let reviews = []; + // find the reviews for a user using the email and return them. + // Even though the email has been declared `@external`, it will be available as `parent.email` as it is mentioned in `@requires`. + return reviews +} +self.addGraphQLResolvers({ + "User.reviews": userReviews +}) +``` + +## Generated queries and mutations + +In this section, you will see what all queries and mutations will be available to individual service and to the Apollo gateway. + +Let's take the below schema as an example - + +```graphql +type Mission @key(fields: "id") { + id: ID! + crew: [Astronaut] + designation: String! + startDate: String + endDate: String +} + +type Astronaut @key(fields: "id") @extends { + id: ID! @external + missions: [Mission] +} +``` + +The queries and mutations which are exposed to the gateway are - + +```graphql +type Query { + getMission(id: ID!): Mission + queryMission(filter: MissionFilter, order: MissionOrder, first: Int, offset: Int): [Mission] + aggregateMission(filter: MissionFilter): MissionAggregateResult +} + +type Mutation { + addMission(input: [AddMissionInput!]!): AddMissionPayload + updateMission(input: UpdateMissionInput!): UpdateMissionPayload + deleteMission(filter: MissionFilter!): DeleteMissionPayload + addAstronaut(input: [AddAstronautInput!]!): AddAstronautPayload + updateAstronaut(input: UpdateAstronautInput!): UpdateAstronautPayload + deleteAstronaut(filter: AstronautFilter!): DeleteAstronautPayload +} +``` + +The queries for `Astronaut` are not exposed to the gateway because they are resolved through the `_entities` resolver. However, these queries are available on the Dgraph GraphQL API endpoint. + +## Mutation for `extended` types +If you want to add an object of `Astronaut` type which is extended in this service. +The mutation `addAstronaut` takes `AddAstronautInput`, which is generated as follows: + +```graphql +input AddAstronautInput { + id: ID! + missions: [MissionRef] +} +``` + +The `id` field is of `ID` type, which is usually generated internally by Dgraph. But, In this case, it's provided as an input. The user should provide the same `id` value that is present in the GraphQL service where the type `Astronaut` is defined. + +For example, let's assume that the type `Astronaut` is defined in some other service, `AstronautService`, as follows: + +```graphql +type Astronaut @key(fields: "id") { + id: ID! + name: String! +} +``` + +When adding an object of type `Astronaut`, you should first add it to the `AstronautService` service. Then, you can call the `addAstronaut` mutation with the value of `id` provided as an argument that must be equal to the value in `AstronautService` service. diff --git a/docusaurus-docs/graphql_versioned_docs/version-v25.4/graphql-clients/endpoint/graphql-get-request.md b/docusaurus-docs/graphql_versioned_docs/version-v25.4/graphql-clients/endpoint/graphql-get-request.md new file mode 100644 index 00000000..a84a8a8f --- /dev/null +++ b/docusaurus-docs/graphql_versioned_docs/version-v25.4/graphql-clients/endpoint/graphql-get-request.md @@ -0,0 +1,25 @@ +--- +title: "GET Request" +description: "Get the structure for GraphQL requests and responses, how to enable compression for them, and configuration options for extensions." + +--- + +
+ +GraphQL request may also be sent using an ``HTTP GET`` operation. + +\GET requests must be sent in the following format. The query, variables, and operation are sent as URL-encoded query parameters in the URL. + +``` +http://localhost:8080/graphql?query={...}&variables={...}&operationName=... +``` + +- `query` is mandatory +- `variables` is only required if the query contains GraphQL variables. +- `operationName` is only required if there are multiple operations in the query; in which case, operations must also be named. + +
+ + + + diff --git a/docusaurus-docs/graphql_versioned_docs/version-v25.4/graphql-clients/endpoint/graphql-request.md b/docusaurus-docs/graphql_versioned_docs/version-v25.4/graphql-clients/endpoint/graphql-request.md new file mode 100644 index 00000000..96405ac7 --- /dev/null +++ b/docusaurus-docs/graphql_versioned_docs/version-v25.4/graphql-clients/endpoint/graphql-request.md @@ -0,0 +1,374 @@ +--- +title: "POST Request" +description: "Get the structure for GraphQL requests and responses, how to enable compression for them, and configuration options for extensions." + +--- + + +## POST ``/graphql`` + +### Headers + + +| Header | Optionality | Value | +|:------|:------|:------| +| Content-Type | mandatory | `application/graphql` or `application/json` | +| Content-Encoding | optional | `gzip` to send compressed data | +| Accept-Encoding | optional | `gzip` to enabled data compression on response| +| X-Dgraph-AccessToken | if ``ACL`` is enabled | pass the access token you got in the login response to access predicates protected by an ACL| +| X-Auth-Token | if ``anonymous access`` is disabled |Admin Key or Client key| +| header as set in ``Dgraph.Authorization`` | if GraphQL ``Dgraph.Authorization`` is set | valid JWT used by @auth directives | + + + + +:::note +Refer to GraphQL [security](/graphql/security) settings for explanations about ``anonymous access`` and ``Dgraph.Authorization``. +::: + + +### Payload format +POST requests sent with the Content-Type header `application/graphql` must have a POST body content as a GraphQL query string. For example, the following is a valid POST body for a query: + +```graphql +query { + getTask(id: "0x3") { + id + title + completed + user { + username + name + } + } +} +``` + +POST requests sent with the Content-Type header `application/json` must have a POST body in the following JSON format: + +```json +{ + "query": "...", + "operationName": "...", + "variables": { "var": "val", ... } +} +``` + +GraphQL requests can contain one or more operations. Operations include `query`, `mutation`, or `subscription`. If a request only has one operation, then it can be unnamed like the following: + +## Single Operation + +The most basic request contains a single anonymous (unnamed) operation. Each operation can have one or more queries within in. For example, the following query has `query` operation running the queries "getTask" and "getUser": + +```graphql +query { + getTask(id: "0x3") { + id + title + completed + } + getUser(username: "dgraphlabs") { + username + } +} +``` + +Response: + +```json +{ + "data": { + "getTask": { + "id": "0x3", + "title": "GraphQL docs example", + "completed": true + }, + "getUser": { + "username": "dgraphlabs" + } + } +} +``` + +You can optionally name the operation as well, though it's not required if the request only has one operation as it's clear what needs to be executed. + +### Query Shorthand + +If a request only has a single query operation, then you can use the short-hand form of omitting the "query" keyword: + +```graphql +{ + getTask(id: "0x3") { + id + title + completed + } + getUser(username: "dgraphlabs") { + username + } +} +``` + +This simplifies queries when a query doesn't require an operation name or variables. + +## Multiple Operations + +If a request has two or more operations, then each operation must have a name. A request can only execute one operation, so you must also include the operation name to execute in the request. Every operation name in a request must be unique. + +For example, in the following request has the operation names "getTaskAndUser" and "completedTasks". + +```graphql +query getTaskAndUser { + getTask(id: "0x3") { + id + title + completed + } + queryUser(filter: {username: {eq: "dgraphlabs"}}) { + username + name + } +} + +query completedTasks { + queryTask(filter: {completed: true}) { + title + completed + } +} +``` + +When executing the following request (as an HTTP POST request in JSON format), specifying the "getTaskAndUser" operation executes the first query: + +```json +{ + "query": "query getTaskAndUser { getTask(id: \"0x3\") { id title completed } queryUser(filter: {username: {eq: \"dgraphlabs\"}}) { username name }\n}\n\nquery completedTasks { queryTask(filter: {completed: true}) { title completed }}", + "operationName": "getTaskAndUser" +} +``` + +```json +{ + "data": { + "getTask": { + "id": "0x3", + "title": "GraphQL docs example", + "completed": true + }, + "queryUser": [ + { + "username": "dgraphlabs", + "name": "Dgraph Labs" + } + ] + } +} +``` + +And specifying the "completedTasks" operation executes the second query: + +```json +{ + "query": "query getTaskAndUser { getTask(id: \"0x3\") { id title completed } queryUser(filter: {username: {eq: \"dgraphlabs\"}}) { username name }\n}\n\nquery completedTasks { queryTask(filter: {completed: true}) { title completed }}", + "operationName": "completedTasks" +} +``` + +```json +{ + "data": { + "queryTask": [ + { + "title": "GraphQL docs example", + "completed": true + }, + { + "title": "Show second operation", + "completed": true + } + ] + } +} +``` + +### multiple queries execution + +When an operation contains multiple queries, they are run concurrently and independently in a Dgraph readonly transaction per query. + +When an operation contains multiple mutations, they are run serially, in the order listed in the request, and in a transaction per mutation. If a mutation fails, the following mutations are not executed, and previous mutations are not rolled back. + + +### Variables + +Variables simplify GraphQL queries and mutations by letting you pass data separately. A GraphQL request can be split into two sections: one for the query or mutation, and another for variables. + +Variables can be declared after the `query` or `mutation` and are passed like arguments to a function and begin with `$`. + +#### Query Example + +```graphql +query post($filter: PostFilter) { + queryPost(filter: $filter) { + title + text + author { + name + } + } +} +``` + +**Variables** + +```graphql +{ + "filter": { + "title": { + "eq": "First Post" + } + } +} +``` + + +#### Mutation Example + +```graphql +mutation addAuthor($author: AddAuthorInput!) { + addAuthor(input: [$author]) { + author { + name + posts { + title + text + } + } + } +} +``` + +**Variables** + +```graphql +{ + "author": { + "name": "A.N. Author", + "dob": "2000-01-01", + "posts": [{ + "title": "First Post", + "text": "Hello world!" + }] + } +} +``` + + +### Fragments +A GraphQL fragment is associated with a type and is a reusable subset of the fields from this type. +Here, we declare a `postData` fragment that can be used with any `Post` object: + +```graphql +fragment postData on Post { + id + title + text + author { + username + displayName + } +} +query allPosts { + queryPost(order: { desc: title }) { + ...postData + } +} +mutation addPost($post: AddPostInput!) { + addPost(input: [$post]) { + post { + ...postData + } + } +} +``` + + + +### Using fragments with interfaces + +It is possible to define fragments on interfaces. +Here's an example of a query that includes in-line fragments: + +**Schema** + +```graphql +interface Employee { + ename: String! +} +interface Character { + id: ID! + name: String! @search(by: [exact]) +} +type Human implements Character & Employee { + totalCredits: Float +} +type Droid implements Character { + primaryFunction: String +} +``` + +**Query** + +```graphql +query allCharacters { + queryCharacter { + name + __typename + ... on Human { + totalCredits + } + ... on Droid { + primaryFunction + } + } +} +``` + +The `allCharacters` query returns a list of `Character` objects. Since `Human` and `Droid` implements the `Character` interface, the fields in the result would be returned according to the type of object. + +**Result** + +```graphql +{ + "data": { + "queryCharacter": [ + { + "name": "Human1", + "__typename": "Human", + "totalCredits": 200.23 + }, + { + "name": "Human2", + "__typename": "Human", + "totalCredits": 2.23 + }, + { + "name": "Droid1", + "__typename": "Droid", + "primaryFunction": "Code" + }, + { + "name": "Droid2", + "__typename": "Droid", + "primaryFunction": "Automate" + } + ] + } +} +``` + + + + + + + diff --git a/docusaurus-docs/graphql_versioned_docs/version-v25.4/graphql-clients/endpoint/graphql-response.md b/docusaurus-docs/graphql_versioned_docs/version-v25.4/graphql-clients/endpoint/graphql-response.md new file mode 100644 index 00000000..67356a23 --- /dev/null +++ b/docusaurus-docs/graphql_versioned_docs/version-v25.4/graphql-clients/endpoint/graphql-response.md @@ -0,0 +1,182 @@ +--- +title: "HTTP Response" +description: "Get the structure for GraphQL requests and responses, how to enable compression for them, and configuration options for extensions." + +--- + +
+ + +### Responses +All responses, including errors, always return HTTP 200 OK status codes. + +The response is a JSON map including the fields `"data"`, `"errors"`, or `"extensions"` following the GraphQL specification. They follow the following formats. + +Successful queries are in the following format: + +```json +{ + "data": { ... }, + "extensions": { ... } +} +``` + +Queries that have errors are in the following format. + +```json +{ + "errors": [ ... ], +} +``` + + +#### "data" field + +The "data" field contains the result of your GraphQL request. The response has exactly the same shape as the result. For example, notice that for the following query, the response includes the data in the exact shape as the query. + +Query: + +```graphql +query { + getTask(id: "0x3") { + id + title + completed + user { + username + name + } + } +} +``` + +Response: + +```json +{ + "data": { + "getTask": { + "id": "0x3", + "title": "GraphQL docs example", + "completed": true, + "user": { + "username": "dgraphlabs", + "name": "Dgraph Labs" + } + } + } +} +``` + +#### "errors" field + +The "errors" field is a JSON list where each entry has a `"message"` field that describes the error and optionally has a `"locations"` array to list the specific line and column number of the request that points to the error described. For example, here's a possible error for the following query, where `getTask` needs to have an `id` specified as input: + +Query: +```graphql +query { + getTask() { + id + } +} +``` + +Response: +```json +{ + "errors": [ + { + "message": "Field \"getTask\" argument \"id\" of type \"ID!\" is required but not provided.", + "locations": [ + { + "line": 2, + "column": 3 + } + ] + } + ] +} +``` +#### Error propagation +Before returning query and mutation results, Dgraph uses the types in the schema to apply GraphQL [value completion](https://graphql.github.io/graphql-spec/June2018/#sec-Value-Completion) and [error handling](https://graphql.github.io/graphql-spec/June2018/#sec-Errors-and-Non-Nullability). That is, `null` values for non-nullable fields, e.g. `String!`, cause error propagation to parent fields. + +In short, the GraphQL value completion and error propagation mean the following. + +* Fields marked as nullable (i.e. without `!`) can return `null` in the json response. +* For fields marked as non-nullable (i.e. with `!`) Dgraph never returns null for that field. +* If an instance of type has a non-nullable field that has evaluated to null, the whole instance results in null. +* Reducing an object to null might cause further error propagation. For example, querying for a post that has an author with a null name results in null: the null name (`name: String!`) causes the author to result in null, and a null author causes the post (`author: Author!`) to result in null. +* Error propagation for lists with nullable elements, e.g. `friends [Author]`, can result in nulls inside the result list. +* Error propagation for lists with non-nullable elements results in null for `friends [Author!]` and would cause further error propagation for `friends [Author!]!`. + +Note that, a query that results in no values for a list will always return the empty list `[]`, not `null`, regardless of the nullability. For example, given a schema for an author with `posts: [Post!]!`, if an author has not posted anything and we queried for that author, the result for the posts field would be `posts: []`. + +A list can, however, result in null due to GraphQL error propagation. For example, if the definition is `posts: [Post!]`, and we queried for an author who has a list of posts. If one of those posts happened to have a null title (title is non-nullable `title: String!`), then that post would evaluate to null, the `posts` list can't contain nulls and so the list reduces to null. + +#### "extensions" field + +The "extensions" field contains extra metadata for the request with metrics and trace information for the request. + +- `"touched_uids"`: The number of nodes that were touched to satisfy the request. This is a good metric to gauge the complexity of the query. +- `"tracing"`: Displays performance tracing data in [Apollo Tracing][apollo-tracing] format. This includes the duration of the whole query and the duration of each operation. + +[apollo-tracing]: https://github.com/apollographql/apollo-tracing + +Here's an example of a query response with the extensions field: + +```json +{ + "data": { + "getTask": { + "id": "0x3", + "title": "GraphQL docs example", + "completed": true, + "user": { + "username": "dgraphlabs", + "name": "Dgraph Labs" + } + } + }, + "extensions": { + "touched_uids": 9, + "tracing": { + "version": 1, + "startTime": "2020-07-29T05:54:27.784837196Z", + "endTime": "2020-07-29T05:54:27.787239465Z", + "duration": 2402299, + "execution": { + "resolvers": [ + { + "path": [ + "getTask" + ], + "parentType": "Query", + "fieldName": "getTask", + "returnType": "Task", + "startOffset": 122073, + "duration": 2255955, + "dgraph": [ + { + "label": "query", + "startOffset": 171684, + "duration": 2154290 + } + ] + } + ] + } + } + } +} +``` + +**Turn off extensions** + +To turn off extensions set the +`--graphql` superflag's `extensions` option to false (`--graphql extensions=false`) +when running Dgraph Alpha. +
+ + + + diff --git a/docusaurus-docs/graphql_versioned_docs/version-v25.4/graphql-clients/endpoint/index.md b/docusaurus-docs/graphql_versioned_docs/version-v25.4/graphql-clients/endpoint/index.md new file mode 100644 index 00000000..e846051e --- /dev/null +++ b/docusaurus-docs/graphql_versioned_docs/version-v25.4/graphql-clients/endpoint/index.md @@ -0,0 +1,21 @@ +--- +title: "/graphql endpoint" +description: "Get the structure for GraphQL requests and responses, how to enable compression for them, and configuration options for extensions." + +--- +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + + + + +When you deploy a GraphQL schema, Dgraph serves the corresponding [spec-compliant GraphQL](https://graphql.github.io/graphql-spec/June2018/) API at the HTTP endpoint `/graphql`. GraphQL requests can be sent via HTTP POST or HTTP GET requests. + + + + + + + + + diff --git a/docusaurus-docs/graphql_versioned_docs/version-v25.4/graphql-clients/graphql-ide.md b/docusaurus-docs/graphql_versioned_docs/version-v25.4/graphql-clients/graphql-ide.md new file mode 100644 index 00000000..6338a463 --- /dev/null +++ b/docusaurus-docs/graphql_versioned_docs/version-v25.4/graphql-clients/graphql-ide.md @@ -0,0 +1,21 @@ +--- +title: "GraphQL IDEs" +description: "Dgraph" + +--- + + +As Dgraph serves a [spec-compliant GraphQL](https://graphql.github.io/graphql-spec/June2018/) API, you can use your favorite GraphQL IDE. + +- Postman +- Insomnia +- GraphiQL +- VSCode with GraphQL extensions + +### General IDE setup +- Copy Dgraph GraphQL endpoint. +- Set the security header as required. +- use IDE instrospection capability. + +You are ready to write GraphQL queries and mutation and to run them against Dgraph cluster. + diff --git a/docusaurus-docs/graphql_versioned_docs/version-v25.4/graphql-clients/graphql-ui.md b/docusaurus-docs/graphql_versioned_docs/version-v25.4/graphql-clients/graphql-ui.md new file mode 100644 index 00000000..d239e8c3 --- /dev/null +++ b/docusaurus-docs/graphql_versioned_docs/version-v25.4/graphql-clients/graphql-ui.md @@ -0,0 +1,18 @@ +--- +title: "Client libraries" +description: "Dgraph" + +--- + + +When building an application in React, Vue, Svelte or any of you favorite framework, using a GraphQL client library is a must. + +As Dgraph serves a [spec-compliant GraphQL](https://graphql.github.io/graphql-spec/June2018/) API from your schema, supports instropection and GraphQL subscriptions, the integration with GraphQL UI client libraries is seamless. + +Here is a not limited list of popular GraphQL UI clients that you can use with Dgraph to build applications: +- [graphql-request](https://github.com/jasonkuhrt/graphql-request) +- [URQL](https://github.com/urql-graphql/urql) +- [Apollo client](https://github.com/apollographql/apollo-client) + + + diff --git a/docusaurus-docs/graphql_versioned_docs/version-v25.4/graphql-clients/index.md b/docusaurus-docs/graphql_versioned_docs/version-v25.4/graphql-clients/index.md new file mode 100644 index 00000000..a48b5db6 --- /dev/null +++ b/docusaurus-docs/graphql_versioned_docs/version-v25.4/graphql-clients/index.md @@ -0,0 +1,6 @@ +--- +title: "GraphQL Client" + +--- + +### In this section \ No newline at end of file diff --git a/docusaurus-docs/graphql_versioned_docs/version-v25.4/graphql-dql/dql-for-graphql.md b/docusaurus-docs/graphql_versioned_docs/version-v25.4/graphql-dql/dql-for-graphql.md new file mode 100644 index 00000000..820e2283 --- /dev/null +++ b/docusaurus-docs/graphql_versioned_docs/version-v25.4/graphql-dql/dql-for-graphql.md @@ -0,0 +1,18 @@ +--- +title: "Use DQL in GraphQL" + +--- + + + +Dgraph Query Language ([DQL](/dql/)) can be used to extend GraphQL API capabilities when writing: + +- [custom DQL resolvers](/graphql/custom) +- [subscriptions on DQL queries](/graphql/schema/directives/directive-withsubscription) + + + +When writing custom DQL query resolvers, you must understand the [GraphQL - DQL schema mapping](/graphql/graphql-dql/graphql-dql-schema) to use proper aliases inside DQL queries to map them to the GraphQL response. + + + diff --git a/docusaurus-docs/graphql_versioned_docs/version-v25.4/graphql-dql/graphql-data-loading.md b/docusaurus-docs/graphql_versioned_docs/version-v25.4/graphql-dql/graphql-data-loading.md new file mode 100644 index 00000000..bc5f2264 --- /dev/null +++ b/docusaurus-docs/graphql_versioned_docs/version-v25.4/graphql-dql/graphql-data-loading.md @@ -0,0 +1,17 @@ +--- +title: "Data loading" + +--- + + + +After you have deployed your first GraphQL Schema, you get a GraphQL API served on ``/graphql`` endpoint and an empty backend. You can populate the graph database using the mutations operations on the GraphQL API. + +A more efficient way to populate the database is to use the Dgraph's [import tools](/migration/import-data). + +The first step is to understand the [schema mapping](/graphql/graphql-dql/graphql-dql-schema) and to prepare your RDF files or JSON files to follow the internal Dgraph predicates names. +You also have to make sure that you properly generate data for the `dgraph.type` predicate so that each node is asscociated with it's type. + +If you are using the [initial import](/migration/bulk-loader) tool, you can provide the GraphQL schema along with the data to import when executing the bulk load. + +If you are using the [live import](/migration/live-loader) tool, you must first deploy your GraphQL Schema and then proceed with the import. Deploying the schema first, will generate the predicates indexes and reduce the loading time. \ No newline at end of file diff --git a/docusaurus-docs/graphql_versioned_docs/version-v25.4/graphql-dql/graphql-data-migration.md b/docusaurus-docs/graphql_versioned_docs/version-v25.4/graphql-dql/graphql-data-migration.md new file mode 100644 index 00000000..bddd4bd8 --- /dev/null +++ b/docusaurus-docs/graphql_versioned_docs/version-v25.4/graphql-dql/graphql-data-migration.md @@ -0,0 +1,51 @@ +--- +title: "GraphQL data migration" + +--- + + + +When deploying a new version of your GraphQL Schema, Dgraph will update the underlying DQL Schema but will not alter the data. + +As explained in [GraphQL and DQL Schemas](/graphql/graphql-dql/graphql-dql-schema) overview, Dgraph has no constraints at the database level and any node with predicates is valid. + +You may face with several data GraphQL API and data discrepancies. + +### unused fields +For example, let's assume that you have deployed the following schema: +```graphql +type TestDataMigration { + id: ID! + someInfo: String! + someOtherInfo: String +} +``` + +Then you create a `TestDataMigration` with `someOtherInfo` value. + +Then you update the Schema and remove the field. +```graphql +type TestDataMigration { + id: ID! + someInfo: String! +} +``` + +The data you have previously created is still in the graph database ! + +Moreover if you delete the `TestDataMigration` object using its `id`, the GraphQL API delete operation will be successful. + +If you followed the [GraphQL - DQL Schema mapping](/graphql/graphql-dql/graphql-dql-schema), you understand that Dgraph has used the list the known list of predicates (id, someInfo) and removed them. In fact, Dgraph also removed the `dgraph.type` predicate and so this `TestDataMigration` node is not visible anymore to the GraphQL API. + +The point is that a node with this `uid` exists and has a predicate `someOtherInfo`. This is because this data has been created initially and nothing in the process of deploying a new version and then using a delete operation by ID instructed Dgraph to delete this predicate. + +You end up with a node without type (i.e without a `dgraph.type` predicate) and with an old predicate value which is 'invisible' to your GraphQL API! + +When doing a GraphQL schema deployement, you must take care of the data cleaning and data migration. +The good news is that DQL offers you the tools to identify (search) potential issues and to correct the data (mutations). + +In the previous case, you can alter the database and completely delete the predicate or you can write an 'upsert' DQL query that will search the nodes of interest and delete the unused predicate for those nodes. + +### new non-nullable field +Another obvious example appears if you deploy a new version containing a new non-nullable field for an existing type. The existing 'nodes' of the same type in the graph do not have this predicate. A Gra[hQL query reaching those nodes will return a list of errors. You can easily write an 'upsert' DQL mutation to find all node of this type not having the new predicate and update them with a default value. + diff --git a/docusaurus-docs/graphql_versioned_docs/version-v25.4/graphql-dql/graphql-dgraph.md b/docusaurus-docs/graphql_versioned_docs/version-v25.4/graphql-dql/graphql-dgraph.md new file mode 100644 index 00000000..016532fa --- /dev/null +++ b/docusaurus-docs/graphql_versioned_docs/version-v25.4/graphql-dql/graphql-dgraph.md @@ -0,0 +1,440 @@ +--- +title: "GraphQL on Existing Dgraph" + +--- + +### How to use GraphQL on an existing Dgraph instance + +In the case where you have an existing Dgraph instance which has been created using a DQL Schema (and populated with Dgraph import tools) and you want to expose some or all of the data using a GraphQL API, you can use the [@dgraph directive](/graphql/schema/directives/directive-dgraph/) to customize how Dgraph maps GraphQL type names and fields names to DQL types and predicates. + + + +### Language support in GraphQL + +In your GraphQL schema, you need to define a field for each language that you want to use. +In addition, you also need to apply the `@dgraph(pred: "...")` directive on that field, with the `pred` argument set to point to the correct DQL predicate with a language tag for the language that you want to use it for. +Dgraph will automatically add a `@lang` directive in the DQL schema for the corresponding predicate. + +:::tip +By default, the DQL predicate for a GraphQL field is generated as `Typename.FieldName`. +::: + +For example: + +```graphql +type Person { + name: String # Person.name is the auto-generated DQL predicate for this GraphQL field, unless overridden using @dgraph(pred: "...") + nameHi: String @dgraph(pred:"Person.name@hi") # this field exposes the value for the language tag `@hi` for the DQL predicate `Person.name` to GraphQL + nameEn: String @dgraph(pred:"Person.name@en") + nameHi_En: String @dgraph(pred:"Person.name@hi:en") # this field uses multiple language tags: `@hi` and `@en` + nameHi_En_untag: String @dgraph(pred:"Person.name@hi:en:.") # as this uses `.`, it will give untagged values if there is no value for `@hi` or `@en` + } +``` + +If a GraphQL field uses more than one language tag, then it won't be part of any mutation input. Like, in the above example the fields `nameHi_En` and `nameHi_En_untag` can't be given as an input to any mutation. Only the fields which use one or no language can be given in a mutation input, like `name`, `nameHi`, and `nameEn`. + +All the fields can be queried, irrespective of whether they use one language or more. + +:::note +GraphQL won’t be able to query `Person.name@*` type of language tags because of the structural requirements of GraphQL. +::: + +### Bidirectional Relationships in Dgraph GraphQL + +Dgraph provides two approaches for defining bidirectional relationships in GraphQL schemas. Each +approach has different tradeoffs regarding storage, DQL compatibility, and consistency management. + +#### Overview + +| Approach | Directive | DQL Predicates | Storage | DQL `~` Support | +| ------------------- | ----------------------- | ----------------- | ------- | --------------- | +| **Dual Predicates** | `@hasInverse` | 2 separate | Doubled | ❌ | +| **Reverse Index** | `@dgraph(pred: "~...")` | 1 with `@reverse` | Single | ✅ | + +--- + +#### Approach 1: `@hasInverse` (Dual Predicates) + +##### GraphQL Schema + +```graphql +type Post { + id: ID! + title: String! + author: Author @hasInverse(field: "posts") +} + +type Author { + id: ID! + name: String! + posts: [Post] +} +``` + +##### Generated DQL Schema + +```dql +Post.title: string . +Post.author: uid . +Author.name: string . +Author.posts: [uid] . + +type Post { + Post.title + Post.author +} + +type Author { + Author.name + Author.posts +} +``` + +##### How It Works + +1. **Two separate predicates** are created: `Post.author` and `Author.posts` +2. The GraphQL **mutation layer maintains consistency** - when you set `Post.author`, it + automatically updates `Author.posts` +3. Data is stored **twice** (once in each direction) + +##### GraphQL Mutations + +```graphql +mutation { + addPost(input: [{ title: "My Post", author: { id: "0x123" } }]) { + post { + id + } + } +} +``` + +This generates DQL mutations for **both** directions: + +```dql +{ + set { + _:post "My Post" . + _:post <0x123> . + <0x123> _:post . + } +} +``` + +##### DQL Queries + +```dql +# Forward: Post → Author +{ + posts(func: type(Post)) { + Post.title + Post.author { + Author.name + } + } +} + +# Reverse: Author → Posts (uses separate predicate) +{ + authors(func: type(Author)) { + Author.name + Author.posts { + Post.title + } + } +} + +# This does NOT work with @hasInverse: +{ + authors(func: type(Author)) { + Author.name + ~Post.author { # No @reverse index exists! + Post.title + } + } +} +``` + +##### When to Use + +- You need the GraphQL layer to manage relationship consistency +- You're primarily using GraphQL and rarely use DQL directly +- You want simpler GraphQL schema syntax + +--- + +#### Approach 2: `@dgraph(pred: "~...")` (Reverse Index) + +##### GraphQL Schema + +```graphql +type Post { + id: ID! + title: String! + author: Author # Default predicate: Post.author +} + +type Author { + id: ID! + name: String! + posts: [Post] @dgraph(pred: "~Post.author") +} +``` + +> **Note:** The `@dgraph(pred: "Post.author")` directive on the forward edge is optional. Dgraph +> automatically names predicates as `TypeName.fieldName`, so `Post.author` is the default. You only +> need `@dgraph(pred: ...)` on the forward edge if you want a custom predicate name. + +##### Generated DQL Schema + +```dql +Post.title: string . +Post.author: uid @reverse . +Author.name: string . + +type Post { + Post.title + Post.author +} + +type Author { + Author.name +} +``` + +##### How It Works + +1. **One predicate** is created: `Post.author` with `@reverse` index +2. `Author.posts` is a **virtual field** - it maps to `~Post.author` +3. Data is stored **once**; reverse traversal uses the index +4. DQL automatically maintains consistency + +##### GraphQL Mutations + +```graphql +mutation { + addPost(input: [{ title: "My Post", author: { id: "0x123" } }]) { + post { + id + } + } +} +``` + +This generates a DQL mutation for **only the forward edge**: + +```dql +{ + set { + _:post "My Post" . + _:post <0x123> . + } +} +``` + +The reverse is automatically available via `~Post.author`. + +### DQL Queries + +```dql +# Forward: Post → Author +{ + posts(func: type(Post)) { + Post.title + Post.author { + Author.name + } + } +} + +# Reverse: Author → Posts (uses @reverse index) +{ + authors(func: type(Author)) { + Author.name + ~Post.author { + Post.title + } + } +} +``` + +##### When to Use + +- You need DQL compatibility with `~predicate` syntax +- You want to minimize storage (data stored once, not twice) +- You're using both GraphQL and DQL APIs +- You prefer DQL's automatic consistency over GraphQL mutation rewriting + +--- + +#### Important Constraints + +##### Cannot Combine Both Approaches + +You **cannot** use `@hasInverse` and `@dgraph(pred: "~...")` on the same relationship: + +```graphql +# INVALID - Will produce a validation error +type Post { + author: Author @hasInverse(field: "posts") @dgraph(pred: "~Author.posts") +} +``` + +Error: + +``` +@hasInverse directive is not allowed when pred argument in @dgraph directive starts with a ~ +``` + +##### Predicate Naming + +Dgraph automatically names predicates as `TypeName.fieldName`. The reverse field references this +default name: + +```graphql +# Simple: Use default predicate naming +type Post { + author: Author +} +type Author { + posts: [Post] @dgraph(pred: "~Post.author") +} + +# Also valid: Explicit predicate name (useful for custom naming) +type Post { + author: Author @dgraph(pred: "wrote") +} +type Author { + posts: [Post] @dgraph(pred: "~wrote") +} +``` + +--- + +#### Examples + +##### Blog System + +**Using `@hasInverse`:** + +```graphql +type BlogPost { + id: ID! + title: String! @search(by: [term]) + content: String + author: User @hasInverse(field: "posts") + comments: [Comment] @hasInverse(field: "post") +} + +type User { + id: ID! + username: String! @id + posts: [BlogPost] +} + +type Comment { + id: ID! + text: String! + post: BlogPost + author: User @hasInverse(field: "comments") +} +``` + +**Using `@dgraph(pred: "~...")`:** + +```graphql +type BlogPost { + id: ID! + title: String! @search(by: [term]) + content: String + author: User @dgraph(pred: "BlogPost.author") + comments: [Comment] @dgraph(pred: "BlogPost.comments") +} + +type User { + id: ID! + username: String! @id + posts: [BlogPost] @dgraph(pred: "~BlogPost.author") + comments: [Comment] @dgraph(pred: "~Comment.author") +} + +type Comment { + id: ID! + text: String! + post: BlogPost @dgraph(pred: "~BlogPost.comments") + author: User @dgraph(pred: "Comment.author") +} +``` + +##### Movie Database + +**Using `@dgraph(pred: "~...")` for DQL compatibility:** + +```graphql +type Movie { + id: ID! + title: String! @search(by: [term, fulltext]) + releaseYear: Int @search + director: Person @dgraph(pred: "Movie.director") + actors: [Person] @dgraph(pred: "Movie.actors") +} + +type Person { + id: ID! + name: String! @search(by: [hash, term]) + directed: [Movie] @dgraph(pred: "~Movie.director") + actedIn: [Movie] @dgraph(pred: "~Movie.actors") +} +``` + +**DQL queries for this schema:** + +```dql +# Find all movies directed by a person +{ + person(func: eq(Person.name, "Christopher Nolan")) { + Person.name + ~Movie.director { + Movie.title + Movie.releaseYear + } + } +} + +# Find all actors in a movie +{ + movie(func: eq(Movie.title, "Inception")) { + Movie.title + Movie.actors { + Person.name + } + } +} +``` + +--- + +#### Migration Guide + +##### From `@hasInverse` to `@dgraph(pred: "~...")` + +If you need to migrate an existing schema: + +1. **Export your data** using the export API +2. **Update your GraphQL schema** to use `@dgraph(pred: "~...")` +3. **Transform exported data** to remove duplicate inverse edges +4. **Re-import data** + +**Warning:** This is a breaking change. The DQL predicate structure changes, and existing DQL +queries may need updates. + +##### Recommended Approach for New Projects + +For new projects that will use both GraphQL and DQL: + +- Use `@dgraph(pred: "~...")` for better DQL compatibility +- Explicitly name all predicates with `@dgraph(pred: "...")` for clarity + +For GraphQL-only projects: + +- Use `@hasInverse` for simpler syntax and automatic consistency management diff --git a/docusaurus-docs/graphql_versioned_docs/version-v25.4/graphql-dql/graphql-dql-schema.md b/docusaurus-docs/graphql_versioned_docs/version-v25.4/graphql-dql/graphql-dql-schema.md new file mode 100644 index 00000000..91df57b9 --- /dev/null +++ b/docusaurus-docs/graphql_versioned_docs/version-v25.4/graphql-dql/graphql-dql-schema.md @@ -0,0 +1,164 @@ +--- +title: "GraphQL and DQL schemas" + +--- + +The first step in mastering DQL in the context of GraphQL API is probably to understand the fundamental difference between GraphQL schema and DQL schema. + +### In GraphQL, the schema is a central notion. +GraphQL is a strongly typed language. Contrary to REST which is organized in terms of endpoints, GraphQL APIs are organized in terms of types and fields. The type system is used to define the schema, which is a contract between client and server. +GraphQL uses types to ensure Apps only ask for what’s possible and provide clear and helpful errors. + +In the [GraphQL Quick start](/graphql/quick-start), we have used a schema to generate a GraphQL API: + ```graphql +type Product { + productID: ID! + name: String @search(by: [term]) + reviews: [Review] @hasInverse(field: about) +} + +type Customer { + username: String! @id @search(by: [hash, regexp]) + reviews: [Review] @hasInverse(field: by) +} + +type Review { + id: ID! + about: Product! + by: Customer! + comment: String @search(by: [fulltext]) + rating: Int @search +} +``` + +The API and the engine logic are generated from the schema defining the types of objects we are dealing with, the fields, and the relationships in the form of fields referencing other types. + + +### In DQL, the schema described the predicates + +Dgraph maintains a list of all predicates names with their type and indexes in the [Dgraph types schema](/dql/dql-schema). + + +### Schema mapping + +When deploying a GraphQL Schema, Dgraph will generates DQL predicates and types for the graph backend. +In order to distinguish a field ``name`` from a type ``Person`` from the field ``name`` of different type (they may have different indexes), Dgraph is using a dotted notation for the DQL schema. + +For example, deploying the following GraphQL Schema +```graphql +type Person { + id: ID + name: String! + friends: [Person] +} +``` + +will lead the the declaration of 3 predicates in the DQL Schema: + +- ``Person.id default`` +- ``Person.name string`` +- ``Person.friends [uid]`` + +and one DQL type +``` +type Person { + Person.name + Person.friends +} +``` + +Once again, the DQL type is just a declaration of the list of predicates that one can expect to be present in a node of having ``dgraph.type`` equal ``Person``. + +The default mapping can be customized by using the [@dgraph directive](/graphql/schema/directives/directive-dgraph/). + + +#### GraphQL ID type and Dgraph `uid` +Person.id is not part of the Person DQL type: internally Dgraph is using ``uid`` predicate as unique identifier for every node in the graph. Dgraph returns the value of ``uid`` when a GraphQL field of type ID is requested. + +#### @search directive and predicate indexes + +`@search` directive tells Dgraph what search to build into your GraphQL API. +```graphql +type Person { + name: String @search(by: [hash]) + ... +``` +Is simply translated into a prediate index specification in the Dgraph schema: +``` +Person.name: string @index(hash) . +``` + +#### Constraints +DQL does not have 'non nullable' constraint ``!`` nor 'unique' constraint. Constraints on the graph are handled by correctly using ``upsert`` operation in DQL. + +#### DQL queries +You can use DQL to query the data generated by the GraphQL API operations. +For example the GraphQL Query +```graphql +query { + queryPerson { + id + name + friends { + id + name + } + } +} +``` +can be executed in DQL +```graphql +{ + queryPerson(func: type(Person)) { + id: uid + name: Person.name + friends: Person.friends { + id: uid + name: Person.name + } + } +} +``` + +Note that in this query, we are using ``aliases`` such as ``name: Person.name`` to name the predicates in the JSON response,as they are declared in the GraphQL schema. + +#### GraphQL Interface +DQL does not have the concept of interfaces. + +Considering the following GraphQL schema : +```graphql +interface Location { + id: ID! + geoloc: Point +} + +type Property implements Location { + price: Float +} +``` +The predicates and types generated for a ``Property`` are: + + +```graphql +Location.geoloc: geo . +Location.name: string . +Property.price: float . +type Property { + Location.name + Location.geoloc + Property.price +} +``` + +### Consequences +The fact that the GraphQL API backend is a graph in Dgraph, implies that you can use Dgraph DQL on the data that is also served by the GraphQL API operations. + +In particular, you can +- use Dgraph DQL mutations but also Dgraph's [import tools](/migration/import-data) to populate the graph after you have deployed a GraphQL Schema. See [GraphQL data loading](/graphql/graphql-dql/graphql-data-loading) +- use DQL to query the graph in the context of authorization rules and custom resolvers. +- add knowledge to your graph such as meta-data, score, annotations, ..., but also relationships or relationships attributes (facets) that could be the result of similarity computation, threat detection a.s.o. The added data could be hidden from your GraphQL API clients but be available to logic written with DQL clients. +- break things using DQL: DQL is powerful and is bypassing constraints expressed in the GraphQL schema. You can for example delete a node predicate that is mandatory in the GraphQL API! Hopefully there are ways to secure who can read/write/delete predicates. ( see the [ACL](/admin/admin-tasks/user-management-access-control)) section. +- fix things using DQL: this is especially useful when doing GraphQL Schema updates which require some [data migrations](/graphql/graphql-dql/graphql-data-migration). + + + diff --git a/docusaurus-docs/graphql_versioned_docs/version-v25.4/graphql-dql/index.md b/docusaurus-docs/graphql_versioned_docs/version-v25.4/graphql-dql/index.md new file mode 100644 index 00000000..eb4086fb --- /dev/null +++ b/docusaurus-docs/graphql_versioned_docs/version-v25.4/graphql-dql/index.md @@ -0,0 +1,15 @@ +--- +title: "GraphQL - DQL interoperability" +description: "Dgraph Query Language (DQL) is Dgraph's proprietary language to add, modify, delete and fetch data." + +--- + +As aGraphQL developer, you can deploy a GraphQL Schema in Dgraph and immediately get a GraphQL API served on ``/graphql`` endpoint and a backend; you don't need to concern yourself with the powerful graph database running in the background. + +However, by leveraging the graph database and using Dgraph Query Language (DQL), the Dgraph’s proprietary language, you can address advanced use cases and overcome some limitations of the GraphQL specification. + +This section covers how to use DQL in the conjunction with GraphQL API, what are the best parctices and the points of attention. + +### In this section + + diff --git a/docusaurus-docs/graphql_versioned_docs/version-v25.4/index.md b/docusaurus-docs/graphql_versioned_docs/version-v25.4/index.md new file mode 100644 index 00000000..ac839d04 --- /dev/null +++ b/docusaurus-docs/graphql_versioned_docs/version-v25.4/index.md @@ -0,0 +1,17 @@ +--- +title: "GraphQL API" +description: "Generate a GraphQL API and a graph backend from a single GraphQL schema." + +--- + +Dgraph lets you generate a GraphQL API and a graph backend from a single [GraphQL schema](/graphql/schema/dgraph-schema), no resolvers or custom queries are needed. Dgraph automatically generates the GraphQL operations for [queries](/graphql/queries/) and [mutations](/graphql/mutations/) + +GraphQL developers can [get started](/graphql/quick-start/) in minutes, and need not concern themselves with the powerful graph database running in the background. + +Dgraph extends the [GraphQL specifications](https://spec.graphql.org/) with [directives](/graphql/schema/directives/) and allows you to customize the behavior of GraphQL operations using [custom resolvers](/graphql/custom/) or to write you own resolver logic with [Lambda resolvers](/graphql/lambda/lambda-overview). + +Dgraph also supports +- [GraphQL subscriptions](/graphql/subscriptions/) with the `@withSubscription` directive: a client application can execute a subscription query and receive real-time updates when the subscription query result is updated. +- [Apollo federation](/graphql/federation/) : you can create a gateway GraphQL service that includes the Dgraph GraphQL API and other GraphQL services. + +Refer to the following pages for more details: \ No newline at end of file diff --git a/docusaurus-docs/graphql_versioned_docs/version-v25.4/lambda/field.md b/docusaurus-docs/graphql_versioned_docs/version-v25.4/lambda/field.md new file mode 100644 index 00000000..9a974cd8 --- /dev/null +++ b/docusaurus-docs/graphql_versioned_docs/version-v25.4/lambda/field.md @@ -0,0 +1,202 @@ +--- +title: "Lambda Fields" +description: "Start with lambda resolvers by defining it in your GraphQL schema. Then define your JavaScript mutation function and add it as a resolver in your JS source code." + +--- + +### Schema + +To set up a lambda function, first you need to define it on your GraphQL schema by using the `@lambda` directive. + +For example, to define a lambda function for the `rank` and `bio` fields in `Author`: + +```graphql +type Author { + id: ID! + name: String! @search(by: [hash, trigram]) + dob: DateTime @search + reputation: Float @search + bio: String @lambda + rank: Int @lambda + isMe: Boolean @lambda +} +``` + +You can also define `@lambda` fields on interfaces, as follows: + +```graphql +interface Character { + id: ID! + name: String! @search(by: [exact]) + bio: String @lambda +} + +type Human implements Character { + totalCredits: Float +} + +type Droid implements Character { + primaryFunction: String +} +``` + +### Resolvers + +After the schema is ready, you can define your JavaScript mutation function and add it as a resolver in your JS source code. +To add the resolver you can use either the `addGraphQLResolvers` or `addMultiParentGraphQLResolvers` methods. + +:::note +A Lambda Field resolver can use a combination of `parents`, `parent`, `dql`, or `graphql` inside the function. +::: + +:::tip +This example uses `parent` for the resolver function. You can find additional resolver examples using `dql` in the [Lambda queries article](/graphql/lambda/query), and using `graphql` in the [Lambda mutations article](/graphql/lambda/mutation). +::: + +For example, to define JavaScript lambda functions for... +- `Author`, +- `Character`, +- `Human`, and +- `Droid` + +...and add them as resolvers, do the following: + +```javascript +const authorBio = ({parent: {name, dob}}) => `My name is ${name} and I was born on ${dob}.` +const characterBio = ({parent: {name}}) => `My name is ${name}.` +const humanBio = ({parent: {name, totalCredits}}) => `My name is ${name}. I have ${totalCredits} credits.` +const droidBio = ({parent: {name, primaryFunction}}) => `My name is ${name}. My primary function is ${primaryFunction}.` + +self.addGraphQLResolvers({ + "Author.bio": authorBio, + "Character.bio": characterBio, + "Human.bio": humanBio, + "Droid.bio": droidBio +}) +``` + +For example, you can add a resolver for `rank` using a `graphql` call, as follows: + +```javascript +async function rank({parents}) { + const idRepList = parents.map(function (parent) { + return {id: parent.id, rep: parent.reputation} + }); + const idRepMap = {}; + idRepList.sort((a, b) => a.rep > b.rep ? -1 : 1) + .forEach((a, i) => idRepMap[a.id] = i + 1) + return parents.map(p => idRepMap[p.id]) +} + +self.addMultiParentGraphQLResolvers({ + "Author.rank": rank +}) +``` + +The following example demonstrates using the client-provided JWT to return `true` if the custom claim +for `USER` from the JWT matches the `id` of the `Author`. + +```javascript +async function isMe({ parent, authHeader }) { + if (!authHeader) return false; + if (!authHeader.value) return false; + const headerValue = authHeader.value; + if (headerValue === "") return false; + const base64Url = headerValue.split(".")[1]; + const base = base64Url.replace(/-/g, "+").replace(/_/g, "/"); + const allClaims = JSON.parse(atob(base64)); + if (!allClaims["https://my.app.io/jwt/claims"]) return false; + const customClaims = allClaims["https://my.app.io/jwt/claims"]; + return customClaims.USER === parent.id; +} + +self.addGraphQLResolvers({ + "Author.isMe": isMe, +}); +``` + +### Example + +For example, if you execute the following GraphQL query: + +```graphql +query { + queryAuthor { + name + bio + rank + isMe + } +} +``` + +...you should see a response such as the following: + +```json +{ + "queryAuthor": [ + { + "name":"Ann Author", + "bio":"My name is Ann Author and I was born on 2000-01-01T00:00:00Z.", + "rank":3, + "isMe": false + } + ] +} +``` + +In the same way, if you execute the following GraphQL query on the `Character` interface: + +```graphql +query { + queryCharacter { + name + bio + } +} +``` + +...you should see a response such as the following: + +```json +{ + "queryCharacter": [ + { + "name":"Han", + "bio":"My name is Han." + }, + { + "name":"R2-D2", + "bio":"My name is R2-D2." + } + ] +} +``` + +:::note +The `Human` and `Droid` types will inherit the `bio` lambda field from the `Character` interface. +::: + +For example, if you execute a `queryHuman` query with a selection set containing `bio`, then the lambda function registered for `Human.bio` is executed, as follows: + +```graphql +query { + queryHuman { + name + bio + } +} +``` + +This query generates the following response: + +```json +{ + "queryHuman": [ + { + "name": "Han", + "bio": "My name is Han. I have 10 credits." + } + ] +} +``` diff --git a/docusaurus-docs/graphql_versioned_docs/version-v25.4/lambda/index.md b/docusaurus-docs/graphql_versioned_docs/version-v25.4/lambda/index.md new file mode 100644 index 00000000..480879ff --- /dev/null +++ b/docusaurus-docs/graphql_versioned_docs/version-v25.4/lambda/index.md @@ -0,0 +1,4 @@ +--- +title: "Lambda Resolvers" + +--- diff --git a/docusaurus-docs/graphql_versioned_docs/version-v25.4/lambda/lambda-overview.md b/docusaurus-docs/graphql_versioned_docs/version-v25.4/lambda/lambda-overview.md new file mode 100644 index 00000000..05f3fdcd --- /dev/null +++ b/docusaurus-docs/graphql_versioned_docs/version-v25.4/lambda/lambda-overview.md @@ -0,0 +1,300 @@ +--- +title: "Dgraph Lambda Overview" +description: "Lambda provides a way to write custom logic in JavaScript, integrate it with your GraphQL schema, and execute it using the GraphQL API in a few easy steps." + +--- + +Lambda provides a way to write your custom logic in JavaScript, integrate it with your GraphQL schema, and execute it using the GraphQL API in a few easy steps: + +1. Set up a Dgraph cluster with a working lambda server (not required for [Dgraph Cloud](https://dgraph.io/cloud) users) +2. Declare lambda queries, mutations, and fields in your GraphQL schema as needed +3. Define lambda resolvers for them in a JavaScript file + +This also simplifies the job of developers, as they can build a complex backend that is rich with business logic, without setting up multiple different services. Also, you can build your backend in JavaScript, which means you can build both your frontend and backend using the same language. + +Dgraph doesn't execute your custom logic itself. It makes external HTTP requests to a user-defined lambda server. [Dgraph Cloud](https://dgraph.io/cloud) will do all of this for you. + +:::tip +If you want to deploy your own lambda server, you can find the implementation of Dgraph Lambda in our [open-source repository](https://github.com/dgraph-io/dgraph-lambda). Please refer to the documentation on [setting up a lambda server](/installation/lambda-server) for more details. +::: + +:::note +If you're using [Dgraph Cloud](https://dgraph.io/cloud), the final compiled script file must be under 500Kb +::: + +## Declaring lambda in a GraphQL schema + +There are three places where you can use the `@lambda` directive and thus tell Dgraph where to apply custom JavaScript logic. + +- You can add lambda fields to your types and interfaces, as follows: + +```graphql +type MyType { + ... + customField: String @lambda +} +``` + +- You can add lambda queries to the Query type, as follows: + +```graphql +type Query { + myCustomQuery(...): QueryResultType @lambda +} +``` + +- You can add lambda mutations to the Mutation type, as follows: + +```graphql +type Mutation { + myCustomMutation(...): MutationResult @lambda +} +``` + +## Defining lambda resolvers in JavaScript + +A lambda resolver is a user-defined JavaScript function that performs custom actions over the GraphQL types, interfaces, queries, and mutations. There are two methods to register JavaScript resolvers: + +- `self.addGraphQLResolvers` +- `self.addMultiParentGraphQLResolvers` + +:::tip +Functions `self.addGraphQLResolvers` and `self.addMultiParentGraphQLResolvers` can be called multiple times in your resolver code. +::: + +### addGraphQLResolvers + +The `self.addGraphQLResolvers` method takes an object as an argument, which maps a resolver name to the resolver function that implements it. The resolver functions registered using `self.addGraphQLResolvers` receive `{ parent, args, graphql, dql }` as argument: + +- `parent`, the parent object for which to resolve the current lambda field registered using `addGraphQLResolver`. +The `parent` receives all immediate fields of that object, whether or not they were actually queried. +Available only for types and interfaces (`null` for queries and mutations) +- `args`, the set of arguments for lambda queries and mutations +- `graphql`, a function to execute auto-generated GraphQL API calls from the lambda server. The user's auth header is passed back to the `graphql` function, so this can be used securely +- `dql`, provides an API to execute DQL from the lambda server +- `authHeader`, provides the JWT key and value of the auth header passed from + the client + +The `addGraphQLResolvers` can be represented with the following TypeScript types: + +```TypeScript +type GraphQLResponse { + data?: Record + errors?: { message: string }[] +} + +type AuthHeader { + key: string + value: string +} + +type GraphQLEventWithParent = { + parent: Record | null + args: Record + graphql: (query: string, vars?: Record, authHeader?: AuthHeader) => Promise + dql: { + query: (dql: string, vars?: Record) => Promise + mutate: (dql: string) => Promise + } + authHeader: AuthHeader +} + +function addGraphQLResolvers(resolvers: { + [key: string]: (e: GraphQLEventWithParent) => any; +}): void +``` + +:::tip +`self.addGraphQLResolvers` is the default choice for registering resolvers when the result of the lambda for each parent is independent of other parents. +::: + +Each resolver function should return data in the exact format as the return type of GraphQL field, query, or mutation for which it is being registered. + +In the following example, the resolver function `myTypeResolver` registered for the `customField` field in `MyType` returns a string because the return type of that field in the GraphQL schema is `String`: + +```javascript +const myTypeResolver = ({parent: {customField}}) => `My value is ${customField}.` + +self.addGraphQLResolvers({ + "MyType.customField": myTypeResolver +}) +``` + +Another resolver example using a `graphql` call: + +```javascript +async function todoTitles({ graphql }) { + const results = await graphql('{ queryTodo { title } }') + return results.data.queryTodo.map(t => t.title) +} + +self.addGraphQLResolvers({ + "Query.todoTitles": todoTitles +}) +``` + +### addMultiParentGraphQLResolvers + +The `self.addMultiParentGraphQLResolvers` is useful in scenarios where you want to perform computations involving all the parents returned from Dgraph for a lambda field. This is useful in two scenarios: + +- When you want to perform a computation between parents +- When you want to execute a complex query, and want to optimize it by firing a single query for all the parents + +This method takes an object as an argument, which maps a resolver name to the resolver function that implements it. The resolver functions registered using this method receive `{ parents, args, graphql, dql }` as argument: + +- `parents`, a list of parent objects for which to resolve the current lambda field registered using `addMultiParentGraphQLResolvers`. Available only for types and interfaces (`null` for queries and mutations) +- `args`, the set of arguments for lambda queries and mutations (`null` for types and interfaces) +- `graphql`, a function to execute auto-generated GraphQL API calls from the lambda server +- `dql`, provides an API to execute DQL from the lambda server +- `authHeader`, provides the JWT key and value of the auth header passed from + the client + +The `addMultiParentGraphQLResolvers` can be represented with the following TypeScript types: + +```TypeScript +type GraphQLResponse { + data?: Record + errors?: { message: string }[] +} + +type AuthHeader { + key: string + value: string +} + +type GraphQLEventWithParents = { + parents: (Record)[] | null + args: Record + graphql: (query: string, vars?: Record, authHeader?: AuthHeader) => Promise + dql: { + query: (dql: string, vars?: Record) => Promise + mutate: (dql: string) => Promise + } + authHeader: AuthHeader +} + +function addMultiParentGraphQLResolvers(resolvers: { + [key: string]: (e: GraphQLEventWithParents) => any; +}): void +``` + +:::note +This method should not be used for lambda queries or lambda mutations. +::: + +Each resolver function should return data as a list of the return type of GraphQL field for which it is being registered. + +In the following example, the resolver function `rank()` registered for the `rank` field in `Author`, returns a list of integers because the return type of that field in the GraphQL schema is `Int`: + +```graphql +type Author { + id: ID! + name: String! @search(by: [hash, trigram]) + reputation: Float @search + rank: Int @lambda +} +``` + +```javascript +import { sortBy } from 'lodash'; + +/* +This function computes the rank of each author based on the reputation of the author relative to other authors. +*/ +async function rank({parents}) { + const idRepMap = {}; + sortBy(parents, 'reputation').forEach((parent, i) => idRepMap[parent.id] = parents.length - i) + return parents.map(p => idRepMap[p.id]) +} + +self.addMultiParentGraphQLResolvers({ + "Author.rank": rank +}) +``` + +:::note +Scripts containing import packages (such as the example above) require compilation using Webpack. +::: + +The following example resolver uses a `dql` call: + +```javascript +async function reallyComplexDql({parents, dql}) { + const ids = parents.map(p => p.id); + const someComplexResults = await dql.query(`really-complex-query-here with ${ids}`); + return parents.map(parent => someComplexResults[parent.id]) +} + +self.addMultiParentGraphQLResolvers({ + "MyType.reallyComplexProperty": reallyComplexDql +}) +``` + +The following resolver example uses a `graphql` call and manually overrides the `authHeader` provided by the client: + +```javascript +async function secretGraphQL({ parents, graphql }) { + const ids = parents.map((p) => p.id); + const secretResults = await graphql( + `query myQueryName ($ids: [ID!]) { + queryMyType(filter: { id: $ids }) { + id + controlledEdge { + myField + } + } + }`, + { ids }, + { + key: 'X-My-App-Auth' + value: 'eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJodHRwczovL215LmFwcC5pby9qd3QvY2xhaW1zIjp7IlVTRVIiOiJmb28ifSwiZXhwIjoxODAwMDAwMDAwLCJzdWIiOiJ0ZXN0IiwibmFtZSI6IkpvaG4gRG9lIDIiLCJpYXQiOjE1MTYyMzkwMjJ9.wI3857KzwjtZAtOjng6MnzKVhFSqS1vt1SjxUMZF4jc' + } + ); + return parents.map((parent) => { + const secretRes = secretResults.data.find(res => res.id === parent.id) + parent.secretProperty = null + if (secretRes) { + if (secretRes.controlledEdge) { + parent.secretProperty = secretRes.controlledEdge.myField + } + } + return parent + }); +} +self.addMultiParentGraphQLResolvers({ + "MyType.secretProperty": secretGraphQL, +}); +``` + +## Example + +For example, if you execute the following lambda query: + +```graphql +query { + queryMyType { + customField + } +} +``` + +...you should see a response such as the following: + +```json +{ + "queryMyType": [ + { + "customField":"My value is Lambda Example" + } + ] +} +``` + +## Learn more + +To learn more about the `@lambda` directive, see: + +* [Lambda fields](/graphql/lambda/field) +* [Lambda queries](/graphql/lambda/query) +* [Lambda mutations](/graphql/lambda/mutation) +* [Lambda server setup](/installation/lambda-server) diff --git a/docusaurus-docs/graphql_versioned_docs/version-v25.4/lambda/mutation.md b/docusaurus-docs/graphql_versioned_docs/version-v25.4/lambda/mutation.md new file mode 100644 index 00000000..15440e67 --- /dev/null +++ b/docusaurus-docs/graphql_versioned_docs/version-v25.4/lambda/mutation.md @@ -0,0 +1,99 @@ +--- +title: "Lambda Mutations" +description: "Ready to use lambdas for mutations? This documentation takes you through the schemas, resolvers, and examples." + +--- + +### Schema + +To set up a lambda mutation, first you need to define it on your GraphQL schema by using the `@lambda` directive. + +:::note +`add`, `update`, and `delete` are reserved prefixes and they can't be used to define Lambda mutations. +::: + +For example, to define a lambda mutation for `Author` that creates a new author with a default `reputation` of `3.0` given just the `name`: + +```graphql +type Author { + id: ID! + name: String! @search(by: [hash, trigram]) + dob: DateTime + reputation: Float +} + +type Mutation { + newAuthor(name: String!): ID! @lambda +} +``` + +### Resolver + +Once the schema is ready, you can define your JavaScript mutation function and add it as resolver in your JS source code. +To add the resolver you can use either the `addGraphQLResolvers` or `addMultiParentGraphQLResolvers` methods. + +:::note +A Lambda Mutation resolver can use a combination of `parents`, `args`, `dql`, or `graphql` inside the function. +::: + +:::tip +This example uses `graphql` for the resolver function. You can find additional resolver examples using `dql` in the [Lambda queries article](/graphql/lambda/query), and using `parent` in the [Lambda fields article](/graphql/lambda/field). +::: + +For example, to define the JavaScript `newAuthor()` lambda function and add it as resolver: + +```javascript +async function newAuthor({args, graphql}) { + // lets give every new author a reputation of 3 by default + const results = await graphql(`mutation ($name: String!) { + addAuthor(input: [{name: $name, reputation: 3.0 }]) { + author { + id + reputation + } + } + }`, {"name": args.name}) + return results.data.addAuthor.author[0].id +} + +self.addGraphQLResolvers({ + "Mutation.newAuthor": newAuthor +}) +``` + +Alternatively, you can use `dql.mutate` to achieve the same results: + +```javascript +async function newAuthor({args, dql, graphql}) { + // lets give every new author a reputation of 3 by default + const res = await dql.mutate(`{ + set { + _:newAuth "${args.name}" . + _:newAuth "3.0" . + _:newAuth "Author" . + } + }`); + return res.data.uids.newAuth +} +``` + +### Example + +Finally, if you execute this lambda mutation a new author `Ken Addams` with `reputation=3.0` should be added to the database: + +```graphql +mutation { + newAuthor(name: "Ken Addams") +} +``` + +Afterwards, if you query the GraphQL database for `Ken Addams`, you would see: + +```json +{ + "getAuthor": { + "name":"Ken Addams", + "reputation":3.0 + } +} +``` diff --git a/docusaurus-docs/graphql_versioned_docs/version-v25.4/lambda/query.md b/docusaurus-docs/graphql_versioned_docs/version-v25.4/lambda/query.md new file mode 100644 index 00000000..1f22f87d --- /dev/null +++ b/docusaurus-docs/graphql_versioned_docs/version-v25.4/lambda/query.md @@ -0,0 +1,88 @@ +--- +title: "Lambda Queries" +description: "Get started with the @lambda directive for queries. This documentation takes you through the schemas, resolvers, and examples." + +--- + +### Schema + +To set up a lambda query, first you need to define it on your GraphQL schema by using the `@lambda` directive. + +:::note +`get`, `query`, and `aggregate` are reserved prefixes and they can't be used to define Lambda queries. +::: + +For example, to define a lambda query for `Author` that finds out authors given an author's `name`: + +```graphql +type Author { + id: ID! + name: String! @search(by: [hash, trigram]) + dob: DateTime + reputation: Float +} + +type Query { + authorsByName(name: String!): [Author] @lambda +} +``` + +### Resolver + +Once the schema is ready, you can define your JavaScript query function and add it as resolver in your JS source code. +To add the resolver you can use either the `addGraphQLResolvers` or `addMultiParentGraphQLResolvers` methods. + +:::note +A Lambda Query resolver can use a combination of `parents`, `args`, `dql`, or `graphql` inside the function. +::: + +:::tip +This example uses `dql` for the resolver function. You can find additional resolver examples using `parent` in the [Lambda fields article](/graphql/lambda/field), and using `graphql` in the [Lambda mutations article](/graphql/lambda/mutation). +::: + +For example, to define the JavaScript `authorsByName()` lambda function and add it as resolver: + +```javascript +async function authorsByName({args, dql}) { + const results = await dql.query(`query queryAuthor($name: string) { + queryAuthor(func: type(Author)) @filter(eq(Author.name, $name)) { + name: Author.name + dob: Author.dob + reputation: Author.reputation + } + }`, {"$name": args.name}) + return results.data.queryAuthor +} + +self.addGraphQLResolvers({ + "Query.authorsByName": authorsByName, +}) +``` + +### Example + +Finally, if you execute this lambda query + +```graphql +query { + authorsByName(name: "Ann Author") { + name + dob + reputation + } +} +``` + +You should see a response such as + +```json +{ + "authorsByName": [ + { + "name":"Ann Author", + "dob":"2000-01-01T00:00:00Z", + "reputation":6.6 + } + ] +} +``` diff --git a/docusaurus-docs/graphql_versioned_docs/version-v25.4/lambda/webhook.md b/docusaurus-docs/graphql_versioned_docs/version-v25.4/lambda/webhook.md new file mode 100644 index 00000000..78cd09f4 --- /dev/null +++ b/docusaurus-docs/graphql_versioned_docs/version-v25.4/lambda/webhook.md @@ -0,0 +1,101 @@ +--- +title: "Lambda Webhooks" +description: "Ready to use lambdas for webhooks? This documentation takes you through the schemas, resolvers, and examples." + +--- + +### Schema + +To set up a lambda webhook, you need to define it in your GraphQL schema by using the `@lambdaOnMutate` directive along with the mutation events (`add`/`update`/`delete`) you want to listen on. + +:::note +Lambda webhooks only listen for events from the root mutation. You can create a schema that is capable of creating deeply nested objects, but only the parent level webhooks will be evoked for the mutation. +::: + +For example, to define a lambda webhook for all mutation events (`add`/`update`/`delete`) on any `Author` object: + +```graphql +type Author @lambdaOnMutate(add: true, update: true, delete: true) { + id: ID! + name: String! @search(by: [hash, trigram]) + dob: DateTime + reputation: Float +} +``` + +### Resolver + +Once the schema is ready, you can define your JavaScript functions and add those as resolvers in your JS source code. +To add the resolvers you should use the `addWebHookResolvers`method. + +:::note +A Lambda Webhook resolver can use a combination of `event`, `dql`, `graphql` or `authHeader` inside the function. +::: + +#### Event object + +You also have access to the `event` object within the resolver. Depending on the value of `operation` field, only one of the fields (`add`/`update`/`delete`) will be part of the `event` object. The definition of `event` is as follows: + +``` +"event": { + "__typename": "", + "operation": "", + "commitTs": + "add": { + "rootUIDs": [], + "input": [] + }, + "update": { + "rootUIDs": [], + "setPatch": , + "removePatch": + }, + "delete": { + "rootUIDs": [] + } +``` + +#### Resolver examples + +For example, to define JavaScript lambda functions for each mutation event for which `@lambdaOnMutate` is enabled and add those as resolvers: + +```javascript +async function addAuthorWebhook({event, dql, graphql, authHeader}) { + // execute what you want on addition of an author + // maybe send a welcome mail to the author + +} + +async function updateAuthorWebhook({event, dql, graphql, authHeader}) { + // execute what you want on update of an author + // maybe send a mail to the author informing that few details have been updated + +} + +async function deleteAuthorWebhook({event, dql, graphql, authHeader}) { + // execute what you want on deletion of an author + // maybe mail the author saying they have been removed from the platform + +} + +self.addWebHookResolvers({ + "Author.add": addAuthorWebhook, + "Author.update": updateAuthorWebhook, + "Author.delete": deleteAuthorWebhook, +}) +``` + +### Example + +Finally, if you execute an `addAuthor` mutation, the `add` operation mapped to the `addAuthorWebhook` resolver will be triggered: + +```graphql +mutation { + addAuthor(input:[{name: "Ken Addams"}]) { + author { + id + name + } + } +} +``` diff --git a/docusaurus-docs/graphql_versioned_docs/version-v25.4/mutations/add.md b/docusaurus-docs/graphql_versioned_docs/version-v25.4/mutations/add.md new file mode 100644 index 00000000..061bff60 --- /dev/null +++ b/docusaurus-docs/graphql_versioned_docs/version-v25.4/mutations/add.md @@ -0,0 +1,84 @@ +--- +title: "Add Mutations" +description: "Add mutations allows you to add new objects of a particular type. Dgraph automatically generates input and return types in the schema for the add mutation" + +--- + +Add mutations allow you to add new objects of a particular type. + +We use the following schema to demonstrate some examples. + +**Schema**: +```graphql +type Author { + id: ID! + name: String! @search(by: [hash]) + dob: DateTime + posts: [Post] +} + +type Post { + postID: ID! + title: String! @search(by: [term, fulltext]) + text: String @search(by: [fulltext, term]) + datePublished: DateTime +} +``` + +Dgraph automatically generates input and return types in the schema for the `add` mutation, +as shown below: +```graphql +addPost(input: [AddPostInput!]!): AddPostPayload + +input AddPostInput { + title: String! + text: String + datePublished: DateTime +} + +type AddPostPayload { + post(filter: PostFilter, order: PostOrder, first: Int, offset: Int): [Post] + numUids: Int +} +``` + +**Example**: Add mutation on single type with embedded value +```graphql +mutation { + addAuthor(input: [{ name: "A.N. Author", posts: []}]) { + author { + id + name + } + } +} +``` + +**Example**: Add mutation on single type using variables +```graphql +mutation addAuthor($author: [AddAuthorInput!]!) { + addAuthor(input: $author) { + author { + id + name + } + } +} +``` +Variables: +```json +{ "author": + { "name": "A.N. Author", + "dob": "2000-01-01", + "posts": [] + } +} +``` + +:::note +You can convert an `add` mutation to an `upsert` mutation by setting the value of the input variable `upsert` to `true`. For more information, see [Upsert Mutations](/graphql/mutations/upsert). +::: + +## Examples + +You can refer to the following [link](https://github.com/dgraph-io/dgraph/blob/main/graphql/resolve/add_mutation_test.yaml) for more examples. diff --git a/docusaurus-docs/graphql_versioned_docs/version-v25.4/mutations/deep.md b/docusaurus-docs/graphql_versioned_docs/version-v25.4/mutations/deep.md new file mode 100644 index 00000000..a1106634 --- /dev/null +++ b/docusaurus-docs/graphql_versioned_docs/version-v25.4/mutations/deep.md @@ -0,0 +1,98 @@ +--- +title: "Deep Mutations" +description: "You can perform deep mutations at multiple levels. Deep mutations do not alter linked objects, but they can add deeply-nested new objects or link to existing objects." + +--- + +You can perform deep mutations at multiple levels. Deep mutations do not alter linked objects, but they can add deeply-nested new objects or link to existing objects. To update an existing nested object, use the update mutation for its type. + +We use the following schema to demonstrate some examples. + +## **Schema**: +```graphql +type Author { + id: ID! + name: String! @search(by: [hash]) + dob: DateTime + posts: [Post] +} + +type Post { + postID: ID! + title: String! @search(by: [term, fulltext]) + text: String @search(by: [fulltext, term]) + datePublished: DateTime +} +``` + +### **Example**: Adding deeply nested post with new author mutation using variables +```graphql +mutation addAuthorWithPost($author: addAuthorInput!) { + addAuthor(input: [$author]) { + author { + id + name + posts { + title + text + } + } + } +} +``` + +Variables: + +```json +{ "author": + { "name": "A.N. Author", + "dob": "2000-01-01", + "posts": [ + { + "title": "New post", + "text": "A really new post" + } + ] + } +} +``` + +### **Example**: Update mutation on deeply nested post and link to an existing author using variables + +The following example assumes that the post with the postID of `0x456` already exists, and is not currently nested under the author having the id of `0x123`. + +:::note +This syntax does not remove any other existing posts, it just adds the existing post to any that may already be nested. +::: + +```graphql +mutation updateAuthorWithExistingPost($patch: UpdateAuthorInput!) { + updateAuthor(input: $patch) { + author { + id + posts { + title + text + } + } + } +} +``` +Variables: +```json +{ "patch": + { "filter": { + "id": ["0x123"] + }, + "set": { + "posts": [ + { + "postID": "0x456" + } + ] + } + } +} +``` + +The example query above can't modify the existing post's title or text. To modify the post's title or text, use the `updatePost` mutation either alongside the mutation above, or as a separate transaction. diff --git a/docusaurus-docs/graphql_versioned_docs/version-v25.4/mutations/delete.md b/docusaurus-docs/graphql_versioned_docs/version-v25.4/mutations/delete.md new file mode 100644 index 00000000..de9ddb85 --- /dev/null +++ b/docusaurus-docs/graphql_versioned_docs/version-v25.4/mutations/delete.md @@ -0,0 +1,60 @@ +--- +title: "Delete Mutations" + +--- + +Delete Mutations allow you to delete objects of a particular type. + +We use the following schema to demonstrate some examples. + +**Schema**: +```graphql +type Author { + id: ID! + name: String! @search(by: [hash]) + dob: DateTime + posts: [Post] +} + +type Post { + postID: ID! + title: String! @search(by: [term, fulltext]) + text: String @search(by: [fulltext, term]) + datePublished: DateTime +} +``` + +Dgraph automatically generates input and return types in the schema for the `delete` mutation. +Delete mutations take `filter` as an input to select specific objects and returns the state of the objects before deletion. +```graphql +deleteAuthor(filter: AuthorFilter!): DeleteAuthorPayload + +type DeleteAuthorPayload { + author(filter: AuthorFilter, order: AuthorOrder, first: Int, offset: Int): [Author] + msg: String + numUids: Int +} +``` + +**Example**: Delete mutation using variables +```graphql +mutation deleteAuthor($filter: AuthorFilter!) { + deleteAuthor(filter: $filter) { + msg + author { + name + dob + } + } +} +``` +Variables: +```json +{ "filter": + { "name": { "eq": "A.N. Author" } } +} +``` + +## Examples + +You can refer to the following [link](https://github.com/dgraph-io/dgraph/blob/main/graphql/resolve/delete_mutation_test.yaml) for more examples. diff --git a/docusaurus-docs/graphql_versioned_docs/version-v25.4/mutations/index.md b/docusaurus-docs/graphql_versioned_docs/version-v25.4/mutations/index.md new file mode 100644 index 00000000..ffbad157 --- /dev/null +++ b/docusaurus-docs/graphql_versioned_docs/version-v25.4/mutations/index.md @@ -0,0 +1,4 @@ +--- +title: "Mutations" + +--- diff --git a/docusaurus-docs/graphql_versioned_docs/version-v25.4/mutations/mutations-overview.md b/docusaurus-docs/graphql_versioned_docs/version-v25.4/mutations/mutations-overview.md new file mode 100644 index 00000000..e355d8fb --- /dev/null +++ b/docusaurus-docs/graphql_versioned_docs/version-v25.4/mutations/mutations-overview.md @@ -0,0 +1,251 @@ +--- +title: "Mutations Overview" +description: "Mutations can be used to insert, update, or delete data. Dgraph automatically generates GraphQL mutation for each type that you define in your schema." + +--- + +Mutations allow you to modify server-side data, and it also returns an object based on the operation performed. It can be used to insert, update, or delete data. Dgraph automatically generates GraphQL mutations for each type that you define in your schema. The mutation field returns an object type that allows you to query for nested fields. This can be useful for fetching an object's new state after an add/update, or to get the old state of an object before a delete. + +**Example** + +```graphql +type Author { + id: ID! + name: String! @search(by: [hash]) + dob: DateTime + posts: [Post] +} + +type Post { + postID: ID! + title: String! @search(by: [term, fulltext]) + text: String @search(by: [fulltext, term]) + datePublished: DateTime +} +``` + +The following mutations would be generated from the above schema. + +```graphql +type Mutation { + addAuthor(input: [AddAuthorInput!]!): AddAuthorPayload + updateAuthor(input: UpdateAuthorInput!): UpdateAuthorPayload + deleteAuthor(filter: AuthorFilter!): DeleteAuthorPayload + addPost(input: [AddPostInput!]!): AddPostPayload + updatePost(input: UpdatePostInput!): UpdatePostPayload + deletePost(filter: PostFilter!): DeletePostPayload +} + +type AddAuthorPayload { + author(filter: AuthorFilter, order: AuthorOrder, first: Int, offset: Int): [Author] + numUids: Int +} + +type AddPostPayload { + post(filter: PostFilter, order: PostOrder, first: Int, offset: Int): [Post] + numUids: Int +} + +type DeleteAuthorPayload { + author(filter: AuthorFilter, order: AuthorOrder, first: Int, offset: Int): [Author] + msg: String + numUids: Int +} + +type DeletePostPayload { + post(filter: PostFilter, order: PostOrder, first: Int, offset: Int): [Post] + msg: String + numUids: Int +} + +type UpdateAuthorPayload { + author(filter: AuthorFilter, order: AuthorOrder, first: Int, offset: Int): [Author] + numUids: Int +} + +type UpdatePostPayload { + post(filter: PostFilter, order: PostOrder, first: Int, offset: Int): [Post] + numUids: Int +} +``` + +## Input objects +Mutations require input data, such as the data, to create a new object or an object's ID to delete. Dgraph auto-generates the input object type for every type in the schema. + +```graphql +input AddAuthorInput { + name: String! + dob: DateTime + posts: [PostRef] +} + +mutation { + addAuthor( + input: { + name: "A.N. Author", + lastName: "2000-01-01", + } + ) + { + ... + } +} +``` + +## Return fields +Each mutation provides a set of fields that can be returned in the response. Dgraph auto-generates the return payload object type for every type in the schema. + +```graphql +type AddAuthorPayload { + author(filter: AuthorFilter, order: AuthorOrder, first: Int, offset: Int): [Author] + numUids: Int +} +``` + +## Multiple fields in mutations +A mutation can contain multiple fields, just like a query. While query fields are executed in parallel, mutation fields run in series, one after the other. This means that if we send two `updateAuthor` mutations in one request, the first is guaranteed to finish before the second begins. This ensures that we don't end up with a race condition with ourselves. If one of the mutations is aborted due error like transaction conflict, we continue performing the next mutations. + +**Example**: Mutation on multiple types +```graphql +mutation ($post: AddPostInput!, $author: AddAuthorInput!) { + addAuthor(input: [$author]) { + author { + name + } + } + addPost(input: [$post]) { + post { + postID + title + text + } + } +} +``` + +Variables: + +```json +{ + "author": { + "name": "A.N. Author", + "dob": "2000-01-01", + "posts": [] + }, + "post": { + "title": "Exciting post", + "text": "A really good post", + "author": { + "name": "A.N. Author" + } + } +} +``` + +## Union mutations + +Mutations can be used to add a node to a `union` field in a type. + +For the following schema, + +```graphql +enum Category { + Fish + Amphibian + Reptile + Bird + Mammal + InVertebrate +} + +interface Animal { + id: ID! + category: Category @search +} + +type Dog implements Animal { + breed: String @search +} + +type Parrot implements Animal { + repeatsWords: [String] +} + +type Human { + name: String! + pets: [Animal!]! +} + +union HomeMember = Dog | Parrot | Human + +type Home { + id: ID! + address: String + members: [HomeMember] +} +``` + +This is the mutation for adding `members` to the `Home` type: + +```graphql +mutation { + addHome(input: [ + { + "address": "United Street", + "members": [ + { "dogRef": { "category": Mammal, "breed": "German Shepherd"} }, + { "parrotRef": { "category": Bird, "repeatsWords": ["squawk"]} }, + { "humanRef": { "name": "Han Solo"} } + ] + } + ]) { + home { + address + members { + ... on Dog { + breed + } + ... on Parrot { + repeatsWords + } + ... on Human { + name + } + } + } + } +} +``` + +## Vector Embedding mutations + +For types with vector embeddings Dgraph automatically generates the add mutation. For this example of add mutation we use the following schema. + +```graphql +type User { + userID: ID! + name: String! + name_v: [Float!] @embedding @search(by: ["hnsw(metric: euclidean, exponent: 4)"]) +} + +mutation { +addUser(input: [ +{ name: "iCreate with a Mini iPad", name_v: [0.12, 0.53, 0.9, 0.11, 0.32] }, +{ name: "Resistive Touchscreen", name_v: [0.72, 0.89, 0.54, 0.15, 0.26] }, +{ name: "Fitness Band", name_v: [0.56, 0.91, 0.93, 0.71, 0.24] }, +{ name: "Smart Ring", name_v: [0.38, 0.62, 0.99, 0.44, 0.25] }]) + { + project { + id + name + name_v + } + } +} +``` + +Note: The embeddings are generated outside of Dgraph using any suitable machine learning model. + +## Examples + +You can refer to the following [link](https://github.com/dgraph-io/dgraph/tree/main/graphql/schema/testdata/schemagen) for more examples. diff --git a/docusaurus-docs/graphql_versioned_docs/version-v25.4/mutations/update.md b/docusaurus-docs/graphql_versioned_docs/version-v25.4/mutations/update.md new file mode 100644 index 00000000..7023a658 --- /dev/null +++ b/docusaurus-docs/graphql_versioned_docs/version-v25.4/mutations/update.md @@ -0,0 +1,107 @@ +--- +title: "Update Mutations" +description: "Update mutations let you to update existing objects of a particular type. With update mutations, you can filter nodes and set and remove any field belonging to a type." + +--- + +Update mutations let you update existing objects of a particular type. With update mutations, you can filter nodes and set or remove any field belonging to a type. + +We use the following schema to demonstrate some examples. + +**Schema**: +```graphql +type Author { + id: ID! + name: String! @search(by: [hash]) + dob: DateTime + posts: [Post] +} + +type Post { + postID: ID! + title: String! @search(by: [term, fulltext]) + text: String @search(by: [fulltext, term]) + datePublished: DateTime +} +``` + +Dgraph automatically generates input and return types in the schema for the `update` mutation. Update mutations take `filter` as an input to select specific objects. You can specify `set` and `remove` operations on fields belonging to the filtered objects. It returns the state of the objects after updating. + +:::note +Executing an empty `remove {}` or an empty `set{}` doesn't have any effect on the update mutation. +::: + +```graphql +updatePost(input: UpdatePostInput!): UpdatePostPayload + +input UpdatePostInput { + filter: PostFilter! + set: PostPatch + remove: PostPatch +} + +type UpdatePostPayload { + post(filter: PostFilter, order: PostOrder, first: Int, offset: Int): [Post] + numUids: Int +} +``` + +### Set + +For example, an update `set` mutation using variables: + +```graphql +mutation updatePost($patch: UpdatePostInput!) { + updatePost(input: $patch) { + post { + postID + title + text + } + } +} +``` +Variables: +```json +{ "patch": + { "filter": { + "postID": ["0x123", "0x124"] + }, + "set": { + "text": "updated text" + } + } +} +``` + +### Remove + +For example an update `remove` mutation using variables: + +```graphql +mutation updatePost($patch: UpdatePostInput!) { + updatePost(input: $patch) { + post { + postID + title + text + } + } +} +``` +Variables: +```json +{ "patch": + { "filter": { + "postID": ["0x123", "0x124"] + }, + "remove": { + "text": "delete this text" + } + } +} +``` + +### Examples + +You can refer to the following [link](https://github.com/dgraph-io/dgraph/blob/main/graphql/resolve/update_mutation_test.yaml) for more examples. diff --git a/docusaurus-docs/graphql_versioned_docs/version-v25.4/mutations/upsert.md b/docusaurus-docs/graphql_versioned_docs/version-v25.4/mutations/upsert.md new file mode 100644 index 00000000..41183f50 --- /dev/null +++ b/docusaurus-docs/graphql_versioned_docs/version-v25.4/mutations/upsert.md @@ -0,0 +1,106 @@ +--- +title: "Upsert Mutations" +description: "Upsert mutations allow you to perform `add` or `update` operations based on whether a particular ID exists in the database" + +--- + +Upsert mutations allow you to perform `add` or `update` operations based on whether a particular `ID` exists in the database. The IDs must be external IDs, defined using the `@id` directive in the schema. + +For example, to demonstrate how upserts work in GraphQL, take the following schema: + +**Schema** +```graphql +type Author { + id: String! @id + name: String! @search(by: [hash]) + posts: [Post] @hasInverse(field: author) +} + +type Post { + postID: String! @id + title: String! @search(by: [term, fulltext]) + text: String @search(by: [fulltext, term]) + author: Author! +} +``` + +Dgraph automatically generates input and return types in the schema for the `add` mutation, as shown below: + +```graphql +addPost(input: [AddPostInput!]!, upsert: Boolean): AddPostPayload + +input AddPostInput { + postID: String! + title: String! + text: String + author: AuthorRef! +} +``` + +Suppose you want to update the `text` field of a post with the ID `mm2`. But you also want to create a new post with that ID in case it doesn't already exist. To do this, you use the `addPost` mutation, but with an additional input variable `upsert`. + +This is a `Boolean` variable. Setting it to `true` will result in an upsert operation. + +It will perform an `update` mutation and carry out the changes you specify in your request if the particular ID exists. Otherwise, it will fall back to a default `add` operation and create a new `Post` with that ID and the details you provide. + +Setting `upsert` to `false` is the same as using a plain `add` operation—it'll either fail or succeed, depending on whether the ID exists or not. + +**Example**: Add mutation with `upsert` true: + +```graphql +mutation($post: [AddPostInput!]!) { + addPost(input: $post, upsert: true) { + post { + postID + title + text + author { + id + } + } + } +} +``` + +With variables: + +```json +{ + "post": + { + "postID": "mm2", + "title": "Second Post", + "text": "This is my second post, and updated with some new information.", + "author": { + "id": "micky" + } + } +} +``` + +If a post with the ID `mm2` exists, it will update the post with the new details. Otherwise, it'll create a new `Post` with that ID and the values you provided. In either case, you'll get the following response back: + +```graphql +"data": { + "addPost": { + "post": [ + { + "postID": "mm2", + "title": "Second Post", + "text": "This is my second post, and updated with some new information.", + "author": { + "id": "micky" + } + } + ] + } + } +``` + +:::note +* The default value of `upsert` will be `false`, for backward compatibility. +* The current behavior of `Add` and `Update` mutations is such that they do not update deep level nodes. So Add mutations with `upsert` set to `true` will only update values at the root level. +::: + +## Examples +You can refer to the following [link](https://github.com/dgraph-io/dgraph/blob/main/graphql/resolve/add_mutation_test.yaml) for more examples. diff --git a/docusaurus-docs/graphql_versioned_docs/version-v25.4/queries/aggregate.md b/docusaurus-docs/graphql_versioned_docs/version-v25.4/queries/aggregate.md new file mode 100644 index 00000000..35f42223 --- /dev/null +++ b/docusaurus-docs/graphql_versioned_docs/version-v25.4/queries/aggregate.md @@ -0,0 +1,209 @@ +--- +title: "Aggregate Queries" +description: "Dgraph automatically generates aggregate queries for GraphQL schemas. These are compatible with the @auth directive." + +--- + +Dgraph automatically generates aggregate queries for GraphQL schemas. +Aggregate queries fetch aggregate data, including the following: + +* *Count queries* that let you count fields +satisfying certain criteria specified using a filter. +* *Advanced aggregate queries* that let you calculate the maximum, minimum, sum +and average of specified fields. + +Aggregate queries are compatible with the `@auth` directive and follow the same +authorization rules as the `query` keyword. You can also use filters with +aggregate queries, as shown in some of the examples provided below. + +## Count queries at root + +For every `type` defined in a GraphQL schema, Dgraph generates an aggregate query +`aggregate`. This query includes a `count` field, as well as +[advanced aggregate query fields](#advanced-aggregate-queries-at-root). + +### Examples + +Example: Fetch the total number of `posts`. + +```graphql + query { + aggregatePost { + count + } + } +``` + +Example: Fetch the number of `posts` whose titles contain `GraphQL`. + +```graphql + query { + aggregatePost(filter: { + title: { + anyofterms: "GraphQL" + } + }) { + count + } + } +``` + + +## Count queries for child nodes + +Dgraph also defines `Aggregate` fields for every field which +is of type `List[Type/Interface]` inside `query` queries, allowing +you to do a `count` on fields, or to use the [advanced aggregate queries](#advanced-aggregate-queries-for-child-nodes). + +### Examples + +Example: Fetch the number of `posts` for all authors along with their `name`. + +```graphql + query { + queryAuthor { + name + postsAggregate { + count + } + } + } +``` + +Example: Fetch the number of `posts` with a `score` greater than `10` for all +authors, along with their `name` + +```graphql + query { + queryAuthor { + name + postsAggregate(filter: { + score: { + gt: 10 + } + }) { + count + } + } + } +``` + +## Advanced aggregate queries at root + +For every `type` defined in the GraphQL schema, Dgraph generates an aggregate +query `aggregate` that includes advanced aggregate query +fields, and also includes a `count` field (see [Count queries at root](#count-queries-at-root)). Dgraph generates one or more advanced aggregate +query fields (`Min`, `Max`, `Sum` and +`Avg`) for fields in the schema that are typed as `Int`, `Float`, +`String` and `Datetime`. + +:::note +Advanced aggregate query fields are generated according to a field's type. +Fields typed as `Int` and `Float` get the following query fields: +`Max`, `Min`, `Sum` and `Avg`. +Fields typed as `String` and `Datetime` only get the `Max`, + `Min` query fields. +::: + +### Examples + +Example: Fetch the average number of `posts` written by authors: + +```graphql + query { + aggregateAuthor { + numPostsAvg + } + } +``` + +Example: Fetch the total number of `posts` by all authors, and the maximum +number of `posts` by any single `Author`: + +```graphql + query { + aggregateAuthor { + numPostsSum + numPostsMax + } + } +``` + +Example: Fetch the average number of `posts` for authors with more than 20 +`friends`: + +```graphql + query { + aggregateAuthor (filter: { + friends: { + gt: 20 + } + }) { + numPostsAvg + } + } +``` + + +## Advanced aggregate queries for child nodes + +Dgraph also defines aggregate `Aggregate` fields for child nodes +within `query` queries. This is done for each field that is of type +`List[Type/Interface]` inside `query` queries, letting you fetch +minimums, maximums, averages and sums for those fields. + +:::note +Aggregate query fields are generated according to a field's type. Fields typed +as `Int` and `Float` get the following query fields:`Max`, +`Min`, `Sum` and `Avg`. Fields typed as +`String` and `Datetime` only get the `Max`, `Min` query +fields. +::: + +### Examples + +Example: Fetch the minimum, maximum and average `score` of the `posts` for each +`Author`, along with each author's `name`. + +```graphql + query { + queryAuthor { + name + postsAggregate { + scoreMin + scoreMax + scoreAvg + } + } + } +``` + +Example: Fetch the date of the most recent post with a `score` greater than +`10` for all authors, along with the author's `name`. + +```graphql + query { + queryAuthor { + name + postsAggregate(filter: { + score: { + gt: 10 + } + }) { + datePublishedMax + } + } + } +``` + +## Aggregate queries on null data + +Aggregate queries against empty data return `null`. This is true for both the +`Aggregate` fields and `aggregate` queries generated by +Dgraph. + +So, in the examples above, the following is true: +* If there are no nodes of type `Author`, the `aggregateAuthor` query will + return null. +* If an `Author` has not written any posts, the field `postsAggregate` will be + null for that `Author`. diff --git a/docusaurus-docs/graphql_versioned_docs/version-v25.4/queries/and-or-not.md b/docusaurus-docs/graphql_versioned_docs/version-v25.4/queries/and-or-not.md new file mode 100644 index 00000000..545dd42f --- /dev/null +++ b/docusaurus-docs/graphql_versioned_docs/version-v25.4/queries/and-or-not.md @@ -0,0 +1,92 @@ +--- +title: "And, Or and Not Operators in GraphQL" +description: "Every GraphQL search filter can use AND, OR and NOT operators." + +--- + +Every GraphQL search filter can use `and`, `or`, and `not` operators. + +GraphQL syntax uses infix notation, so: "a and b" is `a, and: { b }`, "a or b or c" is `a, or: { b, or: c }`, and "not" is a prefix (`not:`). + +The following example queries demonstrate the use of `and`, `or`, and `not` operators: + +Example: _"Posts that do not have "GraphQL" in the title"_ + +```graphql +queryPost(filter: { not: { title: { allofterms: "GraphQL"} } } ) { ... } +``` + +Example: _"Posts that have "GraphQL" or "Dgraph" in the title"_ + +```graphql +queryPost(filter: { + title: { allofterms: "GraphQL"}, + or: { title: { allofterms: "Dgraph" } } +} ) { ... } +``` + +Example: _"Posts that have "GraphQL" and "Dgraph" in the title"_ + +```graphql +queryPost(filter: { + title: { allofterms: "GraphQL"}, + and: { title: { allofterms: "Dgraph" } } +} ) { ... } +``` + +The `and` operator is implicit for a single filter object, if the fields don't overlap. For example, above the `and` is required because `title` is in both filters; whereas below, `and` is not required. + +```graphql +queryPost(filter: { + title: { allofterms: "GraphQL" }, + datePublished: { ge: "2020-06-15" } +} ) { ... } +``` + +Example: _"Posts that have "GraphQL" in the title, or have the tag "GraphQL" and mention "Dgraph" in the title"_ + +```graphql +queryPost(filter: { + title: { allofterms: "GraphQL"}, + or: { title: { allofterms: "Dgraph" }, tags: { eq: "GraphQL" } } +} ) { ... } +``` + +The `and` and `or` filter both accept a list of filters. Per the GraphQL specification, non-list filters are coerced into a list. This provides backwards-compatibility while allowing for more complex filters. + +Example: _"Query for posts that have `GraphQL` in the title but that lack the `GraphQL` tag, or that have `Dgraph` in the title but lack the `Dgraph` tag"_ + +```graphql +queryPost(filter: { + or: [ + { and: [{ title: { allofterms: "GraphQL" } }, { not: { tags: { eq: "GraphQL" } } }] } + { and: [{ title: { allofterms: "Dgraph" } }, { not: { tags: { eq: "Dgraph" } } }] } + ] +} ) { ... } +``` + +### Nesting + +Nested logic with the same `and`/`or` conjunction can be simplified into a single list. + +For example, the following complex query: + +``` +queryPost(filter: { + or: [ + { or: [ { foo: { eq: "A" } }, { bar: { eq: "B" } } ] }, + { or: [ { baz: { eq: "C" } }, { quz: { eq: "D" } } ] } + ] +} ) { ... } +``` +...can be simplified into the following simplified query syntax: +``` +queryPost(filter: { + or: [ + { foo: { eq: "A" } }, + { bar: { eq: "B" } }, + { baz: { eq: "C" } }, + { quz: { eq: "D" } } + ] +} ) { ... } +``` diff --git a/docusaurus-docs/graphql_versioned_docs/version-v25.4/queries/cached-results.md b/docusaurus-docs/graphql_versioned_docs/version-v25.4/queries/cached-results.md new file mode 100644 index 00000000..f3228dad --- /dev/null +++ b/docusaurus-docs/graphql_versioned_docs/version-v25.4/queries/cached-results.md @@ -0,0 +1,41 @@ +--- +title: "Cached Results" +description: "Cached results can serve read-heavy workloads with complex queries to improve performance. This refers to external caching at the browser/CDN level" + +--- + +Cached results can be used to serve read-heavy workloads with complex queries to improve performance. When cached results are enabled for a query, the stored results are served if queried within the defined time-to-live (TTL) of the cached query. + +When using cached results, Dgraph will add the appropriate HTTP headers so the caching can be done at the browser or content delivery network (CDN) level. + + +:::note +Caching refers to external caching at the browser/CDN level. Internal caching at the database layer is not currently supported. +::: + +### Enabling cached results + +To enable the external result cache you need to add the `@cacheControl(maxAge: int)` directive at the top of your query. This directive adds the appropriate `Cache-Control` HTTP headers to the response, so that browsers and CDNs can cache the results. + +For example, the following query defines a cache with TTL of 15 seconds. + +```graphql +query @cacheControl(maxAge: 15){ + queryReview(filter: { comment: {alloftext: "Fantastic"}}) { + comment + by { + username + } + about { + name + } + } +} +``` + +Dgraph's returned HTTP headers: + +``` +Cache-Control: public,max-age=15 +Vary: Accept-Encoding +``` diff --git a/docusaurus-docs/graphql_versioned_docs/version-v25.4/queries/cascade.md b/docusaurus-docs/graphql_versioned_docs/version-v25.4/queries/cascade.md new file mode 100644 index 00000000..b9e7d8dc --- /dev/null +++ b/docusaurus-docs/graphql_versioned_docs/version-v25.4/queries/cascade.md @@ -0,0 +1,136 @@ +--- +title: "@cascade Directive" +description: "The @cascade directive can be applied to fields. With the @cascade directive, nodes that don’t have all fields specified in the query are removed." + +--- + +The `@cascade` directive can be applied to fields. With the `@cascade` +directive, nodes that don’t have all fields specified in the query are removed. +This can be useful in cases where some filter was applied and some nodes might not +have all the listed fields. + +For example, the query below only returns the authors which have both `reputation` +and `posts`, where posts have `text`. Note that `@cascade` trickles down so if it's applied at the `queryAuthor` +level, it will automatically be applied at the `posts` level too. + +```graphql +{ + queryAuthor @cascade { + reputation + posts { + text + } + } +} +``` + +### Pagination + +Starting from v21.03, the `@cascade` directive supports pagination of query results. + +For example, to get to get the next 5 results after skipping the first 2 with all the fields non-null: + +```graphql +query { + queryTask(first: 5, offset: 2) @cascade { + name + completed + } +} +``` + +### Nested `@cascade` + +`@cascade` can also be used at nested levels, so the query below would return all authors +but only those posts which have both `text` and `id`. + +```graphql +{ + queryAuthor { + reputation + posts @cascade { + id + text + } + } +} +``` + +### Parameterized `@cascade` + +The `@cascade` directive can optionally take a list of fields as an argument. This changes the default behavior, considering only the supplied fields as mandatory instead of all the fields for a type. +Listed fields are automatically cascaded as a required argument to nested selection sets. + +In the example below, `name` is supplied in the `fields` argument. For an author to be in the query response, it must have a `name`, and if it has a `country` subfield, then that subfield must also have `name`. + +```graphql +{ + queryAuthor @cascade(fields:["name"]) { + reputation + name + country{ + Id + name + } + } +} +``` + +The query below only return those `posts` which have a non-null `text` field. + +```graphql +{ + queryAuthor { + reputation + name + posts @cascade(fields:["text"]) { + title + text + } + } +} +``` + +#### Nesting + +The cascading nature of field selection is overwritten by a nested `@cascade`. + +For example, the query below ensures that an author has the `reputation` and `name` fields, and, if it has a `posts` subfield, then that subfield must have a `text` field. + +```graphql +{ + queryAuthor @cascade(fields:["reputation","name"]) { + reputation + name + dob + posts @cascade(fields:["text"]) { + title + text + } + } +} +``` + + +#### Filtering + +Filters can be used with the `@cascade` directive if they are placed before it: + +```graphql +{ + queryAuthor (filter: { + name: { + anyofterms: "Alice Bob" + } + }) @cascade(fields:["reputation","name"]) { + reputation + name + dob + posts @cascade(fields:["text"]) { + title + text + } + } +} +``` + diff --git a/docusaurus-docs/graphql_versioned_docs/version-v25.4/queries/index.md b/docusaurus-docs/graphql_versioned_docs/version-v25.4/queries/index.md new file mode 100644 index 00000000..e7738e80 --- /dev/null +++ b/docusaurus-docs/graphql_versioned_docs/version-v25.4/queries/index.md @@ -0,0 +1,4 @@ +--- +title: "Queries" + +--- \ No newline at end of file diff --git a/docusaurus-docs/graphql_versioned_docs/version-v25.4/queries/order-page.md b/docusaurus-docs/graphql_versioned_docs/version-v25.4/queries/order-page.md new file mode 100644 index 00000000..4a6c57ae --- /dev/null +++ b/docusaurus-docs/graphql_versioned_docs/version-v25.4/queries/order-page.md @@ -0,0 +1,30 @@ +--- +title: "Order and Pagination" +description: "Every type with fields whose types can be ordered gets ordering built into the query and any list fields of that type." + +--- + +Every type with fields whose types can be ordered (`Int`, `Float`, `String`, `DateTime`) gets +ordering built into the query and any list fields of that type. Every query and list field +gets pagination with `first` and `offset` and ordering with `order` parameter. + +The `order` parameter is not required for pagination. + +For example, find the most recent 5 posts. + +```graphql +queryPost(order: { desc: datePublished }, first: 5) { ... } +``` + +Skip the first five recent posts and then get the next 10. + +```graphql +queryPost(order: { desc: datePublished }, offset: 5, first: 10) { ... } +``` + +It's also possible to give multiple orders. For example, sort by date and within each +date order the posts by number of likes. + +```graphql +queryPost(order: { desc: datePublished, then: { desc: numLikes } }, first: 5) { ... } +``` diff --git a/docusaurus-docs/graphql_versioned_docs/version-v25.4/queries/persistent-queries.md b/docusaurus-docs/graphql_versioned_docs/version-v25.4/queries/persistent-queries.md new file mode 100644 index 00000000..7e89d2ea --- /dev/null +++ b/docusaurus-docs/graphql_versioned_docs/version-v25.4/queries/persistent-queries.md @@ -0,0 +1,68 @@ +--- +title: "Persistent Queries" +description: "Persistent queries significantly improve the performance of an application as the smaller hash signature reduces bandwidth utilization." + +--- + +Dgraph supports Persistent Queries. When a client uses persistent queries, the client only sends the hash of a query to the server. The server has a list of known hashes and uses the associated query accordingly. + +Persistent queries significantly improve the performance and the security of an application since the smaller hash signature reduces bandwidth utilization and speeds up client loading times. + +### Persisted Query logic + +The execution of Persistent Queries follows this logic: + +- If the `extensions` key is not provided in the `GET` request, Dgraph will process the request as usual +- If a `persistedQuery ` exists under the `extensions` key, Dgraph will try to process a Persisted Query: + - if no `sha256` hash is provided, process the query without persisting + - if the `sha256` hash is provided, try to retrieve the persisted query + +Example: + +```json +{ + "persistedQuery":{ + "sha256Hash":"b952c19b894e1aa89dc05b7d53e15ab34ee0b3a3f11cdf3486acef4f0fe85c52" + } +} +``` + +### Create + +To create a Persistent Query, both `query` and `sha256` must be provided. + +Dgraph will verify the hash and perform a lookup. If the query doesn't exist, Dgraph will store the query, provided that the `sha256` of the query is correct. Finally, Dgraph will process the query and return the results. + +Example: + +```sh +curl -g 'http://localhost:8080/graphql/?query={sample_query}&extensions={"persistedQuery":{"sha256Hash":"b952c19b894e1aa89dc05b7d53e15ab34ee0b3a3f11cdf3486acef4f0fe85c52"}}' +``` + +### Lookup + +If only a `sha256` is provided, Dgraph will do a look-up, and process the query if found. Otherwise you'll get a `PersistedQueryNotFound` error. + +Example: + +```sh +curl -g 'http://localhost:8080/graphql/?extensions={"persistedQuery":{"sha256Hash":"b952c19b894e1aa89dc05b7d53e15ab34ee0b3a3f11cdf3486acef4f0fe85c52"}}' +``` + +### Usage with Apollo + +You can create an [Apollo GraphQL](https://www.apollographql.com/) client with persisted queries enabled. In the background, Apollo will send the same requests like the ones previously shown. + +For example: + +```go +import { createPersistedQueryLink } from "apollo-link-persisted-queries"; +import { createHttpLink } from "apollo-link-http"; +import { InMemoryCache } from "apollo-cache-inmemory"; +import ApolloClient from "apollo-client"; +const link = createPersistedQueryLink().concat(createHttpLink({ uri: "/graphql" })); +const client = new ApolloClient({ + cache: new InMemoryCache(), + link: link, +}); +``` diff --git a/docusaurus-docs/graphql_versioned_docs/version-v25.4/queries/queries-overview.md b/docusaurus-docs/graphql_versioned_docs/version-v25.4/queries/queries-overview.md new file mode 100644 index 00000000..f91a9f1b --- /dev/null +++ b/docusaurus-docs/graphql_versioned_docs/version-v25.4/queries/queries-overview.md @@ -0,0 +1,50 @@ +--- +title: "Overview" +description: "Dgraph automatically generates GraphQL queries for each type that you define in your schema. There are three types of queries generated for each type." + +--- + +How to use queries to fetch data from Dgraph. + +Dgraph automatically generates GraphQL queries for each type that you define in +your schema. There are three types of queries generated for each type. + +Example + +```graphql +type Post { + id: ID! + title: String! @search + text: String + score: Float @search + completed: Boolean @search + datePublished: DateTime @search(by: [year]) + author: Author! +} + +type Author { + id: ID! + name: String! @search + posts: [Post!] + friends: [Author] +} +``` + +With the above schema, there would be three queries generated for Post and three +for Author. Here are the queries that are generated for the Post type: + +```graphql +getPost(postID: ID!): Post +queryPost(filter: PostFilter, order: PostOrder, first: Int, offset: Int): [Post] +aggregatePost(filter: PostFilter): PostAggregateResult +``` + +The first query allows you to fetch a post and its related fields given an ID. +The second query allows you to fetch a list of posts based on some filters, sorting and +pagination parameters. The third query allows you to fetch aggregate parameters +like count of nodes based on filters. + +Additionally, a `checkPassword` query is generated for types that have been specified with a `@secret` directive. + +You can look at all the queries that are generated by using any +GraphQL client such as Insomnia or GraphQL playground. diff --git a/docusaurus-docs/graphql_versioned_docs/version-v25.4/queries/search-filtering.md b/docusaurus-docs/graphql_versioned_docs/version-v25.4/queries/search-filtering.md new file mode 100644 index 00000000..98749a76 --- /dev/null +++ b/docusaurus-docs/graphql_versioned_docs/version-v25.4/queries/search-filtering.md @@ -0,0 +1,329 @@ +--- +title: "Search and Filtering" +description: " Queries generated for a GraphQL type allow you to generate a single list of objects for a type. You can also query a list of objects using GraphQL." + +--- + +Queries generated for a GraphQL type allow you to generate a single list of +objects for a type. + +### Get a single object + +Fetch the `title`, `text` and `datePublished` for a post with id `0x1`. + +```graphql +query { + getPost(id: "0x1") { + title + text + datePublished + } +} +``` + +Fetching nested linked objects, while using `get` queries is also easy. For +example, this is how you would fetch the authors for a post and their friends. + +```graphql +query { + getPost(id: "0x1") { + id + title + text + datePublished + author { + name + friends { + name + } + } + } +} +``` + +While fetching nested linked objects, you can also apply a filter on them. + +For example, the following query fetches the author with the `id` 0x1 and their +posts about `GraphQL`. + +```graphql +query { + getAuthor(id: "0x1") { + name + posts(filter: { + title: { + allofterms: "GraphQL" + } + }) { + title + text + datePublished + } + } +} +``` + +If your type has a field with the `@id` directive applied to it, you can also fetch objects using that. + +For example, given the following schema, the query below fetches a user's `name` and `age` by `userID` (which has the `@id` directive): + +**Schema**: + +```graphql +type User { + userID: String! @id + name: String! + age: String +} +``` + +**Query**: + +```graphql +query { + getUser(userID: "0x2") { + name + age + } +} +``` + +:::note +The `get` API on interfaces containing fields with the `@id` directive is being deprecated and will be removed in v21.11. +Users are advised to use the `query` API instead. +::: + +### Query a list of objects + +You can query a list of objects using GraphQL. For example, the following query fetches the `title`, `text` and and `datePublished` for all posts: + +```graphql +query { + queryPost { + id + title + text + datePublished + } +} +``` + +The following example query fetches a list of posts by their post `id`: + +```graphql +query { + queryPost(filter: { + id: ["0x1", "0x2", "0x3", "0x4"], + }) { + id + title + text + datePublished + } +} +``` + +### Query that filters objects by predicate + +Before filtering an object by a predicate, you need to add a `@search` directive to the field that will be used to filter the results. + +For example, if you wanted to query events between two dates, or events that fall within a certain radius of a point, you could have an `Event` schema, as follows: + +``` +type Event { + id: ID! + date: DateTime! @search + location: Point @search +} +``` + +The search directive would let you filter events that fall within a date range, as follows: + +``` +query { + queryEvent (filter: { date: { between: { min: "2020-01-01", max: "2020-02-01" } } }) { + id + } +} +``` + +You can also filter events that have a location near a certain point with the following query: + +``` +query { + queryEvent (filter: { location: { near: { coordinate: { latitude: 37.771935, longitude: -122.469829 }, distance: 1000 } } }) { + id + } +} +``` + + +You can also use connectors such as the `and` keyword to show results with multiple filters applied. In the query below, we fetch posts that have `GraphQL` in their title and have a `score > 100`. + +This example assumes that the `Post` type has a `@search` directive applied to the `title` field and the `score` field. + + + +```graphql +query { + queryPost(filter: { + title: { + anyofterms: "GraphQL" + }, + and: { + score: { + gt: 100 + } + } + }) { + id + title + text + datePublished + } +} +``` + +### Filter a query for a list of objects + +You can also filter nested objects while querying for a list of objects. + +For example, the following query fetches all of the authors whose name contains +`Lee` and with their `completed` posts that have a score greater than `10`: + +```graphql +query { + queryAuthor(filter: { + name: { + anyofterms: "Lee" + } + }) { + name + posts(filter: { + score: { + gt: 10 + }, + and: { + completed: true + } + }) { + title + text + datePublished + } + } +} +``` + +### Filter a query for a range of objects with `between` + +You can filter query results within an inclusive range of indexed and typed +scalar values using the `between` keyword. + +:::tipThis keyword is also supported for DQL; to learn more, see +[DQL Functions: `between`](/dql/query/functions#between).::: + + +For example, you might start with the following example schema used to track +students at a school: + +**Schema**: + +```graphql +type Student{ + age: Int @search + name: String @search(by: [exact]) +} +``` +Using the `between` filter, you could fetch records for students who are between +10 and 20 years of age: + +**Query**: + +```graphql +queryStudent(filter: {age: between: {min: 10, max: 20}}){ + age + name +} +``` + +You could also use this filter to fetch records for students whose names fall +alphabetically between `ba` and `hz`: + +**Query**: + +```graphql +queryStudent(filter: {name: between: {min: "ba", max: "hz"}}){ + age + name +} +``` + +### Filter to match specified field values with `in` + +You can filter query results to find objects with one or more specified values using the +`in` keyword. This keyword can find matches for fields with the `@id` directive +applied. The `in` filter is supported for all data types such as `string`, `enum`, `Int`, `Int64`, `Float`, and `DateTime`. + +For example, let's say that your schema defines a `State` type that has the +`@id` directive applied to the `code` field: + +```graphql +type State { + code: String! @id + name: String! + capital: String +} +``` + +Using the `in` keyword, you can query for a list of states that have the postal +code **WA** or **VA** using the following query: + +```graphql +query { + queryState(filter: {code: {in : ["WA", "VA"]}}){ + code + name + } + } +``` + +### Filter for objects with specified non-null fields using `has` + +You can filter queries to find objects with a non-null value in a specified +field using the `has` keyword. The `has` keyword can only check whether a field +returns a non-null value, not for specific field values. + +For example, your schema might define a `Student` type that has basic +information about each student; such as their ID number, age, name, and email address: + +```graphql +type Student { + tid: ID! + age: Int! + name: String + email: String +} +``` + +To find those students who have a non-null `name`, run the following query: + +```graphql +queryStudent(filter: { has : name } ){ + tid + age + name +} +``` +You can also specify a list of fields, like the following: + +```graphql +queryStudent(filter: { has : [name, email] } ){ + tid + age + name + email +} +``` + +This would return `Student` objects where both `name` and `email` fields are non-null. diff --git a/docusaurus-docs/graphql_versioned_docs/version-v25.4/queries/skip-include.md b/docusaurus-docs/graphql_versioned_docs/version-v25.4/queries/skip-include.md new file mode 100644 index 00000000..bcedac61 --- /dev/null +++ b/docusaurus-docs/graphql_versioned_docs/version-v25.4/queries/skip-include.md @@ -0,0 +1,61 @@ +--- +title: "@skip and @include Directives" +description: "@skip and @include directives can be applied to query fields. They let you skip or include a field based on the value of the if argument." + +--- + +`@skip` and `@include` directives can be applied to query fields. +They allow you to skip or include a field based on the value of the `if` argument +that is passed to the directive. + +## @skip + +In the query below, we fetch posts and decide whether to fetch the title for them or not +based on the `skipTitle` GraphQL variable. + +GraphQL query + +```graphql +query ($skipTitle: Boolean!) { + queryPost { + id + title @skip(if: $skipTitle) + text + } +} +``` + +GraphQL variables +```json +{ + "skipTitle": true +} +``` + +## @include + +Similarly, the `@include` directive can be used to include a field based on the value of +the `if` argument. The query below would only include the authors for a post if `includeAuthor` +GraphQL variable has value true. + +GraphQL Query +```graphql +query ($includeAuthor: Boolean!) { + queryPost { + id + title + text + author @include(if: $includeAuthor) { + id + name + } + } +} +``` + +GraphQL variables +```json +{ + "includeAuthor": false +} +``` diff --git a/docusaurus-docs/graphql_versioned_docs/version-v25.4/queries/vector-similarity.md b/docusaurus-docs/graphql_versioned_docs/version-v25.4/queries/vector-similarity.md new file mode 100644 index 00000000..d5dcf52a --- /dev/null +++ b/docusaurus-docs/graphql_versioned_docs/version-v25.4/queries/vector-similarity.md @@ -0,0 +1,61 @@ +--- +title: "Similarity Search" +description: "Dgraph automatically generates GraphQL queries for each vector index that you define in your schema. There are two types of queries generated for each index." + +--- + +Dgraph automatically generates two GraphQL similarity queries for each type that have at least one [vector predicate](/graphql/schema/types/#vectors) with `@search` directive. + +For example + +```graphql +type User { + id: ID! + name: String! + name_v: [Float!] @embedding @search(by: ["hnsw(metric: euclidean, exponent: 4)"]) +} +``` + +With the above schema, the auto-generated `querySimilarByEmbedding` query allows us to run similarity search using the vector index specified in our schema. + +```graphql +getSimilarByEmbedding( + by: vector_predicate, + topK: n, + vector: searchVector): [User] +``` + +For example in order to find top 3 users with names similar to a given user name embedding the following query function can be used. + +```graphql +querySimilarUserByEmbedding(by: name_v, topK: 3, vector: [0.1, 0.2, 0.3, 0.4, 0.5]) { + id + name + vector_distance + } +``` +The results obtained for this query includes the 3 closest Users ordered by vector_distance. The vector_distance is the Euclidean distance between the name_v embedding vector and the input vector used in our query. + +Note: you can omit vector_distance predicate in the query, the result will still be ordered by vector_distance. + +The distance metric used is specified in the index creation. + +Similarly, the auto-generated `querySimilarById` query allows us to search for similar objects to an existing object, given it’s Id. using the function. + +```graphql +getSimilarById( + by: vector_predicate, + topK: n, + id: userID): [User] +``` + +For example the following query searches for top 3 users whose names are most similar to the name of the user with id "0xef7". + +```graphql +querySimilarUserById(by: name_v, topK: 3, id: "0xef7") { + id + name + vector_distance +} +``` + diff --git a/docusaurus-docs/graphql_versioned_docs/version-v25.4/quick-start/index.md b/docusaurus-docs/graphql_versioned_docs/version-v25.4/quick-start/index.md new file mode 100644 index 00000000..6778712e --- /dev/null +++ b/docusaurus-docs/graphql_versioned_docs/version-v25.4/quick-start/index.md @@ -0,0 +1,253 @@ +--- +title: "Quick Start" +description: "Go from an empty Dgraph database to a running GraphQL API in just one step; just define the schema of your graph and how you’d like to search it; Dgraph does the rest." + +--- + + +## Overview + +Traditional GraphQL implementations require significant overhead when building on top of REST endpoints or relational databases. Developers must manually translate REST/relational data models into graph structures, implement resolvers for each field, and manage the numerous queries generated during this translation process. + +Dgraph simplifies this workflow by providing a schema-first approach. By deploying your GraphQL schema, Dgraph automatically generates a fully functional GraphQL API with a native graph database backend—eliminating the need for manual resolver implementation and data translation layers. + +## Step 1: Run Dgraph + +The easiest way to get Dgraph up and running is to install a [Learning Environment](/installation/single-host-setup). + + +## Step 2: Deploy a GraphQL Schema + +1. Create a file schema.graphql with the following content. + + + ```graphql + type Product { + productID: ID! + name: String @search(by: [term]) + reviews: [Review] @hasInverse(field: about) + } + + type Customer { + username: String! @id @search(by: [hash, regexp]) + reviews: [Review] @hasInverse(field: by) + } + + type Review { + id: ID! + about: Product! + by: Customer! + comment: String @search(by: [fulltext]) + rating: Int @search + } + ``` + +2. Push the schema to Dgraph + +From a terminal window + +``` +curl --data-binary '@./schema.graphql --header 'content-type: application/octet-stream' http://localhost:8080/admin/schema + +``` + + +## Step 3: Test your GraphQL API + +You can access the `GraphQL endpoint` with any GraphQL clients such as [GraphQL Playground](https://github.com/prisma-labs/graphql-playground), [Insomnia](https://insomnia.rest/), [GraphiQL](https://github.com/graphql/graphiql), [Altair](https://github.com/imolorhe/altair) or Postman. + + +You may want to use the introspection capability of the client to explore the schema, queries, and mutations that were generated by Dgraph. + +### A first GraphQL mutation +To populate the database, + + ```graphql + mutation { + addProduct( + input: [ + { name: "GraphQL on Dgraph" } + { name: "Dgraph: The GraphQL Database" } + ] + ) { + product { + productID + name + } + } + addCustomer(input: [{ username: "Michael" }]) { + customer { + username + } + } + } + ``` + + +The GraphQL server returns a json response similar to this: + +```json +{ + "data": { + "addProduct": { + "product": [ + { + "productID": "0x2", + "name": "GraphQL on Dgraph" + }, + { + "productID": "0x3", + "name": "Dgraph: The GraphQL Database" + } + ] + }, + "addCustomer": { + "customer": [ + { + "username": "Michael" + } + ] + } + }, + "extensions": { + "requestID": "b155867e-4241-4cfb-a564-802f2d3808a6" + } +} +``` + + +### A second GraphQL mutation +Because the schema defined Customer with the field `username: String! @id`, the `username` field acts like an ID, so we can identify customers just with their names. + +Products, on the other hand, had `productID: ID!`, so they'll get an auto-generated ID which are returned by the mutation. + + +Your ID for the product might be different than `0x2`. Make sure to replace the product ID with the ID from the response of the previous mutation. + +Execute the mutation + + +```graphql +mutation { + addReview(input: [{ + by: {username: "Michael"}, + about: { productID: "0x2"}, + comment: "Fantastic, easy to install, worked great. Best GraphQL server available", + rating: 10}]) + { + review { + comment + rating + by { username } + about { name } + } + } +} +``` + +This time, the mutation result queries for the author making the review and the product being reviewed, so it's gone deeper into the graph to get the result than just the mutation data. + +```json +{ + "data": { + "addReview": { + "review": [ + { + "comment": "Fantastic, easy to install, worked great. Best GraphQL server available", + "rating": 10, + "by": { + "username": "Michael" + }, + "about": { + "name": "GraphQL on Dgraph" + } + } + ] + } + }, + "extensions": { + "requestID": "11bc2841-8c19-45a6-bb31-7c37c9b027c9" + } +} +``` + + + +### GraphQL Queries + +With Dgraph, you get powerful graph search built into your GraphQL API. The schema for search is generated from the schema document that we started with and automatically added to the GraphQL API for you. + +Remember the definition of a review. + +``` +type Review { + ... + comment: String @search(by: [fulltext]) + ... +} +``` + +The directive `@search(by: [fulltext])` tells Dgraph we want to be able to search for comments with full-text search. + +Dgraph took that directive and the other information in the schema, and built queries and search into the API. + +Let's find all the products that were easy to install. + +Execute the query + +```graphql +query { + queryReview(filter: { comment: {alloftext: "easy to install"}}) { + comment + by { + username + } + about { + name + } + } +} +``` + +What reviews did you get back? It'll depend on the data you added, but you'll at least get the initial review we added. + +Maybe you want to find reviews that describe best GraphQL products and give a high rating. + +```graphql +query { + queryReview(filter: { comment: {alloftext: "best GraphQL"}, rating: { ge: 10 }}) { + comment + by { + username + } + about { + name + } + } +} +``` + +How about we find the customers with names starting with "Mich" and the five products that each of those liked the most. + +```graphql +query { + queryCustomer(filter: { username: { regexp: "/Mich.*/" } }) { + username + reviews(order: { asc: rating }, first: 5) { + comment + rating + about { + name + } + } + } +} +``` +## Conclusion + +Dgraph allows you to have a fully functional GraphQL API in minutes with a highly performant graph backend to serve complex nested queries. Moreover, you can update or change your schema freely and just re-deploy new versions. For GraphQL in Dgraph, you just concentrate on defining the schema of your graph and how you'd like to search that graph; Dgraph does the rest. + + +## What's Next +- Learn more about [GraphQL schema](/graphql/schema/) and Dgraph directives. + diff --git a/docusaurus-docs/graphql_versioned_docs/version-v25.4/schema/dgraph-schema.md b/docusaurus-docs/graphql_versioned_docs/version-v25.4/schema/dgraph-schema.md new file mode 100644 index 00000000..b4712368 --- /dev/null +++ b/docusaurus-docs/graphql_versioned_docs/version-v25.4/schema/dgraph-schema.md @@ -0,0 +1,270 @@ +--- +title: "Dgraph Schema Fragment" +description: "While editing your schema, this GraphQL schema fragment can be useful. It sets up the definitions of the directives that you’ll use in your schema." + +--- + +While editing your schema, you might find it useful to include this GraphQL schema fragment. It sets up the definitions of the directives, etc. (like `@search`) that you'll use in your schema. If your editor is GraphQL aware, it may give you errors if you don't have this available and context sensitive help if you do. + +Don't include it in your input schema to Dgraph - use your editing environment to set it up as an import. The details will depend on your setup. + +```graphql +""" +The Int64 scalar type represents a signed 64‐bit numeric non‐fractional value. +Int64 can represent values in range [-(2^63),(2^63 - 1)]. +""" +scalar Int64 + +""" +The DateTime scalar type represents date and time as a string in RFC3339 format. +For example: "1985-04-12T23:20:50.52Z" represents 20 minutes and 50.52 seconds after the 23rd hour of April 12th, 1985 in UTC. +""" +scalar DateTime + +input IntRange{ + min: Int! + max: Int! +} + +input FloatRange{ + min: Float! + max: Float! +} + +input Int64Range{ + min: Int64! + max: Int64! +} + +input DateTimeRange{ + min: DateTime! + max: DateTime! +} + +input StringRange{ + min: String! + max: String! +} + +enum DgraphIndex { + int + int64 + float + bool + hash + exact + term + fulltext + trigram + ngram + regexp + year + month + day + hour + geo +} + +input AuthRule { + and: [AuthRule] + or: [AuthRule] + not: AuthRule + rule: String +} + +enum HTTPMethod { + GET + POST + PUT + PATCH + DELETE +} + +enum Mode { + BATCH + SINGLE +} + +input CustomHTTP { + url: String! + method: HTTPMethod! + body: String + graphql: String + mode: Mode + forwardHeaders: [String!] + secretHeaders: [String!] + introspectionHeaders: [String!] + skipIntrospection: Boolean +} + +type Point { + longitude: Float! + latitude: Float! +} + +input PointRef { + longitude: Float! + latitude: Float! +} + +input NearFilter { + distance: Float! + coordinate: PointRef! +} + +input PointGeoFilter { + near: NearFilter + within: WithinFilter +} + +type PointList { + points: [Point!]! +} + +input PointListRef { + points: [PointRef!]! +} + +type Polygon { + coordinates: [PointList!]! +} + +input PolygonRef { + coordinates: [PointListRef!]! +} + +type MultiPolygon { + polygons: [Polygon!]! +} + +input MultiPolygonRef { + polygons: [PolygonRef!]! +} + +input WithinFilter { + polygon: PolygonRef! +} + +input ContainsFilter { + point: PointRef + polygon: PolygonRef +} + +input IntersectsFilter { + polygon: PolygonRef + multiPolygon: MultiPolygonRef +} + +input PolygonGeoFilter { + near: NearFilter + within: WithinFilter + contains: ContainsFilter + intersects: IntersectsFilter +} + +input GenerateQueryParams { + get: Boolean + query: Boolean + password: Boolean + aggregate: Boolean +} + +input GenerateMutationParams { + add: Boolean + update: Boolean + delete: Boolean +} + +directive @hasInverse(field: String!) on FIELD_DEFINITION +directive @search(by: [DgraphIndex!]) on FIELD_DEFINITION +directive @dgraph(type: String, pred: String) on OBJECT | INTERFACE | FIELD_DEFINITION +directive @id(interface: Boolean) on FIELD_DEFINITION +directive @withSubscription on OBJECT | INTERFACE | FIELD_DEFINITION +directive @secret(field: String!, pred: String) on OBJECT | INTERFACE +directive @auth( + password: AuthRule + query: AuthRule, + add: AuthRule, + update: AuthRule, + delete: AuthRule) on OBJECT | INTERFACE +directive @custom(http: CustomHTTP, dql: String) on FIELD_DEFINITION +directive @remote on OBJECT | INTERFACE | UNION | INPUT_OBJECT | ENUM +directive @remoteResponse(name: String) on FIELD_DEFINITION +directive @cascade(fields: [String]) on FIELD +directive @lambda on FIELD_DEFINITION +directive @lambdaOnMutate(add: Boolean, update: Boolean, delete: Boolean) on OBJECT | INTERFACE +directive @cacheControl(maxAge: Int!) on QUERY +directive @generate( + query: GenerateQueryParams, + mutation: GenerateMutationParams, + subscription: Boolean) on OBJECT | INTERFACE + +input IntFilter { + eq: Int + in: [Int] + le: Int + lt: Int + ge: Int + gt: Int + between: IntRange +} + +input Int64Filter { + eq: Int64 + in: [Int64] + le: Int64 + lt: Int64 + ge: Int64 + gt: Int64 + between: Int64Range +} + +input FloatFilter { + eq: Float + in: [Float] + le: Float + lt: Float + ge: Float + gt: Float + between: FloatRange +} + +input DateTimeFilter { + eq: DateTime + in: [DateTime] + le: DateTime + lt: DateTime + ge: DateTime + gt: DateTime + between: DateTimeRange +} + +input StringTermFilter { + allofterms: String + anyofterms: String +} + +input StringRegExpFilter { + regexp: String +} + +input StringFullTextFilter { + alloftext: String + anyoftext: String +} + +input StringExactFilter { + eq: String + in: [String] + le: String + lt: String + ge: String + gt: String + between: StringRange +} + +input StringHashFilter { + eq: String + in: [String] +} +``` diff --git a/docusaurus-docs/graphql_versioned_docs/version-v25.4/schema/directives/auth.md b/docusaurus-docs/graphql_versioned_docs/version-v25.4/schema/directives/auth.md new file mode 100644 index 00000000..00950314 --- /dev/null +++ b/docusaurus-docs/graphql_versioned_docs/version-v25.4/schema/directives/auth.md @@ -0,0 +1,12 @@ +--- +title: "@auth" + +--- + +`@auth` allows you to define how to apply authorization rules on the queries/mutation for a type. + +Refer to [graphql endpoint security](/graphql/security/), [RBAC rules](/graphql/security/RBAC-rules) and [Graph traversal rules](/graphql/security/graphtraversal-rules) for details. + + +`@auth` directive is not supported on `union` and `@remote` types. + diff --git a/docusaurus-docs/graphql_versioned_docs/version-v25.4/schema/directives/deprecated.md b/docusaurus-docs/graphql_versioned_docs/version-v25.4/schema/directives/deprecated.md new file mode 100644 index 00000000..c3cb7c5e --- /dev/null +++ b/docusaurus-docs/graphql_versioned_docs/version-v25.4/schema/directives/deprecated.md @@ -0,0 +1,22 @@ +--- +title: "@deprecated" + +--- + +The `@deprecated` directive allows you to tag the schema definition of a field or enum value as deprecated with an optional reason. + +When you use the `@deprecated` directive, GraphQL users can deprecate their use of the deprecated field or `enum` value. +Most GraphQL tools and clients will pick up this notification and give you a warning if you try to use a deprecated field. + +### Example + +For example, to mark `oldField` in the schema as deprecated: + +```graphql +type MyType { + id: ID! + oldField: String @deprecated(reason: "oldField is deprecated. Use newField instead.") + newField: String + deprecatedField: String @deprecated +} +``` diff --git a/docusaurus-docs/graphql_versioned_docs/version-v25.4/schema/directives/directive-dgraph.md b/docusaurus-docs/graphql_versioned_docs/version-v25.4/schema/directives/directive-dgraph.md new file mode 100644 index 00000000..3ba27f01 --- /dev/null +++ b/docusaurus-docs/graphql_versioned_docs/version-v25.4/schema/directives/directive-dgraph.md @@ -0,0 +1,58 @@ +--- +title: "@dgraph" + +--- + + +The `@dgraph` directive customizes the name of the types and predicates generated in Dgraph when deploying a GraphQL Schema. + +* `type @dgraph(type: "TypeNameToUseInDgraph")` controls what Dgraph type is used for a GraphQL type. +* `field: SomeType @dgraph(pred: "DgraphPredicate")` controls what Dgraph predicate is mapped to a GraphQL field. + +For example, if you have existing types that don't match GraphQL requirements, you can create a schema like the following. + +```graphql +type Person @dgraph(type: "Human-Person") { + name: String @search(by: [hash]) @dgraph(pred: "name") + age: Int +} + +type Movie @dgraph(type: "film") { + name: String @search(by: [term]) @dgraph(pred: "film.name") +} +``` + +Which maps to the Dgraph schema: + +```graphql +type Human-Person { + name + Person.age +} + +type film { + film.name +} + +name string @index(hash) . +Person.age: int . +film.name string @index(term) . +``` + +You might also have the situation where you have used `name` for both movie names and people's names. In this case you can map fields in two different GraphQL types to the one Dgraph predicate. + +```graphql +type Person { + name: String @dgraph(pred: "name") + ... +} + +type Movie { + name: String @dgraph(pred: "name") + ... +} +``` + +:::note +In Dgraph's current GraphQL implementation, if two fields are mapped to the same Dgraph predicate, both should have the same `@search` directive. +::: \ No newline at end of file diff --git a/docusaurus-docs/graphql_versioned_docs/version-v25.4/schema/directives/directive-withsubscription.md b/docusaurus-docs/graphql_versioned_docs/version-v25.4/schema/directives/directive-withsubscription.md new file mode 100644 index 00000000..6f07480b --- /dev/null +++ b/docusaurus-docs/graphql_versioned_docs/version-v25.4/schema/directives/directive-withsubscription.md @@ -0,0 +1,24 @@ +--- +title: "@withSubscription" + +--- + + +The `@withSubscription` directive enables **subscription** operation on a GraphQL type. + +A subscription notifies your client with changes to back-end data using the WebSocket protocol. +Subscriptions are useful to get low-latency, real-time updates. + +To enable subscriptions on any type add the `@withSubscription` directive to the schema as part of the type definition, as in the following example: + +```graphql +type Todo @withSubscription { + id: ID! + title: String! + description: String! + completed: Boolean! +} +``` + +Refer to [GraphQL Subscriptions](/graphql/subscriptions) to learn how to use subscriptions in you client application. + diff --git a/docusaurus-docs/graphql_versioned_docs/version-v25.4/schema/directives/embedding.md b/docusaurus-docs/graphql_versioned_docs/version-v25.4/schema/directives/embedding.md new file mode 100644 index 00000000..f4c0efcd --- /dev/null +++ b/docusaurus-docs/graphql_versioned_docs/version-v25.4/schema/directives/embedding.md @@ -0,0 +1,11 @@ +--- +title: "@embedding" + +--- + + +A Float array can be used as a vector using `@embedding` directive. It denotes a vector of floating point numbers, i.e an ordered array of float32. + +The embeddings can be defined on one or more predicates of a type and they are generated using suitable machine learning models. + +This directive is used in conjunction with `@search` directive to declare the HNSW index. For more information see: [@search](/graphql/schema/directives/search/#vector-embedding) directive for vector embeddings. \ No newline at end of file diff --git a/docusaurus-docs/graphql_versioned_docs/version-v25.4/schema/directives/generate.md b/docusaurus-docs/graphql_versioned_docs/version-v25.4/schema/directives/generate.md new file mode 100644 index 00000000..333bcb2f --- /dev/null +++ b/docusaurus-docs/graphql_versioned_docs/version-v25.4/schema/directives/generate.md @@ -0,0 +1,56 @@ +--- +title: "@generate" +description: "The @generate directive specifies which GraphQL APIs are generated for a given type. Without it, all queries & mutations are generated except subscription." + +--- + +The `@generate` directive is used to specify which GraphQL APIs are generated for a given type. + +Here's the GraphQL definition of the directive +```graphql +input GenerateQueryParams { + get: Boolean + query: Boolean + password: Boolean + aggregate: Boolean +} + +input GenerateMutationParams { + add: Boolean + update: Boolean + delete: Boolean +} +directive @generate( + query: GenerateQueryParams, + mutation: GenerateMutationParams, + subscription: Boolean) on OBJECT | INTERFACE + +``` + +The corresponding APIs are generated by setting the `Boolean` variables inside the `@generate` directive to `true`. Passing `false` forbids the generation of the corresponding APIs. + +The default value of the `subscription` variable is `false` while the default value of all +other variables is `true`. Therefore, if no `@generate` directive is specified for a type, all queries and mutations except `subscription` are generated. + +## Example of @generate directive + +```graphql +type Person @generate( + query: { + get: false, + query: true, + aggregate: false + }, + mutation: { + add: true, + delete: false + }, + subscription: false +) { + id: ID! + name: String! +} +``` + +The GraphQL schema above will generate a `queryPerson` query and `addPerson`, `updatePerson` mutations. It won't generate `getPerson`, `aggregatePerson` queries nor a `deletePerson` mutation as these have been marked as `false` using the `@generate` directive. +Note that the `updatePerson` mutation is generated because the default value of the `update` variable is `true`. diff --git a/docusaurus-docs/graphql_versioned_docs/version-v25.4/schema/directives/ids.md b/docusaurus-docs/graphql_versioned_docs/version-v25.4/schema/directives/ids.md new file mode 100644 index 00000000..68c93693 --- /dev/null +++ b/docusaurus-docs/graphql_versioned_docs/version-v25.4/schema/directives/ids.md @@ -0,0 +1,115 @@ +--- +title: "@id" +description: "Dgraph database provides two types of identifiers: the ID scalar type and the @id directive." + +--- + +Dgraph provides two types of built-in identifiers: the `ID` scalar type and the `@id` directive. + +* The `ID` scalar type is used when you don't need to set an identifier outside of Dgraph. +* The `@id` directive is used for external identifiers, such as email addresses. + + +## The `@id` directive + +For some types, you'll need a unique identifier set from outside Dgraph. A common example is a username. + +The `@id` directive tells Dgraph to keep that field's values unique and use them as identifiers. + +For example, you might set the following type in a schema: + +```graphql +type User { + username: String! @id + ... +} +``` + +Dgraph requires a unique username when creating a new user. It generates the input type for `addUser` with `username: String!`, so you can't make an add mutation without setting a username; and when processing the mutation, Dgraph will ensure that the username isn't already set for another node of the `User` type. + +In a single-page app, you could render the page for `http://.../user/Erik` when a user clicks to view the author bio page for that user. Your app can then use a `getUser(username: "Erik") { ... }` GraphQL query to fetch the data and generate the page. + +Identities created with `@id` are reusable. If you delete an existing user, you can reuse the username. + +Fields with the `@id` directive must have the type `String!`. + +As with `ID` types, Dgraph generates queries and mutations so you can query, update, and delete data in nodes, using the fields with the `@id` directive as references. + +It's possible to use the `@id` directive on more than one field in a type. For example, you can define a type like the following: + +```graphql +type Book { + name: String! @id + isbn: String! @id + genre: String! + ... +} +``` + +You can then use multiple `@id` fields in arguments to `get` queries, and while searching, these fields will be combined with the `AND` operator, resulting in a Boolean `AND` operation. For example, for the above schema, you can send a `getBook` query like the following: + +```graphql +query { + getBook(name: "The Metamorphosis", isbn: "9871165072") { + name + genre + ... + } +} +``` + +This will yield a positive response if both the `name` **and** `isbn` match any data in the database. + +### `@id` and interfaces + +By default, if used in an interface, the `@id` directive will ensure field uniqueness for each implementing type separately. +In this case, the `@id` field in the interface won't be unique for the interface but for each of its implementing types. +This allows two different types implementing the same interface to have the same value for the inherited `@id` field. + +There are scenarios where this behavior might not be desired, and you may want to constrain the `@id` field to be unique across all the implementing types. In that case, you can set the `interface` argument of the `@id` directive to `true`, and Dgraph will ensure that the field has unique values across all the implementing types of an interface. + +For example: + +```graphql +interface Item { + refID: Int! @id(interface: true) # if there is a Book with refID = 1, then there can't be a chair with that refID. + itemID: Int! @id # If there is a Book with itemID = 1, there can still be a Chair with the same itemID. +} + +type Book implements Item { ... } +type Chair implements Item { ... } +``` + +In the above example, `itemID` won't be present as an argument to the `getItem` query as it might return more than one `Item`. + +:::note +`get` queries generated for an interface will have only the `@id(interface: true)` fields as arguments. +::: + +## Combining `ID` and `@id` + +You can use both the `ID` type and the `@id` directive on another field definition to have both a unique identifier and a generated identifier. + +For example, you might define the following type in a schema: + +```graphql +type User { + id: ID! + username: String! @id + ... +} +``` + +With this schema, Dgraph requires a unique `username` when creating a new user. This schema provides the benefits of both of the previous examples above. Your app can then use the `getUser(...) { ... }` query to provide either the Dgraph-generated `id` or the externally-generated `username`. + +:::note +If in a type there are multiple `@id` fields, then in a `get` query these arguments will be optional. If in a type there's only one field defined with either `@id` or `ID`, then that will be a required field in the `get` query's arguments. +::: + + diff --git a/docusaurus-docs/graphql_versioned_docs/version-v25.4/schema/directives/index.md b/docusaurus-docs/graphql_versioned_docs/version-v25.4/schema/directives/index.md new file mode 100644 index 00000000..71439a36 --- /dev/null +++ b/docusaurus-docs/graphql_versioned_docs/version-v25.4/schema/directives/index.md @@ -0,0 +1,116 @@ +--- +title: "Directives" + +--- + +The list of all directives supported by Dgraph. + +### @auth + +`@auth` allows you to define how to apply authorization rules on the queries/mutation for a type. + +Reference: [Auth directive](auth) + +### @cascade + +`@cascade` allows you to filter out certain nodes within a query. + +Reference: [Cascade](/graphql/queries/cascade) + +### @custom + +`@custom` directive is used to define custom queries, mutations and fields. + +Reference: [Custom directive](/graphql/custom/directive) + +### @deprecated + +The `@deprecated` directive lets you mark the schema definition of a field or `enum` value as deprecated, and also lets you provide an optional reason for the deprecation. + +Reference: [Deprecation]((deprecated) + +### @dgraph + +`@dgraph` directive tells us how to map fields within a type to existing predicates inside Dgraph. + +Reference: [@dgraph directive](directive-dgraph) + +### @embedding + +`@embedding` directive designates one or more fields as vector embeddings. + +Reference: [@embedding directive](embedding) + +### @generate + +The `@generate` directive is used to specify which GraphQL APIs are generated for a type. + +Reference: [Generate directive](generate) + +### @hasInverse + +`@hasInverse` is used to setup up two way edges such that adding a edge in +one direction automatically adds the one in the inverse direction. + +Reference: [Linking nodes in the graph](/graphql/schema/graph-links) + +### @id + +`@id` directive is used to annotate a field which represents a unique identifier coming from outside + of Dgraph. + +Reference: [Identity](ids) + +### @include + +The `@include` directive can be used to include a field based on the value of an `if` argument. + +Reference: [Include directive](/graphql/queries/skip-include) + +### @lambda + +The `@lambda` directive allows you to call custom JavaScript resolvers. The `@lambda` queries, mutations, and fields are resolved through the lambda functions implemented on a given lambda server. + +Reference: [Lambda directive](/graphql/lambda/lambda-overview) + +### @remote + +`@remote` directive is used to annotate types for which data is not stored in Dgraph. These types +are typically used with custom queries and mutations. + + +### @remoteResponse + +The `@remoteResponse` directive allows you to annotate the fields of a `@remote` type in order to map a custom query's JSON key response to a GraphQL field. + + +### @search + +`@search` allows you to perform filtering on a field while querying for nodes. + +Reference: [Search](search) + +### @secret + +`@secret` directive is used to store secret information, it gets encrypted and then stored in Dgraph. + +Reference: [Password Type](/graphql/schema/types#password-type) + +### @skip + +The `@skip` directive can be used to fetch a field based on the value of a user-defined GraphQL variable. + +Reference: [Skip directive](/graphql/queries/skip-include) + +### @withSubscription + +`@withSubscription` directive when applied on a type, generates subscription queries for it. + +Reference: [Subscriptions](/graphql/subscriptions) + +### @lambdaOnMutate + +The `@lambdaOnMutate` directive allows you to listen to mutation events(`add`/`update`/`delete`). Depending on the defined events and the occurrence of a mutation event, `@lambdaOnMutate` triggers the appropriate lambda function implemented on a given lambda server. + +Reference: [LambdaOnMutate directive](/graphql/lambda/webhook) + diff --git a/docusaurus-docs/graphql_versioned_docs/version-v25.4/schema/directives/search.md b/docusaurus-docs/graphql_versioned_docs/version-v25.4/schema/directives/search.md new file mode 100644 index 00000000..1dbfa371 --- /dev/null +++ b/docusaurus-docs/graphql_versioned_docs/version-v25.4/schema/directives/search.md @@ -0,0 +1,671 @@ +--- +title: "Search and Filtering" +description: "What search can you build into your GraphQL API? Dgraph builds search into the fields of each type, so searching is available at deep levels in a query." + +--- + +The `@search` directive tells Dgraph what search to build into your GraphQL API. + +When a type contains an `@search` directive, Dgraph constructs a search input type and a query in the GraphQL `Query` type. For example, if the schema contains + +```graphql +type Post { + ... +} +``` + +then Dgraph constructs a `queryPost` GraphQL query for querying posts. The `@search` directives in the `Post` type control how Dgraph builds indexes and what kinds of search it builds into `queryPost`. If the type contains + +```graphql +type Post { + ... + datePublished: DateTime @search +} +``` + +then it's possible to filter posts with a date-time search like: + +```graphql +query { + queryPost(filter: { datePublished: { ge: "2020-06-15" }}) { + ... + } +} +``` + +If the type tells Dgraph to build search capability based on a term (word) index for the `title` field + +```graphql +type Post { + ... + title: String @search(by: [term]) +} +``` + +then, the generated GraphQL API will allow search by terms in the title. + +```graphql +query { + queryPost(filter: { title: { anyofterms: "GraphQL" }}) { + ... + } +} +``` + +Dgraph also builds search into the fields of each type, so searching is available at deep levels in a query. For example, if the schema contained these types + +```graphql +type Post { + ... + title: String @search(by: [term]) +} + +type Author { + name: String @search(by: [hash]) + posts: [Post] +} +``` + +then Dgraph builds GraphQL search such that a query can, for example, find an author by name (from the hash search on `name`) and return only their posts that contain the term "GraphQL". + +```graphql +queryAuthor(filter: { name: { eq: "Diggy" } } ) { + posts(filter: { title: { anyofterms: "GraphQL" }}) { + title + } +} +``` + +Dgraph can build search types with the ability to search between a range. For example with the above Post type with datePublished field, a query can find publish dates within a range + +```graphql +query { + queryPost(filter: { datePublished: { between: { min: "2020-06-15", max: "2020-06-16" }}}) { + ... + } +} +``` + +Dgraph can also build GraphQL search ability to find match a value from a list. For example with the above Author type with the name field, a query can return the Authors that match a list + +```graphql +queryAuthor(filter: { name: { in: ["Diggy", "Jarvis"] } } ) { + ... +} +``` + +There's different search possible for each type as explained below. + +### Int, Float and DateTime + +| argument | constructed filter | +|----------|----------------------| +| none | `lt`, `le`, `eq`, `in`, `between`, `ge`, and `gt` | + +Search for fields of types `Int`, `Float` and `DateTime` is enabled by adding `@search` to the field with no arguments. For example, if a schema contains: + +```graphql +type Post { + ... + numLikes: Int @search +} +``` + +Dgraph generates search into the API for `numLikes` in two ways: a query for posts and field search on any post list. + +A field `queryPost` is added to the `Query` type of the schema. + +```graphql +type Query { + ... + queryPost(filter: PostFilter, order: PostOrder, first: Int, offset: Int): [Post] +} +``` + +`PostFilter` will contain less than `lt`, less than or equal to `le`, equal `eq`, in list `in`, between range `between`, greater than or equal to `ge`, and greater than `gt` search on `numLikes`. Allowing for example: + +```graphql +query { + queryPost(filter: { numLikes: { gt: 50 }}) { + ... + } +} +``` + +Also, any field with a type of list of posts has search options added to it. For example, if the input schema also contained: + +```graphql +type Author { + ... + posts: [Post] +} +``` + +Dgraph would insert search into `posts`, with + +```graphql +type Author { + ... + posts(filter: PostFilter, order: PostOrder, first: Int, offset: Int): [Post] +} +``` + +That allows search within the GraphQL query. For example, to find Diggy's posts with more than 50 likes. + +```graphql +queryAuthor(filter: { name: { eq: "Diggy" } } ) { + ... + posts(filter: { numLikes: { gt: 50 }}) { + title + text + } +} +``` + +### DateTime + +| argument | constructed filters | +|----------|----------------------| +| `year`, `month`, `day`, or `hour` | `lt`, `le`, `eq`, `in`, `between`, `ge`, and `gt` | + +As well as `@search` with no arguments, `DateTime` also allows specifying how the search index should be built: by year, month, day or hour. `@search` defaults to year, but once you understand your data and query patterns, you might want to changes that like `@search(by: [day])`. + +### Boolean + +| argument | constructed filter | +|----------|----------------------| +| none | `true` and `false` | + +Booleans can only be tested for true or false. If `isPublished: Boolean @search` is in the schema, then the search allows + +```graphql +filter: { isPublished: true } +``` + +and + +```graphql +filter: { isPublished: false } +``` + +### String + +Strings allow a wider variety of search options than other types. For strings, you have the following options as arguments to `@search`. + +| argument | constructed searches | +|----------|----------------------| +| `hash` | `eq` and `in` | +| `exact` | `lt`, `le`, `eq`, `in`, `between`, `ge`, and `gt` (lexicographically) | +| `regexp` | `regexp` (regular expressions) | +| `term` | `allofterms` and `anyofterms` | +| `fulltext` | `alloftext` and `anyoftext` | +| `ngram` | `ngram` | + +* *Schema rule*: `hash` and `exact` can't be used together. + +#### String exact and hash search + +Exact and hash search has the standard lexicographic meaning. + +```graphql +query { + queryAuthor(filter: { name: { eq: "Diggy" } }) { ... } +} +``` + +And for exact search + +```graphql +query { + queryAuthor(filter: { name: { gt: "Diggy" } }) { ... } +} +``` + +to find users with names lexicographically after "Diggy". + +#### String regular expression search + +Search by regular expression requires bracketing the expression with `/` and `/`. For example, query for "Diggy" and anyone else with "iggy" in their name: + +```graphql +query { + queryAuthor(filter: { name: { regexp: "/.*iggy.*/" } }) { ... } +} +``` + +#### String term and fulltext search + +If the schema has + +```graphql +type Post { + title: String @search(by: [term]) + text: String @search(by: [fulltext]) + ... +} +``` + +then + +```graphql +query { + queryPost(filter: { title: { `allofterms: "GraphQL tutorial"` } } ) { ... } +} +``` + +will match all posts with both "GraphQL and "tutorial" in the title, while `anyofterms: "GraphQL tutorial"` would match posts with either "GraphQL" or "tutorial". + +`fulltext` search is Google-stye text search with stop words, stemming. etc. So `alloftext: "run woman"` would match "run" as well as "running", etc. For example, to find posts that talk about fantastic GraphQL tutorials: + +```graphql +query { + queryPost(filter: { title: { `alloftext: "fantastic GraphQL tutorials"` } } ) { ... } +} +``` + +#### String ngram search + +The `ngram` index tokenizes a string into contiguous sequences of n words, with +support for stop word removal and stemming. N-gram search matches if the indexed +string contains the given sequence of terms. + +If the schema has + +```graphql +type Post { + title: String @search(by: [ngram]) + ... +} +``` + +then + +```graphql +query { + queryPost(filter: { title: { ngram: "quick brown fox" } } ) { ... } +} +``` + +will match all posts that contain the contiguous sequence "quick brown fox" in +the title. + + +#### Strings with multiple searches + +It's possible to add multiple string indexes to a field. For example to search for authors by `eq` and regular expressions, add both options to the type definition, as follows. + +```graphql +type Author { + ... + name: String! @search(by: [hash, regexp]) +} +``` + +### Enums + +| argument | constructed searches | +|----------|----------------------| +| none | `eq` and `in` | +| `hash` | `eq` and `in` | +| `exact` | `lt`, `le`, `eq`, `in`, `between`, `ge`, and `gt` (lexicographically) | +| `regexp` | `regexp` (regular expressions) | + +Enums are serialized in Dgraph as strings. `@search` with no arguments is the same as `@search(by: [hash])` and provides `eq` and `in` searches. Also available for enums are `exact` and `regexp`. For hash and exact search on enums, the literal enum value, without quotes `"..."`, is used, for regexp, strings are required. For example: + +```graphql +enum Tag { + GraphQL + Database + Question + ... +} + +type Post { + ... + tags: [Tag!]! @search +} +``` + +would allow + +```graphql +query { + queryPost(filter: { tags: { eq: GraphQL } } ) { ... } +} +``` + +Which would find any post with the `GraphQL` tag. + +While `@search(by: [exact, regexp]` would also admit `lt` etc. and + +```graphql +query { + queryPost(filter: { tags: { regexp: "/.*aph.*/" } } ) { ... } +} +``` + +which is helpful for example if the enums are something like product codes where regular expressions can match a number of values. + +### Geolocation + +There are 3 Geolocation types: `Point`, `Polygon` and `MultiPolygon`. All of them are searchable. + +The following table lists the generated filters for each type when you include `@search` on the corresponding field: + +| type | constructed searches | +|----------|----------------------| +| `Point` | `near`, `within` | +| `Polygon` | `near`, `within`, `contains`, `intersects` | +| `MultiPolygon` | `near`, `within`, `contains`, `intersects` | + +#### Example + +Take for example a `Hotel` type that has a `location` and an `area`: + +```graphql +type Hotel { + id: ID! + name: String! + location: Point @search + area: Polygon @search +} +``` + +#### near + +The `near` filter matches all entities where the location given by a field is within a distance `meters` from a coordinate. + +```graphql +queryHotel(filter: { + location: { + near: { + coordinate: { + latitude: 37.771935, + longitude: -122.469829 + }, + distance: 1000 + } + } +}) { + name +} +``` + +#### within + +The `within` filter matches all entities where the location given by a field is within a defined `polygon`. + +```graphql +queryHotel(filter: { + location: { + within: { + polygon: { + coordinates: [{ + points: [{ + latitude: 11.11, + longitude: 22.22 + }, { + latitude: 15.15, + longitude: 16.16 + }, { + latitude: 20.20, + longitude: 21.21 + }, { + latitude: 11.11, + longitude: 22.22 + }] + }], + } + } + } +}) { + name +} +``` + +#### contains + +The `contains` filter matches all entities where the `Polygon` or `MultiPolygon` field contains another given `point` or `polygon`. + +:::tip +Only one `point` or `polygon` can be taken inside the `ContainsFilter` at a time. +::: + +A `contains` example using `point`: + +```graphql +queryHotel(filter: { + area: { + contains: { + point: { + latitude: 0.5, + longitude: 2.5 + } + } + } +}) { + name +} +``` + +A `contains` example using `polygon`: + +```graphql + queryHotel(filter: { + area: { + contains: { + polygon: { + coordinates: [{ + points:[{ + latitude: 37.771935, + longitude: -122.469829 + }] + }], + } + } + } +}) { + name +} +``` + +#### intersects + +The `intersects` filter matches all entities where the `Polygon` or `MultiPolygon` field intersects another given `polygon` or `multiPolygon`. + +:::tip +Only one `polygon` or `multiPolygon` can be given inside the `IntersectsFilter` at a time. +::: + +```graphql + queryHotel(filter: { + area: { + intersects: { + multiPolygon: { + polygons: [{ + coordinates: [{ + points: [{ + latitude: 11.11, + longitude: 22.22 + }, { + latitude: 15.15, + longitude: 16.16 + }, { + latitude: 20.20, + longitude: 21.21 + }, { + latitude: 11.11, + longitude: 22.22 + }] + }, { + points: [{ + latitude: 11.18, + longitude: 22.28 + }, { + latitude: 15.18, + longitude: 16.18 + }, { + latitude: 20.28, + longitude: 21.28 + }, { + latitude: 11.18, + longitude: 22.28 + }] + }] + }, { + coordinates: [{ + points: [{ + latitude: 91.11, + longitude: 92.22 + }, { + latitude: 15.15, + longitude: 16.16 + }, { + latitude: 20.20, + longitude: 21.21 + }, { + latitude: 91.11, + longitude: 92.22 + }] + }, { + points: [{ + latitude: 11.18, + longitude: 22.28 + }, { + latitude: 15.18, + longitude: 16.18 + }, { + latitude: 20.28, + longitude: 21.28 + }, { + latitude: 11.18, + longitude: 22.28 + }] + }] + }] + } + } + } + }) { + name + } +``` + +### Union + +Unions can be queried only as a field of a type. Union queries can't be ordered, but you can filter and paginate them. + +:::note +Union queries do not support the `order` argument. +The results will be ordered by the `uid` of each node in ascending order. +::: + +For example, the following schema will enable to query the `members` union field in the `Home` type with filters and pagination. + +```graphql +union HomeMember = Dog | Parrot | Human + +type Home { + id: ID! + address: String + + members(filter: HomeMemberFilter, first: Int, offset: Int): [HomeMember] +} + +# Not specifying a field in the filter input will be considered as a null value for that field. +input HomeMemberFilter { + # `homeMemberTypes` is used to specify which types to report back. + homeMemberTypes: [HomeMemberType] + + # specifying a null value for this field means query all dogs + dogFilter: DogFilter + + # specifying a null value for this field means query all parrots + parrotFilter: ParrotFilter + # note that there is no HumanFilter because the Human type wasn't filterable +} + +enum HomeMemberType { + dog + parrot + human +} + +input DogFilter { + id: [ID!] + category: Category_hash + breed: StringTermFilter + and: DogFilter + or: DogFilter + not: DogFilter +} + +input ParrotFilter { + id: [ID!] + category: Category_hash + and: ParrotFilter + or: ParrotFilter + not: ParrotFilter +} +``` + +:::tip +Not specifying any filter at all or specifying any of the `null` values for a filter will query all members. +::: + + + +The same example, but this time with filter and pagination arguments: + +```graphql +query { + queryHome { + address + members ( + filter: { + homeMemberTypes: [dog, parrot] # means we don't want to query humans + dogFilter: { + # means in Dogs, we only want to query "German Shepherd" breed + breed: { allofterms: "German Shepherd"} + } + # not specifying any filter for parrots means we want to query all parrots + } + first: 5 + offset: 10 + ) { + ... on Animal { + category + } + ... on Dog { + breed + } + ... on Parrot { + repeatsWords + } + ... on HomeMember { + name + } + } + } +} +``` + +### Vector embedding + +The `@search` directive is used in conjunction with `@embeding` directive to define the HNSW index on vector embeddings. These vector embeddings are obtained from external Machine Learning models. + +```graphql +type User { + userID: ID! + name: String! + name_v: [Float!] @embedding @search(by: ["hnsw(metric: euclidean, exponent: 4)"]) +} +``` + +In this schema, the field `name_v` is an embedding on which the HNSW algorithm is used to create a vector search index. + +The metric used to compute the distance between vectors (in this example) is Euclidean distance. Other possible metrics are `cosine` and `dotproduct`. + +The directive, `@embedding`, designates one or more fields as vector embeddings. + +The `exponent` value is used to set reasonable defaults for HNSW internal tuning parameters. It is an integer representing an approximate number for the vectors expected in the index, in terms of power of 10. Default is “4” (10^4 vectors). \ No newline at end of file diff --git a/docusaurus-docs/graphql_versioned_docs/version-v25.4/schema/documentation.mdx b/docusaurus-docs/graphql_versioned_docs/version-v25.4/schema/documentation.mdx new file mode 100644 index 00000000..b69673f2 --- /dev/null +++ b/docusaurus-docs/graphql_versioned_docs/version-v25.4/schema/documentation.mdx @@ -0,0 +1,61 @@ +--- +title: "Documentation and Comments" +description: "Dgraph accepts GraphQL documentation comments, which get passed through to the generated API and shown as documentation in GraphQL tools." + +--- + +## Schema Documentation Processed by Generated API +Dgraph accepts GraphQL documentation comments (e.g. `""" This is a graphql comment """`), which get passed through to the generated API and thus shown as documentation in GraphQL tools like GraphiQL, GraphQL Playground, Insomnia etc. + +## Schema Documentation Ignored by Generated API +You can also add `# ...` comments where ever you like. These comments are not passed via the generated API and are not visible in the API docs. + +## Reserved Namespace in Dgraph +Any comment starting with `# Dgraph.` is **reserved** and **should not be used** to document your input schema. + +## An Example +An example that adds comments to a type as well as fields within the type would be as below. + +```graphql +""" +Author of questions and answers in a website +""" +type Author { +# ... username is the author name , this is an example of a dropped comment + username: String! @id +""" +The questions submitted by this author +""" + questions: [Question] @hasInverse(field: author) +""" +The answers submitted by this author +""" + answers: [Answer] @hasInverse(field: author) +} +``` + +It is also possible to add comments for queries or mutations that have been added via the custom directive. +```graphql +type Query { +""" +This query involves a custom directive, and gets top authors. +""" +getTopAuthors(id: ID!): [Author] @custom(http: { + url: "http://api.github.com/topAuthors", + method: "POST", + introspectionHeaders: ["Github-Api-Token"], + secretHeaders: ["Authorization:Github-Api-Token"] + }) +} +``` +The screenshots below shows how the documentation appear in a Graphql API explorer. + + +Schema Documentation on Types +![Schema Documentation On Types](/images/graphql/authors1.png) + + +Schema Documentation on Custom directive +![Schema Documentation On Custom Directive](/images/graphql/CustomDirectiveDocumentation.png) + + diff --git a/docusaurus-docs/graphql_versioned_docs/version-v25.4/schema/graph-links.md b/docusaurus-docs/graphql_versioned_docs/version-v25.4/schema/graph-links.md new file mode 100644 index 00000000..da675882 --- /dev/null +++ b/docusaurus-docs/graphql_versioned_docs/version-v25.4/schema/graph-links.md @@ -0,0 +1,110 @@ +--- +title: "Relationships" +description: "All the data in your app form a GraphQL data graph. That graph has nodes of particular types and relationships between the nodes to form the data graph." + +--- + +All the data in your app form a GraphQL data graph. That graph has nodes of particular types and relationships between the nodes to form the data graph. + +Dgraph uses the types and fields in the schema to work out how to link that graph, what to accept for mutations and what shape responses should take. + +Relationships in that graph are directed: either pointing in one direction or two. You use the `@hasInverse` directive to tell Dgraph how to handle two-way relationship. + +### One-way relationship + +If you only ever need to traverse the graph between nodes in a particular direction, then your schema can simply contain the types and the relationship. + +In this schema, posts have an author - each post in the graph is linked to its author - but that relationship is one-way. + +```graphql +type Author { + ... +} + +type Post { + ... + author: Author +} +``` + +You'll be able to traverse the graph from a Post to its author, but not able to traverse from an author to all their posts. Sometimes that's the right choice, but mostly, you'll want two way relationships. + +Note: Dgraph won't store the reverse direction, so if you change your schema to include a `@hasInverse`, you'll need to migrate the data to add the reverse edges. + +### Two-way relationship + + +In Dgraph, the directive `@hasInverse` is used to create a two-way relationship. + +```graphql +type Author { + ... + posts: [Post] @hasInverse(field: author) +} + +type Post { + ... + author: Author +} +``` + +With that, `posts` and `author` are just two directions of the same link in the graph. For example, adding a new post with + +```graphql +mutation { + addPost(input: [ + { ..., author: { username: "diggy" }} + ]) { + ... + } +} +``` + +will automatically add it to Diggy's list of `posts`. Deleting the post will remove it from Diggy's `posts`. Similarly, using an update mutation on an author to insert a new post will automatically add Diggy as the author + +```graphql +mutation { + updateAuthor(input: { + filter: { username: { eq: "diggy "}}, + set: { posts: [ {... new post ...}]} + }) { + ... + } +} +``` + +### Many edges + +It's not really possible to auto-detect what a schema designer meant for two-way edges. There's not even only one possible relationship between two types. Consider, for example, if an app recorded the posts an `Author` had recently liked (so it can suggest interesting material) and just a tally of all likes on a post. + +```graphql +type Author { + ... + posts: [Post] + recentlyLiked: [Post] +} + +type Post { + ... + author: Author + numLikes: Int +} +``` + +It's not possible to detect what is meant here as a one-way edge, or which edges are linked as a two-way connection. That's why `@hasInverse` is needed - so you can enforce the semantics your app needs. + +```graphql +type Author { + ... + posts: [Post] @hasInverse(field: author) + recentlyLiked: [Post] +} + +type Post { + ... + author: Author + numLikes: Int +} +``` + +Now, Dgraph will manage the connection between posts and authors and you can get on with concentrating on what your app needs to to - suggesting them interesting content. diff --git a/docusaurus-docs/graphql_versioned_docs/version-v25.4/schema/index.md b/docusaurus-docs/graphql_versioned_docs/version-v25.4/schema/index.md new file mode 100644 index 00000000..3c6dec89 --- /dev/null +++ b/docusaurus-docs/graphql_versioned_docs/version-v25.4/schema/index.md @@ -0,0 +1,12 @@ +--- +title: "Schema" + +--- + +This section describes all the things you can put in your input GraphQL schema, and what gets generated from that. + +The process for serving GraphQL with Dgraph is to add a set of GraphQL type definitions using the `/admin` endpoint. Dgraph takes those definitions, generates queries and mutations, and serves the generated GraphQL schema. + +The input schema may contain interfaces, types and enums that follow the usual GraphQL syntax and validation rules. + +If you want to make your schema editing experience nicer, you should use an editor that does syntax highlighting for GraphQL. With that, you may also want to include the definitions [here](/graphql/schema/dgraph-schema) as an import. diff --git a/docusaurus-docs/graphql_versioned_docs/version-v25.4/schema/migration.md b/docusaurus-docs/graphql_versioned_docs/version-v25.4/schema/migration.md new file mode 100644 index 00000000..480b4cad --- /dev/null +++ b/docusaurus-docs/graphql_versioned_docs/version-v25.4/schema/migration.md @@ -0,0 +1,215 @@ +--- +title: "Schema Migration" +description: "This document describes all the things that you need to take care while doing a schema update or migration." + +--- + +In every app's development lifecycle, there's a point where the underlying schema doesn't fit the requirements and must be changed for good. +That requires a migration for both schema and the underlying data. +This article will guide you through common migration scenarios you can encounter with Dgraph and help you avoid any pitfalls around them. + +These are the most common scenarios that can occur: +* Renaming a type +* Renaming a field +* Changing a field's type +* Adding `@id` to an existing field + +:::note +As long as you can avoid migration, avoid it. +Because there can be scenarios where you might need to update downstream clients, which can be hard. +So, its always best to try out things first, once you are confident enough, then only push them to +production. +::: + +### Renaming a type + +Let's say you had the following schema: + +```graphql +type User { + id: ID! + name: String +} +``` + +and you had your application working fine with it. Now, you feel that the name `AppUser` would be +more sensible than the name `User` because `User` seems a bit generic to you. Then you are in a +situation where you need migration. + +This can be handled in a couple of ways: +1. Migrate all the data for type `User` to use the new name `AppUser`. OR, +2. Just use the [`@dgraph(type: ...)`](directives/directive-dgraph) directive to maintain backward compatibility + with the existing data. + +Depending on your use-case, you might find option 1 or 2 better for you. For example, if you +have accumulated very little data for the `User` type till now, then you might want to go with +option #1. But, if you have an active application with a very large dataset then updating the +node of each user may not be a thing you might want to commit to, as that can require some +maintenance downtime. So, option #2 could be a better choice in such conditions. + +Option #2 makes your new schema compatible with your existing data. Here's an example: + +```graphql +type AppUser @dgraph(type: "User") { + id: ID! + name: String +} +``` + +So, no downtime required. Migration is done by just updating your schema. Fast, easy, and simple. + +Note that, irrespective of what option you choose for migration on Dgraph side, you will still +need to migrate your GraphQL clients to use the new name in queries/mutations. For example, the +query `getUser` would now be renamed to `getAppUser`. So, your downstream clients need to update +that bit in the code. + +### Renaming a field + +Just like renaming a type, let's say you had the following working schema: + +```graphql +type User { + id: ID! + name: String + phone: String +} +``` + +and now you figured that it would be better to call `phone` as `tel`. You need migration. + +You have the same two choices as before: +1. Migrate all the data for the field `phone` to use the new name `tel`. OR, +2. Just use the [`@dgraph(pred: ...)`](directives/directive-dgraph) directive to maintain backward compatibility + with the existing data. + +Here's an example if you want to go with option #2: + +```graphql +type User { + id: ID! + name: String + tel: String @dgraph(pred: "User.phone") +} +``` + +Again, note that, irrespective of what option you choose for migration on Dgraph side, you will +still need to migrate your GraphQL clients to use the new name in queries/mutations. For example, +the following query: + +```graphql +query { + getUser(id: "0x05") { + name + phone + } +} +``` + +would now have to be changed to: + +```graphql +query { + getUser(id: "0x05") { + name + tel + } +} +``` + +So, your downstream clients need to update that bit in the code. + +### Changing a field's type + +There can be multiple scenarios in this category: +* List -> Single item +* `String` -> `Int` +* Any other combination you can imagine + +It is strictly advisable that you figure out a solid schema before going in production, so that +you don't have to deal with such cases later. Nevertheless, if you ended up in such a situation, you +have to migrate your data to fit the new schema. There is no easy way around here. + +An example scenario is, if you initially had this schema: + +```graphql +type Todo { + id: ID! + task: String + owner: Owner +} + +type Owner { + name: String! @id + todo: [Todo] @hasInverse(field:"owner") +} +``` + +and later you decided that you want an owner to have only one todo at a time. So, you want to +make your schema look like this: + +```graphql +type Todo { + id: ID! + task: String + owner: Owner +} + +type Owner { + name: String! @id + todo: Todo @hasInverse(field:"owner") +} +``` + +If you try updating your schema, you may end up getting an error like this: + +```txt +resolving updateGQLSchema failed because succeeded in saving GraphQL schema but failed to alter Dgraph schema - GraphQL layer may exhibit unexpected behavior, reapplying the old GraphQL schema may prevent any issues: Schema change not allowed from [uid] => uid without deleting pred: owner.todo +``` + +That is a red flag. As the error message says, you should revert to the old schema to make your +clients work correctly. In such cases, you should have migrated your data to fit the new schema +_before_ applying the new schema. The steps for such a data migration varies from case to case, +and so can't all be listed down here, but you need to migrate your data first, is all you need +to keep in mind while making such changes. + +### Adding `@id` to an existing field + +Let's say you had the following schema: + +```graphql +type User { + id: ID! + username: String +} +``` + +and now you think that `username` must be unique for every user. So, you change the schema to this: + +```graphql +type User { + id: ID! + username: String! @id +} +``` + +Now, here's the catch: with the old schema, it was possible that there could have existed +multiple users with the username `Alice`. If that was true, then the queries would break in such +cases. Like, if you run this query after the schema change: + +```graphql +query { + getUser(username: "Alice") { + id + } +} +``` + +Then it might error out saying: + +```txt +A list was returned, but GraphQL was expecting just one item. This indicates an internal error - probably a mismatch between the GraphQL and Dgraph/remote schemas. The value was resolved as null (which may trigger GraphQL error propagation) and as much other data as possible returned. +``` + +So, while making such a schema change, you need to make sure that the underlying data really +honors the uniqueness constraint on the username field. If not, you need to do a data migration +to honor such constraints. diff --git a/docusaurus-docs/graphql_versioned_docs/version-v25.4/schema/reserved.md b/docusaurus-docs/graphql_versioned_docs/version-v25.4/schema/reserved.md new file mode 100644 index 00000000..fea6d011 --- /dev/null +++ b/docusaurus-docs/graphql_versioned_docs/version-v25.4/schema/reserved.md @@ -0,0 +1,53 @@ +--- +title: "Reserved Names" +description: "This document provides the full list of names that are reserved and can’t be used to define any other identifiers." + +--- + +The following names are reserved and can't be used to define any other identifiers: + +- `Int` +- `Float` +- `Boolean` +- `String` +- `DateTime` +- `ID` +- `uid` +- `Subscription` +- `as` (case-insensitive) +- `Query` +- `Mutation` +- `Point` +- `PointList` +- `Polygon` +- `MultiPolygon` +- `Aggregate` (as a suffix of any identifier name) + + +For each type, Dgraph generates a number of GraphQL types needed to operate the GraphQL API, these generated type names also can't be present in the input schema. For example, for a type `Author`, Dgraph generates: + +- `AuthorFilter` +- `AuthorOrderable` +- `AuthorOrder` +- `AuthorRef` +- `AddAuthorInput` +- `UpdateAuthorInput` +- `AuthorPatch` +- `AddAuthorPayload` +- `DeleteAuthorPayload` +- `UpdateAuthorPayload` +- `AuthorAggregateResult` + +**Mutations** + +- `addAuthor` +- `updateAuthor` +- `deleteAuthor` + +**Queries** + +- `getAuthor` +- `queryAuthor` +- `aggregateAuthor` + +Thus if `Author` is present in the input schema, all of those become reserved type names. diff --git a/docusaurus-docs/graphql_versioned_docs/version-v25.4/schema/types.md b/docusaurus-docs/graphql_versioned_docs/version-v25.4/schema/types.md new file mode 100644 index 00000000..f038513f --- /dev/null +++ b/docusaurus-docs/graphql_versioned_docs/version-v25.4/schema/types.md @@ -0,0 +1,465 @@ +--- +title: "Types" +description: "How to use GraphQL types to set a GraphQL schema for the Dgraph database. Includes scalars, enums, types, interfaces, union, password, & geolocation types." + +--- + +This page describes how to use GraphQL types to set the a GraphQL schema for +Dgraph database. + +### Scalars + +Dgraph's GraphQL implementation comes with the standard GraphQL scalar types: +`Int`, `Float`, `String`, `Boolean` and `ID`. There's also an `Int64` scalar, +and a `DateTime` scalar type that is represented as a string in RFC3339 format. + +Scalar types, including `Int`, `Int64`, `Float`, `String` and `DateTime`; can be +used in lists. Lists behave like an unordered set in Dgraph. For example: +`["e1", "e1", "e2"]` may get stored as `["e2", "e1"]`, so duplicate values will +not be stored and order might not be preserved. All scalars may be nullable or +non-nullable. + +:::noteThe `Int64` type introduced in release v20.11 represents +a signed integer ranging between `-(2^63)` and `(2^63 -1)`. Signed `Int64` values +in this range will be parsed correctly by Dgraph as long as the client can +serialize the number correctly in JSON. For example, a JavaScript client might +need to use a serialization library such as +[`json-bigint`](https://www.npmjs.com/package/json-bigint) to correctly +write an `Int64` value in JSON.::: + +The `ID` type is special. IDs are auto-generated, immutable, and can be treated as strings. Fields of type `ID` can be listed as nullable in a schema, but Dgraph will never return null. + +* *Schema rule*: `ID` lists aren't allowed - e.g. `tags: [String]` is valid, but `ids: [ID]` is not. +* *Schema rule*: Each type you define can have at most one field with type `ID`. That includes IDs implemented through interfaces. + +It's not possible to define further scalars - you'll receive an error if the input schema contains the definition of a new scalar. + +For example, the following GraphQL type uses all of the available scalars. + +```graphql +type User { + userID: ID! + name: String! + lastSignIn: DateTime + recentScores: [Float] + reputation: Int + active: Boolean +} +``` + +Scalar lists in Dgraph act more like sets, so `tags: [String]` would always contain unique tags. Similarly, `recentScores: [Float]` could never contain duplicate scores. + +### Vectors + +A Float array can be used as a vector using `@embedding` directive. It denotes a vector of floating point numbers, i.e an ordered array of float32. A type can contain more than one vector predicate. + +Vectors are normaly used to store embeddings obtained from an ML model. + +When a Float vector is indexed, the GraphQL `querySimilarByEmbedding` and `querySimilarById` functions can be used for [similarity search](/graphql/queries/vector-similarity). + +A simple example of adding a vector embedding on `name` to `User` type is shown below. + +```graphql +type User { + userID: ID! + name: String! + name_v: [Float!] @embedding @search(by: ["hnsw(metric: euclidean, exponent: 4)"]) +} +``` + +In this schema, the field `name_v` is an embedding on which the [@search ](/graphql/schema/directives/search/#vector-embedding) directive for vector embeddings is used. +For a full description of the supported arguments to the embedding index definition, see [this document](/dql/predicate-indexing#vector-indices). + +### The `ID` type + +In Dgraph, every node has a unique 64-bit identifier that you can expose in GraphQL using the `ID` type. An `ID` is auto-generated, immutable and never reused. Each type can have at most one `ID` field. + +The `ID` type works great when you need to use an identifier on nodes and don't need to set that identifier externally (for example, posts and comments). + +For example, you might set the following type in a schema: + +```graphql +type Post { + id: ID! + ... +} +``` + +In a single-page app, you could generate the page for `http://.../posts/0x123` when a user clicks to view the post with `ID` 0x123. Your app can then use a `getPost(id: "0x123") { ... }` GraphQL query to fetch the data used to generate the page. + +For input and output, `ID`s are treated as strings. + +You can also update and delete posts by `ID`. + +### Enums + +You can define enums in your input schema. For example: + +```graphql +enum Tag { + GraphQL + Database + Question + ... +} + +type Post { + ... + tags: [Tag!]! +} +``` + +### Types + +From the built-in scalars and the enums you add, you can generate types in the usual way for GraphQL. For example: + +```graphql +enum Tag { + GraphQL + Database + Dgraph +} + +type Post { + id: ID! + title: String! + text: String + datePublished: DateTime + tags: [Tag!]! + author: Author! +} + +type Author { + id: ID! + name: String! + posts: [Post!] + friends: [Author] +} +``` + +* *Schema rule*: Lists of lists aren't accepted. For example: `multiTags: [[Tag!]]` isn't valid. +* *Schema rule*: Fields with arguments are not accepted in the input schema unless the field is implemented using the `@custom` directive. + +### Interfaces + +GraphQL interfaces allow you to define a generic pattern that multiple types follow. When a type implements an interface, that means it has all fields of the interface and some extras. + +According to GraphQL specifications, you can have the same fields in implementing types as the interface. In such cases, the GraphQL layer will generate the correct Dgraph schema without duplicate fields. + +If you repeat a field name in a type, it must be of the same type (including list or scalar types), and it must have the same nullable condition as the interface's field. Note that if the interface's field has a directive like `@search` then it will be inherited by the implementing type's field. + +For example: + +```graphql +interface Fruit { + id: ID! + price: Int! +} + +type Apple implements Fruit { + id: ID! + price: Int! + color: String! +} + +type Banana implements Fruit { + id: ID! + price: Int! +} +``` + +:::tip +GraphQL will generate the correct Dgraph schema where fields occur only once. +::: + +The following example defines the schema for posts with comment threads. As mentioned, Dgraph will fill in the `Question` and `Comment` types to make the full GraphQL types. + +```graphql +interface Post { + id: ID! + text: String + datePublished: DateTime +} + +type Question implements Post { + title: String! +} +type Comment implements Post { + commentsOn: Post! +} +``` + +The generated schema will contain the full types, for example, `Question` and `Comment` get expanded as: + +```graphql +type Question implements Post { + id: ID! + text: String + datePublished: DateTime + title: String! +} + +type Comment implements Post { + id: ID! + text: String + datePublished: DateTime + commentsOn: Post! +} +``` + +:::note +If you have a type that implements two interfaces, Dgraph won't allow a field of the same name in both interfaces, except for the `ID` field. +::: + +Dgraph currently allows this behavior for `ID` type fields since the `ID` type field is not a predicate. Note that in both interfaces and the implementing type, the nullable condition and type (list or scalar) for the `ID` field should be the same. For example: + +```graphql +interface Shape { + id: ID! + shape: String! +} + +interface Color { + id: ID! + color: String! +} + +type Figure implements Shape & Color { + id: ID! + shape: String! + color: String! + size: Int! +} +``` + +### Union type + +GraphQL Unions represent an object that could be one of a list of GraphQL Object types, but provides for no guaranteed fields between those types. So no fields may be queried on this type without the use of type refining fragments or inline fragments. + +Union types have the potential to be invalid if incorrectly defined: + +- A `Union` type must include one or more unique member types. +- The member types of a `Union` type must all be Object base types; [Scalar](#scalars), [Interface](#interfaces) and `Union` types must not be member types of a Union. Similarly, wrapping types must not be member types of a Union. + + +For example, the following defines the `HomeMember` union type: + +```graphql +enum Category { + Fish + Amphibian + Reptile + Bird + Mammal + InVertebrate +} + +interface Animal { + id: ID! + category: Category @search +} + +type Dog implements Animal { + breed: String @search +} + +type Parrot implements Animal { + repeatsWords: [String] +} + +type Cheetah implements Animal { + speed: Float +} + +type Human { + name: String! + pets: [Animal!]! +} + +union HomeMember = Dog | Parrot | Human + +type Zoo { + id: ID! + animals: [Animal] + city: String +} + +type Home { + id: ID! + address: String + members: [HomeMember] +} +``` + +So, when you want to query members in a `Home`, you will be able to do a GraphQL query like this: + +```graphql +query { + queryHome { + address + members { + ... on Animal { + category + } + ... on Dog { + breed + } + ... on Parrot { + repeatsWords + } + ... on Human { + name + } + } + } +} +``` + +And the results of the GraphQL query will look like the following: + +```json +{ + "data": { + "queryHome": { + "address": "Earth", + "members": [ + { + "category": "Mammal", + "breed": "German Shepherd" + }, { + "category": "Bird", + "repeatsWords": ["Good Morning!", "I am a GraphQL parrot"] + }, { + "name": "Alice" + } + ] + } + } +} +``` + +### Password type + +A password for an entity is set with setting the schema for the node type with `@secret` directive. Passwords cannot be queried directly, only checked for a match using the `checkTypePassword` function where `Type` is the node type. +The passwords are encrypted using [Bcrypt](https://en.wikipedia.org/wiki/Bcrypt). + +:::note +For security reasons, Dgraph enforces a minimum password length of 6 characters on `@secret` fields. +::: + +For example, to set a password, first set schema: + +1. Cut-and-paste the following schema into a file called `schema.graphql` + ```graphql + type Author @secret(field: "pwd") { + name: String! @id + } + ``` + +2. Run the following curl request: + ```bash + curl -X POST localhost:8080/admin/schema --data-binary '@schema.graphql' + ``` + +3. Set the password by pointing to the `graphql` endpoint (http://localhost:8080/graphql): + ```graphql + mutation { + addAuthor(input: [{name:"myname", pwd:"mypassword"}]) { + author { + name + } + } + } + ``` + +The output should look like: +```json +{ + "data": { + "addAuthor": { + "author": [ + { + "name": "myname" + } + ] + } + } +} +``` + +You can check a password: +```graphql +query { + checkAuthorPassword(name: "myname", pwd: "mypassword") { + name + } +} +``` + +output: +```json +{ + "data": { + "checkAuthorPassword": { + "name": "myname" + } + } +} +``` + +If the password is wrong you will get the following response: +```json +{ + "data": { + "checkAuthorPassword": null + } +} +``` + +### Geolocation types + +Dgraph GraphQL comes with built-in types to store Geolocation data. Currently, it supports `Point`, `Polygon` and `MultiPolygon`. These types are useful in scenarios like storing a location's GPS coordinates, representing a city on the map, etc. + +For example: + +```graphql +type Hotel { + id: ID! + name: String! + location: Point + area: Polygon +} +``` + +#### Point + +```graphql +type Point { + longitude: Float! + latitude: Float! +} +``` + +#### PointList + +```graphql +type PointList { + points: [Point!]! +} +``` + +#### Polygon + +```graphql +type Polygon { + coordinates: [PointList!]! +} +``` + +#### MultiPolygon + +```graphql +type MultiPolygon { + polygons: [Polygon!]! +} +``` diff --git a/docusaurus-docs/graphql_versioned_docs/version-v25.4/security/RBAC-rules.md b/docusaurus-docs/graphql_versioned_docs/version-v25.4/security/RBAC-rules.md new file mode 100644 index 00000000..6667ea48 --- /dev/null +++ b/docusaurus-docs/graphql_versioned_docs/version-v25.4/security/RBAC-rules.md @@ -0,0 +1,121 @@ +--- +title: "RBAC rules" +description: "Dgraph support Role Based Access Control (RBAC) on GraphQL API operations." + +--- + +Dgraph support Role Based Access Control (RBAC) on GraphQL API operations: you can specify who can invoke query, add, update and delete operations on each type of your GraphQL schema based on JWT claims, using the ``@auth`` directive. + + +To implement Role Based Access Control on GraphQL API operations : +1. Ensure your have configured the GraphQL schema to [Handle JWT tokens](/graphql/security/jwt) using ``# Dgraph.Authorization`` + This step is important to be able to use the [JWT claims](/graphql/security/#jwt-claims) +2. Annotate the Types in the GraphQL schema with the `@auth` directive and specify conditions to be met for `query`, `add`, `update` or `delete` operations. +3. Deploy the GraphQL schema either with a [schema update](/graphql/admin/#using-updategqlschema-to-add-or-modify-a-schema) or via the Cloud console's [Schema](https://cloud.dgraph.io/_/schema) page. + + + +The generic format of RBAC rule is as follow +```graphql +type User @auth( + query: { rule: "{$: { eq: \"\" } }" }, + add: { rule: "{$: { in: [\"\",...] } }" }, + update: ... + delete: ... +) +``` +RBAC rule supports ``eq`` or ``in`` functions to test the value of a [JWT claim](/graphql/security/#jwt-claims) from the JWT token payload. + +The claim value may be a string or array of strings. + +For example the following schema has a @auth directive specifying that a delete operation on a User object can only be done if the connected user has a 'ROLE' claim in the JWT token with the value "admin" : +```graphql +type User @auth( + delete: { rule: "{$ROLE: { eq: \"admin\" } }"} + ) { + username: String! + @id todos: [Todo] +} +``` +The following JWT token payload will pass the test (provided that Dgraph.Authorization is configured correctly with the right namespace) +```json +{ + "aud": "dgraph", + "exp": 1695359621, + "https://dgraph.io/jwt/claims": { + "ROLE": "admin", + "USERID": "testuser@dgraph.io" + }, + "iat": 1695359591, + ... +} +``` +The rule is also working with an array of roles in the JWT token: +```json +{ + "aud": "dgraph", + "exp": 1695359621, + "https://dgraph.io/jwt/claims": { + "ROLE": ["admin","user"] + "USERID": "testuser@dgraph.io" + }, + "iat": 1695359591, + ... +} +``` +In the case of an array used with the "in" function, the rule is valid is at least one of the claim value is "in" the provided list. + +For example, with the following rule, the previous token will be valid because one of the ROLE is in the authorized roles. +```graphql +type User @auth( + delete: { rule: "{$ROLE: { in: [\"admin\",\"superadmin\"] } }"} + ) { + username: String! + @id todos: [Todo] +} +``` + +## rules combination + +Rules can be combined with the logical connectives ``and``, ``or`` and ``not``. +A permission can be a mixture of graph traversals and role based rules. + +In the todo app, you can express, for example, that you can delete a `Todo` if you are the author, or are the site admin. + +```graphql +type Todo @auth( + delete: { or: [ + { rule: "query ($USER: String!) { ... }" }, # you are the author graph query + { rule: "{$ROLE: { eq: \"ADMIN\" } }" } + ]} +) +``` + + +## claims + +Rules may use claims from the namespace specified by the [# Dgraph.Authorization](/graphql/security/jwt) or claims present at the root level of the JWT payload. + +For example, given the following JWT payload + +```json +{ + "https://xyz.io/jwt/claims": [ + "ROLE": "ADMIN" + ], + "email": "random@example.com" +} +``` + +If `https://xyz.io/jwt/claims` is declared as the namespace to use, the authorization rules can use ``$ROLE`` but also ``$email``. + +In cases where the same claim is present in the namespace and at the root level, the claim value in the namespace takes precedence. + +## `@auth` on Interfaces + +The rules provided inside the `@auth` directive on an interface will be applied as an `AND` rule to those on the implementing types. + +A type inherits the `@auth` rules of all the implemented interfaces. The final authorization rule is an `AND` of the type's `@auth` rule and of all the implemented interfaces. + + + diff --git a/docusaurus-docs/graphql_versioned_docs/version-v25.4/security/auth-tips.md b/docusaurus-docs/graphql_versioned_docs/version-v25.4/security/auth-tips.md new file mode 100644 index 00000000..d6388762 --- /dev/null +++ b/docusaurus-docs/graphql_versioned_docs/version-v25.4/security/auth-tips.md @@ -0,0 +1,79 @@ +--- +title: "Authorization tips" +description: "Given an authentication mechanism and a signed JSON Web Token (JWT), the @auth directive tells Dgraph how to apply authorization." + +--- + +## Public Data + +Many apps have data that can be accessed by anyone, logged in or not. That also works nicely with Dgraph auth rules. + +For example, in Twitter, StackOverflow, etc. you can see authors and posts without being signed it - but you'd need to be signed in to add a post. With Dgraph auth rules, if a type doesn't have, for example, a `query` auth rule or the auth rule doesn't depend on a JWT value, then the data can be accessed without a signed JWT. + +For example, the todo app might allow anyone, logged in or not, to view any author, but not make any mutations unless logged in as the author or an admin. That would be achieved by rules like the following. + +```graphql +type User @auth( + # no query rule + add: { rule: "{$ROLE: { eq: \"ADMIN\" } }" }, + update: ... + delete: ... +) { + username: String! @id + todos: [Todo] +} +``` + +Maybe some todos can be marked as public and users you aren't logged in can see those. + +```graphql +type Todo @auth( + query: { or: [ + # you are the author + { rule: ... }, + # or, the todo is marked as public + { rule: """query { + queryTodo(filter: { isPublic: { eq: true } } ) { + id + } + }"""} + ]} +) { + ... + isPublic: Boolean +} + +``` + +Because the rule doesn't depend on a JWT value, it can be successfully evaluated for users who aren't logged in. + +Ensuring that requests are from an authenticated JWT, and no further restrictions, can be done by arranging the JWT to contain a value like `"isAuthenticated": "true"`. For example, + + +```graphql +type User @auth( + query: { rule: "{$isAuthenticated: { eq: \"true\" } }" }, +) { + username: String! @id + todos: [Todo] +} +``` + +specifies that only authenticated users can query other users. + +### blocking an operation of everyone + +If the `ROLE` claim isn't present in a JWT, any rule that relies on `ROLE` simply evaluates to false. + +You can also simply disallow some queries and mutations by using a condition on a non-existing claim: + +If you know that your JWTs never contain the claim `DENIED`, then a rule such as + +```graphql +type User @auth( + delete: { rule: "{$DENIED: { eq: \"DENIED\" } }"} +) { + ... +} +``` +will block the delete operation for everyone. \ No newline at end of file diff --git a/docusaurus-docs/graphql_versioned_docs/version-v25.4/security/cors.md b/docusaurus-docs/graphql_versioned_docs/version-v25.4/security/cors.md new file mode 100644 index 00000000..88c8df14 --- /dev/null +++ b/docusaurus-docs/graphql_versioned_docs/version-v25.4/security/cors.md @@ -0,0 +1,24 @@ +--- +title: "Restrict origins" + +--- + +To restrict origins of HTTP requests : + +1. Add lines starting with `# Dgraph.Allow-Origin` at the end of your GraphQL schema specifying the origins allowed. +2. Deploy the GraphQL schema either with a [schema update](/graphql/admin/#using-updategqlschema-to-add-or-modify-a-schema) or via the Cloud console's [Schema](https://cloud.dgraph.io/_/schema) page. + +For example, the following will restrict all origins except the ones specified. + +``` +# Dgraph.Allow-Origin "https://example.com" +# Dgraph.Allow-Origin "https://www.example.com" +``` + + +`https://cloud.dgraph.io` is always allowed so that ``API explorer``, in Dgraph Cloud console, continues to work. + +:::note +- CORS restrictions only apply to browsers. +- By default, ``/graphql`` endpoint does not limit the request origin (`Access-Control-Allow-Origin: *`). +::: \ No newline at end of file diff --git a/docusaurus-docs/graphql_versioned_docs/version-v25.4/security/graphtraversal-rules.md b/docusaurus-docs/graphql_versioned_docs/version-v25.4/security/graphtraversal-rules.md new file mode 100644 index 00000000..6febc4cf --- /dev/null +++ b/docusaurus-docs/graphql_versioned_docs/version-v25.4/security/graphtraversal-rules.md @@ -0,0 +1,140 @@ +--- +title: "ABAC rules" +description: "Dgraph support Attribute Based Access Control (ABAC) on GraphQL API operations: you can specify which data a user can query, add, update or delete for each type of your GraphQL schema based on JWT claims, using the ``@auth`` directive and graph traversal queries." + +--- + +Dgraph support Attribute Based Access Control (ABAC) on GraphQL API operations: you can specify which data a user can query, add, update or delete for each type of your GraphQL schema based on JWT claims, using the ``@auth`` directive and graph traversal queries. + + +To implement graph traversal rule on GraphQL API operations : +1. Ensure your have configured the GraphQL schema to [Handle JWT tokens](/graphql/security/jwt) using ``# Dgraph.Authorization`` + This step is important to be able to use the [JWT claims](/graphql/security/#jwt-claims) +2. Annotate the Types in the GraphQL schema with the `@auth` directive and specify conditions to be met for `query`, `add`, `update` or `delete` operations. +3. Deploy the GraphQL schema either with a [schema update](/graphql/admin/#using-updategqlschema-to-add-or-modify-a-schema) or via the Cloud console's [Schema](https://cloud.dgraph.io/_/schema) page. + + +A graph traversal rule is expressed as GraphQL query on the type on which the @auth directive applies. + +For example, a rule on ``Contact`` type can only use a ``queryContact`` query : + +```graphql +type Contact @auth( + query: { rule: "query { queryContact(filter: { isPublic: true }) { id } }" }, + add: ... + update: ... + delete: ... +) { + + ... +} +``` + +You can use triple quotation marks. In that case the query can be defined on multiple lines. + +The following schema is also valid: +```graphql +type Contact @auth( + query: { rule: """query { + queryContact(filter: { isPublic: true }) { + id + } + } """ +}) { + + ... +} +``` + +The rules are expressed as GraphQL queries, so they can also have a name and parameters: +```graphql +type Todo @auth( + query: { rule: """ + query ($USER: String!) { + queryTodo(filter: { owner: { eq: $USER } } ) { + id + } + }""" + } +){ + id: ID! + text: String! @search(by: [term]) + owner: String! @search(by: [hash]) +} +``` + +The parameters are replaced at runtime by the corresponding ``claims`` found in the JWT token. In the previous case, the query will be executed with the value of the `USER` claim. + +When a user sends a request on `/graphql` endpoint for a `get` or `query` operation, Dgraph executes the query specified in the @auth directive of the `Type` to build a list of "authorized" UIDs. Dgraph returns only the data matching both the requested data and the "authorized" list. That means that the client can apply any filter condition, the result will be the intersection of the data matching the filter and the "authorized" data. + +The same logic applies for update<Type> and delete<Type>: only the data matching the @auth query are affected. +```graphql +type Todo @auth( + delete: { or: [ + { rule: """query ($USER: String!) { + queryTodo(filter: { owner: { eq: $USER } } ) { + __typename + } + } """ + }, # you are the author graph query + { rule: "{$ROLE: { eq: \"ADMIN\" } }" } + ]} +) +``` + +In the context of @auth directive, Dgraph executes the @auth query differently that a normal query : if the query has nested blocks, all levels must match existing data. Dgraph internally applies a `@cascade` directive, making the directive more like a **pattern matching** condition. + +For example, in the cases of `Todo`, the access will depend not on a value in the todo, but on checking which owner it's linked to. +This means our auth rule must make a step further into the graph to check who the owner is : + +```graphql +type User { + username: String! @id + todos: [Todo] +} + +type Todo @auth( + query: { rule: """ + query ($USER: String!) { + queryTodo { + owner(filter: { username: { eq: $USER } } ) { + __typename + } + } + }""" + } +){ + id: ID! + text: String! + owner: User +} +``` + +The @auth query rule will only return ``Todos`` having an owner matching the condition: the owner ``username`` must be equal the the JWT claim ``USER``. + +All blocks must return some data for the query to succeed. You may want to use the field `__typename` in the most inner block to ensure a data match at this level. + + +### rules combination + +Rules can be combined with the logical connectives ``and``, ``or`` and ``not``. +A permission can be a mixture of graph traversals and role based rules. + +### `@auth` on Interfaces + +The rules provided inside the `@auth` directive on an interface will be applied as an `AND` rule to those on the implementing types. + +A type inherits the `@auth` rules of all the implemented interfaces. The final authorization rule is an `AND` of the type's `@auth` rule and of all the implemented interfaces. + +### claims + +Rules may use claims from the namespace specified by the [# Dgraph.Authorization](/graphql/security/jwt) or claims present at the root level of the JWT payload. + +### error handling + +When deploying the schema, Dgraph tests if you are using valid queries in your @auth directive. + +For example, using ``queryFilm`` for a rule on a type ``Actor`` will lead to an error: +``` +resolving updateGQLSchema failed because Type Actor: @auth: expected only queryActor rules,but found queryFilm +``` \ No newline at end of file diff --git a/docusaurus-docs/graphql_versioned_docs/version-v25.4/security/index.md b/docusaurus-docs/graphql_versioned_docs/version-v25.4/security/index.md new file mode 100644 index 00000000..9c74f7eb --- /dev/null +++ b/docusaurus-docs/graphql_versioned_docs/version-v25.4/security/index.md @@ -0,0 +1,81 @@ +--- +title: "Security" +description: "Dgraph's GraphQL implementation comes with built-in authorization, and supports various authentication methods, so you can annotate your schema with rules that determine who can access or mutate the data." + +--- + +When you deploy a GraphQL schema, Dgraph automatically generates the query and mutation operations for each type and exposes them as a GraphQL API on the ``/graphql`` endpoint. + + +Dgraph's GraphQL authorization features let you specify : +- if the client requires an API key or notif **anonymous access** is allowed to invoke a specific operation of the API. +- if a client must present an identity in the form of a **JWT token** to use the API. +- **RBAC rules** (Role Based Access Control) at operation level based on the claims included in the client JWT token. +- **ABAC rules** (Attribute Based Access COntrol) at data level using graph traversal queries. + + +:::note +By default all operations are accessible to anonymous clients, no JWT token is required and no authorization rules are applied. +It is your responsibility to correctly configure the authorization for the ``/graphql`` endpoint. +::: + +Refer to the following documentation to set your ``/graphql`` endpoint security : + +- [Handle JWT token](/graphql/security/jwt) + +- [RBAC rules](/graphql/security/RBAC-rules) + +- [ABAC rules](/graphql/security/graphtraversal-rules) + +### ``/graphql`` security flow +In summary, the Dgraph security flow on ``/graphql`` endpoint is as follow: + +![graphql endpoint security](/images/graphql/RBAC.jpeg) + +### CORS +Additionally, you can [restrict the origins](/graphql/security/cors) that ``/graphql`` endpoint responds to. + +This is a best practice to prevent XSS exploits. + +## Authentication + +Dgraph's GraphQL authorization relies on the presence of a valid JWT token in the request. + +Dgraph supports both symmetric (HS256) and asymmetric (RS256) encryption and accepts JSON Web Key (JWK) URL or signed JSON Web Token (JWT). + +You can use any authentication method that is capable of generating such JWT token (Auth0, Cognito, Firebase, etc...) including Dgraph login mechanism. + + +### ACL +Note that another token may be needed to access the system if ACL security is also enabled. See the [ACLs](/installation/configuration/enable-acl) section for details. The ACLs are a separate security mechanism. + +### JWT Claims + +In JSON web tokens (JWTs) (https://www.rfc-editor.org/rfc/rfc7519) , a claim appears as a name/value pair. + +When we talk about a claim in the context of a JWT, we are referring to the name (or key). For example, the following JSON object contains three claims ``sub``, ``name`` and ``admin``: +```json +{ +"sub": "1234567890", +"name": "John Doe", +"admin": true +} +``` + +So that different organizations can specify different claims without conflicting, claims typically have a namespace, and it's a good practice to specify the namespace of your claims. put specific claims in a nested structure called a namespace. +``` +{ + "https://mycompany.org/jwt/claims": { + "username": "auth0|63fe77f32cef38f4fa3dab34", + "role": "Admin" + }, + "name": "raph@dgraph.io", + "email": "raph@dgraph.io", + "email_verified": false, + "iss": "https://dev-5q3n8cc7nckhu5w8.us.auth0.com/", + "aud": "aqk1CSVtliyoXUfLaaLKSKUtkaIel6Vd", + "iat": 1677705681, + "exp": 1677741681 +} +``` +This json is a JWT token payload containing a namespace ``https://mycompany.org/jwt/claims`` having a ``username`` claim and a ``role`` claim. diff --git a/docusaurus-docs/graphql_versioned_docs/version-v25.4/security/jwt.md b/docusaurus-docs/graphql_versioned_docs/version-v25.4/security/jwt.md new file mode 100644 index 00000000..6ce90579 --- /dev/null +++ b/docusaurus-docs/graphql_versioned_docs/version-v25.4/security/jwt.md @@ -0,0 +1,158 @@ +--- +title: "Handle JWT Token" + +--- + +When deploying a GraphQL schema, the admin user can set a ``# Dgraph.Authorization`` line at the bottom of the schema to specify how JWT tokens present in the HTTP header requests are extracted, validated and used. + +This line must start with the exact string ``# Dgraph.Authorization`` and be at the bottom of the schema file. + + +## Configure JWT token handling + +To configure how Dgraph should handle JWT token for ``/graphql`` endpoint : +1. Add a line starting with ``# Dgraph.Authorization`` and with the following parameters at the very end of your GraphQL schema. + The `Dgraph.Authorization` object uses the following syntax: + + ``` + # Dgraph.Authorization {"VerificationKey":"","Header":"X-My-App-Auth","Namespace":"https://my.app.io/jwt/claims","Algo":"HS256","Audience":["aud1"],"ClosedByDefault":true} + ``` + +Dgraph.Authorization object contains the following parameters: +* `Header` name of the header field used by the client to send the token. + :::note + Do not use `Dg-Auth`, `X-Auth-Token` or `Authorization` headers which are used by Dgraph for other purposes. +::: +* `Namespace` is the key inside the JWT that contains the claims relevant to Dgraph authorization. +* `Algo` is the JWT verification algorithm which can be either `HS256` or `RS256`. +* `VerificationKey` is the string value of the key, with newlines replaced with `\n` and the key string wrapped in `""`: + * **For asymmetric encryption**: `VerificationKey` contains the public key string. + * **For symmetric (secret-based) encryption**: `VerificationKey` is the secret key. +* `JWKURL`/`JWKURLs` is the URL for the JSON Web Key sets. If you want to pass multiple URLs, use `JWKURLs` as an array of multiple JWK URLs for the JSON Web Key sets. You can only use one authentication connection method, either JWT (`Header`), a single JWK URL, or multiple JWK URLs. +* `Audience` is used to verify the `aud` field of a JWT, which is used by certain providers to indicate the intended audience for the JWT. When doing authentication with `JWKURL`, this field is mandatory. +* `ClosedByDefault`, if set to `true`, requires authorization for all requests even if the GraphQL type does not specify rules. If omitted, the default setting is `false`. + +2. Deploy the GraphQL schema either with a [schema update](/graphql/admin/#using-updategqlschema-to-add-or-modify-a-schema) or via the Cloud console's [Schema](https://cloud.dgraph.io/_/schema) page. + + +When the `# Dgraph.Authorization` line is present in the GraphQL schema, Dgraph will use the settings in that line to +- read the specified header in each HTTP request sent on the /graphql endpoint, +- decode that header as a JWT token using the specified algorithm (Algo) +- validate the token signature and the audience +- extract the JWT claims present in the specified namespace and at the root level + +These claims will then be accessible to any @auth schema directives (a GraphQL schema directive specific to Dgraph) that are associated with GraphQL types in the schema file. + +See the [RBAC rules](/graphql/security/RBAC-rules) and [Graph traversal rules](/graphql/security/graphtraversal-rules) for details on how to restrict data access using the @auth directive on a per-type basis. + +### Require JWT token +To not only accept but to require the JWT token regardless of @auth directives in your GraphQL schema, set option "ClosedByDefault" to true in the `# Dgraph.Authorization` line. + +## Working with Authentication providers +Dgraph.Authorization is fully configurable to work with various authentication providers. +Authentication providers have options to configure how to generate JWT tokens. + +Here are some configuration examples. + +### Clerk.com + +In your clerk dashboard, Access `JWT Templates` and create a template for Dgraph. + +Your template must have an `aud` (audience), this is mandatory for Dgraph when the token is verified using JWKURL. + +Decide on a claim namespace and add the information you want to use in your RBAC rules. + +We are using 'https://dgraph.io/jwt/claims' namespace in this example and have decided to get the user current organization, role ( clerk has currently two roles 'admin' and 'basic_member') and email. + +This is our JWT Template in Clerk: +```json +{ + "aud": "dgraph", + "https://dgraph.io/jwt/claims": { + "org": "{{org.name}}", + "role": "{{org.role}}", + "userid": "{{user.primary_email_address}}" + } +} +``` + +In the same configuration panel +- set the **token lifetime** +- copy the **JWKS Endpoint** + +Configure your Dgraph GraphQL schema with the following authorization +``` +# Dgraph.Authorization {"header":"X-Dgraph-AuthToken","namespace":"https://dgraph.io/jwt/claims","jwkurl":"https://<>.clerk.accounts.dev/.well-known/jwks.json","audience":["dgraph"],"closedbydefault":true} +``` +Note that +- **namespace** matches the namespace used in the JWT Template +- **audience** is an array and contains the **aud** used in the JWT token +- **jwkurl** is the **JWKS Endpoint** from Clerk + +You can select the header to receive the JWT token from your client app, `X-Dgraph-AuthToken` is a header authorized by default by Dgraph GraphQL API to pass CORS requirements. + + +## Other Dgraph.Authorization Examples + +To use a single JWK URL: + +``` +# Dgraph.Authorization {"VerificationKey":"","Header":"X-My-App-Auth", "jwkurl":"https://www.googleapis.com/service_accounts/v1/jwk/securetoken@system.gserviceaccount.com", "Namespace":"https://xyz.io/jwt/claims","Algo":"","Audience":["fir-project1-259e7", "HhaXkQVRBn5e0K3DmMp2zbjI8i1wcv2e"]} +``` + +To use multiple JWK URL: + +``` +# Dgraph.Authorization {"VerificationKey":"","Header":"X-My-App-Auth","jwkurls":["https://www.googleapis.com/service_accounts/v1/jwk/securetoken@system.gserviceaccount.com","https://dev-hr2kugfp.us.auth0.com/.well-known/jwks.json"], "Namespace":"https://xyz.io/jwt/claims","Algo":"","Audience":["fir-project1-259e7", "HhaXkQVRBn5e0K3DmMp2zbjI8i1wcv2e"]} +``` + +Using HMAC-SHA256 token in `X-My-App-Auth` header and authorization claims in `https://my.app.io/jwt/claims` namespace: + + +``` +# Dgraph.Authorization {"VerificationKey":"secretkey","Header":"X-My-App-Auth","Namespace":"https://my.app.io/jwt/claims","Algo":"HS256"} +``` + +Using HMAC-SHA256 token in `X-My-App-Auth` header and authorization claims in `https://my.app.io/jwt/claims` namespace: + +``` +# Dgraph.Authorization {"VerificationKey":"-----BEGIN PUBLIC KEY-----\n...\n-----END PUBLIC KEY-----","Header":"X-My-App-Auth","Namespace":"https://my.app.io/jwt/claims","Algo":"RS256"} +``` + +### JWT format + +The value of the JWT ``header`` is expected to be in one of the following forms: +* Bare token. + For example: + ``` + eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyLCJodHRwczovL215LmFwcC5pby9qd3QvY2xhaW1zIjp7fX0.Pjlxpf-3FhH61EtHBRo2g1amQPRi0pNwoLUooGbxIho + ``` + +* A Bearer token, e.g., a JWT prepended with `Bearer ` prefix (including space). + For example: + ``` + Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyLCJodHRwczovL215LmFwcC5pby9qd3QvY2xhaW1zIjp7fX0.Pjlxpf-3FhH61EtHBRo2g1amQPRi0pNwoLUooGbxIho + ``` + +### Error handling + +If ClosedByDefault is set to true, and the JWT is not present or if the JWT token does not include the proper audience information, or is not properly encoded, or is expired, Dgraph replies to requests on `/graphql` endpoint with an error message rejecting the operation similar to: +``` +{ + "errors": [ + { + "message": "couldn't rewrite query queryContact because a valid JWT is required but was not provided", + "path": [ + "queryContact" + ] + } + ], + "data": { + "queryContact": [] + },... +``` +**Error messages** +- "couldn't rewrite query queryContact because a valid JWT is required but was not provided" +- "couldn't rewrite query queryMessage because unable to parse jwt token:token is expired by 5h49m46.236018623s" +- "couldn't rewrite query queryMessage because JWT `aud` value doesn't match with the audience" +- "couldn't rewrite query queryMessage because unable to parse jwt token:token signature is invalid" diff --git a/docusaurus-docs/graphql_versioned_docs/version-v25.4/security/mutations.md b/docusaurus-docs/graphql_versioned_docs/version-v25.4/security/mutations.md new file mode 100644 index 00000000..299a1db3 --- /dev/null +++ b/docusaurus-docs/graphql_versioned_docs/version-v25.4/security/mutations.md @@ -0,0 +1,117 @@ +--- +title: "Mutations and GraphQL Authorization" +description: "Mutations with authorization work like queries. But mutations involve a state change in the database, so you need to understand when the rules are applied." + +--- + +Mutations with authorization work like queries. But because mutations involve a state change in the database, it's important to understand when the authorization rules are applied and what they mean. + +## Add + +Rules for `add` authorization state that the rule must hold of nodes created by the mutation data once committed to the database. + +For example, a rule such as the following: + +```graphql +type Todo @auth( + add: { rule: """ + query ($USER: String!) { + queryTodo { + owner(filter: { username: { eq: $USER } } ) { + username + } + } + }""" + } +){ + id: ID! + text: String! + owner: User +} +type User { + username: String! @id + todos: [Todo] +} +``` + +... states that if you add a new to-do list item, then that new to-do must satisfy the `add` rule, in this case saying that you can only add to-do list items with yourself as the author. + +## Delete + +Delete rules filter the nodes that can be deleted. A user can only ever delete a subset of the nodes that the `delete` rules allow. + +For example, the following rule states that a user can delete a to-do list item if they own it, or they have the `ADMIN` role: + +```graphql +type Todo @auth( + delete: { or: [ + { rule: """ + query ($USER: String!) { + queryTodo { + owner(filter: { username: { eq: $USER } } ) { + username + } + } + }""" + }, + { rule: "{$ROLE: { eq: \"ADMIN\" } }"} + ]} +){ + id: ID! + text: String! @search(by: [term]) + owner: User +} + +type User { + username: String! @id + todos: [Todo] +} +``` + +When using these types of rules, a mutation such as the one shown below will behave differently. +depending on which user is running it: +* For most users, the following mutation deletes the posts that contain the + term "graphql" and are owned by the user who runs the mutation, but doesn't + affect any other user's to-do list items +* For an admin user, the following mutation deletes any posts that contain the + term "graphql", regardless of which user owns these posts + +```graphql +mutation { + deleteTodo(filter: { text: { anyofterms: "graphql" } }) { + numUids + } +} +``` + +When adding data, what matters is the resulting state of the database, when deleting, +what matters is the state before the delete occurs. + +## Update + +Updates have both a before and after state that can be important for authorization. + +For example, consider a rule stating that you can only update your own to-do list items. If evaluated in the database before the mutation (like the delete rules) it would prevent you from updating anyone elses to-do list items, but it does not stop you from updating your own to-do items to have a different `owner`. If evaluated in the database after the mutation occurs, like for add rules, it would prevent setting the `owner` to another user, but would not prevent editing other's posts. + +Currently, Dgraph evaluates `update` rules _before_ the mutation. + +## Update and add mutations + +Update mutations can also insert new data. For example, you might allow a mutation that runs an update mutation to add a new to-do list item: + +```graphql +mutation { + updateUser(input: { + filter: { username: { eq: "aUser" }}, + set: { todos: [ { text: "do this new todo"} ] } + }) { + ... + } +} +``` + +Because a mutation updates a user's to-do list by inserting a new to-do list item, it +would have to satisfy the rules to update the author _and_ the rules to add a +to-do list item. If either fail, the mutation has no effect. + +--- \ No newline at end of file diff --git a/docusaurus-docs/graphql_versioned_docs/version-v25.4/subscriptions/index.md b/docusaurus-docs/graphql_versioned_docs/version-v25.4/subscriptions/index.md new file mode 100644 index 00000000..2bc2a342 --- /dev/null +++ b/docusaurus-docs/graphql_versioned_docs/version-v25.4/subscriptions/index.md @@ -0,0 +1,185 @@ +--- +title: "GraphQL Subscriptions" +description: "Subscriptions allow clients to listen to real-time messages from the server. In GraphQL, it’s straightforward to enable subscriptions on any type." + +--- + +Subscriptions allow clients to listen to real-time messages from the server. The client connects to the server with a bi-directional communication channel using the WebSocket protocol and sends a subscription query that specifies which event it is interested in. When an event is triggered, the server executes the stored GraphQL query, and the result is sent back to the client using the same communication channel. + +The client can unsubscribe by sending a message to the server. The server can also unsubscribe at any time due to errors or timeouts. A significant difference between queries or mutations and subscriptions is that subscriptions are stateful and require maintaining the GraphQL document, variables, and context over the lifetime of the subscription. + +![Subscription](/images/graphql/subscription_flow.png "Subscription in GraphQL") + +## Enable subscriptions in GraphQL + +In GraphQL, it's straightforward to enable subscriptions on any type. You can add the `@withSubscription` directive to the schema as part of the type definition, as in the following example: + +```graphql +type Todo @withSubscription { + id: ID! + title: String! + description: String! + completed: Boolean! +} +``` + +## @withSubscription with @auth + +You can use [@auth](/graphql/schema/directives/auth) access control rules in conjunction with `@withSubscription`. + + +Consider following Schema that has both the `@withSubscription` and `@auth` directives defined on type `Todo`. + +```graphql +type Todo @withSubscription @auth( + query: { rule: """ + query ($USER: String!) { + queryTodo(filter: { owner: { eq: $USER } } ) { + __typename + } + }""" + } + ){ + id: ID! + text: String! @search(by: [term]) + owner: String! @search(by: [hash]) + } +# Dgraph.Authorization {"Header":"X-Dgraph-AuthToken","Namespace":"https://dgraph.io/jwt/claims","jwkurl":"https://xyz.clerk.accounts.dev/.well-known/jwks.json","audience":["dgraph"],"ClosedByDefault":true} +``` +The generated GraphQL API expects a JWT token in the `X-Dgraph-AuthToken` header and uses the `USER` claim to apply a rule based access control (RBAC): the authorization rule enforces that only to-do tasks owned by `$USER` are returned. + + +## WebSocket client +Dgraph uses the websocket subprotocol `subscription-transport-ws`. + +Clients must be instantiated using the WebSocket URL of the GraphQL API which is your [Dgraph GraphQL endpoint](/graphql/graphql-clients/endpoint/) with ``https`` replaced by ``wss``. + +If your Dgraph endpoint is ``https://blue-surf-0033.us-east-1.aws.cloud.dgraph.io/graphql`` +the WebSocket URL is ``wss://blue-surf-0033.us-east-1.aws.cloud.dgraph.io/graphql`` + +If your GraphQL API is configured to expect a JWT token in a header, you must configure the WebSocket client to pass the token. Additionally, the subscription terminates when the JWT expires. + + +Here are some examples of frontend clients setup. + +### URQL client setup in a React application + +In this scenario, we are using [urql client](https://formidable.com/open-source/urql/) and `subscriptions-transport-ws` modules. + +In order to use a GraphQL subscription query in a component, you need to +- instantiate a subscriptionClient +- instantiate a URQL client with a 'subscriptionExchange' using the subscriptionClient + +```js +import { Client, Provider, cacheExchange, fetchExchange, subscriptionExchange } from 'urql'; +import { SubscriptionClient } from 'subscriptions-transport-ws'; + + const subscriptionClient = new SubscriptionClient( + process.env.REACT_APP_DGRAPH_WSS, + { reconnect: true, + connectionParams: {"X-Dgraph-AuthToken" : props.token} + } + ); + + const client = new Client({ + url: process.env.REACT_APP_DGRAPH_ENDPOINT, + fetchOptions: { headers: { "X-Dgraph-AuthToken": `Bearer ${props.token}` } }, + exchanges: [ + cacheExchange, + fetchExchange, + subscriptionExchange({ + forwardSubscription: request => subscriptionClient.request(request), + }) + ]}) + ``` + +In this example, + +- **process.env.REACT_APP_DGRAPH_ENDPOINT** is your [Dgraph GraphQL endpoint](/graphql/graphql-clients/endpoint/) +- **process.env.REACT_APP_DGRAPH_WSS** is the WebSocket URL +- **props.token** is the JWT token of the logged-in user. + +Note that we are passing the JWT token in the GraphQL client using 'fetchOptions' and in the WebSocket client using 'connectionParams'. + +Assuming we are using graphql-codegen, we can define a subcription query: +```js +import { graphql } from "../gql"; + +export const TodoFragment = graphql(` + fragment TodoItem on Todo { + id + text + } +`) + + +export const TodoSubscription = graphql(` + subscription myTodo { + queryTodo(first:100) { + ...TodoItem + } + } +`) +``` +and use it in a React component +```js +import { useQuery, useSubscription } from "urql"; +... +const [messages] = useSubscription({ query: MyMessagesDocument}); + +``` +That's it, the react component is able to use ``messages.data.queryTodo`` to display the updated list of Todos. + + +### Apollo client setup + +To learn about using subscriptions with Apollo client, see a blog post on [GraphQL Subscriptions with Apollo client](https://dgraph.io/blog/post/how-does-graphql-subscription/). + +To pass the user JWT token in the Apollo client,use `connectionParams`, as follows. + +```javascript +const wsLink = new WebSocketLink({ + uri: `wss://${ENDPOINT}`, + options: { + reconnect: true, + connectionParams: { "
": "", },}); +``` + +Use the header expected by the Dgraph.Authorization configuration of your GraphQL schema. + +## Subscriptions to custom DQL + +You can also apply `@withSubscription` directive to custom DQL queries by specifying `@withSubscription` on individual DQL queries in `type Query`, +and those queries will be added to `type subscription`. + +For example, see the custom DQL query `queryUserTweetCounts` below: + +```graphql +type Query { + queryUserTweetCounts: [UserTweetCount] @withSubscription @custom(dql: """ + query { + queryUserTweetCounts(func: type(User)) { + screen_name: User.screen_name + tweetCount: count(User.tweets) + } + } + """) +} +``` + +`queryUserTweetCounts` is added to the `subscription` type, allowing users to subscribe to this query. + +:::note +Currently, Dgraph only supports subscriptions on custom **DQL queries**. You +can't subscribe to custom **HTTP queries**. +::: + + + +:::note +Starting in release v21.03, Dgraph supports compression for subscriptions. +Dgraph uses `permessage-deflate` compression if the GraphQL client's +`Sec-Websocket-Extensions` request header includes `permessage-deflate`, as follows: +`Sec-WebSocket-Extensions: permessage-deflate`. +::: + diff --git a/docusaurus-docs/graphql_versioned_sidebars/version-v25.4-sidebars.json b/docusaurus-docs/graphql_versioned_sidebars/version-v25.4-sidebars.json new file mode 100644 index 00000000..6a2fdb90 --- /dev/null +++ b/docusaurus-docs/graphql_versioned_sidebars/version-v25.4-sidebars.json @@ -0,0 +1,152 @@ +{ + "graphqlSidebar": [ + "index", + { + "type": "category", + "label": "GraphQL", + "items": [ + "quick-start/index", + { + "type": "category", + "label": "Schema", + "items": [ + "schema/index", + "schema/dgraph-schema", + "schema/types", + "schema/graph-links", + "schema/documentation", + "schema/migration", + "schema/reserved", + { + "type": "category", + "label": "Directives", + "items": [ + "schema/directives/index", + "schema/directives/auth", + "schema/directives/deprecated", + "schema/directives/directive-dgraph", + "schema/directives/directive-withsubscription", + "schema/directives/embedding", + "schema/directives/generate", + "schema/directives/ids", + "schema/directives/search" + ] + } + ] + }, + { + "type": "category", + "label": "Queries", + "items": [ + "queries/index", + "queries/queries-overview", + "queries/search-filtering", + "queries/and-or-not", + "queries/order-page", + "queries/aggregate", + "queries/cascade", + "queries/skip-include", + "queries/cached-results", + "queries/persistent-queries", + "queries/vector-similarity" + ] + }, + { + "type": "category", + "label": "Mutations", + "items": [ + "mutations/index", + "mutations/mutations-overview", + "mutations/add", + "mutations/update", + "mutations/delete", + "mutations/upsert", + "mutations/deep" + ] + }, + "subscriptions/index", + { + "type": "category", + "label": "Lambda", + "items": [ + "lambda/index", + "lambda/lambda-overview", + "lambda/query", + "lambda/mutation", + "lambda/field", + "lambda/webhook" + ] + }, + { + "type": "category", + "label": "Custom", + "items": [ + "custom/index", + "custom/custom-overview", + "custom/query", + "custom/mutation", + "custom/field", + "custom/directive", + "custom/custom-dql" + ] + }, + { + "type": "category", + "label": "GraphQL Clients", + "items": [ + "graphql-clients/index", + "graphql-clients/graphql-ui", + "graphql-clients/graphql-ide", + { + "type": "category", + "label": "Endpoint", + "link": { + "type": "doc", + "id": "graphql-clients/endpoint/index" + }, + "items": [ + "graphql-clients/endpoint/graphql-request", + "graphql-clients/endpoint/graphql-response", + "graphql-clients/endpoint/graphql-get-request" + ] + } + ] + }, + { + "type": "category", + "label": "Security", + "items": [ + "security/index", + "security/jwt", + "security/auth-tips", + "security/cors", + "security/mutations", + "security/graphtraversal-rules", + "security/RBAC-rules" + ] + }, + { + "type": "category", + "label": "Admin", + "items": [ + "admin/index", + "admin/admin-api" + ] + }, + "federation/index" + ] + }, + { + "type": "category", + "label": "GraphQL-DQL", + "items": [ + "graphql-dql/index", + "graphql-dql/dql-for-graphql", + "graphql-dql/graphql-dql-schema", + "graphql-dql/graphql-dgraph", + "graphql-dql/graphql-data-loading", + "graphql-dql/graphql-data-migration" + ] + } + ] +} diff --git a/docusaurus-docs/graphql_versions.json b/docusaurus-docs/graphql_versions.json index 63b866f5..a790137b 100644 --- a/docusaurus-docs/graphql_versions.json +++ b/docusaurus-docs/graphql_versions.json @@ -1,4 +1,5 @@ [ + "v25.4", "v25.3", "v25.2", "v25.1",