`.
+The starting node must be an existing absolute znode path. The maximum depth is
+a non-negative integer: 0 prints every non-leaf node, 1 prints the starting
+node and its non-leaf children, 2 adds another level, and so on.
+**Depth limits output only, not traversal or totals.**
+
+`children` is the number of all descendants, not just immediate children.
+`data` is the sum of payload bytes for the node itself and all descendants.
+Null data contributes zero bytes. Leaves contribute to their ancestors' totals
+but are not printed; selecting a leaf produces no summary entries.
+
+For example, if `/app` has 2 payload bytes, `/app/branch` has 3, and
+`/app/branch/leaf` has 5, the summary is:
+
+```text
+ /app
+ children: 2
+ data: 10
+-- /app/branch
+-- children: 1
+-- data: 8
+```
+
+#### Snapshot analysis limitations
+
+Both tools run offline, read files without modifying them, and support
+uncompressed, `.gz`, and `.snappy` snapshots (including mixed formats in the
+comparer). They validate snapshot checksums and report unreadable or corrupt
+input rather than silently producing a successful analysis. Invalid arguments,
+file paths or summary starting paths exit with code 2; snapshot read failures
+exit with code 1. Windows launchers with the same names and a `.cmd` extension
+are also included. The launchers use the existing `zkEnv` configuration.
+
+Both tools **include ephemeral znodes** present in the snapshot; they do not
+report session records. This reflects the upstream traversal behavior, despite
+the original comparer's description claiming that ephemerals were ignored.
+Neither tool compares payload contents, ACLs, versions or other znode metadata.
+**Equal sizes/counts do not prove identical contents.** Snapshots may be fuzzy:
+these tools do not replay transaction logs, reconstruct point-in-time state,
+or establish transaction-consistent equality. The tools load snapshots into
+memory, and recursive traversal still visits the full selected subtree.
+
## Testing
diff --git a/zookeeper-server/src/main/java/org/apache/zookeeper/server/SnapshotComparer.java b/zookeeper-server/src/main/java/org/apache/zookeeper/server/SnapshotComparer.java
new file mode 100644
index 00000000000..39c9616ce27
--- /dev/null
+++ b/zookeeper-server/src/main/java/org/apache/zookeeper/server/SnapshotComparer.java
@@ -0,0 +1,435 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.zookeeper.server;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.PushbackInputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Scanner;
+import java.util.Set;
+import java.util.zip.CheckedInputStream;
+import org.apache.commons.cli.CommandLine;
+import org.apache.commons.cli.DefaultParser;
+import org.apache.commons.cli.HelpFormatter;
+import org.apache.commons.cli.Option;
+import org.apache.commons.cli.Options;
+import org.apache.commons.cli.ParseException;
+import org.apache.jute.BinaryInputArchive;
+import org.apache.jute.InputArchive;
+import org.apache.zookeeper.ZKUtil;
+import org.apache.zookeeper.server.persistence.FileSnap;
+import org.apache.zookeeper.server.persistence.SnapStream;
+import org.apache.zookeeper.util.ServiceUtils;
+
+/**
+ * Compares snapshot subtree data sizes and descendant counts, including ephemeral nodes.
+ * Sessions, payload contents, ACLs and other metadata are not compared. Equal sizes and
+ * counts do not establish identical contents or transaction-consistent state.
+ *
+ * Backported from Apache ZooKeeper commit f90060b83da4bfcca58ada93a57fedb40a069387
+ * (ZOOKEEPER-3427).
+ */
+public class SnapshotComparer {
+
+ private static final String LEFT_OPTION = "left";
+ private static final String RIGHT_OPTION = "right";
+ private static final String BYTE_THRESHOLD_OPTION = "bytes";
+ private static final String NODE_THRESHOLD_OPTION = "nodes";
+ private static final String DEBUG_OPTION = "debug";
+ private static final String INTERACTIVE_OPTION = "interactive";
+
+ private final Options options = new Options();
+
+ private SnapshotComparer() {
+ options.addOption(Option.builder("l").longOpt(LEFT_OPTION).hasArg().required()
+ .argName("LEFT").desc("(Required) The left snapshot file.").build());
+ options.addOption(Option.builder("r").longOpt(RIGHT_OPTION).hasArg().required()
+ .argName("RIGHT").desc("(Required) The right snapshot file.").build());
+ options.addOption(Option.builder("b").longOpt(BYTE_THRESHOLD_OPTION).hasArg().required()
+ .argName("BYTETHRESHOLD")
+ .desc("(Required) The node data delta size threshold, in bytes, for printing the node.").build());
+ options.addOption(Option.builder("n").longOpt(NODE_THRESHOLD_OPTION).hasArg().required()
+ .argName("NODETHRESHOLD")
+ .desc("(Required) The descendant node delta size threshold, in nodes, for printing the node.").build());
+ options.addOption("d", DEBUG_OPTION, false, "Use debug output.");
+ options.addOption("i", INTERACTIVE_OPTION, false, "Enter interactive mode.");
+ }
+
+ private void usage() {
+ new HelpFormatter().printHelp(120, "java -cp " + SnapshotComparer.class.getName(), "", options, "");
+ }
+
+ public static void main(String[] args) {
+ new SnapshotComparer().compareSnapshots(args);
+ }
+
+ private void compareSnapshots(String[] args) {
+ CommandLine parsedOptions;
+ File left;
+ File right;
+ int byteThreshold;
+ int nodeThreshold;
+ try {
+ parsedOptions = new DefaultParser().parse(options, args);
+ if (!parsedOptions.getArgList().isEmpty()) {
+ throw new IllegalArgumentException("Unexpected arguments: " + parsedOptions.getArgList());
+ }
+ byteThreshold = Integer.parseInt(parsedOptions.getOptionValue(BYTE_THRESHOLD_OPTION));
+ nodeThreshold = Integer.parseInt(parsedOptions.getOptionValue(NODE_THRESHOLD_OPTION));
+ if (byteThreshold < 0 || nodeThreshold < 0) {
+ throw new IllegalArgumentException("Thresholds must be non-negative integers.");
+ }
+ left = new File(parsedOptions.getOptionValue(LEFT_OPTION));
+ right = new File(parsedOptions.getOptionValue(RIGHT_OPTION));
+ for (File file : new File[]{left, right}) {
+ String error = ZKUtil.validateFileInput(file.toString());
+ if (error != null) {
+ throw new IllegalArgumentException(error);
+ }
+ }
+ } catch (ParseException | IllegalArgumentException e) {
+ System.err.println(e.getMessage());
+ usage();
+ ServiceUtils.requestSystemExit(ExitCode.INVALID_INVOCATION.getValue());
+ return;
+ }
+
+ boolean debug = parsedOptions.hasOption(DEBUG_OPTION);
+ boolean interactive = parsedOptions.hasOption(INTERACTIVE_OPTION);
+ System.out.println("Successfully parsed options!");
+ try {
+ TreeInfo leftTree = new TreeInfo(left);
+ TreeInfo rightTree = new TreeInfo(right);
+ System.out.println(leftTree);
+ System.out.println(rightTree);
+ compareTrees(leftTree, rightTree, byteThreshold, nodeThreshold, debug, interactive);
+ } catch (IOException e) {
+ System.err.println("Unable to read snapshot: " + e.getMessage());
+ ServiceUtils.requestSystemExit(ExitCode.UNEXPECTED_ERROR.getValue());
+ }
+ }
+
+ private static class TreeInfo {
+
+ private static class TreeNode {
+
+ final String label;
+ final long size;
+ final List children = new ArrayList<>();
+ long descendantSize;
+ long descendantCount;
+
+ TreeNode(String label, long size) {
+ this.label = label;
+ this.size = size;
+ }
+
+ void populateChildren(DataTree dataTree, TreeInfo treeInfo, int depth) {
+ DataNode node = dataTree.getNode(label);
+ Set childLabels;
+ synchronized (node) {
+ childLabels = node.getChildren();
+ }
+ for (String childName : childLabels) {
+ String childPath = label + "/" + childName;
+ DataNode childNode = dataTree.getNode(childPath);
+ long childSize;
+ synchronized (childNode) {
+ childSize = childNode.data == null ? 0 : childNode.data.length;
+ }
+ TreeNode child = new TreeNode(childPath, childSize);
+ child.populateChildren(dataTree, treeInfo, depth + 1);
+ children.add(child);
+ }
+ descendantSize = size;
+ descendantCount = children.size();
+ for (TreeNode child : children) {
+ descendantSize += child.descendantSize;
+ descendantCount += child.descendantCount;
+ }
+ treeInfo.registerNode(this, depth);
+ }
+
+ }
+
+ final TreeNode root;
+ long count;
+ final List> nodesAtDepths = new ArrayList<>();
+ final Map nodesByName = new HashMap<>();
+
+ TreeInfo(File snapshot) throws IOException {
+ long beginning = System.nanoTime();
+ DataTree dataTree = getSnapshot(snapshot);
+ System.out.printf("Deserialized snapshot in %s in %f seconds%n",
+ snapshot.getName(), (System.nanoTime() - beginning) / 1_000_000_000.0);
+ beginning = System.nanoTime();
+ DataNode rootNode = dataTree.getNode("");
+ long size;
+ synchronized (rootNode) {
+ size = rootNode.data == null ? 0 : rootNode.data.length;
+ }
+ root = new TreeNode("", size);
+ root.populateChildren(dataTree, this, 0);
+ System.out.printf("Processed data tree in %f seconds%n", (System.nanoTime() - beginning) / 1_000_000_000.0);
+ }
+
+ void registerNode(TreeNode node, int depth) {
+ while (depth >= nodesAtDepths.size()) {
+ nodesAtDepths.add(new ArrayList<>());
+ }
+ nodesAtDepths.get(depth).add(node);
+ nodesByName.put(node.label, node);
+ count++;
+ }
+
+ @Override
+ public String toString() {
+ StringBuilder builder = new StringBuilder();
+ builder.append(String.format("Node count: %d%n", count));
+ builder.append(String.format("Total size: %d%n", root.descendantSize));
+ builder.append(String.format("Max depth: %d%n", nodesAtDepths.size()));
+ for (int i = 0; i < nodesAtDepths.size(); i++) {
+ builder.append(String.format("Count of nodes at depth %d: %d%n", i, nodesAtDepths.get(i).size()));
+ }
+ return builder.toString();
+ }
+
+ }
+
+ static DataTree getSnapshot(File file) throws IOException {
+ try (CheckedInputStream stream = SnapStream.getInputStream(file);
+ PushbackInputStream input = new PushbackInputStream(stream)) {
+ InputArchive archive = BinaryInputArchive.getArchive(input);
+ DataTree dataTree = new DataTree();
+ new FileSnap(null).deserialize(dataTree, new HashMap(), archive);
+ checkSealIntegrity(stream, archive);
+
+ // Distinguish an absent digest in older snapshots from a truncated digest.
+ int next = input.read();
+ if (next != -1) {
+ input.unread(next);
+ dataTree.new ZxidDigest().deserialize(archive);
+ checkSealIntegrity(stream, archive);
+ if (input.read() != -1) {
+ throw new IOException("Unexpected data after snapshot");
+ }
+ }
+ return dataTree;
+ } catch (IOException | NumberFormatException e) {
+ throw new IOException(file + ": " + e, e);
+ }
+ }
+
+ private static void checkSealIntegrity(CheckedInputStream stream, InputArchive archive) throws IOException {
+ // SnapStream's seal checker is package-private on this branch.
+ long checksum = stream.getChecksum().getValue();
+ long expected = archive.readLong("val");
+ String path = archive.readString("path");
+ if (checksum != expected || !"/".equals(path)) {
+ throw new IOException("CRC corruption or invalid snapshot seal");
+ }
+ }
+
+ private static void printThresholdInfo(int byteThreshold, int nodeThreshold) {
+ System.out.printf("Printing analysis for nodes difference larger than %d bytes or node count difference larger than %d.%n",
+ byteThreshold, nodeThreshold);
+ }
+
+ private static void compareTrees(
+ TreeInfo left,
+ TreeInfo right,
+ int byteThreshold,
+ int nodeThreshold,
+ boolean debug,
+ boolean interactive) {
+ int maxDepth = Math.max(left.nodesAtDepths.size(), right.nodesAtDepths.size());
+ if (!interactive) {
+ printThresholdInfo(byteThreshold, nodeThreshold);
+ for (int i = 0; i < maxDepth; i++) {
+ System.out.printf("Analysis for depth %d%n", i);
+ compareLine(left, right, i, byteThreshold, nodeThreshold, debug, false);
+ }
+ } else {
+ try (Scanner scanner = new Scanner(System.in)) {
+ int currentDepth = 0;
+ while (currentDepth < maxDepth) {
+ System.out.printf("Current depth is %d%n", currentDepth);
+ System.out.println("- Press enter to move to print current depth layer;\n"
+ + "- Type a number to jump to and print all nodes at a given depth;\n"
+ + "- Enter an ABSOLUTE path to print the immediate subtree of a node. Path must start with '/'.");
+ if (!scanner.hasNextLine()) {
+ System.out.println("End of input.");
+ return;
+ }
+ String input = scanner.nextLine();
+ printThresholdInfo(byteThreshold, nodeThreshold);
+ if (input.isEmpty()) {
+ System.out.printf("Analysis for depth %d%n", currentDepth);
+ compareLine(left, right, currentDepth, byteThreshold, nodeThreshold, debug, true);
+ currentDepth++;
+ } else if (input.startsWith("/")) {
+ System.out.printf("Analysis for node %s%n", input);
+ compareSubtree(left, right, input, byteThreshold, nodeThreshold, debug);
+ } else {
+ try {
+ int depth = Integer.parseInt(input);
+ if (depth < 0 || depth >= maxDepth) {
+ System.out.printf("Depth must be in range [%d, %d]%n", 0, maxDepth - 1);
+ continue;
+ }
+ currentDepth = depth;
+ System.out.printf("Analysis for depth %d%n", currentDepth);
+ compareLine(left, right, currentDepth, byteThreshold, nodeThreshold, debug, true);
+ } catch (NumberFormatException e) {
+ System.out.printf("Input %s is not valid. Depth must be in range [%d, %d]. "
+ + "Path must be an absolute path which starts with '/'.%n",
+ input, 0, maxDepth - 1);
+ }
+ }
+ System.out.println();
+ }
+ }
+ }
+ System.out.println("All layers compared.");
+ }
+
+ private static void compareSubtree(
+ TreeInfo left,
+ TreeInfo right,
+ String path,
+ int byteThreshold,
+ int nodeThreshold,
+ boolean debug) {
+ String label = "/".equals(path) ? "" : path;
+ TreeInfo.TreeNode leftRoot = left.nodesByName.get(label);
+ TreeInfo.TreeNode rightRoot = right.nodesByName.get(label);
+ if (leftRoot == null && rightRoot == null) {
+ System.out.printf("Path %s is neither found in left tree nor right tree.%n", path);
+ } else {
+ List leftList = leftRoot == null ? Collections.emptyList() : leftRoot.children;
+ List rightList = rightRoot == null ? Collections.emptyList() : rightRoot.children;
+ compareNodes(leftList, rightList, byteThreshold, nodeThreshold, debug, true);
+ }
+ }
+
+ private static void compareLine(
+ TreeInfo left,
+ TreeInfo right,
+ int depth,
+ int byteThreshold,
+ int nodeThreshold,
+ boolean debug,
+ boolean interactive) {
+ List leftList = depth >= left.nodesAtDepths.size()
+ ? Collections.emptyList() : left.nodesAtDepths.get(depth);
+ List rightList = depth >= right.nodesAtDepths.size()
+ ? Collections.emptyList() : right.nodesAtDepths.get(depth);
+ compareNodes(leftList, rightList, byteThreshold, nodeThreshold, debug, interactive);
+ }
+
+ private static void compareNodes(
+ List leftList,
+ List rightList,
+ int byteThreshold,
+ int nodeThreshold,
+ boolean debug,
+ boolean interactive) {
+ Comparator comparator = Comparator.comparing(node -> node.label);
+ Collections.sort(leftList, comparator);
+ Collections.sort(rightList, comparator);
+ int leftIndex = 0;
+ int rightIndex = 0;
+ while (leftIndex < leftList.size() || rightIndex < rightList.size()) {
+ TreeInfo.TreeNode leftNode = leftIndex < leftList.size() ? leftList.get(leftIndex) : null;
+ TreeInfo.TreeNode rightNode = rightIndex < rightList.size() ? rightList.get(rightIndex) : null;
+ if (leftNode != null && rightNode != null) {
+ if (debug) {
+ System.out.printf("Comparing %s to %s%n", leftNode.label, rightNode.label);
+ }
+ int result = comparator.compare(leftNode, rightNode);
+ if (result < 0) {
+ if (debug) {
+ System.out.println("left is less");
+ }
+ printOnly(leftNode, "left", byteThreshold, nodeThreshold, debug, interactive);
+ leftIndex++;
+ } else if (result > 0) {
+ if (debug) {
+ System.out.println("right is less");
+ }
+ printOnly(rightNode, "right", byteThreshold, nodeThreshold, debug, interactive);
+ rightIndex++;
+ } else {
+ if (debug) {
+ System.out.println("same");
+ }
+ printBoth(leftNode, rightNode, byteThreshold, nodeThreshold, debug, interactive);
+ leftIndex++;
+ rightIndex++;
+ }
+ } else if (leftNode != null) {
+ printOnly(leftNode, "left", byteThreshold, nodeThreshold, debug, interactive);
+ leftIndex++;
+ } else {
+ printOnly(rightNode, "right", byteThreshold, nodeThreshold, debug, interactive);
+ rightIndex++;
+ }
+ }
+ }
+
+ private static void printOnly(
+ TreeInfo.TreeNode node,
+ String side,
+ int byteThreshold,
+ int nodeThreshold,
+ boolean debug,
+ boolean interactive) {
+ if (node.descendantSize > byteThreshold || node.descendantCount > nodeThreshold) {
+ System.out.printf("Node %s found only in %s tree. Descendant size: %d. Descendant count: %d%n",
+ node.label, side, node.descendantSize, node.descendantCount);
+ } else if (debug || interactive) {
+ System.out.printf("Filtered %s node %s of size %d%n", side, node.label, node.descendantSize);
+ }
+ }
+
+ private static void printBoth(
+ TreeInfo.TreeNode left,
+ TreeInfo.TreeNode right,
+ int byteThreshold,
+ int nodeThreshold,
+ boolean debug,
+ boolean interactive) {
+ long byteDelta = right.descendantSize - left.descendantSize;
+ long nodeDelta = right.descendantCount - left.descendantCount;
+ if (Math.abs(byteDelta) > byteThreshold || Math.abs(nodeDelta) > nodeThreshold) {
+ System.out.printf("Node %s found in both trees. Delta: %d bytes, %d descendants%n",
+ left.label, byteDelta, nodeDelta);
+ } else if (debug || interactive) {
+ System.out.printf("Filtered node %s of left size %d, right size %d%n",
+ left.label, left.descendantSize, right.descendantSize);
+ }
+ }
+
+}
diff --git a/zookeeper-server/src/main/java/org/apache/zookeeper/server/SnapshotRecursiveSummary.java b/zookeeper-server/src/main/java/org/apache/zookeeper/server/SnapshotRecursiveSummary.java
new file mode 100644
index 00000000000..d34bc38c4b6
--- /dev/null
+++ b/zookeeper-server/src/main/java/org/apache/zookeeper/server/SnapshotRecursiveSummary.java
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.zookeeper.server;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.Collections;
+import java.util.Set;
+import java.util.TreeSet;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.apache.zookeeper.ZKUtil;
+import org.apache.zookeeper.common.PathUtils;
+import org.apache.zookeeper.util.ServiceUtils;
+
+/**
+ * Recursively summarizes snapshot subtree data sizes and descendant counts.
+ * Only non-leaf nodes are printed, but totals include all descendants and ephemeral nodes.
+ * The maximum depth limits output, not traversal; zero means unlimited output depth.
+ *
+ * Backported from Apache ZooKeeper commit 05b215994f5e145c2758c4089828b57ba471b329
+ * (ZOOKEEPER-4566).
+ */
+@InterfaceAudience.Public
+public class SnapshotRecursiveSummary {
+
+ public static void main(String[] args) {
+ if (args.length != 3) {
+ System.err.println(getUsage());
+ ServiceUtils.requestSystemExit(ExitCode.INVALID_INVOCATION.getValue());
+ return;
+ }
+ try {
+ new SnapshotRecursiveSummary().run(args[0], args[1], Integer.parseInt(args[2]));
+ } catch (IllegalArgumentException e) {
+ System.err.println(e.getMessage());
+ System.err.println(getUsage());
+ ServiceUtils.requestSystemExit(ExitCode.INVALID_INVOCATION.getValue());
+ } catch (IOException e) {
+ System.err.println("Unable to read snapshot: " + e.getMessage());
+ ServiceUtils.requestSystemExit(ExitCode.UNEXPECTED_ERROR.getValue());
+ }
+ }
+
+ public void run(String snapshotFileName, String startingNode, int maxDepth) throws IOException {
+ PathUtils.validatePath(startingNode);
+ if (maxDepth < 0) {
+ throw new IllegalArgumentException("max_depth must be a non-negative integer.");
+ }
+ String error = ZKUtil.validateFileInput(snapshotFileName);
+ if (error != null) {
+ throw new IllegalArgumentException(error);
+ }
+ DataTree dataTree = SnapshotComparer.getSnapshot(new File(snapshotFileName));
+ if (dataTree.getNode(startingNode) == null) {
+ throw new IllegalArgumentException("Starting node does not exist: " + startingNode);
+ }
+ StringBuilder builder = new StringBuilder();
+ printZnode(dataTree, startingNode, builder, 0, maxDepth);
+ System.out.println(builder);
+ }
+
+ private long[] printZnode(DataTree dataTree, String name, StringBuilder builder, int level, int maxDepth) {
+ DataNode node = dataTree.getNode(name);
+ Set children;
+ long dataSize;
+ synchronized (node) {
+ dataSize = node.data == null ? 0 : node.data.length;
+ children = new TreeSet<>(node.getChildren());
+ }
+ long[] result = {1L, dataSize};
+ if (children.isEmpty()) {
+ return result;
+ }
+ StringBuilder childBuilder = new StringBuilder();
+ for (String child : children) {
+ long[] childResult = printZnode(dataTree, name + (name.equals("/") ? "" : "/") + child,
+ childBuilder, level + 1, maxDepth);
+ result[0] += childResult[0];
+ result[1] += childResult[1];
+ }
+ if (maxDepth == 0 || level <= maxDepth) {
+ String indent = String.join("", Collections.nCopies(level, "--"));
+ builder.append(indent).append(" ").append(name).append("\n");
+ builder.append(indent).append(" children: ").append(result[0] - 1).append("\n");
+ builder.append(indent).append(" data: ").append(result[1]).append("\n");
+ builder.append(childBuilder);
+ }
+ return result;
+ }
+
+ public static String getUsage() {
+ String newLine = System.lineSeparator();
+ return String.join(newLine,
+ "USAGE:",
+ "",
+ "SnapshotRecursiveSummary ",
+ "",
+ "snapshot_file: path to the zookeeper snapshot",
+ "starting_node: the absolute path in the zookeeper tree where traversal should begin",
+ "max_depth: non-negative output depth. 0 displays every non-leaf node; "
+ + "1 displays the starting node and its non-leaf children; 2 adds another level, and so on. "
+ + "This ONLY affects the level of details displayed, NOT the calculation.");
+ }
+
+}
diff --git a/zookeeper-server/src/test/java/org/apache/zookeeper/server/SnapshotComparerTest.java b/zookeeper-server/src/test/java/org/apache/zookeeper/server/SnapshotComparerTest.java
new file mode 100644
index 00000000000..f44c75893b2
--- /dev/null
+++ b/zookeeper-server/src/test/java/org/apache/zookeeper/server/SnapshotComparerTest.java
@@ -0,0 +1,434 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.zookeeper.server;
+
+import static org.hamcrest.CoreMatchers.containsString;
+import static org.hamcrest.CoreMatchers.not;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertThat;
+import static org.junit.Assert.assertTrue;
+import java.io.ByteArrayInputStream;
+import java.io.File;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+import org.apache.commons.io.FileUtils;
+import org.apache.jute.BinaryInputArchive;
+import org.apache.zookeeper.ZKTestCase;
+import org.apache.zookeeper.ZooDefs;
+import org.apache.zookeeper.server.persistence.FileSnap;
+import org.apache.zookeeper.server.persistence.SnapStream;
+import org.apache.zookeeper.server.persistence.SnapStream.StreamMode;
+import org.apache.zookeeper.test.ClientBase;
+import org.junit.After;
+import org.junit.Assume;
+import org.junit.Before;
+import org.junit.Test;
+
+public class SnapshotComparerTest extends ZKTestCase {
+
+ private SnapshotToolTestSupport support;
+
+ @Before
+ public void setUp() throws IOException {
+ support = new SnapshotToolTestSupport();
+ }
+
+ @After
+ public void tearDown() throws IOException {
+ support.close();
+ }
+
+ @Test
+ public void testAddedAndDeletedSubtrees() throws Exception {
+ DataTree left = new DataTree();
+ support.addNode(left, "/app", new byte[1]);
+ support.addNode(left, "/app/removed", new byte[2]);
+ support.addNode(left, "/app/removed/leaf", new byte[3]);
+ DataTree right = new DataTree();
+ support.addNode(right, "/app", new byte[1]);
+ support.addNode(right, "/app/added", new byte[4]);
+ support.addNode(right, "/app/added/a", new byte[5]);
+ support.addNode(right, "/app/added/b", null);
+
+ String output = compare(left, right, "0", "0", "-d");
+
+ assertThat(output, containsString("Node found in both trees. Delta: 4 bytes, 1 descendants"));
+ assertThat(output, containsString("Node /app found in both trees. Delta: 4 bytes, 1 descendants"));
+ assertThat(output, containsString("Node /app/removed found only in left tree. Descendant size: 5. Descendant count: 1"));
+ assertThat(output, containsString("Node /app/removed/leaf found only in left tree. Descendant size: 3. Descendant count: 0"));
+ assertThat(output, containsString("Node /app/added found only in right tree. Descendant size: 9. Descendant count: 2"));
+ assertThat(output, containsString("Node /app/added/a found only in right tree. Descendant size: 5. Descendant count: 0"));
+ assertThat(output, containsString("Filtered right node /app/added/b of size 0"));
+ assertTrue(output, output.indexOf("Node /app/added found") < output.indexOf("Node /app/removed found"));
+ assertThat(output, containsString("All layers compared."));
+ }
+
+ @Test
+ public void testSignedByteDeltas() throws Exception {
+ DataTree left = new DataTree();
+ support.addNode(left, "/grow", new byte[1]);
+ support.addNode(left, "/shrink", new byte[4]);
+ DataTree right = new DataTree();
+ support.addNode(right, "/grow", new byte[4]);
+ support.addNode(right, "/shrink", new byte[1]);
+
+ String output = compare(left, right, "0", "2147483647");
+
+ assertThat(output, containsString("Node /grow found in both trees. Delta: 3 bytes, 0 descendants"));
+ assertThat(output, containsString("Node /shrink found in both trees. Delta: -3 bytes, 0 descendants"));
+ }
+
+ @Test
+ public void testByteThresholdIsExclusive() throws Exception {
+ DataTree left = new DataTree();
+ support.addNode(left, "/node", new byte[1]);
+ DataTree right = new DataTree();
+ support.addNode(right, "/node", new byte[5]);
+
+ String filtered = compare(left, right, "4", "0", "--debug");
+ String included = compare(left, right, "3", "0");
+
+ assertThat(filtered, containsString("Filtered node /node of left size 1, right size 5"));
+ assertThat(filtered, not(containsString("Node /node found")));
+ assertThat(included, containsString("Node /node found in both trees. Delta: 4 bytes, 0 descendants"));
+ }
+
+ @Test
+ public void testDescendantThresholdWithNullData() throws Exception {
+ DataTree left = new DataTree();
+ support.addNode(left, "/app", null);
+ DataTree right = new DataTree();
+ support.addNode(right, "/app", null);
+ support.addNode(right, "/app/child", null);
+
+ String filtered = compare(left, right, "2147483647", "1", "-d");
+ String included = compare(left, right, "2147483647", "0");
+ String removed = compare(right, left, "2147483647", "0");
+
+ assertThat(filtered, containsString("Filtered node /app of left size 0, right size 0"));
+ assertThat(filtered, not(containsString("Node /app found")));
+ assertThat(included, containsString("Node /app found in both trees. Delta: 0 bytes, 1 descendants"));
+ assertThat(removed, containsString("Node /app found in both trees. Delta: 0 bytes, -1 descendants"));
+ }
+
+ @Test
+ public void testEphemeralNodesAreIncluded() throws Exception {
+ DataTree right = new DataTree();
+ right.createNode("/ephemeral", new byte[7], ZooDefs.Ids.OPEN_ACL_UNSAFE, 0x123L, -1, 1, 1);
+
+ String output = compare(new DataTree(), right, "0", "0");
+
+ assertThat(output, containsString("Node /ephemeral found only in right tree. Descendant size: 7. Descendant count: 0"));
+ assertThat(output, containsString("Node found in both trees. Delta: 7 bytes, 1 descendants"));
+ }
+
+ @Test
+ public void testEqualLengthPayloadsAreOnlySizeCompared() throws Exception {
+ DataTree left = new DataTree();
+ support.addNode(left, "/equal", "aa".getBytes(StandardCharsets.UTF_8));
+ DataTree right = new DataTree();
+ support.addNode(right, "/equal", "bb".getBytes(StandardCharsets.UTF_8));
+
+ String output = compare(left, right, "0", "0", "-d");
+
+ assertThat(output, containsString("Filtered node /equal of left size 2, right size 2"));
+ assertThat(output, not(containsString("Node /equal found")));
+ assertThat(output, containsString("All layers compared."));
+ }
+
+ @Test
+ public void testMixedCompressionFormats() throws Exception {
+ DataTree leftTree = new DataTree();
+ support.addNode(leftTree, "/node", new byte[1]);
+ DataTree rightTree = new DataTree();
+ support.addNode(rightTree, "/node", new byte[3]);
+ File left = support.snapshot(leftTree, StreamMode.CHECKED, true);
+ for (StreamMode mode : StreamMode.values()) {
+ File right = support.snapshot(rightTree, mode, true);
+ String output = run("", arguments(left, right, "0", "0")).success();
+ assertThat(output, containsString("Node /node found in both trees. Delta: 2 bytes, 0 descendants"));
+ }
+ }
+
+ @Test
+ public void testSnapshotsWithoutDigest() throws Exception {
+ DataTree rightTree = new DataTree();
+ support.addNode(rightTree, "/node", new byte[3]);
+ File left = support.snapshot(new DataTree(), StreamMode.CHECKED, false);
+ File right = support.snapshot(rightTree, StreamMode.CHECKED, false);
+
+ String output = run("", arguments(left, right, "0", "0")).success();
+
+ assertThat(output, containsString("Node /node found only in right tree. Descendant size: 3. Descendant count: 0"));
+ }
+
+ @Test
+ public void testInteractivePathsAndDepths() throws Exception {
+ DataTree leftTree = new DataTree();
+ support.addNode(leftTree, "/app", null);
+ support.addNode(leftTree, "/app/leaf", new byte[1]);
+ DataTree rightTree = new DataTree();
+ support.addNode(rightTree, "/app", null);
+ support.addNode(rightTree, "/app/leaf", new byte[3]);
+ File left = support.snapshot(leftTree, StreamMode.CHECKED, true);
+ File right = support.snapshot(rightTree, StreamMode.CHECKED, true);
+
+ String output = run("/\n/app\n/app/leaf\n/missing\n-1\n999\nbad\n1\n\n\n",
+ arguments(left, right, "0", "0", "--interactive")).success();
+
+ assertThat(output, containsString("Analysis for node /"));
+ assertThat(output, not(containsString("Path / is neither found")));
+ assertThat(output, containsString("Node /app found in both trees. Delta: 2 bytes, 0 descendants"));
+ assertThat(output, containsString("Node /app/leaf found in both trees. Delta: 2 bytes, 0 descendants"));
+ assertThat(output, not(containsString("Path /app/leaf is neither found")));
+ assertThat(output, containsString("Path /missing is neither found in left tree nor right tree."));
+ assertThat(output, containsString("Depth must be in range [0, 2]"));
+ assertThat(output, containsString("Input bad is not valid."));
+ assertThat(output, containsString("Analysis for depth 1"));
+ assertThat(output, containsString("All layers compared."));
+ }
+
+ @Test
+ public void testInteractiveEndOfInput() throws Exception {
+ File snapshot = support.snapshot(new DataTree(), StreamMode.CHECKED, true);
+
+ String output = run("", arguments(snapshot, snapshot, "0", "0", "-i")).success();
+
+ assertThat(output, containsString("End of input."));
+ assertThat(output, not(containsString("All layers compared.")));
+ }
+
+ @Test
+ public void testInvalidArguments() throws Exception {
+ File snapshot = support.snapshot(new DataTree(), StreamMode.CHECKED, true);
+ run("").invalidInvocation();
+ run("", "-l", snapshot.toString()).invalidInvocation();
+ for (String invalid : Arrays.asList("-1", "NaN", "2147483648")) {
+ run("", arguments(snapshot, snapshot, invalid, "0")).invalidInvocation();
+ run("", arguments(snapshot, snapshot, "0", invalid)).invalidInvocation();
+ }
+ run("", arguments(snapshot, snapshot, "0", "0", "--unknown")).invalidInvocation();
+ run("", arguments(snapshot, snapshot, "0", "0", "extra")).invalidInvocation();
+ }
+
+ @Test
+ public void testInvalidFiles() throws Exception {
+ File snapshot = support.snapshot(new DataTree(), StreamMode.CHECKED, true);
+ File missing = new File(support.directory, "missing");
+ for (File invalid : Arrays.asList(missing, support.directory)) {
+ run("", arguments(invalid, snapshot, "0", "0")).invalidInvocation();
+ run("", arguments(snapshot, invalid, "0", "0")).invalidInvocation();
+ }
+ }
+
+ @Test
+ public void testCorruptSnapshots() throws Exception {
+ File valid = support.snapshot(new DataTree(), StreamMode.CHECKED, true);
+ for (File corrupt : support.corruptSnapshots()) {
+ SnapshotToolTestSupport.Result result = run("", arguments(corrupt, valid, "0", "0"));
+ result.readFailure();
+ assertThat(result.output, not(containsString("All layers compared.")));
+ }
+ }
+
+ @Test
+ public void testShellLauncher() throws Exception {
+ Assume.assumeFalse(System.getProperty("os.name").startsWith("Windows"));
+ DataTree rightTree = new DataTree();
+ support.addNode(rightTree, "/node", new byte[3]);
+ File left = support.snapshot(new DataTree(), StreamMode.CHECKED, true);
+ File right = support.snapshot(rightTree, StreamMode.GZIP, true);
+ for (String flags : Arrays.asList("", "-Xms32m -Xmx128m")) {
+ String output = support.runLauncher("zkSnapshotComparer.sh", flags,
+ "-l", left.toString(), "-r", right.toString(), "-b", "0", "-n", "0").success();
+ assertThat(output, containsString("Node /node found only in right tree. Descendant size: 3. Descendant count: 0"));
+ }
+ }
+
+ private String compare(DataTree left, DataTree right, String bytes, String nodes, String... flags) throws Exception {
+ return run("", arguments(support.snapshot(left, StreamMode.CHECKED, true),
+ support.snapshot(right, StreamMode.CHECKED, true),
+ bytes, nodes, flags)).success();
+ }
+
+ private String[] arguments(File left, File right, String bytes, String nodes, String... flags) {
+ List args = new ArrayList<>(Arrays.asList("--left", left.toString(), "--right", right.toString(),
+ "--bytes", bytes, "--nodes", nodes));
+ args.addAll(Arrays.asList(flags));
+ return args.toArray(new String[0]);
+ }
+
+ private SnapshotToolTestSupport.Result run(String input, String... args) throws Exception {
+ return support.run("org.apache.zookeeper.server.SnapshotComparer", input, args);
+ }
+
+ static class SnapshotToolTestSupport {
+
+ final File directory = ClientBase.createEmptyTestDir();
+ private int nextFile;
+
+ SnapshotToolTestSupport() throws IOException {
+ }
+
+ void addNode(DataTree tree, String path, byte[] data) throws Exception {
+ tree.createNode(path, data, ZooDefs.Ids.OPEN_ACL_UNSAFE, 0, -1, 1, 1);
+ }
+
+ File snapshot(DataTree tree, StreamMode mode, boolean digest) throws IOException {
+ File snapshots = new File(directory, "snapshots with spaces");
+ assertTrue(snapshots.isDirectory() || snapshots.mkdir());
+ File file = new File(snapshots, "snapshot." + Integer.toHexString(++nextFile) + mode.getFileExtension());
+ StreamMode previousMode = SnapStream.getStreamMode();
+ boolean previousDigest = ZooKeeperServer.isDigestEnabled();
+ try {
+ SnapStream.setStreamMode(mode);
+ ZooKeeperServer.setDigestEnabled(digest);
+ new FileSnap(null).serialize(tree, Collections.singletonMap(0x123L, 30000), file, false);
+ } finally {
+ SnapStream.setStreamMode(previousMode);
+ ZooKeeperServer.setDigestEnabled(previousDigest);
+ }
+ return file;
+ }
+
+ List corruptSnapshots() throws IOException {
+ List files = new ArrayList<>();
+ File badHeader = snapshot(new DataTree(), StreamMode.CHECKED, true);
+ byte[] bytes = Files.readAllBytes(badHeader.toPath());
+ bytes[0] ^= 1;
+ Files.write(badHeader.toPath(), bytes);
+ files.add(badHeader);
+
+ File truncated = snapshot(new DataTree(), StreamMode.CHECKED, true);
+ bytes = Files.readAllBytes(truncated.toPath());
+ Files.write(truncated.toPath(), Arrays.copyOf(bytes, bytes.length / 2));
+ files.add(truncated);
+
+ File checksum = snapshot(new DataTree(), StreamMode.CHECKED, true);
+ bytes = Files.readAllBytes(checksum.toPath());
+ ByteArrayInputStream input = new ByteArrayInputStream(bytes);
+ new FileSnap(null).deserialize(new DataTree(), new HashMap(),
+ BinaryInputArchive.getArchive(input));
+ int sealOffset = bytes.length - input.available();
+ bytes[sealOffset] ^= 1;
+ Files.write(checksum.toPath(), bytes);
+ files.add(checksum);
+
+ File partialDigest = snapshot(new DataTree(), StreamMode.CHECKED, true);
+ bytes = Files.readAllBytes(partialDigest.toPath());
+ Files.write(partialDigest.toPath(), Arrays.copyOf(bytes, sealOffset + 14));
+ files.add(partialDigest);
+
+ File digestSeal = snapshot(new DataTree(), StreamMode.CHECKED, true);
+ bytes = Files.readAllBytes(digestSeal.toPath());
+ bytes[bytes.length - 13] ^= 1;
+ Files.write(digestSeal.toPath(), bytes);
+ files.add(digestSeal);
+
+ File gzipTrailer = snapshot(new DataTree(), StreamMode.GZIP, true);
+ bytes = Files.readAllBytes(gzipTrailer.toPath());
+ bytes[bytes.length - 1] ^= 1;
+ Files.write(gzipTrailer.toPath(), bytes);
+ files.add(gzipTrailer);
+ return files;
+ }
+
+ Result run(String mainClass, String input, String... args) throws Exception {
+ List command = new ArrayList<>(Arrays.asList(
+ new File(System.getProperty("java.home"), "bin/java").toString(),
+ "-Xmx128m",
+ "-Djava.io.tmpdir=" + System.getProperty("java.io.tmpdir"),
+ "-cp",
+ System.getProperty("surefire.test.class.path", System.getProperty("java.class.path")),
+ mainClass));
+ command.addAll(Arrays.asList(args));
+ return runProcess(new ProcessBuilder(command), input);
+ }
+
+ Result runLauncher(String launcher, String flags, String... args) throws Exception {
+ File script = new File(System.getProperty("basedir", "."), "../bin/" + launcher);
+ List command = new ArrayList<>(Arrays.asList("bash", script.getCanonicalPath()));
+ command.addAll(Arrays.asList(args));
+ ProcessBuilder builder = new ProcessBuilder(command);
+ builder.environment().put("JAVA_HOME", System.getProperty("java.home"));
+ builder.environment().put("CLASSPATH",
+ System.getProperty("surefire.test.class.path", System.getProperty("java.class.path")));
+ builder.environment().put("JVMFLAGS", flags);
+ return runProcess(builder, "");
+ }
+
+ private Result runProcess(ProcessBuilder builder, String input) throws Exception {
+ File outputFile = new File(directory, "output-" + ++nextFile);
+ File inputFile = new File(directory, "input-" + nextFile);
+ Files.write(inputFile.toPath(), input.getBytes(StandardCharsets.UTF_8));
+ Process process = builder
+ .redirectInput(inputFile)
+ .redirectErrorStream(true)
+ .redirectOutput(outputFile)
+ .start();
+ try {
+ assertTrue("CLI did not exit: " + builder.command(), process.waitFor(30, TimeUnit.SECONDS));
+ return new Result(process.exitValue(),
+ new String(Files.readAllBytes(outputFile.toPath()), StandardCharsets.UTF_8));
+ } finally {
+ if (process.isAlive()) {
+ process.destroyForcibly();
+ }
+ }
+ }
+
+ void close() throws IOException {
+ FileUtils.deleteDirectory(directory);
+ }
+
+ static class Result {
+
+ final int exitCode;
+ final String output;
+
+ Result(int exitCode, String output) {
+ this.exitCode = exitCode;
+ this.output = output;
+ }
+
+ String success() {
+ assertEquals(output, ExitCode.EXECUTION_FINISHED.getValue(), exitCode);
+ return output;
+ }
+
+ void invalidInvocation() {
+ assertEquals(output, ExitCode.INVALID_INVOCATION.getValue(), exitCode);
+ assertThat(output.toLowerCase(), containsString("usage:"));
+ }
+
+ void readFailure() {
+ assertEquals(output, ExitCode.UNEXPECTED_ERROR.getValue(), exitCode);
+ assertThat(output, containsString("Unable to read snapshot"));
+ }
+ }
+ }
+
+}
diff --git a/zookeeper-server/src/test/java/org/apache/zookeeper/server/SnapshotRecursiveSummaryTest.java b/zookeeper-server/src/test/java/org/apache/zookeeper/server/SnapshotRecursiveSummaryTest.java
new file mode 100644
index 00000000000..2e326ee1fc6
--- /dev/null
+++ b/zookeeper-server/src/test/java/org/apache/zookeeper/server/SnapshotRecursiveSummaryTest.java
@@ -0,0 +1,198 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.zookeeper.server;
+
+import static org.hamcrest.CoreMatchers.containsString;
+import static org.hamcrest.CoreMatchers.not;
+import static org.junit.Assert.assertThat;
+import java.io.File;
+import java.io.IOException;
+import java.util.Arrays;
+import org.apache.zookeeper.ZKTestCase;
+import org.apache.zookeeper.ZooDefs;
+import org.apache.zookeeper.server.SnapshotComparerTest.SnapshotToolTestSupport;
+import org.apache.zookeeper.server.persistence.SnapStream.StreamMode;
+import org.junit.After;
+import org.junit.Assume;
+import org.junit.Before;
+import org.junit.Test;
+
+public class SnapshotRecursiveSummaryTest extends ZKTestCase {
+
+ private SnapshotToolTestSupport support;
+
+ @Before
+ public void setUp() throws IOException {
+ support = new SnapshotToolTestSupport();
+ }
+
+ @After
+ public void tearDown() throws IOException {
+ support.close();
+ }
+
+ @Test
+ public void testRecursiveSubtreeTotals() throws Exception {
+ String output = summarize(tree(), "/app", "0");
+
+ assertThat(output, containsString(" /app\n children: 5\n data: 17"));
+ assertThat(output, containsString("-- /app/branch\n-- children: 2\n-- data: 8"));
+ assertThat(output, containsString("---- /app/branch/deep\n---- children: 1\n---- data: 5"));
+ assertThat(output, not(containsString(" /app/leaf\n")));
+ assertThat(output, not(containsString(" /app/empty\n")));
+ assertThat(output, not(containsString(" /zookeeper\n")));
+ }
+
+ @Test
+ public void testDepthOnlyLimitsOutput() throws Exception {
+ String output = summarize(tree(), "/app", "1");
+
+ assertThat(output, containsString(" /app\n children: 5\n data: 17"));
+ assertThat(output, containsString("-- /app/branch\n-- children: 2\n-- data: 8"));
+ assertThat(output, not(containsString(" /app/branch/deep\n")));
+ }
+
+ @Test
+ public void testRootIncludesItsOwnData() throws Exception {
+ DataTree tree = tree();
+ tree.setData("/", new byte[11], 1, 1, 1);
+
+ String output = summarize(tree, "/", "1");
+
+ assertThat(output, containsString(" /\n children: 9\n data: 28"));
+ assertThat(output, containsString("-- /app\n-- children: 5\n-- data: 17"));
+ assertThat(output, containsString("-- /zookeeper\n-- children: 2\n-- data: 0"));
+ }
+
+ @Test
+ public void testSelectedLeafIsNotPrinted() throws Exception {
+ String output = summarize(tree(), "/app/leaf", "0");
+
+ assertThat(output, not(containsString("children:")));
+ assertThat(output, not(containsString("data:")));
+ }
+
+ @Test
+ public void testNullDataContributesZeroBytes() throws Exception {
+ DataTree tree = new DataTree();
+ support.addNode(tree, "/app", null);
+ support.addNode(tree, "/app/leaf", null);
+
+ String output = summarize(tree, "/app", "0");
+
+ assertThat(output, containsString(" /app\n children: 1\n data: 0"));
+ }
+
+ @Test
+ public void testEphemeralDescendantsAreIncluded() throws Exception {
+ DataTree tree = new DataTree();
+ support.addNode(tree, "/app", new byte[2]);
+ tree.createNode("/app/ephemeral", new byte[7], ZooDefs.Ids.OPEN_ACL_UNSAFE, 0x123L, -1, 1, 1);
+
+ String output = summarize(tree, "/app", "0");
+
+ assertThat(output, containsString(" /app\n children: 1\n data: 9"));
+ }
+
+ @Test
+ public void testCompressedSnapshots() throws Exception {
+ for (StreamMode mode : StreamMode.values()) {
+ File snapshot = support.snapshot(tree(), mode, true);
+ String output = run(snapshot.toString(), "/app", "0").success();
+ assertThat(output, containsString(" /app\n children: 5\n data: 17"));
+ }
+ }
+
+ @Test
+ public void testSnapshotWithoutDigest() throws Exception {
+ File snapshot = support.snapshot(tree(), StreamMode.CHECKED, false);
+
+ String output = run(snapshot.toString(), "/app", "0").success();
+
+ assertThat(output, containsString(" /app\n children: 5\n data: 17"));
+ }
+
+ @Test
+ public void testInvalidArguments() throws Exception {
+ File snapshot = support.snapshot(tree(), StreamMode.CHECKED, true);
+ run().invalidInvocation();
+ run(snapshot.toString(), "/app").invalidInvocation();
+ run(snapshot.toString(), "/app", "0", "extra").invalidInvocation();
+ for (String invalid : Arrays.asList("-1", "NaN", "2147483648")) {
+ run(snapshot.toString(), "/app", invalid).invalidInvocation();
+ }
+ }
+
+ @Test
+ public void testInvalidZnodePaths() throws Exception {
+ File snapshot = support.snapshot(tree(), StreamMode.CHECKED, true);
+ for (String invalid : Arrays.asList("", "app", "/app/", "/app//leaf", "/app/..", "/missing")) {
+ SnapshotToolTestSupport.Result result = run(snapshot.toString(), invalid, "0");
+ result.invalidInvocation();
+ assertThat(result.output, not(containsString("NullPointerException")));
+ }
+ }
+
+ @Test
+ public void testInvalidFiles() throws Exception {
+ run(new File(support.directory, "missing").toString(), "/", "0").invalidInvocation();
+ run(support.directory.toString(), "/", "0").invalidInvocation();
+ }
+
+ @Test
+ public void testCorruptSnapshots() throws Exception {
+ for (File corrupt : support.corruptSnapshots()) {
+ SnapshotToolTestSupport.Result result = run(corrupt.toString(), "/", "0");
+ result.readFailure();
+ assertThat(result.output, not(containsString("children:")));
+ }
+ }
+
+ @Test
+ public void testShellLauncher() throws Exception {
+ Assume.assumeFalse(System.getProperty("os.name").startsWith("Windows"));
+ File snapshot = support.snapshot(tree(), StreamMode.GZIP, true);
+ for (String flags : Arrays.asList("", "-Xms32m -Xmx128m")) {
+ String output = support.runLauncher("zkSnapshotRecursiveSummaryToolkit.sh", flags,
+ snapshot.toString(), "/app", "0").success();
+ assertThat(output, containsString(" /app\n children: 5\n data: 17"));
+ }
+ }
+
+ private DataTree tree() throws Exception {
+ DataTree tree = new DataTree();
+ support.addNode(tree, "/app", new byte[2]);
+ support.addNode(tree, "/app/branch", new byte[3]);
+ support.addNode(tree, "/app/branch/deep", null);
+ support.addNode(tree, "/app/branch/deep/leaf", new byte[5]);
+ support.addNode(tree, "/app/empty", null);
+ support.addNode(tree, "/app/leaf", new byte[7]);
+ return tree;
+ }
+
+ private String summarize(DataTree tree, String path, String depth) throws Exception {
+ File snapshot = support.snapshot(tree, StreamMode.CHECKED, true);
+ return run(snapshot.toString(), path, depth).success();
+ }
+
+ private SnapshotToolTestSupport.Result run(String... args) throws Exception {
+ return support.run("org.apache.zookeeper.server.SnapshotRecursiveSummary", "", args);
+ }
+
+}
From e0adf9f30fb94560010881f6c8d78976a68fe89e Mon Sep 17 00:00:00 2001
From: Arkesh Mishra <118651144+arkmish@users.noreply.github.com>
Date: Fri, 25 Sep 2026 15:07:42 +0530
Subject: [PATCH 2/9] Add validated snapshot-only metadata export (D2)
Stream versioned node, namespace and true-session metadata without values or credential-bearing ACL IDs. Validate the selected source, enforce optional output budgets and publish a checksummed manifest atomically last.
Reuse the D1 native snapshot loader and expose iterative preorder traversal without changing the summary CLI. Keep capture and owner uncertainty explicit; do not replay transaction logs or infer recovery completeness.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---
bin/zkOfflineAudit.cmd | 23 +
bin/zkOfflineAudit.sh | 29 +
.../main/resources/markdown/zookeeperTools.md | 267 ++++-
.../server/OfflineAuditExporter.java | 653 ++++++++++++
.../zookeeper/server/SnapshotComparer.java | 68 +-
.../server/SnapshotRecursiveSummary.java | 109 +-
.../server/OfflineAuditExporterTest.java | 939 ++++++++++++++++++
7 files changed, 2056 insertions(+), 32 deletions(-)
create mode 100644 bin/zkOfflineAudit.cmd
create mode 100755 bin/zkOfflineAudit.sh
create mode 100644 zookeeper-server/src/main/java/org/apache/zookeeper/server/OfflineAuditExporter.java
create mode 100644 zookeeper-server/src/test/java/org/apache/zookeeper/server/OfflineAuditExporterTest.java
diff --git a/bin/zkOfflineAudit.cmd b/bin/zkOfflineAudit.cmd
new file mode 100644
index 00000000000..b8adc0e3d3f
--- /dev/null
+++ b/bin/zkOfflineAudit.cmd
@@ -0,0 +1,23 @@
+@echo off
+REM Licensed to the Apache Software Foundation (ASF) under one or more
+REM contributor license agreements. See the NOTICE file distributed with
+REM this work for additional information regarding copyright ownership.
+REM The ASF licenses this file to You under the Apache License, Version 2.0
+REM (the "License"); you may not use this file except in compliance with
+REM the License. You may obtain a copy of the License at
+REM
+REM http://www.apache.org/licenses/LICENSE-2.0
+REM
+REM Unless required by applicable law or agreed to in writing, software
+REM distributed under the License is distributed on an "AS IS" BASIS,
+REM WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+REM See the License for the specific language governing permissions and
+REM limitations under the License.
+
+setlocal
+call "%~dp0zkEnv.cmd"
+
+set ZOOMAIN=org.apache.zookeeper.server.OfflineAuditExporter
+call %JAVA% -cp "%CLASSPATH%" %JVMFLAGS% %ZOOMAIN% %*
+
+endlocal & exit /b %ERRORLEVEL%
diff --git a/bin/zkOfflineAudit.sh b/bin/zkOfflineAudit.sh
new file mode 100755
index 00000000000..94625b018b3
--- /dev/null
+++ b/bin/zkOfflineAudit.sh
@@ -0,0 +1,29 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+ZOOBIN="${BASH_SOURCE-$0}"
+ZOOBIN="$(dirname "${ZOOBIN}")"
+ZOOBINDIR="$(cd "${ZOOBIN}"; pwd)"
+
+if [ -e "$ZOOBIN/../libexec/zkEnv.sh" ]; then
+ . "$ZOOBINDIR"/../libexec/zkEnv.sh
+else
+ . "$ZOOBINDIR"/zkEnv.sh
+fi
+
+"$JAVA" -cp "$CLASSPATH" $JVMFLAGS \
+ org.apache.zookeeper.server.OfflineAuditExporter "$@"
diff --git a/zookeeper-docs/src/main/resources/markdown/zookeeperTools.md b/zookeeper-docs/src/main/resources/markdown/zookeeperTools.md
index 7fc685d9e8f..0fa468e110a 100644
--- a/zookeeper-docs/src/main/resources/markdown/zookeeperTools.md
+++ b/zookeeper-docs/src/main/resources/markdown/zookeeperTools.md
@@ -25,6 +25,7 @@ limitations under the License.
* [zkSnapShotToolkit.sh](#zkSnapShotToolkit)
* [zkSnapshotComparer.sh](#zkSnapshotComparer)
* [zkSnapshotRecursiveSummaryToolkit.sh](#zkSnapshotRecursiveSummaryToolkit)
+ * [zkOfflineAudit.sh](#zkOfflineAudit)
* [Testing](#Testing)
* [Jepsen Test](#jepsen-test)
@@ -304,7 +305,271 @@ Neither tool compares payload contents, ACLs, versions or other znode metadata.
**Equal sizes/counts do not prove identical contents.** Snapshots may be fuzzy:
these tools do not replay transaction logs, reconstruct point-in-time state,
or establish transaction-consistent equality. The tools load snapshots into
-memory, and recursive traversal still visits the full selected subtree.
+memory, and traversal still visits the full selected subtree.
+
+
+
+### zkOfflineAudit.sh
+
+Export versioned, non-value metadata from **one explicitly selected snapshot**.
+The Windows equivalent is `bin/zkOfflineAudit.cmd`; the Java entry point is
+`org.apache.zookeeper.server.OfflineAuditExporter`.
+
+```bash
+bin/zkOfflineAudit.sh --snapshot-file snapshot.100000001.gz --output-dir audit-run
+```
+
+The output directory **must not already exist**, even if empty, and its parent
+must exist. Use a new, owned directory for each invocation. Existing files,
+directories, hard links and symbolic links at the destination are rejected
+without modifying them. Filesystem path resolution preserves symbolic-link
+parent semantics; it does not lexically remove `..` and select a different file.
+The tool never renames, rewrites or deletes the source, connects to a ZooKeeper
+server, replays transaction logs, or selects a fallback snapshot.
+
+The shared native reader accepts format-version 2 snapshots, uncompressed or
+with the native `.gz`/`.snappy` suffix. It checks the header, body seal and any
+optional digest seal, rejects partial trailers, invalid roots and duplicate node
+records, and consumes the decoded stream to its end. A digest is recorded as metadata, **not** compared
+as a transaction-consistent recovery digest. Fuzzy snapshots do not need
+transaction-log coverage for this export.
+
+#### Resources, publication and failures
+
+The native reader loads one complete `DataTree`, its ACL cache and session
+indexes. Budget heap for the **decoded** tree, not just the compressed file or
+stored value bytes. The exporter adds an iterative traversal stack, sorted child
+name references for the active ancestors, one JSON record at a time, and a copy
+of one session's ephemeral path set at a time. It does not build a second tree
+or retain all output records. Native quota initialization can still recurse
+during loading; an unusually deep quota tree can exhaust the JVM stack.
+
+Output can exceed snapshot size substantially: every node includes full paths
+and metadata, and deep paths are repeated. Disk policy must also allow filesystem
+allocation overhead. The optional `--max-output-bytes BYTES` is a **positive,
+decimal signed-64-bit integer** bounding the combined UTF-8 bytes of the three
+NDJSON files and the manifest, including newlines. No production budget is
+assumed when it is absent. This is not an input-size, heap, stack, runtime or
+filesystem-block quota. Supply approved worker budgets and JVM limits through
+the existing launcher environment; do not infer safe limits from this example.
+
+The tool creates a private directory (mode 0700 on POSIX), creates output files
+without replacement, closes and forces the data files, stages the manifest,
+rechecks source identity, and **atomically publishes `manifest.json` last**.
+Atomic move support is required; there is no non-atomic fallback. Source checks
+compare SHA-256 of the original file bytes, resolved path, size, precise mtime,
+creation time, available file key and Unix change time where supported, including
+stat checks around hashing. Access time is deliberately not compared because
+reading may update it. Use immutable, owned input copies: these checks are not
+a lock against concurrent writers, and identity/time precision depends on the
+filesystem.
+
+Exit 0 means publication succeeded. Invalid arguments, decoder properties or an
+existing destination exit 2 with usage; snapshot, source-change, output-budget
+and I/O failures exit 1 with diagnostics. JVM resource errors remain visible
+and nonzero rather than becoming an empty-database success. A failed started
+export can leave partial files and `.manifest.json.inprogress` for diagnosis;
+**neither is a published result** and the directory cannot be reused. An
+existing destination is rejected *before an export is attempted*: its prior
+manifest is left untouched, not relabeled as this invocation's result. Consumers
+must require a successful invocation into their unique run directory, read only
+`manifest.json`, and verify its source identity, schema, file counts and hashes.
+The file descriptors omit the manifest itself to avoid a checksum cycle.
+
+#### Decoder configuration and owner uncertainty
+
+Use JVM system properties (for example through `JVMFLAGS`) only when the source
+settings are known:
+
+* `zookeeper.extendedTypesEnabled`: `true` or `false`, case-insensitive.
+* `zookeeper.emulate353TTLNodes`: `true` or `false`, case-insensitive. Explicit
+ `true` requires explicit `zookeeper.extendedTypesEnabled=true`.
+* `jute.maxbuffer`: positive 32-bit integer in the native Java numeric-property
+ syntax (`Integer.decode`: decimal, `0x`/`#` hexadecimal or leading-zero octal).
+ When absent, the native `BinaryInputArchive.maxBuffer` default applies.
+* `zookeeper.jute.maxbuffer.extrasize`: non-negative 32-bit integer in the same
+ syntax. When absent it defaults to `jute.maxbuffer`; the native minimum extra
+ padding of 1024 bytes is applied. The effective sum must fit a signed 32-bit
+ length. These decoder bounds are not estimates of server response limits.
+
+Malformed properties are rejected instead of silently using Java's fallback.
+Compression is detected by the selected file's suffix, not the output
+compression system property. The optional digest is read regardless of the
+local digest-calculation setting. Arbitrary JVM properties are not exported.
+
+Snapshots do not persist the source's type settings. Absent source properties
+are therefore `null` in the manifest, although the native decoder's effective
+boolean defaults are false. Owner 0 is persistent, `0x8000000000000000` is a
+container, and nonzero sign-bit-clear owners are normal ephemerals. Other owners
+are classified as follows:
+
+| Known source configuration | Classification of other negative owners |
+| --- | --- |
+| Extended types unspecified | `unknown`; never grouped as sessions |
+| Extended types explicitly false | Native normal ephemeral/session |
+| Extended types true, emulation unspecified | `unknown`; legacy interpretation is unresolved |
+| Extended types true, emulation false | Native modern TTL or normal ephemeral; unsupported extended feature bits fail |
+| Extended types true, emulation true | Native 3.5.3-emulated TTL |
+
+`ttl_ms` uses this decoder's `EphemeralType.TTL.getValue(owner)` (low 40 bits),
+including emulation; it does not reconstruct a historical implementation's
+different value mask. It is a duration, **not** a calculated expiry time.
+Neither an owner in `sessions.ndjson` nor membership in the snapshot session
+table proves a currently live session.
+
+#### Schema version 1
+
+All files are UTF-8. Each NDJSON line is one complete JSON object; `manifest.json`
+is one JSON object. `null` means unknown/not applicable, not zero. `integer`
+below means a JSON integral number (64-bit arithmetic for counts, lengths,
+byte totals and times); schema/format/digest versions, node version counters,
+session timeouts and quota counts are 32-bit integers. No field is a JVM-memory
+estimate. **All 64-bit zxids, owner/session identifiers and digest values are
+lowercase, zero-padded, 16-digit hex strings prefixed with `0x`**, preserving
+their unsigned bit patterns. SHA-256 values are 64 lowercase hex digits without
+a prefix. Consumers must not convert identifiers to floating-point numbers.
+
+**`nodes.ndjson`** includes the root exactly once as `/`, all customer nodes and
+the reserved subtree. Records are emitted in deterministic depth-first
+**preorder** with sorted children. Each parent's descendants are contiguous,
+supporting streaming subtree intervals. This is not a global lexical path sort:
+`/a/x` follows `/a` before the traversal advances to `/a-`.
+
+| Field | Type | Exact meaning |
+| --- | --- | --- |
+| `schema_version` | integer | `1` |
+| `path` | string | Full absolute decoded znode path; root is `/`, never `""` |
+| `data_length` | integer | This decoded node's payload length in bytes; null data contributes 0 |
+| `num_children` | integer | Number of **immediate** children, not descendants |
+| `getchildren_response_bytes` | integer | Full successful getChildren reply, defined below |
+| `getchildren2_response_bytes` | integer | Same reply plus the 68-byte Stat |
+| `ctime_ms`, `mtime_ms` | integer | Stored node creation/data-modification timestamps, epoch milliseconds |
+| `czxid`, `mzxid`, `pzxid` | hex string | Stored creation, data-modification and child-modification zxids |
+| `data_version`, `acl_version` | integer | Persisted data/ACL version counters |
+| `persisted_cversion` | integer | Raw persisted child-create counter; not the client Stat cversion conversion |
+| `path_utf8_bytes` | integer | Full absolute path's UTF-8 length, including leading slash, without a length prefix |
+| `node_type` | string | `persistent`, `ephemeral`, `container`, `ttl` or `unknown` |
+| `ephemeral_owner` | hex string | Raw persisted owner bits, including 0, TTL/container and unknown encodings |
+| `owner_encoding` | string | `none`, `session`, `container`, `ttl`, `ttl-3.5.3` or `unknown` |
+| `ttl_ms` | integer or null | Native decoded TTL duration for known TTL types only |
+| `acl_risk_flags` | array of strings | Non-secret review hints below; empty means no listed hint, not proven safe |
+
+The getChildren size is
+`16 + 4 + sum(4 + UTF-8(child-name).length)`: the 16-byte ReplyHeader, four-byte
+vector count, and four-byte length plus bytes for each **child name**, not full
+path. getChildren2 adds 68 bytes. Both **exclude** the outer four-byte framing
+prefix and transport/TLS overhead. Empty vectors are 20/88 bytes; names `é`, `a`
+are 31/99 bytes. Counts use `long` arithmetic, independent of any packet limit.
+Actual Jute serialization agrees for valid znode names. This branch's older
+Jute encoder treats surrogate pairs differently from standard UTF-8, but native
+`PathUtils` prohibits those characters in znode paths; malformed decoded paths
+are rejected, not silently relabeled.
+
+ACL hints do not disclose ACL scheme strings or IDs, change permissions, or
+perform authentication/authorization:
+
+| Flag | Meaning |
+| --- | --- |
+| `world_read` | `world:anyone` has READ |
+| `world_write` | `world:anyone` has any of WRITE, CREATE or DELETE |
+| `world_admin` | `world:anyone` has ADMIN |
+| `auth` | An `auth` scheme entry remains in the decoded ACL |
+| `unknown_scheme` | Scheme outside the known set `world`, `auth`, `digest`, `ip`, `sasl`, `x509`; custom providers require review |
+| `missing_acl` | Decoded ACL list is null or empty |
+| `invalid_permissions` | Permission bits outside `ZooDefs.Perms.ALL` |
+| `invalid_identity` | Missing identity/scheme/ID, or a `world` identity other than `anyone` |
+
+Native deserialization can reject malformed ACL structures before flags can be
+produced. Node values and raw ACL identities are never written to the export.
+Paths themselves can be sensitive; protect the output as metadata.
+
+**`namespaces.ndjson`** contains **every** immediate root child except exactly
+`/zookeeper`, including leaves and `/zookeeper-client`. Counts/bytes include the
+namespace node and all descendants of every node type. Root data is not assigned
+to a namespace. Records follow traversal order.
+
+| Field | Type | Exact meaning |
+| --- | --- | --- |
+| `schema_version` | integer | `1` |
+| `path` | string | Top-level customer namespace path |
+| `subtree_node_count` | integer | Inclusive subtree node count |
+| `payload_bytes` | integer | Sum of decoded payload lengths; null contributes 0 |
+| `path_utf8_bytes` | integer | Sum of full absolute path lengths in UTF-8 |
+| `quota_status` | string | `absent`, `valid` or `invalid` for the exact namespace limit znode |
+| `quota_limit_count` | integer or null | Parsed `count`, including -1 (unset); null if absent/invalid |
+| `quota_limit_bytes` | integer or null | Parsed `bytes`, including -1 (unset); null if absent/invalid |
+| `quota_stat_present` | boolean | Exact namespace quota stat znode exists; not evidence of freshness |
+
+Quota availability refers specifically to
+`/zookeeper/quota/zookeeper_limits` and `zookeeper_stats`; it does not
+claim absence of quotas on descendants or model inherited enforcement. A valid
+limit has the native `count=,bytes=` format with values at least -1.
+Malformed limits are flagged without exporting their raw value. Native DataTree
+loading **rebuilds quota-stat payloads in memory**; node lengths under the
+reserved quota subtree describe that decoded tree, not necessarily the original
+stat-node payload lengths. Customer subtree totals are computed independently
+from the decoded customer nodes; no rebuilt quota count is trusted for them.
+
+**`sessions.ndjson`** has one record per owner with decoded normal ephemerals.
+It excludes containers, known TTLs, unknown encodings and session-table entries
+with no normal ephemeral nodes. Session ordering is unspecified; records are
+streamed from the native index, including ephemerals in the reserved subtree.
+
+| Field | Type | Exact meaning |
+| --- | --- | --- |
+| `schema_version` | integer | `1` |
+| `session_id` | hex string | Normal ephemeral owner under the known/unambiguous decoder interpretation |
+| `ephemeral_node_count` | integer | Number of nodes owned by that session |
+| `payload_bytes` | integer | Sum of those nodes' payload lengths |
+| `path_utf8_bytes` | integer | Sum of those nodes' full UTF-8 path lengths |
+| `present_in_snapshot_session_table` | boolean | Owner also appears in the serialized session/timeout table |
+| `timeout_ms` | integer or null | Serialized timeout if present; null otherwise, not a remaining lifetime |
+
+**`manifest.json`** is the sole publication marker. Export completeness means
+complete traversal/output of this decoded snapshot, **not** current-state,
+backup completeness or transaction-consistent recovery.
+
+| Field | Type | Exact meaning |
+| --- | --- | --- |
+| `schema_version` | integer | `1` |
+| `tool` | object | `name` = `OfflineAuditExporter`; `version` = native `Version.getFullVersion()` string |
+| `export_id` | string | New UUID identifying this published export |
+| `export_complete` | boolean | `true` in a successfully published manifest only |
+| `recovery_scope` | string | Always `snapshot-only` |
+| `transaction_logs_replayed` | boolean | Always `false` |
+| `source_capture_time_ms` | null | Capture time unknown; never inferred from file or node timestamps |
+| `source_capture_provenance` | null | No authoritative capture provenance supplied to this CLI |
+| `source_server_version` | null | Snapshot header version does not identify the source server release |
+| `source` | object | Source descriptor defined below |
+| `decoder` | object | Decoder settings defined below |
+| `snapshot_zxid` | hex string or null | Label parsed only from `snapshot.<1..16 hex digits>[.gz or .snappy]`; renamed/range filenames yield null; not a consistent endpoint |
+| `last_observed_zxid` | hex string | Unsigned maximum of all exported nodes' `czxid`, `mzxid`, `pzxid`; excludes filename/digest zxids |
+| `last_observed_zxid_scope` | string | Always `maximum-node-czxid-mzxid-pzxid-unsigned` |
+| `snapshot_digest` | object or null | Optional stored digest metadata, not recovery evidence |
+| `unknown_owner_nodes` | integer | Count of node records with `node_type=unknown` |
+| `max_output_bytes` | integer or null | Requested output byte budget, or no CLI budget |
+| `files` | array of objects | Exactly three output descriptors, in nodes/namespaces/sessions order |
+
+Nested objects have these exact fields:
+
+* `source`: `path` (resolved absolute filename string), `sha256` (hash of the
+ original **compressed bytes**, if compressed), `size_bytes` (integer),
+ `mtime_ms` (integer epoch milliseconds), `change_time_ms` (Unix inode-change
+ time in epoch milliseconds, or null if unavailable), `compression` (`checked`
+ = uncompressed, `gzip` or `snappy`), `format_version` (integer, currently 2),
+ `database_id` (raw header dbid as a hex string). None of the file times is
+ snapshot capture time.
+* `decoder`: `source_extended_types_enabled` and `source_emulate_353_ttl_nodes`
+ (boolean or null); `effective_extended_types_enabled` and
+ `effective_emulate_353_ttl_nodes` (boolean); `jute_maxbuffer` and
+ `jute_extra_maxbuffer` (integers after native default/minimum handling).
+* `snapshot_digest`, when present: `zxid` (hex string), `version` (integer),
+ `value` (hex string), `seal_validated` (true),
+ `transaction_consistency_verified` (false). A native zero/empty digest is
+ retained as zero, not promoted to completeness evidence.
+* Each `files` entry: `name` (exactly `nodes.ndjson`, `namespaces.ndjson` or
+ `sessions.ndjson`), `records` (integer line count), `size_bytes` (integer
+ physical file length), `sha256` (hash of all file bytes, including newlines).
diff --git a/zookeeper-server/src/main/java/org/apache/zookeeper/server/OfflineAuditExporter.java b/zookeeper-server/src/main/java/org/apache/zookeeper/server/OfflineAuditExporter.java
new file mode 100644
index 00000000000..ac2fbaebe0d
--- /dev/null
+++ b/zookeeper-server/src/main/java/org/apache/zookeeper/server/OfflineAuditExporter.java
@@ -0,0 +1,653 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.zookeeper.server;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import java.io.BufferedOutputStream;
+import java.io.Closeable;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.nio.channels.Channels;
+import java.nio.channels.FileChannel;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.LinkOption;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.nio.file.StandardCopyOption;
+import java.nio.file.StandardOpenOption;
+import java.nio.file.attribute.BasicFileAttributes;
+import java.nio.file.attribute.FileTime;
+import java.nio.file.attribute.PosixFilePermissions;
+import java.security.DigestOutputStream;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.Set;
+import java.util.UUID;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import org.apache.commons.cli.CommandLine;
+import org.apache.commons.cli.DefaultParser;
+import org.apache.commons.cli.HelpFormatter;
+import org.apache.commons.cli.Option;
+import org.apache.commons.cli.Options;
+import org.apache.commons.cli.ParseException;
+import org.apache.jute.BinaryInputArchive;
+import org.apache.zookeeper.Quotas;
+import org.apache.zookeeper.StatsTrack;
+import org.apache.zookeeper.Version;
+import org.apache.zookeeper.ZKUtil;
+import org.apache.zookeeper.ZooDefs;
+import org.apache.zookeeper.data.ACL;
+import org.apache.zookeeper.data.Id;
+import org.apache.zookeeper.data.StatPersisted;
+import org.apache.zookeeper.server.SnapshotComparer.SnapshotData;
+import org.apache.zookeeper.server.persistence.SnapStream;
+import org.apache.zookeeper.util.ServiceUtils;
+
+/**
+ * Streams non-value metadata from one validated snapshot. No transaction logs are
+ * replayed and neither the snapshot filename nor its digest establishes capture time
+ * or transaction-consistent state. See zookeeperTools.md for the versioned schema.
+ */
+public final class OfflineAuditExporter {
+
+ private static final int SCHEMA_VERSION = 1;
+ private static final ObjectMapper JSON = new ObjectMapper();
+ private static final Pattern SNAPSHOT_NAME = Pattern.compile("snapshot\\.([0-9a-fA-F]{1,16})(?:\\.(?:gz|snappy))?");
+ private static final Pattern QUOTA_VALUE = Pattern.compile("count=-?[0-9]+,bytes=-?[0-9]+");
+ private static final Set ACL_SCHEMES = new HashSet<>(Arrays.asList("world", "auth", "digest", "ip", "sasl", "x509"));
+
+ private OfflineAuditExporter() {
+ }
+
+ public static void main(String[] args) {
+ Options options = new Options();
+ options.addOption(Option.builder().longOpt("snapshot-file").hasArg().required().argName("SNAPSHOT")
+ .desc("Read only this snapshot file.").build());
+ options.addOption(Option.builder().longOpt("output-dir").hasArg().required().argName("DIRECTORY")
+ .desc("Create a new directory; its parent must already exist.").build());
+ options.addOption(Option.builder().longOpt("max-output-bytes").hasArg().argName("BYTES")
+ .desc("Optional positive byte budget, including the manifest. No default policy limit.").build());
+ Path source;
+ Path output;
+ Long limit;
+ DecoderSettings decoder;
+ try {
+ CommandLine command = new DefaultParser().parse(options, args);
+ if (!command.getArgList().isEmpty()) {
+ throw new IllegalArgumentException("Unexpected positional arguments");
+ }
+ Set supplied = new HashSet<>();
+ for (Option option : command.getOptions()) {
+ if (!supplied.add(option.getLongOpt())) {
+ throw new IllegalArgumentException("Repeated option: --" + option.getLongOpt());
+ }
+ }
+ limit = command.hasOption("max-output-bytes")
+ ? Long.valueOf(command.getOptionValue("max-output-bytes")) : null;
+ if (limit != null && limit <= 0) {
+ throw new IllegalArgumentException("max-output-bytes must be a positive 64-bit integer");
+ }
+ decoder = DecoderSettings.read();
+ source = Paths.get(command.getOptionValue("snapshot-file")).toAbsolutePath();
+ String error = ZKUtil.validateFileInput(source.toString());
+ if (error != null) {
+ throw new IllegalArgumentException(error);
+ }
+ output = Paths.get(command.getOptionValue("output-dir")).toAbsolutePath();
+ if (Files.exists(output, LinkOption.NOFOLLOW_LINKS)) {
+ throw new IllegalArgumentException("Output already exists; no export attempted and prior files left untouched: " + output);
+ }
+ Path parent = output.getParent();
+ if (parent == null || !Files.isDirectory(parent)) {
+ throw new IllegalArgumentException("Output parent must be an existing directory: " + output);
+ }
+ output = parent.toRealPath().resolve(fileName(output));
+ } catch (ParseException | IllegalArgumentException | IOException e) {
+ System.err.println(e.getMessage());
+ new HelpFormatter().printHelp("zkOfflineAudit.sh", options, true);
+ ServiceUtils.requestSystemExit(ExitCode.INVALID_INVOCATION.getValue());
+ return;
+ }
+ try {
+ export(source, output, decoder, limit);
+ System.out.println("Snapshot-only export complete: " + output);
+ } catch (IOException | IllegalArgumentException | ArithmeticException e) {
+ System.err.println("Unable to export snapshot: " + e.getMessage());
+ ServiceUtils.requestSystemExit(ExitCode.UNEXPECTED_ERROR.getValue());
+ }
+ }
+
+ /**
+ * Returns the entire successful getChildren/getChildren2 reply packet size,
+ * including the 16-byte ReplyHeader but excluding the outer four-byte frame.
+ */
+ public static long childResponseBytes(Iterable children, boolean includeStat) {
+ long bytes = includeStat ? 88L : 20L;
+ for (String child : children) {
+ bytes = Math.addExact(bytes, 4L + child.getBytes(StandardCharsets.UTF_8).length);
+ }
+ return bytes;
+ }
+
+ private static void export(Path source, Path output, DecoderSettings decoder, Long limit) throws IOException {
+ SourceIdentity identity = SourceIdentity.read(source);
+ SnapshotData snapshot = SnapshotComparer.readSnapshot(identity.path.toFile());
+ if (output.getFileSystem().supportedFileAttributeViews().contains("posix")) {
+ Files.createDirectory(output, PosixFilePermissions.asFileAttribute(PosixFilePermissions.fromString("rwx------")));
+ } else {
+ Files.createDirectory(output);
+ }
+ OutputBudget budget = new OutputBudget(limit);
+ ExportRecords records = new ExportRecords(snapshot, decoder);
+ List> files = new ArrayList<>();
+ try (RecordWriter nodes = new RecordWriter(output.resolve("nodes.ndjson"), budget);
+ RecordWriter namespaces = new RecordWriter(output.resolve("namespaces.ndjson"), budget);
+ RecordWriter sessions = new RecordWriter(output.resolve("sessions.ndjson"), budget)) {
+ records.write(nodes, namespaces, sessions);
+ files.add(nodes.finish());
+ files.add(namespaces.finish());
+ files.add(sessions.finish());
+ }
+
+ Map manifest = record();
+ manifest.put("tool", fields("name", "OfflineAuditExporter", "version", Version.getFullVersion()));
+ manifest.put("export_id", UUID.randomUUID().toString());
+ manifest.put("export_complete", true);
+ manifest.put("recovery_scope", "snapshot-only");
+ manifest.put("transaction_logs_replayed", false);
+ manifest.put("source_capture_time_ms", null);
+ manifest.put("source_capture_provenance", null);
+ manifest.put("source_server_version", null);
+ manifest.put("source", identity.metadata(snapshot));
+ manifest.put("decoder", decoder.metadata());
+ manifest.put("snapshot_zxid", snapshotZxid(identity.path));
+ manifest.put("last_observed_zxid", hex(records.lastObservedZxid));
+ manifest.put("last_observed_zxid_scope", "maximum-node-czxid-mzxid-pzxid-unsigned");
+ manifest.put("snapshot_digest", snapshot.digest == null ? null
+ : fields("zxid", hex(snapshot.digest.getZxid()), "version", snapshot.digest.getDigestVersion(),
+ "value", hex(snapshot.digest.getDigest()), "seal_validated", true,
+ "transaction_consistency_verified", false));
+ manifest.put("unknown_owner_nodes", records.unknownOwnerNodes);
+ manifest.put("max_output_bytes", limit);
+ manifest.put("files", files);
+ Path pending = output.resolve(".manifest.json.inprogress");
+ try (RecordWriter writer = new RecordWriter(pending, budget)) {
+ writer.write(manifest);
+ writer.finish();
+ }
+ identity.verify();
+ Files.move(pending, output.resolve("manifest.json"), StandardCopyOption.ATOMIC_MOVE);
+ }
+
+ private enum NodeType {
+ PERSISTENT("persistent", "none"),
+ EPHEMERAL("ephemeral", "session"),
+ CONTAINER("container", "container"),
+ TTL("ttl", "ttl"),
+ TTL_353("ttl", "ttl-3.5.3"),
+ UNKNOWN("unknown", "unknown");
+
+ final String type;
+ final String encoding;
+
+ NodeType(String type, String encoding) {
+ this.type = type;
+ this.encoding = encoding;
+ }
+ }
+
+ private static final class DecoderSettings {
+
+ final Boolean extended;
+ final Boolean emulate353;
+ final int maxBuffer;
+ final int extraBuffer;
+
+ DecoderSettings(Boolean extended, Boolean emulate353, int maxBuffer, int extraBuffer) {
+ this.extended = extended;
+ this.emulate353 = emulate353;
+ this.maxBuffer = maxBuffer;
+ this.extraBuffer = extraBuffer;
+ }
+
+ static DecoderSettings read() {
+ Boolean extended = booleanProperty(EphemeralType.EXTENDED_TYPES_ENABLED_PROPERTY).orElse(null);
+ Boolean emulate353 = booleanProperty(EphemeralType.TTL_3_5_3_EMULATION_PROPERTY).orElse(null);
+ if (Boolean.TRUE.equals(emulate353) && !Boolean.TRUE.equals(extended)) {
+ throw new IllegalArgumentException("emulate353TTLNodes=true requires explicit extendedTypesEnabled=true");
+ }
+ String configuredMax = System.getProperty("jute.maxbuffer");
+ if (configuredMax != null && Integer.decode(configuredMax) <= 0) {
+ throw new IllegalArgumentException("jute.maxbuffer must be a positive integer");
+ }
+ int max = BinaryInputArchive.maxBuffer;
+ String configuredExtra = System.getProperty("zookeeper.jute.maxbuffer.extrasize");
+ int extra = configuredExtra == null ? max : Integer.decode(configuredExtra);
+ if (extra < 0) {
+ throw new IllegalArgumentException("zookeeper.jute.maxbuffer.extrasize must not be negative");
+ }
+ extra = Math.max(1024, extra);
+ if ((long) max + extra > Integer.MAX_VALUE) {
+ throw new IllegalArgumentException("Combined Jute buffer limits exceed a signed 32-bit length");
+ }
+ return new DecoderSettings(extended, emulate353, max, extra);
+ }
+
+ NodeType type(long owner) {
+ if (owner == 0) {
+ return NodeType.PERSISTENT;
+ }
+ if (owner == EphemeralType.CONTAINER_EPHEMERAL_OWNER) {
+ return NodeType.CONTAINER;
+ }
+ // Negative encodings depend on source properties that are not stored in snapshots.
+ if (owner < 0 && (extended == null || (extended && emulate353 == null))) {
+ return NodeType.UNKNOWN;
+ }
+ if (EphemeralType.get(owner) == EphemeralType.TTL) {
+ return Boolean.TRUE.equals(emulate353) ? NodeType.TTL_353 : NodeType.TTL;
+ }
+ return NodeType.EPHEMERAL;
+ }
+
+ Map metadata() {
+ return fields("source_extended_types_enabled", extended, "source_emulate_353_ttl_nodes", emulate353,
+ "effective_extended_types_enabled", Boolean.TRUE.equals(extended),
+ "effective_emulate_353_ttl_nodes", Boolean.TRUE.equals(emulate353),
+ "jute_maxbuffer", maxBuffer, "jute_extra_maxbuffer", extraBuffer);
+ }
+
+ private static Optional booleanProperty(String name) {
+ String value = System.getProperty(name);
+ if (value == null) {
+ return Optional.empty();
+ }
+ if (!"true".equalsIgnoreCase(value) && !"false".equalsIgnoreCase(value)) {
+ throw new IllegalArgumentException(name + " must be true or false");
+ }
+ return Optional.of(Boolean.valueOf(value));
+ }
+ }
+
+ private static final class ExportRecords {
+
+ final SnapshotData snapshot;
+ final DecoderSettings decoder;
+ long lastObservedZxid;
+ long unknownOwnerNodes;
+
+ ExportRecords(SnapshotData snapshot, DecoderSettings decoder) {
+ this.snapshot = snapshot;
+ this.decoder = decoder;
+ }
+
+ void write(RecordWriter nodes, RecordWriter namespaces, RecordWriter sessions) throws IOException {
+ SnapshotRecursiveSummary.walk(snapshot.tree, "/", new SnapshotRecursiveSummary.TreeVisitor() {
+ @Override
+ public void visit(String path, DataNode node, int depth) throws IOException {
+ nodes.write(nodeRecord(path, node));
+ }
+
+ @Override
+ public void leave(String path, int depth, long count, long payloadBytes, long pathBytes) throws IOException {
+ if (depth == 0 && count != (long) snapshot.tree.getNodeCount() - 1) {
+ throw new IOException("Snapshot contains unreachable nodes or invalid root aliases");
+ }
+ if (depth == 1 && !Quotas.procZookeeper.equals(path)) {
+ Map namespace = record();
+ namespace.put("path", path);
+ namespace.put("subtree_node_count", count);
+ namespace.put("payload_bytes", payloadBytes);
+ namespace.put("path_utf8_bytes", pathBytes);
+ namespace.putAll(quota(path));
+ namespaces.write(namespace);
+ }
+ }
+ });
+ writeSessions(sessions);
+ }
+
+ Map nodeRecord(String path, DataNode node) {
+ Map record = record();
+ synchronized (node) {
+ StatPersisted stat = node.stat;
+ long owner = stat.getEphemeralOwner();
+ NodeType type = decoder.type(owner);
+ if (type == NodeType.UNKNOWN) {
+ unknownOwnerNodes++;
+ }
+ observe(stat.getCzxid());
+ observe(stat.getMzxid());
+ observe(stat.getPzxid());
+ long responseBytes = childResponseBytes(node.getChildren(), false);
+ record.put("path", path);
+ record.put("data_length", node.data == null ? 0L : (long) node.data.length);
+ record.put("num_children", (long) node.getChildren().size());
+ record.put("getchildren_response_bytes", responseBytes);
+ record.put("getchildren2_response_bytes", Math.addExact(responseBytes, 68L));
+ record.put("ctime_ms", stat.getCtime());
+ record.put("mtime_ms", stat.getMtime());
+ record.put("czxid", hex(stat.getCzxid()));
+ record.put("mzxid", hex(stat.getMzxid()));
+ record.put("pzxid", hex(stat.getPzxid()));
+ record.put("data_version", stat.getVersion());
+ record.put("acl_version", stat.getAversion());
+ record.put("persisted_cversion", stat.getCversion());
+ record.put("path_utf8_bytes", (long) path.getBytes(StandardCharsets.UTF_8).length);
+ record.put("node_type", type.type);
+ record.put("ephemeral_owner", hex(owner));
+ record.put("owner_encoding", type.encoding);
+ record.put("ttl_ms", type == NodeType.TTL || type == NodeType.TTL_353
+ ? EphemeralType.TTL.getValue(owner) : null);
+ record.put("acl_risk_flags", aclFlags(snapshot.tree.getACL(node)));
+ }
+ return record;
+ }
+
+ private void observe(long zxid) {
+ if (Long.compareUnsigned(zxid, lastObservedZxid) > 0) {
+ lastObservedZxid = zxid;
+ }
+ }
+
+ private Map quota(String path) {
+ DataNode limit = snapshot.tree.getNode(Quotas.quotaPath(path));
+ boolean statsPresent = snapshot.tree.getNode(Quotas.statPath(path)) != null;
+ String status = limit == null ? "absent" : "invalid";
+ Integer count = null;
+ Long bytes = null;
+ String value = null;
+ if (limit != null) {
+ synchronized (limit) {
+ if (limit.data != null) {
+ value = new String(limit.data, StandardCharsets.UTF_8);
+ }
+ }
+ }
+ if (value != null && QUOTA_VALUE.matcher(value).matches()) {
+ try {
+ StatsTrack stats = new StatsTrack(value);
+ if (stats.getCount() >= -1 && stats.getBytes() >= -1) {
+ count = stats.getCount();
+ bytes = stats.getBytes();
+ status = "valid";
+ }
+ } catch (NumberFormatException e) {
+ // Malformed quota metadata is a review flag, never a raw-value diagnostic.
+ status = "invalid";
+ }
+ }
+ return fields("quota_status", status, "quota_limit_count", count, "quota_limit_bytes", bytes,
+ "quota_stat_present", statsPresent);
+ }
+
+ private void writeSessions(RecordWriter writer) throws IOException {
+ for (long owner : snapshot.tree.getSessions()) {
+ if (decoder.type(owner) != NodeType.EPHEMERAL) {
+ continue;
+ }
+ Set paths = snapshot.tree.getEphemerals(owner);
+ long payload = 0;
+ long pathBytes = 0;
+ for (String path : paths) {
+ DataNode node = snapshot.tree.getNode(path);
+ if (node == null) {
+ throw new IOException("Missing ephemeral snapshot node: " + path);
+ }
+ synchronized (node) {
+ payload = Math.addExact(payload, node.data == null ? 0 : node.data.length);
+ }
+ pathBytes = Math.addExact(pathBytes, path.getBytes(StandardCharsets.UTF_8).length);
+ }
+ Map session = record();
+ session.put("session_id", hex(owner));
+ session.put("ephemeral_node_count", (long) paths.size());
+ session.put("payload_bytes", payload);
+ session.put("path_utf8_bytes", pathBytes);
+ session.put("present_in_snapshot_session_table", snapshot.sessions.containsKey(owner));
+ session.put("timeout_ms", snapshot.sessions.get(owner));
+ writer.write(session);
+ }
+ }
+ }
+
+ private static Set aclFlags(List acls) {
+ Set flags = new LinkedHashSet<>();
+ if (acls == null || acls.isEmpty()) {
+ flags.add("missing_acl");
+ return flags;
+ }
+ for (ACL acl : acls) {
+ Id id = acl.getId();
+ int permissions = acl.getPerms();
+ if ((permissions & ~ZooDefs.Perms.ALL) != 0) {
+ flags.add("invalid_permissions");
+ }
+ if (id == null || id.getScheme() == null || id.getId() == null) {
+ flags.add("invalid_identity");
+ continue;
+ }
+ if ("world".equals(id.getScheme()) && !"anyone".equals(id.getId())) {
+ flags.add("invalid_identity");
+ }
+ if ("world".equals(id.getScheme()) && "anyone".equals(id.getId())) {
+ if ((permissions & ZooDefs.Perms.READ) != 0) {
+ flags.add("world_read");
+ }
+ if ((permissions & (ZooDefs.Perms.WRITE | ZooDefs.Perms.CREATE | ZooDefs.Perms.DELETE)) != 0) {
+ flags.add("world_write");
+ }
+ if ((permissions & ZooDefs.Perms.ADMIN) != 0) {
+ flags.add("world_admin");
+ }
+ }
+ if ("auth".equals(id.getScheme())) {
+ flags.add("auth");
+ }
+ if (!ACL_SCHEMES.contains(id.getScheme())) {
+ flags.add("unknown_scheme");
+ }
+ }
+ return flags;
+ }
+
+ private static final class SourceIdentity {
+
+ final Path selected;
+ final Path path;
+ final BasicFileAttributes attributes;
+ final FileTime changeTime;
+ final String sha256;
+
+ SourceIdentity(Path selected, Path path, BasicFileAttributes attributes, FileTime changeTime, String sha256) {
+ this.selected = selected;
+ this.path = path;
+ this.attributes = attributes;
+ this.changeTime = changeTime;
+ this.sha256 = sha256;
+ }
+
+ static SourceIdentity read(Path selected) throws IOException {
+ Path path = selected.toRealPath();
+ BasicFileAttributes before = Files.readAttributes(path, BasicFileAttributes.class, LinkOption.NOFOLLOW_LINKS);
+ FileTime changeTime = changeTime(path);
+ if (!before.isRegularFile()) {
+ throw new IOException("Snapshot must be a regular file: " + selected);
+ }
+ MessageDigest digest = sha256();
+ try (InputStream input = Files.newInputStream(path)) {
+ byte[] buffer = new byte[65536];
+ int count;
+ while ((count = input.read(buffer)) != -1) {
+ digest.update(buffer, 0, count);
+ }
+ }
+ BasicFileAttributes after = Files.readAttributes(path, BasicFileAttributes.class, LinkOption.NOFOLLOW_LINKS);
+ if (!path.equals(selected.toRealPath()) || !sameAttributes(before, after)
+ || !Objects.equals(changeTime, changeTime(path))) {
+ throw new IOException("Source changed while hashing: " + selected);
+ }
+ return new SourceIdentity(selected, path, before, changeTime, hexBytes(digest.digest()));
+ }
+
+ void verify() throws IOException {
+ SourceIdentity current = read(selected);
+ if (!path.equals(current.path) || !sameAttributes(attributes, current.attributes)
+ || !Objects.equals(changeTime, current.changeTime) || !sha256.equals(current.sha256)) {
+ throw new IOException("Source changed during export: " + selected);
+ }
+ }
+
+ Map metadata(SnapshotData snapshot) {
+ return fields("path", path.toString(), "sha256", sha256, "size_bytes", attributes.size(),
+ "mtime_ms", attributes.lastModifiedTime().toMillis(),
+ "change_time_ms", changeTime == null ? null : changeTime.toMillis(),
+ "compression", SnapStream.getStreamMode(fileName(path)).name().toLowerCase(Locale.ROOT),
+ "format_version", snapshot.header.getVersion(), "database_id", hex(snapshot.header.getDbid()));
+ }
+
+ private static FileTime changeTime(Path path) throws IOException {
+ return path.getFileSystem().supportedFileAttributeViews().contains("unix")
+ ? (FileTime) Files.getAttribute(path, "unix:ctime", LinkOption.NOFOLLOW_LINKS) : null;
+ }
+
+ private static boolean sameAttributes(BasicFileAttributes left, BasicFileAttributes right) {
+ return right.isRegularFile() && left.size() == right.size()
+ && left.lastModifiedTime().equals(right.lastModifiedTime())
+ && left.creationTime().equals(right.creationTime())
+ && Objects.equals(left.fileKey(), right.fileKey());
+ }
+ }
+
+ private static final class OutputBudget {
+
+ final Long limit;
+ long written;
+
+ OutputBudget(Long limit) {
+ this.limit = limit;
+ }
+
+ void reserve(long count) throws IOException {
+ long next = Math.addExact(written, count);
+ if (limit != null && next > limit) {
+ throw new IOException("Output byte budget exceeded (max-output-bytes=" + limit + ")");
+ }
+ written = next;
+ }
+ }
+
+ private static final class RecordWriter implements Closeable {
+
+ final Path path;
+ final OutputBudget budget;
+ final MessageDigest digest = sha256();
+ final FileChannel channel;
+ final OutputStream output;
+ long records;
+ long bytes;
+
+ RecordWriter(Path path, OutputBudget budget) throws IOException {
+ this.path = path;
+ this.budget = budget;
+ channel = FileChannel.open(path, StandardOpenOption.CREATE_NEW, StandardOpenOption.WRITE);
+ output = new BufferedOutputStream(new DigestOutputStream(Channels.newOutputStream(channel), digest));
+ }
+
+ void write(Map record) throws IOException {
+ byte[] encoded = JSON.writeValueAsBytes(record);
+ long size = (long) encoded.length + 1;
+ budget.reserve(size);
+ output.write(encoded);
+ output.write('\n');
+ bytes = Math.addExact(bytes, size);
+ records++;
+ }
+
+ Map finish() throws IOException {
+ output.flush();
+ channel.force(true);
+ return fields("name", fileName(path), "records", records,
+ "size_bytes", bytes, "sha256", hexBytes(digest.digest()));
+ }
+
+ @Override
+ public void close() throws IOException {
+ output.close();
+ }
+ }
+
+ private static String snapshotZxid(Path source) {
+ Matcher matcher = SNAPSHOT_NAME.matcher(fileName(source));
+ return matcher.matches() ? hex(Long.parseUnsignedLong(matcher.group(1), 16)) : null;
+ }
+
+ private static String fileName(Path path) {
+ Path name = path.getFileName();
+ if (name == null) {
+ throw new IllegalArgumentException("Path must identify a file: " + path);
+ }
+ return name.toString();
+ }
+
+ private static Map record() {
+ return fields("schema_version", SCHEMA_VERSION);
+ }
+
+ private static Map fields(Object... fields) {
+ Map result = new LinkedHashMap<>();
+ for (int i = 0; i < fields.length; i += 2) {
+ result.put((String) fields[i], fields[i + 1]);
+ }
+ return result;
+ }
+
+ private static String hex(long value) {
+ return String.format(Locale.ROOT, "0x%016x", value);
+ }
+
+ private static String hexBytes(byte[] bytes) {
+ StringBuilder result = new StringBuilder(bytes.length * 2);
+ for (byte value : bytes) {
+ result.append(String.format(Locale.ROOT, "%02x", value & 0xff));
+ }
+ return result.toString();
+ }
+
+ private static MessageDigest sha256() {
+ try {
+ return MessageDigest.getInstance("SHA-256");
+ } catch (NoSuchAlgorithmException e) {
+ throw new IllegalStateException("JVM does not provide SHA-256", e);
+ }
+ }
+}
diff --git a/zookeeper-server/src/main/java/org/apache/zookeeper/server/SnapshotComparer.java b/zookeeper-server/src/main/java/org/apache/zookeeper/server/SnapshotComparer.java
index 39c9616ce27..41d491744f5 100644
--- a/zookeeper-server/src/main/java/org/apache/zookeeper/server/SnapshotComparer.java
+++ b/zookeeper-server/src/main/java/org/apache/zookeeper/server/SnapshotComparer.java
@@ -18,8 +18,10 @@
package org.apache.zookeeper.server;
+import java.io.DataInputStream;
import java.io.File;
import java.io.IOException;
+import java.io.InputStream;
import java.io.PushbackInputStream;
import java.util.ArrayList;
import java.util.Collections;
@@ -38,9 +40,12 @@
import org.apache.commons.cli.ParseException;
import org.apache.jute.BinaryInputArchive;
import org.apache.jute.InputArchive;
+import org.apache.jute.Record;
import org.apache.zookeeper.ZKUtil;
+import org.apache.zookeeper.server.persistence.FileHeader;
import org.apache.zookeeper.server.persistence.FileSnap;
import org.apache.zookeeper.server.persistence.SnapStream;
+import org.apache.zookeeper.server.util.SerializeUtils;
import org.apache.zookeeper.util.ServiceUtils;
/**
@@ -219,25 +224,78 @@ public String toString() {
}
static DataTree getSnapshot(File file) throws IOException {
+ return readSnapshot(file).tree;
+ }
+
+ static final class SnapshotData {
+
+ final DataTree tree;
+ final Map sessions;
+ final FileHeader header;
+ final DataTree.ZxidDigest digest;
+
+ SnapshotData(DataTree tree, Map sessions, FileHeader header, DataTree.ZxidDigest digest) {
+ this.tree = tree;
+ this.sessions = sessions;
+ this.header = header;
+ this.digest = digest;
+ }
+
+ }
+
+ private static final class SnapshotInputArchive extends BinaryInputArchive {
+
+ long nodeRecords;
+
+ SnapshotInputArchive(InputStream input) {
+ super(new DataInputStream(input));
+ }
+
+ @Override
+ public void readRecord(Record record, String tag) throws IOException {
+ super.readRecord(record, tag);
+ if (record instanceof DataNode) {
+ nodeRecords++;
+ }
+ }
+
+ }
+
+ static SnapshotData readSnapshot(File file) throws IOException {
try (CheckedInputStream stream = SnapStream.getInputStream(file);
PushbackInputStream input = new PushbackInputStream(stream)) {
- InputArchive archive = BinaryInputArchive.getArchive(input);
+ SnapshotInputArchive archive = new SnapshotInputArchive(input);
+ FileHeader header = new FileHeader();
+ header.deserialize(archive, "fileheader");
+ if (header.getMagic() != FileSnap.SNAP_MAGIC || header.getVersion() != 2) {
+ throw new IOException("Unsupported snapshot magic or format version: " + header.getVersion());
+ }
DataTree dataTree = new DataTree();
- new FileSnap(null).deserialize(dataTree, new HashMap(), archive);
+ Map sessions = new HashMap<>();
+ SerializeUtils.deserializeSnapshot(dataTree, archive, sessions);
+ if (dataTree.getNode("") == null || dataTree.getNode("") != dataTree.getNode("/")) {
+ throw new IOException("Invalid snapshot root");
+ }
+ // DataTree overwrites duplicate paths; count records without a second path index.
+ if (archive.nodeRecords != (long) dataTree.getNodeCount() - 1) {
+ throw new IOException("Duplicate snapshot node records");
+ }
checkSealIntegrity(stream, archive);
// Distinguish an absent digest in older snapshots from a truncated digest.
+ DataTree.ZxidDigest digest = null;
int next = input.read();
if (next != -1) {
input.unread(next);
- dataTree.new ZxidDigest().deserialize(archive);
+ digest = dataTree.new ZxidDigest();
+ digest.deserialize(archive);
checkSealIntegrity(stream, archive);
if (input.read() != -1) {
throw new IOException("Unexpected data after snapshot");
}
}
- return dataTree;
- } catch (IOException | NumberFormatException e) {
+ return new SnapshotData(dataTree, sessions, header, digest);
+ } catch (IOException | IllegalArgumentException e) {
throw new IOException(file + ": " + e, e);
}
}
diff --git a/zookeeper-server/src/main/java/org/apache/zookeeper/server/SnapshotRecursiveSummary.java b/zookeeper-server/src/main/java/org/apache/zookeeper/server/SnapshotRecursiveSummary.java
index d34bc38c4b6..e0d1b06feac 100644
--- a/zookeeper-server/src/main/java/org/apache/zookeeper/server/SnapshotRecursiveSummary.java
+++ b/zookeeper-server/src/main/java/org/apache/zookeeper/server/SnapshotRecursiveSummary.java
@@ -20,8 +20,11 @@
import java.io.File;
import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayDeque;
import java.util.Collections;
-import java.util.Set;
+import java.util.Deque;
+import java.util.Iterator;
import java.util.TreeSet;
import org.apache.yetus.audience.InterfaceAudience;
import org.apache.zookeeper.ZKUtil;
@@ -71,37 +74,91 @@ public void run(String snapshotFileName, String startingNode, int maxDepth) thro
throw new IllegalArgumentException("Starting node does not exist: " + startingNode);
}
StringBuilder builder = new StringBuilder();
- printZnode(dataTree, startingNode, builder, 0, maxDepth);
+ Deque children = new ArrayDeque<>();
+ walk(dataTree, startingNode, new TreeVisitor() {
+ @Override
+ public void visit(String path, DataNode node, int depth) {
+ children.push(new StringBuilder());
+ }
+
+ @Override
+ public void leave(String path, int depth, long nodes, long payloadBytes, long pathBytes) {
+ StringBuilder childSummary = children.pop();
+ StringBuilder summary = new StringBuilder();
+ if (nodes > 1 && (maxDepth == 0 || depth <= maxDepth)) {
+ String indent = String.join("", Collections.nCopies(depth, "--"));
+ summary.append(indent).append(" ").append(path).append("\n");
+ summary.append(indent).append(" children: ").append(nodes - 1).append("\n");
+ summary.append(indent).append(" data: ").append(payloadBytes).append("\n");
+ summary.append(childSummary);
+ }
+ (children.isEmpty() ? builder : children.peek()).append(summary);
+ }
+ });
System.out.println(builder);
}
- private long[] printZnode(DataTree dataTree, String name, StringBuilder builder, int level, int maxDepth) {
- DataNode node = dataTree.getNode(name);
- Set children;
- long dataSize;
- synchronized (node) {
- dataSize = node.data == null ? 0 : node.data.length;
- children = new TreeSet<>(node.getChildren());
- }
- long[] result = {1L, dataSize};
- if (children.isEmpty()) {
- return result;
+ interface TreeVisitor {
+
+ void visit(String path, DataNode node, int depth) throws IOException;
+
+ void leave(String path, int depth, long nodes, long payloadBytes, long pathBytes) throws IOException;
+
+ }
+
+ private static final class Frame {
+
+ final String path;
+ final int depth;
+ final Iterator children;
+ long nodes = 1;
+ long payloadBytes;
+ long pathBytes;
+
+ Frame(String path, int depth, DataNode node) {
+ this.path = path;
+ this.depth = depth;
+ synchronized (node) {
+ children = new TreeSet<>(node.getChildren()).iterator();
+ payloadBytes = node.data == null ? 0 : node.data.length;
+ }
+ pathBytes = path.getBytes(StandardCharsets.UTF_8).length;
}
- StringBuilder childBuilder = new StringBuilder();
- for (String child : children) {
- long[] childResult = printZnode(dataTree, name + (name.equals("/") ? "" : "/") + child,
- childBuilder, level + 1, maxDepth);
- result[0] += childResult[0];
- result[1] += childResult[1];
+
+ }
+
+ /** Visits the offline tree once, with inclusive subtree totals on exit and no second tree. */
+ static void walk(DataTree tree, String startingNode, TreeVisitor visitor) throws IOException {
+ String start = startingNode.isEmpty() ? "/" : startingNode;
+ Deque stack = new ArrayDeque<>();
+ enter(tree, start, 0, visitor, stack);
+ while (!stack.isEmpty()) {
+ Frame current = stack.peek();
+ if (current.children.hasNext()) {
+ String child = current.path + (current.path.equals("/") ? "" : "/") + current.children.next();
+ enter(tree, child, current.depth + 1, visitor, stack);
+ } else {
+ stack.pop();
+ visitor.leave(current.path, current.depth, current.nodes, current.payloadBytes, current.pathBytes);
+ if (!stack.isEmpty()) {
+ Frame parent = stack.peek();
+ parent.nodes = Math.addExact(parent.nodes, current.nodes);
+ parent.payloadBytes = Math.addExact(parent.payloadBytes, current.payloadBytes);
+ parent.pathBytes = Math.addExact(parent.pathBytes, current.pathBytes);
+ }
+ }
}
- if (maxDepth == 0 || level <= maxDepth) {
- String indent = String.join("", Collections.nCopies(level, "--"));
- builder.append(indent).append(" ").append(name).append("\n");
- builder.append(indent).append(" children: ").append(result[0] - 1).append("\n");
- builder.append(indent).append(" data: ").append(result[1]).append("\n");
- builder.append(childBuilder);
+ }
+
+ private static void enter(DataTree tree, String path, int depth, TreeVisitor visitor, Deque stack)
+ throws IOException {
+ PathUtils.validatePath(path);
+ DataNode node = tree.getNode(path);
+ if (node == null) {
+ throw new IOException("Missing snapshot node: " + path);
}
- return result;
+ stack.push(new Frame(path, depth, node));
+ visitor.visit(path, node, depth);
}
public static String getUsage() {
diff --git a/zookeeper-server/src/test/java/org/apache/zookeeper/server/OfflineAuditExporterTest.java b/zookeeper-server/src/test/java/org/apache/zookeeper/server/OfflineAuditExporterTest.java
new file mode 100644
index 00000000000..342ae1b109e
--- /dev/null
+++ b/zookeeper-server/src/test/java/org/apache/zookeeper/server/OfflineAuditExporterTest.java
@@ -0,0 +1,939 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.zookeeper.server;
+
+import static org.hamcrest.CoreMatchers.containsString;
+import static org.hamcrest.CoreMatchers.not;
+import static org.junit.Assert.assertArrayEquals;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertThat;
+import static org.junit.Assert.assertTrue;
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import java.io.BufferedReader;
+import java.io.ByteArrayOutputStream;
+import java.io.File;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.StandardCopyOption;
+import java.nio.file.attribute.FileTime;
+import java.nio.file.attribute.PosixFilePermissions;
+import java.security.MessageDigest;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Deque;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+import org.apache.jute.BinaryOutputArchive;
+import org.apache.jute.OutputArchive;
+import org.apache.zookeeper.Quotas;
+import org.apache.zookeeper.ZKTestCase;
+import org.apache.zookeeper.ZooDefs;
+import org.apache.zookeeper.data.ACL;
+import org.apache.zookeeper.data.Id;
+import org.apache.zookeeper.data.Stat;
+import org.apache.zookeeper.proto.GetChildren2Response;
+import org.apache.zookeeper.proto.GetChildrenResponse;
+import org.apache.zookeeper.proto.ReplyHeader;
+import org.apache.zookeeper.server.SnapshotComparerTest.SnapshotToolTestSupport;
+import org.apache.zookeeper.server.persistence.FileHeader;
+import org.apache.zookeeper.server.persistence.FileSnap;
+import org.apache.zookeeper.server.persistence.SnapStream.StreamMode;
+import org.junit.After;
+import org.junit.Assume;
+import org.junit.Before;
+import org.junit.Test;
+
+public class OfflineAuditExporterTest extends ZKTestCase {
+
+ private static final ObjectMapper JSON = new ObjectMapper();
+ private static final String MAIN = "org.apache.zookeeper.server.OfflineAuditExporter";
+ private SnapshotToolTestSupport support;
+ private int nextOutput;
+
+ @Before
+ public void setUp() throws IOException {
+ support = new SnapshotToolTestSupport();
+ }
+
+ @After
+ public void tearDown() throws IOException {
+ support.close();
+ }
+
+ @Test
+ public void childResponseSizesIncludeProtocolOverhead() throws Exception {
+ List names = Arrays.asList("\u00e9", "a");
+ assertEquals(31L, OfflineAuditExporter.childResponseBytes(names, false));
+ assertEquals(99L, OfflineAuditExporter.childResponseBytes(names, true));
+ assertEquals(20L, OfflineAuditExporter.childResponseBytes(Collections.emptyList(), false));
+ assertEquals(88L, OfflineAuditExporter.childResponseBytes(Collections.emptyList(), true));
+ assertEquals(28L, OfflineAuditExporter.childResponseBytes(Collections.singletonList("\ud83d\ude00"), false));
+ }
+
+ @Test
+ public void childResponseSizesMatchActualJuteSerialization() throws Exception {
+ for (List names : Arrays.asList(Collections.emptyList(),
+ Arrays.asList("a", "\u00e9", "\u4e2d", "quote\""))) {
+ for (boolean includeStat : new boolean[]{false, true}) {
+ ByteArrayOutputStream bytes = new ByteArrayOutputStream();
+ OutputArchive archive = BinaryOutputArchive.getArchive(bytes);
+ new ReplyHeader(123, 456L, 0).serialize(archive, "header");
+ if (includeStat) {
+ new GetChildren2Response(names, new Stat()).serialize(archive, "response");
+ } else {
+ new GetChildrenResponse(names).serialize(archive, "response");
+ }
+ assertEquals(bytes.size(), OfflineAuditExporter.childResponseBytes(names, includeStat));
+ }
+ }
+ }
+
+ @Test
+ public void childResponseSizesUseLongArithmetic() throws Exception {
+ char[] name = new char[65532];
+ Arrays.fill(name, 'a');
+ assertEquals(2147483668L, OfflineAuditExporter.childResponseBytes(Collections.nCopies(32768, new String(name)), false));
+ }
+
+ @Test
+ public void exportsRootUnicodeAndPersistedMetadataWithoutSecrets() throws Exception {
+ DataTree tree = customerTree();
+ tree.setData("/app/\u00e9", "private-synthetic-data".getBytes(StandardCharsets.UTF_8), 2, 3, 1234);
+ DataNode unicode = tree.getNode("/app/\u00e9");
+ unicode.stat.setCzxid(0x2000000000000001L);
+ unicode.stat.setMzxid(0x7ffffffffffffff0L);
+ unicode.stat.setPzxid(0x8000000000000001L);
+ unicode.stat.setCtime(1000);
+ tree.setACL("/app/\u00e9", Collections.singletonList(
+ new ACL(ZooDefs.Perms.READ, new Id("digest", "synthetic-user:do-not-export"))), 4);
+ File snapshot = support.snapshot(tree, StreamMode.CHECKED, true);
+ byte[] original = Files.readAllBytes(snapshot.toPath());
+ FileTime mtime = Files.getLastModifiedTime(snapshot.toPath());
+ Path output = output();
+
+ run(snapshot, output).success();
+
+ JsonNode root = node(output, "/");
+ assertEquals(5L, root.get("data_length").asLong());
+ assertEquals(4L, root.get("num_children").asLong());
+ assertEquals(1L, root.get("path_utf8_bytes").asLong());
+ JsonNode record = node(output, "/app/\u00e9");
+ assertEquals(1, record.get("schema_version").asInt());
+ assertEquals(22L, record.get("data_length").asLong());
+ assertEquals(7L, record.get("path_utf8_bytes").asLong());
+ assertEquals(0L, record.get("num_children").asLong());
+ assertEquals(20L, record.get("getchildren_response_bytes").asLong());
+ assertEquals(88L, record.get("getchildren2_response_bytes").asLong());
+ assertEquals(1000L, record.get("ctime_ms").asLong());
+ assertEquals(1234L, record.get("mtime_ms").asLong());
+ assertEquals("0x2000000000000001", record.get("czxid").asText());
+ assertEquals("0x7ffffffffffffff0", record.get("mzxid").asText());
+ assertEquals("0x8000000000000001", record.get("pzxid").asText());
+ assertEquals(2, record.get("data_version").asInt());
+ assertEquals(4, record.get("acl_version").asInt());
+ assertEquals(0, record.get("persisted_cversion").asInt());
+ assertEquals("persistent", record.get("node_type").asText());
+ assertEquals("0x0000000000000000", record.get("ephemeral_owner").asText());
+ assertTrue(record.get("ttl_ms").isNull());
+ assertEquals(9, records(output.resolve("nodes.ndjson")).size());
+ String all = text(output.resolve("nodes.ndjson"));
+ assertThat(all, not(containsString("private-synthetic-data")));
+ assertThat(all, not(containsString("synthetic-user")));
+ assertThat(all, not(containsString("do-not-export")));
+ assertThat(all, not(containsString("\"path\":\"\"")));
+ assertEquals("0x8000000000000001", manifest(output).get("last_observed_zxid").asText());
+ assertArrayEquals(original, Files.readAllBytes(snapshot.toPath()));
+ assertEquals(mtime, Files.getLastModifiedTime(snapshot.toPath()));
+ }
+
+ @Test
+ public void nodeOrderIsDeterministicPreorderWithContiguousSubtrees() throws Exception {
+ DataTree tree = new DataTree();
+ for (String path : Arrays.asList("/a-", "/a-/y", "/a", "/a/x", "/a/x/z")) {
+ support.addNode(tree, path, null);
+ }
+ File snapshot = support.snapshot(tree, StreamMode.CHECKED, false);
+ Path first = output();
+ Path second = output();
+ run(snapshot, first).success();
+ run(snapshot, second).success();
+ List paths = new ArrayList<>();
+ for (JsonNode node : records(first.resolve("nodes.ndjson"))) {
+ paths.add(node.get("path").asText());
+ }
+ assertEquals(Arrays.asList("/", "/a", "/a/x", "/a/x/z", "/a-", "/a-/y",
+ "/zookeeper", "/zookeeper/config", "/zookeeper/quota"), paths);
+ assertArrayEquals(Files.readAllBytes(first.resolve("nodes.ndjson")), Files.readAllBytes(second.resolve("nodes.ndjson")));
+ }
+
+ @Test
+ public void namespaceTotalsAreInclusiveAndExcludeOnlyReservedSubtree() throws Exception {
+ DataTree tree = customerTree();
+ addQuota(tree, "/app", "count=50,bytes=1000");
+ Path output = output();
+
+ run(support.snapshot(tree, StreamMode.CHECKED, false), output).success();
+
+ assertEquals(3, records(output.resolve("namespaces.ndjson")).size());
+ JsonNode app = record(output.resolve("namespaces.ndjson"), "path", "/app");
+ assertEquals(3L, app.get("subtree_node_count").asLong());
+ assertEquals(12L, app.get("payload_bytes").asLong());
+ assertEquals(17L, app.get("path_utf8_bytes").asLong());
+ assertEquals("valid", app.get("quota_status").asText());
+ assertEquals(50, app.get("quota_limit_count").asInt());
+ assertEquals(1000L, app.get("quota_limit_bytes").asLong());
+ assertTrue(app.get("quota_stat_present").asBoolean());
+ JsonNode client = record(output.resolve("namespaces.ndjson"), "path", "/zookeeper-client");
+ assertEquals(1L, client.get("subtree_node_count").asLong());
+ assertEquals(11L, client.get("payload_bytes").asLong());
+ assertEquals(17L, client.get("path_utf8_bytes").asLong());
+ assertEquals("absent", client.get("quota_status").asText());
+ assertTrue(client.get("quota_limit_bytes").isNull());
+ assertFalse(client.get("quota_stat_present").asBoolean());
+ assertThat(text(output.resolve("namespaces.ndjson")), not(containsString("\"path\":\"/zookeeper\"")));
+ assertNotNull(node(output, "/zookeeper/quota/app/zookeeper_limits"));
+ }
+
+ @Test
+ public void quotaAvailabilityDoesNotInventOrLeakMalformedLimits() throws Exception {
+ DataTree tree = customerTree();
+ addQuota(tree, "/app", "private-malformed-quota");
+ addQuota(tree, "/other", "count=-1,bytes=999");
+ Path output = output();
+
+ run(support.snapshot(tree, StreamMode.CHECKED, false), output).success();
+
+ JsonNode invalid = record(output.resolve("namespaces.ndjson"), "path", "/app");
+ assertEquals("invalid", invalid.get("quota_status").asText());
+ assertTrue(invalid.get("quota_limit_count").isNull());
+ assertTrue(invalid.get("quota_limit_bytes").isNull());
+ JsonNode valid = record(output.resolve("namespaces.ndjson"), "path", "/other");
+ assertEquals("valid", valid.get("quota_status").asText());
+ assertEquals(-1, valid.get("quota_limit_count").asInt());
+ assertEquals(999, valid.get("quota_limit_bytes").asLong());
+ assertThat(text(output.resolve("namespaces.ndjson")), not(containsString("private-malformed-quota")));
+ }
+
+ @Test
+ public void aclFlagsAreReviewHintsWithoutRawIdentities() throws Exception {
+ DataTree tree = new DataTree();
+ List> acls = Arrays.asList(
+ ZooDefs.Ids.OPEN_ACL_UNSAFE,
+ Collections.singletonList(new ACL(ZooDefs.Perms.READ, new Id("world", "anyone"))),
+ Collections.singletonList(new ACL(ZooDefs.Perms.READ, new Id("auth", ""))),
+ Collections.singletonList(new ACL(ZooDefs.Perms.ALL, new Id("custom-secret-scheme", "private-id"))),
+ Collections.emptyList(),
+ Collections.singletonList(new ACL(64, new Id("digest", "private-digest"))),
+ Collections.singletonList(new ACL(ZooDefs.Perms.READ, new Id("world", "private-invalid-identity"))));
+ List> expected = Arrays.asList(
+ flags("world_read", "world_write", "world_admin"), flags("world_read"),
+ flags("auth"), flags("unknown_scheme"), flags("missing_acl"), flags("invalid_permissions"), flags("invalid_identity"));
+ for (int i = 0; i < acls.size(); i++) {
+ tree.createNode("/acl" + i, null, acls.get(i), 0, -1, 1, 1);
+ }
+ Path output = output();
+
+ run(support.snapshot(tree, StreamMode.CHECKED, false), output).success();
+
+ for (int i = 0; i < expected.size(); i++) {
+ Set actual = new HashSet<>();
+ for (JsonNode flag : node(output, "/acl" + i).get("acl_risk_flags")) {
+ actual.add(flag.asText());
+ }
+ assertEquals(expected.get(i), actual);
+ assertEquals(0, node(output, "/acl" + i).get("data_length").asLong());
+ }
+ assertThat(text(output.resolve("nodes.ndjson")), not(containsString("private-")));
+ assertThat(text(output.resolve("nodes.ndjson")), not(containsString("custom-secret-scheme")));
+ }
+
+ @Test
+ public void unknownConfigurationDoesNotTurnTtlOrUnknownOwnersIntoSessions() throws Exception {
+ Path output = output();
+ run(support.snapshot(ownerTree(true), StreamMode.CHECKED, true), output).success();
+
+ assertType(output, "/session", "ephemeral", "session");
+ assertType(output, "/container", "container", "container");
+ for (String path : Arrays.asList("/ttl", "/legacy", "/unknown")) {
+ assertType(output, path, "unknown", "unknown");
+ }
+ List sessions = records(output.resolve("sessions.ndjson"));
+ assertEquals(1, sessions.size());
+ JsonNode session = sessions.get(0);
+ assertEquals("0x0000000000000123", session.get("session_id").asText());
+ assertEquals(2, session.get("ephemeral_node_count").asLong());
+ assertEquals(12, session.get("payload_bytes").asLong());
+ assertEquals(15, session.get("path_utf8_bytes").asLong());
+ assertTrue(session.get("present_in_snapshot_session_table").asBoolean());
+ assertEquals(30000, session.get("timeout_ms").asInt());
+ assertEquals(3, manifest(output).get("unknown_owner_nodes").asLong());
+ assertTrue(manifest(output).get("decoder").get("source_extended_types_enabled").isNull());
+ }
+
+ @Test
+ public void explicitModernConfigurationDecodesTtlAndContainer() throws Exception {
+ Path output = output();
+ run(support.snapshot(ownerTree(false), StreamMode.CHECKED, true), output,
+ Arrays.asList("-Dzookeeper.extendedTypesEnabled=true", "-Dzookeeper.emulate353TTLNodes=false")).success();
+
+ assertType(output, "/ttl", "ttl", "ttl");
+ assertEquals(42, node(output, "/ttl").get("ttl_ms").asLong());
+ assertType(output, "/legacy", "ephemeral", "session");
+ assertType(output, "/container", "container", "container");
+ assertEquals(2, records(output.resolve("sessions.ndjson")).size());
+ JsonNode orphan = record(output.resolve("sessions.ndjson"), "session_id", "0x800000000000002a");
+ assertFalse(orphan.get("present_in_snapshot_session_table").asBoolean());
+ assertTrue(orphan.get("timeout_ms").isNull());
+ assertTrue(manifest(output).get("decoder").get("source_extended_types_enabled").asBoolean());
+ assertFalse(manifest(output).get("decoder").get("source_emulate_353_ttl_nodes").asBoolean());
+ }
+
+ @Test
+ public void explicitLegacyConfigurationUsesNativeTtlValueSemantics() throws Exception {
+ DataTree tree = ownerTree(false);
+ tree.getNode("/legacy").stat.setEphemeralOwner(0x800001000000002aL);
+ Path output = output();
+ run(support.snapshot(tree, StreamMode.CHECKED, false), output,
+ Arrays.asList("-Dzookeeper.extendedTypesEnabled=true", "-Dzookeeper.emulate353TTLNodes=true")).success();
+
+ assertType(output, "/legacy", "ttl", "ttl-3.5.3");
+ assertEquals(42, node(output, "/legacy").get("ttl_ms").asLong());
+ assertEquals(1, records(output.resolve("sessions.ndjson")).size());
+ }
+
+ @Test
+ public void explicitDisabledExtendedTypesPreservesNegativeSessionIds() throws Exception {
+ Path output = output();
+ run(support.snapshot(ownerTree(true), StreamMode.CHECKED, false), output,
+ Collections.singletonList("-Dzookeeper.extendedTypesEnabled=false")).success();
+
+ assertType(output, "/ttl", "ephemeral", "session");
+ assertType(output, "/unknown", "ephemeral", "session");
+ assertType(output, "/container", "container", "container");
+ assertEquals(4, records(output.resolve("sessions.ndjson")).size());
+ }
+
+ @Test
+ public void missingLegacySettingRemainsUnknownForNegativeEncodings() throws Exception {
+ Path output = output();
+ run(support.snapshot(ownerTree(false), StreamMode.CHECKED, false), output,
+ Collections.singletonList("-Dzookeeper.extendedTypesEnabled=true")).success();
+
+ assertType(output, "/legacy", "unknown", "unknown");
+ assertType(output, "/ttl", "unknown", "unknown");
+ assertEquals(1, records(output.resolve("sessions.ndjson")).size());
+ }
+
+ @Test
+ public void unsupportedKnownExtendedEncodingFailsClosed() throws Exception {
+ Path output = output();
+ failure(run(support.snapshot(ownerTree(true), StreamMode.CHECKED, false), output,
+ Arrays.asList("-Dzookeeper.extendedTypesEnabled=true", "-Dzookeeper.emulate353TTLNodes=false")));
+ assertFalse(Files.exists(output.resolve("manifest.json")));
+ }
+
+ @Test
+ public void invalidDecoderPropertiesAreNotSilentlyDefaulted() throws Exception {
+ File snapshot = support.snapshot(new DataTree(), StreamMode.CHECKED, false);
+ for (List properties : Arrays.asList(
+ Collections.singletonList("-Dzookeeper.extendedTypesEnabled=maybe"),
+ Collections.singletonList("-Dzookeeper.emulate353TTLNodes=yes"),
+ Collections.singletonList("-Dzookeeper.emulate353TTLNodes=true"),
+ Collections.singletonList("-Djute.maxbuffer=not-a-number"),
+ Collections.singletonList("-Djute.maxbuffer=-1"),
+ Arrays.asList("-Djute.maxbuffer=2147483647", "-Dzookeeper.jute.maxbuffer.extrasize=1024"),
+ Collections.singletonList("-Dzookeeper.jute.maxbuffer.extrasize=-1"))) {
+ Path output = output();
+ run(snapshot, output, properties).invalidInvocation();
+ assertFalse(Files.exists(output));
+ }
+ }
+
+ @Test
+ public void nativeJutePropertySyntaxAndMinimumExtraPaddingAreRecorded() throws Exception {
+ DataTree tree = new DataTree();
+ support.addNode(tree, "/large", new byte[8000]);
+ File snapshot = support.snapshot(tree, StreamMode.CHECKED, false);
+ Path output = output();
+ run(snapshot, output, Arrays.asList("-Djute.maxbuffer=0x1000", "-Dzookeeper.jute.maxbuffer.extrasize=010000")).success();
+ assertEquals(8000, node(output, "/large").get("data_length").asLong());
+ assertEquals(4096, manifest(output).get("decoder").get("jute_maxbuffer").asInt());
+ assertEquals(4096, manifest(output).get("decoder").get("jute_extra_maxbuffer").asInt());
+ File small = support.snapshot(new DataTree(), StreamMode.CHECKED, false);
+ Path padded = output();
+ run(small, padded, Arrays.asList("-Djute.maxbuffer=1024", "-Dzookeeper.jute.maxbuffer.extrasize=0")).success();
+ assertEquals(1024, manifest(padded).get("decoder").get("jute_extra_maxbuffer").asInt());
+ }
+
+ @Test
+ public void manifestRecordsChecksumsAndUnknownCaptureRatherThanRecoveryClaims() throws Exception {
+ for (StreamMode mode : StreamMode.values()) {
+ for (boolean digest : new boolean[]{false, true}) {
+ File snapshot = support.snapshot(customerTree(), mode, digest);
+ Path output = output();
+ run(snapshot, output).success();
+ JsonNode manifest = manifest(output);
+ assertEquals(1, manifest.get("schema_version").asInt());
+ assertTrue(manifest.get("export_complete").asBoolean());
+ assertEquals("snapshot-only", manifest.get("recovery_scope").asText());
+ assertFalse(manifest.get("transaction_logs_replayed").asBoolean());
+ assertTrue(manifest.get("source_capture_time_ms").isNull());
+ assertTrue(manifest.get("source_capture_provenance").isNull());
+ assertTrue(manifest.get("source_server_version").isNull());
+ assertTrue(manifest.get("max_output_bytes").isNull());
+ assertEquals(snapshot.getCanonicalPath(), manifest.get("source").get("path").asText());
+ assertEquals(Files.size(snapshot.toPath()), manifest.get("source").get("size_bytes").asLong());
+ assertEquals(snapshot.lastModified(), manifest.get("source").get("mtime_ms").asLong());
+ assertEquals(sha256(snapshot.toPath()), manifest.get("source").get("sha256").asText());
+ assertEquals(2, manifest.get("source").get("format_version").asInt());
+ assertEquals("OfflineAuditExporter", manifest.get("tool").get("name").asText());
+ assertFalse(manifest.get("tool").get("version").asText().isEmpty());
+ assertTrue(manifest.get("snapshot_zxid").asText().startsWith("0x"));
+ assertEquals(!digest, manifest.get("snapshot_digest").isNull());
+ if (digest) {
+ assertTrue(manifest.get("snapshot_digest").get("seal_validated").asBoolean());
+ assertFalse(manifest.get("snapshot_digest").get("transaction_consistency_verified").asBoolean());
+ }
+ Set names = new HashSet<>();
+ for (JsonNode descriptor : manifest.get("files")) {
+ Path file = output.resolve(descriptor.get("name").asText());
+ names.add(file.getFileName().toString());
+ assertEquals(Files.size(file), descriptor.get("size_bytes").asLong());
+ assertEquals(sha256(file), descriptor.get("sha256").asText());
+ assertEquals(records(file).size(), descriptor.get("records").asLong());
+ }
+ assertEquals(flags("nodes.ndjson", "namespaces.ndjson", "sessions.ndjson"), names);
+ }
+ }
+ }
+
+ @Test
+ public void fuzzyDigestDoesNotRequireTransactionLogsOrBecomeAnEndpoint() throws Exception {
+ DataTree tree = new DataTree() {
+ @Override
+ public boolean serializeZxidDigest(OutputArchive archive) throws IOException {
+ new ZxidDigest(0x123456789abcdef0L, 2, 0xfedcba9876543210L).serialize(archive);
+ return true;
+ }
+ };
+ support.addNode(tree, "/customer", new byte[3]);
+ File snapshot = support.snapshot(tree, StreamMode.CHECKED, true);
+ Path output = output();
+
+ run(snapshot, output).success();
+
+ JsonNode manifest = manifest(output);
+ assertEquals("0x0000000000000001", manifest.get("last_observed_zxid").asText());
+ assertEquals("0x123456789abcdef0", manifest.get("snapshot_digest").get("zxid").asText());
+ assertEquals("0xfedcba9876543210", manifest.get("snapshot_digest").get("value").asText());
+ Path renamed = snapshot.toPath().resolveSibling("snapshot.11-22");
+ Files.copy(snapshot.toPath(), renamed);
+ Path renamedOutput = output();
+ run(renamed.toFile(), renamedOutput).success();
+ assertTrue(manifest(renamedOutput).get("snapshot_zxid").isNull());
+ }
+
+ @Test
+ public void rejectsCorruptionWithoutFallbackToAnotherSnapshot() throws Exception {
+ support.snapshot(customerTree(), StreamMode.CHECKED, true);
+ for (File snapshot : support.corruptSnapshots()) {
+ Path output = output();
+ failure(run(snapshot, output));
+ assertFalse(Files.exists(output.resolve("manifest.json")));
+ }
+ }
+
+ @Test
+ public void rejectsSealedUnsupportedFormatAndMissingRoot() throws Exception {
+ File unsupported = new File(support.directory, "snapshot.abc");
+ new FileSnap(null) {
+ @Override
+ protected void serialize(DataTree tree, Map sessions, OutputArchive archive,
+ FileHeader header) throws IOException {
+ super.serialize(tree, sessions, archive, new FileHeader(FileSnap.SNAP_MAGIC, 3, -1));
+ }
+ }.serialize(new DataTree(), Collections.emptyMap(), unsupported, false);
+ Path unsupportedOutput = output();
+ failure(run(unsupported, unsupportedOutput));
+ assertFalse(Files.exists(unsupportedOutput.resolve("manifest.json")));
+ DataTree missingRoot = new DataTree() {
+ @Override
+ public void serializeNodes(OutputArchive archive) throws IOException {
+ archive.writeString("/", "path");
+ }
+ };
+ Path rootOutput = output();
+ failure(run(support.snapshot(missingRoot, StreamMode.CHECKED, false), rootOutput));
+ assertFalse(Files.exists(rootOutput.resolve("manifest.json")));
+ }
+
+ @Test
+ public void rejectsSealedDuplicateNodesInsteadOfDoubleCountingOwners() throws Exception {
+ DataTree tree = new DataTree() {
+ @Override
+ public void serializeNodes(OutputArchive archive) throws IOException {
+ for (String path : Arrays.asList("", "/zookeeper", "/zookeeper/config", "/zookeeper/quota", "/same")) {
+ serializeNodeData(archive, path, getNode(path));
+ }
+ DataNode duplicate = new DataNode(new byte[7], getNode("/same").acl, createStat(2, 2, 0x456L));
+ serializeNodeData(archive, "/same", duplicate);
+ archive.writeString("/", "path");
+ }
+ };
+ tree.createNode("/same", new byte[5], ZooDefs.Ids.OPEN_ACL_UNSAFE, 0x123L, -1, 1, 1);
+ Path output = output();
+ failure(run(support.snapshot(tree, StreamMode.CHECKED, true), output));
+ assertFalse(Files.exists(output.resolve("manifest.json")));
+ }
+
+ @Test
+ public void rejectsInvalidCliArgumentsBeforeCreatingOutput() throws Exception {
+ File snapshot = support.snapshot(customerTree(), StreamMode.CHECKED, false);
+ execute(Collections.emptyList()).invalidInvocation();
+ execute(Collections.emptyList(), "--snapshot-file", snapshot.toString()).invalidInvocation();
+ for (String limit : Arrays.asList("0", "-1", "NaN", "9223372036854775808")) {
+ Path output = output();
+ run(snapshot, output, Collections.emptyList(), "--max-output-bytes", limit).invalidInvocation();
+ assertFalse(Files.exists(output));
+ }
+ Path output = output();
+ run(snapshot, output, Collections.emptyList(), "unexpected").invalidInvocation();
+ run(snapshot, output, Collections.emptyList(), "--unknown").invalidInvocation();
+ run(snapshot, output, Collections.emptyList(),
+ "--snapshot-file", snapshot.toString()).invalidInvocation();
+ run(new File(support.directory, "missing"), output).invalidInvocation();
+ run(support.directory, output).invalidInvocation();
+ assertFalse(Files.exists(output));
+ }
+
+ @Test
+ public void rejectsExistingOutputAndAliasesWithoutTouchingUnrelatedFiles() throws Exception {
+ File snapshot = support.snapshot(customerTree(), StreamMode.CHECKED, true);
+ byte[] original = Files.readAllBytes(snapshot.toPath());
+ Path output = output();
+ run(snapshot, output).success();
+ byte[] priorManifest = Files.readAllBytes(output.resolve("manifest.json"));
+ run(snapshot, output).invalidInvocation();
+ assertArrayEquals(priorManifest, Files.readAllBytes(output.resolve("manifest.json")));
+ run(snapshot, snapshot.toPath()).invalidInvocation();
+ run(snapshot, snapshot.toPath().getParent()).invalidInvocation();
+ Path existing = output();
+ Files.createDirectory(existing);
+ Path unrelated = existing.resolve("unrelated.txt");
+ Files.write(unrelated, new byte[]{1, 2, 3});
+ run(snapshot, existing).invalidInvocation();
+ assertArrayEquals(new byte[]{1, 2, 3}, Files.readAllBytes(unrelated));
+ Path link = support.directory.toPath().resolve("source-hardlink");
+ Files.createLink(link, snapshot.toPath());
+ run(snapshot, link).invalidInvocation();
+ assertArrayEquals(original, Files.readAllBytes(snapshot.toPath()));
+ }
+
+ @Test
+ public void rejectsSymbolicOutputAliases() throws Exception {
+ Assume.assumeFalse(System.getProperty("os.name").startsWith("Windows"));
+ File snapshot = support.snapshot(customerTree(), StreamMode.CHECKED, true);
+ Path link = support.directory.toPath().resolve("output-link");
+ Path dangling = support.directory.toPath().resolve("dangling-link");
+ try {
+ Files.createSymbolicLink(link, snapshot.toPath().getParent());
+ run(snapshot, link).invalidInvocation();
+ Files.createSymbolicLink(dangling, support.directory.toPath().resolve("missing-target"));
+ run(snapshot, dangling).invalidInvocation();
+ assertTrue(Files.isSymbolicLink(dangling));
+ } finally {
+ Files.deleteIfExists(link);
+ Files.deleteIfExists(dangling);
+ }
+ }
+
+ @Test
+ public void symlinkParentComponentsDoNotSelectADifferentSnapshotOrDestination() throws Exception {
+ Assume.assumeFalse(System.getProperty("os.name").startsWith("Windows"));
+ Path real = support.directory.toPath().resolve("real");
+ Files.createDirectories(real.resolve("inner"));
+ Path alias = support.directory.toPath().resolve("alias");
+ DataTree selectedTree = new DataTree();
+ support.addNode(selectedTree, "/selected", new byte[7]);
+ Path selected = real.resolve("snapshot.55");
+ Files.copy(support.snapshot(selectedTree, StreamMode.CHECKED, false).toPath(), selected);
+ DataTree decoy = new DataTree();
+ support.addNode(decoy, "/selected", new byte[17]);
+ Files.copy(support.snapshot(decoy, StreamMode.CHECKED, false).toPath(),
+ support.directory.toPath().resolve("snapshot.55"));
+ try {
+ Files.createSymbolicLink(alias, real.resolve("inner"));
+ Path input = alias.resolve("../snapshot.55");
+ Path output = alias.resolve("../export");
+ run(input.toFile(), output).success();
+ assertTrue(Files.exists(real.resolve("export/manifest.json")));
+ assertEquals(selected.toRealPath().toString(), manifest(output).get("source").get("path").asText());
+ assertEquals(7, node(output, "/selected").get("data_length").asLong());
+ assertFalse(Files.exists(support.directory.toPath().resolve("export")));
+ } finally {
+ Files.deleteIfExists(alias);
+ }
+ }
+
+ @Test
+ public void outputBudgetFailuresNeverPublishSuccessEvenAfterAllDataFiles() throws Exception {
+ File snapshot = support.snapshot(customerTree(), StreamMode.CHECKED, false);
+ Path complete = output();
+ run(snapshot, complete).success();
+ long dataBytes = 0;
+ for (JsonNode file : manifest(complete).get("files")) {
+ dataBytes += file.get("size_bytes").asLong();
+ }
+ for (long limit : new long[]{1, 1500, dataBytes + 1}) {
+ Path output = output();
+ failure(run(snapshot, output, Collections.emptyList(),
+ "--max-output-bytes", Long.toString(limit)));
+ assertFalse(Files.exists(output.resolve("manifest.json")));
+ assertTrue(Files.exists(output.resolve("nodes.ndjson")));
+ assertTrue(Files.size(output.resolve("nodes.ndjson")) <= limit);
+ if (limit == dataBytes + 1) {
+ assertTrue(Files.exists(output.resolve("sessions.ndjson")));
+ assertEquals(Files.size(complete.resolve("nodes.ndjson")), Files.size(output.resolve("nodes.ndjson")));
+ }
+ run(snapshot, output).invalidInvocation();
+ }
+ Path bounded = output();
+ run(snapshot, bounded, Collections.emptyList(), "--max-output-bytes", "1000000").success();
+ assertEquals(1000000L, manifest(bounded).get("max_output_bytes").asLong());
+ }
+
+ @Test
+ public void largePayloadHonorsNativeJuteDecoderLimits() throws Exception {
+ DataTree tree = new DataTree();
+ support.addNode(tree, "/large", new byte[3000000]);
+ File snapshot = support.snapshot(tree, StreamMode.CHECKED, false);
+ Path tooSmall = output();
+ failure(run(snapshot, tooSmall));
+ assertFalse(Files.exists(tooSmall.resolve("manifest.json")));
+ Path output = output();
+ run(snapshot, output, Collections.singletonList("-Djute.maxbuffer=4194304")).success();
+ assertEquals(3000000L, node(output, "/large").get("data_length").asLong());
+ assertEquals(4194304, manifest(output).get("decoder").get("jute_maxbuffer").asInt());
+ assertEquals(4194304, manifest(output).get("decoder").get("jute_extra_maxbuffer").asInt());
+ assertTrue(Files.size(output.resolve("nodes.ndjson")) < 10000);
+ }
+
+ @Test
+ public void largeFanoutReportsImmediateChildrenAndCompleteNamespaceTotals() throws Exception {
+ DataTree tree = new DataTree();
+ support.addNode(tree, "/fanout", null);
+ for (int i = 0; i < 4096; i++) {
+ support.addNode(tree, String.format("/fanout/child-%04d", i), new byte[1]);
+ }
+ Path output = output();
+ run(support.snapshot(tree, StreamMode.CHECKED, false), output).success();
+ JsonNode node = node(output, "/fanout");
+ assertEquals(4096, node.get("num_children").asLong());
+ assertEquals(57364L, node.get("getchildren_response_bytes").asLong());
+ assertEquals(57432L, node.get("getchildren2_response_bytes").asLong());
+ JsonNode namespace = record(output.resolve("namespaces.ndjson"), "path", "/fanout");
+ assertEquals(4097L, namespace.get("subtree_node_count").asLong());
+ assertEquals(4096L, namespace.get("payload_bytes").asLong());
+ assertEquals(73735L, namespace.get("path_utf8_bytes").asLong());
+ }
+
+ @Test
+ public void outputFilesystemFailureDoesNotPublishAResult() throws Exception {
+ Assume.assumeTrue(support.directory.toPath().getFileSystem().supportedFileAttributeViews().contains("posix"));
+ Path parent = support.directory.toPath().resolve("unwritable");
+ Files.createDirectory(parent);
+ File snapshot = support.snapshot(customerTree(), StreamMode.CHECKED, false);
+ try {
+ Files.setPosixFilePermissions(parent, PosixFilePermissions.fromString("r-x------"));
+ Assume.assumeFalse(Files.isWritable(parent));
+ Path output = parent.resolve("export");
+ failure(run(snapshot, output));
+ assertFalse(Files.exists(output.resolve("manifest.json")));
+ } finally {
+ Files.setPosixFilePermissions(parent, PosixFilePermissions.fromString("rwx------"));
+ }
+ }
+
+ @Test
+ public void rejectsInvalidPersistedPathsRatherThanPublishingNormalizedNames() throws Exception {
+ for (String path : Arrays.asList("/bad\nname", "/bad\u0000name", "/bad\ud83d\ude00")) {
+ DataTree tree = new DataTree();
+ support.addNode(tree, path, null);
+ Path output = output();
+ failure(run(support.snapshot(tree, StreamMode.CHECKED, false), output));
+ assertFalse(Files.exists(output.resolve("manifest.json")));
+ }
+ }
+
+ @Test
+ public void traversalHandlesDeepTreesWithoutRecursiveExporterStack() throws Exception {
+ DataTree tree = new DataTree() {
+ @Override
+ public void serializeNodes(OutputArchive archive) throws IOException {
+ Deque paths = new ArrayDeque<>();
+ paths.push("");
+ while (!paths.isEmpty()) {
+ String path = paths.pop();
+ DataNode node = getNode(path);
+ serializeNodeData(archive, path, node);
+ for (String child : node.getChildren()) {
+ paths.push(path + "/" + child);
+ }
+ }
+ archive.writeString("/", "path");
+ }
+ };
+ String path = "";
+ for (int i = 0; i < 1800; i++) {
+ path += "/d";
+ support.addNode(tree, path, new byte[1]);
+ }
+ Path output = output();
+
+ run(support.snapshot(tree, StreamMode.CHECKED, false), output,
+ Collections.singletonList("-Xss256k")).success();
+
+ JsonNode namespace = record(output.resolve("namespaces.ndjson"), "path", "/d");
+ assertEquals(1800, namespace.get("subtree_node_count").asLong());
+ assertEquals(1800, namespace.get("payload_bytes").asLong());
+ assertEquals(3241800L, namespace.get("path_utf8_bytes").asLong());
+ assertEquals(1804L, manifest(output).get("files").get(0).get("records").asLong());
+ }
+
+ @Test
+ public void detectsSourceByteChangesEvenWhenSizeAndMtimeAreRestored() throws Exception {
+ sourceMutation(0);
+ }
+
+ @Test
+ public void detectsSourceReplacementEvenWhenBytesAndMtimeAreIdentical() throws Exception {
+ sourceMutation(1);
+ }
+
+ @Test
+ public void detectsSourceRewriteThenRestoreWhenChangeTimeIsAvailable() throws Exception {
+ Assume.assumeTrue(support.directory.toPath().getFileSystem().supportedFileAttributeViews().contains("unix"));
+ sourceMutation(2);
+ }
+
+ @Test
+ public void shellLauncherForwardsPathsAndJvmFlags() throws Exception {
+ Assume.assumeFalse(System.getProperty("os.name").startsWith("Windows"));
+ File snapshot = support.snapshot(ownerTree(false), StreamMode.GZIP, true);
+ for (String flags : Arrays.asList("", "-Xms32m -Xmx128m -Dzookeeper.extendedTypesEnabled=true"
+ + " -Dzookeeper.emulate353TTLNodes=false")) {
+ Path output = output();
+ support.runLauncher("zkOfflineAudit.sh", flags, "--snapshot-file", snapshot.toString(),
+ "--output-dir", output.toString()).success();
+ assertType(output, "/ttl", flags.isEmpty() ? "unknown" : "ttl", flags.isEmpty() ? "unknown" : "ttl");
+ }
+ }
+
+ private void sourceMutation(int action) throws Exception {
+ DataTree tree = new DataTree();
+ support.addNode(tree, "/fanout", null);
+ for (int i = 0; i < 12000; i++) {
+ support.addNode(tree, "/fanout/child-" + i, new byte[1]);
+ }
+ File snapshot = support.snapshot(tree, StreamMode.CHECKED, true);
+ byte[] bytes = Files.readAllBytes(snapshot.toPath());
+ FileTime mtime = Files.getLastModifiedTime(snapshot.toPath());
+ Path output = output();
+ Path log = support.directory.toPath().resolve("mutation-log-" + nextOutput);
+ Process process = process(Collections.emptyList(), "--snapshot-file", snapshot.toString(),
+ "--output-dir", output.toString()).redirectErrorStream(true).redirectOutput(log.toFile()).start();
+ try {
+ long deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(20);
+ while (!Files.exists(output.resolve("nodes.ndjson")) && process.isAlive() && System.nanoTime() < deadline) {
+ Thread.sleep(1);
+ }
+ assertTrue("Exporter never started writing: " + text(log), Files.exists(output.resolve("nodes.ndjson")));
+ if (action == 1) {
+ Path replacement = support.directory.toPath().resolve("replacement");
+ Files.write(replacement, bytes);
+ Files.setLastModifiedTime(replacement, mtime);
+ Files.move(replacement, snapshot.toPath(), StandardCopyOption.REPLACE_EXISTING);
+ } else {
+ bytes[bytes.length - 1] ^= 1;
+ Files.write(snapshot.toPath(), bytes);
+ if (action == 2) {
+ bytes[bytes.length - 1] ^= 1;
+ Files.write(snapshot.toPath(), bytes);
+ }
+ Files.setLastModifiedTime(snapshot.toPath(), mtime);
+ }
+ assertTrue(process.waitFor(30, TimeUnit.SECONDS));
+ assertEquals(text(log), 1, process.exitValue());
+ assertThat(text(log), containsString("Source changed"));
+ assertFalse(Files.exists(output.resolve("manifest.json")));
+ } finally {
+ if (process.isAlive()) {
+ process.destroyForcibly();
+ process.waitFor(10, TimeUnit.SECONDS);
+ }
+ }
+ }
+
+ private DataTree customerTree() throws Exception {
+ DataTree tree = new DataTree();
+ tree.setData("/", new byte[5], 1, 1, 1);
+ support.addNode(tree, "/app", new byte[2]);
+ support.addNode(tree, "/app/\u00e9", new byte[3]);
+ tree.createNode("/app/e", new byte[7], ZooDefs.Ids.OPEN_ACL_UNSAFE, 0x123L, -1, 1, 1);
+ support.addNode(tree, "/other", new byte[4]);
+ support.addNode(tree, "/zookeeper-client", new byte[11]);
+ return tree;
+ }
+
+ private DataTree ownerTree(boolean unknown) throws Exception {
+ String previous = System.getProperty(EphemeralType.EXTENDED_TYPES_ENABLED_PROPERTY);
+ System.setProperty(EphemeralType.EXTENDED_TYPES_ENABLED_PROPERTY, "false");
+ try {
+ DataTree tree = new DataTree();
+ tree.createNode("/session", new byte[5], ZooDefs.Ids.OPEN_ACL_UNSAFE, 0x123L, -1, 1, 1);
+ tree.createNode("/second", new byte[7], ZooDefs.Ids.OPEN_ACL_UNSAFE, 0x123L, -1, 1, 1);
+ tree.createNode("/container", new byte[2], ZooDefs.Ids.OPEN_ACL_UNSAFE, Long.MIN_VALUE, -1, 1, 1);
+ tree.createNode("/ttl", new byte[3], ZooDefs.Ids.OPEN_ACL_UNSAFE, 0xff0000000000002aL, -1, 1, 1);
+ tree.createNode("/legacy", new byte[4], ZooDefs.Ids.OPEN_ACL_UNSAFE, 0x800000000000002aL, -1, 1, 1);
+ if (unknown) {
+ tree.createNode("/unknown", new byte[6], ZooDefs.Ids.OPEN_ACL_UNSAFE, 0xff0001000000002aL, -1, 1, 1);
+ }
+ return tree;
+ } finally {
+ if (previous == null) {
+ System.clearProperty(EphemeralType.EXTENDED_TYPES_ENABLED_PROPERTY);
+ } else {
+ System.setProperty(EphemeralType.EXTENDED_TYPES_ENABLED_PROPERTY, previous);
+ }
+ }
+ }
+
+ private void addQuota(DataTree tree, String path, String limits) throws Exception {
+ support.addNode(tree, Quotas.quotaZookeeper + path, null);
+ support.addNode(tree, Quotas.quotaPath(path), limits.getBytes(StandardCharsets.UTF_8));
+ support.addNode(tree, Quotas.statPath(path), "count=0,bytes=0".getBytes(StandardCharsets.UTF_8));
+ }
+
+ private Path output() {
+ return support.directory.toPath().resolve("export with spaces " + ++nextOutput);
+ }
+
+ private SnapshotToolTestSupport.Result run(File snapshot, Path output) throws Exception {
+ return run(snapshot, output, Collections.emptyList());
+ }
+
+ private SnapshotToolTestSupport.Result run(File snapshot, Path output, List properties,
+ String... extra) throws Exception {
+ List args = new ArrayList<>(Arrays.asList("--snapshot-file", snapshot.toString(),
+ "--output-dir", output.toString()));
+ args.addAll(Arrays.asList(extra));
+ return execute(properties, args.toArray(new String[0]));
+ }
+
+ private SnapshotToolTestSupport.Result execute(List properties, String... args) throws Exception {
+ Path log = support.directory.toPath().resolve("cli-log-" + ++nextOutput);
+ Process process = process(properties, args).redirectErrorStream(true).redirectOutput(log.toFile()).start();
+ try {
+ assertTrue("CLI timed out", process.waitFor(30, TimeUnit.SECONDS));
+ return new SnapshotToolTestSupport.Result(process.exitValue(), text(log));
+ } finally {
+ if (process.isAlive()) {
+ process.destroyForcibly();
+ process.waitFor(10, TimeUnit.SECONDS);
+ }
+ }
+ }
+
+ private ProcessBuilder process(List properties, String... args) {
+ List command = new ArrayList<>(Arrays.asList(
+ new File(System.getProperty("java.home"), "bin/java").toString(), "-Xmx128m",
+ "-Djava.io.tmpdir=" + System.getProperty("java.io.tmpdir")));
+ command.addAll(properties);
+ command.addAll(Arrays.asList("-cp",
+ System.getProperty("surefire.test.class.path", System.getProperty("java.class.path")), MAIN));
+ command.addAll(Arrays.asList(args));
+ return new ProcessBuilder(command);
+ }
+
+ private static void failure(SnapshotToolTestSupport.Result result) {
+ assertEquals(result.output, 1, result.exitCode);
+ assertThat(result.output, containsString("Unable to export snapshot"));
+ }
+
+ private static JsonNode manifest(Path output) throws IOException {
+ return JSON.readTree(output.resolve("manifest.json").toFile());
+ }
+
+ private static JsonNode node(Path output, String path) throws IOException {
+ return record(output.resolve("nodes.ndjson"), "path", path);
+ }
+
+ private static JsonNode record(Path file, String key, String value) throws IOException {
+ for (JsonNode record : records(file)) {
+ if (record.get(key).asText().equals(value)) {
+ return record;
+ }
+ }
+ throw new AssertionError("Missing " + value + " in " + file);
+ }
+
+ private static List records(Path file) throws IOException {
+ List records = new ArrayList<>();
+ try (BufferedReader reader = Files.newBufferedReader(file, StandardCharsets.UTF_8)) {
+ String line;
+ while ((line = reader.readLine()) != null) {
+ records.add(JSON.readTree(line));
+ }
+ }
+ return records;
+ }
+
+ private static void assertType(Path output, String path, String type, String encoding) throws IOException {
+ JsonNode record = node(output, path);
+ assertEquals(path, type, record.get("node_type").asText());
+ assertEquals(path, encoding, record.get("owner_encoding").asText());
+ }
+
+ private static Set flags(String... flags) {
+ return new HashSet<>(Arrays.asList(flags));
+ }
+
+ private static String text(Path path) throws IOException {
+ return new String(Files.readAllBytes(path), StandardCharsets.UTF_8);
+ }
+
+ private static String sha256(Path path) throws Exception {
+ byte[] digest = MessageDigest.getInstance("SHA-256").digest(Files.readAllBytes(path));
+ StringBuilder result = new StringBuilder();
+ for (byte value : digest) {
+ result.append(String.format("%02x", value & 0xff));
+ }
+ return result.toString();
+ }
+}
From f468b5339314f327b1469ce8ff656de36e774987 Mon Sep 17 00:00:00 2001
From: Arkesh Mishra <118651144+arkmish@users.noreply.github.com>
Date: Fri, 25 Sep 2026 16:20:56 +0530
Subject: [PATCH 3/9] Add opt-in write size and outcome audit metadata (C1)
Decode typed requests without changing their buffers, cover TTL writes, correlate multi members by position and distinguish atomic rollback from commit. Add safe enhanced ACL metadata, deletion identity and observable audit failures while retaining legacy logging defaults.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---
.../resources/markdown/zookeeperAuditLogs.md | 127 +++-
.../zookeeper/audit/AuditConstants.java | 2 +
.../apache/zookeeper/audit/AuditEvent.java | 25 +-
.../apache/zookeeper/audit/AuditHelper.java | 410 ++++++++----
.../zookeeper/audit/ZKAuditProvider.java | 63 +-
.../org/apache/zookeeper/server/DataTree.java | 15 +-
.../zookeeper/server/ServerMetrics.java | 2 +
.../zookeeper/audit/AuditEventTest.java | 53 ++
.../zookeeper/audit/AuditHelperTest.java | 597 ++++++++++++++++++
.../zookeeper/audit/Slf4JAuditLoggerTest.java | 121 ++--
.../audit/StandaloneServerAuditTest.java | 239 ++++++-
11 files changed, 1430 insertions(+), 224 deletions(-)
create mode 100644 zookeeper-server/src/test/java/org/apache/zookeeper/audit/AuditHelperTest.java
diff --git a/zookeeper-docs/src/main/resources/markdown/zookeeperAuditLogs.md b/zookeeper-docs/src/main/resources/markdown/zookeeperAuditLogs.md
index b3954ad676d..a58c8f19aa3 100644
--- a/zookeeper-docs/src/main/resources/markdown/zookeeperAuditLogs.md
+++ b/zookeeper-docs/src/main/resources/markdown/zookeeperAuditLogs.md
@@ -17,7 +17,8 @@ limitations under the License.
# ZooKeeper Audit Logging
* [ZooKeeper Audit Logs](#ch_auditLogs)
-* [ZooKeeper Audit Log Configuration](#ch_reconfig_format)
+* [ZooKeeper Audit Log Configuration](#ch_auditConfig)
+* [Enhanced audit metadata (schema v2)](#ch_auditV2)
* [Who is taken as user in audit logs?](#ch_zkAuditUser)
@@ -36,9 +37,9 @@ The audit log captures detailed information for the operations that are selected
|session | client session id |
|user | comma separated list of users who are associate with a client session. For more on this, see [Who is taken as user in audit logs](#ch_zkAuditUser).
|ip | client IP address
-|operation | any one of the selected operations for audit. Possible values are(serverStart, serverStop, create, delete, setData, setAcl, multiOperation, reconfig, ephemeralZNodeDeleteOnSessionClose)
+|operation | any one of the selected operations for audit. Possible values are(serverStart, serverStop, create, delete, setData, setAcl, multiOperation, reconfig, ephemeralZNodeDeletionOnSessionCloseOrExpire)
|znode | path of the znode
-|znode type | type of znode in case of creation operation
+|znode_type | type of znode in case of creation operation
|acl | String representation of znode ACL like cdrwa(create, delete,read, write, admin). This is logged only for setAcl operation
|result | result of the operation. Possible values are (success/failure/invoked). Result "invoked" is used for serverStop operation because stop is logged before ensuring that server actually stopped.
@@ -99,6 +100,126 @@ Audit logging is done using logback. Following is the default logback configurat
Change above configuration to customize the auditlog file, number of backups, max file size, custom audit logger etc.
+
+
+## Enhanced audit metadata (schema v2)
+
+The optional Java system property `zookeeper.audit.enhanced.enable` defaults to
+`false`. Set `-Dzookeeper.audit.enhanced.enable=true` in addition to enabling
+`zookeeper.audit.enable` to emit v2 records. Enhanced logging does not enable
+audit logging by itself. With enhanced logging off, ordinary legacy formatting
+and the existing logging APIs are preserved. TTL creates are audited in both
+modes, including sequential TTL creates.
+
+V2 keeps the existing field names and `result=success/failure/invoked`, and adds:
+
+| Key | Meaning |
+| --- | --- |
+| `schema_version` | `2`. Its absence identifies legacy output. |
+| `data_length` | Attempted payload size in bytes for create variants and `setData`, not request size, character count, or resulting znode size. Null and empty payloads are `0`; unavailable or undecodable payloads omit this field. A failed attempt can have a nonzero length. |
+| `error_code` | Known numeric ZooKeeper code, such as `0` (OK), `-101` (NONODE), `-102` (NOAUTH), `-103` (BADVERSION), or `-110` (NODEEXISTS). Omitted when unavailable. |
+| `outcome` | `committed`, `failed`, `rolled_back`, or `unknown`, as described below. |
+| `cxid` | Known client request ID in decimal. All members of a multi share this ID. |
+| `zxid` | Known transaction ID in decimal, not the server's latest zxid at reply time. A failed transaction may also have a zxid. |
+| `multi_index` | Zero-based position in the **complete** multi request, including non-mutating checks. Omitted on single operations and multi parent records. |
+
+No znode payloads are included. For enhanced `setAcl` records, `acl` retains
+schemes and permissions, but identities are conservative: `world:anyone` is
+retained, valid built-in digest identities use the provider's username extraction
+(not the digest), and other identities are `[redacted]`. This also applies to
+failed ACL attempts, where identities might be malformed or contain credentials.
+Unknown custom identity representations are not printed as ACL identities.
+Custom callers of the logging APIs must supply sanitized user and ACL metadata;
+the string-based APIs cannot infer credentials from arbitrary strings.
+
+### Results and transaction outcomes
+
+* `committed` means the available transaction result reports an applied write.
+ These records have `result=success` and `error_code=0`. It does not guarantee
+ that the reply reached the client or the log reached durable storage.
+* `failed` means the operation or whole multi failed according to its known
+ result. For rejected single transactions, the request exception takes
+ precedence when the reply path uses it instead of the transaction error.
+* `rolled_back` means a mutation was not applied because its atomic multi
+ failed. This includes prepared members discarded by rollback and members
+ skipped after the failure. They have `result=failure`, **even when their
+ individual `error_code` is `0`**. Later skipped members normally have code
+ `-2` (RUNTIMEINCONSISTENCY); a first failure with that code is still `failed`.
+* `unknown` means the available metadata cannot establish an operation's
+ outcome. Missing or mismatched multi results are not treated as commits,
+ and untrustworthy per-member error codes are omitted. If the whole multi
+ is known to have failed, these members still have `result=failure`;
+ otherwise unknown operations use `result=invoked`.
+
+Failed multis keep their `multiOperation` parent failure record, followed in v2
+by records for the attempted mutations when the request is decodable. The parent
+is retained even if request decoding fails. Successful multis emit individual
+mutation records, not a parent success record. Request and result members are
+paired by position, so repeated paths cannot overwrite each other's metadata.
+Successful sequential creates use the final path returned by the transaction;
+failed or rolled-back creates use the attempted path.
+
+### Supported operations and identity
+
+The audited operations remain creates (`create`, `create2`, container and TTL
+variants), deletes (including container deletes), `setData`, `setAcl`, write
+multis, reconfiguration, server start/stop, and existing system ephemeral-node
+deletions. Ordinary reads, read multis and checks do not produce audit records.
+Checks inside a write multi still occupy an index.
+
+Existing lifecycle events and callers of the legacy logging overloads receive
+`outcome=unknown` in v2 when they do not supply transaction metadata; their
+existing `result` is unchanged. No session/authentication binding events are
+added by this schema.
+
+System ephemeral-node deletion records preserve the **server actor**, the
+affected session and path, and omit client IP. V2 adds the deletion transaction's
+zxid, known error code and outcome. A client request ID is not available at this
+deletion hook, so it is omitted. These system records can be emitted on multiple
+replicas or during replay. Downstream consumers can deduplicate using ensemble
+identity plus `zxid`, `operation`, `session` and `znode`; client multi mutations
+also require `multi_index` to distinguish repeated paths.
+
+### Escaping and parsing
+
+Fields are separated by literal tabs. Split each field at its **first** equals
+sign; equals signs inside values are unchanged. In v2 only, value formatting
+reversibly escapes backslash as `\\`, tab as `\t`, carriage return as `\r`, and
+newline as `\n`. A literal backslash followed by `t` is therefore `\\t`, distinct
+from an escaped tab. Decode left to right, consuming one escape pair at a time;
+do not use successive global replacements to unescape. For example, the user
+value containing a tab and newline is rendered as `user=team\tname\n` on a single
+physical log line. Enum-derived keys and values are independent of the server's
+default locale. Custom `AuditLogger` implementations receive raw event values;
+`AuditEvent.toString()` is the production text-formatting path.
+
+### Mixed versions, rollout, rollback and coverage
+
+Consumers should recognize v2 per record using `schema_version=2`, tolerate
+unknown additive fields, and accept omitted optional fields. Do not apply v2
+unescaping to legacy records. Prepare consumers for mixed legacy/v2 output before
+enabling the property on servers, then roll it out gradually. Roll back emission
+by removing the enhanced property or setting it to `false` on restart. The
+enhanced gate is checked when emitting events; the base audit enablement retains
+its startup behavior. No wire, persistence, authentication, ACL, quota or payload
+limit changes are required.
+
+The bounded-cardinality server counter `audit_errors` counts detected audit
+metadata extraction/correlation failures and runtime failures reported by the
+audit logger. Such failures are logged without rejecting an otherwise valid
+client operation. A metadata error can still yield an audit record with omitted
+fields, so this counter is **not** a dropped-record count. It has no per-path,
+per-user or per-session labels.
+
+Audit coverage is limited to the existing transaction-audit hooks. Rejections
+before a hook, connection-level failures and later reply/send failures need
+separate instrumentation; these records are not a complete request-delivery
+ledger. Log delivery is also a separate concern: an asynchronous appender using
+`neverBlock=true` can discard records without reporting an exception.
+`audit_errors` cannot measure those silent drops, filtering, retention loss, or
+downstream delivery gaps. Validate source-log coverage and delivery independently
+before relying on the stream for complete accounting.
+
## Who is taken as user in audit logs?
diff --git a/zookeeper-server/src/main/java/org/apache/zookeeper/audit/AuditConstants.java b/zookeeper-server/src/main/java/org/apache/zookeeper/audit/AuditConstants.java
index 22fd8567abe..1bfe96b363b 100644
--- a/zookeeper-server/src/main/java/org/apache/zookeeper/audit/AuditConstants.java
+++ b/zookeeper-server/src/main/java/org/apache/zookeeper/audit/AuditConstants.java
@@ -18,6 +18,8 @@
package org.apache.zookeeper.audit;
public final class AuditConstants {
+ public static final String SCHEMA_VERSION = "2";
+
private AuditConstants() {
//Utility classes should not have public constructors
}
diff --git a/zookeeper-server/src/main/java/org/apache/zookeeper/audit/AuditEvent.java b/zookeeper-server/src/main/java/org/apache/zookeeper/audit/AuditEvent.java
index e499552a948..bd3a7842f95 100644
--- a/zookeeper-server/src/main/java/org/apache/zookeeper/audit/AuditEvent.java
+++ b/zookeeper-server/src/main/java/org/apache/zookeeper/audit/AuditEvent.java
@@ -18,6 +18,7 @@
package org.apache.zookeeper.audit;
import java.util.LinkedHashMap;
+import java.util.Locale;
import java.util.Map;
import java.util.Set;
@@ -43,12 +44,12 @@ public Set> getLogEntries() {
void addEntry(FieldName fieldName, String value) {
if (value != null) {
- logEntries.put(fieldName.name().toLowerCase(), value);
+ logEntries.put(fieldName.name().toLowerCase(Locale.ROOT), value);
}
}
public String getValue(FieldName fieldName) {
- return logEntries.get(fieldName.name().toLowerCase());
+ return logEntries.get(fieldName.name().toLowerCase(Locale.ROOT));
}
public Result getResult() {
@@ -63,6 +64,7 @@ public Result getResult() {
@Override
public String toString() {
StringBuilder buffer = new StringBuilder();
+ boolean enhanced = AuditConstants.SCHEMA_VERSION.equals(getValue(FieldName.SCHEMA_VERSION));
boolean first = true;
for (Map.Entry entry : logEntries.entrySet()) {
String key = entry.getKey();
@@ -75,7 +77,7 @@ public String toString() {
buffer.append(PAIR_SEPARATOR);
}
buffer.append(key).append(KEY_VAL_SEPARATOR)
- .append(value);
+ .append(enhanced ? escape(value) : value);
}
}
//add result field
@@ -83,16 +85,27 @@ public String toString() {
buffer.append(PAIR_SEPARATOR);
}
buffer.append("result").append(KEY_VAL_SEPARATOR)
- .append(result.name().toLowerCase());
+ .append(result.name().toLowerCase(Locale.ROOT));
return buffer.toString();
}
+ private static String escape(String value) {
+ return value.replace("\\", "\\\\")
+ .replace("\t", "\\t")
+ .replace("\r", "\\r")
+ .replace("\n", "\\n");
+ }
+
public enum FieldName {
- USER, OPERATION, IP, ACL, ZNODE, SESSION, ZNODE_TYPE
+ USER, OPERATION, IP, ACL, ZNODE, SESSION, ZNODE_TYPE,
+ SCHEMA_VERSION, DATA_LENGTH, ERROR_CODE, OUTCOME, CXID, ZXID, MULTI_INDEX
}
public enum Result {
SUCCESS, FAILURE, INVOKED
}
-}
+ public enum Outcome {
+ COMMITTED, FAILED, ROLLED_BACK, UNKNOWN
+ }
+}
diff --git a/zookeeper-server/src/main/java/org/apache/zookeeper/audit/AuditHelper.java b/zookeeper-server/src/main/java/org/apache/zookeeper/audit/AuditHelper.java
index ce0d58a87ba..c2811048aba 100644
--- a/zookeeper-server/src/main/java/org/apache/zookeeper/audit/AuditHelper.java
+++ b/zookeeper-server/src/main/java/org/apache/zookeeper/audit/AuditHelper.java
@@ -18,23 +18,34 @@
package org.apache.zookeeper.audit;
import java.io.IOException;
-import java.util.HashMap;
-import java.util.Map;
+import java.nio.ByteBuffer;
+import java.util.List;
+import java.util.Locale;
import org.apache.jute.Record;
import org.apache.zookeeper.CreateMode;
import org.apache.zookeeper.KeeperException;
+import org.apache.zookeeper.KeeperException.Code;
import org.apache.zookeeper.MultiOperationRecord;
import org.apache.zookeeper.Op;
import org.apache.zookeeper.ZKUtil;
-import org.apache.zookeeper.ZooDefs;
+import org.apache.zookeeper.ZooDefs.OpCode;
+import org.apache.zookeeper.audit.AuditEvent.Outcome;
import org.apache.zookeeper.audit.AuditEvent.Result;
+import org.apache.zookeeper.data.ACL;
+import org.apache.zookeeper.data.Id;
import org.apache.zookeeper.proto.CreateRequest;
+import org.apache.zookeeper.proto.CreateTTLRequest;
import org.apache.zookeeper.proto.DeleteRequest;
import org.apache.zookeeper.proto.SetACLRequest;
import org.apache.zookeeper.proto.SetDataRequest;
import org.apache.zookeeper.server.ByteBufferInputStream;
import org.apache.zookeeper.server.DataTree.ProcessTxnResult;
import org.apache.zookeeper.server.Request;
+import org.apache.zookeeper.server.ServerMetrics;
+import org.apache.zookeeper.server.auth.AuthenticationProvider;
+import org.apache.zookeeper.server.auth.DigestAuthenticationProvider;
+import org.apache.zookeeper.server.auth.ProviderRegistry;
+import org.apache.zookeeper.server.util.AuthUtil;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -53,174 +64,295 @@ public static void addAuditLog(Request request, ProcessTxnResult rc) {
*
* @param request user request
* @param txnResult ProcessTxnResult
- * @param failedTxn whether audit is being done failed transaction for normal transaction
+ * @param failedTxn whether the transaction was rejected before applying the requested operation
*/
public static void addAuditLog(Request request, ProcessTxnResult txnResult, boolean failedTxn) {
if (!ZKAuditProvider.isAuditEnabled()) {
return;
}
- String op = null;
- //For failed transaction rc.path is null
- String path = txnResult.path;
- String acls = null;
- String createMode = null;
try {
- switch (request.type) {
- case ZooDefs.OpCode.create:
- case ZooDefs.OpCode.create2:
- case ZooDefs.OpCode.createContainer:
- op = AuditConstants.OP_CREATE;
- if (failedTxn) {
- CreateRequest createRequest = new CreateRequest();
- deserialize(request, createRequest);
- path = createRequest.getPath();
- createMode =
- getCreateMode(createRequest);
- } else {
- createMode = getCreateMode(request);
- }
- break;
- case ZooDefs.OpCode.delete:
- case ZooDefs.OpCode.deleteContainer:
- op = AuditConstants.OP_DELETE;
- if (failedTxn) {
- DeleteRequest deleteRequest = new DeleteRequest();
- deserialize(request, deleteRequest);
- path = deleteRequest.getPath();
- }
- break;
- case ZooDefs.OpCode.setData:
- op = AuditConstants.OP_SETDATA;
- if (failedTxn) {
- SetDataRequest setDataRequest = new SetDataRequest();
- deserialize(request, setDataRequest);
- path = setDataRequest.getPath();
- }
- break;
- case ZooDefs.OpCode.setACL:
- op = AuditConstants.OP_SETACL;
- if (failedTxn) {
- SetACLRequest setACLRequest = new SetACLRequest();
- deserialize(request, setACLRequest);
- path = setACLRequest.getPath();
- acls = ZKUtil.aclToString(setACLRequest.getAcl());
- } else {
- acls = getACLs(request);
+ if (request.type == OpCode.multi) {
+ logMultiOperation(request, txnResult, failedTxn);
+ return;
+ }
+ String operation = operationFor(request.type);
+ if (operation == null) {
+ return;
+ }
+ Integer error = errorCode(request, txnResult, failedTxn);
+ Outcome outcome = outcome(txnResult, failedTxn, error);
+ RequestMetadata metadata = new RequestMetadata();
+ boolean enhanced = ZKAuditProvider.isEnhancedAuditEnabled();
+ if (isCreate(request.type) || request.type == OpCode.setACL
+ || (request.type != OpCode.reconfig && (enhanced || outcome != Outcome.COMMITTED))) {
+ try {
+ metadata = metadata(request.type, readRequestRecord(request), enhanced);
+ } catch (IOException e) {
+ auditError(request.type, e);
+ }
+ }
+ String path = txnResult == null ? null : txnResult.path;
+ if (outcome != Outcome.COMMITTED || path == null) {
+ path = metadata.path == null ? path : metadata.path;
+ }
+ log(request, txnResult, path, operation, metadata, result(outcome), error, outcome, null);
+ } catch (RuntimeException e) {
+ auditError(request.type, e);
+ }
+ }
+
+ private static Record readRequestRecord(Request request) throws IOException {
+ Record record;
+ switch (request.type) {
+ case OpCode.create:
+ case OpCode.create2:
+ case OpCode.createContainer:
+ record = new CreateRequest();
+ break;
+ case OpCode.createTTL:
+ record = new CreateTTLRequest();
+ break;
+ case OpCode.delete:
+ case OpCode.deleteContainer:
+ record = new DeleteRequest();
+ break;
+ case OpCode.setData:
+ record = new SetDataRequest();
+ break;
+ case OpCode.setACL:
+ record = new SetACLRequest();
+ break;
+ case OpCode.multi:
+ record = new MultiOperationRecord();
+ break;
+ default:
+ throw new IOException("Unsupported audit request type: " + request.type);
+ }
+ if (request.request == null) {
+ throw new IOException("Audit request record is unavailable");
+ }
+ ByteBuffer buffer = request.request.duplicate();
+ buffer.rewind();
+ ByteBufferInputStream.byteBuffer2Record(buffer, record);
+ return record;
+ }
+
+ private static void logMultiOperation(Request request, ProcessTxnResult rc, boolean failedTxn) {
+ boolean enhanced = ZKAuditProvider.isEnhancedAuditEnabled();
+ Integer error = errorCode(request, rc, failedTxn);
+ boolean failed = failedTxn || (error != null && error != Code.OK.intValue());
+ int failureIndex = -1;
+ if (rc != null && rc.multiResult != null) {
+ for (int index = 0; index < rc.multiResult.size(); index++) {
+ ProcessTxnResult subResult = rc.multiResult.get(index);
+ if (subResult != null && (subResult.type == OpCode.error || subResult.err != Code.OK.intValue())) {
+ failed = true;
+ if (failureIndex < 0 && subResult.err != Code.OK.intValue()) {
+ failureIndex = index;
}
- break;
- case ZooDefs.OpCode.multi:
- if (failedTxn) {
- op = AuditConstants.OP_MULTI_OP;
- } else {
- logMultiOperation(request, txnResult);
- //operation si already logged
- return;
+ if (error == null || error == Code.OK.intValue()) {
+ error = subResult.err;
}
+ }
+ }
+ }
+ // Emit the failed parent before decoding so malformed metadata cannot hide the failure.
+ if (failed) {
+ log(request, rc, rc == null ? null : rc.path, AuditConstants.OP_MULTI_OP,
+ new RequestMetadata(), Result.FAILURE, error, Outcome.FAILED, null);
+ if (!enhanced) {
+ return;
+ }
+ }
+ MultiOperationRecord multiRequest;
+ try {
+ multiRequest = (MultiOperationRecord) readRequestRecord(request);
+ } catch (IOException e) {
+ auditError(request.type, e);
+ return;
+ }
+ boolean complete = rc != null && rc.type == OpCode.multi && rc.multiResult != null
+ && rc.multiResult.size() == multiRequest.size();
+ if (complete) {
+ int index = 0;
+ for (Op op : multiRequest) {
+ ProcessTxnResult subResult = rc.multiResult.get(index++);
+ if (subResult == null || (subResult.type != OpCode.error
+ && subResult.type != op.getType()
+ && !(subResult.type == OpCode.create2 && op.getType() == OpCode.create))) {
+ complete = false;
break;
- case ZooDefs.OpCode.reconfig:
- op = AuditConstants.OP_RECONFIG;
- break;
- default:
- //Not an audit log operation
- return;
+ }
}
- Result result = getResult(txnResult, failedTxn);
- log(request, path, op, acls, createMode, result);
- } catch (Throwable e) {
- LOG.error("Failed to audit log request {}", request.type, e);
+ }
+ if (!complete) {
+ auditError(request.type, new IOException("Audit multi request/result positions do not match"));
+ }
+ int index = 0;
+ for (Op op : multiRequest) {
+ ProcessTxnResult subResult = complete ? rc.multiResult.get(index) : null;
+ String operation = operationFor(op.getType());
+ if (operation != null) {
+ Outcome outcome;
+ if (!complete) {
+ outcome = Outcome.UNKNOWN;
+ } else if (failed) {
+ // A zero-code error member was rolled back, not committed. Later members are not applied.
+ outcome = subResult.err == Code.OK.intValue()
+ || (index > failureIndex && subResult.err == Code.RUNTIMEINCONSISTENCY.intValue())
+ ? Outcome.ROLLED_BACK : Outcome.FAILED;
+ } else {
+ outcome = Outcome.COMMITTED;
+ }
+ RequestMetadata metadata = metadata(op.getType(), op.toRequestRecord(), enhanced);
+ String path = outcome == Outcome.COMMITTED ? subResult.path : op.getPath();
+ log(request, rc, path, operation, metadata, failed ? Result.FAILURE : result(outcome),
+ subResult == null ? null : subResult.err, outcome, index);
+ }
+ index++;
}
}
- private static void deserialize(Request request, Record record) throws IOException {
- request.request.rewind();
- ByteBufferInputStream.byteBuffer2Record(request.request.slice(), record);
- }
-
- private static Result getResult(ProcessTxnResult rc, boolean failedTxn) {
- if (failedTxn) {
- return Result.FAILURE;
- } else {
- return rc.err == KeeperException.Code.OK.intValue() ? Result.SUCCESS : Result.FAILURE;
+ private static RequestMetadata metadata(int type, Record record, boolean enhanced) {
+ RequestMetadata metadata = new RequestMetadata();
+ switch (type) {
+ case OpCode.create:
+ case OpCode.create2:
+ case OpCode.createContainer:
+ CreateRequest create = (CreateRequest) record;
+ metadata.path = create.getPath();
+ metadata.dataLength = length(create.getData());
+ metadata.createMode = createMode(type, create.getFlags());
+ break;
+ case OpCode.createTTL:
+ CreateTTLRequest ttl = (CreateTTLRequest) record;
+ metadata.path = ttl.getPath();
+ metadata.dataLength = length(ttl.getData());
+ metadata.createMode = createMode(type, ttl.getFlags());
+ break;
+ case OpCode.setData:
+ SetDataRequest setData = (SetDataRequest) record;
+ metadata.path = setData.getPath();
+ metadata.dataLength = length(setData.getData());
+ break;
+ case OpCode.delete:
+ case OpCode.deleteContainer:
+ metadata.path = ((DeleteRequest) record).getPath();
+ break;
+ case OpCode.setACL:
+ SetACLRequest setAcl = (SetACLRequest) record;
+ metadata.path = setAcl.getPath();
+ if (setAcl.getAcl() != null) {
+ metadata.acl = enhanced ? safeAclToString(setAcl.getAcl()) : ZKUtil.aclToString(setAcl.getAcl());
+ }
+ break;
+ default:
+ break;
}
+ return metadata;
}
- private static void logMultiOperation(Request request, ProcessTxnResult rc) throws IOException, KeeperException {
- Map createModes = AuditHelper.getCreateModes(request);
- boolean multiFailed = false;
- for (ProcessTxnResult subTxnResult : rc.multiResult) {
- switch (subTxnResult.type) {
- case ZooDefs.OpCode.create:
- case ZooDefs.OpCode.create2:
- case ZooDefs.OpCode.createTTL:
- case ZooDefs.OpCode.createContainer:
- log(request, subTxnResult.path, AuditConstants.OP_CREATE, null,
- createModes.get(subTxnResult.path), Result.SUCCESS);
- break;
- case ZooDefs.OpCode.delete:
- case ZooDefs.OpCode.deleteContainer:
- log(request, subTxnResult.path, AuditConstants.OP_DELETE, null,
- null, Result.SUCCESS);
- break;
- case ZooDefs.OpCode.setData:
- log(request, subTxnResult.path, AuditConstants.OP_SETDATA, null,
- null, Result.SUCCESS);
- break;
- case ZooDefs.OpCode.error:
- multiFailed = true;
- break;
- default:
- // Do nothing, it ok, we do not log all multi operations
+ private static String safeAclToString(List acls) {
+ StringBuilder value = new StringBuilder();
+ for (ACL acl : acls) {
+ Id id = acl.getId();
+ String user = "[redacted]";
+ if ("world".equals(id.getScheme()) && "anyone".equals(id.getId())) {
+ user = "anyone";
+ } else if ("digest".equals(id.getScheme())) {
+ AuthenticationProvider provider = ProviderRegistry.getProvider(id.getScheme());
+ if (provider != null && provider.getClass() == DigestAuthenticationProvider.class
+ && id.getId() != null && provider.isValid(id.getId())) {
+ user = AuthUtil.getUser(id);
+ }
}
+ value.append(id.getScheme()).append(':').append(user).append(':')
+ .append(ZKUtil.getPermString(acl.getPerms()));
}
- if (multiFailed) {
- log(request, rc.path, AuditConstants.OP_MULTI_OP, null,
- null, Result.FAILURE);
- }
+ return value.toString();
}
- private static void log(Request request, String path, String op, String acls, String createMode, Result result) {
- log(request.getUsers(), op, path, acls, createMode,
- request.cnxn.getSessionIdHex(), request.cnxn.getHostAddress(), result);
+ private static String createMode(int type, int flags) {
+ try {
+ return CreateMode.fromFlag(flags).name().toLowerCase(Locale.ROOT);
+ } catch (KeeperException e) {
+ auditError(type, e);
+ return null;
+ }
}
- private static void log(String user, String operation, String znode, String acl,
- String createMode, String session, String ip, Result result) {
- ZKAuditProvider.log(user, operation, znode, acl, createMode, session, ip, result);
+ private static int length(byte[] data) {
+ return data == null ? 0 : data.length;
}
- private static String getACLs(Request request) throws IOException {
- SetACLRequest setACLRequest = new SetACLRequest();
- deserialize(request, setACLRequest);
- return ZKUtil.aclToString(setACLRequest.getAcl());
+ private static Integer errorCode(Request request, ProcessTxnResult rc, boolean failedTxn) {
+ if (failedTxn && request.getException() != null) {
+ return request.getException().code().intValue();
+ }
+ return rc == null || rc.type == 0 ? null : rc.err;
}
- private static String getCreateMode(Request request) throws IOException, KeeperException {
- CreateRequest createRequest = new CreateRequest();
- deserialize(request, createRequest);
- return getCreateMode(createRequest);
+ private static Outcome outcome(ProcessTxnResult rc, boolean failedTxn, Integer error) {
+ if (failedTxn || (rc != null && rc.type == OpCode.error)
+ || (error != null && error != Code.OK.intValue())) {
+ return Outcome.FAILED;
+ }
+ return error == null ? Outcome.UNKNOWN : Outcome.COMMITTED;
}
- private static String getCreateMode(CreateRequest createRequest) throws KeeperException {
- return CreateMode.fromFlag(createRequest.getFlags()).toString().toLowerCase();
+ private static Result result(Outcome outcome) {
+ return outcome == Outcome.COMMITTED ? Result.SUCCESS : outcome == Outcome.UNKNOWN ? Result.INVOKED : Result.FAILURE;
}
- private static Map getCreateModes(Request request)
- throws IOException, KeeperException {
- Map createModes = new HashMap<>();
- if (!ZKAuditProvider.isAuditEnabled()) {
- return createModes;
+ private static String operationFor(int type) {
+ switch (type) {
+ case OpCode.create:
+ case OpCode.create2:
+ case OpCode.createTTL:
+ case OpCode.createContainer:
+ return AuditConstants.OP_CREATE;
+ case OpCode.delete:
+ case OpCode.deleteContainer:
+ return AuditConstants.OP_DELETE;
+ case OpCode.setData:
+ return AuditConstants.OP_SETDATA;
+ case OpCode.setACL:
+ return AuditConstants.OP_SETACL;
+ case OpCode.reconfig:
+ return AuditConstants.OP_RECONFIG;
+ default:
+ return null;
}
- MultiOperationRecord multiRequest = new MultiOperationRecord();
- deserialize(request, multiRequest);
- for (Op op : multiRequest) {
- if (op.getType() == ZooDefs.OpCode.create || op.getType() == ZooDefs.OpCode.create2
- || op.getType() == ZooDefs.OpCode.createContainer) {
- CreateRequest requestRecord = (CreateRequest) op.toRequestRecord();
- createModes.put(requestRecord.getPath(),
- getCreateMode(requestRecord));
- }
+ }
+
+ private static boolean isCreate(int type) {
+ return type == OpCode.create || type == OpCode.create2 || type == OpCode.createTTL || type == OpCode.createContainer;
+ }
+
+ private static void log(Request request, ProcessTxnResult rc, String path, String operation,
+ RequestMetadata metadata, Result result, Integer error, Outcome outcome, Integer index) {
+ Long zxid = null;
+ if (request.getHdr() != null) {
+ zxid = request.getHdr().getZxid();
+ } else if (rc != null && rc.type != 0) {
+ zxid = rc.zxid;
+ } else if (request.zxid >= 0) {
+ zxid = request.zxid;
}
- return createModes;
+ ZKAuditProvider.log(request.getUsers(), operation, path, metadata.acl, metadata.createMode,
+ request.cnxn.getSessionIdHex(), request.cnxn.getHostAddress(), result,
+ metadata.dataLength, error, outcome, request.cxid, zxid, index);
}
+ private static void auditError(int type, Exception e) {
+ ServerMetrics.getMetrics().AUDIT_ERRORS.add(1);
+ LOG.error("Failed to audit log request {}", type, e);
+ }
+
+ private static final class RequestMetadata {
+ private String path;
+ private String acl;
+ private String createMode;
+ private Integer dataLength;
+ }
}
diff --git a/zookeeper-server/src/main/java/org/apache/zookeeper/audit/ZKAuditProvider.java b/zookeeper-server/src/main/java/org/apache/zookeeper/audit/ZKAuditProvider.java
index 864d2366c97..701ab71a622 100644
--- a/zookeeper-server/src/main/java/org/apache/zookeeper/audit/ZKAuditProvider.java
+++ b/zookeeper-server/src/main/java/org/apache/zookeeper/audit/ZKAuditProvider.java
@@ -19,13 +19,17 @@
import static org.apache.zookeeper.audit.AuditEvent.FieldName;
import java.lang.reflect.Constructor;
+import java.util.Locale;
+import org.apache.zookeeper.audit.AuditEvent.Outcome;
import org.apache.zookeeper.audit.AuditEvent.Result;
import org.apache.zookeeper.server.ServerCnxnFactory;
+import org.apache.zookeeper.server.ServerMetrics;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
public class ZKAuditProvider {
static final String AUDIT_ENABLE = "zookeeper.audit.enable";
+ static final String AUDIT_ENHANCED_ENABLE = "zookeeper.audit.enhanced.enable";
static final String AUDIT_IMPL_CLASS = "zookeeper.audit.impl.class";
private static final Logger LOG = LoggerFactory.getLogger(ZKAuditProvider.class);
// By default audit logging is disabled
@@ -66,9 +70,28 @@ public static boolean isAuditEnabled() {
return auditEnabled;
}
+ public static boolean isEnhancedAuditEnabled() {
+ return auditEnabled && Boolean.getBoolean(AUDIT_ENHANCED_ENABLE);
+ }
+
public static void log(String user, String operation, String znode, String acl,
String createMode, String session, String ip, Result result) {
- auditLogger.logAuditEvent(createLogEvent(user, operation, znode, acl, createMode, session, ip, result));
+ log(user, operation, znode, acl, createMode, session, ip, result, null, null, null, null, null, null);
+ }
+
+ /**
+ * Logs optional operation metadata only when enhanced audit logging is enabled.
+ * Null metadata is unavailable, not zero. Callers must sanitize ACL identities.
+ */
+ public static void log(String user, String operation, String znode, String acl,
+ String createMode, String session, String ip, Result result,
+ Integer dataLength, Integer errorCode, Outcome outcome,
+ Integer cxid, Long zxid, Integer multiIndex) {
+ if (!isAuditEnabled()) {
+ return;
+ }
+ logAuditEvent(createLogEvent(user, operation, znode, acl, createMode, session, ip, result,
+ dataLength, errorCode, outcome, cxid, zxid, multiIndex));
}
/**
@@ -78,6 +101,7 @@ static AuditEvent createLogEvent(String user, String operation, Result result) {
AuditEvent event = new AuditEvent(result);
event.addEntry(FieldName.USER, user);
event.addEntry(FieldName.OPERATION, operation);
+ addMetadata(event, null, null, null, null, null, null);
return event;
}
@@ -86,6 +110,14 @@ static AuditEvent createLogEvent(String user, String operation, Result result) {
*/
static AuditEvent createLogEvent(String user, String operation, String znode, String acl,
String createMode, String session, String ip, Result result) {
+ return createLogEvent(user, operation, znode, acl, createMode, session, ip, result,
+ null, null, null, null, null, null);
+ }
+
+ static AuditEvent createLogEvent(String user, String operation, String znode, String acl,
+ String createMode, String session, String ip, Result result,
+ Integer dataLength, Integer errorCode, Outcome outcome,
+ Integer cxid, Long zxid, Integer multiIndex) {
AuditEvent event = new AuditEvent(result);
event.addEntry(FieldName.SESSION, session);
event.addEntry(FieldName.USER, user);
@@ -94,9 +126,36 @@ static AuditEvent createLogEvent(String user, String operation, String znode, St
event.addEntry(FieldName.ZNODE, znode);
event.addEntry(FieldName.ZNODE_TYPE, createMode);
event.addEntry(FieldName.ACL, acl);
+ addMetadata(event, dataLength, errorCode, outcome, cxid, zxid, multiIndex);
return event;
}
+ private static void addMetadata(AuditEvent event, Integer dataLength, Integer errorCode, Outcome outcome,
+ Integer cxid, Long zxid, Integer multiIndex) {
+ if (isEnhancedAuditEnabled()) {
+ event.addEntry(FieldName.SCHEMA_VERSION, AuditConstants.SCHEMA_VERSION);
+ event.addEntry(FieldName.DATA_LENGTH, valueOf(dataLength));
+ event.addEntry(FieldName.ERROR_CODE, valueOf(errorCode));
+ event.addEntry(FieldName.OUTCOME, (outcome == null ? Outcome.UNKNOWN : outcome).name().toLowerCase(Locale.ROOT));
+ event.addEntry(FieldName.CXID, valueOf(cxid));
+ event.addEntry(FieldName.ZXID, valueOf(zxid));
+ event.addEntry(FieldName.MULTI_INDEX, valueOf(multiIndex));
+ }
+ }
+
+ private static String valueOf(Number value) {
+ return value == null ? null : value.toString();
+ }
+
+ private static void logAuditEvent(AuditEvent event) {
+ try {
+ auditLogger.logAuditEvent(event);
+ } catch (RuntimeException e) {
+ ServerMetrics.getMetrics().AUDIT_ERRORS.add(1);
+ LOG.error("Failed to write audit log for operation {}", event.getValue(FieldName.OPERATION), e);
+ }
+ }
+
/**
* Add audit log for server start and register server stop log.
*/
@@ -119,7 +178,7 @@ public static void addServerStartFailureAuditLog() {
}
private static void log(String user, String operation, Result result) {
- auditLogger.logAuditEvent(createLogEvent(user, operation, result));
+ logAuditEvent(createLogEvent(user, operation, result));
}
/**
diff --git a/zookeeper-server/src/main/java/org/apache/zookeeper/server/DataTree.java b/zookeeper-server/src/main/java/org/apache/zookeeper/server/DataTree.java
index 1a5d1304120..80d0173afe6 100644
--- a/zookeeper-server/src/main/java/org/apache/zookeeper/server/DataTree.java
+++ b/zookeeper-server/src/main/java/org/apache/zookeeper/server/DataTree.java
@@ -59,6 +59,7 @@
import org.apache.zookeeper.ZooDefs;
import org.apache.zookeeper.ZooDefs.OpCode;
import org.apache.zookeeper.audit.AuditConstants;
+import org.apache.zookeeper.audit.AuditEvent.Outcome;
import org.apache.zookeeper.audit.AuditEvent.Result;
import org.apache.zookeeper.audit.ZKAuditProvider;
import org.apache.zookeeper.common.PathTrie;
@@ -1451,15 +1452,11 @@ void deleteNodes(long session, long zxid, Iterable paths2Delete) {
path, sessionHex);
}
if (ZKAuditProvider.isAuditEnabled()) {
- if (deleted) {
- ZKAuditProvider.log(ZKAuditProvider.getZKUser(),
- AuditConstants.OP_DEL_EZNODE_EXP, path, null, null,
- sessionHex, null, Result.SUCCESS);
- } else {
- ZKAuditProvider.log(ZKAuditProvider.getZKUser(),
- AuditConstants.OP_DEL_EZNODE_EXP, path, null, null,
- sessionHex, null, Result.FAILURE);
- }
+ ZKAuditProvider.log(ZKAuditProvider.getZKUser(),
+ AuditConstants.OP_DEL_EZNODE_EXP, path, null, null,
+ sessionHex, null, deleted ? Result.SUCCESS : Result.FAILURE,
+ null, deleted ? Code.OK.intValue() : Code.NONODE.intValue(),
+ deleted ? Outcome.COMMITTED : Outcome.FAILED, null, zxid, null);
}
}
}
diff --git a/zookeeper-server/src/main/java/org/apache/zookeeper/server/ServerMetrics.java b/zookeeper-server/src/main/java/org/apache/zookeeper/server/ServerMetrics.java
index c9cfcc12a52..4b5ba818106 100644
--- a/zookeeper-server/src/main/java/org/apache/zookeeper/server/ServerMetrics.java
+++ b/zookeeper-server/src/main/java/org/apache/zookeeper/server/ServerMetrics.java
@@ -227,6 +227,7 @@ private ServerMetrics(MetricsProvider metricsProvider) {
STALE_REQUESTS = metricsContext.getCounter("stale_requests");
STALE_REQUESTS_DROPPED = metricsContext.getCounter("stale_requests_dropped");
STALE_REPLIES = metricsContext.getCounter("stale_replies");
+ AUDIT_ERRORS = metricsContext.getCounter("audit_errors");
REQUEST_THROTTLE_WAIT_COUNT = metricsContext.getCounter("request_throttle_wait_count");
LARGE_REQUESTS_REJECTED = metricsContext.getCounter("large_requests_rejected");
@@ -443,6 +444,7 @@ private ServerMetrics(MetricsProvider metricsProvider) {
public final Counter STALE_REQUESTS;
public final Counter STALE_REQUESTS_DROPPED;
public final Counter STALE_REPLIES;
+ public final Counter AUDIT_ERRORS;
public final Counter REQUEST_THROTTLE_WAIT_COUNT;
public final Counter LARGE_REQUESTS_REJECTED;
diff --git a/zookeeper-server/src/test/java/org/apache/zookeeper/audit/AuditEventTest.java b/zookeeper-server/src/test/java/org/apache/zookeeper/audit/AuditEventTest.java
index 02d9ac0bb85..d5f371a7a6a 100644
--- a/zookeeper-server/src/test/java/org/apache/zookeeper/audit/AuditEventTest.java
+++ b/zookeeper-server/src/test/java/org/apache/zookeeper/audit/AuditEventTest.java
@@ -18,6 +18,8 @@
package org.apache.zookeeper.audit;
import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import java.util.Locale;
import org.apache.zookeeper.audit.AuditEvent.Result;
import org.junit.Test;
@@ -42,4 +44,55 @@ public void testFormatShouldIgnoreKeyIfValueIsNull() {
String expected = "operation=Value2\tresult=success";
assertEquals(expected, actual);
}
+
+ @Test
+ public void testProtocolNamesDoNotDependOnDefaultLocale() {
+ Locale previous = Locale.getDefault();
+ try {
+ Locale.setDefault(new Locale("tr", "TR"));
+ AuditEvent event = new AuditEvent(Result.FAILURE);
+ event.addEntry(AuditEvent.FieldName.IP, "127.0.0.1");
+ assertEquals("ip=127.0.0.1\tresult=failure", event.toString());
+ assertEquals("result=invoked", new AuditEvent(Result.INVOKED).toString());
+ } finally {
+ Locale.setDefault(previous);
+ }
+ }
+
+ @Test
+ public void testEnhancedFormattingEscapesValuesReversibly() {
+ String previousAudit = System.getProperty(ZKAuditProvider.AUDIT_ENABLE);
+ String previousEnhanced = System.getProperty(AuditHelperTest.ENHANCED_ENABLE);
+ try {
+ System.setProperty(ZKAuditProvider.AUDIT_ENABLE, "true");
+ System.setProperty(AuditHelperTest.ENHANCED_ENABLE, "true");
+ AuditEvent event = ZKAuditProvider.createLogEvent("team\tname\r\n\\t", "setData",
+ "/name=value\\child", null, null, null, null, Result.FAILURE);
+ String log = event.toString();
+ assertEquals("2", AuditHelperTest.fields(log).get("schema_version"));
+ assertEquals("team\\tname\\r\\n\\\\t", AuditHelperTest.fields(log).get("user"));
+ assertEquals("/name=value\\\\child", AuditHelperTest.fields(log).get("znode"));
+ assertEquals("team\tname\r\n\\t", event.getValue(AuditEvent.FieldName.USER));
+ assertFalse(log.contains("\n"));
+ assertFalse(log.contains("\r"));
+ } finally {
+ AuditHelperTest.restoreProperty(ZKAuditProvider.AUDIT_ENABLE, previousAudit);
+ AuditHelperTest.restoreProperty(AuditHelperTest.ENHANCED_ENABLE, previousEnhanced);
+ }
+ }
+
+ @Test
+ public void testLegacyValuesAreNotEscaped() {
+ AuditEvent event = new AuditEvent(Result.SUCCESS);
+ event.addEntry(AuditEvent.FieldName.USER, "team\tname\n\\t");
+ assertEquals("user=team\tname\n\\t\tresult=success", event.toString());
+ }
+
+ @Test
+ public void testSchemaMarkerEscapesPreviouslyAddedValues() {
+ AuditEvent event = new AuditEvent(Result.SUCCESS);
+ event.addEntry(AuditEvent.FieldName.USER, "team\tname\n");
+ event.addEntry(AuditEvent.FieldName.SCHEMA_VERSION, "2");
+ assertEquals("user=team\\tname\\n\tschema_version=2\tresult=success", event.toString());
+ }
}
diff --git a/zookeeper-server/src/test/java/org/apache/zookeeper/audit/AuditHelperTest.java b/zookeeper-server/src/test/java/org/apache/zookeeper/audit/AuditHelperTest.java
new file mode 100644
index 00000000000..a85b1c9b9f7
--- /dev/null
+++ b/zookeeper-server/src/test/java/org/apache/zookeeper/audit/AuditHelperTest.java
@@ -0,0 +1,597 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.zookeeper.audit;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertNull;
+import static org.junit.Assert.assertTrue;
+import static org.junit.Assert.fail;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+import ch.qos.logback.classic.Level;
+import ch.qos.logback.classic.Logger;
+import ch.qos.logback.classic.spi.ILoggingEvent;
+import ch.qos.logback.core.AppenderBase;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.lang.reflect.Field;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+import org.apache.jute.BinaryOutputArchive;
+import org.apache.jute.Record;
+import org.apache.zookeeper.CreateMode;
+import org.apache.zookeeper.KeeperException;
+import org.apache.zookeeper.KeeperException.Code;
+import org.apache.zookeeper.MultiOperationRecord;
+import org.apache.zookeeper.Op;
+import org.apache.zookeeper.ZooDefs;
+import org.apache.zookeeper.ZooDefs.OpCode;
+import org.apache.zookeeper.audit.AuditEvent.Result;
+import org.apache.zookeeper.data.ACL;
+import org.apache.zookeeper.data.Id;
+import org.apache.zookeeper.metrics.MetricsUtils;
+import org.apache.zookeeper.proto.CreateRequest;
+import org.apache.zookeeper.proto.CreateTTLRequest;
+import org.apache.zookeeper.proto.SetACLRequest;
+import org.apache.zookeeper.proto.SetDataRequest;
+import org.apache.zookeeper.server.DataTree;
+import org.apache.zookeeper.server.DataTree.ProcessTxnResult;
+import org.apache.zookeeper.server.Request;
+import org.apache.zookeeper.server.ServerCnxn;
+import org.apache.zookeeper.txn.CheckVersionTxn;
+import org.apache.zookeeper.txn.CloseSessionTxn;
+import org.apache.zookeeper.txn.CreateTTLTxn;
+import org.apache.zookeeper.txn.CreateTxn;
+import org.apache.zookeeper.txn.DeleteTxn;
+import org.apache.zookeeper.txn.ErrorTxn;
+import org.apache.zookeeper.txn.MultiTxn;
+import org.apache.zookeeper.txn.SetDataTxn;
+import org.apache.zookeeper.txn.Txn;
+import org.apache.zookeeper.txn.TxnHeader;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+import org.slf4j.LoggerFactory;
+
+public class AuditHelperTest {
+ static final String ENHANCED_ENABLE = "zookeeper.audit.enhanced.enable";
+ private static final long SESSION = 0x123;
+ private AuditCapture capture;
+ private DataTree tree;
+ private ServerCnxn cnxn;
+ private String previousEnhanced;
+ private String previousAudit;
+ private String previousExtendedTypes;
+
+ @Before
+ public void setUp() {
+ previousAudit = System.getProperty(ZKAuditProvider.AUDIT_ENABLE);
+ previousEnhanced = System.getProperty(ENHANCED_ENABLE);
+ previousExtendedTypes = System.getProperty("zookeeper.extendedTypesEnabled");
+ System.setProperty(ZKAuditProvider.AUDIT_ENABLE, "true");
+ System.setProperty(ENHANCED_ENABLE, "true");
+ System.setProperty("zookeeper.extendedTypesEnabled", "true");
+ assertTrue(ZKAuditProvider.isAuditEnabled());
+ capture = new AuditCapture();
+ tree = new DataTree();
+ cnxn = mock(ServerCnxn.class);
+ when(cnxn.getSessionIdHex()).thenReturn("0x123");
+ when(cnxn.getHostAddress()).thenReturn("127.0.0.1");
+ }
+
+ @After
+ public void tearDown() {
+ capture.close();
+ restoreProperty(ZKAuditProvider.AUDIT_ENABLE, previousAudit);
+ restoreProperty(ENHANCED_ENABLE, previousEnhanced);
+ restoreProperty("zookeeper.extendedTypesEnabled", previousExtendedTypes);
+ }
+
+ @Test
+ public void testCreateAndSetDataLengthsArePayloadBytes() throws Exception {
+ byte[][] data = {null, new byte[0], new byte[1], new byte[1024],
+ new byte[65536], new byte[1048575], "\u00e9\ud83d\ude00".getBytes(StandardCharsets.UTF_8)};
+ String[] lengths = {"0", "0", "1", "1024", "65536", "1048575", "6"};
+ for (int i = 0; i < data.length; i++) {
+ String path = "/length-" + i;
+ Request create = request(OpCode.create, createRecord(path, data[i], CreateMode.PERSISTENT));
+ ProcessTxnResult created = apply(create, OpCode.create, createTxn(path, data[i], false));
+ assertEquals(0, created.err);
+ AuditHelper.addAuditLog(create, created);
+ Map fields = fields(capture.read(1).get(0));
+ assertWrite(fields, "create", path, lengths[i], "committed", "0");
+ assertEquals("41", fields.get("cxid"));
+ assertEquals("66", fields.get("zxid"));
+ assertNull(fields.get("multi_index"));
+
+ Request setData = request(OpCode.setData, new SetDataRequest(path, data[i], -1));
+ ProcessTxnResult changed = apply(setData, OpCode.setData, new SetDataTxn(path, data[i], 1));
+ assertEquals(0, changed.err);
+ AuditHelper.addAuditLog(setData, changed);
+ assertWrite(fields(capture.read(1).get(0)), "setData", path, lengths[i], "committed", "0");
+ }
+ }
+
+ @Test
+ public void testCreateTtlUsesTypedRecord() throws Exception {
+ byte[] data = {1, 2, 3};
+ Request request = request(OpCode.createTTL, new CreateTTLRequest(
+ "/ttl", data, ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT_WITH_TTL.toFlag(), 60000));
+ ProcessTxnResult result = apply(request, OpCode.createTTL,
+ new CreateTTLTxn("/ttl", data, ZooDefs.Ids.OPEN_ACL_UNSAFE, -1, 60000));
+ assertEquals(0, result.err);
+ AuditHelper.addAuditLog(request, result);
+ Map fields = fields(capture.read(1).get(0));
+ assertWrite(fields, "create", "/ttl", "3", "committed", "0");
+ assertEquals("persistent_with_ttl", fields.get("znode_type"));
+ }
+
+ @Test
+ public void testFailureUsesAttemptedLengthAndReplyException() throws Exception {
+ Request request = request(OpCode.create, createRecord("/denied", new byte[7], CreateMode.PERSISTENT));
+ ProcessTxnResult result = apply(request, OpCode.error, new ErrorTxn(Code.SESSIONEXPIRED.intValue()));
+ request.setException(KeeperException.create(Code.NOAUTH));
+ AuditHelper.addAuditLog(request, result, true);
+ assertWrite(fields(capture.read(1).get(0)), "create", "/denied", "7", "failed", "-102");
+ assertNull(tree.getNode("/denied"));
+ }
+
+ @Test
+ public void testFailedMultiNeverCommitsZeroCodeErrorMembers() throws Exception {
+ Request request = request(OpCode.multi, new MultiOperationRecord(Arrays.asList(
+ Op.create("/rolled", new byte[1], ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT),
+ Op.check("/", -1),
+ Op.setData("/missing", new byte[3], -1),
+ Op.create("/later", new byte[2], ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT))));
+ ProcessTxnResult result = apply(request, OpCode.multi, new MultiTxn(Arrays.asList(
+ txn(OpCode.create, createTxn("/rolled", new byte[1], false)),
+ txn(OpCode.check, new CheckVersionTxn("/", -1)),
+ txn(OpCode.error, new ErrorTxn(Code.NONODE.intValue())),
+ txn(OpCode.error, new ErrorTxn(Code.RUNTIMEINCONSISTENCY.intValue())))));
+ assertEquals(-101, result.err);
+ assertEquals(OpCode.error, result.multiResult.get(0).type);
+ assertEquals(0, result.multiResult.get(0).err);
+ assertNull(tree.getNode("/rolled"));
+ assertNull(tree.getNode("/later"));
+
+ AuditHelper.addAuditLog(request, result);
+ List logs = capture.read(4);
+ assertWrite(fields(logs.get(0)), "multiOperation", null, null, "failed", "-101");
+ assertWrite(fields(logs.get(1)), "create", "/rolled", "1", "rolled_back", "0");
+ assertWrite(fields(logs.get(2)), "setData", "/missing", "3", "failed", "-101");
+ assertWrite(fields(logs.get(3)), "create", "/later", "2", "rolled_back", "-2");
+ assertEquals("0", fields(logs.get(1)).get("multi_index"));
+ assertEquals("2", fields(logs.get(2)).get("multi_index"));
+ assertEquals("3", fields(logs.get(3)).get("multi_index"));
+ for (String log : logs) {
+ assertFalse(log, log.contains("result=success"));
+ assertFalse(log, log.contains("outcome=committed"));
+ }
+ }
+
+ @Test
+ public void testMultiMatchesRepeatedPathsAndTtlByPosition() throws Exception {
+ Request request = request(OpCode.multi, new MultiOperationRecord(Arrays.asList(
+ Op.create("/same", new byte[1], ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT),
+ Op.delete("/same", -1),
+ Op.check("/", -1),
+ Op.create("/same", new byte[2], ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.EPHEMERAL),
+ Op.create("/seq-", new byte[3], ZooDefs.Ids.OPEN_ACL_UNSAFE,
+ CreateMode.PERSISTENT_SEQUENTIAL_WITH_TTL, 60000))));
+ ProcessTxnResult result = apply(request, OpCode.multi, new MultiTxn(Arrays.asList(
+ txn(OpCode.create, createTxn("/same", new byte[1], false)),
+ txn(OpCode.delete, new DeleteTxn("/same")),
+ txn(OpCode.check, new CheckVersionTxn("/", -1)),
+ txn(OpCode.create, createTxn("/same", new byte[2], true)),
+ txn(OpCode.createTTL, new CreateTTLTxn(
+ "/seq-0000000002", new byte[3], ZooDefs.Ids.OPEN_ACL_UNSAFE, -1, 60000)))));
+ assertEquals(0, result.err);
+ assertNotNull(tree.getNode("/seq-0000000002"));
+
+ AuditHelper.addAuditLog(request, result);
+ List logs = capture.read(4);
+ assertWrite(fields(logs.get(0)), "create", "/same", "1", "committed", "0");
+ assertEquals("persistent", fields(logs.get(0)).get("znode_type"));
+ assertEquals("ephemeral", fields(logs.get(2)).get("znode_type"));
+ assertEquals("3", fields(logs.get(2)).get("multi_index"));
+ assertWrite(fields(logs.get(3)), "create", "/seq-0000000002", "3", "committed", "0");
+ assertEquals("persistent_sequential_with_ttl", fields(logs.get(3)).get("znode_type"));
+ assertEquals("4", fields(logs.get(3)).get("multi_index"));
+ }
+
+ @Test
+ public void testIncompleteMultiResultsDoNotMisattributeCodes() throws Exception {
+ Request request = request(OpCode.multi, new MultiOperationRecord(Arrays.asList(
+ Op.create("/incomplete", new byte[2], ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT),
+ Op.check("/", -1))));
+ ProcessTxnResult result = apply(request, OpCode.multi, new MultiTxn(Arrays.asList(
+ txn(OpCode.create, createTxn("/incomplete", new byte[2], false)),
+ txn(OpCode.check, new CheckVersionTxn("/", -1)))));
+ assertEquals(0, result.err);
+ result.multiResult.remove(0);
+ long before = auditErrors();
+ AuditHelper.addAuditLog(request, result);
+ assertWrite(fields(capture.read(1).get(0)), "create", "/incomplete", "2", "unknown", null);
+ assertEquals(before + 1, auditErrors());
+ }
+
+ @Test
+ public void testFirstRuntimeInconsistencyIsFailureNotSkippedMember() throws Exception {
+ Request request = request(OpCode.multi, new MultiOperationRecord(Arrays.asList(
+ Op.create("/first", new byte[1], ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT),
+ Op.create("/skipped", new byte[2], ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT))));
+ ProcessTxnResult result = apply(request, OpCode.multi, new MultiTxn(Arrays.asList(
+ txn(OpCode.error, new ErrorTxn(Code.RUNTIMEINCONSISTENCY.intValue())),
+ txn(OpCode.error, new ErrorTxn(Code.RUNTIMEINCONSISTENCY.intValue())))));
+ assertNull(tree.getNode("/first"));
+ assertNull(tree.getNode("/skipped"));
+ AuditHelper.addAuditLog(request, result);
+ List logs = capture.read(3);
+ assertWrite(fields(logs.get(1)), "create", "/first", "1", "failed", "-2");
+ assertWrite(fields(logs.get(2)), "create", "/skipped", "2", "rolled_back", "-2");
+ }
+
+ @Test
+ public void testDecodingPreservesPositionLimitAndMark() throws Exception {
+ for (int kind = 0; kind < 3; kind++) {
+ String path = "/buffer-" + kind;
+ byte[] bytes = serialize(createRecord(path, new byte[3], CreateMode.PERSISTENT));
+ ByteBuffer storage = kind == 0
+ ? ByteBuffer.allocateDirect(bytes.length + 12) : ByteBuffer.allocate(bytes.length + 12);
+ storage.position(4);
+ storage.put(bytes);
+ storage.position(4);
+ ByteBuffer buffer = storage.slice();
+ buffer.limit(bytes.length);
+ if (kind == 2) {
+ buffer = buffer.asReadOnlyBuffer();
+ }
+ buffer.position(2);
+ buffer.mark();
+ buffer.position(bytes.length);
+ Request request = request(OpCode.create, buffer);
+ ProcessTxnResult result = apply(request, OpCode.create, createTxn(path, new byte[3], false));
+ AuditHelper.addAuditLog(request, result);
+ assertEquals(bytes.length, buffer.position());
+ assertEquals(bytes.length, buffer.limit());
+ buffer.reset();
+ assertEquals(2, buffer.position());
+ assertWrite(fields(capture.read(1).get(0)), "create", path, "3", "committed", "0");
+ }
+ }
+
+ @Test
+ public void testMultiDecodingPreservesPositionLimitAndMark() throws Exception {
+ Request request = request(OpCode.multi, new MultiOperationRecord(Collections.singletonList(
+ Op.create("/buffer-multi", new byte[2], ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT))));
+ request.request.position(3);
+ request.request.mark();
+ int limit = request.request.limit();
+ request.request.position(limit);
+ ProcessTxnResult result = apply(request, OpCode.multi, new MultiTxn(Collections.singletonList(
+ txn(OpCode.create, createTxn("/buffer-multi", new byte[2], false)))));
+ AuditHelper.addAuditLog(request, result);
+ assertEquals(limit, request.request.position());
+ assertEquals(limit, request.request.limit());
+ request.request.reset();
+ assertEquals(3, request.request.position());
+ assertWrite(fields(capture.read(1).get(0)), "create", "/buffer-multi", "2", "committed", "0");
+ }
+
+ @Test
+ public void testUnavailablePayloadIsOmittedAndDecodeErrorsAreCounted() throws Exception {
+ for (ByteBuffer buffer : Arrays.asList(null, ByteBuffer.wrap(new byte[] {0, 0, 0, 10, 1}))) {
+ String path = buffer == null ? "/unavailable" : "/undecodable";
+ Request request = request(OpCode.create, buffer);
+ ProcessTxnResult result = apply(request, OpCode.create, createTxn(path, new byte[5], false));
+ long before = auditErrors();
+ AuditHelper.addAuditLog(request, result);
+ assertWrite(fields(capture.read(1).get(0)), "create", path, null, "committed", "0");
+ assertEquals(before + 1, auditErrors());
+ assertNotNull(tree.getNode(path));
+ }
+ }
+
+ @Test
+ public void testFailedMultiParentSurvivesUndecodableRequest() throws Exception {
+ Request request = request(OpCode.multi, ByteBuffer.wrap(new byte[] {1}));
+ ProcessTxnResult result = apply(request, OpCode.multi, new MultiTxn(Collections.singletonList(
+ txn(OpCode.error, new ErrorTxn(Code.NOAUTH.intValue())))));
+ long before = auditErrors();
+ AuditHelper.addAuditLog(request, result);
+ assertWrite(fields(capture.read(1).get(0)), "multiOperation", null, null, "failed", "-102");
+ assertEquals(before + 1, auditErrors());
+ }
+
+ @Test
+ public void testMissingResultDoesNotInventSuccessOrZxid() throws Exception {
+ Request request = request(OpCode.setData, new SetDataRequest("/unknown", new byte[4], -1));
+ try {
+ AuditHelper.addAuditLog(request, null);
+ } catch (RuntimeException e) {
+ fail("Unavailable transaction results must not escape the audit boundary: " + e);
+ }
+ Map fields = fields(capture.read(1).get(0));
+ assertWrite(fields, "setData", "/unknown", "4", "unknown", null);
+ assertEquals("invoked", fields.get("result"));
+ assertNull(fields.get("zxid"));
+ }
+
+ @Test
+ public void testEnhancedSetAclDoesNotExposeCredentials() throws Exception {
+ List acls = Arrays.asList(
+ new ACL(ZooDefs.Perms.ALL, new Id("world", "anyone")),
+ new ACL(ZooDefs.Perms.READ, new Id("digest", "alice:synthetic-digest-secret")),
+ new ACL(ZooDefs.Perms.WRITE, new Id("custom", "synthetic-token-secret")),
+ new ACL(ZooDefs.Perms.READ, new Id("x509", "-----BEGIN CERTIFICATE-----synthetic-body")));
+ Request request = request(OpCode.setACL, new SetACLRequest("/acl", acls, -1));
+ ProcessTxnResult result = apply(request, OpCode.error, new ErrorTxn(Code.INVALIDACL.intValue()));
+ AuditHelper.addAuditLog(request, result, true);
+ String log = capture.read(1).get(0);
+ assertWrite(fields(log), "setAcl", "/acl", null, "failed", "-114");
+ assertTrue(log, log.contains("world:anyone:cdrwa"));
+ assertTrue(log, log.contains("digest:alice:r"));
+ assertFalse(log, log.contains("synthetic-digest-secret"));
+ assertFalse(log, log.contains("synthetic-token-secret"));
+ assertFalse(log, log.contains("BEGIN CERTIFICATE"));
+ assertFalse(log, log.contains("synthetic-body"));
+ }
+
+ @Test
+ public void testMalformedDigestIdentityIsNotMistakenForAUser() throws Exception {
+ Request request = request(OpCode.setACL, new SetACLRequest("/acl", Collections.singletonList(
+ new ACL(ZooDefs.Perms.READ, new Id("digest", "synthetic-digest-token"))), -1));
+ ProcessTxnResult result = apply(request, OpCode.error, new ErrorTxn(Code.INVALIDACL.intValue()));
+ AuditHelper.addAuditLog(request, result, true);
+ String log = capture.read(1).get(0);
+ assertFalse(log, log.contains("synthetic-digest-token"));
+ assertWrite(fields(log), "setAcl", "/acl", null, "failed", "-114");
+ }
+
+ @Test
+ public void testAuditDisabledSkipsRequestsAndProvider() throws Exception {
+ Object previous = replaceProviderField("auditEnabled", false);
+ long before = auditErrors();
+ try {
+ AuditHelper.addAuditLog(null, null);
+ ZKAuditProvider.log("user", "create", "/disabled", null, null, null, null, Result.SUCCESS);
+ capture.read(0);
+ assertEquals(before, auditErrors());
+ } finally {
+ replaceProviderField("auditEnabled", previous);
+ }
+ }
+
+ @Test
+ public void testDefaultAndExplicitLegacyOutputAreUnchanged() throws Exception {
+ for (String setting : Arrays.asList(null, "false")) {
+ restoreProperty(ENHANCED_ENABLE, setting);
+ ZKAuditProvider.log("user", "create", "/legacy", null, "persistent", "0x123",
+ "127.0.0.1", Result.SUCCESS);
+ assertEquals("session=0x123\tuser=user\tip=127.0.0.1\toperation=create"
+ + "\tznode=/legacy\tznode_type=persistent\tresult=success", capture.read(1).get(0));
+ }
+ }
+
+ @Test
+ public void testLegacyFailedMultiStillHasOnlyParent() throws Exception {
+ System.clearProperty(ENHANCED_ENABLE);
+ Request request = request(OpCode.multi, new MultiOperationRecord(Collections.singletonList(
+ Op.setData("/absent", new byte[2], -1))));
+ ProcessTxnResult result = apply(request, OpCode.multi, new MultiTxn(Collections.singletonList(
+ txn(OpCode.error, new ErrorTxn(Code.NONODE.intValue())))));
+ AuditHelper.addAuditLog(request, result);
+ Map fields = fields(capture.read(1).get(0));
+ assertEquals("multiOperation", fields.get("operation"));
+ assertEquals("failure", fields.get("result"));
+ assertNull(fields.get("schema_version"));
+ }
+
+ @Test
+ public void testReadsRemainUnaudited() {
+ for (int type : new int[] {OpCode.getData, OpCode.getChildren, OpCode.exists, OpCode.multiRead}) {
+ AuditHelper.addAuditLog(request(type, ByteBuffer.wrap(new byte[] {1})), new ProcessTxnResult());
+ }
+ capture.read(0);
+ }
+
+ @Test
+ public void testSystemDeletionHasStableIdentityAndSystemActor() throws Exception {
+ Request create = request(OpCode.create, createRecord("/ephemeral", new byte[0], CreateMode.EPHEMERAL));
+ assertEquals(0, apply(create, OpCode.create, createTxn("/ephemeral", new byte[0], true)).err);
+ tree.processTxn(new TxnHeader(SESSION, -11, 80, 1000, OpCode.closeSession),
+ new CloseSessionTxn(Collections.singletonList("/ephemeral")));
+ Map fields = fields(capture.read(1).get(0));
+ assertWrite(fields, "ephemeralZNodeDeletionOnSessionCloseOrExpire",
+ "/ephemeral", null, "committed", "0");
+ assertEquals(ZKAuditProvider.getZKUser(), fields.get("user"));
+ assertEquals("0x123", fields.get("session"));
+ assertEquals("80", fields.get("zxid"));
+ assertNull(fields.get("cxid"));
+ assertNull(fields.get("ip"));
+ assertNull(tree.getNode("/ephemeral"));
+ }
+
+ @Test
+ public void testLoggerFailureDoesNotInterruptSystemDeletion() throws Exception {
+ for (String path : Arrays.asList("/ephemeral-a", "/ephemeral-b")) {
+ Request request = request(OpCode.create, createRecord(path, new byte[0], CreateMode.EPHEMERAL));
+ assertEquals(0, apply(request, OpCode.create, createTxn(path, new byte[0], true)).err);
+ }
+ AuditLogger failingLogger = event -> {
+ throw new IllegalStateException("synthetic audit sink failure");
+ };
+ Object previous = replaceProviderField("auditLogger", failingLogger);
+ long before = auditErrors();
+ try {
+ try {
+ tree.processTxn(new TxnHeader(SESSION, -11, 81, 1000, OpCode.closeSession),
+ new CloseSessionTxn(Arrays.asList("/ephemeral-a", "/ephemeral-b")));
+ } catch (RuntimeException e) {
+ fail("Audit sink failure must not interrupt committed deletion: " + e);
+ }
+ assertNull(tree.getNode("/ephemeral-a"));
+ assertNull(tree.getNode("/ephemeral-b"));
+ assertEquals(before + 2, auditErrors());
+ } finally {
+ replaceProviderField("auditLogger", previous);
+ }
+ }
+
+ private Request request(int type, Record record) throws IOException {
+ return request(type, ByteBuffer.wrap(serialize(record)));
+ }
+
+ private Request request(int type, ByteBuffer buffer) {
+ return new Request(cnxn, SESSION, 41, type, buffer,
+ Collections.singletonList(new Id("ip", "127.0.0.1")));
+ }
+
+ private ProcessTxnResult apply(Request request, int type, Record txn) {
+ TxnHeader header = new TxnHeader(SESSION, 41, 66, 1000, type);
+ request.setHdr(header);
+ request.setTxn(txn);
+ return tree.processTxn(header, txn);
+ }
+
+ private static CreateRequest createRecord(String path, byte[] data, CreateMode mode) {
+ return new CreateRequest(path, data, ZooDefs.Ids.OPEN_ACL_UNSAFE, mode.toFlag());
+ }
+
+ private static CreateTxn createTxn(String path, byte[] data, boolean ephemeral) {
+ return new CreateTxn(path, data, ZooDefs.Ids.OPEN_ACL_UNSAFE, ephemeral, -1);
+ }
+
+ private static Txn txn(int type, Record record) throws IOException {
+ return new Txn(type, serialize(record));
+ }
+
+ private static byte[] serialize(Record record) throws IOException {
+ ByteArrayOutputStream out = new ByteArrayOutputStream();
+ record.serialize(BinaryOutputArchive.getArchive(out), "request");
+ return out.toByteArray();
+ }
+
+ static long auditErrors() {
+ return ((Number) MetricsUtils.currentServerMetrics().getOrDefault("audit_errors", 0L)).longValue();
+ }
+
+ static Object replaceProviderField(String name, Object value) throws ReflectiveOperationException {
+ Field field = ZKAuditProvider.class.getDeclaredField(name);
+ field.setAccessible(true);
+ Object previous = field.get(null);
+ field.set(null, value);
+ return previous;
+ }
+
+ static void restoreProperty(String name, String value) {
+ if (value == null) {
+ System.clearProperty(name);
+ } else {
+ System.setProperty(name, value);
+ }
+ }
+
+ static Map fields(String log) {
+ Map fields = new LinkedHashMap<>();
+ for (String pair : log.split("\t")) {
+ int separator = pair.indexOf('=');
+ assertTrue(log, separator > 0);
+ String previous = fields.put(pair.substring(0, separator), pair.substring(separator + 1));
+ assertNull("Duplicate audit field in " + log, previous);
+ }
+ return fields;
+ }
+
+ static void assertWrite(Map fields, String operation, String path,
+ String length, String outcome, String error) {
+ assertEquals("2", fields.get("schema_version"));
+ assertEquals(operation, fields.get("operation"));
+ assertEquals(path, fields.get("znode"));
+ assertEquals(length, fields.get("data_length"));
+ assertEquals(outcome, fields.get("outcome"));
+ assertEquals(error, fields.get("error_code"));
+ assertEquals("committed".equals(outcome) ? "success" : "unknown".equals(outcome) ? "invoked" : "failure",
+ fields.get("result"));
+ }
+
+ static final class AuditCapture extends AppenderBase implements AutoCloseable {
+ private final Logger logger = (Logger) LoggerFactory.getLogger(Slf4jAuditLogger.class);
+ private final Level previousLevel = logger.getLevel();
+ private final List messages = new ArrayList<>();
+ private boolean overflow;
+
+ AuditCapture() {
+ setContext(logger.getLoggerContext());
+ logger.setLevel(Level.INFO);
+ logger.addAppender(this);
+ start();
+ }
+
+ @Override
+ protected synchronized void append(ILoggingEvent event) {
+ if (messages.size() < 128) {
+ messages.add(event.getFormattedMessage());
+ } else {
+ overflow = true;
+ }
+ notifyAll();
+ }
+
+ synchronized List read(int expected) {
+ assertFalse("Audit capture exceeded its bounded capacity", overflow);
+ List result = new ArrayList<>(messages);
+ messages.clear();
+ assertEquals(result.toString(), expected, result.size());
+ return result;
+ }
+
+ synchronized void clear() {
+ assertFalse("Audit capture exceeded its bounded capacity", overflow);
+ messages.clear();
+ }
+
+ synchronized List await(int expected, long timeoutMillis) throws InterruptedException {
+ long deadline = System.nanoTime() + TimeUnit.MILLISECONDS.toNanos(timeoutMillis);
+ while (messages.size() < expected) {
+ long remaining = deadline - System.nanoTime();
+ if (remaining <= 0) {
+ break;
+ }
+ TimeUnit.NANOSECONDS.timedWait(this, remaining);
+ }
+ return read(expected);
+ }
+
+ @Override
+ public void close() {
+ logger.detachAppender(this);
+ logger.setLevel(previousLevel);
+ stop();
+ }
+ }
+}
diff --git a/zookeeper-server/src/test/java/org/apache/zookeeper/audit/Slf4JAuditLoggerTest.java b/zookeeper-server/src/test/java/org/apache/zookeeper/audit/Slf4JAuditLoggerTest.java
index f8d3c7b47af..0f426e073f7 100644
--- a/zookeeper-server/src/test/java/org/apache/zookeeper/audit/Slf4JAuditLoggerTest.java
+++ b/zookeeper-server/src/test/java/org/apache/zookeeper/audit/Slf4JAuditLoggerTest.java
@@ -19,14 +19,12 @@
import static org.apache.zookeeper.test.ClientBase.CONNECTION_TIMEOUT;
import static org.junit.Assert.assertEquals;
-import java.io.ByteArrayOutputStream;
import java.io.IOException;
-import java.io.LineNumberReader;
-import java.io.StringReader;
import java.net.InetAddress;
import java.net.InetSocketAddress;
import java.util.ArrayList;
import java.util.List;
+import java.util.Map;
import org.apache.zookeeper.CreateMode;
import org.apache.zookeeper.KeeperException;
import org.apache.zookeeper.KeeperException.Code;
@@ -36,6 +34,7 @@
import org.apache.zookeeper.ZooDefs;
import org.apache.zookeeper.ZooKeeper;
import org.apache.zookeeper.audit.AuditEvent.Result;
+import org.apache.zookeeper.audit.AuditHelperTest.AuditCapture;
import org.apache.zookeeper.data.ACL;
import org.apache.zookeeper.data.Stat;
import org.apache.zookeeper.server.Request;
@@ -43,7 +42,6 @@
import org.apache.zookeeper.server.quorum.QuorumPeerTestBase;
import org.apache.zookeeper.test.ClientBase;
import org.apache.zookeeper.test.ClientBase.CountdownWatcher;
-import org.apache.zookeeper.test.LoggerTestTool;
import org.junit.AfterClass;
import org.junit.Assert;
import org.junit.Before;
@@ -57,14 +55,14 @@ public class Slf4JAuditLoggerTest extends QuorumPeerTestBase {
private static int SERVER_COUNT = 3;
private static MainThread[] mt;
private static ZooKeeper zk;
- private static ByteArrayOutputStream os;
+ private static AuditCapture os;
@BeforeClass
public static void setUpBeforeClass() throws Exception {
System.setProperty(ZKAuditProvider.AUDIT_ENABLE, "true");
+ System.setProperty("zookeeper.extendedTypesEnabled", "true");
// setup the logger to capture all logs
- LoggerTestTool loggerTestTool = new LoggerTestTool(Slf4jAuditLogger.class);
- os = loggerTestTool.getOutputStream();
+ os = new AuditCapture();
mt = startQuorum();
zk = ClientBase.createZKClient("127.0.0.1:" + mt[0].getQuorumPeer().getClientPort());
//Verify start audit log here itself
@@ -75,7 +73,7 @@ public static void setUpBeforeClass() throws Exception {
@Before
public void setUp() {
- os.reset();
+ os.clear();
}
@Test
@@ -102,13 +100,47 @@ public void testCreateAuditLogs()
null, createMode), readAuditLog(os));
}
+ @Test
+ public void testCreateWithTtlAuditLogs() throws Exception {
+ String path = zk.create("/createTtlPath", new byte[0], ZooDefs.Ids.OPEN_ACL_UNSAFE,
+ CreateMode.PERSISTENT_WITH_TTL, null, 60000);
+ verifyLog(getAuditLog(AuditConstants.OP_CREATE, path, Result.SUCCESS,
+ null, "persistent_with_ttl"), readAuditLog(os));
+ }
+
+ @Test
+ public void testCreateSequentialWithTtlAuditLogs() throws Exception {
+ String path = zk.create("/createTtlSeqPath", new byte[0], ZooDefs.Ids.OPEN_ACL_UNSAFE,
+ CreateMode.PERSISTENT_SEQUENTIAL_WITH_TTL, null, 60000);
+ verifyLog(getAuditLog(AuditConstants.OP_CREATE, path, Result.SUCCESS,
+ null, "persistent_sequential_with_ttl"), readAuditLog(os));
+ }
+
+ @Test
+ public void testEnhancedEscapingThroughSlf4j() {
+ String previous = System.getProperty(AuditHelperTest.ENHANCED_ENABLE);
+ try (AuditHelperTest.AuditCapture capture = new AuditHelperTest.AuditCapture()) {
+ System.setProperty(AuditHelperTest.ENHANCED_ENABLE, "true");
+ ZKAuditProvider.log("team\tname\r\n\\t", "create", "/name=value\\child",
+ null, "persistent", "0x123", "127.0.0.1", Result.SUCCESS);
+ String log = capture.read(1).get(0);
+ assertEquals("2", AuditHelperTest.fields(log).get("schema_version"));
+ assertEquals("team\\tname\\r\\n\\\\t", AuditHelperTest.fields(log).get("user"));
+ assertEquals("/name=value\\\\child", AuditHelperTest.fields(log).get("znode"));
+ Assert.assertFalse(log.contains("\n"));
+ Assert.assertFalse(log.contains("\r"));
+ } finally {
+ AuditHelperTest.restoreProperty(AuditHelperTest.ENHANCED_ENABLE, previous);
+ }
+ }
+
@Test
public void testDeleteAuditLogs()
throws InterruptedException, IOException, KeeperException {
String path = "/deletePath";
zk.create(path, "".getBytes(), ZooDefs.Ids.OPEN_ACL_UNSAFE,
CreateMode.PERSISTENT);
- os.reset();
+ os.clear();
try {
zk.delete(path, -100);
} catch (KeeperException exception) {
@@ -129,7 +161,7 @@ public void testSetDataAuditLogs()
String path = "/setDataPath";
zk.create(path, "".getBytes(), ZooDefs.Ids.OPEN_ACL_UNSAFE,
CreateMode.PERSISTENT);
- os.reset();
+ os.clear();
try {
zk.setData(path, "newData".getBytes(), -100);
} catch (KeeperException exception) {
@@ -151,7 +183,7 @@ public void testSetACLAuditLogs()
String path = "/aclPath";
zk.create(path, "".getBytes(), ZooDefs.Ids.OPEN_ACL_UNSAFE,
CreateMode.PERSISTENT);
- os.reset();
+ os.clear();
try {
zk.setACL(path, openAclUnsafe, -100);
} catch (KeeperException exception) {
@@ -243,6 +275,32 @@ public void testEphemralZNodeAuditLogs()
ZKAuditProvider.getZKUser(), null), readAuditLog(os, SERVER_COUNT));
}
+ @Test
+ public void testEnhancedSystemDeletionIdentityAcrossReplicas() throws Exception {
+ String previous = System.getProperty(AuditHelperTest.ENHANCED_ENABLE);
+ System.setProperty(AuditHelperTest.ENHANCED_ENABLE, "true");
+ try (ZooKeeper client = ClientBase.createZKClient("127.0.0.1:" + mt[0].getQuorumPeer().getClientPort())) {
+ client.create("/enhanced-ephemeral", new byte[1], ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.EPHEMERAL);
+ String session = "0x" + Long.toHexString(client.getSessionId());
+ os.read(1);
+ client.close();
+ List logs = os.await(SERVER_COUNT, CONNECTION_TIMEOUT);
+ String zxid = Long.toString(zk.exists("/", false).getPzxid());
+ for (String log : logs) {
+ Map fields = AuditHelperTest.fields(log);
+ AuditHelperTest.assertWrite(fields, AuditConstants.OP_DEL_EZNODE_EXP,
+ "/enhanced-ephemeral", null, "committed", "0");
+ assertEquals(zxid, fields.get("zxid"));
+ assertEquals(session, fields.get("session"));
+ assertEquals(ZKAuditProvider.getZKUser(), fields.get("user"));
+ Assert.assertNull(fields.get("cxid"));
+ Assert.assertNull(fields.get("ip"));
+ }
+ } finally {
+ AuditHelperTest.restoreProperty(AuditHelperTest.ENHANCED_ENABLE, previous);
+ }
+ }
+
private static String getStartLog() {
// user=userName operation=ZooKeeperServer start result=success
@@ -309,10 +367,7 @@ private ServerCnxn getServerCnxn() {
}
private static void verifyLog(String expectedLog, String log) {
- String searchString = " - ";
- int logStartIndex = log.indexOf(searchString);
- String auditLog = log.substring(logStartIndex + searchString.length());
- Assert.assertTrue(auditLog.endsWith(expectedLog));
+ Assert.assertTrue(log, log.endsWith(expectedLog));
}
private static void verifyLogs(String expectedLog, List logs) {
@@ -321,37 +376,14 @@ private static void verifyLogs(String expectedLog, List logs) {
}
}
- private String readAuditLog(ByteArrayOutputStream os) throws IOException {
+ private String readAuditLog(AuditCapture os) throws IOException {
return readAuditLog(os, 1).get(0);
}
- private static List readAuditLog(ByteArrayOutputStream os,
+ private static List readAuditLog(AuditCapture os,
int numberOfLogEntry)
throws IOException {
- return readAuditLog(os, numberOfLogEntry, false);
- }
-
- private static List readAuditLog(ByteArrayOutputStream os,
- int numberOfLogEntry,
- boolean skipEphemralDeletion) throws IOException {
- List logs = new ArrayList<>();
- LineNumberReader r = new LineNumberReader(
- new StringReader(os.toString()));
- String line;
- while ((line = r.readLine()) != null) {
- if (skipEphemralDeletion
- && line.contains(AuditConstants.OP_DEL_EZNODE_EXP)) {
- continue;
- }
- logs.add(line);
- }
- os.reset();
- assertEquals(
- "Expected number of log entries are not generated. Logs are "
- + logs,
- numberOfLogEntry, logs.size());
- return logs;
-
+ return os.read(numberOfLogEntry);
}
private static MainThread[] startQuorum() throws IOException {
@@ -411,6 +443,7 @@ private void waitForDeletion(ZooKeeper zooKeeper, String path)
@AfterClass
public static void tearDownAfterClass() {
System.clearProperty(ZKAuditProvider.AUDIT_ENABLE);
+ System.clearProperty("zookeeper.extendedTypesEnabled");
for (int i = 0; i < SERVER_COUNT; i++) {
try {
if (mt[i] != null) {
@@ -419,11 +452,7 @@ public static void tearDownAfterClass() {
} catch (InterruptedException e) {
e.printStackTrace();
}
- try {
- os.close();
- } catch (IOException e) {
- e.printStackTrace();
- }
}
+ os.close();
}
}
diff --git a/zookeeper-server/src/test/java/org/apache/zookeeper/audit/StandaloneServerAuditTest.java b/zookeeper-server/src/test/java/org/apache/zookeeper/audit/StandaloneServerAuditTest.java
index cf8ca8eea36..702857018d0 100644
--- a/zookeeper-server/src/test/java/org/apache/zookeeper/audit/StandaloneServerAuditTest.java
+++ b/zookeeper-server/src/test/java/org/apache/zookeeper/audit/StandaloneServerAuditTest.java
@@ -19,34 +19,63 @@
package org.apache.zookeeper.audit;
+import static org.apache.zookeeper.audit.AuditHelperTest.assertWrite;
+import static org.apache.zookeeper.audit.AuditHelperTest.fields;
+import static org.junit.Assert.assertArrayEquals;
import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertNull;
import static org.junit.Assert.assertTrue;
-import java.io.ByteArrayOutputStream;
+import static org.junit.Assert.fail;
import java.io.IOException;
-import java.io.LineNumberReader;
-import java.io.StringReader;
-import java.util.ArrayList;
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import java.util.Collections;
import java.util.List;
+import java.util.Map;
import org.apache.zookeeper.CreateMode;
import org.apache.zookeeper.KeeperException;
+import org.apache.zookeeper.KeeperException.Code;
+import org.apache.zookeeper.Op;
+import org.apache.zookeeper.OpResult;
import org.apache.zookeeper.ZooDefs;
import org.apache.zookeeper.ZooKeeper;
+import org.apache.zookeeper.audit.AuditHelperTest.AuditCapture;
+import org.apache.zookeeper.data.ACL;
+import org.apache.zookeeper.data.Id;
+import org.apache.zookeeper.data.Stat;
+import org.apache.zookeeper.server.auth.DigestAuthenticationProvider;
import org.apache.zookeeper.test.ClientBase;
-import org.apache.zookeeper.test.LoggerTestTool;
+import org.junit.After;
import org.junit.AfterClass;
+import org.junit.Before;
import org.junit.BeforeClass;
import org.junit.Test;
public class StandaloneServerAuditTest extends ClientBase {
- private static ByteArrayOutputStream os;
+ private AuditCapture capture;
+ private String previousEnhanced;
+ private String previousExtendedTypes;
@BeforeClass
public static void setup() {
System.setProperty(ZKAuditProvider.AUDIT_ENABLE, "true");
- LoggerTestTool loggerTestTool = new LoggerTestTool(Slf4jAuditLogger.class);
- os = loggerTestTool.getOutputStream();
+ }
+
+ @Before
+ public void captureAuditLogs() {
+ previousEnhanced = System.getProperty(AuditHelperTest.ENHANCED_ENABLE);
+ previousExtendedTypes = System.getProperty("zookeeper.extendedTypesEnabled");
+ capture = new AuditCapture();
+ }
+
+ @After
+ public void restoreAuditSettings() {
+ capture.close();
+ AuditHelperTest.restoreProperty(AuditHelperTest.ENHANCED_ENABLE, previousEnhanced);
+ AuditHelperTest.restoreProperty("zookeeper.extendedTypesEnabled", previousExtendedTypes);
}
@AfterClass
@@ -60,21 +89,193 @@ public void testCreateAuditLog() throws KeeperException, InterruptedException, I
String path = "/createPath";
zk.create(path, "".getBytes(), ZooDefs.Ids.OPEN_ACL_UNSAFE,
CreateMode.PERSISTENT);
- List logs = readAuditLog(os);
+ List logs = capture.read(1);
assertEquals(1, logs.size());
assertTrue(logs.get(0).endsWith("operation=create\tznode=/createPath\tznode_type=persistent\tresult=success"));
}
- private static List readAuditLog(ByteArrayOutputStream os) throws IOException {
- List logs = new ArrayList<>();
- LineNumberReader r = new LineNumberReader(
- new StringReader(os.toString()));
- String line;
- while ((line = r.readLine()) != null) {
- logs.add(line);
+ @Test
+ public void testEnhancedWriteResultsMatchClientResponses() throws Exception {
+ System.setProperty(AuditHelperTest.ENHANCED_ENABLE, "true");
+ ZooKeeper zk = createClient();
+ byte[] data = "audit-secret".getBytes(StandardCharsets.UTF_8);
+ zk.create("/enhanced", data, ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT);
+ String log = capture.read(1).get(0);
+ Map created = fields(log);
+ assertWrite(created, "create", "/enhanced", "12", "committed", "0");
+ assertFalse(log.contains("audit-secret"));
+ assertEquals(Long.toString(zk.exists("/enhanced", false).getCzxid()), created.get("zxid"));
+
+ try {
+ zk.create("/enhanced", new byte[7], ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT);
+ fail("Duplicate create must fail");
+ } catch (KeeperException e) {
+ assertEquals(Code.NODEEXISTS, e.code());
+ }
+ assertWrite(fields(capture.read(1).get(0)), "create", "/enhanced", "7", "failed", "-110");
+
+ byte[] unicode = "\u00e9\ud83d\ude00".getBytes(StandardCharsets.UTF_8);
+ Stat changed = zk.setData("/enhanced", unicode, -1);
+ Map changedLog = fields(capture.read(1).get(0));
+ assertWrite(changedLog, "setData", "/enhanced", "6", "committed", "0");
+ assertEquals(Long.toString(changed.getMzxid()), changedLog.get("zxid"));
+ try {
+ zk.setData("/enhanced", new byte[1], -100);
+ fail("Invalid version must fail");
+ } catch (KeeperException e) {
+ assertEquals(Code.BADVERSION, e.code());
+ }
+ assertWrite(fields(capture.read(1).get(0)), "setData", "/enhanced", "1", "failed", "-103");
+ assertArrayEquals(unicode, zk.getData("/enhanced", false, null));
+ zk.getChildren("/", false);
+ zk.exists("/enhanced", false);
+ capture.read(0);
+ }
+
+ @Test
+ public void testEnhancedCreateVariants() throws Exception {
+ System.setProperty(AuditHelperTest.ENHANCED_ENABLE, "true");
+ System.setProperty("zookeeper.extendedTypesEnabled", "true");
+ ZooKeeper zk = createClient();
+ Stat stat = new Stat();
+ zk.create("/create2", null, ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT, stat);
+ assertWrite(fields(capture.read(1).get(0)), "create", "/create2", "0", "committed", "0");
+ zk.create("/container", new byte[2], ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.CONTAINER);
+ Map container = fields(capture.read(1).get(0));
+ assertWrite(container, "create", "/container", "2", "committed", "0");
+ assertEquals("container", container.get("znode_type"));
+ String path = zk.create("/ttl-", new byte[3], ZooDefs.Ids.OPEN_ACL_UNSAFE,
+ CreateMode.PERSISTENT_SEQUENTIAL_WITH_TTL, null, 60000);
+ Map ttl = fields(capture.read(1).get(0));
+ assertWrite(ttl, "create", path, "3", "committed", "0");
+ assertEquals("persistent_sequential_with_ttl", ttl.get("znode_type"));
+
+ zk.create("/ttl", new byte[0], ZooDefs.Ids.OPEN_ACL_UNSAFE,
+ CreateMode.PERSISTENT_WITH_TTL, null, 60000);
+ assertWrite(fields(capture.read(1).get(0)), "create", "/ttl", "0", "committed", "0");
+ try {
+ zk.create("/ttl", new byte[4], ZooDefs.Ids.OPEN_ACL_UNSAFE,
+ CreateMode.PERSISTENT_WITH_TTL, null, 60000);
+ fail("Duplicate TTL create must fail");
+ } catch (KeeperException e) {
+ assertEquals(Code.NODEEXISTS, e.code());
}
- os.reset();
- return logs;
+ Map failedTtl = fields(capture.read(1).get(0));
+ assertWrite(failedTtl, "create", "/ttl", "4", "failed", "-110");
+ assertEquals("persistent_with_ttl", failedTtl.get("znode_type"));
+ }
+
+ @Test
+ public void testEnhancedMultiUsesReturnedPathsAndCompleteIndexes() throws Exception {
+ System.setProperty(AuditHelperTest.ENHANCED_ENABLE, "true");
+ System.setProperty("zookeeper.extendedTypesEnabled", "true");
+ ZooKeeper zk = createClient();
+ List results = zk.multi(Arrays.asList(
+ Op.create("/same", new byte[1], ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT),
+ Op.delete("/same", -1),
+ Op.check("/", -1),
+ Op.create("/same", new byte[2], ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.EPHEMERAL),
+ Op.create("/seq-", new byte[3], ZooDefs.Ids.OPEN_ACL_UNSAFE,
+ CreateMode.PERSISTENT_SEQUENTIAL_WITH_TTL, 60000)));
+ assertEquals(5, results.size());
+ String finalPath = ((OpResult.CreateResult) results.get(4)).getPath();
+ assertFalse("/seq-".equals(finalPath));
+ List logs = capture.read(4);
+ assertWrite(fields(logs.get(0)), "create", "/same", "1", "committed", "0");
+ assertEquals("persistent", fields(logs.get(0)).get("znode_type"));
+ assertWrite(fields(logs.get(1)), "delete", "/same", null, "committed", "0");
+ assertWrite(fields(logs.get(2)), "create", "/same", "2", "committed", "0");
+ assertEquals("ephemeral", fields(logs.get(2)).get("znode_type"));
+ assertEquals("3", fields(logs.get(2)).get("multi_index"));
+ assertWrite(fields(logs.get(3)), "create", finalPath, "3", "committed", "0");
+ assertEquals("4", fields(logs.get(3)).get("multi_index"));
+ assertEquals("persistent_sequential_with_ttl", fields(logs.get(3)).get("znode_type"));
+ assertArrayEquals(new byte[3], zk.getData(finalPath, false, null));
+ }
+
+ @Test
+ public void testEnhancedFailedMultiMatchesAtomicRollback() throws Exception {
+ System.setProperty(AuditHelperTest.ENHANCED_ENABLE, "true");
+ ZooKeeper zk = createClient();
+ try {
+ zk.multi(Arrays.asList(
+ Op.create("/rolled", new byte[1], ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT),
+ Op.check("/", -1),
+ Op.setData("/missing", new byte[3], -1),
+ Op.create("/later", new byte[2], ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT)));
+ fail("Missing node must abort the multi");
+ } catch (KeeperException e) {
+ assertEquals(Code.NONODE, e.code());
+ }
+ assertNull(zk.exists("/rolled", false));
+ assertNull(zk.exists("/later", false));
+ List logs = capture.read(4);
+ assertWrite(fields(logs.get(0)), "multiOperation", null, null, "failed", "-101");
+ assertWrite(fields(logs.get(1)), "create", "/rolled", "1", "rolled_back", "0");
+ assertWrite(fields(logs.get(2)), "setData", "/missing", "3", "failed", "-101");
+ assertWrite(fields(logs.get(3)), "create", "/later", "2", "rolled_back", "-2");
+ assertEquals("0", fields(logs.get(1)).get("multi_index"));
+ assertEquals("2", fields(logs.get(2)).get("multi_index"));
+ assertEquals("3", fields(logs.get(3)).get("multi_index"));
+ }
+
+ @Test
+ public void testFailedCheckRollsBackMutationsWithoutACheckEvent() throws Exception {
+ System.setProperty(AuditHelperTest.ENHANCED_ENABLE, "true");
+ ZooKeeper zk = createClient();
+ try {
+ zk.multi(Arrays.asList(
+ Op.create("/rolled", new byte[1], ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT),
+ Op.check("/absent", -1),
+ Op.setData("/rolled", new byte[2], -1)));
+ fail("Missing check target must abort the multi");
+ } catch (KeeperException e) {
+ assertEquals(Code.NONODE, e.code());
+ }
+ assertNull(zk.exists("/rolled", false));
+ List logs = capture.read(3);
+ assertWrite(fields(logs.get(0)), "multiOperation", null, null, "failed", "-101");
+ assertWrite(fields(logs.get(1)), "create", "/rolled", "1", "rolled_back", "0");
+ assertWrite(fields(logs.get(2)), "setData", "/rolled", "2", "rolled_back", "-2");
+ assertEquals("2", fields(logs.get(2)).get("multi_index"));
+ }
+
+ @Test
+ public void testEnhancedAclRedactionDoesNotChangeStoredAcl() throws Exception {
+ System.setProperty(AuditHelperTest.ENHANCED_ENABLE, "true");
+ ZooKeeper zk = createClient();
+ String credentials = "alice:synthetic-password";
+ String digest = DigestAuthenticationProvider.generateDigest(credentials);
+ zk.addAuthInfo("digest", credentials.getBytes(StandardCharsets.UTF_8));
+ zk.create("/acl", new byte[0], ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT);
+ capture.read(1);
+ List acls = Collections.singletonList(new ACL(ZooDefs.Perms.ALL, new Id("digest", digest)));
+ zk.setACL("/acl", acls, -1);
+ String log = capture.read(1).get(0);
+ assertWrite(fields(log), "setAcl", "/acl", null, "committed", "0");
+ assertEquals("digest:alice:cdrwa", fields(log).get("acl"));
+ assertFalse(log.contains(digest));
+ assertFalse(log.contains("synthetic-password"));
+ assertEquals(acls, zk.getACL("/acl", new Stat()));
+ capture.read(0);
}
-}
+ @Test
+ public void testAuditFailureDoesNotRejectValidWrite() throws Exception {
+ System.setProperty(AuditHelperTest.ENHANCED_ENABLE, "true");
+ ZooKeeper zk = createClient();
+ AuditLogger failingLogger = event -> {
+ throw new IllegalStateException("synthetic audit sink failure");
+ };
+ Object previous = AuditHelperTest.replaceProviderField("auditLogger", failingLogger);
+ long before = AuditHelperTest.auditErrors();
+ try {
+ assertEquals("/valid", zk.create("/valid", new byte[2],
+ ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT));
+ assertArrayEquals(new byte[2], zk.getData("/valid", false, null));
+ assertEquals(before + 1, AuditHelperTest.auditErrors());
+ } finally {
+ AuditHelperTest.replaceProviderField("auditLogger", previous);
+ }
+ }
+}
From 1ec62e586c4ee41bb7810b8a36b419c1bc86c489 Mon Sep 17 00:00:00 2001
From: Arkesh Mishra <118651144+arkmish@users.noreply.github.com>
Date: Fri, 25 Sep 2026 16:55:17 +0530
Subject: [PATCH 4/9] Redact untrusted enhanced audit users (C1 I1)
Use validated concrete built-in authentication providers for enhanced user extraction, redact custom and unknown identity representations, and preserve legacy behavior. Cover the registered default-getUserName leak with real client authentication.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---
.../resources/markdown/zookeeperAuditLogs.md | 11 +++
.../apache/zookeeper/audit/AuditHelper.java | 59 +++++++++++++---
.../zookeeper/audit/AuditHelperTest.java | 69 +++++++++++++++++++
.../audit/StandaloneServerAuditTest.java | 34 +++++++++
4 files changed, 165 insertions(+), 8 deletions(-)
diff --git a/zookeeper-docs/src/main/resources/markdown/zookeeperAuditLogs.md b/zookeeper-docs/src/main/resources/markdown/zookeeperAuditLogs.md
index a58c8f19aa3..b9683a4fbb0 100644
--- a/zookeeper-docs/src/main/resources/markdown/zookeeperAuditLogs.md
+++ b/zookeeper-docs/src/main/resources/markdown/zookeeperAuditLogs.md
@@ -129,6 +129,17 @@ retained, valid built-in digest identities use the provider's username extractio
(not the digest), and other identities are `[redacted]`. This also applies to
failed ACL attempts, where identities might be malformed or contain credentials.
Unknown custom identity representations are not printed as ACL identities.
+
+For server-generated v2 records, the `user` field also uses a conservative
+provider policy rather than trusting a custom provider's default `getUserName`.
+Only the concrete built-in digest, IP, SASL and X509 providers are trusted, and
+their identity syntax must validate before their username extraction is used.
+Digest credentials are reduced to usernames; X509 identities are distinguished
+names, not certificate bodies. Unknown or malformed identities, custom providers
+and provider subclasses are represented as `[redacted]`. Merely overriding
+`getUserName` does not opt a custom provider into this trust policy. Legacy user
+extraction is unchanged when enhanced logging is off.
+
Custom callers of the logging APIs must supply sanitized user and ACL metadata;
the string-based APIs cannot infer credentials from arbitrary strings.
diff --git a/zookeeper-server/src/main/java/org/apache/zookeeper/audit/AuditHelper.java b/zookeeper-server/src/main/java/org/apache/zookeeper/audit/AuditHelper.java
index c2811048aba..855fd68f4e8 100644
--- a/zookeeper-server/src/main/java/org/apache/zookeeper/audit/AuditHelper.java
+++ b/zookeeper-server/src/main/java/org/apache/zookeeper/audit/AuditHelper.java
@@ -44,8 +44,10 @@
import org.apache.zookeeper.server.ServerMetrics;
import org.apache.zookeeper.server.auth.AuthenticationProvider;
import org.apache.zookeeper.server.auth.DigestAuthenticationProvider;
+import org.apache.zookeeper.server.auth.IPAuthenticationProvider;
import org.apache.zookeeper.server.auth.ProviderRegistry;
-import org.apache.zookeeper.server.util.AuthUtil;
+import org.apache.zookeeper.server.auth.SASLAuthenticationProvider;
+import org.apache.zookeeper.server.auth.X509AuthenticationProvider;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -54,6 +56,7 @@
*/
public final class AuditHelper {
private static final Logger LOG = LoggerFactory.getLogger(AuditHelper.class);
+ private static final String REDACTED = "[redacted]";
public static void addAuditLog(Request request, ProcessTxnResult rc) {
addAuditLog(request, rc, false);
@@ -95,7 +98,7 @@ public static void addAuditLog(Request request, ProcessTxnResult txnResult, bool
if (outcome != Outcome.COMMITTED || path == null) {
path = metadata.path == null ? path : metadata.path;
}
- log(request, txnResult, path, operation, metadata, result(outcome), error, outcome, null);
+ log(request, txnResult, path, operation, metadata, result(outcome), error, outcome, null, enhanced);
} catch (RuntimeException e) {
auditError(request.type, e);
}
@@ -159,7 +162,7 @@ private static void logMultiOperation(Request request, ProcessTxnResult rc, bool
// Emit the failed parent before decoding so malformed metadata cannot hide the failure.
if (failed) {
log(request, rc, rc == null ? null : rc.path, AuditConstants.OP_MULTI_OP,
- new RequestMetadata(), Result.FAILURE, error, Outcome.FAILED, null);
+ new RequestMetadata(), Result.FAILURE, error, Outcome.FAILED, null, enhanced);
if (!enhanced) {
return;
}
@@ -207,7 +210,7 @@ private static void logMultiOperation(Request request, ProcessTxnResult rc, bool
RequestMetadata metadata = metadata(op.getType(), op.toRequestRecord(), enhanced);
String path = outcome == Outcome.COMMITTED ? subResult.path : op.getPath();
log(request, rc, path, operation, metadata, failed ? Result.FAILURE : result(outcome),
- subResult == null ? null : subResult.err, outcome, index);
+ subResult == null ? null : subResult.err, outcome, index, enhanced);
}
index++;
}
@@ -256,14 +259,14 @@ private static String safeAclToString(List acls) {
StringBuilder value = new StringBuilder();
for (ACL acl : acls) {
Id id = acl.getId();
- String user = "[redacted]";
+ String user = REDACTED;
if ("world".equals(id.getScheme()) && "anyone".equals(id.getId())) {
user = "anyone";
} else if ("digest".equals(id.getScheme())) {
AuthenticationProvider provider = ProviderRegistry.getProvider(id.getScheme());
if (provider != null && provider.getClass() == DigestAuthenticationProvider.class
&& id.getId() != null && provider.isValid(id.getId())) {
- user = AuthUtil.getUser(id);
+ user = provider.getUserName(id.getId());
}
}
value.append(id.getScheme()).append(':').append(user).append(':')
@@ -272,6 +275,45 @@ private static String safeAclToString(List acls) {
return value.toString();
}
+ private static String getUsers(Request request, boolean enhanced) {
+ if (!enhanced) {
+ return request.getUsers();
+ }
+ if (request.authInfo == null) {
+ return null;
+ }
+ StringBuilder users = new StringBuilder();
+ boolean first = true;
+ for (Id id : request.authInfo) {
+ if (!first) {
+ users.append(',');
+ }
+ first = false;
+ users.append(safeUser(id));
+ }
+ return users.toString();
+ }
+
+ private static String safeUser(Id id) {
+ if (id == null || id.getScheme() == null || id.getId() == null) {
+ return REDACTED;
+ }
+ AuthenticationProvider provider = ProviderRegistry.getProvider(id.getScheme());
+ if (provider == null) {
+ return REDACTED;
+ }
+ Class providerClass = provider.getClass();
+ // Custom providers, including subclasses, do not establish safe identity representations.
+ if ((providerClass == DigestAuthenticationProvider.class
+ || providerClass == IPAuthenticationProvider.class
+ || providerClass == SASLAuthenticationProvider.class
+ || providerClass == X509AuthenticationProvider.class)
+ && provider.isValid(id.getId())) {
+ return provider.getUserName(id.getId());
+ }
+ return REDACTED;
+ }
+
private static String createMode(int type, int flags) {
try {
return CreateMode.fromFlag(flags).name().toLowerCase(Locale.ROOT);
@@ -330,7 +372,8 @@ private static boolean isCreate(int type) {
}
private static void log(Request request, ProcessTxnResult rc, String path, String operation,
- RequestMetadata metadata, Result result, Integer error, Outcome outcome, Integer index) {
+ RequestMetadata metadata, Result result, Integer error, Outcome outcome,
+ Integer index, boolean enhanced) {
Long zxid = null;
if (request.getHdr() != null) {
zxid = request.getHdr().getZxid();
@@ -339,7 +382,7 @@ private static void log(Request request, ProcessTxnResult rc, String path, Strin
} else if (request.zxid >= 0) {
zxid = request.zxid;
}
- ZKAuditProvider.log(request.getUsers(), operation, path, metadata.acl, metadata.createMode,
+ ZKAuditProvider.log(getUsers(request, enhanced), operation, path, metadata.acl, metadata.createMode,
request.cnxn.getSessionIdHex(), request.cnxn.getHostAddress(), result,
metadata.dataLength, error, outcome, request.cxid, zxid, index);
}
diff --git a/zookeeper-server/src/test/java/org/apache/zookeeper/audit/AuditHelperTest.java b/zookeeper-server/src/test/java/org/apache/zookeeper/audit/AuditHelperTest.java
index a85b1c9b9f7..7ddd49596f7 100644
--- a/zookeeper-server/src/test/java/org/apache/zookeeper/audit/AuditHelperTest.java
+++ b/zookeeper-server/src/test/java/org/apache/zookeeper/audit/AuditHelperTest.java
@@ -62,6 +62,8 @@
import org.apache.zookeeper.server.DataTree.ProcessTxnResult;
import org.apache.zookeeper.server.Request;
import org.apache.zookeeper.server.ServerCnxn;
+import org.apache.zookeeper.server.auth.AuthenticationProvider;
+import org.apache.zookeeper.server.auth.ProviderRegistry;
import org.apache.zookeeper.txn.CheckVersionTxn;
import org.apache.zookeeper.txn.CloseSessionTxn;
import org.apache.zookeeper.txn.CreateTTLTxn;
@@ -372,6 +374,45 @@ public void testMalformedDigestIdentityIsNotMistakenForAUser() throws Exception
assertWrite(fields(log), "setAcl", "/acl", null, "failed", "-114");
}
+ @Test
+ public void testEnhancedUsersRedactRegisteredDefaultProvider() throws Exception {
+ String property = ProviderRegistry.AUTHPROVIDER_PROPERTY_PREFIX + "c1-audit-user";
+ String previous = System.getProperty(property);
+ System.setProperty(property, CredentialAuthenticationProvider.class.getName());
+ ProviderRegistry.initialize();
+ try {
+ Request request = new Request(cnxn, SESSION, 41, OpCode.create,
+ ByteBuffer.wrap(serialize(createRecord("/custom-user", new byte[1], CreateMode.PERSISTENT))),
+ Arrays.asList(new Id("ip", "127.0.0.1"),
+ new Id("audit-test-custom", "alice:synthetic-password")));
+ ProcessTxnResult result = apply(request, OpCode.create, createTxn("/custom-user", new byte[1], false));
+ assertEquals(0, result.err);
+ AuditHelper.addAuditLog(request, result);
+ String log = capture.read(1).get(0);
+ assertWrite(fields(log), "create", "/custom-user", "1", "committed", "0");
+ assertEquals("127.0.0.1,[redacted]", fields(log).get("user"));
+ assertFalse(log.contains("synthetic-password"));
+ } finally {
+ ProviderRegistry.removeProvider("audit-test-custom");
+ restoreProperty(property, previous);
+ }
+ }
+
+ @Test
+ public void testEnhancedUsersRedactUnknownAndMalformedIdentities() throws Exception {
+ Request request = new Request(cnxn, SESSION, 41, OpCode.create,
+ ByteBuffer.wrap(serialize(createRecord("/unknown-users", new byte[1], CreateMode.PERSISTENT))),
+ Arrays.asList(new Id("unregistered", "synthetic-token"),
+ new Id("digest", "synthetic-digest-token"),
+ new Id("ip", "alice:synthetic-password")));
+ ProcessTxnResult result = apply(request, OpCode.create, createTxn("/unknown-users", new byte[1], false));
+ AuditHelper.addAuditLog(request, result);
+ String log = capture.read(1).get(0);
+ assertWrite(fields(log), "create", "/unknown-users", "1", "committed", "0");
+ assertEquals("[redacted],[redacted],[redacted]", fields(log).get("user"));
+ assertFalse(log.contains("synthetic"));
+ }
+
@Test
public void testAuditDisabledSkipsRequestsAndProvider() throws Exception {
Object previous = replaceProviderField("auditEnabled", false);
@@ -539,6 +580,34 @@ static void assertWrite(Map fields, String operation, String pat
fields.get("result"));
}
+ public static class CredentialAuthenticationProvider implements AuthenticationProvider {
+ @Override
+ public String getScheme() {
+ return "audit-test-custom";
+ }
+
+ @Override
+ public Code handleAuthentication(ServerCnxn connection, byte[] authData) {
+ connection.addAuthInfo(new Id(getScheme(), new String(authData, StandardCharsets.UTF_8)));
+ return Code.OK;
+ }
+
+ @Override
+ public boolean matches(String id, String aclExpr) {
+ return id.equals(aclExpr);
+ }
+
+ @Override
+ public boolean isAuthenticated() {
+ return true;
+ }
+
+ @Override
+ public boolean isValid(String id) {
+ return true;
+ }
+ }
+
static final class AuditCapture extends AppenderBase implements AutoCloseable {
private final Logger logger = (Logger) LoggerFactory.getLogger(Slf4jAuditLogger.class);
private final Level previousLevel = logger.getLevel();
diff --git a/zookeeper-server/src/test/java/org/apache/zookeeper/audit/StandaloneServerAuditTest.java b/zookeeper-server/src/test/java/org/apache/zookeeper/audit/StandaloneServerAuditTest.java
index 702857018d0..3a0180f31a2 100644
--- a/zookeeper-server/src/test/java/org/apache/zookeeper/audit/StandaloneServerAuditTest.java
+++ b/zookeeper-server/src/test/java/org/apache/zookeeper/audit/StandaloneServerAuditTest.java
@@ -41,10 +41,12 @@
import org.apache.zookeeper.ZooDefs;
import org.apache.zookeeper.ZooKeeper;
import org.apache.zookeeper.audit.AuditHelperTest.AuditCapture;
+import org.apache.zookeeper.audit.AuditHelperTest.CredentialAuthenticationProvider;
import org.apache.zookeeper.data.ACL;
import org.apache.zookeeper.data.Id;
import org.apache.zookeeper.data.Stat;
import org.apache.zookeeper.server.auth.DigestAuthenticationProvider;
+import org.apache.zookeeper.server.auth.ProviderRegistry;
import org.apache.zookeeper.test.ClientBase;
import org.junit.After;
import org.junit.AfterClass;
@@ -260,6 +262,38 @@ public void testEnhancedAclRedactionDoesNotChangeStoredAcl() throws Exception {
capture.read(0);
}
+ @Test
+ public void testRegisteredCustomUserIsRedactedOnlyInEnhancedMode() throws Exception {
+ String property = ProviderRegistry.AUTHPROVIDER_PROPERTY_PREFIX + "c1-audit-user";
+ String previous = System.getProperty(property);
+ System.setProperty(property, CredentialAuthenticationProvider.class.getName());
+ ProviderRegistry.initialize();
+ try {
+ System.setProperty(AuditHelperTest.ENHANCED_ENABLE, "true");
+ ZooKeeper zk = createClient();
+ zk.addAuthInfo("audit-test-custom", "alice:synthetic-password".getBytes(StandardCharsets.UTF_8));
+ zk.create("/custom-user", new byte[1], ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT);
+ String enhanced = capture.read(1).get(0);
+ assertWrite(fields(enhanced), "create", "/custom-user", "1", "committed", "0");
+ assertFalse(enhanced.contains("synthetic-password"));
+ List enhancedUsers = Arrays.asList(fields(enhanced).get("user").split(","));
+ Collections.sort(enhancedUsers);
+ assertEquals(Arrays.asList("127.0.0.1", "[redacted]"), enhancedUsers);
+
+ System.setProperty(AuditHelperTest.ENHANCED_ENABLE, "false");
+ zk.setData("/custom-user", new byte[2], -1);
+ Map legacy = fields(capture.read(1).get(0));
+ assertNull(legacy.get("schema_version"));
+ List legacyUsers = Arrays.asList(legacy.get("user").split(","));
+ Collections.sort(legacyUsers);
+ assertEquals(Arrays.asList("127.0.0.1", "alice:synthetic-password"), legacyUsers);
+ assertArrayEquals(new byte[2], zk.getData("/custom-user", false, null));
+ } finally {
+ ProviderRegistry.removeProvider("audit-test-custom");
+ AuditHelperTest.restoreProperty(property, previous);
+ }
+ }
+
@Test
public void testAuditFailureDoesNotRejectValidWrite() throws Exception {
System.setProperty(AuditHelperTest.ENHANCED_ENABLE, "true");
From cebdf2db40d671b87fa4a5794daa3cc8e325a7c2 Mon Sep 17 00:00:00 2001
From: Arkesh Mishra <118651144+arkmish@users.noreply.github.com>
Date: Fri, 25 Sep 2026 16:58:31 +0530
Subject: [PATCH 5/9] Isolate failures in audit error reporting (C1 I2)
Report audit failures through independently best-effort counter and diagnostic operations without recursive retries. Verify an applied write still receives success and multiple system deletions complete when both the audit sink and error counter fail.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---
.../resources/markdown/zookeeperAuditLogs.md | 4 +
.../apache/zookeeper/audit/AuditHelper.java | 4 +-
.../zookeeper/audit/ZKAuditProvider.java | 14 +++-
.../zookeeper/audit/AuditHelperTest.java | 78 +++++++++++++++++++
.../audit/StandaloneServerAuditTest.java | 33 ++++++++
5 files changed, 129 insertions(+), 4 deletions(-)
diff --git a/zookeeper-docs/src/main/resources/markdown/zookeeperAuditLogs.md b/zookeeper-docs/src/main/resources/markdown/zookeeperAuditLogs.md
index b9683a4fbb0..8c0bd0e4234 100644
--- a/zookeeper-docs/src/main/resources/markdown/zookeeperAuditLogs.md
+++ b/zookeeper-docs/src/main/resources/markdown/zookeeperAuditLogs.md
@@ -221,6 +221,10 @@ audit logger. Such failures are logged without rejecting an otherwise valid
client operation. A metadata error can still yield an audit record with omitted
fields, so this counter is **not** a dropped-record count. It has no per-path,
per-user or per-session labels.
+The counter update and diagnostic error logging are independently best-effort:
+runtime failures in either reporting backend are isolated without retries, so
+they cannot reject an applied write or interrupt system deletion. A failing
+metrics backend can therefore also leave detected audit errors uncounted.
Audit coverage is limited to the existing transaction-audit hooks. Rejections
before a hook, connection-level failures and later reply/send failures need
diff --git a/zookeeper-server/src/main/java/org/apache/zookeeper/audit/AuditHelper.java b/zookeeper-server/src/main/java/org/apache/zookeeper/audit/AuditHelper.java
index 855fd68f4e8..6a1f5e13bc2 100644
--- a/zookeeper-server/src/main/java/org/apache/zookeeper/audit/AuditHelper.java
+++ b/zookeeper-server/src/main/java/org/apache/zookeeper/audit/AuditHelper.java
@@ -41,7 +41,6 @@
import org.apache.zookeeper.server.ByteBufferInputStream;
import org.apache.zookeeper.server.DataTree.ProcessTxnResult;
import org.apache.zookeeper.server.Request;
-import org.apache.zookeeper.server.ServerMetrics;
import org.apache.zookeeper.server.auth.AuthenticationProvider;
import org.apache.zookeeper.server.auth.DigestAuthenticationProvider;
import org.apache.zookeeper.server.auth.IPAuthenticationProvider;
@@ -388,8 +387,7 @@ private static void log(Request request, ProcessTxnResult rc, String path, Strin
}
private static void auditError(int type, Exception e) {
- ServerMetrics.getMetrics().AUDIT_ERRORS.add(1);
- LOG.error("Failed to audit log request {}", type, e);
+ ZKAuditProvider.reportAuditError(LOG, "Failed to audit log request {}", type, e);
}
private static final class RequestMetadata {
diff --git a/zookeeper-server/src/main/java/org/apache/zookeeper/audit/ZKAuditProvider.java b/zookeeper-server/src/main/java/org/apache/zookeeper/audit/ZKAuditProvider.java
index 701ab71a622..d108ffa65e9 100644
--- a/zookeeper-server/src/main/java/org/apache/zookeeper/audit/ZKAuditProvider.java
+++ b/zookeeper-server/src/main/java/org/apache/zookeeper/audit/ZKAuditProvider.java
@@ -151,8 +151,20 @@ private static void logAuditEvent(AuditEvent event) {
try {
auditLogger.logAuditEvent(event);
} catch (RuntimeException e) {
+ reportAuditError(LOG, "Failed to write audit log for operation {}", event.getValue(FieldName.OPERATION), e);
+ }
+ }
+
+ static void reportAuditError(Logger logger, String message, Object context, Exception error) {
+ try {
ServerMetrics.getMetrics().AUDIT_ERRORS.add(1);
- LOG.error("Failed to write audit log for operation {}", event.getValue(FieldName.OPERATION), e);
+ } catch (RuntimeException ignored) {
+ // A failed metrics backend must not prevent the diagnostic or change the operation's result.
+ }
+ try {
+ logger.error(message, context, error);
+ } catch (RuntimeException ignored) {
+ // Reporting is best-effort; retrying through the same failing logger could escape or recurse.
}
}
diff --git a/zookeeper-server/src/test/java/org/apache/zookeeper/audit/AuditHelperTest.java b/zookeeper-server/src/test/java/org/apache/zookeeper/audit/AuditHelperTest.java
index 7ddd49596f7..2f3e4fdd139 100644
--- a/zookeeper-server/src/test/java/org/apache/zookeeper/audit/AuditHelperTest.java
+++ b/zookeeper-server/src/test/java/org/apache/zookeeper/audit/AuditHelperTest.java
@@ -41,6 +41,7 @@
import java.util.List;
import java.util.Map;
import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
import org.apache.jute.BinaryOutputArchive;
import org.apache.jute.Record;
import org.apache.zookeeper.CreateMode;
@@ -53,6 +54,7 @@
import org.apache.zookeeper.audit.AuditEvent.Result;
import org.apache.zookeeper.data.ACL;
import org.apache.zookeeper.data.Id;
+import org.apache.zookeeper.metrics.Counter;
import org.apache.zookeeper.metrics.MetricsUtils;
import org.apache.zookeeper.proto.CreateRequest;
import org.apache.zookeeper.proto.CreateTTLRequest;
@@ -62,6 +64,7 @@
import org.apache.zookeeper.server.DataTree.ProcessTxnResult;
import org.apache.zookeeper.server.Request;
import org.apache.zookeeper.server.ServerCnxn;
+import org.apache.zookeeper.server.ServerMetrics;
import org.apache.zookeeper.server.auth.AuthenticationProvider;
import org.apache.zookeeper.server.auth.ProviderRegistry;
import org.apache.zookeeper.txn.CheckVersionTxn;
@@ -503,6 +506,57 @@ public void testLoggerFailureDoesNotInterruptSystemDeletion() throws Exception {
}
}
+ @Test
+ public void testFailingErrorCounterDoesNotEscapeMetadataFailure() throws Exception {
+ Request request = request(OpCode.create, ByteBuffer.wrap(new byte[] {1}));
+ ProcessTxnResult result = apply(request, OpCode.create, createTxn("/reporter-metadata", new byte[2], false));
+ FailingCounter counter = new FailingCounter();
+ Counter previousCounter = replaceAuditErrorCounter(counter);
+ try {
+ RuntimeException escaped = null;
+ try {
+ AuditHelper.addAuditLog(request, result);
+ } catch (RuntimeException e) {
+ escaped = e;
+ }
+ assertNotNull(tree.getNode("/reporter-metadata"));
+ assertNull("The error reporter must not escape the audit boundary", escaped);
+ assertWrite(fields(capture.read(1).get(0)), "create", "/reporter-metadata", null, "committed", "0");
+ assertEquals("Do not retry a failing error reporter", 1, counter.get());
+ } finally {
+ replaceAuditErrorCounter(previousCounter);
+ }
+ }
+
+ @Test
+ public void testFailingErrorCounterDoesNotInterruptSystemDeletions() throws Exception {
+ for (String path : Arrays.asList("/reporter-a", "/reporter-b")) {
+ Request request = request(OpCode.create, createRecord(path, new byte[0], CreateMode.EPHEMERAL));
+ assertEquals(0, apply(request, OpCode.create, createTxn(path, new byte[0], true)).err);
+ }
+ FailingCounter counter = new FailingCounter();
+ Counter previousCounter = replaceAuditErrorCounter(counter);
+ Object previousLogger = replaceProviderField("auditLogger", (AuditLogger) event -> {
+ throw new IllegalStateException("synthetic audit sink failure");
+ });
+ try {
+ RuntimeException escaped = null;
+ try {
+ tree.processTxn(new TxnHeader(SESSION, -11, 81, 1000, OpCode.closeSession),
+ new CloseSessionTxn(Arrays.asList("/reporter-a", "/reporter-b")));
+ } catch (RuntimeException e) {
+ escaped = e;
+ }
+ assertNull("Reporting a sink failure must not escape system deletion", escaped);
+ assertNull(tree.getNode("/reporter-a"));
+ assertNull(tree.getNode("/reporter-b"));
+ assertEquals("One best-effort report per deletion, with no retries", 2, counter.get());
+ } finally {
+ replaceProviderField("auditLogger", previousLogger);
+ replaceAuditErrorCounter(previousCounter);
+ }
+ }
+
private Request request(int type, Record record) throws IOException {
return request(type, ByteBuffer.wrap(serialize(record)));
}
@@ -549,6 +603,15 @@ static Object replaceProviderField(String name, Object value) throws ReflectiveO
return previous;
}
+ static Counter replaceAuditErrorCounter(Counter counter) throws ReflectiveOperationException {
+ ServerMetrics metrics = ServerMetrics.getMetrics();
+ Field field = ServerMetrics.class.getField("AUDIT_ERRORS");
+ field.setAccessible(true);
+ Counter previous = (Counter) field.get(metrics);
+ field.set(metrics, counter);
+ return previous;
+ }
+
static void restoreProperty(String name, String value) {
if (value == null) {
System.clearProperty(name);
@@ -608,6 +671,21 @@ public boolean isValid(String id) {
}
}
+ static final class FailingCounter implements Counter {
+ private final AtomicInteger attempts = new AtomicInteger();
+
+ @Override
+ public void add(long delta) {
+ attempts.incrementAndGet();
+ throw new IllegalStateException("synthetic audit counter failure");
+ }
+
+ @Override
+ public long get() {
+ return attempts.get();
+ }
+ }
+
static final class AuditCapture extends AppenderBase implements AutoCloseable {
private final Logger logger = (Logger) LoggerFactory.getLogger(Slf4jAuditLogger.class);
private final Level previousLevel = logger.getLevel();
diff --git a/zookeeper-server/src/test/java/org/apache/zookeeper/audit/StandaloneServerAuditTest.java b/zookeeper-server/src/test/java/org/apache/zookeeper/audit/StandaloneServerAuditTest.java
index 3a0180f31a2..851cc1e8f0c 100644
--- a/zookeeper-server/src/test/java/org/apache/zookeeper/audit/StandaloneServerAuditTest.java
+++ b/zookeeper-server/src/test/java/org/apache/zookeeper/audit/StandaloneServerAuditTest.java
@@ -42,9 +42,11 @@
import org.apache.zookeeper.ZooKeeper;
import org.apache.zookeeper.audit.AuditHelperTest.AuditCapture;
import org.apache.zookeeper.audit.AuditHelperTest.CredentialAuthenticationProvider;
+import org.apache.zookeeper.audit.AuditHelperTest.FailingCounter;
import org.apache.zookeeper.data.ACL;
import org.apache.zookeeper.data.Id;
import org.apache.zookeeper.data.Stat;
+import org.apache.zookeeper.metrics.Counter;
import org.apache.zookeeper.server.auth.DigestAuthenticationProvider;
import org.apache.zookeeper.server.auth.ProviderRegistry;
import org.apache.zookeeper.test.ClientBase;
@@ -312,4 +314,35 @@ public void testAuditFailureDoesNotRejectValidWrite() throws Exception {
AuditHelperTest.replaceProviderField("auditLogger", previous);
}
}
+
+ @Test
+ public void testAuditReporterFailureDoesNotRejectAppliedWrite() throws Exception {
+ ZooKeeper zk = createClient();
+ FailingCounter counter = new FailingCounter();
+ Counter previousCounter = AuditHelperTest.replaceAuditErrorCounter(counter);
+ Object previousLogger = AuditHelperTest.replaceProviderField("auditLogger", (AuditLogger) event -> {
+ throw new IllegalStateException("synthetic audit sink failure");
+ });
+ try {
+ for (String enhanced : Arrays.asList("false", "true")) {
+ System.setProperty(AuditHelperTest.ENHANCED_ENABLE, enhanced);
+ String path = "/reporter-" + enhanced;
+ Code replyError = null;
+ String created = null;
+ long before = counter.get();
+ try {
+ created = zk.create(path, new byte[2], ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT);
+ } catch (KeeperException e) {
+ replyError = e.code();
+ }
+ assertArrayEquals(new byte[2], zk.getData(path, false, null));
+ assertNull("Audit reporting changed the reply after the write was applied", replyError);
+ assertEquals(path, created);
+ assertEquals("Do not retry a failing error reporter", before + 1, counter.get());
+ }
+ } finally {
+ AuditHelperTest.replaceProviderField("auditLogger", previousLogger);
+ AuditHelperTest.replaceAuditErrorCounter(previousCounter);
+ }
+ }
}
From 8434eafcea63b6df69f7ada39d83f217e799355b Mon Sep 17 00:00:00 2001
From: Arkesh Mishra <118651144+arkmish@users.noreply.github.com>
Date: Fri, 25 Sep 2026 17:02:08 +0530
Subject: [PATCH 6/9] Keep one audit mode through metadata and emission (C1 I3)
Carry the captured enhanced mode into event construction without rereading the property. Preserve public delegating overloads and keep every multi parent/member on the same mode. Cover deterministic ACL off-to-on and multi mode transitions with subsequent genuine v2 emission.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---
.../resources/markdown/zookeeperAuditLogs.md | 8 +-
.../apache/zookeeper/audit/AuditHelper.java | 2 +-
.../zookeeper/audit/ZKAuditProvider.java | 29 ++++-
.../zookeeper/audit/AuditHelperTest.java | 111 ++++++++++++++++++
4 files changed, 142 insertions(+), 8 deletions(-)
diff --git a/zookeeper-docs/src/main/resources/markdown/zookeeperAuditLogs.md b/zookeeper-docs/src/main/resources/markdown/zookeeperAuditLogs.md
index 8c0bd0e4234..4e03b13466d 100644
--- a/zookeeper-docs/src/main/resources/markdown/zookeeperAuditLogs.md
+++ b/zookeeper-docs/src/main/resources/markdown/zookeeperAuditLogs.md
@@ -210,8 +210,12 @@ Consumers should recognize v2 per record using `schema_version=2`, tolerate
unknown additive fields, and accept omitted optional fields. Do not apply v2
unescaping to legacy records. Prepare consumers for mixed legacy/v2 output before
enabling the property on servers, then roll it out gradually. Roll back emission
-by removing the enhanced property or setting it to `false` on restart. The
-enhanced gate is checked when emitting events; the base audit enablement retains
+by removing the enhanced property or setting it to `false` on restart. Each
+audited request captures the enhanced mode before metadata extraction, and uses
+that same decision for user/ACL sanitization, schema construction and every
+parent/member record of a multi. An in-flight request keeps its captured mode if
+the property changes; a subsequent request captures the new value. Direct
+provider logging captures its mode per event. The base audit enablement retains
its startup behavior. No wire, persistence, authentication, ACL, quota or payload
limit changes are required.
diff --git a/zookeeper-server/src/main/java/org/apache/zookeeper/audit/AuditHelper.java b/zookeeper-server/src/main/java/org/apache/zookeeper/audit/AuditHelper.java
index 6a1f5e13bc2..4731cbe58a3 100644
--- a/zookeeper-server/src/main/java/org/apache/zookeeper/audit/AuditHelper.java
+++ b/zookeeper-server/src/main/java/org/apache/zookeeper/audit/AuditHelper.java
@@ -383,7 +383,7 @@ private static void log(Request request, ProcessTxnResult rc, String path, Strin
}
ZKAuditProvider.log(getUsers(request, enhanced), operation, path, metadata.acl, metadata.createMode,
request.cnxn.getSessionIdHex(), request.cnxn.getHostAddress(), result,
- metadata.dataLength, error, outcome, request.cxid, zxid, index);
+ metadata.dataLength, error, outcome, request.cxid, zxid, index, enhanced);
}
private static void auditError(int type, Exception e) {
diff --git a/zookeeper-server/src/main/java/org/apache/zookeeper/audit/ZKAuditProvider.java b/zookeeper-server/src/main/java/org/apache/zookeeper/audit/ZKAuditProvider.java
index d108ffa65e9..f05062f9d1c 100644
--- a/zookeeper-server/src/main/java/org/apache/zookeeper/audit/ZKAuditProvider.java
+++ b/zookeeper-server/src/main/java/org/apache/zookeeper/audit/ZKAuditProvider.java
@@ -90,8 +90,19 @@ public static void log(String user, String operation, String znode, String acl,
if (!isAuditEnabled()) {
return;
}
+ log(user, operation, znode, acl, createMode, session, ip, result,
+ dataLength, errorCode, outcome, cxid, zxid, multiIndex, isEnhancedAuditEnabled());
+ }
+
+ static void log(String user, String operation, String znode, String acl,
+ String createMode, String session, String ip, Result result,
+ Integer dataLength, Integer errorCode, Outcome outcome,
+ Integer cxid, Long zxid, Integer multiIndex, boolean enhanced) {
+ if (!isAuditEnabled()) {
+ return;
+ }
logAuditEvent(createLogEvent(user, operation, znode, acl, createMode, session, ip, result,
- dataLength, errorCode, outcome, cxid, zxid, multiIndex));
+ dataLength, errorCode, outcome, cxid, zxid, multiIndex, enhanced));
}
/**
@@ -101,7 +112,7 @@ static AuditEvent createLogEvent(String user, String operation, Result result) {
AuditEvent event = new AuditEvent(result);
event.addEntry(FieldName.USER, user);
event.addEntry(FieldName.OPERATION, operation);
- addMetadata(event, null, null, null, null, null, null);
+ addMetadata(event, null, null, null, null, null, null, isEnhancedAuditEnabled());
return event;
}
@@ -118,6 +129,14 @@ static AuditEvent createLogEvent(String user, String operation, String znode, St
String createMode, String session, String ip, Result result,
Integer dataLength, Integer errorCode, Outcome outcome,
Integer cxid, Long zxid, Integer multiIndex) {
+ return createLogEvent(user, operation, znode, acl, createMode, session, ip, result,
+ dataLength, errorCode, outcome, cxid, zxid, multiIndex, isEnhancedAuditEnabled());
+ }
+
+ private static AuditEvent createLogEvent(String user, String operation, String znode, String acl,
+ String createMode, String session, String ip, Result result,
+ Integer dataLength, Integer errorCode, Outcome outcome,
+ Integer cxid, Long zxid, Integer multiIndex, boolean enhanced) {
AuditEvent event = new AuditEvent(result);
event.addEntry(FieldName.SESSION, session);
event.addEntry(FieldName.USER, user);
@@ -126,13 +145,13 @@ static AuditEvent createLogEvent(String user, String operation, String znode, St
event.addEntry(FieldName.ZNODE, znode);
event.addEntry(FieldName.ZNODE_TYPE, createMode);
event.addEntry(FieldName.ACL, acl);
- addMetadata(event, dataLength, errorCode, outcome, cxid, zxid, multiIndex);
+ addMetadata(event, dataLength, errorCode, outcome, cxid, zxid, multiIndex, enhanced);
return event;
}
private static void addMetadata(AuditEvent event, Integer dataLength, Integer errorCode, Outcome outcome,
- Integer cxid, Long zxid, Integer multiIndex) {
- if (isEnhancedAuditEnabled()) {
+ Integer cxid, Long zxid, Integer multiIndex, boolean enhanced) {
+ if (enhanced) {
event.addEntry(FieldName.SCHEMA_VERSION, AuditConstants.SCHEMA_VERSION);
event.addEntry(FieldName.DATA_LENGTH, valueOf(dataLength));
event.addEntry(FieldName.ERROR_CODE, valueOf(errorCode));
diff --git a/zookeeper-server/src/test/java/org/apache/zookeeper/audit/AuditHelperTest.java b/zookeeper-server/src/test/java/org/apache/zookeeper/audit/AuditHelperTest.java
index 2f3e4fdd139..2dc2ab9f47a 100644
--- a/zookeeper-server/src/test/java/org/apache/zookeeper/audit/AuditHelperTest.java
+++ b/zookeeper-server/src/test/java/org/apache/zookeeper/audit/AuditHelperTest.java
@@ -66,6 +66,7 @@
import org.apache.zookeeper.server.ServerCnxn;
import org.apache.zookeeper.server.ServerMetrics;
import org.apache.zookeeper.server.auth.AuthenticationProvider;
+import org.apache.zookeeper.server.auth.DigestAuthenticationProvider;
import org.apache.zookeeper.server.auth.ProviderRegistry;
import org.apache.zookeeper.txn.CheckVersionTxn;
import org.apache.zookeeper.txn.CloseSessionTxn;
@@ -74,6 +75,7 @@
import org.apache.zookeeper.txn.DeleteTxn;
import org.apache.zookeeper.txn.ErrorTxn;
import org.apache.zookeeper.txn.MultiTxn;
+import org.apache.zookeeper.txn.SetACLTxn;
import org.apache.zookeeper.txn.SetDataTxn;
import org.apache.zookeeper.txn.Txn;
import org.apache.zookeeper.txn.TxnHeader;
@@ -416,6 +418,115 @@ public void testEnhancedUsersRedactUnknownAndMalformedIdentities() throws Except
assertFalse(log.contains("synthetic"));
}
+ @Test
+ public void testAclModeSnapshotSurvivesOffToOnInterleaving() throws Exception {
+ Request create = request(OpCode.create, createRecord("/mode-acl", new byte[0], CreateMode.PERSISTENT));
+ assertEquals(0, apply(create, OpCode.create, createTxn("/mode-acl", new byte[0], false)).err);
+ String digest = DigestAuthenticationProvider.generateDigest("alice:synthetic-password");
+ List acls = Collections.singletonList(new ACL(ZooDefs.Perms.ALL, new Id("digest", digest)));
+ SetACLRequest record = new SetACLRequest("/mode-acl", acls, -1);
+ AtomicInteger transitions = new AtomicInteger();
+ Request switching = new Request(cnxn, SESSION, 41, OpCode.setACL, ByteBuffer.wrap(serialize(record)),
+ Collections.singletonList(new Id("ip", "127.0.0.1"))) {
+ @Override
+ public String getUsers() {
+ System.setProperty(ENHANCED_ENABLE, "true");
+ transitions.incrementAndGet();
+ return super.getUsers();
+ }
+ };
+ ProcessTxnResult changed = apply(switching, OpCode.setACL, new SetACLTxn("/mode-acl", acls, 1));
+ assertEquals(0, changed.err);
+ System.setProperty(ENHANCED_ENABLE, "false");
+ AuditHelper.addAuditLog(switching, changed);
+ Map legacy = fields(capture.read(1).get(0));
+ assertEquals(1, transitions.get());
+ assertEquals("true", System.getProperty(ENHANCED_ENABLE));
+ assertNull("An in-flight legacy record must not be relabeled v2", legacy.get("schema_version"));
+ assertEquals("digest:" + digest + ":cdrwa", legacy.get("acl"));
+ assertEquals("success", legacy.get("result"));
+
+ Request enhanced = request(OpCode.setACL, record);
+ ProcessTxnResult updated = apply(enhanced, OpCode.setACL, new SetACLTxn("/mode-acl", acls, 2));
+ assertEquals(0, updated.err);
+ AuditHelper.addAuditLog(enhanced, updated);
+ String log = capture.read(1).get(0);
+ assertWrite(fields(log), "setAcl", "/mode-acl", null, "committed", "0");
+ assertEquals("digest:alice:cdrwa", fields(log).get("acl"));
+ assertFalse(log.contains(digest));
+ assertFalse(log.contains("synthetic-password"));
+ }
+
+ @Test
+ public void testSuccessfulMultiKeepsOneModeAcrossMembers() throws Exception {
+ Request request = request(OpCode.multi, new MultiOperationRecord(Arrays.asList(
+ Op.create("/mode-multi", new byte[1], ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT),
+ Op.setData("/mode-multi", new byte[2], -1))));
+ ProcessTxnResult result = apply(request, OpCode.multi, new MultiTxn(Arrays.asList(
+ txn(OpCode.create, createTxn("/mode-multi", new byte[1], false)),
+ txn(OpCode.setData, new SetDataTxn("/mode-multi", new byte[2], 1)))));
+ assertEquals(0, result.err);
+ System.setProperty(ENHANCED_ENABLE, "false");
+ AtomicInteger events = new AtomicInteger();
+ AuditLogger delegate = new Slf4jAuditLogger();
+ Object previousLogger = replaceProviderField("auditLogger", (AuditLogger) event -> {
+ delegate.logAuditEvent(event);
+ if (events.incrementAndGet() == 1) {
+ System.setProperty(ENHANCED_ENABLE, "true");
+ }
+ });
+ try {
+ AuditHelper.addAuditLog(request, result);
+ List logs = capture.read(2);
+ assertEquals(2, events.get());
+ for (String log : logs) {
+ assertNull("A multi must keep its captured legacy mode", fields(log).get("schema_version"));
+ assertEquals("success", fields(log).get("result"));
+ }
+
+ Request following = request(OpCode.setData, new SetDataRequest("/mode-multi", new byte[3], -1));
+ ProcessTxnResult updated = apply(following, OpCode.setData, new SetDataTxn("/mode-multi", new byte[3], 2));
+ assertEquals(0, updated.err);
+ AuditHelper.addAuditLog(following, updated);
+ assertWrite(fields(capture.read(1).get(0)), "setData", "/mode-multi", "3", "committed", "0");
+ } finally {
+ replaceProviderField("auditLogger", previousLogger);
+ }
+ }
+
+ @Test
+ public void testFailedMultiKeepsParentModeForRolledBackMembers() throws Exception {
+ Request request = request(OpCode.multi, new MultiOperationRecord(Arrays.asList(
+ Op.create("/mode-rolled", new byte[1], ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT),
+ Op.check("/missing", -1),
+ Op.setData("/mode-rolled", new byte[2], -1))));
+ ProcessTxnResult result = apply(request, OpCode.multi, new MultiTxn(Arrays.asList(
+ txn(OpCode.create, createTxn("/mode-rolled", new byte[1], false)),
+ txn(OpCode.error, new ErrorTxn(Code.NONODE.intValue())),
+ txn(OpCode.error, new ErrorTxn(Code.RUNTIMEINCONSISTENCY.intValue())))));
+ assertEquals(-101, result.err);
+ assertNull(tree.getNode("/mode-rolled"));
+ AtomicInteger events = new AtomicInteger();
+ AuditLogger delegate = new Slf4jAuditLogger();
+ Object previousLogger = replaceProviderField("auditLogger", (AuditLogger) event -> {
+ delegate.logAuditEvent(event);
+ if (events.incrementAndGet() == 1) {
+ System.setProperty(ENHANCED_ENABLE, "false");
+ }
+ });
+ try {
+ AuditHelper.addAuditLog(request, result);
+ List logs = capture.read(3);
+ assertEquals(3, events.get());
+ assertWrite(fields(logs.get(0)), "multiOperation", null, null, "failed", "-101");
+ assertWrite(fields(logs.get(1)), "create", "/mode-rolled", "1", "rolled_back", "0");
+ assertWrite(fields(logs.get(2)), "setData", "/mode-rolled", "2", "rolled_back", "-2");
+ assertEquals("2", fields(logs.get(2)).get("multi_index"));
+ } finally {
+ replaceProviderField("auditLogger", previousLogger);
+ }
+ }
+
@Test
public void testAuditDisabledSkipsRequestsAndProvider() throws Exception {
Object previous = replaceProviderField("auditEnabled", false);
From ba846669d544313e2b800713e79204dc9d7eb075 Mon Sep 17 00:00:00 2001
From: Arkesh Mishra <118651144+arkmish@users.noreply.github.com>
Date: Fri, 25 Sep 2026 18:01:46 +0530
Subject: [PATCH 7/9] Add enhanced session and authentication audit bindings
(C2)
Emit sanitized scalar auth_scheme/user bindings for valid attachments and actual authentication outcomes. Preserve C1 feature gates, mode snapshots and best-effort reporting; account explicitly for setup bindings in write-focused tests.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---
.../resources/markdown/zookeeperAuditLogs.md | 125 ++-
.../zookeeper/audit/AuditConstants.java | 2 +
.../apache/zookeeper/audit/AuditEvent.java | 2 +-
.../apache/zookeeper/audit/AuditHelper.java | 63 ++
.../zookeeper/audit/ZKAuditProvider.java | 13 +
.../zookeeper/server/ZooKeeperServer.java | 6 +
.../zookeeper/audit/SessionAuthAuditTest.java | 953 ++++++++++++++++++
.../zookeeper/audit/Slf4JAuditLoggerTest.java | 3 +
.../audit/StandaloneServerAuditTest.java | 25 +-
9 files changed, 1177 insertions(+), 15 deletions(-)
create mode 100644 zookeeper-server/src/test/java/org/apache/zookeeper/audit/SessionAuthAuditTest.java
diff --git a/zookeeper-docs/src/main/resources/markdown/zookeeperAuditLogs.md b/zookeeper-docs/src/main/resources/markdown/zookeeperAuditLogs.md
index 4e03b13466d..25f43951054 100644
--- a/zookeeper-docs/src/main/resources/markdown/zookeeperAuditLogs.md
+++ b/zookeeper-docs/src/main/resources/markdown/zookeeperAuditLogs.md
@@ -19,6 +19,7 @@ limitations under the License.
* [ZooKeeper Audit Logs](#ch_auditLogs)
* [ZooKeeper Audit Log Configuration](#ch_auditConfig)
* [Enhanced audit metadata (schema v2)](#ch_auditV2)
+* [Session and authentication bindings](#ch_auditBindings)
* [Who is taken as user in audit logs?](#ch_zkAuditUser)
@@ -35,9 +36,9 @@ The audit log captures detailed information for the operations that are selected
| Key | Value |
| ----- | ----- |
|session | client session id |
-|user | comma separated list of users who are associate with a client session. For more on this, see [Who is taken as user in audit logs](#ch_zkAuditUser).
+|user | comma separated list of users associated with a write request, or one sanitized principal on each v2 [binding record](#ch_auditBindings). For more on this, see [Who is taken as user in audit logs](#ch_zkAuditUser).
|ip | client IP address
-|operation | any one of the selected operations for audit. Possible values are(serverStart, serverStop, create, delete, setData, setAcl, multiOperation, reconfig, ephemeralZNodeDeletionOnSessionCloseOrExpire)
+|operation | any one of the selected operations for audit. Possible values are(serverStart, serverStop, create, delete, setData, setAcl, multiOperation, reconfig, ephemeralZNodeDeletionOnSessionCloseOrExpire). Enhanced logging also adds sessionEstablished and authentication.
|znode | path of the znode
|znode_type | type of znode in case of creation operation
|acl | String representation of znode ACL like cdrwa(create, delete,read, write, admin). This is logged only for setAcl operation
@@ -122,6 +123,7 @@ V2 keeps the existing field names and `result=success/failure/invoked`, and adds
| `cxid` | Known client request ID in decimal. All members of a multi share this ID. |
| `zxid` | Known transaction ID in decimal, not the server's latest zxid at reply time. A failed transaction may also have a zxid. |
| `multi_index` | Zero-based position in the **complete** multi request, including non-mutating checks. Omitted on single operations and multi parent records. |
+| `auth_scheme` | Authentication scheme paired with the single sanitized `user` on a session/authentication binding record. Omitted when unknown and on write records. |
No znode payloads are included. For enhanced `setAcl` records, `acl` retains
schemes and permissions, but identities are conservative: `world:anyone` is
@@ -156,11 +158,13 @@ the string-based APIs cannot infer credentials from arbitrary strings.
skipped after the failure. They have `result=failure`, **even when their
individual `error_code` is `0`**. Later skipped members normally have code
`-2` (RUNTIMEINCONSISTENCY); a first failure with that code is still `failed`.
-* `unknown` means the available metadata cannot establish an operation's
- outcome. Missing or mismatched multi results are not treated as commits,
+* For transaction records, `unknown` means the available metadata cannot
+ establish the transaction's outcome. Missing or mismatched multi results are not treated as commits,
and untrustworthy per-member error codes are omitted. If the whole multi
is known to have failed, these members still have `result=failure`;
- otherwise unknown operations use `result=invoked`.
+ otherwise unknown transactions use `result=invoked`. Non-transactional
+ lifecycle and successful binding events also use `outcome=unknown`; their
+ separate `result` retains the known lifecycle or authentication result.
Failed multis keep their `multiOperation` parent failure record, followed in v2
by records for the attempted mutations when the request is decodable. The parent
@@ -172,7 +176,7 @@ failed or rolled-back creates use the attempted path.
### Supported operations and identity
-The audited operations remain creates (`create`, `create2`, container and TTL
+The audited write operations remain creates (`create`, `create2`, container and TTL
variants), deletes (including container deletes), `setData`, `setAcl`, write
multis, reconfiguration, server start/stop, and existing system ephemeral-node
deletions. Ordinary reads, read multis and checks do not produce audit records.
@@ -180,8 +184,8 @@ Checks inside a write multi still occupy an index.
Existing lifecycle events and callers of the legacy logging overloads receive
`outcome=unknown` in v2 when they do not supply transaction metadata; their
-existing `result` is unchanged. No session/authentication binding events are
-added by this schema.
+existing `result` is unchanged. Enhanced session/authentication events are
+described below; they do not claim a committed transaction.
System ephemeral-node deletion records preserve the **server actor**, the
affected session and path, and omit client IP. V2 adds the deletion transaction's
@@ -191,6 +195,102 @@ replicas or during replay. Downstream consumers can deduplicate using ensemble
identity plus `zxid`, `operation`, `session` and `znode`; client multi mutations
also require `multi_index` to distinguish repeated paths.
+
+
+### Session and authentication bindings
+
+Both existing audit gates must be enabled for these additional operations.
+With audit logging disabled or enhanced logging disabled, neither operation is
+emitted. Authentication, SASL failure policy, TLS, ACL evaluation and client
+responses are unchanged.
+
+* `operation=sessionEstablished`, `result=success` records a **valid session
+ attachment**, including reconnects and movement to another server. It does
+ not necessarily mean a new session was created. The hook runs after the
+ connect response has been queued and before receiving further requests is
+ enabled. Invalid session/password revalidation does not produce these records.
+ Each record binds one identity already attached to that connection.
+* `operation=authentication` records an explicit auth provider's accepted or
+ rejected outcome, or SASL completion/failure. Intermediate SASL challenges
+ do not produce successes. Successful binding records snapshot the accepted
+ **scheme's currently attached identities**, including earlier identities of
+ the same scheme; they are not a delta or proof that each principal was
+ newly authenticated by this packet. A failure record does not extract a
+ principal from credentials or blame an identity already on the connection.
+
+These operations emit one record per distinct sanitized `(auth_scheme, user)`
+binding. `auth_scheme` and the scalar `user` belong to the **same record**;
+neither field is a parallel comma-separated list. For example, one attachment
+with IP, digest and X509 identities produces three records:
+
+```text
+session=0x123 user=127.0.0.1 ip=127.0.0.1 operation=sessionEstablished schema_version=2 outcome=unknown auth_scheme=ip result=success
+session=0x123 user=alice ip=127.0.0.1 operation=sessionEstablished schema_version=2 outcome=unknown auth_scheme=digest result=success
+session=0x123 user=CN=Bob,OU=Audit ip=127.0.0.1 operation=sessionEstablished schema_version=2 outcome=unknown auth_scheme=x509 result=success
+```
+
+Binding order is unspecified. Duplicate sanitized pairs within a single hook
+are collapsed, including distinct credentials for the same username or
+multiple untrusted identities that redact to the same pair. A later attachment
+or auth outcome can emit that pair again. Record counts are therefore **not**
+session-creation or authentication-attempt counts.
+
+Use ordinary v2 field unescaping only; there is no JSON grammar inside `user`.
+Commas, quotes and equals signs are part of a scalar principal, not binding
+separators. Write-record `user` formatting is unchanged.
+
+An attachment without identities emits one record with `user` and `auth_scheme`
+omitted. Ordinary anonymous network clients normally still have an `ip`
+identity; this is not proof of credential-based authentication. Unknown fields
+are omitted, not encoded as a literal `null` username. Failed digest
+authentication has `auth_scheme=digest` and no `user`. A provider that accepts
+authentication without attaching a matching identity also has no `user`.
+A missing scheme is omitted. An existing but untrusted or
+malformed identity is `[redacted]`, using the same conservative concrete-provider
+policy as other server-generated v2 records. Custom providers and subclasses
+do not opt into trusted extraction through `getUserName`. No auth packet bytes,
+SASL tokens, passwords, digest secrets or certificate bodies are logged.
+
+| Field | Binding-event meaning |
+| --- | --- |
+| `session` | Known connection session ID in hexadecimal. Omitted when the ID is not assigned (zero), including pre-session authentication failures. No replacement session ID is invented. |
+| `auth_scheme` | Scheme of the identity or the authentication outcome. Omitted when unknown. |
+| `user` | One sanitized principal bound to `auth_scheme`. Omitted for failed authentication or when no matching identity is known. |
+| `ip` | Client address available on the connection. |
+| `result` | `success` for valid attachments or accepted authentication; `failure` for rejected authentication, even if SASL policy allows the connection to remain open. |
+| `error_code` | Authentication outcome: `0` for success, `-115` (AUTHFAILED) for failure. Omitted on attachment events. This is not the SASL policy's eventual reply code: allowed failed clients can receive OK, while required-SASL failures can receive SESSIONCLOSEDREQUIRESASLAUTH. |
+| `outcome` | `unknown` for a successful, non-transactional binding; `failed` for authentication failure. Never `committed`. |
+| `cxid` | Actual auth/SASL request ID, when a packet supplied one. These protocol IDs can repeat and are not unique event IDs. Omitted for session attachment. |
+
+There is no binding `zxid`, znode, payload size or multi index. The
+`sessionEstablished` hook is shared by standalone and quorum servers, including
+learner revalidation. An attachment's records describe one snapshot on the
+attached server, not a new binding on every replica. All identities are
+sanitized before the first record is emitted, and the entire group uses one
+captured enhanced mode. A later authentication outcome is a separate stage,
+not a duplicate attachment.
+
+TLS authentication can populate an X509 identity **before** session assignment.
+The later attachment snapshot correlates that identity with the valid session;
+it does not invent an earlier session ID or a second X509 authentication
+outcome. Neither an `x509` nor a `super` identity establishes the transport
+security state. A `super` identity remains conservatively redacted. DN-based
+X509 identities use the existing built-in validation; SAN-derived identities
+that do not validate as DNs and custom TLS identities remain redacted.
+
+Request-time write identities remain authoritative for those writes. Later
+authentication does not rewrite earlier audit history, and reconnecting does
+not transfer another connection's authentication information. Consumers must
+not treat an attachment snapshot as the identity for every subsequent write.
+
+These are best-effort server observations, not evidence of client receipt or
+durable log delivery. TLS handshake failures before session initialization,
+malformed packets that never reach an auth outcome, and a missing SASL server
+are not newly instrumented here. Metadata failures can omit a binding record;
+the existing best-effort `audit_errors` reporter is reused without changing
+authentication or session behavior, even if the counter or diagnostic logger
+also fails.
+
### Escaping and parsing
Fields are separated by literal tabs. Split each field at its **first** equals
@@ -213,7 +313,8 @@ enabling the property on servers, then roll it out gradually. Roll back emission
by removing the enhanced property or setting it to `false` on restart. Each
audited request captures the enhanced mode before metadata extraction, and uses
that same decision for user/ACL sanitization, schema construction and every
-parent/member record of a multi. An in-flight request keeps its captured mode if
+parent/member record of a multi. A connection-binding hook likewise captures
+one mode before identity extraction and carries it through emission. An in-flight request keeps its captured mode if
the property changes; a subsequent request captures the new value. Direct
provider logging captures its mode per event. The base audit enablement retains
its startup behavior. No wire, persistence, authentication, ACL, quota or payload
@@ -230,8 +331,8 @@ runtime failures in either reporting backend are isolated without retries, so
they cannot reject an applied write or interrupt system deletion. A failing
metrics backend can therefore also leave detected audit errors uncounted.
-Audit coverage is limited to the existing transaction-audit hooks. Rejections
-before a hook, connection-level failures and later reply/send failures need
+Audit coverage is limited to the selected transaction and binding hooks. Rejections
+before a hook, other connection-level failures and later reply/send failures need
separate instrumentation; these records are not a complete request-delivery
ledger. Log delivery is also a separate concern: an asynchronous appender using
`neverBlock=true` can discard records without reporting an exception.
@@ -254,7 +355,7 @@ User is decided based on the configured authentication provider:
* When IPAuthenticationProvider is configured then authenticated IP is taken as user
* When SASLAuthenticationProvider is configured then client principal is taken as user
-* When X509AuthenticationProvider is configured then client certificate is taken as user
+* When X509AuthenticationProvider is configured then its certificate-derived identity (normally the subject DN) is taken as user, not the certificate body
* When DigestAuthenticationProvider is configured then authenticated user is user
Custom authentication provider can override org.apache.zookeeper.server.auth.AuthenticationProvider.getUserName(String id)
diff --git a/zookeeper-server/src/main/java/org/apache/zookeeper/audit/AuditConstants.java b/zookeeper-server/src/main/java/org/apache/zookeeper/audit/AuditConstants.java
index 1bfe96b363b..2d5e3eef79f 100644
--- a/zookeeper-server/src/main/java/org/apache/zookeeper/audit/AuditConstants.java
+++ b/zookeeper-server/src/main/java/org/apache/zookeeper/audit/AuditConstants.java
@@ -33,4 +33,6 @@ private AuditConstants() {
public static final String OP_MULTI_OP = "multiOperation";
public static final String OP_RECONFIG = "reconfig";
public static final String OP_DEL_EZNODE_EXP = "ephemeralZNodeDeletionOnSessionCloseOrExpire";
+ public static final String OP_SESSION_ESTABLISHED = "sessionEstablished";
+ public static final String OP_AUTHENTICATION = "authentication";
}
diff --git a/zookeeper-server/src/main/java/org/apache/zookeeper/audit/AuditEvent.java b/zookeeper-server/src/main/java/org/apache/zookeeper/audit/AuditEvent.java
index bd3a7842f95..d830570b72b 100644
--- a/zookeeper-server/src/main/java/org/apache/zookeeper/audit/AuditEvent.java
+++ b/zookeeper-server/src/main/java/org/apache/zookeeper/audit/AuditEvent.java
@@ -98,7 +98,7 @@ private static String escape(String value) {
public enum FieldName {
USER, OPERATION, IP, ACL, ZNODE, SESSION, ZNODE_TYPE,
- SCHEMA_VERSION, DATA_LENGTH, ERROR_CODE, OUTCOME, CXID, ZXID, MULTI_INDEX
+ SCHEMA_VERSION, DATA_LENGTH, ERROR_CODE, OUTCOME, CXID, ZXID, MULTI_INDEX, AUTH_SCHEME
}
public enum Result {
diff --git a/zookeeper-server/src/main/java/org/apache/zookeeper/audit/AuditHelper.java b/zookeeper-server/src/main/java/org/apache/zookeeper/audit/AuditHelper.java
index 4731cbe58a3..47aaf1c760d 100644
--- a/zookeeper-server/src/main/java/org/apache/zookeeper/audit/AuditHelper.java
+++ b/zookeeper-server/src/main/java/org/apache/zookeeper/audit/AuditHelper.java
@@ -19,8 +19,13 @@
import java.io.IOException;
import java.nio.ByteBuffer;
+import java.util.Collections;
+import java.util.LinkedHashMap;
+import java.util.LinkedHashSet;
import java.util.List;
import java.util.Locale;
+import java.util.Map;
+import java.util.Set;
import org.apache.jute.Record;
import org.apache.zookeeper.CreateMode;
import org.apache.zookeeper.KeeperException;
@@ -41,6 +46,7 @@
import org.apache.zookeeper.server.ByteBufferInputStream;
import org.apache.zookeeper.server.DataTree.ProcessTxnResult;
import org.apache.zookeeper.server.Request;
+import org.apache.zookeeper.server.ServerCnxn;
import org.apache.zookeeper.server.auth.AuthenticationProvider;
import org.apache.zookeeper.server.auth.DigestAuthenticationProvider;
import org.apache.zookeeper.server.auth.IPAuthenticationProvider;
@@ -57,6 +63,63 @@ public final class AuditHelper {
private static final Logger LOG = LoggerFactory.getLogger(AuditHelper.class);
private static final String REDACTED = "[redacted]";
+ /**
+ * Records the identities on a valid session attachment, including reconnects.
+ * The caller must have validated and assigned the session before invoking this hook.
+ */
+ public static void addSessionEstablishedLog(ServerCnxn cnxn) {
+ logConnection(cnxn, AuditConstants.OP_SESSION_ESTABLISHED, null, Result.SUCCESS, null, null);
+ }
+
+ /**
+ * Records an actual authentication outcome, not an intermediate SASL challenge.
+ * Success includes the accepted scheme's current bindings; failure does not infer a principal.
+ */
+ public static void addAuthenticationLog(ServerCnxn cnxn, String scheme, Code code, int cxid) {
+ logConnection(cnxn, AuditConstants.OP_AUTHENTICATION, scheme,
+ code == Code.OK ? Result.SUCCESS : Result.FAILURE, code == null ? null : code.intValue(), cxid);
+ }
+
+ private static void logConnection(ServerCnxn cnxn, String operation, String scheme,
+ Result result, Integer error, Integer cxid) {
+ try {
+ boolean enhanced = ZKAuditProvider.isEnhancedAuditEnabled();
+ if (!enhanced) {
+ return;
+ }
+ long sessionId = cnxn.getSessionId();
+ String session = sessionId == 0 ? null : "0x" + Long.toHexString(sessionId);
+ String ip = cnxn.getHostAddress();
+ Map> bindings = connectionBindings(cnxn, operation, scheme, result);
+ for (Map.Entry> binding : bindings.entrySet()) {
+ for (String user : binding.getValue()) {
+ ZKAuditProvider.logConnection(user, operation, binding.getKey(),
+ session, ip, result, error, cxid, enhanced);
+ }
+ }
+ } catch (RuntimeException e) {
+ ZKAuditProvider.reportAuditError(LOG, "Failed to audit log operation {}", operation, e);
+ }
+ }
+
+ private static Map> connectionBindings(
+ ServerCnxn cnxn, String operation, String scheme, Result result) {
+ boolean attachment = AuditConstants.OP_SESSION_ESTABLISHED.equals(operation);
+ Map> bindings = new LinkedHashMap<>();
+ if (result == Result.SUCCESS) {
+ for (Id id : cnxn.getAuthInfo()) {
+ if (attachment || (id != null && scheme != null && scheme.equals(id.getScheme()))) {
+ bindings.computeIfAbsent(id == null ? null : id.getScheme(), key -> new LinkedHashSet<>())
+ .add(safeUser(id));
+ }
+ }
+ }
+ if (bindings.isEmpty()) {
+ bindings.put(scheme, Collections.singleton(null));
+ }
+ return bindings;
+ }
+
public static void addAuditLog(Request request, ProcessTxnResult rc) {
addAuditLog(request, rc, false);
}
diff --git a/zookeeper-server/src/main/java/org/apache/zookeeper/audit/ZKAuditProvider.java b/zookeeper-server/src/main/java/org/apache/zookeeper/audit/ZKAuditProvider.java
index f05062f9d1c..f76258b6b38 100644
--- a/zookeeper-server/src/main/java/org/apache/zookeeper/audit/ZKAuditProvider.java
+++ b/zookeeper-server/src/main/java/org/apache/zookeeper/audit/ZKAuditProvider.java
@@ -105,6 +105,19 @@ static void log(String user, String operation, String znode, String acl,
dataLength, errorCode, outcome, cxid, zxid, multiIndex, enhanced));
}
+ static void logConnection(String user, String operation, String authScheme,
+ String session, String ip, Result result,
+ Integer errorCode, Integer cxid, boolean enhanced) {
+ if (!isAuditEnabled() || !enhanced) {
+ return;
+ }
+ AuditEvent event = createLogEvent(user, operation, null, null, null, session, ip, result,
+ null, errorCode, result == Result.FAILURE ? Outcome.FAILED : Outcome.UNKNOWN,
+ cxid, null, null, enhanced);
+ event.addEntry(FieldName.AUTH_SCHEME, authScheme);
+ logAuditEvent(event);
+ }
+
/**
* A helper api for creating an AuditEvent object.
*/
diff --git a/zookeeper-server/src/main/java/org/apache/zookeeper/server/ZooKeeperServer.java b/zookeeper-server/src/main/java/org/apache/zookeeper/server/ZooKeeperServer.java
index 2b2a5e5f569..09e4c70eb92 100644
--- a/zookeeper-server/src/main/java/org/apache/zookeeper/server/ZooKeeperServer.java
+++ b/zookeeper-server/src/main/java/org/apache/zookeeper/server/ZooKeeperServer.java
@@ -49,6 +49,7 @@
import org.apache.zookeeper.ZooDefs;
import org.apache.zookeeper.ZooDefs.OpCode;
import org.apache.zookeeper.ZookeeperBanner;
+import org.apache.zookeeper.audit.AuditHelper;
import org.apache.zookeeper.common.Time;
import org.apache.zookeeper.data.ACL;
import org.apache.zookeeper.data.Id;
@@ -1108,6 +1109,7 @@ public void finishSessionInit(ServerCnxn cnxn, boolean valid) {
Long.toHexString(cnxn.getSessionId()),
cnxn.getSessionTimeout(),
cnxn.getRemoteSocketAddress());
+ AuditHelper.addSessionEstablishedLog(cnxn);
cnxn.enableRecv();
} else {
@@ -1658,9 +1660,11 @@ public void processPacket(ServerCnxn cnxn, ByteBuffer incomingBuffer) throws IOE
if (authReturn == KeeperException.Code.OK) {
LOG.debug("Authentication succeeded for scheme: {}", scheme);
LOG.info("auth success {}", cnxn.getRemoteSocketAddress());
+ AuditHelper.addAuthenticationLog(cnxn, scheme, Code.OK, h.getXid());
ReplyHeader rh = new ReplyHeader(h.getXid(), 0, KeeperException.Code.OK.intValue());
cnxn.sendResponse(rh, null, null);
} else {
+ AuditHelper.addAuthenticationLog(cnxn, scheme, Code.AUTHFAILED, h.getXid());
if (ap == null) {
LOG.warn(
"No authentication provider for scheme: {} has {}",
@@ -1738,8 +1742,10 @@ private void processSasl(ByteBuffer incomingBuffer, ServerCnxn cnxn, RequestHead
&& authorizationID.equals(System.getProperty("zookeeper.superUser"))) {
cnxn.addAuthInfo(new Id("super", ""));
}
+ AuditHelper.addAuthenticationLog(cnxn, SASL_AUTH_SCHEME, Code.OK, requestHeader.getXid());
}
} catch (SaslException e) {
+ AuditHelper.addAuthenticationLog(cnxn, SASL_AUTH_SCHEME, Code.AUTHFAILED, requestHeader.getXid());
LOG.warn("Client {} failed to SASL authenticate: {}", cnxn.getRemoteSocketAddress(), e);
if (shouldAllowSaslFailedClientsConnect() && !shouldRequireClientSaslAuth()) {
LOG.warn("Maintaining client connection despite SASL authentication failure.");
diff --git a/zookeeper-server/src/test/java/org/apache/zookeeper/audit/SessionAuthAuditTest.java b/zookeeper-server/src/test/java/org/apache/zookeeper/audit/SessionAuthAuditTest.java
new file mode 100644
index 00000000000..c1f89d4118f
--- /dev/null
+++ b/zookeeper-server/src/test/java/org/apache/zookeeper/audit/SessionAuthAuditTest.java
@@ -0,0 +1,953 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.zookeeper.audit;
+
+import static org.apache.zookeeper.audit.AuditHelperTest.fields;
+import static org.junit.Assert.assertArrayEquals;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertNull;
+import static org.junit.Assert.assertTrue;
+import static org.junit.Assert.fail;
+import static org.mockito.Mockito.CALLS_REAL_METHODS;
+import static org.mockito.Mockito.mock;
+import ch.qos.logback.classic.Level;
+import ch.qos.logback.classic.Logger;
+import ch.qos.logback.classic.spi.ILoggingEvent;
+import ch.qos.logback.core.AppenderBase;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.lang.reflect.Field;
+import java.net.InetSocketAddress;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.security.cert.X509Certificate;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+import javax.net.ssl.X509TrustManager;
+import javax.security.auth.callback.Callback;
+import javax.security.auth.callback.CallbackHandler;
+import javax.security.auth.callback.NameCallback;
+import javax.security.auth.callback.PasswordCallback;
+import javax.security.auth.callback.UnsupportedCallbackException;
+import javax.security.sasl.AuthorizeCallback;
+import javax.security.sasl.RealmCallback;
+import javax.security.sasl.Sasl;
+import javax.security.sasl.SaslClient;
+import javax.security.sasl.SaslServer;
+import org.apache.jute.BinaryOutputArchive;
+import org.apache.jute.Record;
+import org.apache.zookeeper.CreateMode;
+import org.apache.zookeeper.KeeperException;
+import org.apache.zookeeper.KeeperException.Code;
+import org.apache.zookeeper.TestableZooKeeper;
+import org.apache.zookeeper.ZKTestCase;
+import org.apache.zookeeper.ZooDefs;
+import org.apache.zookeeper.ZooDefs.OpCode;
+import org.apache.zookeeper.ZooKeeper;
+import org.apache.zookeeper.audit.AuditHelperTest.AuditCapture;
+import org.apache.zookeeper.audit.AuditHelperTest.CredentialAuthenticationProvider;
+import org.apache.zookeeper.audit.AuditHelperTest.FailingCounter;
+import org.apache.zookeeper.data.Id;
+import org.apache.zookeeper.data.Stat;
+import org.apache.zookeeper.metrics.Counter;
+import org.apache.zookeeper.proto.AuthPacket;
+import org.apache.zookeeper.proto.GetSASLRequest;
+import org.apache.zookeeper.proto.ReplyHeader;
+import org.apache.zookeeper.proto.RequestHeader;
+import org.apache.zookeeper.proto.SetSASLResponse;
+import org.apache.zookeeper.server.MockServerCnxn;
+import org.apache.zookeeper.server.ServerCnxn;
+import org.apache.zookeeper.server.ZooKeeperSaslServer;
+import org.apache.zookeeper.server.ZooKeeperServer;
+import org.apache.zookeeper.server.auth.DigestAuthenticationProvider;
+import org.apache.zookeeper.server.auth.ProviderRegistry;
+import org.apache.zookeeper.server.auth.SASLAuthenticationProvider;
+import org.apache.zookeeper.server.auth.X509AuthenticationProvider;
+import org.apache.zookeeper.test.ClientBase;
+import org.apache.zookeeper.test.QuorumUtil;
+import org.apache.zookeeper.test.X509AuthTest.TestCertificate;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+import org.slf4j.LoggerFactory;
+
+public class SessionAuthAuditTest extends ZKTestCase {
+ private static final long SESSION = 0x123;
+ private static final int AUTH_XID = -4;
+ private static final int SASL_XID = -33;
+ private static final String SECRET = "SYNTHETIC_C2_PASSWORD";
+ private static final String TOKEN = "SYNTHETIC_C2_TOKEN";
+ private static final String CERTIFICATE = "SYNTHETIC_C2_CERTIFICATE_BODY";
+ private static final String PROVIDER_PREFIX = "zookeeper.authProvider.auditC2";
+ private final Map previousProperties = new LinkedHashMap<>();
+ private final List captured = new ArrayList<>();
+ private AuditCapture capture;
+ private ZooKeeperServer server;
+ private Object previousAuditEnabled;
+ private Object previousAuditLogger;
+
+ @Before
+ public void setUp() throws Exception {
+ property(ZKAuditProvider.AUDIT_ENABLE, "true");
+ property(AuditHelperTest.ENHANCED_ENABLE, "true");
+ property(ZooKeeperServer.ALLOW_SASL_FAILED_CLIENTS, "false");
+ property(ZooKeeperServer.SESSION_REQUIRE_CLIENT_SASL_AUTH, "false");
+ property(PROVIDER_PREFIX + "Sasl", SASLAuthenticationProvider.class.getName());
+ property(PROVIDER_PREFIX + "X509", X509AuthenticationProvider.class.getName());
+ property(PROVIDER_PREFIX + "Custom", CredentialAuthenticationProvider.class.getName());
+ property(PROVIDER_PREFIX + "Empty", EmptyAuthenticationProvider.class.getName());
+ ProviderRegistry.reset();
+ ProviderRegistry.initialize();
+ previousAuditEnabled = AuditHelperTest.replaceProviderField("auditEnabled", true);
+ previousAuditLogger = AuditHelperTest.replaceProviderField("auditLogger", new Slf4jAuditLogger());
+ capture = new AuditCapture();
+ server = new ZooKeeperServer();
+ }
+
+ @After
+ public void tearDown() throws Exception {
+ try {
+ for (String log : captured) {
+ assertFalse(log, log.contains(SECRET));
+ assertFalse(log, log.contains(TOKEN));
+ assertFalse(log, log.contains(CERTIFICATE));
+ assertFalse(log, log.contains(DigestAuthenticationProvider.generateDigest("alice:" + SECRET)));
+ }
+ } finally {
+ capture.close();
+ AuditHelperTest.replaceProviderField("auditLogger", previousAuditLogger);
+ AuditHelperTest.replaceProviderField("auditEnabled", previousAuditEnabled);
+ previousProperties.forEach(AuditHelperTest::restoreProperty);
+ ProviderRegistry.reset();
+ }
+ }
+
+ @Test
+ public void testAnonymousAttachmentDoesNotClaimAuthenticationOrNewSession() throws Exception {
+ RecordingCnxn connection = new RecordingCnxn(SESSION);
+ server.finishSessionInit(connection, true);
+ String first = read(1).get(0);
+ assertBindingEvent(first, "sessionEstablished", "0x123", "success", null, "unknown", null);
+ assertBindings(first, new String[][]{{null, null}});
+
+ server.finishSessionInit(connection, true);
+ String reconnect = read(1).get(0);
+ assertBindingEvent(reconnect, "sessionEstablished", "0x123", "success", null, "unknown", null);
+ assertBindings(reconnect, new String[][]{{null, null}});
+ }
+
+ @Test
+ public void testRejectedAttachmentDoesNotBindAnUnvalidatedSessionId() {
+ server.finishSessionInit(new RecordingCnxn(SESSION), false);
+ read(0);
+ }
+
+ @Test
+ public void testX509IdentityBeforeAssignmentIsBoundWithoutClaimingTls() throws Exception {
+ RecordingCnxn connection = new RecordingCnxn(0);
+ X509Certificate certificate = new TestCertificate("CLIENT,OU=Audit") {
+ @Override
+ public byte[] getEncoded() {
+ return CERTIFICATE.getBytes(StandardCharsets.UTF_8);
+ }
+ };
+ connection.clientChain = new X509Certificate[]{certificate};
+ X509AuthenticationProvider provider = new X509AuthenticationProvider(mock(X509TrustManager.class), null);
+ assertEquals(Code.OK, provider.handleAuthentication(connection, null));
+ assertEquals(0, connection.getSessionId());
+ read(0);
+
+ connection.sessionId = SESSION;
+ connection.addAuthInfo(new Id("super", TOKEN));
+ server.finishSessionInit(connection, true);
+ List logs = read(2);
+ assertBindings(logs, new String[][]{{"x509", "CN=CLIENT,OU=Audit"}, {"super", "[redacted]"}});
+ assertFalse(connection.isSecure());
+ for (String log : logs) {
+ assertBindingEvent(log, "sessionEstablished", "0x123", "success", null, "unknown", null);
+ assertNull(fields(log).get("tls"));
+ assertNull(fields(log).get("secure"));
+ }
+ }
+
+ @Test
+ public void testMultipleSchemesRemainPairedAndUntrustedIdentitiesAreRedacted() throws Exception {
+ RecordingCnxn connection = new RecordingCnxn(SESSION);
+ connection.addAuthInfo(new Id("ip", "127.0.0.1"));
+ connection.addAuthInfo(new Id("digest", DigestAuthenticationProvider.generateDigest("alice:" + SECRET)));
+ connection.addAuthInfo(new Id("sasl", "service/host@EXAMPLE.COM"));
+ connection.addAuthInfo(new Id("x509", "CN=Bob,OU=Audit"));
+ connection.addAuthInfo(new Id("x509", "-----BEGIN CERTIFICATE-----\n" + CERTIFICATE));
+ connection.addAuthInfo(new Id("audit-test-custom", "alice:" + SECRET));
+ connection.addAuthInfo(new Id("missing", TOKEN));
+ connection.addAuthInfo(new Id("digest", TOKEN));
+ server.finishSessionInit(connection, true);
+ List logs = read(8);
+ assertBindings(logs, new String[][]{
+ {"ip", "127.0.0.1"}, {"digest", "alice"}, {"sasl", "service/host@EXAMPLE.COM"},
+ {"x509", "CN=Bob,OU=Audit"}, {"audit-test-custom", "[redacted]"},
+ {"x509", "[redacted]"},
+ {"missing", "[redacted]"}, {"digest", "[redacted]"}});
+ for (String log : logs) {
+ assertBindingEvent(log, "sessionEstablished", "0x123", "success", null, "unknown", null);
+ }
+ }
+
+ @Test
+ public void testExplicitAuthRecordsOnlyTheAcceptedSchemeAndNotCredentials() throws Exception {
+ RecordingCnxn connection = new RecordingCnxn(SESSION);
+ connection.addAuthInfo(new Id("ip", "127.0.0.1"));
+ authenticate(connection, "digest", "alice:" + SECRET);
+ assertEquals(Code.OK.intValue(), connection.lastReply().getErr());
+ String log = read(1).get(0);
+ assertBindingEvent(log, "authentication", "0x123", "success", "0", "unknown", "-4");
+ assertBindings(log, new String[][]{{"digest", "alice"}});
+ assertTrue(connection.getAuthInfo().contains(
+ new Id("digest", DigestAuthenticationProvider.generateDigest("alice:" + SECRET))));
+ }
+
+ @Test
+ public void testPreSessionAuthFailureHasNoInventedSessionOrPrincipal() throws Exception {
+ RecordingCnxn connection = new RecordingCnxn(0);
+ authenticate(connection, "missing", TOKEN);
+ assertEquals(Code.AUTHFAILED.intValue(), connection.lastReply().getErr());
+ String log = read(1).get(0);
+ assertBindingEvent(log, "authentication", null, "failure", "-115", "failed", "-4");
+ assertBindings(log, new String[][]{{"missing", null}});
+ }
+
+ @Test
+ public void testAuthFailureDoesNotBlameAPreviouslyAttachedPrincipal() throws Exception {
+ RecordingCnxn connection = new RecordingCnxn(SESSION);
+ connection.addAuthInfo(new Id("sasl", "previous@EXAMPLE.COM"));
+ authenticate(connection, "sasl", TOKEN);
+ assertEquals(Code.AUTHFAILED.intValue(), connection.lastReply().getErr());
+ String log = read(1).get(0);
+ assertBindings(log, new String[][]{{"sasl", null}});
+ assertFalse(log.contains("previous@EXAMPLE.COM"));
+ }
+
+ @Test
+ public void testAcceptedAuthWithoutAnIdentityRetainsUnknownPrincipal() throws Exception {
+ RecordingCnxn connection = new RecordingCnxn(SESSION);
+ connection.addAuthInfo(new Id("ip", "127.0.0.1"));
+ authenticate(connection, "audit-c2-empty", TOKEN);
+ assertEquals(Code.OK.intValue(), connection.lastReply().getErr());
+ String log = read(1).get(0);
+ assertBindingEvent(log, "authentication", "0x123", "success", "0", "unknown", "-4");
+ assertBindings(log, new String[][]{{"audit-c2-empty", null}});
+ }
+
+ @Test
+ public void testScalarUserRoundTripsDelimitersThroughRealSlf4jFormatting() throws Exception {
+ String username = "a,b=\"c\\d\t\r\n";
+ RecordingCnxn connection = new RecordingCnxn(SESSION);
+ authenticate(connection, "digest", username + ":" + SECRET);
+ String log = read(1).get(0);
+ assertBindings(log, new String[][]{{"digest", "a,b=\"c\\d\t\r\n"}});
+ assertFalse(log.contains("\n"));
+ assertFalse(log.contains("\r"));
+ assertEquals(10, fields(log).size());
+ }
+
+ @Test
+ public void testAuthSchemeUsesTheExistingVersionedEscaping() throws Exception {
+ RecordingCnxn connection = new RecordingCnxn(SESSION);
+ authenticate(connection, "missing,scheme=\"value\\t\t\r\n", TOKEN);
+ String log = read(1).get(0);
+ assertBindingEvent(log, "authentication", "0x123", "failure", "-115", "failed", "-4");
+ assertBindings(log, new String[][]{{"missing,scheme=\"value\\t\t\r\n", null}});
+ assertFalse(log.contains("\n"));
+ assertFalse(log.contains("\r"));
+ assertFalse(fields(log).containsKey("user"));
+ }
+
+ @Test
+ public void testRepeatedSanitizedPrincipalIsBoundOncePerAuthOutcome() throws Exception {
+ RecordingCnxn connection = new RecordingCnxn(SESSION);
+ authenticate(connection, "digest", "alice:" + SECRET);
+ assertBindings(read(1).get(0), new String[][]{{"digest", "alice"}});
+ authenticate(connection, "digest", "alice:" + TOKEN);
+ assertEquals(2, connection.getAuthInfo().size());
+ String log = read(1).get(0);
+ assertBindingEvent(log, "authentication", "0x123", "success", "0", "unknown", "-4");
+ assertBindings(log, new String[][]{{"digest", "alice"}});
+ }
+
+ @Test
+ public void testRedactedIdentitiesDoNotProduceDuplicateBindings() throws Exception {
+ RecordingCnxn connection = new RecordingCnxn(SESSION);
+ connection.addAuthInfo(new Id("missing", SECRET));
+ connection.addAuthInfo(new Id("missing", TOKEN));
+ server.finishSessionInit(connection, true);
+ assertBindings(read(1).get(0), new String[][]{{"missing", "[redacted]"}});
+ }
+
+ @Test
+ public void testOneModeSnapshotCoversAllIdentityBindings() throws Exception {
+ RecordingCnxn connection = new RecordingCnxn(SESSION);
+ connection.addAuthInfo(new Id("ip", "127.0.0.1"));
+ connection.addAuthInfo(new Id("digest", DigestAuthenticationProvider.generateDigest("alice:" + SECRET)));
+ Slf4jAuditLogger logger = new Slf4jAuditLogger();
+ AtomicInteger emitted = new AtomicInteger();
+ Object previous = AuditHelperTest.replaceProviderField("auditLogger", (AuditLogger) event -> {
+ logger.logAuditEvent(event);
+ if (emitted.incrementAndGet() == 1) {
+ System.setProperty(AuditHelperTest.ENHANCED_ENABLE, "false");
+ }
+ });
+ try {
+ server.finishSessionInit(connection, true);
+ List logs = read(2);
+ assertBindings(logs, new String[][]{{"ip", "127.0.0.1"}, {"digest", "alice"}});
+ for (String log : logs) {
+ assertBindingEvent(log, "sessionEstablished", "0x123", "success", null, "unknown", null);
+ }
+ server.finishSessionInit(connection, true);
+ read(0);
+ } finally {
+ AuditHelperTest.replaceProviderField("auditLogger", previous);
+ }
+ }
+
+ @Test
+ public void testSinkAndCounterFailuresDoNotDiscardRemainingBindings() throws Exception {
+ RecordingCnxn connection = new RecordingCnxn(SESSION);
+ connection.addAuthInfo(new Id("ip", "127.0.0.1"));
+ connection.addAuthInfo(new Id("digest", DigestAuthenticationProvider.generateDigest("alice:" + SECRET)));
+ FailingCounter counter = new FailingCounter();
+ Counter previousCounter = AuditHelperTest.replaceAuditErrorCounter(counter);
+ Slf4jAuditLogger logger = new Slf4jAuditLogger();
+ Object previousLogger = AuditHelperTest.replaceProviderField("auditLogger", (AuditLogger) event -> {
+ logger.logAuditEvent(event);
+ throw new IllegalStateException("synthetic per-binding sink failure");
+ });
+ try {
+ server.finishSessionInit(connection, true);
+ assertNull(connection.disconnectReason);
+ assertBindings(read(2), new String[][]{{"ip", "127.0.0.1"}, {"digest", "alice"}});
+ assertEquals(2, counter.get());
+ } finally {
+ AuditHelperTest.replaceProviderField("auditLogger", previousLogger);
+ AuditHelperTest.replaceAuditErrorCounter(previousCounter);
+ }
+ }
+
+ @Test
+ public void testDisabledGatesDoNotReadBindingMetadataOrChangeAuthentication() throws Exception {
+ RecordingCnxn connection = new RecordingCnxn(SESSION);
+ connection.authInfoRead = () -> {
+ throw new IllegalStateException("binding metadata must not be read");
+ };
+ property(AuditHelperTest.ENHANCED_ENABLE, null);
+ server.finishSessionInit(connection, true);
+ authenticate(connection, "digest", "alice:" + SECRET);
+ assertEquals(Code.OK.intValue(), connection.lastReply().getErr());
+ read(0);
+
+ property(AuditHelperTest.ENHANCED_ENABLE, "true");
+ Object previous = AuditHelperTest.replaceProviderField("auditEnabled", false);
+ try {
+ server.finishSessionInit(connection, true);
+ authenticate(connection, "digest", "alice:" + SECRET);
+ assertEquals(Code.OK.intValue(), connection.lastReply().getErr());
+ read(0);
+ } finally {
+ AuditHelperTest.replaceProviderField("auditEnabled", previous);
+ }
+ }
+
+ @Test
+ public void testBindingKeepsCapturedModeWhenIdentityExtractionChangesTheGate() throws Exception {
+ RecordingCnxn connection = new RecordingCnxn(SESSION);
+ connection.addAuthInfo(new Id("audit-test-custom", "alice:" + SECRET));
+ connection.authInfoRead = () -> System.setProperty(AuditHelperTest.ENHANCED_ENABLE, "false");
+ server.finishSessionInit(connection, true);
+ String log = read(1).get(0);
+ assertBindingEvent(log, "sessionEstablished", "0x123", "success", null, "unknown", null);
+ assertBindings(log, new String[][]{{"audit-test-custom", "[redacted]"}});
+ server.finishSessionInit(connection, true);
+ read(0);
+ }
+
+ @Test
+ public void testAuditGateReadFailureDoesNotAbortAcceptedAuthentication() throws Exception {
+ RecordingCnxn connection = new RecordingCnxn(SESSION);
+ long errors = AuditHelperTest.auditErrors();
+ Properties previous = failEnhancedGateRead();
+ try {
+ try {
+ authenticate(connection, "digest", "alice:" + SECRET);
+ } catch (RuntimeException e) {
+ fail("Audit gate access must not abort accepted authentication: " + e);
+ }
+ assertEquals(Code.OK.intValue(), connection.lastReply().getErr());
+ assertEquals(errors + 1, AuditHelperTest.auditErrors());
+ read(0);
+ } finally {
+ System.setProperties(previous);
+ }
+ }
+
+ @Test
+ public void testAuditGateReadFailureDoesNotCloseAValidSession() {
+ RecordingCnxn connection = new RecordingCnxn(SESSION);
+ long errors = AuditHelperTest.auditErrors();
+ Properties previous = failEnhancedGateRead();
+ try {
+ server.finishSessionInit(connection, true);
+ assertNull("Audit gate access must not close a valid session", connection.disconnectReason);
+ assertEquals(errors + 1, AuditHelperTest.auditErrors());
+ read(0);
+ } finally {
+ System.setProperties(previous);
+ }
+ }
+
+ @Test
+ public void testMetadataFailureAndBothBrokenReportersCannotChangeAuthOrSession() throws Exception {
+ RecordingCnxn connection = new RecordingCnxn(SESSION);
+ connection.authInfoRead = () -> {
+ throw new IllegalStateException("synthetic binding metadata failure");
+ };
+ FailingCounter counter = new FailingCounter();
+ Counter previousCounter = AuditHelperTest.replaceAuditErrorCounter(counter);
+ Logger logger = (Logger) LoggerFactory.getLogger(AuditHelper.class);
+ Level previousLevel = logger.getLevel();
+ AtomicInteger reports = new AtomicInteger();
+ AppenderBase failingAppender = new AppenderBase() {
+ @Override
+ public void doAppend(ILoggingEvent event) {
+ reports.incrementAndGet();
+ throw new IllegalStateException("synthetic diagnostic failure");
+ }
+
+ @Override
+ protected void append(ILoggingEvent event) {
+ }
+ };
+ logger.setLevel(Level.ERROR);
+ logger.addAppender(failingAppender);
+ try {
+ server.finishSessionInit(connection, true);
+ authenticate(connection, "digest", "alice:" + SECRET);
+ assertNull(connection.disconnectReason);
+ assertEquals(Code.OK.intValue(), connection.lastReply().getErr());
+ assertEquals(2, counter.get());
+ assertEquals(2, reports.get());
+ read(0);
+ connection.authInfoRead = null;
+ assertTrue(connection.getAuthInfo().contains(
+ new Id("digest", DigestAuthenticationProvider.generateDigest("alice:" + SECRET))));
+ } finally {
+ logger.detachAppender(failingAppender);
+ logger.setLevel(previousLevel);
+ AuditHelperTest.replaceAuditErrorCounter(previousCounter);
+ }
+ }
+
+ @Test
+ public void testAuthenticationKeepsCapturedModeDuringBindingExtraction() throws Exception {
+ RecordingCnxn connection = new RecordingCnxn(SESSION);
+ connection.authInfoRead = () -> System.setProperty(AuditHelperTest.ENHANCED_ENABLE, "false");
+ authenticate(connection, "audit-test-custom", "alice:" + SECRET);
+ String log = read(1).get(0);
+ assertBindingEvent(log, "authentication", "0x123", "success", "0", "unknown", "-4");
+ assertBindings(log, new String[][]{{"audit-test-custom", "[redacted]"}});
+ assertEquals(Code.OK.intValue(), connection.lastReply().getErr());
+ }
+
+ @Test
+ public void testSaslCompletionDoesNotAuditIntermediateChallenges() throws Exception {
+ RecordingCnxn connection = new RecordingCnxn(SESSION);
+ try (SaslExchange sasl = new SaslExchange(connection, SECRET)) {
+ byte[] response = sasl.challenge();
+ assertFalse(sasl.server.isComplete());
+ read(0);
+ sasl.respond(response);
+ assertTrue(sasl.server.isComplete());
+ assertTrue(sasl.client.isComplete());
+ assertEquals(Code.OK.intValue(), connection.lastReply().getErr());
+ String log = read(1).get(0);
+ assertBindingEvent(log, "authentication", "0x123", "success", "0", "unknown", "-33");
+ assertBindings(log, new String[][]{{"sasl", "alice"}});
+ assertTrue(connection.getAuthInfo().contains(new Id("sasl", "alice")));
+ }
+ }
+
+ @Test
+ public void testSaslFailurePreservesDefaultClosingPolicy() throws Exception {
+ assertSaslFailure(SESSION, false, false, Code.AUTHFAILED, true);
+ }
+
+ @Test
+ public void testAllowedSaslFailureIsNotMistakenForSuccessfulAuthentication() throws Exception {
+ assertSaslFailure(SESSION, true, false, Code.OK, false);
+ }
+
+ @Test
+ public void testRequiredSaslFailurePreservesItsDistinctClosingCode() throws Exception {
+ assertSaslFailure(SESSION, true, true, Code.SESSIONCLOSEDREQUIRESASLAUTH, true);
+ }
+
+ @Test
+ public void testPreSessionSaslFailureOmitsSessionIdentity() throws Exception {
+ assertSaslFailure(0, false, false, Code.AUTHFAILED, true);
+ }
+
+ @Test
+ public void testMissingSaslServerDoesNotFabricateAuthenticationSuccess() throws Exception {
+ RecordingCnxn connection = new RecordingCnxn(SESSION);
+ server.processPacket(connection, packet(OpCode.sasl, SASL_XID, new GetSASLRequest(new byte[0])));
+ assertEquals(Code.OK.intValue(), connection.lastReply().getErr());
+ read(0);
+ }
+
+ @Test
+ public void testFailingAuditSinkAndCounterPreserveBothSaslFailurePolicies() throws Exception {
+ FailingCounter counter = new FailingCounter();
+ Counter previousCounter = AuditHelperTest.replaceAuditErrorCounter(counter);
+ Slf4jAuditLogger logger = new Slf4jAuditLogger();
+ Object previousLogger = AuditHelperTest.replaceProviderField("auditLogger", (AuditLogger) event -> {
+ logger.logAuditEvent(event);
+ throw new IllegalStateException("synthetic C2 SASL audit sink failure");
+ });
+ try {
+ assertSaslFailure(SESSION, false, false, Code.AUTHFAILED, true);
+ assertSaslFailure(SESSION, true, false, Code.OK, false);
+ assertEquals(2, counter.get());
+ } finally {
+ AuditHelperTest.replaceProviderField("auditLogger", previousLogger);
+ AuditHelperTest.replaceAuditErrorCounter(previousCounter);
+ }
+ }
+
+ @Test
+ public void testRealStandaloneAuthChangesDoNotRewriteEarlierWriteUsers() throws Exception {
+ Standalone fixture = new Standalone();
+ fixture.setUp();
+ try {
+ ZooKeeper client = fixture.connect();
+ client.exists("/", false);
+ String session = "0x" + Long.toHexString(client.getSessionId());
+ String attachment = read(1).get(0);
+ assertBindingEvent(attachment, "sessionEstablished", session, "success", null, "unknown", null);
+ assertBindings(attachment, new String[][]{{"ip", "127.0.0.1"}});
+
+ client.create("/before-auth", new byte[0], ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT);
+ String before = read(1).get(0);
+ assertEquals("127.0.0.1", fields(before).get("user"));
+ client.addAuthInfo("digest", ("alice:" + SECRET).getBytes(StandardCharsets.UTF_8));
+ client.exists("/", false);
+ assertBindings(read(1).get(0), new String[][]{{"digest", "alice"}});
+ client.setData("/before-auth", new byte[]{1}, -1);
+ assertUsers(read(1).get(0), "127.0.0.1", "alice");
+
+ client.addAuthInfo("digest", ("bob:" + SECRET).getBytes(StandardCharsets.UTF_8));
+ client.exists("/", false);
+ assertBindings(read(2), new String[][]{{"digest", "alice"}, {"digest", "bob"}});
+ client.setData("/before-auth", new byte[]{2}, -1);
+ assertUsers(read(1).get(0), "127.0.0.1", "alice", "bob");
+ assertEquals("127.0.0.1", fields(before).get("user"));
+ assertArrayEquals(new byte[]{2}, client.getData("/before-auth", false, null));
+ } finally {
+ fixture.tearDown();
+ }
+ }
+
+ @Test
+ public void testRealAuthFailurePreservesTheClientError() throws Exception {
+ Standalone fixture = new Standalone();
+ fixture.setUp();
+ try {
+ ZooKeeper client = fixture.connect();
+ client.exists("/", false);
+ String session = "0x" + Long.toHexString(client.getSessionId());
+ read(1);
+ client.addAuthInfo("missing", TOKEN.getBytes(StandardCharsets.UTF_8));
+ try {
+ client.exists("/", false);
+ fail("Unknown authentication provider must still fail the client");
+ } catch (KeeperException.AuthFailedException expected) {
+ assertEquals(Code.AUTHFAILED, expected.code());
+ }
+ String log = read(1).get(0);
+ assertBindingEvent(log, "authentication", session, "failure", "-115", "failed", "-4");
+ assertBindings(log, new String[][]{{"missing", null}});
+ } finally {
+ fixture.tearDown();
+ }
+ }
+
+ @Test
+ public void testRealQuorumMovementRebindsTheSameSessionWithoutOldConnectionAuth() throws Exception {
+ QuorumUtil quorum = new QuorumUtil(1);
+ TestableZooKeeper original = null;
+ ZooKeeper reconnected = null;
+ try {
+ quorum.startAll();
+ capture.clear();
+ ClientBase.CountdownWatcher watcher = new ClientBase.CountdownWatcher();
+ original = new TestableZooKeeper(
+ quorum.getConnectString(quorum.getLeaderQuorumPeer()), ClientBase.CONNECTION_TIMEOUT, watcher);
+ watcher.waitForConnected(ClientBase.CONNECTION_TIMEOUT);
+ original.exists("/", false);
+ long session = original.getSessionId();
+ byte[] passwd = original.getSessionPasswd();
+ String sessionText = "0x" + Long.toHexString(session);
+ assertBindingEvent(read(1).get(0), "sessionEstablished", sessionText, "success", null, "unknown", null);
+ original.addAuthInfo("digest", ("alice:" + SECRET).getBytes(StandardCharsets.UTF_8));
+ original.create("/moving", new byte[]{1}, ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.EPHEMERAL);
+ List before = read(2);
+ assertBindings(before.get(0), new String[][]{{"digest", "alice"}});
+ original.disconnect();
+
+ watcher = new ClientBase.CountdownWatcher();
+ reconnected = new ZooKeeper(
+ quorum.getConnectString(quorum.getFollowerQuorumPeers().get(0)),
+ ClientBase.CONNECTION_TIMEOUT, watcher, session, passwd);
+ watcher.waitForConnected(ClientBase.CONNECTION_TIMEOUT);
+ assertEquals(session, reconnected.getSessionId());
+ assertArrayEquals(new byte[]{1}, reconnected.getData("/moving", false, null));
+ String attachment = read(1).get(0);
+ assertBindingEvent(attachment, "sessionEstablished", sessionText, "success", null, "unknown", null);
+ assertBindings(attachment, new String[][]{{"ip", "127.0.0.1"}});
+ reconnected.addAuthInfo("digest", ("bob:" + SECRET).getBytes(StandardCharsets.UTF_8));
+ reconnected.setData("/moving", new byte[]{2}, -1);
+ List after = read(2);
+ assertBindings(after.get(0), new String[][]{{"digest", "bob"}});
+ assertUsers(after.get(1), "127.0.0.1", "bob");
+ assertArrayEquals(new byte[]{2}, reconnected.getData("/moving", false, null));
+ read(0);
+ } finally {
+ if (reconnected != null) {
+ reconnected.close();
+ }
+ if (original != null) {
+ original.close();
+ }
+ quorum.tearDown();
+ }
+ }
+
+ @Test
+ public void testFailingAuditSinkAndCounterDoNotRejectRealSessionAuthOrWrite() throws Exception {
+ FailingCounter counter = new FailingCounter();
+ Counter previousCounter = AuditHelperTest.replaceAuditErrorCounter(counter);
+ Slf4jAuditLogger logger = new Slf4jAuditLogger();
+ Object previousLogger = AuditHelperTest.replaceProviderField("auditLogger", (AuditLogger) event -> {
+ logger.logAuditEvent(event);
+ throw new IllegalStateException("synthetic C2 audit sink failure");
+ });
+ Standalone fixture = new Standalone();
+ try {
+ fixture.setUp();
+ ZooKeeper client = fixture.connect();
+ client.addAuthInfo("digest", ("alice:" + SECRET).getBytes(StandardCharsets.UTF_8));
+ client.create("/audit-failure", new byte[]{9}, ZooDefs.Ids.CREATOR_ALL_ACL, CreateMode.PERSISTENT);
+ assertArrayEquals(new byte[]{9}, client.getData("/audit-failure", false, null));
+ List logs = read(3);
+ assertEquals("sessionEstablished", fields(logs.get(0)).get("operation"));
+ assertBindings(logs.get(1), new String[][]{{"digest", "alice"}});
+ assertEquals("committed", fields(logs.get(2)).get("outcome"));
+ assertEquals(3, counter.get());
+ } finally {
+ AuditHelperTest.replaceProviderField("auditLogger", previousLogger);
+ AuditHelperTest.replaceAuditErrorCounter(previousCounter);
+ fixture.tearDown();
+ }
+ }
+
+ private void assertSaslFailure(long session, boolean allow, boolean require, Code replyCode, boolean closed)
+ throws Exception {
+ property(ZooKeeperServer.ALLOW_SASL_FAILED_CLIENTS, Boolean.toString(allow));
+ property(ZooKeeperServer.SESSION_REQUIRE_CLIENT_SASL_AUTH, Boolean.toString(require));
+ RecordingCnxn connection = new RecordingCnxn(session);
+ try (SaslExchange sasl = new SaslExchange(connection, TOKEN)) {
+ byte[] response = sasl.challenge();
+ read(0);
+ sasl.respond(response);
+ assertFalse(sasl.server.isComplete());
+ assertEquals(replyCode.intValue(), connection.lastReply().getErr());
+ assertEquals(2, connection.replies.size());
+ assertEquals(closed, connection.sessionClosed);
+ String log = read(1).get(0);
+ assertBindingEvent(log, "authentication", session == 0 ? null : "0x123",
+ "failure", "-115", "failed", "-33");
+ assertBindings(log, new String[][]{{"sasl", null}});
+ }
+ }
+
+ private void authenticate(RecordingCnxn connection, String scheme, String credentials) throws IOException {
+ server.processPacket(connection, packet(OpCode.auth, AUTH_XID,
+ new AuthPacket(0, scheme, credentials.getBytes(StandardCharsets.UTF_8))));
+ }
+
+ private static ByteBuffer packet(int type, int xid, Record record) throws IOException {
+ ByteArrayOutputStream bytes = new ByteArrayOutputStream();
+ BinaryOutputArchive archive = BinaryOutputArchive.getArchive(bytes);
+ new RequestHeader(xid, type).serialize(archive, "header");
+ record.serialize(archive, "request");
+ return ByteBuffer.wrap(bytes.toByteArray());
+ }
+
+ private List read(int count) {
+ List logs = capture.read(count);
+ captured.addAll(logs);
+ return logs;
+ }
+
+ private void property(String name, String value) {
+ if (!previousProperties.containsKey(name)) {
+ previousProperties.put(name, System.getProperty(name));
+ }
+ AuditHelperTest.restoreProperty(name, value);
+ }
+
+ private static Properties failEnhancedGateRead() {
+ Properties previous = System.getProperties();
+ Properties failing = new Properties() {
+ @Override
+ public String getProperty(String key) {
+ if (AuditHelperTest.ENHANCED_ENABLE.equals(key)) {
+ throw new SecurityException("synthetic audit gate access failure");
+ }
+ return super.getProperty(key);
+ }
+ };
+ failing.putAll(previous);
+ System.setProperties(failing);
+ return previous;
+ }
+
+ private static void assertBindingEvent(String log, String operation, String session, String result,
+ String error, String outcome, String cxid) {
+ Map event = fields(log);
+ assertEquals("2", event.get("schema_version"));
+ assertEquals(operation, event.get("operation"));
+ assertEquals(session, event.get("session"));
+ assertEquals(result, event.get("result"));
+ assertEquals(error, event.get("error_code"));
+ assertEquals(outcome, event.get("outcome"));
+ assertEquals(cxid, event.get("cxid"));
+ assertEquals("127.0.0.1", event.get("ip"));
+ assertNull(event.get("zxid"));
+ assertNull(event.get("data_length"));
+ assertNull(event.get("znode"));
+ }
+
+ private static void assertBindings(String log, String[][] expected) {
+ assertBindings(Collections.singletonList(log), expected);
+ }
+
+ private static void assertBindings(List logs, String[][] expected) {
+ assertEquals(logs.toString(), expected.length, logs.size());
+ Set> actual = new HashSet<>();
+ for (String log : logs) {
+ Map binding = fields(log);
+ assertEquals("2", binding.get("schema_version"));
+ actual.add(Arrays.asList(unescape(binding.get("auth_scheme")), unescape(binding.get("user"))));
+ }
+ assertEquals("Duplicate scheme/principal binding", logs.size(), actual.size());
+ Set> wanted = new HashSet<>();
+ for (String[] binding : expected) {
+ wanted.add(Arrays.asList(binding));
+ }
+ assertEquals(wanted, actual);
+ }
+
+ private static String unescape(String encoded) {
+ if (encoded == null) {
+ return null;
+ }
+ StringBuilder decoded = new StringBuilder();
+ for (int i = 0; i < encoded.length(); i++) {
+ char c = encoded.charAt(i);
+ if (c == '\\') {
+ char escaped = encoded.charAt(++i);
+ switch (escaped) {
+ case '\\': c = '\\'; break;
+ case 't': c = '\t'; break;
+ case 'r': c = '\r'; break;
+ case 'n': c = '\n'; break;
+ default: fail("Unexpected v2 escape: " + escaped);
+ }
+ }
+ decoded.append(c);
+ }
+ return decoded.toString();
+ }
+
+ private static void assertUsers(String log, String... users) {
+ assertEquals(new HashSet<>(Arrays.asList(users)),
+ new HashSet<>(Arrays.asList(fields(log).get("user").split(","))));
+ }
+
+ public static class EmptyAuthenticationProvider extends CredentialAuthenticationProvider {
+ @Override
+ public String getScheme() {
+ return "audit-c2-empty";
+ }
+
+ @Override
+ public Code handleAuthentication(ServerCnxn connection, byte[] data) {
+ return Code.OK;
+ }
+ }
+
+ private static final class Standalone extends ClientBase {
+ private TestableZooKeeper connect() throws IOException, InterruptedException {
+ return createClient();
+ }
+ }
+
+ private static final class RecordingCnxn extends MockServerCnxn {
+ private final List replies = new ArrayList<>();
+ private long sessionId;
+ private boolean sessionClosed;
+ private DisconnectReason disconnectReason;
+ private SetSASLResponse saslResponse;
+ private Runnable authInfoRead;
+
+ private RecordingCnxn(long sessionId) {
+ this.sessionId = sessionId;
+ }
+
+ @Override
+ public long getSessionId() {
+ return sessionId;
+ }
+
+ @Override
+ public InetSocketAddress getRemoteSocketAddress() {
+ return new InetSocketAddress("127.0.0.1", 2181);
+ }
+
+ @Override
+ public List getAuthInfo() {
+ if (authInfoRead != null) {
+ authInfoRead.run();
+ }
+ return super.getAuthInfo();
+ }
+
+ @Override
+ public void sendResponse(ReplyHeader header, Record record, String tag,
+ String cacheKey, Stat stat, int opCode) {
+ replies.add(header);
+ if (record instanceof SetSASLResponse) {
+ saslResponse = (SetSASLResponse) record;
+ }
+ }
+
+ @Override
+ public void sendCloseSession() {
+ sessionClosed = true;
+ }
+
+ @Override
+ public void close(DisconnectReason reason) {
+ disconnectReason = reason;
+ }
+
+ private ReplyHeader lastReply() {
+ assertFalse("Server must produce a protocol reply", replies.isEmpty());
+ return replies.get(replies.size() - 1);
+ }
+
+ private void saslServer(ZooKeeperSaslServer saslServer) {
+ zooKeeperSaslServer = saslServer;
+ }
+ }
+
+ private final class SaslExchange implements AutoCloseable {
+ private final SaslServer server;
+ private final SaslClient client;
+ private final RecordingCnxn connection;
+
+ private SaslExchange(RecordingCnxn connection, String clientPassword) throws Exception {
+ this.connection = connection;
+ Map properties = Collections.singletonMap(Sasl.QOP, "auth");
+ server = Sasl.createSaslServer("DIGEST-MD5", "zookeeper", "localhost",
+ properties, callbacks(SECRET));
+ client = Sasl.createSaslClient(new String[]{"DIGEST-MD5"}, null, "zookeeper", "localhost",
+ properties, callbacks(clientPassword));
+ assertNotNull("JDK must provide DIGEST-MD5", server);
+ assertNotNull("JDK must provide DIGEST-MD5", client);
+ // The package-private wrapper constructor requires a JAAS Login; keep its methods and the JDK engine real.
+ ZooKeeperSaslServer wrapper = mock(ZooKeeperSaslServer.class, CALLS_REAL_METHODS);
+ Field engine = ZooKeeperSaslServer.class.getDeclaredField("saslServer");
+ engine.setAccessible(true);
+ engine.set(wrapper, server);
+ connection.saslServer(wrapper);
+ }
+
+ private byte[] challenge() throws IOException {
+ SessionAuthAuditTest.this.server.processPacket(connection,
+ packet(OpCode.sasl, SASL_XID, new GetSASLRequest(new byte[0])));
+ assertEquals(Code.OK.intValue(), connection.lastReply().getErr());
+ assertNotNull(connection.saslResponse.getToken());
+ return client.evaluateChallenge(connection.saslResponse.getToken());
+ }
+
+ private void respond(byte[] response) throws IOException {
+ SessionAuthAuditTest.this.server.processPacket(connection,
+ packet(OpCode.sasl, SASL_XID, new GetSASLRequest(response)));
+ if (server.isComplete()) {
+ client.evaluateChallenge(connection.saslResponse.getToken());
+ }
+ }
+
+ @Override
+ public void close() throws Exception {
+ client.dispose();
+ server.dispose();
+ }
+ }
+
+ private static CallbackHandler callbacks(String password) {
+ return callbacks -> {
+ for (Callback callback : callbacks) {
+ if (callback instanceof NameCallback) {
+ ((NameCallback) callback).setName("alice");
+ } else if (callback instanceof PasswordCallback) {
+ ((PasswordCallback) callback).setPassword(password.toCharArray());
+ } else if (callback instanceof RealmCallback) {
+ RealmCallback realm = (RealmCallback) callback;
+ realm.setText(realm.getDefaultText());
+ } else if (callback instanceof AuthorizeCallback) {
+ AuthorizeCallback authorization = (AuthorizeCallback) callback;
+ authorization.setAuthorized(
+ authorization.getAuthenticationID().equals(authorization.getAuthorizationID()));
+ } else {
+ throw new UnsupportedCallbackException(callback);
+ }
+ }
+ };
+ }
+}
diff --git a/zookeeper-server/src/test/java/org/apache/zookeeper/audit/Slf4JAuditLoggerTest.java b/zookeeper-server/src/test/java/org/apache/zookeeper/audit/Slf4JAuditLoggerTest.java
index 0f426e073f7..7e04b448627 100644
--- a/zookeeper-server/src/test/java/org/apache/zookeeper/audit/Slf4JAuditLoggerTest.java
+++ b/zookeeper-server/src/test/java/org/apache/zookeeper/audit/Slf4JAuditLoggerTest.java
@@ -280,6 +280,9 @@ public void testEnhancedSystemDeletionIdentityAcrossReplicas() throws Exception
String previous = System.getProperty(AuditHelperTest.ENHANCED_ENABLE);
System.setProperty(AuditHelperTest.ENHANCED_ENABLE, "true");
try (ZooKeeper client = ClientBase.createZKClient("127.0.0.1:" + mt[0].getQuorumPeer().getClientPort())) {
+ Map binding = AuditHelperTest.fields(os.await(1, CONNECTION_TIMEOUT).get(0));
+ assertEquals(AuditConstants.OP_SESSION_ESTABLISHED, binding.get("operation"));
+ assertEquals("ip", binding.get("auth_scheme"));
client.create("/enhanced-ephemeral", new byte[1], ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.EPHEMERAL);
String session = "0x" + Long.toHexString(client.getSessionId());
os.read(1);
diff --git a/zookeeper-server/src/test/java/org/apache/zookeeper/audit/StandaloneServerAuditTest.java b/zookeeper-server/src/test/java/org/apache/zookeeper/audit/StandaloneServerAuditTest.java
index 851cc1e8f0c..344d6e07da3 100644
--- a/zookeeper-server/src/test/java/org/apache/zookeeper/audit/StandaloneServerAuditTest.java
+++ b/zookeeper-server/src/test/java/org/apache/zookeeper/audit/StandaloneServerAuditTest.java
@@ -102,6 +102,7 @@ public void testCreateAuditLog() throws KeeperException, InterruptedException, I
public void testEnhancedWriteResultsMatchClientResponses() throws Exception {
System.setProperty(AuditHelperTest.ENHANCED_ENABLE, "true");
ZooKeeper zk = createClient();
+ assertSessionEstablished();
byte[] data = "audit-secret".getBytes(StandardCharsets.UTF_8);
zk.create("/enhanced", data, ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT);
String log = capture.read(1).get(0);
@@ -141,6 +142,7 @@ public void testEnhancedCreateVariants() throws Exception {
System.setProperty(AuditHelperTest.ENHANCED_ENABLE, "true");
System.setProperty("zookeeper.extendedTypesEnabled", "true");
ZooKeeper zk = createClient();
+ assertSessionEstablished();
Stat stat = new Stat();
zk.create("/create2", null, ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT, stat);
assertWrite(fields(capture.read(1).get(0)), "create", "/create2", "0", "committed", "0");
@@ -174,6 +176,7 @@ public void testEnhancedMultiUsesReturnedPathsAndCompleteIndexes() throws Except
System.setProperty(AuditHelperTest.ENHANCED_ENABLE, "true");
System.setProperty("zookeeper.extendedTypesEnabled", "true");
ZooKeeper zk = createClient();
+ assertSessionEstablished();
List results = zk.multi(Arrays.asList(
Op.create("/same", new byte[1], ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT),
Op.delete("/same", -1),
@@ -201,6 +204,7 @@ public void testEnhancedMultiUsesReturnedPathsAndCompleteIndexes() throws Except
public void testEnhancedFailedMultiMatchesAtomicRollback() throws Exception {
System.setProperty(AuditHelperTest.ENHANCED_ENABLE, "true");
ZooKeeper zk = createClient();
+ assertSessionEstablished();
try {
zk.multi(Arrays.asList(
Op.create("/rolled", new byte[1], ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT),
@@ -227,6 +231,7 @@ public void testEnhancedFailedMultiMatchesAtomicRollback() throws Exception {
public void testFailedCheckRollsBackMutationsWithoutACheckEvent() throws Exception {
System.setProperty(AuditHelperTest.ENHANCED_ENABLE, "true");
ZooKeeper zk = createClient();
+ assertSessionEstablished();
try {
zk.multi(Arrays.asList(
Op.create("/rolled", new byte[1], ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT),
@@ -248,11 +253,15 @@ public void testFailedCheckRollsBackMutationsWithoutACheckEvent() throws Excepti
public void testEnhancedAclRedactionDoesNotChangeStoredAcl() throws Exception {
System.setProperty(AuditHelperTest.ENHANCED_ENABLE, "true");
ZooKeeper zk = createClient();
+ assertSessionEstablished();
String credentials = "alice:synthetic-password";
String digest = DigestAuthenticationProvider.generateDigest(credentials);
zk.addAuthInfo("digest", credentials.getBytes(StandardCharsets.UTF_8));
zk.create("/acl", new byte[0], ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT);
- capture.read(1);
+ List initialLogs = capture.read(2);
+ assertEquals(AuditConstants.OP_AUTHENTICATION, fields(initialLogs.get(0)).get("operation"));
+ assertEquals("digest", fields(initialLogs.get(0)).get("auth_scheme"));
+ assertEquals(AuditConstants.OP_CREATE, fields(initialLogs.get(1)).get("operation"));
List acls = Collections.singletonList(new ACL(ZooDefs.Perms.ALL, new Id("digest", digest)));
zk.setACL("/acl", acls, -1);
String log = capture.read(1).get(0);
@@ -273,9 +282,13 @@ public void testRegisteredCustomUserIsRedactedOnlyInEnhancedMode() throws Except
try {
System.setProperty(AuditHelperTest.ENHANCED_ENABLE, "true");
ZooKeeper zk = createClient();
+ assertSessionEstablished();
zk.addAuthInfo("audit-test-custom", "alice:synthetic-password".getBytes(StandardCharsets.UTF_8));
zk.create("/custom-user", new byte[1], ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT);
- String enhanced = capture.read(1).get(0);
+ List initialLogs = capture.read(2);
+ assertEquals(AuditConstants.OP_AUTHENTICATION, fields(initialLogs.get(0)).get("operation"));
+ assertEquals("audit-test-custom", fields(initialLogs.get(0)).get("auth_scheme"));
+ String enhanced = initialLogs.get(1);
assertWrite(fields(enhanced), "create", "/custom-user", "1", "committed", "0");
assertFalse(enhanced.contains("synthetic-password"));
List enhancedUsers = Arrays.asList(fields(enhanced).get("user").split(","));
@@ -300,6 +313,7 @@ public void testRegisteredCustomUserIsRedactedOnlyInEnhancedMode() throws Except
public void testAuditFailureDoesNotRejectValidWrite() throws Exception {
System.setProperty(AuditHelperTest.ENHANCED_ENABLE, "true");
ZooKeeper zk = createClient();
+ assertSessionEstablished();
AuditLogger failingLogger = event -> {
throw new IllegalStateException("synthetic audit sink failure");
};
@@ -345,4 +359,11 @@ public void testAuditReporterFailureDoesNotRejectAppliedWrite() throws Exception
AuditHelperTest.replaceAuditErrorCounter(previousCounter);
}
}
+
+ private void assertSessionEstablished() throws InterruptedException {
+ Map binding = fields(capture.await(1, CONNECTION_TIMEOUT).get(0));
+ assertEquals(AuditConstants.OP_SESSION_ESTABLISHED, binding.get("operation"));
+ assertEquals("ip", binding.get("auth_scheme"));
+ assertEquals("success", binding.get("result"));
+ }
}
From 59fdc4ae62319ed4a785ffcf86759f301300c7a0 Mon Sep 17 00:00:00 2001
From: Arkesh Mishra <118651144+arkmish@users.noreply.github.com>
Date: Fri, 25 Sep 2026 18:35:47 +0530
Subject: [PATCH 8/9] Add allowlisted metadata-only quota telemetry (B1)
Expose quota_stats through AdminServer with a default-empty exact-path JSON allowlist. Sample existing quota records without subtree traversal or write-path changes, report unavailable data explicitly, and cover validation, concurrent changes and restart behavior.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---
.../main/resources/markdown/zookeeperAdmin.md | 100 ++++++
.../org/apache/zookeeper/server/DataTree.java | 98 ++++++
.../zookeeper/server/admin/Commands.java | 83 +++++
.../apache/zookeeper/server/DataTreeTest.java | 296 +++++++++++++++++
.../zookeeper/server/admin/CommandsTest.java | 302 ++++++++++++++++++
5 files changed, 879 insertions(+)
diff --git a/zookeeper-docs/src/main/resources/markdown/zookeeperAdmin.md b/zookeeper-docs/src/main/resources/markdown/zookeeperAdmin.md
index d015079d7dc..73ba3891938 100644
--- a/zookeeper-docs/src/main/resources/markdown/zookeeperAdmin.md
+++ b/zookeeper-docs/src/main/resources/markdown/zookeeperAdmin.md
@@ -1875,6 +1875,14 @@ options are used to configure the [AdminServer](#sc_adminserver).
The URL for listing and issuing commands relative to the
root URL. Defaults to "/commands".
+* *zookeeper.quotaStats.allowedNamespaces* :
+ (Java system property)
+ A JSON array of exact namespace paths that may be queried by
+ [quota_stats](#sc_quota_stats). Defaults to `[]`, which denies every
+ namespace. Keep this list empty until the namespaces and operational
+ access have been explicitly approved. This is a metadata-disclosure
+ allowlist, not a quota limit or an authentication mechanism.
+
### Metrics Providers
**New in 3.6.0:** The following options are used to configure metrics.
@@ -2326,6 +2334,12 @@ Available commands include:
Reset all observer connection statistics. Companion command to *observers*.
No new fields returned.
+* *quota_stats* :
+ Read only the existing quota usage and limit metadata for one explicitly
+ allowlisted namespace. Requires a `path` parameter. See
+ [bounded quota telemetry](#sc_quota_stats) for configuration, schema,
+ unavailable results and sampling limitations.
+
* *ruok* :
No-op command, check if the server is running.
A response does not necessarily indicate that the
@@ -2391,6 +2405,92 @@ Available commands include:
Peers can be in one of these phases: ELECTION, DISCOVERY, SYNCHRONIZATION, BROADCAST.
Returns fields "voting" and "zabstate".
+
+
+##### Bounded quota telemetry
+
+`quota_stats?path=/example-quota` runs through the existing AdminServer
+command URL, for example `/commands/quota_stats?path=/example-quota`.
+`/example-quota` is illustrative, not an approved namespace. Use the existing
+protected administrative access path; this command adds no authentication,
+ACL policy, transport listener or four-letter command.
+
+The JVM property `zookeeper.quotaStats.allowedNamespaces` must contain one
+complete JSON array of strings, for example `["/example-quota"]`. An unset
+property means `[]` and rejects all requests. The command reads the property
+once per request and validates the entire array before reading quota metadata;
+an invalid later entry cannot be hidden by an earlier match. Non-array JSON,
+non-string entries, malformed JSON, trailing JSON values, invalid paths and
+unreadable configuration produce a normal command error rather than permissive
+fallback. Duplicate valid entries have no additional effect.
+
+Both configured and requested paths must pass ZooKeeper's existing znode path
+validation. Root `/`, `/zookeeper` and its descendants are excluded;
+`/zookeeper-client` is not excluded by that reserved-path rule. Paths are
+matched exactly, with no trimming, normalization, ancestor inheritance,
+prefix matching or wildcard expansion. A literal `*` in a valid znode name is
+just a character, not a pattern. A child namespace needs its own explicit
+allowlist entry and its own quota metadata. Encode the `path` query parameter
+normally when its characters require URL encoding.
+
+Invalid/missing paths, invalid configuration and disallowed paths return only
+the usual `command` and non-null `error` fields. An uninitialized or stopped
+server is rejected by the existing command-framework availability guard.
+Successful requests, including unavailable quota samples, have this schema:
+
+| Field | Type | Meaning |
+| --- | --- | --- |
+| `command` | string | `quota_stats` |
+| `error` | null | Request/configuration validation succeeded |
+| `schema_version` | integer | `1` |
+| `path` | string | The exact requested, allowlisted namespace |
+| `count_used` | integer or null | Existing quota-stat node count, including the namespace itself |
+| `bytes_used` | integer or null | Existing quota-stat total znode payload bytes, not path, ACL, packet or JVM-memory bytes |
+| `count_limit` | integer or null | Existing quota count limit; native `-1` is reported as null |
+| `bytes_limit` | integer or null | Existing quota byte limit; native `-1` is reported as null |
+| `available` | boolean | Both exact metadata records are present and valid in this sample |
+| `reason` | string or null | Unavailability reason below, or null when available |
+
+Counts use the native signed 32-bit integer range; byte values use signed
+64-bit integers. Metadata must use the native `count=,bytes=`
+format with exact field names/order and decimal digits. Usage must be
+non-negative; limits must be at least `-1`. Null/empty payloads, malformed
+records and numeric overflow are invalid, not zero usage. An available
+sample can have one or both limits null (unset/unlimited in native quota
+metadata); null is not approved infinite headroom. Zero limits remain zero.
+The command computes no utilization ratios or headroom, and clients must
+not divide by zero or treat unknown values as an approved budget.
+
+An unavailable sample sets **all four numeric fields to null** and uses one
+of these reasons:
+
+| `reason` | Meaning |
+| --- | --- |
+| `namespace_missing` | The namespace does not exist when sampling starts, even if orphan quota metadata remains |
+| `quota_missing` | Both exact quota metadata nodes are absent; an ancestor's quota is not substituted |
+| `quota_incomplete` | Exactly one of the stat/limit nodes is absent |
+| `invalid_quota_stats` | The stat record is null, malformed, negative or out of range |
+| `invalid_quota_limits` | The limit record is null, malformed or out of range |
+| `quota_changed` | Removal/replacement of the namespace or a sampled metadata node was detected during sampling |
+
+The only payloads read are the exact
+`/zookeeper/quota/zookeeper_stats` and
+`/zookeeper/quota/zookeeper_limits` records. Namespace existence and
+node-identity checks use a bounded number of direct lookups. The command
+does not walk the live subtree, search ancestor quotas, create watches,
+repair accounting, alter enforcement or register per-path metrics.
+
+**This is not an atomic or historical snapshot.** Stat and limit records are
+sampled independently with no tree-wide lock; they can reflect different
+instants and existing quota-accounting lag. Identity rechecks detect some
+removals/replacements, not every concurrent change. `available=true` is a
+metadata-availability statement, not a freshness, consistency or safety
+guarantee. Missing or invalid data must remain unknown.
+
+Continue using `monitor/mntr` for existing response-size, session/connection,
+latency and queue metrics, and `watch_summary/wchs` for watch totals.
+`quota_stats` does not duplicate those metrics or introduce namespace labels.
+
diff --git a/zookeeper-server/src/main/java/org/apache/zookeeper/server/DataTree.java b/zookeeper-server/src/main/java/org/apache/zookeeper/server/DataTree.java
index 80d0173afe6..82afce2f567 100644
--- a/zookeeper-server/src/main/java/org/apache/zookeeper/server/DataTree.java
+++ b/zookeeper-server/src/main/java/org/apache/zookeeper/server/DataTree.java
@@ -23,6 +23,7 @@
import java.io.PrintWriter;
import java.io.Serializable;
import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
import java.util.ArrayList;
import java.util.Collection;
import java.util.Collections;
@@ -37,6 +38,7 @@
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.atomic.AtomicInteger;
import java.util.concurrent.atomic.AtomicLong;
+import java.util.regex.Pattern;
import org.apache.jute.BinaryInputArchive;
import org.apache.jute.BinaryOutputArchive;
import org.apache.jute.InputArchive;
@@ -107,6 +109,8 @@ public class DataTree {
private static final Logger LOG = LoggerFactory.getLogger(DataTree.class);
+ private static final Pattern QUOTA_VALUE = Pattern.compile("count=-?[0-9]+,bytes=-?[0-9]+");
+
private final RateLogger RATE_LOGGER = new RateLogger(LOG, 15 * 60 * 1000);
/**
@@ -761,6 +765,100 @@ public String getMaxPrefixWithQuota(String path) {
}
}
+ /**
+ * Samples only the exact namespace's existing quota metadata, without traversing
+ * or updating the tree. The two metadata reads are not an atomic snapshot.
+ *
+ * @param path a validated absolute, non-root, non-reserved namespace path
+ */
+ public QuotaStats getQuotaStats(String path) {
+ DataNode namespace = getNode(path);
+ if (namespace == null) {
+ return new QuotaStats(null, null, "namespace_missing");
+ }
+ String statPath = Quotas.statPath(path);
+ String limitPath = Quotas.quotaPath(path);
+ DataNode statNode = getNode(statPath);
+ DataNode limitNode = getNode(limitPath);
+ if (statNode == null && limitNode == null) {
+ return new QuotaStats(null, null, "quota_missing");
+ }
+ if (statNode == null || limitNode == null) {
+ return new QuotaStats(null, null, "quota_incomplete");
+ }
+ StatsTrack usage = readQuotaMetadata(statNode, 0);
+ StatsTrack limits = readQuotaMetadata(limitNode, -1);
+ if (getNode(path) != namespace || getNode(statPath) != statNode || getNode(limitPath) != limitNode) {
+ return new QuotaStats(null, null, "quota_changed");
+ }
+ if (usage == null) {
+ return new QuotaStats(null, null, "invalid_quota_stats");
+ }
+ if (limits == null) {
+ return new QuotaStats(null, null, "invalid_quota_limits");
+ }
+ return new QuotaStats(usage, limits, null);
+ }
+
+ private static StatsTrack readQuotaMetadata(DataNode node, int minimum) {
+ byte[] data = node.getData();
+ if (data == null) {
+ return null;
+ }
+ String value = new String(data, StandardCharsets.UTF_8);
+ // StatsTrack accepts arbitrary field names; validate the metadata format first.
+ if (!QUOTA_VALUE.matcher(value).matches()) {
+ return null;
+ }
+ try {
+ StatsTrack stats = new StatsTrack(value);
+ return stats.getCount() < minimum || stats.getBytes() < minimum ? null : stats;
+ } catch (NumberFormatException e) {
+ return null;
+ }
+ }
+
+ /**
+ * A quota metadata sample; unavailable samples expose no numeric values.
+ */
+ public static final class QuotaStats {
+
+ private final StatsTrack usage;
+ private final StatsTrack limits;
+ private final String reason;
+
+ private QuotaStats(StatsTrack usage, StatsTrack limits, String reason) {
+ this.usage = usage;
+ this.limits = limits;
+ this.reason = reason;
+ }
+
+ public Integer getCountUsed() {
+ return usage == null ? null : usage.getCount();
+ }
+
+ public Long getBytesUsed() {
+ return usage == null ? null : usage.getBytes();
+ }
+
+ public Integer getCountLimit() {
+ return limits == null || limits.getCount() == -1 ? null : limits.getCount();
+ }
+
+ public Long getBytesLimit() {
+ return limits == null || limits.getBytes() == -1 ? null : limits.getBytes();
+ }
+
+ public boolean isAvailable() {
+ return reason == null;
+ }
+
+ public String getReason() {
+ return reason;
+ }
+
+ }
+
public void addWatch(String basePath, Watcher watcher, int mode) {
WatcherMode watcherMode = WatcherMode.fromZooDef(mode);
dataWatches.addWatch(basePath, watcher, watcherMode);
diff --git a/zookeeper-server/src/main/java/org/apache/zookeeper/server/admin/Commands.java b/zookeeper-server/src/main/java/org/apache/zookeeper/server/admin/Commands.java
index 1476e5049b7..dc363b96676 100644
--- a/zookeeper-server/src/main/java/org/apache/zookeeper/server/admin/Commands.java
+++ b/zookeeper-server/src/main/java/org/apache/zookeeper/server/admin/Commands.java
@@ -19,7 +19,11 @@
package org.apache.zookeeper.server.admin;
import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.core.JsonFactory;
+import com.fasterxml.jackson.core.JsonParser;
+import com.fasterxml.jackson.core.JsonToken;
import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
+import java.io.IOException;
import java.net.InetSocketAddress;
import java.util.Arrays;
import java.util.Collections;
@@ -34,8 +38,11 @@
import java.util.stream.Collectors;
import org.apache.zookeeper.Environment;
import org.apache.zookeeper.Environment.Entry;
+import org.apache.zookeeper.Quotas;
import org.apache.zookeeper.Version;
+import org.apache.zookeeper.common.PathUtils;
import org.apache.zookeeper.server.DataTree;
+import org.apache.zookeeper.server.DataTree.QuotaStats;
import org.apache.zookeeper.server.ServerCnxnFactory;
import org.apache.zookeeper.server.ServerMetrics;
import org.apache.zookeeper.server.ZooKeeperServer;
@@ -142,6 +149,7 @@ public static Command getCommand(String cmdName) {
registerCommand(new LeaderCommand());
registerCommand(new MonitorCommand());
registerCommand(new ObserverCnxnStatResetCommand());
+ registerCommand(new QuotaStatsCommand());
registerCommand(new RuokCommand());
registerCommand(new SetTraceMaskCommand());
registerCommand(new SrvrCommand());
@@ -480,6 +488,81 @@ public CommandResponse run(ZooKeeperServer zkServer, Map kwargs)
}
+ /**
+ * Samples the existing quota metadata for one explicitly allowlisted namespace.
+ */
+ public static class QuotaStatsCommand extends CommandBase {
+
+ private static final String ALLOWED_NAMESPACES = "zookeeper.quotaStats.allowedNamespaces";
+ private static final JsonFactory JSON = new JsonFactory();
+
+ public QuotaStatsCommand() {
+ super(Collections.singletonList("quota_stats"));
+ }
+
+ @Override
+ public CommandResponse run(ZooKeeperServer zkServer, Map kwargs) {
+ String path = kwargs == null ? null : kwargs.get("path");
+ if (!isValidNamespacePath(path)) {
+ return new CommandResponse(getPrimaryName(), "quota_stats requires a valid namespace path");
+ }
+ final boolean allowed;
+ try {
+ allowed = isAllowed(path, System.getProperty(ALLOWED_NAMESPACES, "[]"));
+ } catch (IOException | IllegalArgumentException | SecurityException e) {
+ return new CommandResponse(getPrimaryName(), "Invalid " + ALLOWED_NAMESPACES
+ + ": expected a JSON array of valid namespace paths");
+ }
+ if (!allowed) {
+ return new CommandResponse(getPrimaryName(), "Path is not allowlisted for quota_stats");
+ }
+
+ QuotaStats stats = zkServer.getZKDatabase().getDataTree().getQuotaStats(path);
+ CommandResponse response = initializeResponse();
+ response.put("schema_version", 1);
+ response.put("path", path);
+ response.put("count_used", stats.getCountUsed());
+ response.put("bytes_used", stats.getBytesUsed());
+ response.put("count_limit", stats.getCountLimit());
+ response.put("bytes_limit", stats.getBytesLimit());
+ response.put("available", stats.isAvailable());
+ response.put("reason", stats.getReason());
+ return response;
+ }
+
+ private static boolean isValidNamespacePath(String path) {
+ try {
+ PathUtils.validatePath(path);
+ } catch (IllegalArgumentException e) {
+ return false;
+ }
+ return !"/".equals(path)
+ && !Quotas.procZookeeper.equals(path)
+ && !path.startsWith(Quotas.procZookeeper + "/");
+ }
+
+ private static boolean isAllowed(String path, String configuration) throws IOException {
+ boolean allowed = false;
+ try (JsonParser parser = JSON.createParser(configuration)) {
+ if (parser.nextToken() != JsonToken.START_ARRAY) {
+ throw new IllegalArgumentException();
+ }
+ JsonToken token;
+ while ((token = parser.nextToken()) != JsonToken.END_ARRAY) {
+ if (token != JsonToken.VALUE_STRING || !isValidNamespacePath(parser.getText())) {
+ throw new IllegalArgumentException();
+ }
+ allowed |= path.equals(parser.getText());
+ }
+ if (parser.nextToken() != null) {
+ throw new IllegalArgumentException();
+ }
+ }
+ return allowed;
+ }
+
+ }
+
/**
* No-op command, check if the server is running
*/
diff --git a/zookeeper-server/src/test/java/org/apache/zookeeper/server/DataTreeTest.java b/zookeeper-server/src/test/java/org/apache/zookeeper/server/DataTreeTest.java
index 2aad3b79e8c..5892e81bd8d 100644
--- a/zookeeper-server/src/test/java/org/apache/zookeeper/server/DataTreeTest.java
+++ b/zookeeper-server/src/test/java/org/apache/zookeeper/server/DataTreeTest.java
@@ -18,6 +18,7 @@
package org.apache.zookeeper.server;
+import static org.junit.Assert.assertArrayEquals;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertFalse;
import static org.junit.Assert.assertNotEquals;
@@ -32,10 +33,16 @@
import java.io.PrintWriter;
import java.io.StringWriter;
import java.lang.reflect.Field;
+import java.nio.charset.StandardCharsets;
import java.util.ArrayList;
+import java.util.Arrays;
import java.util.List;
import java.util.Map;
import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
import java.util.concurrent.Semaphore;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicBoolean;
@@ -262,6 +269,295 @@ public void testPathTrieClearOnDeserialize() throws Exception {
assertEquals("/bug is still in pTrie", "/", pTrie.findMaxPrefix("/bug"));
}
+ @Test
+ public void testQuotaStatsRejectsMalformedUsage() throws Exception {
+ DataTree tree = new DataTree();
+ createQuotaTree(tree);
+ String[] values = {
+ "other=1,bytes=3", "count=1,other=3", "bytes=3,count=1", "count=1",
+ "count=1,bytes=3,", "count=1,bytes=3,extra=4", "count=1=2,bytes=3",
+ "count=+1,bytes=3", "count=1,bytes=+3", "count=1,bytes=3\n",
+ " count=1,bytes=3", "count=1,bytes=3.0", "count=\u0661,bytes=3", "sensitive-token"
+ };
+ for (String value : values) {
+ setQuotaData(tree, Quotas.statPath("/quota-test"), value);
+ assertQuotaUnavailable(sampleQuota(tree), "invalid_quota_stats");
+ }
+ tree.setData(Quotas.statPath("/quota-test"), new byte[]{(byte) 0xc3, (byte) 0x28}, 1, 2, 2);
+ assertQuotaUnavailable(sampleQuota(tree), "invalid_quota_stats");
+ }
+
+ @Test
+ public void testQuotaStatsRejectsMalformedLimits() throws Exception {
+ DataTree tree = new DataTree();
+ createQuotaTree(tree);
+ for (String value : Arrays.asList("other=10,bytes=100", "bytes=100,count=10",
+ "count=10,bytes=", "count=10,bytes=100,",
+ "count=10,bytes=100=sensitive-token", "sensitive-token")) {
+ setQuotaData(tree, Quotas.quotaPath("/quota-test"), value);
+ assertQuotaUnavailable(sampleQuota(tree), "invalid_quota_limits");
+ }
+ }
+
+ @Test
+ public void testQuotaStatsRejectsNullOrEmptyMetadata() throws Exception {
+ DataTree tree = new DataTree();
+ createQuotaTree(tree);
+ for (String value : Arrays.asList(null, "")) {
+ setQuotaData(tree, Quotas.statPath("/quota-test"), value);
+ assertQuotaUnavailable(sampleQuota(tree), "invalid_quota_stats");
+ }
+ setQuotaData(tree, Quotas.statPath("/quota-test"), "count=1,bytes=3");
+ for (String value : Arrays.asList(null, "")) {
+ setQuotaData(tree, Quotas.quotaPath("/quota-test"), value);
+ assertQuotaUnavailable(sampleQuota(tree), "invalid_quota_limits");
+ }
+ }
+
+ @Test
+ public void testQuotaStatsRejectsOutOfRangeUsage() throws Exception {
+ DataTree tree = new DataTree();
+ createQuotaTree(tree);
+ for (String value : Arrays.asList("count=-1,bytes=3", "count=1,bytes=-1",
+ "count=-2,bytes=3", "count=2147483648,bytes=3",
+ "count=1,bytes=9223372036854775808")) {
+ setQuotaData(tree, Quotas.statPath("/quota-test"), value);
+ assertQuotaUnavailable(sampleQuota(tree), "invalid_quota_stats");
+ }
+ }
+
+ @Test
+ public void testQuotaStatsRejectsOutOfRangeLimits() throws Exception {
+ DataTree tree = new DataTree();
+ createQuotaTree(tree);
+ for (String value : Arrays.asList("count=-2,bytes=100", "count=10,bytes=-2",
+ "count=2147483648,bytes=100",
+ "count=10,bytes=9223372036854775808")) {
+ setQuotaData(tree, Quotas.quotaPath("/quota-test"), value);
+ assertQuotaUnavailable(sampleQuota(tree), "invalid_quota_limits");
+ }
+ }
+
+ @Test
+ public void testQuotaStatsIncompleteMetadata() throws Exception {
+ DataTree tree = new DataTree();
+ createQuotaTree(tree);
+ tree.deleteNode(Quotas.statPath("/quota-test"), 2);
+ assertQuotaUnavailable(sampleQuota(tree), "quota_incomplete");
+ tree.createNode(Quotas.statPath("/quota-test"), new byte[0], null, 0, -1, 3, 3);
+ tree.deleteNode(Quotas.quotaPath("/quota-test"), 4);
+ assertQuotaUnavailable(sampleQuota(tree), "quota_incomplete");
+ tree.deleteNode(Quotas.statPath("/quota-test"), 5);
+ assertQuotaUnavailable(sampleQuota(tree), "quota_missing");
+ }
+
+ @Test
+ public void testQuotaStatsNamespaceRemoval() throws Exception {
+ DataTree tree = new DataTree();
+ createQuotaTree(tree);
+ tree.deleteNode("/quota-test", 2);
+ assertNotNull(tree.getNode(Quotas.statPath("/quota-test")));
+ assertNotNull(tree.getNode(Quotas.quotaPath("/quota-test")));
+ assertQuotaUnavailable(sampleQuota(tree), "namespace_missing");
+ tree.createNode("/quota-test", new byte[5], null, 0, -1, 3, 3);
+ DataTree.QuotaStats sample = sampleQuota(tree);
+ assertTrue(sample.isAvailable());
+ assertEquals(Integer.valueOf(1), sample.getCountUsed());
+ assertEquals(Long.valueOf(5), sample.getBytesUsed());
+ }
+
+ @Test
+ public void testQuotaStatsDetectsRemovalDuringSample() throws Exception {
+ for (String path : Arrays.asList("/quota-test", Quotas.statPath("/quota-test"), Quotas.quotaPath("/quota-test"))) {
+ ChangingQuotaTree tree = new ChangingQuotaTree();
+ createQuotaTree(tree);
+ tree.changePath = path;
+ assertQuotaUnavailable(sampleQuota(tree), "quota_changed");
+ }
+ }
+
+ @Test
+ public void testQuotaStatsDetectsReplacementDuringSample() throws Exception {
+ for (String path : Arrays.asList("/quota-test", Quotas.statPath("/quota-test"), Quotas.quotaPath("/quota-test"))) {
+ ChangingQuotaTree tree = new ChangingQuotaTree();
+ createQuotaTree(tree);
+ tree.changePath = path;
+ tree.replace = true;
+ assertQuotaUnavailable(sampleQuota(tree), "quota_changed");
+ }
+ }
+
+ @Test
+ public void testQuotaStatsDoesNotTraverseMissingQuota() throws Exception {
+ BoundedQuotaTree tree = new BoundedQuotaTree();
+ tree.createNode("/quota-test", new byte[3], null, 0, -1, 1, 1);
+ for (int i = 0; i < 100; i++) {
+ tree.createNode("/quota-test/child" + i, new byte[4], null, 0, -1, 1, 1);
+ }
+ tree.sampling = true;
+ assertQuotaUnavailable(sampleQuota(tree), "quota_missing");
+ assertTrue(tree.lookups <= 6);
+ }
+
+ @Test
+ public void testQuotaStatsSamplesMetadataWithoutRecounting() throws Exception {
+ BoundedQuotaTree tree = new BoundedQuotaTree();
+ createQuotaTree(tree);
+ setQuotaData(tree, Quotas.statPath("/quota-test"), "count=19,bytes=999");
+ byte[] data = tree.getNode(Quotas.statPath("/quota-test")).getData().clone();
+ long digest = tree.getTreeDigest();
+ long size = tree.cachedApproximateDataSize();
+ int watches = tree.getWatchCount();
+ Map metrics = MetricsUtils.currentServerMetrics();
+ tree.sampling = true;
+ DataTree.QuotaStats sample = sampleQuota(tree);
+ tree.sampling = false;
+
+ assertTrue(sample.isAvailable());
+ assertEquals(Integer.valueOf(19), sample.getCountUsed());
+ assertEquals(Long.valueOf(999), sample.getBytesUsed());
+ assertTrue(tree.lookups <= 6);
+ assertArrayEquals(data, tree.getNode(Quotas.statPath("/quota-test")).getData());
+ assertEquals(digest, tree.getTreeDigest());
+ assertEquals(size, tree.cachedApproximateDataSize());
+ assertEquals(watches, tree.getWatchCount());
+ assertEquals(metrics, MetricsUtils.currentServerMetrics());
+ setQuotaData(tree, Quotas.statPath("/quota-test"), "count=2,bytes=7");
+ assertEquals(Integer.valueOf(19), sample.getCountUsed());
+ assertEquals(Long.valueOf(999), sample.getBytesUsed());
+ }
+
+ @Test
+ public void testQuotaStatsNumericBoundaries() throws Exception {
+ DataTree tree = new DataTree();
+ createQuotaTree(tree);
+ setQuotaData(tree, Quotas.statPath("/quota-test"), "count=2147483647,bytes=9223372036854775807");
+ setQuotaData(tree, Quotas.quotaPath("/quota-test"), "count=2147483647,bytes=9223372036854775807");
+ DataTree.QuotaStats sample = sampleQuota(tree);
+ assertTrue(sample.isAvailable());
+ assertEquals(Integer.valueOf(Integer.MAX_VALUE), sample.getCountUsed());
+ assertEquals(Long.valueOf(Long.MAX_VALUE), sample.getBytesUsed());
+ assertEquals(Integer.valueOf(Integer.MAX_VALUE), sample.getCountLimit());
+ assertEquals(Long.valueOf(Long.MAX_VALUE), sample.getBytesLimit());
+ setQuotaData(tree, Quotas.statPath("/quota-test"), "count=000,bytes=000");
+ sample = sampleQuota(tree);
+ assertTrue(sample.isAvailable());
+ assertEquals(Integer.valueOf(0), sample.getCountUsed());
+ assertEquals(Long.valueOf(0), sample.getBytesUsed());
+ }
+
+ @Test(timeout = 30000)
+ public void testQuotaStatsConcurrentMetadataUpdates() throws Exception {
+ DataTree tree = new DataTree();
+ createQuotaTree(tree);
+ setQuotaData(tree, Quotas.statPath("/quota-test"), "count=2,bytes=20");
+ setQuotaData(tree, Quotas.quotaPath("/quota-test"), "count=5,bytes=50");
+ CountDownLatch start = new CountDownLatch(1);
+ ExecutorService writer = Executors.newSingleThreadExecutor();
+ try {
+ Future updates = writer.submit(() -> {
+ start.await();
+ for (int i = 0; i < 1000; i++) {
+ setQuotaData(tree, Quotas.statPath("/quota-test"), i % 2 == 0 ? "count=3,bytes=30" : "count=2,bytes=20");
+ setQuotaData(tree, Quotas.quotaPath("/quota-test"), i % 2 == 0 ? "count=7,bytes=70" : "count=5,bytes=50");
+ }
+ return null;
+ });
+ start.countDown();
+ for (int i = 0; i < 1000; i++) {
+ DataTree.QuotaStats sample = sampleQuota(tree);
+ assertTrue(sample.isAvailable());
+ assertTrue((sample.getCountUsed() == 2 && sample.getBytesUsed() == 20)
+ || (sample.getCountUsed() == 3 && sample.getBytesUsed() == 30));
+ assertTrue((sample.getCountLimit() == 5 && sample.getBytesLimit() == 50)
+ || (sample.getCountLimit() == 7 && sample.getBytesLimit() == 70));
+ }
+ updates.get(10, TimeUnit.SECONDS);
+ } finally {
+ writer.shutdownNow();
+ assertTrue(writer.awaitTermination(10, TimeUnit.SECONDS));
+ }
+ }
+
+ private void createQuotaTree(DataTree tree) throws Exception {
+ tree.createNode("/quota-test", new byte[3], null, 0, -1, 1, 1);
+ tree.createNode(Quotas.quotaZookeeper + "/quota-test", null, null, 0, -1, 1, 1);
+ tree.createNode(Quotas.quotaPath("/quota-test"), "count=10,bytes=100".getBytes(StandardCharsets.UTF_8),
+ null, 0, -1, 1, 1);
+ tree.createNode(Quotas.statPath("/quota-test"), new byte[0], null, 0, -1, 1, 1);
+ }
+
+ private void setQuotaData(DataTree tree, String path, String value) throws NoNodeException {
+ tree.setData(path, value == null ? null : value.getBytes(StandardCharsets.UTF_8), 1, 2, 2);
+ }
+
+ private DataTree.QuotaStats sampleQuota(DataTree tree) {
+ try {
+ return tree.getQuotaStats("/quota-test");
+ } catch (RuntimeException e) {
+ throw new AssertionError("Quota telemetry must report unavailable, not throw " + e.getClass().getSimpleName());
+ }
+ }
+
+ private void assertQuotaUnavailable(DataTree.QuotaStats sample, String reason) {
+ assertFalse(sample.isAvailable());
+ assertEquals(reason, sample.getReason());
+ assertNull(sample.getCountUsed());
+ assertNull(sample.getBytesUsed());
+ assertNull(sample.getCountLimit());
+ assertNull(sample.getBytesLimit());
+ }
+
+ private static class BoundedQuotaTree extends DataTree {
+
+ private boolean sampling;
+ private int lookups;
+
+ @Override
+ public DataNode getNode(String path) {
+ if (sampling) {
+ assertTrue("Unexpected subtree lookup", "/quota-test".equals(path)
+ || Quotas.statPath("/quota-test").equals(path)
+ || Quotas.quotaPath("/quota-test").equals(path));
+ assertTrue("Quota telemetry must perform bounded lookups", ++lookups <= 6);
+ }
+ return super.getNode(path);
+ }
+
+ @Override
+ public String getMaxPrefixWithQuota(String path) {
+ assertFalse("Quota telemetry must not search ancestor quotas", sampling);
+ return super.getMaxPrefixWithQuota(path);
+ }
+
+ }
+
+ private static class ChangingQuotaTree extends DataTree {
+
+ private String changePath;
+ private boolean replace;
+
+ @Override
+ public DataNode getNode(String path) {
+ DataNode node = super.getNode(path);
+ if (changePath != null && Quotas.quotaPath("/quota-test").equals(path)) {
+ String changed = changePath;
+ changePath = null;
+ byte[] data = super.getNode(changed).getData();
+ try {
+ super.deleteNode(changed, 2);
+ if (replace) {
+ super.createNode(changed, data, null, 0, -1, 3, 3);
+ }
+ } catch (NoNodeException | NodeExistsException e) {
+ throw new AssertionError(e);
+ }
+ }
+ return node;
+ }
+
+ }
+
/* ZOOKEEPER-3531 - org.apache.zookeeper.server.DataTree#serialize calls the aclCache.serialize when doing
* dataree serialization, however, org.apache.zookeeper.server.ReferenceCountedACLCache#serialize
diff --git a/zookeeper-server/src/test/java/org/apache/zookeeper/server/admin/CommandsTest.java b/zookeeper-server/src/test/java/org/apache/zookeeper/server/admin/CommandsTest.java
index eb8c300f61a..11e273d159f 100644
--- a/zookeeper-server/src/test/java/org/apache/zookeeper/server/admin/CommandsTest.java
+++ b/zookeeper-server/src/test/java/org/apache/zookeeper/server/admin/CommandsTest.java
@@ -20,16 +20,31 @@
import static org.hamcrest.core.Is.is;
import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertNotNull;
import static org.junit.Assert.assertNull;
import static org.junit.Assert.assertThat;
import static org.junit.Assert.assertTrue;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.when;
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
import java.io.IOException;
+import java.io.PrintWriter;
+import java.io.StringWriter;
+import java.nio.charset.StandardCharsets;
+import java.security.Permission;
import java.util.ArrayList;
import java.util.Arrays;
+import java.util.Collections;
import java.util.HashMap;
+import java.util.HashSet;
import java.util.Map;
+import org.apache.zookeeper.CreateMode;
+import org.apache.zookeeper.Quotas;
+import org.apache.zookeeper.ZooDefs.Ids;
+import org.apache.zookeeper.ZooKeeper;
+import org.apache.zookeeper.cli.SetQuotaCommand;
import org.apache.zookeeper.metrics.MetricsUtils;
import org.apache.zookeeper.server.ServerCnxnFactory;
import org.apache.zookeeper.server.ServerStats;
@@ -37,10 +52,30 @@
import org.apache.zookeeper.server.ZooKeeperServer;
import org.apache.zookeeper.server.quorum.BufferStats;
import org.apache.zookeeper.test.ClientBase;
+import org.junit.After;
+import org.junit.Before;
import org.junit.Test;
public class CommandsTest extends ClientBase {
+ private static final String QUOTA_ALLOWLIST = "zookeeper.quotaStats.allowedNamespaces";
+ private String originalQuotaAllowlist;
+
+ @Before
+ public void saveQuotaAllowlist() {
+ originalQuotaAllowlist = System.getProperty(QUOTA_ALLOWLIST);
+ System.clearProperty(QUOTA_ALLOWLIST);
+ }
+
+ @After
+ public void restoreQuotaAllowlist() {
+ if (originalQuotaAllowlist == null) {
+ System.clearProperty(QUOTA_ALLOWLIST);
+ } else {
+ System.setProperty(QUOTA_ALLOWLIST, originalQuotaAllowlist);
+ }
+ }
+
/**
* Checks that running a given Command returns the expected Map. Asserts
* that all specified keys are present with values of the specified types
@@ -259,6 +294,273 @@ public void testVotingViewCommand() throws IOException, InterruptedException {
new Field("current_config", Map.class));
}
+ @Test
+ public void testQuotaStatsExactMetadata() throws Exception {
+ createQuotaFixture();
+ System.setProperty(QUOTA_ALLOWLIST, "[\"/quota-test\"]");
+
+ CommandResponse response = quotaStats("/quota-test");
+ assertQuotaResponse(response, "/quota-test", 2, 7L, 10, 100L, null);
+ StringWriter output = new StringWriter();
+ new JsonOutputter().output(response, new PrintWriter(output));
+ JsonNode json = new ObjectMapper().readTree(output.toString());
+ assertEquals(1, json.get("schema_version").intValue());
+ assertEquals(2, json.get("count_used").intValue());
+ assertEquals(7L, json.get("bytes_used").longValue());
+ assertTrue(json.get("available").booleanValue());
+ assertTrue(json.get("reason").isNull());
+ assertTrue(json.get("error").isNull());
+ }
+
+ @Test
+ public void testQuotaStatsUnlimitedAndZeroLimits() throws Exception {
+ ZooKeeper zk = createQuotaFixture();
+ System.setProperty(QUOTA_ALLOWLIST, "[\"/quota-test\"]");
+ String[] limits = {
+ "count=-1,bytes=100", "count=10,bytes=-1", "count=-1,bytes=-1", "count=0,bytes=0"
+ };
+ Integer[] counts = {null, 10, null, 0};
+ Long[] bytes = {100L, null, null, 0L};
+ for (int i = 0; i < limits.length; i++) {
+ zk.setData(Quotas.quotaPath("/quota-test"), limits[i].getBytes(StandardCharsets.UTF_8), -1);
+ CommandResponse response = quotaStats("/quota-test");
+ assertQuotaResponse(response, "/quota-test", 2, 7L, counts[i], bytes[i], null);
+ StringWriter output = new StringWriter();
+ new JsonOutputter().output(response, new PrintWriter(output));
+ JsonNode json = new ObjectMapper().readTree(output.toString());
+ assertEquals(counts[i] == null, json.get("count_limit").isNull());
+ assertEquals(bytes[i] == null, json.get("bytes_limit").isNull());
+ }
+ }
+
+ @Test
+ public void testQuotaStatsMissingMetadata() throws Exception {
+ ZooKeeper zk = createClient();
+ zk.create("/quota-test", new byte[0], Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT);
+ System.setProperty(QUOTA_ALLOWLIST, "[\"/quota-test\"]");
+ assertQuotaResponse(quotaStats("/quota-test"), "/quota-test", null, null, null, null, "quota_missing");
+ }
+
+ @Test
+ public void testQuotaStatsMissingNamespace() {
+ System.setProperty(QUOTA_ALLOWLIST, "[\"/quota-test\"]");
+ assertQuotaResponse(quotaStats("/quota-test"), "/quota-test", null, null, null, null, "namespace_missing");
+ }
+
+ @Test
+ public void testQuotaStatsDoesNotUseAncestorQuota() throws Exception {
+ createQuotaFixture();
+ System.setProperty(QUOTA_ALLOWLIST, "[\"/quota-test/child\"]");
+ assertQuotaResponse(quotaStats("/quota-test/child"), "/quota-test/child",
+ null, null, null, null, "quota_missing");
+ }
+
+ @Test
+ public void testQuotaStatsDefaultAndEmptyAllowlist() {
+ assertQuotaError(quotaStats("/quota-test"), "not allowlisted");
+ System.setProperty(QUOTA_ALLOWLIST, "[]");
+ assertQuotaError(quotaStats("/quota-test"), "not allowlisted");
+ }
+
+ @Test
+ public void testQuotaStatsExactAllowlist() {
+ System.setProperty(QUOTA_ALLOWLIST, "[\"/quota-test\"]");
+ for (String path : Arrays.asList("/quota-test/child", "/quota-test-other", "/Quota-test")) {
+ assertQuotaError(quotaStats(path), "not allowlisted");
+ }
+ assertQuotaResponse(quotaStats("/quota-test"), "/quota-test", null, null, null, null, "namespace_missing");
+
+ System.setProperty(QUOTA_ALLOWLIST, "[\"/quota-test/*\"]");
+ assertQuotaError(quotaStats("/quota-test/child"), "not allowlisted");
+ assertQuotaResponse(quotaStats("/quota-test/*"), "/quota-test/*",
+ null, null, null, null, "namespace_missing");
+ }
+
+ @Test
+ public void testQuotaStatsRejectsMalformedAllowlist() {
+ String[] configurations = {
+ "", "null", "{}", "\"/quota-test\"", "[1]", "[null]", "[true]", "[[]]",
+ "[\"/quota-test\"", "[\"/quota-test\",]", "[\"/quota-test\"] []",
+ "[\"/quota-test\", \"sensitive-token\"]", "[\"/quota-test\", \"/\"]",
+ "[\"/quota-test\", \"/zookeeper/quota\"]", "[\"/quota-test\", \"/other//child\"]",
+ "['/quota-test']", "/* comment */[\"/quota-test\"]"
+ };
+ for (String configuration : configurations) {
+ System.setProperty(QUOTA_ALLOWLIST, configuration);
+ CommandResponse response = quotaStats("/quota-test");
+ assertQuotaError(response, QUOTA_ALLOWLIST);
+ assertFalse(response.getError().contains("sensitive-token"));
+ }
+ }
+
+ @Test
+ public void testQuotaStatsRejectsMissingOrInvalidPaths() {
+ System.setProperty(QUOTA_ALLOWLIST, "[\"/quota-test\"]");
+ assertQuotaError(Commands.runCommand("quota_stats", serverFactory.getZooKeeperServer(), null), "path");
+ assertQuotaError(Commands.runCommand("quota_stats", serverFactory.getZooKeeperServer(),
+ Collections.emptyMap()), "path");
+ String[] paths = {
+ null, "", "quota-test", "/", "/zookeeper", "/zookeeper/config", "/zookeeper/quota",
+ "/quota-test/", "/quota-test//child", "/quota-test/.", "/quota-test/../child",
+ "/quota-test/\u0000", "/quota-test/\n", "/quota-test/\ud800", "/quota-test/\uffff"
+ };
+ for (String path : paths) {
+ assertQuotaError(quotaStats(path), "path");
+ }
+ }
+
+ @Test
+ public void testQuotaStatsValidPathComponents() throws Exception {
+ String[] paths = {"/zookeeper-client", "/quota-test/child", "/quota-test/..child", "/quota-test/\u00e9"};
+ System.setProperty(QUOTA_ALLOWLIST, new ObjectMapper().writeValueAsString(paths));
+ for (String path : paths) {
+ assertQuotaResponse(quotaStats(path), path, null, null, null, null, "namespace_missing");
+ }
+ System.setProperty(QUOTA_ALLOWLIST, " [ \"/quota-test\", \"/quota-test\" ] ");
+ assertQuotaResponse(quotaStats("/quota-test"), "/quota-test", null, null, null, null, "namespace_missing");
+ }
+
+ @Test
+ public void testQuotaStatsReflectsAllowlistChanges() {
+ System.setProperty(QUOTA_ALLOWLIST, "[\"/quota-test\"]");
+ assertQuotaResponse(quotaStats("/quota-test"), "/quota-test", null, null, null, null, "namespace_missing");
+ System.clearProperty(QUOTA_ALLOWLIST);
+ assertQuotaError(quotaStats("/quota-test"), "not allowlisted");
+ System.setProperty(QUOTA_ALLOWLIST, "invalid");
+ assertQuotaError(quotaStats("/quota-test"), QUOTA_ALLOWLIST);
+ }
+
+ @Test
+ public void testQuotaStatsUninitializedServer() {
+ String expected = Commands.runCommand("ruok", null, null).getError();
+ for (ZooKeeperServer server : Arrays.asList(null, new ZooKeeperServer())) {
+ CommandResponse response = Commands.runCommand("quota_stats", server,
+ Collections.singletonMap("path", "/quota-test"));
+ assertQuotaError(response, expected);
+ }
+ }
+
+ @Test
+ public void testQuotaStatsMalformedMetadataReturnsUnavailable() throws Exception {
+ ZooKeeper zk = createQuotaFixture();
+ System.setProperty(QUOTA_ALLOWLIST, "[\"/quota-test\"]");
+ zk.setData(Quotas.statPath("/quota-test"), "sensitive-token".getBytes(StandardCharsets.UTF_8), -1);
+ CommandResponse response;
+ try {
+ response = quotaStats("/quota-test");
+ } catch (RuntimeException e) {
+ throw new AssertionError("Quota telemetry must report unavailable, not throw " + e.getClass().getSimpleName());
+ }
+ assertQuotaResponse(response, "/quota-test", null, null, null, null, "invalid_quota_stats");
+ StringWriter output = new StringWriter();
+ new JsonOutputter().output(response, new PrintWriter(output));
+ assertFalse(output.toString().contains("sensitive-token"));
+ JsonNode json = new ObjectMapper().readTree(output.toString());
+ assertTrue(json.get("count_used").isNull());
+ assertTrue(json.get("bytes_used").isNull());
+ assertTrue(json.get("count_limit").isNull());
+ assertTrue(json.get("bytes_limit").isNull());
+ assertFalse(json.get("available").booleanValue());
+ }
+
+ @Test
+ public void testQuotaStatsIncompleteMetadataReturnsUnavailable() throws Exception {
+ ZooKeeper zk = createQuotaFixture();
+ System.setProperty(QUOTA_ALLOWLIST, "[\"/quota-test\"]");
+ zk.delete(Quotas.statPath("/quota-test"), -1);
+ assertQuotaResponse(quotaStats("/quota-test"), "/quota-test", null, null, null, null, "quota_incomplete");
+ }
+
+ @Test
+ public void testQuotaStatsDeniedConfigurationAccess() {
+ SecurityManager original = System.getSecurityManager();
+ try {
+ System.setSecurityManager(new SecurityManager() {
+ @Override
+ public void checkPermission(Permission permission) {
+ }
+
+ @Override
+ public void checkPropertyAccess(String key) {
+ if (QUOTA_ALLOWLIST.equals(key)) {
+ throw new SecurityException("sensitive-token");
+ }
+ }
+ });
+ CommandResponse response = quotaStats("/quota-test");
+ assertQuotaError(response, QUOTA_ALLOWLIST);
+ assertFalse(response.getError().contains("sensitive-token"));
+ } finally {
+ System.setSecurityManager(original);
+ }
+ }
+
+ @Test
+ public void testQuotaStatsValidationPrecedesTreeAccess() {
+ ZooKeeperServer server = mock(ZooKeeperServer.class);
+ when(server.isRunning()).thenReturn(true);
+ when(server.getZKDatabase()).thenThrow(new AssertionError("Rejected requests must not access the data tree"));
+ assertQuotaError(Commands.runCommand("quota_stats", server, null), "path");
+ Map request = Collections.singletonMap("path", "/quota-test");
+ assertQuotaError(Commands.runCommand("quota_stats", server, request), "not allowlisted");
+ System.setProperty(QUOTA_ALLOWLIST, "[\"/quota-test\", null]");
+ assertQuotaError(Commands.runCommand("quota_stats", server, request), QUOTA_ALLOWLIST);
+ }
+
+ @Test
+ public void testQuotaStatsAfterServerRestart() throws Exception {
+ createQuotaFixture();
+ System.setProperty(QUOTA_ALLOWLIST, "[\"/quota-test\"]");
+ assertQuotaResponse(quotaStats("/quota-test"), "/quota-test", 2, 7L, 10, 100L, null);
+ stopServer();
+ startServer();
+ stopServer();
+ startServer();
+ assertQuotaResponse(quotaStats("/quota-test"), "/quota-test", 2, 7L, 10, 100L, null);
+ System.clearProperty(QUOTA_ALLOWLIST);
+ assertQuotaError(quotaStats("/quota-test"), "not allowlisted");
+ }
+
+ private ZooKeeper createQuotaFixture() throws Exception {
+ ZooKeeper zk = createClient();
+ zk.create("/quota-test", "abc".getBytes(StandardCharsets.UTF_8), Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT);
+ SetQuotaCommand.createQuota(zk, "/quota-test", 100L, 10);
+ zk.create("/quota-test/child", new byte[4], Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT);
+ return zk;
+ }
+
+ private CommandResponse quotaStats(String path) {
+ return Commands.runCommand("quota_stats", serverFactory.getZooKeeperServer(),
+ Collections.singletonMap("path", path));
+ }
+
+ private void assertQuotaError(CommandResponse response, String message) {
+ assertEquals("quota_stats", response.getCommand());
+ assertNotNull(response.getError());
+ assertTrue(response.getError(), response.getError().contains(message));
+ Map result = response.toMap();
+ assertEquals(response.getError(), result.get("error"));
+ assertEquals(new HashSet<>(Arrays.asList("command", "error")), result.keySet());
+ }
+
+ private void assertQuotaResponse(CommandResponse response, String path, Integer countUsed, Long bytesUsed,
+ Integer countLimit, Long bytesLimit, String reason) {
+ assertEquals("quota_stats", response.getCommand());
+ assertNull(response.getError(), response.getError());
+ Map result = response.toMap();
+ assertEquals(new HashSet<>(Arrays.asList("command", "error", "schema_version", "path",
+ "count_used", "bytes_used", "count_limit", "bytes_limit",
+ "available", "reason")), result.keySet());
+ assertEquals(1, result.get("schema_version"));
+ assertEquals(path, result.get("path"));
+ assertEquals(countUsed, result.get("count_used"));
+ assertEquals(bytesUsed, result.get("bytes_used"));
+ assertEquals(countLimit, result.get("count_limit"));
+ assertEquals(bytesLimit, result.get("bytes_limit"));
+ assertEquals(reason == null, result.get("available"));
+ assertEquals(reason, result.get("reason"));
+ }
+
@Test
public void testConsCommandSecureOnly() {
// Arrange
From e6bdf6d7f77faa585f99e02b60453b9632d6069e Mon Sep 17 00:00:00 2001
From: Arkesh Mishra <118651144+arkmish@users.noreply.github.com>
Date: Sat, 26 Sep 2026 00:27:52 +0530
Subject: [PATCH 9/9] Clarify strict local audit consumer compatibility
boundary
Document that the pinned local JOBS parser rejects unknown fields and staging quarantines those records. Require updating and validating the consumer before enabling additive v2 producer fields; leave runtime behavior unchanged.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---
.../main/resources/markdown/zookeeperAuditLogs.md | 12 +++++++++---
1 file changed, 9 insertions(+), 3 deletions(-)
diff --git a/zookeeper-docs/src/main/resources/markdown/zookeeperAuditLogs.md b/zookeeper-docs/src/main/resources/markdown/zookeeperAuditLogs.md
index 25f43951054..586767933ed 100644
--- a/zookeeper-docs/src/main/resources/markdown/zookeeperAuditLogs.md
+++ b/zookeeper-docs/src/main/resources/markdown/zookeeperAuditLogs.md
@@ -306,9 +306,15 @@ default locale. Custom `AuditLogger` implementations receive raw event values;
### Mixed versions, rollout, rollback and coverage
-Consumers should recognize v2 per record using `schema_version=2`, tolerate
-unknown additive fields, and accept omitted optional fields. Do not apply v2
-unescaping to legacy records. Prepare consumers for mixed legacy/v2 output before
+Consumers should recognize v2 per record using `schema_version=2` and accept
+omitted optional fields. Consumers designed for forward compatibility should
+tolerate unknown additive fields, but the implemented local JOBS consumer is
+pinned to the current v2 schema and strict: its parser rejects unknown fields
+as `unknown_field`, and local staging quarantines those records. Update and
+validate that consumer before enabling additive producer fields, even when
+the producer retains `schema_version=2`.
+
+Do not apply v2 unescaping to legacy records. Prepare consumers for mixed legacy/v2 output before
enabling the property on servers, then roll it out gradually. Roll back emission
by removing the enhanced property or setting it to `false` on restart. Each
audited request captures the enhanced mode before metadata extraction, and uses