smite-scenarios: sync target chain view via RPC after mining - #211
smite-scenarios: sync target chain view via RPC after mining#211NishantBansal2003 wants to merge 2 commits into
Conversation
erickcestari
left a comment
There was a problem hiding this comment.
Good idea! It's much better that it only calls the sync signal once per MineBlock instead of calling it for every block mined.
| assert!( | ||
| out.status.success(), | ||
| "pkill -USR1 ldk-node-wrapper failed: {}", | ||
| String::from_utf8_lossy(&out.stderr) | ||
| ); |
There was a problem hiding this comment.
Should we have an assert here? Imagine the executor crashes the target while executing the input. It would probably fail to sync and panic with a pkill error instead of failing with a Violation::Crashed error.
There was a problem hiding this comment.
Hmm, great point, I think pkill or lightning-cli should always succeed in any case, and they should only error out when the target has already crashed. So, I think we could check whether the target has crashed before asserting this. If it has crashed, we can simply log the failure here, otherwise, we should assert it. I don't think there is any other case where RPC commands should be unresponsive, so I think it's better to be stricter. WDYT?
There was a problem hiding this comment.
Yes, I agree! I also think logging is a good option.
| cmd.env("LD_PRELOAD", handler); | ||
| } | ||
|
|
||
| // Ignore SIGUSR1 for the window between exec and the wrapper blocking |
There was a problem hiding this comment.
Some comments at workloads/ldk/src/main.rs also need to be updated after the remove of SIG_IGN.
81c6746 to
879486b
Compare
|
I updated the CLN I verified this by crashing the target before syncing the chain. CLN and LDK are now able to distinguish between the target having already crashed and other errors, logging the former and hard-asserting on the latter. I haven't done the same for LND and Eclair since they currently don't use any CLI calls. I'll address those in follow-up PRs, as we need to make RPC calls for them as well |
879486b to
9578d23
Compare
erickcestari
left a comment
There was a problem hiding this comment.
LGTM! I think we can optimize this a bit by avoiding spawning the pkill process, consequently reducing the memory overhead and the amount of memory dirtied by process creation.
| fn chain_sync(&mut self) { | ||
| let out = Command::new("pkill") | ||
| .arg("-USR1") | ||
| .arg("-f") | ||
| .arg("^ldk-node-wrapper") | ||
| .output() | ||
| .expect("pkill -USR1 ldk-node-wrapper should not fail"); | ||
|
|
||
| match out.status.code() { | ||
| Some(0) => {} | ||
| // pkill exits 1 when nothing matches, so LDK is gone, check_alive | ||
| // will report the crash at the end. | ||
| Some(1) => log::warn!("ldk-node-wrapper is not running, skipping chain sync"), | ||
| _ => panic!( | ||
| "pkill -USR1 ldk-node-wrapper failed: {}", | ||
| String::from_utf8_lossy(&out.stderr) | ||
| ), | ||
| } | ||
| } |
There was a problem hiding this comment.
We could use the ldk ManagedProcess to get the PID and use the syscall kill directly instead of creating a pkill process to do this for us.
There was a problem hiding this comment.
Done and Rebased!
5931f69 to
9975e59
Compare
Targets previously learned about new blocks through bitcoind's blocknotify, which fires asynchronously. When multiple blocks are mined, this can queue up many sync calls, adding unnecessary load. Instead, explicitly sync the target once after each MineBlocks operation. Introduce a TargetRpc trait with a chain_sync() method, called by the executor immediately after MineBlocks. CLN and LDK sync explicitly via RPC and signals, respectively, while LND and Eclair use ZMQ and remain no-ops. Dropping -blocknotify also removes the burst of asynchronous SIGUSR1s during initial block generation, so LDK no longer needs the pre_exec SIG_IGN. Signed-off-by: Nishant Bansal <nishant.bansal.282003@gmail.com>
9975e59 to
b3a753b
Compare
| assert!( | ||
| response.get("error").is_none(), | ||
| "lightningd rejected {method}: {}", | ||
| response["error"] | ||
| ); | ||
|
|
||
| Ok(response["result"].take()) |
There was a problem hiding this comment.
Should we define an RpcResponse struct like we do for all other RPCs? It would make it clearer what we expect a response to contain.
| #[derive(Debug, Clone)] | ||
| pub struct ClnRpc { | ||
| /// Path to the CLN node's unix RPC socket. | ||
| pub rpc_socket: PathBuf, |
There was a problem hiding this comment.
| pub rpc_socket: PathBuf, | |
| rpc_socket: PathBuf, |
| } | ||
|
|
||
| impl ClnRpc { | ||
| // Bound RPC socket I/O so a stalled lightningd cannot block indefinitely. |
There was a problem hiding this comment.
| // Bound RPC socket I/O so a stalled lightningd cannot block indefinitely. | |
| /// Bound RPC socket I/O so a stalled lightningd cannot block indefinitely. |
| let mut response: serde_json::Value = serde_json::Deserializer::from_reader(&mut sock) | ||
| .into_iter() | ||
| .next() | ||
| .unwrap_or_else(|| panic!("lightningd closed the socket without answering {method}")) | ||
| .unwrap_or_else(|e| panic!("failed to read {method} response from lightningd: {e}")); |
There was a problem hiding this comment.
This code is confusing -- what is being unwrapped twice? Also, the second panic triggers on timeouts too, which seems like it should also be "warn-only" since a crash/hang will be detected later.
Perhaps we should match on the response for clarity and to let us distinguish the timeout error types from other types.
There was a problem hiding this comment.
I updated the error handling to cover possible cases where the target is hung or crashed, whether before the call, during the write, or during the read, and also made it a bit clearer. I also verified some crash cases by crashing CLN before the run, after the write, and after the read, and the handling covers all of them
Wasn't fully sure how to test the hang cases, but I think TimedOut or UnexpectedEof should cover those cases as well
| /// defaults to terminating the process. That distinction is the whole point: an | ||
| /// *ignored* signal is discarded the moment it is delivered, while a *blocked* | ||
| /// one stays pending until `sigwait()` consumes it, regardless of its | ||
| /// disposition. So this call is what makes target's SIGUSR1 observable, and | ||
| /// why nothing here calls `sigaction`: the wait loop below is the only consumer | ||
| /// these signals need. |
There was a problem hiding this comment.
No need to explain the difference between signal-blocking and SIG_IGN anymore.
| /// defaults to terminating the process. That distinction is the whole point: an | |
| /// *ignored* signal is discarded the moment it is delivered, while a *blocked* | |
| /// one stays pending until `sigwait()` consumes it, regardless of its | |
| /// disposition. So this call is what makes target's SIGUSR1 observable, and | |
| /// why nothing here calls `sigaction`: the wait loop below is the only consumer | |
| /// these signals need. | |
| /// defaults to terminating the process. A blocked SIGUSR1 | |
| /// stays pending until `sigwait()` consumes it, regardless of its | |
| /// disposition. So this call is what makes target's SIGUSR1 observable, and | |
| /// why nothing here calls `sigaction`: the wait loop below is the only consumer | |
| /// these signals need. |
| panic!( | ||
| "failed to send SIGUSR1 to ldk-node-wrapper (pid {}): {e}", | ||
| self.pid | ||
| ); |
There was a problem hiding this comment.
Why do we panic for LDK but warn for CLN in the same situation?
There was a problem hiding this comment.
Because bad RPC calls already panic inside self.run for CLN, while cases where CLN has already crashed or hung just return Err so the caller can log them. In the future, I think we'll mostly remove this pattern, and will return Err directly from chain_sync or other RPC calls and instead let ir.rs log the RPC error itself, since we may also run commands like addinvoice and want their responses to proceed further
For LDK, the signal-based approach is essentially fire-and-forget. Even if LDK has crashed or hung, its PID remains until the parent reaps it, so the signal is delivered normally (though it won't be responsive) and the call returns Ok. So, any other error is unexpected, so we panic on it
| impl TargetRpc for LndRpc { | ||
| /// LND receives new blocks directly from bitcoind over ZMQ, so no manual | ||
| /// chain synchronization is required. | ||
| fn chain_sync(&mut self) {} |
There was a problem hiding this comment.
nit: could it make sense to assert the block height here, since after calling this function, I think the caller expects the chain to be synced? But then we would need to pass the expected block height somehow.
There was a problem hiding this comment.
Yeah, I mean that is the defensive way, but the current sync model is essentially fire-and-forget. To actually get the updated block height for each implementation, we would need to wait for some time and then make another RPC call, and I'm not sure how long we should wait or how many times we should retry. In the worst case, we might make several RPC calls, which could hurt the fuzzing itself
So I believe a cheaper way to check whether the blocks have been synced is to run a full funding flow and see if we receive channel_ready from the peer
ref: #195
Targets (CLN, LDK) previously learned about new blocks through bitcoind's
blocknotify, which fires asynchronously. When multiple blocks are mined, this can queue up many sync calls, adding unnecessary load. Instead, explicitly sync the target once after eachMineBlocksoperation.Introduce a
TargetRpctrait with achain_sync()method, called by the executor immediately afterMineBlocks. CLN and LDK sync explicitly via RPC and signals, respectively, while LND and Eclair use ZMQ and remain no-ops.Dropping
-blocknotifyalso removes the burst of asynchronousSIGUSR1s during initial block generation, so LDK no longer needs thepre_execSIG_IGNThe reason I went with an RPC trait is that, in the future, we'll need access to the target's CLI/RPC to add payment hash and preimage information to the target so that it can send an
update_fulfill_htlcmessage.Also, for now, I only added the CLI as a trait implementation for each target to keep the changes contained and concise. In the future, we can definitely add direct RPC calls over the socket as well to save overhead.