Merge #542

542: More resilient `MoneroWalletRpc` startup in the harness r=da-kami a=da-kami

Recently we se this problem on CI quite often:

```
 May 27 01:26:55.898  INFO testcontainers::core::wait_for_message: Found message after comparing 80 lines
May 27 01:27:00.858  INFO testcontainers::core::wait_for_message: Found message after comparing 2 lines
May 27 01:27:00.859  INFO monero_harness: Starting monerod: DQma_monerod
May 27 01:27:08.143  INFO testcontainers::core::wait_for_message: Found message after comparing 183 lines
May 27 01:27:08.204  INFO monero_harness: Starting miner wallet: miner
May 27 01:27:09.832  INFO testcontainers::core::wait_for_message: Found message after comparing 16 lines
May 27 01:27:12.261  INFO monero_harness: Starting wallet: alice
May 27 01:27:14.482  INFO testcontainers::core::wait_for_message: Found message after comparing 16 lines
thread 'alice_punishes_after_restart_if_bob_dead' panicked at 'called `Result::unwrap()` on an `Err` value: error sending request for url (http://127.0.0.1:49177/json_rpc): operation was canceled: connection closed before message completed
```

Given the message `connection closed before message completed` it is likely that the `monero-wallet-rpc` is not fully started yet.

Unfortunately we cannot wait to see a different message in the logs upon container startup, because there are just no further deterministic messages after the one we are currently listening on.

To overcome this problem without extending testcontainers we introduce a retry mechanism when creating the wallet.

Co-authored-by: Daniel Karzel <daniel@comit.network>

monero-harness/src/lib.rs

@@ -31,6 +31,7 @@ use std::time::Duration;
 use testcontainers::clients::Cli;
 use testcontainers::{Container, Docker, RunArgs};
 use tokio::time;
+use tokio::time::sleep;
 
 /// How often we mine a block.
 const BLOCK_TIME_SECS: u64 = 1;
@@ -251,12 +252,24 @@ impl<'c> MoneroWalletRpc {
             .get_host_port(RPC_PORT)
             .context("port not exposed")?;
 
-        // create new wallet
         let client = wallet::Client::localhost(wallet_rpc_port)?;
 
-        client
-            .create_wallet(name.to_owned(), "English".to_owned())
-            .await?;
+        // Create new wallet, the RPC sometimes has startup problems so we allow some
+        // retries
+        tokio::time::timeout(Duration::from_secs(10), async {
+            loop {
+                let result = client
+                    .create_wallet(name.to_owned(), "English".to_owned())
+                    .await;
+
+                match result {
+                    Ok(_) => { break; }
+                    Err(e) => { tracing::warn!("Monero wallet RPC emitted error {} - retrying to create wallet in 2 seconds...", e); }
+                }
+
+                sleep(Duration::from_secs(2)).await;
+            }
+        }).await.context("All retry attempts for creating a wallet exhausted")?;
 
         Ok((
             Self {