Scheduled recovery:

This release contains a scheduled retry for January 14 11:00 UTC. Make sure your node is upgraded by then!

Breaking changes:

• The Wormhole node now supports sending logs to a telemetry server to help network operators with network-wide visibility. The new release either requires a --telemetryKey <key> or a --disableTelemetry flag. We recommend enabling telemetry.

Improvements:

• Eth watcher logging verbosity has increased - logging for Eth observations is now much more fine-grained. If you have previously been using processing new header or similar logs for monitoring, you might have to update your config (note that we do not consider changes to log messages breaking changes and do them all the time - there is no stability guarantee for logs, they're meant for human debugging. We recommend using the metrics endpoint, which has a stability guarantee!).

Bugfixes:

• Refactored the Terra handler to support multiple messages inside a single transaction. This got a couple of transactions unstuck (this was a coordinated hotfix - mainnet is running this already).
• Eth watcher reliability improvements for Polygon.

Breaking changes:

• Oasis support was added via --oasisRPC and --oasisContract flags! 🎉

New features:

• The admin find-missing-messages command now supports a --backfill parameter, which will ask the node to backfill missing messages in the local database by querying a list of public Wormhole RPC node.
• admin list-nodes now supports filtering columns by specifying a comma-separated list of networks using --only.

Improvements:

• The number of retries for Solana transactions and message account fetches were doubled, improving behavior when RPC is slow.

Scheduled recovery:

This release contains a scheduled recovery for 2021-12-23 11:00 UTC, retrying about a dozen transactions which timed out over the past days while RPC performance was degraded due to heavy load.

No backwards incompatible changes (unless you're using Infura, which nobody should do anyway).

New features:

• Batched governance message submission.
  A governance prototxt file can now include multiple messages. This significantly improves the workflow when multiple governance votes are submitted at the same time.

• The guardiand template command has been revamped to support generating full governance VAAs. This makes it easier and less error-prone to produce governance proposals.

• Complain about Infura on mainnet.
  As it turns out, Infura has a bug where it would sometimes incorrectly round block timestamps, which causes consensus issues - the timestamp is part of the VAA and nodes using Infura would sometimes derive an incorrect VAA, accidentally attacking the network by signing a conflicting VAA.

  Node operators do not usually rely on Infura in the first place - doing so is less secure than using your own nodes, since nodes blindly trust the connected nodes to verify on-chain message proofs. Node operators sometimes used Infura as a workaround during migrations where their primary node was offline, causing the aforementioned consensus oddities which were eventually found to be Infura-related. This is generally to the detriment of network security and a judgement call made by individual operators. In the case of Infura, we know it's actively dangerous so we add a warning to make sure that future operators know about the issue before considering to use Infura.

Bugfixes:

• The /readyz endpoint on the status server now correctly reports a failure of BSC or Polygon.

New features:

• A Polygon connection is now supported/required using the --polygonRPC and --polygonContract command line options.

Improvements:

• The schema for injected ContractUpgrade governance VAAs now uses hex-encoded addresses for consistency.
• Harmless received observation by unknown guardian messages were demoted to DEBUG level to avoid confusion.

Backwards-incompatible changes:

• The unused --terraChainID flag was removed.

New features:

• guardiand now supports generating NFT/token bridge upgrade governance VAAs.

• Support for posting alerts to Discord was added (can be seen in action in #alerts on Discord).

• A guardiand admin find-missing-messages command was added, which searches the local VAA store for sequence number gaps for a specific emitter address. This can be used to verify whether the local node is missing any signed VAAs - there should be zero gaps.

  For example, this is how you could search for missing messages for recent emitters on mainnet using this new feature:

  grep -aF "observed and signed confirmed message publication" /var/log/wormhole.log | \
    head -n 500 | grep -Po '{.*}'  | jq -r '.message_id | split("/")[:-1] | join(" ")' | sort | uniq | \
    xargs -n2 echo sudo -u wormhole guardiand admin find-missing-messages --socket /run/guardiand/admin.socket
  

Improvements:

• No longer run the scheduled recovery on startup.
• Some Prometheus per-chain metrics were fixed to include the correct source label rather than "unknown".
• Messages with a sequence number larger than INT64_MAX can now be requested from the API.
• Gossip observations now include a MessageID field, which is logged. This helps debugging observations without local state. The previous release already added a TxHash field for the same reason.

Scheduled recovery:

This release contains a scheduled message recovery. The network failed to reach consensus on seven messages due to Solana RPC timeouts experienced by a superminority of nodes.

On Oct 5 2021 11:00 UTC, the guardiand node fetches a list of transactions from GitHub to re-observe these accounts and achieve consensus. The list of accounts is untrusted - guardiand still relies on the connected RPC node for finality, it merely re-triggers the observation using the same code path.

This release includes retry logic to improve handling of RPC timeouts, so hopefully this is the first and only time we need to use this new feature :-)

Breaking changes:

• The guardiand bridge command was renamed to guardiand node to avoid any confusion with token bridge functionality. The build command is now make node. Two command line flags were also changed:

  • --solanaBridgeAddress is now --solanaContract
  • --bridgeKey is now --guardianKey

  The Heartbeat proto message had one of its fields renamed from BridgeAddress to ContractAddress. If you were using the REST API (rather than the gRPC API, which is unaffected), you might have to update your code to use contract_address instead of bridge_address.

• Support for raw heartbeat streams was removed from the public API. The GetLastHeartbeats / v1/heartbeats API with server-side aggregation should be used instead.

• --publicREST has been extended with grpc-web and TLS support and renamed to --publicWeb (see below).

• Go 1.17 is now required to build the guardiand node.

New features:

• The admin CLI now supports injecting token and NFT bridge chain registration messages.

• admin list-nodes now supports a --showDetails flag to display contract addresses and error counts.

• guardiand now comes with a built-in REST and grpc-web API endpoint for web UIs to retrieve signed messages. Guardians are encouraged to run a public API endpoint for web UIs to use - please refer to the docs for more details.

  • The node exports Prometheus metrics for gRPC requests made to the publicWeb service.

• Signed VAA messages are now broadcast on the gossip network, allowing all nodes to store them even if they missed the original observation.

Improvements:

• guardiand startup time has been improved to a few seconds down from ~15s by improving Ethereum guardian set fetching.

• invalid signed heartbeat received messages are no longer logged at INFO level. These messages occur when nodes that aren't part of the guardian set broadcast signatures, which is normal.

• guardiand now maintains state for messages it observed but failed to reach consensus for up to 24 hours, allowing the network to submit observations late. This is meant to facilitate manual governance consensus and nodes falling behind and catching back up.

• guardiand now retries failed Solana slot and account fetches up to five times.

• A workaround was added for a suspected Solana bug, causing finalized transactions to be missed in rare occasions. This increases the number of GetConfirmedBlock calls by about 10% (skipped blocks are no longer excluded).

• Log output for message observations was improved to include more context for lookups.

This release is suitable for mainnet and testnet.

New features:

• guardiand now maintains state for all active guardian nodes. The admin list-nodes command displays detailed info on node status, and corresponding GetCurrentGuardianSet and GetLastHeartbeats RPC calls were added. The old streaming output with on-the-fly aggregation is still available via admin list-nodes-stream.

• Heartbeats now contain a boot timestamp to calculate uptime.

• Heartbeats now include an error counter.

• Prometheus metrics now include network-wide metrics via wormhole_network_node_height and wormhole_network_node_errors_count.

• The example Grafana dashboard was updated for Wormhole v2.

Bugfixes:

• --statusAddr now works (thanks @winslyn!)
• A bug related Ethereum and BSC guardian set fetching was fixed.

Initial release for the Wormhole v2 mainnet.

This is a testnet release - do not use on mainnet!

This release fixes a panic encountered when Solana watcher requests fail.

This is a testnet release - do not use on mainnet!
Wait for instructions by the Wormhole team before upgrading your nodes.

⚠️ PLEASE USE THE v2.1.1 BUGFIX RELEASE INSTEAD ⚠️

BSC support

Wormhole now supports Binance Smart Chain. The following required parameters were added:

• --bscContract
• --bscRPC

Refer to https://github.com/certusone/wormhole-networks/blob/master/testnetv2/info.md for updated instructions.

New Solana watcher

Wormhole's Solana watcher has been upgraded to a new mechanism, fetching individual slots instead of program accounts (see #248 for details).

This new mechanism significantly improves latency when fetching Solana accounts and reduces the load on your RPC nodes. You need to change your Solana RPC node config: https://github.com/certusone/wormhole/blob/dev.v2/docs/operations.md#solana-node-requirements