Summary

• Major UI Update - Includes Activity, Blockchain, Off-Chain, Tokens, Network Map, and My Node sections (https://github.com/hyperledger/firefly-ui)
• Custom contract APIs complete
• Enhanced subscription filters
• Event API enrichment
• Performance updates
• Bug fixes

Breaking Changes and Migrations

• FIR-9 Enhanced Identity API - custom identities and DID support
  • New namespaced API endpoints to manage identities within a namespace:
    • POST /api/v1/namespaces/identities - async broadcast, with confirm=true option for sync
    • GET /api/v1/namespaces/identities - collection of identities within namespace
    • GET /api/v1/namespaces/identities/{iid} - individual identity
    • GET /api/v1/namespaces/identities/{iid}/did - generated DID document
    • GET /api/v1/namespaces/identities/{iid}/verifiers - collection of keys
  • New non-namespaced endpoint is provided to look up identities by DID:
    • GET /api/v1/network/identities/{did} - efficient cached DID to identity resolution
  • Minor changes to existing APIs
    • GET /api/v1/network/organizations - now returns identity object
    • GET /api/v1/network/nodes - now returns identity object
    • GET /api/v1/status - now returns the did of the local organization
    • The identity field in nodes/orgs has been removed (used to contain the signing key)
    • The did field from identity is newly available on nodes and orgs
  • Existing node and organization identities are migrated to the new identity API
    • No migration support for non-root organizations, as the FIR-9 security model requires a verification transaction from the parent organization that is not available.
    • Existing root organization and node objects are migrated
    • New nodes joining a network can process previously published organization and node identities
    • All organizations now represented as an identity in the ff_system namespace with special types org and node
• Operation type strings have been updated for consistency:
  • blockchain_pin_batch - previously blockchain_batch_pin
  • dataexchange_send_batch - previously dataexchange_batch_send
  • dataexchange_send_blob - previously dataexchange_blob_send
  • sharedstorage_upload_batch - previously sharedstorage_batch_broadcast
  • sharedstorage_upload_blob - new
  • sharedstorage_download_batch - new
  • sharedstorage_download_blob - new
• Blockchain events
  • The /api/v1/contracts/subscriptions endpoint has been renamed to /api/v1/contracts/listeners
  • The /messages/{msgid}/operations endpoint has been removed
  • Events of type blockchain_event are now type blockchain_event_received
• The Ethereum plugin now expects a different input format to generate a FireFly Contract Interface from an Ethereum ABI. Please see the docs for full details.
• The Public Storage plugin interface has been renamed to Shared Storage. The FireFly core config should be updated to replace publicstorage with sharedstorage as a root key. This release of FireFly is still backwards compatible with the old config key.

Updated Dependencies

• firefly-ethconnect v3.1.5
• firefly-fabconnect v0.9.13
• firefly-tokens-erc1155 v0.10.6
• firefly-tokens-erc20-erc721 v0.2.0
• firefly-ui v0.6.0

What's Changed

• FIR-9: Identity enhancements by @peterbroadhurst in #549
• Rename contract subscriptions to contract listeners by @nguyer in #548
• Rename Public Storage to Shared storage by @nguyer in #538
• Add missing itype column in SQLite migration by @peterbroadhurst in #577
• Add Operations Manager, wrap all operations, and add retry functionality by @awrichar in #517
• Handle operations with retries in transaction status by @awrichar in #578
• [charts-by-type] histogram broken down by type by @eberger727 in #547
• Enhance subscription filters & event enrichment by @shorsher in #579
• add fetchreferences param to events api by @shorsher in #587
• Use CLI to deploy test contract by @nguyer in #589
• Update batch logic to only store+hash the manifest by @peterbroadhurst in #582
• Batching message writer routines, to use multi-value insert across API calls on message/data insert by @peterbroadhurst in #592
• Event lookup cache by @peterbroadhurst in #593
• Add DB tuning to keep conns option, and correct cache bypass on group lookup by @peterbroadhurst in #594
• Scope subscription check to the appropriate stream ID by @jebonfig in #591
• Move blob publish to broadcast batch dispatch, ensure public refs on inbound by @peterbroadhurst in #596
• Performance enhancements from flame graph analysis by @peterbroadhurst in #599
• Remove route /messages/{msgid}/operations by @awrichar in #602
• Upgrade dependencies by @nguyer in #605
• Add "info" field to token pools, consume "symbol" from connectors by @awrichar in #606
• [blockchainevents-charts] support for blockchain event charting by @eberger727 in #612
• Add E2E test support for ERC20/ERC721 by @awrichar in #611
• Add firstEvent field to contract listeners by @nguyer in #603
• Dependency updates for v0.14 testing by @peterbroadhurst in #613
• Batch confirm messages, and update events to contain topic+tag by @peterbroadhurst in #600
• Change node/org lookups to support name or ID by @awrichar in #616
• Update microservices versions for v0.14.0 by @nguyer in #615
• Restart stack and re-run E2E tests in daily integration job by @nguyer in #608
• Add Shared Storage Download Manager by @peterbroadhurst in #607
• Add route to lookup identity by DID by @awrichar in #617
• Blockchain metrics for Ethereum by @nguyer in #584
• [ui-v0.6.0] PR for new UI release by @eberger727 in #619
• Custom contracts docs by @nguyer in #604
• [ui-v0.6.0-release] new ui release by @eberger727 in #620

Full Changelog: v0.13.1...v0.14.0

Summary

• Support for token approvals (only supported by erc1155 connector as of this release)
• Bug fixes

Updated Dependencies

• firefly-ethconnect v3.1.4
• firefly-fabconnect v0.9.11
• firefly-tokens-erc1155 v0.10.5
• firefly-tokens-erc20-erc721 v0.1.6

What's Changed

• Add additional backwards compatibility check on ethereum instance path by @nguyer in #531
• Build a predictable response body for async "InvokeContract" calls by @awrichar in #532
• Add endpoint to automatically generate an FFI by @nguyer in #511
• Updates to DX by @gabriel-indik in #534
• Add Transaction ID to Events by @awrichar in #522
• Add query support for Fabric by @nguyer in #533
• [transfer-type-query] allow filtering by type on token transfers by @eberger727 in #544
• Change query method to POST for custom contracts by @nguyer in #542
• Fix empty swagger URLs on contract API creation by @nguyer in #543
• Support for token approvals by @shorsher in #529
• Docs "bundle update" to pick up new nokogiri build by @peterbroadhurst in #552
• Documentation updates from @TheSwarnim by @peterbroadhurst in #553
• Updated documentation to make it more useful by @TheSwarnim in #451
• Roll requestConflictEmptyResult from Group to Data and Message too by @peterbroadhurst in #551
• Explicitly name all subscriptions by @peterbroadhurst in #555
• Update manifest versions by @awrichar in #556

New Contributors

• @gabriel-indik made their first contribution in #534
• @TheSwarnim made their first contribution in #451

Full Changelog: v0.13.0...v0.13.1

Summary

This is an important hardening release - while very little has changed on the surface, it includes a lot of significant rework to the core of FireFly, mostly to fix issues exposed by the performance testing in #316.

Breaking changes:

• The token_pool_rejected event was removed, along with the recently-added contract_interface_rejected and contract_api_rejected (only message_rejected will now be emitted for rejected definitions)
• The expected format of the FireFly contract address in the "instance" config entry for Ethereum blockchains has changed (see "Migrations" below)
• The default data exchange plugin has been renamed from https to ffdx (old name will still be accepted with a warning, but this will eventually become a fatal config error)

Other major items included in this release:

• Support for running on ARM-based M1 processors
• Rewrite of the message batching and event aggregation logic inside FireFly, to fix numerous edge cases with lost or hung messages
• Hardening of operations and transactions to behave more consistently across all types
• Metrics reporting to Prometheus
• Continued development to support custom on-chain logic (still in preview)

Migrations

• The Ethereum plugin now expects just a contract address in the instance field of the FireFly core config file. The value should be a hex string (with the 0x prefix) which is the Ethereum address of the FireFly smart contract. It is recommended to update any existing config file for this release, though this version will attempt to read values from existing configs and determine the contract address from the old value. Note: FireFly will not automatically update the config file as it treats it as read-only.

Here is an example of what your blockchain section should look like (your actual instance value will be a different hex string):

blockchain:
  type: ethereum
  ethereum:
    ethconnect:
      instance: 0x8707d1f1151220430fc8f89836b20bcf05d6eb41

Updated Dependencies

• firefly-ethconnect v3.1.3
• firefly-fabconnect v0.9.9
• firefly-dataexchange-https v0.10.4
• firefly-tokens-erc20-erc721 v0.1.5

What's Changed

• Store all token transfer inputs on operation by @awrichar in #478
• Make the factories act like factories by @awrichar in #474
• Add missing test coverage in txcommon by @awrichar in #476
• Use PreFinalize/Finalize in all definition handlers by @awrichar in #475
• wsReader cannot rely on testing.T as it might extend past test scope by @peterbroadhurst in #472
• Set default keepalive timeout to 5s on clients for Node.js connectors by @peterbroadhurst in #470
• Address some inconsistencies in Operation behavior by @awrichar in #480
• Remove extra "rejected" events by @awrichar in #482
• Update aggregator batch processing to maintain in-memory pin state until OnFinalize by @peterbroadhurst in #483
• Fix fabric event timestamp parsing by @jimthematrix in #491
• Fix Docker build on ARM by @nguyer in #473
• Add extra logging for message/event ids by @peterbroadhurst in #488
• Pass arguments from Makefile to docker build script by @nguyer in #495
• Fix possibility for nil access in sendloop by @peterbroadhurst in #486
• Fix group to be constant throughout test by @peterbroadhurst in #490
• Compile directly with solc to avoid full Node.js npm install of Truffle by @peterbroadhurst in #494
• Rewind when messages appear behind offset by @peterbroadhurst in #496
• Pass requestId to DX and remove backend_id from Operation by @awrichar in #497
• Rename data exchange plugin to "ffdx" by @awrichar in #505
• Misc fixes for operations by @awrichar in #498
• Use inline ABI for all Ethconnect contract interactions by @nguyer in #502
• Prometheus metrics by @eberger727 in #503
• Re-work batch logic for simplicity, efficiency, and restart recovery by @peterbroadhurst in #501
• Add "initEnabled" config to DX by @awrichar in #507
• Move insert of events to a special pre-commit phase of the DB transaction by @peterbroadhurst in #510
• Flatten the DXInfo back to the orginal payload on the ffdx API to connectors by @peterbroadhurst in #514
• Implement and use UpsertOptimization for groups by @peterbroadhurst in #513
• Update components ready for v0.13.0 by @peterbroadhurst in #516
• Update EthConnect to 3.1.3 by @peterbroadhurst in #518
• [metrics-types] using fftypes in metrics manager by @eberger727 in #520
• Provide Migration / Registration / Debugging Utilities in Docker Image by @hfuss in #521
• go-migrate in Dockerfile by @hfuss in #524
• Fix pages build by @peterbroadhurst in #526
• Use bash for variable expansion prior to make by @peterbroadhurst in #528
• Allow building in docker by removing .git from .dockerignore by @peterbroadhurst in #530

Full Changelog: v0.12.0...v0.13.0

Summary

This is a large release as we come closer to the 1.0.0 milestone. All users are recommended to upgrade to this release when possible, taking into account all breaking API changes noted here.

Breaking API changes:

• All APIs deprecated in v0.11.0 or earlier are removed - see detailed migration notes
• Richer query parameters for filtering collections - with breaking changes to the ^ operator

Other major items included in this release:

• Preview of custom on-chain logic
  • Provides flexible methods for interacting with your own contracts, deployed to the blockchain of your choice
  • New APIs are available under /namespaces/{ns}/contracts and/namespaces/{ns}/apis
  • Not yet guaranteed to be stable, and API is subject to change
• Support for new ERC20 / ERC721 connector
• Overhaul of Transaction type, and new BlockchainEvent type
• Support for delivery confirmations via DX plugin
• Support for more flexible signing via new config pointing at an external address resolver
• First-class support for blob filenames on Data objects
• UI enhancements
  • Improved support for navigating and downloading file blobs
  • Filtering for token screens (pools, transfers, accounts)
• Bug fixes and improvements

Updated Dependencies

• firefly-ethconnect v3.1.2
• firefly-fabconnect v0.9.7
• firefly-dataexchange-https v0.10.3
• firefly-tokens-erc1155 v0.10.4
• firefly-tokens-erc20-erc721 v0.1.4
• firefly-ui v0.5.0

What's Changed

• Fixing Websocket Connections when Prometheus Metrics Enabled by @hfuss in #371
• Set fetch-depth to get the full history in GitHub Actions by @nguyer in #377
• Fetch full history on all GitHub Actions by @nguyer in #378
• [helm] Publishing Helm Chart to GHCR via OCI by @hfuss in #359
• [psql-maxconns] default connection limit for postgresql by @eberger727 in #385
• Allow any company name in license header by @nguyer in #379
• Prevent send/receive deadlock in wsclient by @awrichar in #387
• API changes for Blob Friendly Names by @peterbroadhurst in #381
• Fix unit test for wsclient send failure by @awrichar in #388
• Fixing Duplicate Metrics Registration when Unsetting preInit by @hfuss in #389
• Have DB creation script account for username by @drewmarshburn in #375
• Add Performance Testing Details to README by @eberger727 in #373
• Add postgres migrations for onchain-logic by @nguyer in #390
• FIR-4 (rich query updates) and FIR-7 (DX manifests) integration to onchain-logic branch by @peterbroadhurst in #403
• Rectify Transactions and BlockchainEvents by @awrichar in #408
• Build the URL without the firefly port if it was omitted from stack.json by @jebonfig in #411
• Collapse onchain-logic into main by @awrichar in #410
• [erc20-payload] pass name and symbol to CreateTokenPool() by @eberger727 in #413
• Removing Helm Chart and Related CI Workflow by @hfuss in #412
• Update containerd by @peterbroadhurst in #414
• PostgreSQL updates following merging of onchain-logic branch by @peterbroadhurst in #417
• Fix token account listing on Postgres by @awrichar in #422
• Fix intermittent Fabric E2E test failure by @nguyer in #424
• Add ping/pong heartbeating to WSClient, and fix concurrent map on config by @peterbroadhurst in #420
• Fail transaction when token transfer operation fails by @awrichar in #435
• Ready state changes require a bump to the message to re-sequence it by @peterbroadhurst in #438
• JSON Schema validation for FFIs by @nguyer in #419
• Restore Transaction reference on TokenTransfer by @awrichar in #444
• Create a Transaction and Operation for contract invoke requests by @awrichar in #441
• Rename /contracts/events to /blockchainevents by @awrichar in #443
• Add convenience URL /transactions/{txnid}/blockchainevents by @awrichar in #446
• Allow calling JSONObject() on nil JSONAny by @peterbroadhurst in #456
• Add Blockchain Transaction IDs to FireFly Transaction objects and add blockchain IDs to events by @peterbroadhurst in #448
• Fixing E2E Tests for Multi-Member Networks by @hfuss in #459
• Avoid potential double-close of channels due to reconnects after tests complete by @peterbroadhurst in #458
• Add route for /transactions/{txnid}/status by @awrichar in #457
• Add AddressResolver to Ethereum for key-to-address mapping by @peterbroadhurst in #436
• Allow synchronous 200 response for token pool creation by @awrichar in #460
• Remove deprecated APIs by @peterbroadhurst in #452
• ui version 0.5.0 by @shorsher in #464
• Add DefinitionBatchActions to execute after a batch of system definitions by @awrichar in #462
• Use /firefly prefix on local docs dev by @peterbroadhurst in #461
• Allow passing a tokenIndex to "mint" by @awrichar in #465
• [v0.12.0-manifests] manifests for v0.12.0 by @eberger727 in #466

New Contributors

• @drewmarshburn made their first contribution in #375

Full Changelog: v0.11.4...v0.12.0

Summary

• Backported bug fixes

What's Changed

• v0.11.x backport: Fix querying transfers by tx or type, batches by tx by @awrichar in #429
• v0.11.x backport: Fixing Websocket Connections when Prometheus Metrics Enabled + containerd update by @peterbroadhurst in #432
• v0.11.x backport: Fix token account listing on Postgres by @peterbroadhurst in #431
• v0.11.x backport: Add ping/pong heartbeating to WSClient, and fix concurrent map on config by @peterbroadhurst in #430
• v0.11.x backport: Ready state changes require a bump to the message to re-sequence it by @peterbroadhurst in #439

Full Changelog: v0.11.4...v0.11.5

Summary

• Dependency updates and bug fixes

Updated Dependencies

• firefly-fabconnect v0.9.4

What's Changed

• Config map length fix, fix bug with missing events by closing the websocket by @jebonfig in #367
• Update dependencies for next release by @peterbroadhurst in #369
• Fix misspelled filename (bytetypes.go) by @awrichar in #368
• Validate message topics/tag before sending by @awrichar in #365

Full Changelog: v0.11.3...v0.11.4

Summary

• Dependency updates and bug fixes
• Helm/Kubernetes deployment template updated for fabric and prometheus metrics

Updated Dependencies

• firefly-ethconnect v3.1.0
• firefly-fabconnect v0.9.3
• firefly-tokens-erc1155 v0.10.2
• firefly-dataexchange-https v0.9.3

What's Changed

• Fabric connector needs to subscribe from zero by @peterbroadhurst in #361
• Support configuration keys when the value type is a list by @jebonfig in #362
• [helm] Adding Storage for DX Blobs and PVC Templating Improvements by @hfuss in #354
• Fix send-to-self for private messages, and add group query URLs by @peterbroadhurst in #363
• [helm] Support for Prometheus Metrics by @hfuss in #338
• [fabric] Dockerfile for Compiling firefly-go for Linux by @hfuss in #357
• Update manifest ready for v0.11.3 release by @peterbroadhurst in #364
• [helm] Fabconnect Support by @hfuss in #356

Full Changelog: v0.11.2...v0.11.3

Summary

This patch release includes important fixes and hardening for recent features. Major highlights:

• New charts UI and UI fixes for token panels
• Handling for token connectors that don't guarantee a unique URI for each NFT

Updated Dependencies

• firefly-dataexchange-https v0.9.3
• firefly-ui v0.4.3

What's Changed

• Fix Hyperledger Image Refs and Migration Job's Postgres URL in Helm Chart by @hfuss in #340
• Fix image repos by @peterbroadhurst in #288
• Updates to handle recursive structures in swagger - depends on kin-openapi change by @peterbroadhurst in #337
• Return an empty slice (rather than nil) for zero token accounts by @awrichar in #346
• Address coverage drop from #337 by @awrichar in #347
• on transfer/mint/burn fail, set transaction & operation as failed by @shorsher in #351
• update prometheus client version by @shorsher in #352
• Replace UpsertOperation with InsertOperation by @awrichar in #350
• Metrics endpoint to support charting by @eberger727 in #341
• Remove unique index on token URIs by @awrichar in #353
• Reduce overhead of gocritic by @peterbroadhurst in #344
• ui version 0.4.3 by @shorsher in #358
• Update manifest for dataexchange-https by @awrichar in #360

Full Changelog: v0.11.1...v0.11.2

Summary

This patch release includes important fixes and hardening for recent features. Major highlights:

• Fixed bugs with some token transfer events being ignored
• Added support for retrieving the URI attached to each non-fungible token
• Updated EthConnect release with significant new features, particularly when using Kafka
• Added initial prometheus metrics for batch pins

Updated Dependencies

• firefly-ethconnect v3.1.0
• firefly-fabconnect v0.9.2
• firefly-dataexchange-https v0.9.2
• firefly-tokens-erc1155 v0.10.1
• firefly-ui v0.4.2

What's Changed

• Update dependencies for v0.11.0 release by @nguyer in #333
• Remove "skipEventstreamInit" parameters by @awrichar in #329
• add batchpin counter prometheus metric by @shorsher in #334
• Add token URI by @awrichar in #335
• Split event stream helpers into separate files by @awrichar in #323
• Add "Getting Started" documentation for tokens by @awrichar in #306
• Move definitions of datatypes and tokens to their proper namespace by @nguyer in #312
• Leverage new token transfer id from plugin by @awrichar in #336
• bump ui to version 0.4.2 by @shorsher in #339
• Update dependencies for v0.11.1 release by @awrichar in #342

Full Changelog: v0.11.0...v0.11.1

Summary

This release brings a lot of fixes and some exciting new features. Here are some of the highlights:

• Significantly hardened and enhanced token functionality
  • New, more flexible API structure (old APIs deprecated)
  • Coordination of token mint/burn/transfer events with attached messages and off-chain data
  • Proper handling for failed transfers and other error scenarios
• Major web UI overhaul
  • New pluggable architecture, introducing modules
    • Existing FireFly UI is now the data module
    • Added two new modules, tokens and network-map
  • Reworked navigation
  • Updated dashboard
  • Enhanced filtering
• Optimized database operations for increased transactional throughput
• Fixed PostgreSQL database migrations

What's Changed

• Move e2e to new routes, and add private/broadcast strong datatype tests by @peterbroadhurst in #260
• Added more tests to the Fabric plugin for 100% coverage by @jimthematrix in #261
• Do not allow UUIDs to be used as names by @awrichar in #262
• Fix logic for transfer+message with waitConfirm=true by @awrichar in #258
• Do not use tokenIndex in E2E test for fungible tokens by @awrichar in #264
• Use firefly_e2e instead of firefly-e2e for container names by @awrichar in #266
• Adjust version for firefly-tokens-erc1155 by @awrichar in #267
• Allow nested RunAsGroup calls by @awrichar in #270
• Split asset manager into multiple files by @awrichar in #268
• New GET routes for tokens by @awrichar in #275
• Add extra columns to token tables for UI support by @awrichar in #276
• Additional E2E coverage for tokens by @awrichar in #274
• introduce FFTime.Time() by @shorsher in #277
• pools-by-connector Add connector field to TokenPoolQueryFactory by @eberger727 in #280
• tokens-by-id getTokenPoolByNameOrID() and getTokenTransfersByID() by @eberger727 in #278
• Update GitHub actions to build every merge to main by @nguyer in #286
• Group token database calls into RunAsGroup wherever possible by @awrichar in #273
• Fix PSQL migration with invalid AUTOINCREMENT by @peterbroadhurst in #289
• Add topic+tag index to messages by @peterbroadhurst in #290
• Add namespace field to token accounts and transfers by @awrichar in #287
• Fix duplicate db migration 00037 by @awrichar in #295
• Update DX dependency, and go deps including bluemonday for CVE-2021-42576 by @peterbroadhurst in #285
• Update docs and add new guides by @nguyer in #284
• Collapse message bools into a new "state" field by @awrichar in #296
• Run E2E tests with matrix including fabric by @nguyer in #242
• Use single quotes for strings in postgres migration by @awrichar in #302
• Small doc updates by @awrichar in #301
• Replace BeforeSend callback with a Prepare method by @awrichar in #283
• remove-member-field Removing member field from operations table by @eberger727 in #300
• Add ?fetchdata on messages collection to include data, and make consistent with get by ID by @peterbroadhurst in #299
• Allow did:firefly:org/xyz prefix on group input by @peterbroadhurst in #304
• Add scheduled E2E test using latest services by @nguyer in #292
• Add check before writing to websocket channel by @nguyer in #298
• Add jobs for database migration and auto-registration by @peterbroadhurst in #294
• Update assertion on unit test to pass on Go 1.17 by @nguyer in #303
• Only send transfer message if token transfer is successful by @awrichar in #297
• Consume websocket events in "strong datatype" tests by @awrichar in #309
• POST /tokens/<mint | burn | transfers | pools> by @eberger727 in #307
• Split the notions of "token balances" and "token accounts" by @awrichar in #311
• Always pass key and message hash to fftokens connector by @awrichar in #305
• Use WriteByte instead of WriteRune when possible by @awrichar in #314
• Additional cleanup to support tokens UI by @awrichar in #313
• Prometheus Metrics for Mux Router by @hfuss in #321
• Allow specifying pool by name or ID for transfer requests by @awrichar in #320
• Fix COUNT for SQL queries that use DISTINCT by @awrichar in #319
• add build badge by @shorsher in #327
• Replace token pool "validate tx" logic stubs with a new "activate" action by @awrichar in #317
• Upsert optimization by @peterbroadhurst in #325
• Change JSONB columns to BYTEA - issue : Remove use of JSONB columns #324 by @hypefi in #330
• bump UI version to 0.4.1 by @shorsher in #332
• Update dependencies for v0.11.0 release by @nguyer in #333

New Contributors

• @eberger727 made their first contribution in #280
• @hypefi made their first contribution in #330

Full Changelog: v0.10.1...v0.11.0