release v2.1.0-rc.1

release v2.0.2

Please be advised that we have withdrawn version v2.0.0 due to a critical bug affecting upgrades of RW clusters from older versions. We sincerely apologize for the inconvenience.

The PRs for the bug fix are:
#18589
#18581

For installation and running instructions, see Get started.

Main changes

SQL features

• Query syntax:
  • Public preview: Supports AS CHANGELOG to convert any stream into an append-only changelog. #17132
  • Supports time travel query to access historical data at a specific point in time. #17665, #17621.
  • Supports CORRESPONDING specification in set operations. #17891
• SQL commands:
  • Breaking change: DECLARE cursor_name SUBSCRIPTION CURSOR is the same as DECLARE cursor_name SUBSCRIPTION CURSOR since now(), which will be consumed from the current time. DECLARE cursor_name SUBSCRIPTION CURSOR FULL will start consuming data from stock. The type of operation has changed to varchar. It is one of Insert, Delete, UpdateInset, or UpdateDelete. #18217
  • Public preview: Supports managing database credentials securely. #17456
  • SHOW CURSORS will return all query cursors in the current session. SHOW SUBSCRIPTION CURSORS will return all subscription cursorS and subscriptions in the current session. #18217
  • Allows ALTER TABLE on tables with generated columns. #17652
  • Allows dropping generated columns from tables created with a schema registry. #17689
  • Supports using scalar functions with list inputs as aggregate functions. #17622
  • Supports altering the backfill rate limit for materialized views. #17911
• SQL functions & operators:
  • Public preview: Supports approx_percentile(). #17814, #17873.
  • Public preview: Supports native map type and related functions for map, and ingesting AVRO MAP type into RisingWave map type. #17986
  • Public preview: Supports scanning a directory of parquet files. #17811
  • Supports pg_index_column_has_property() to query index column properties. #17275
  • Supports continuous timestamp generation in streaming mode. #17371
  • Supports acosd(). #9876
  • Supports function rw_recovery_status() and pg_is_in_recovery() to retrieve the meta node status. #17641
• System catalog:
  • Adds append_only column in RisingWave catalogs rw_tables and rw_materialized_views. #17598
  • Adds RisingWave catalog rw_catalog.rw_secrets. #17726

Connectors

• Public preview: Supports ingesting Avro map type for source connectors. ****#17980
• Public preview: Supports encoding parquet for file source. #17201
• Public preview: Supports batch reading S3 Parquet files. #17625, #17673.
• Supports AWS Glue schema registry with aws.glue.schema_arn parameter. #17605
• Supports creating tables and sources with format upsert encode protobuf. #17624
• Supports ingesting Avro Union type for source connectors. #17485
• Supports reading files compressed in gzip format. #16538
• Adds the option to use a semicolon as the delimiter for CSV encode. #17356
• Uses OpenDAL to connect to S3 object store state backend. #18011
• Public preview: Supports replicating DDL for MySQL CDC source. #17876
• Supports parameter refresh.interval.sec option for S3, GCS, and POSIX sources. #18184
• Supports parameter group.id.prefix for Kafka sources. #18115
• Validates slot name of PostgreSQL CDC sources. #17949
• Supports altering backfill_rate_limit of CDC tables. #17989
• Public preview: Supports sinking data to file systems in parquet format. #17311
• Supports upsert Protobuf type sinks, which requires KEY ENCODE TEXT. #18024
• Adds option jsonb.handling.mode under WITH options for sinks in JSON format. #17693
• Public preview: Supports Azure Blob sinks. #18244
• Public preview: Supports MongoDB sinks. #17102
• Supports Azure Blob file sources. #18295
• Supports glue catalog for iceberg sink and source. #17477
• Adds jdbc.query.timeout for JDBC sinks to set the timeout for queries. #18430
• Changes default Kafka sink message timeout from five seconds to five minutes. #18304
• Adds new parameters retry_on_conflict, batch_size_kb, batch_num_messages, and concurrent_requests for ElasticSearch sink.#17867
• Supports parameter bigquery.retry_times for BigQuery sink. #17237
• Supports parameter bigquery.auto_create_table for BigQuery sink. #17393
• Supports parameter doris.partial_columns for Doris sink. #16821
• Supports ClickHouse sink checkpoint decouple. #17491
• Sets sink decouple as default for all sinks. #18182
• Uses S3's SQS notification to complete the import of data from Snowflake instead of the Snowflake HTTP client. #17627
• Ensures at-least-once delivery semantic and eventual consistency for Kinesis sink. #17983
• Supports backfilling by consuming a fixed snapshot of upstream table and then the upstream data epoch by epoch. #17735

Installation and deployment

• Supports configuring the SQL metastore using username, password, and database separately. #17530
• Supports more seamless scaling-in in Kubernetes deployments. #17802

Cluster configuration changes

• Breaking change: Refactors streaming_rate_limit into source_rate_limit and backfill_rate_limit. #17796
• Breaking change: Adds a default soft and hard limit on actor count per worker parallelism. When the hard limit is reached, streaming workloads will fail. #18383
• Introduces batch.developer.exchange_connection_pool_size and streaming.developer.exchange_connection_pool_size to configure streaming and batch remote exchange between two nodes #17768
• Introduces system parameter license_key used to enable enterprise features. #17396

Fixes

• Deletes related cursors when deleting a subscription. #17232

Full Changelog: v1.10.1...v2.0.1

release v1.10.2

For installation and running instructions, see Get started.

Main changes

SQL features

• Query syntax:
  • Public preview: Supports AS CHANGELOG to convert any stream into an append-only changelog. #17132
  • Supports time travel query to access historical data at a specific point in time. #17665, #17621.
  • Supports CORRESPONDING specification in set operations. #17891
• SQL commands:
  • Breaking change: DECLARE cursor_name SUBSCRIPTION CURSOR is the same as DECLARE cursor_name SUBSCRIPTION CURSOR since now(), which will be consumed from the current time. DECLARE cursor_name SUBSCRIPTION CURSOR FULL will start consuming data from stock. The type of operation has changed to varchar. It is one of Insert, Delete, UpdateInset, or UpdateDelete. #18217
  • Public preview: Supports managing database credentials securely. #17456
  • SHOW CURSORS will return all query cursors in the current session. SHOW SUBSCRIPTION CURSORS will return all subscription cursorS and subscriptions in the current session. #18217
  • Allows ALTER TABLE on tables with generated columns. #17652
  • Allows dropping generated columns from tables created with a schema registry. #17689
  • Supports using scalar functions with list inputs as aggregate functions. #17622
  • Supports altering the backfill rate limit for materialized views. #17911
• SQL functions & operators:
  • Public preview: Supports approx_percentile(). #17814, #17873.
  • Public preview: Supports native map type and related functions for map, and ingesting AVRO MAP type into RisingWave map type. #17986
  • Public preview: Supports scanning a directory of parquet files. #17811
  • Supports pg_index_column_has_property() to query index column properties. #17275
  • Supports continuous timestamp generation in streaming mode. #17371
  • Supports acosd(). #9876
  • Supports function rw_recovery_status() and pg_is_in_recovery() to retrieve the meta node status. #17641
• System catalog:
  • Adds append_only column in RisingWave catalogs rw_tables and rw_materialized_views. #17598
  • Adds RisingWave catalog rw_catalog.rw_secrets. #17726

Connectors

• Public preview: Supports ingesting Avro map type for source connectors. ****#17980
• Public preview: Supports encoding parquet for file source. #17201
• Public preview: Supports batch reading S3 Parquet files. #17625, #17673.
• Supports AWS Glue schema registry with aws.glue.schema_arn parameter. #17605
• Supports creating tables and sources with format upsert encode protobuf. #17624
• Supports ingesting Avro Union type for source connectors. #17485
• Supports reading files compressed in gzip format. #16538
• Adds the option to use a semicolon as the delimiter for CSV encode. #17356
• Uses OpenDAL to connect to S3 object store state backend. #18011
• Public preview: Supports replicating DDL for MySQL CDC source. #17876
• Supports parameter refresh.interval.sec option for S3, GCS, and POSIX sources. #18184
• Supports parameter group.id.prefix for Kafka sources. #18115
• Validates slot name of PostgreSQL CDC sources. #17949
• Supports altering backfill_rate_limit of CDC tables. #17989
• Public preview: Supports sinking data to file systems in parquet format. #17311
• Supports upsert Protobuf type sinks, which requires KEY ENCODE TEXT. #18024
• Adds option jsonb.handling.mode under WITH options for sinks in JSON format. #17693
• Public preview: Supports Azure Blob sinks. #18244
• Public preview: Supports MongoDB sinks. #17102
• #18295
• Supports glue catalog for iceberg sink and source. #17477
• #18304
• Supports parameter bigquery.retry_times for BigQuery sink connector. #17237
• Supports parameter bigquery.auto_create_table for BigQuery sink. #17393
• Supports parameter doris.partial_columns for Doris sink connector. #16821
• Supports ClickHouse sink checkpoint decouple. #17491
• Sets sink decouple as default for all sinks. #18182
• Uses S3's SQS notification to complete the import of data from Snowflake instead of the Snowflake HTTP client. #17627
• Ensures at-least-once delivery semantic and eventual consistency for Kinesis sink. #17983
• Supports backfilling by consuming a fixed snapshot of upstream table and then the upstream data epoch by epoch. #17735

Installation and deployment

• Supports configuring the SQL metastore using username, password, and database separately. #17530
• Supports more seamless scaling-in in Kubernetes deployments. #17802

Cluster configuration changes

• Breaking change: Refactors streaming_rate_limit into source_rate_limit and backfill_rate_limit. #17796
• Introduces batch.developer.exchange_connection_pool_size and streaming.developer.exchange_connection_pool_size to configure streaming and batch remote exchange between two nodes #17768
• Introduces system parameter license_key used to enable enterprise features. #17396

Fixes

• Deletes related cursors when deleting a subscription. fix(subscription): fix drop subscription not clear cursor

Full Changelog: v1.10.1...v2.0.0-rc.1

release v1.10.1

For installation and running instructions, see Get started.

Main changes

SQL features

• SQL commands:
  • Supports specifying the authorization and omitting the schema name when creating a schema. #16806
  • Supports session window in batch and emit-on-window-close mode. #17098
  • Supports fetching multiple rows from subscription cursor. #16764
• SQL functions & operators:
  • Supports has_table_privilege(), has_schema_privilege(), and has_any_column_privilege(). #16674
  • Supports quote_literal() and quote_nullable(). #16807
  • Supports pg_get_keywords(). #17033
  • Supports jsonb_set(). #17124
  • Supports user-defined aggregate functions for embedded Python and JavaScript UDFs. #16874
  • Allows the specified time zone of AT TIME ZONE to be non-literal. #17395
• System catalog:
  • Supports rw_catalog.actor_id_to_ddl and rw_catalog.fragment_id_to_ddl. #17229

Connectors

• Avro schemas with "default": "NaN" and positive and negative infinities, are supported as float and double types. #17309
• Supports ingesting simple AVRO MAP types as JSONB. #16948
• Supports ingesting avro uuid types as varchar. #17069
• Supports ingesting avro with internal Ref types. #17052
• Adds aws prefix to AWS related parameters when creating a source or sink. #16671
• Supports using AWS IAM to connect to Amazon MSK. #16625
• Adds pubsub.parallelism under the WITH option for Google PubSub source. #16733
• Supports INCLUDE TIMESTAMP [AS] clause for MySQL, PostgreSQL, and MongoDB CDC tables.#16833
• Supports additional metadata columns for CDC tables. #17051
• Automatically maps upstream table schema when creating MySQL and PostgreSQL tables. #16986
• Sets a network timeout for JDBC sink connections. #17244
• Enables sink decouple by default for Kafka, Kinesis, Pulsar, Google Pub/Sub, NATS, MQTT, ClickHouse sinks. #17221
• Supports the KEY ENCODE clause when creating a sink. #16377
• Supports FORMAT PLAIN ENCODE AVRO for Kafka sinks. #17216
• Supports DynamoDB sink. #16670
• Supports Microsoft SQL Server sinks for self-hosted SQL Server and Azure SQL. #17154
• Supports OpenSearch sink. #16330
• Supports checkpoint decouple for StarRocks sinks. #16816
• Supports checkpoint decouple for Delta Lake sinks. #16777
• Supports sinking serial types. #16969

Cluster configuration changes

• Sets arrangement backfill as the default. #14846
• Supports spill hash join to avoid OOM issue. #17122
• Supports spill hash aggregation for batch queries. #16771
• Changes the algorithm that calculates the reserve memory size. #16992

Fixes

• Improves error message and location of cursor. #16959
• Improves error message when trying to create CDC source with columns. #16636
• Allows GRANT and REVOKE privileges on views. #16699

Full Changelog: v1.9.1...v1.10.0

release v1.10.0-rc.3

release v1.10.0-rc.2

release v1.10.0-rc.1