Apache Flink 1.16.1 Release Announcement

Apache Flink 1.16.1 Release Announcement

January 30, 2023 - Martijn Visser (@martijnvisser82)

The Apache Flink Community is pleased to announce the first bug fix release of the Flink 1.16 series.

This release includes 84 bug fixes, vulnerability fixes, and minor improvements for Flink 1.16. Below you will find a list of all bugfixes and improvements (excluding improvements to the build infrastructure and build stability). For a complete list of all changes see: JIRA.

We highly recommend all users upgrade to Flink 1.16.1.

Release Artifacts #

Maven Dependencies # 

<dependency>
  <groupId>org.apache.flink</groupId>
  <artifactId>flink-java</artifactId>
  <version>1.16.1</version>
</dependency>
<dependency>
  <groupId>org.apache.flink</groupId>
  <artifactId>flink-streaming-java</artifactId>
  <version>1.16.1</version>
</dependency>
<dependency>
  <groupId>org.apache.flink</groupId>
  <artifactId>flink-clients</artifactId>
  <version>1.16.1</version>
</dependency>

Binaries #

You can find the binaries on the updated Downloads page.

Docker Images #

PyPi #

Upgrade Notes #

FLINK-28988 - Incorrect result for filter after temporal join The filter will not be pushed down into both inputs of the event time temporal join. This may cause incompatible plan changes compared to Flink 1.16.0, e.g., when the left input is an upsert source (like upsert-kafka connector), the query plan will remove the ChangelogNormalize node in Flink 1.16.1, while it did appear in 1.16.0.

FLINK-29849 - Event time temporal join on an upsert source may produce incorrect execution plan This resolves the correctness issue when doing an event time temporal join with a versioned table backed by an upsert source. When the right input of the join is an upsert source, it no longer generates a ChangelogNormalize node for it. This is an incompatible plan change compared to 1.16.0

FLINK-30383 - UseLogicalIdentifier makes datadog consider metric as custom The Datadog reporter now adds a “flink.” prefix to metric identifiers if “useLogicalIdentifier” is enabled. This is required for these metrics to be recognized as Flink metrics, not custom ones.

Release Notes #

Bug

  • [FLINK-16582] - NettyBufferPoolTest may have warns on NettyBuffer leak
  • [FLINK-26037] - TaskManagerRunner may crash during shutdown sequence
  • [FLINK-26890] - DynamoDB consumer error consuming partitions close to retention
  • [FLINK-27341] - TaskManager running together with JobManager are bind to 127.0.0.1
  • [FLINK-27944] - IO metrics collision happens if a task has union inputs
  • [FLINK-28102] - Flink AkkaRpcSystemLoader fails when temporary directory is a symlink
  • [FLINK-28526] - Fail to lateral join with UDTF from Table with timstamp column
  • [FLINK-28695] - Fail to send partition request to restarted taskmanager
  • [FLINK-28742] - Table.to_pandas fails with lit("xxx")
  • [FLINK-28786] - Cannot run PyFlink 1.16 on MacOS with M1 chip
  • [FLINK-28863] - Snapshot result of RocksDB native savepoint should have empty shared-state
  • [FLINK-28960] - Pulsar throws java.lang.NoClassDefFoundError: javax/xml/bind/annotation/XmlElement
  • [FLINK-28988] - Incorrect result for filter after temporal join
  • [FLINK-29231] - PyFlink UDAF produces different results in the same sliding window
  • [FLINK-29234] - Dead lock in DefaultLeaderElectionService
  • [FLINK-29298] - LocalBufferPool request buffer from NetworkBufferPool hanging
  • [FLINK-29479] - Support whether using system PythonPath for PyFlink jobs
  • [FLINK-29539] - dnsPolicy in FlinkPod is not overridable
  • [FLINK-29615] - MetricStore does not remove metrics of nonexistent subtasks when adaptive scheduler lowers job parallelism
  • [FLINK-29627] - Sink - Duplicate key exception during recover more than 1 committable.
  • [FLINK-29677] - Prevent dropping the current catalog
  • [FLINK-29728] - TablePlanner prevents Flink from starting is working directory is a symbolic link
  • [FLINK-29749] - flink info command support dynamic properties
  • [FLINK-29781] - ChangelogNormalize uses wrong keys after transformation by WatermarkAssignerChangelogNormalizeTransposeRule
  • [FLINK-29803] - Table API Scala APIs lack proper source jars
  • [FLINK-29817] - Published metadata for apache-flink in pypi are inconsistent and causes poetry to fail
  • [FLINK-29827] - [Connector][AsyncSinkWriter] Checkpointed states block writer from sending records
  • [FLINK-29839] - HiveServer2 endpoint doesn't support TGetInfoType value 'CLI_ODBC_KEYWORDS'
  • [FLINK-29849] - Event time temporal join on an upsert source may produce incorrect execution plan
  • [FLINK-29857] - Fix failure to connect to 'HiveServer2Endpoint' when using hive3 beeline
  • [FLINK-29899] - Stacktrace printing in DefaultExecutionGraphCacheTest is confusing maven test log output
  • [FLINK-29923] - Hybrid Shuffle may face deadlock when running a task need to execute big size data
  • [FLINK-29927] - AkkaUtils#getAddress may cause memory leak
  • [FLINK-30030] - Unexpected behavior for overwrite in Hive dialect
  • [FLINK-30133] - HadoopModuleFactory creates error if the security module cannot be loaded
  • [FLINK-30168] - PyFlink Deserialization Error with Object Array
  • [FLINK-30189] - HsSubpartitionFileReader may load data that has been consumed from memory
  • [FLINK-30239] - The flame graph doesn't work due to groupExecutionsByLocation has bug
  • [FLINK-30304] - Possible Deadlock in Kinesis/Firehose/DynamoDB Connector
  • [FLINK-30308] - ClassCastException: class java.io.ObjectStreamClass$Caches$1 cannot be cast to class java.util.Map is showing in the logging when the job shutdown
  • [FLINK-30334] - SourceCoordinator error splitRequest check cause HybridSource loss of data and hang
  • [FLINK-30359] - Encountered NoClassDefFoundError when using flink-sql-connector-elasticsearch6
  • [FLINK-30366] - Python Group Agg failed in cleaning the idle state
  • [FLINK-30525] - Cannot open jobmanager configuration web page
  • [FLINK-30558] - The metric 'numRestarts' reported in SchedulerBase will be overridden by metric 'fullRestarts'
  • [FLINK-30637] - In linux-aarch64 environment, using “is” judgment to match the window type of overwindow have returned incorrect matching results

Improvement

  • [FLINK-27327] - Add description about changing max parallelism explicitly leads to state incompatibility
  • [FLINK-29134] - fetch metrics may cause oom(ThreadPool task pile up)
  • [FLINK-29155] - Improve default config of grpcServer in Process Mode
  • [FLINK-29244] - Add metric lastMaterializationDuration to ChangelogMaterializationMetricGroup
  • [FLINK-29458] - When two tables have the same field, do not specify the table name,Exception will be thrown：SqlValidatorException :Column 'currency' is ambiguous
  • [FLINK-29639] - Add ResourceId in TransportException for debugging
  • [FLINK-29693] - MiniClusterExtension should respect DEFAULT_PARALLELISM if set
  • [FLINK-29834] - Clear static Jackson TypeFactory cache on CL release
  • [FLINK-29966] - Replace and redesign the Python api documentation base
  • [FLINK-30016] - Update Flink 1.16 release notes about updated oshi-core
  • [FLINK-30116] - Don't Show Env Vars in Web UI
  • [FLINK-30183] - We should add a proper error message in case the deprecated reflection-based instantiation of a reporter is triggered
  • [FLINK-30357] - Wrong link in connector/jdbc doc.
  • [FLINK-30436] - Integrate Opensearch connector docs into Flink docs v1.16
  • [FLINK-30592] - The unsupported hive version is not deleted on the hive overview document
  • [FLINK-30633] - Update AWS SDKv2 to v2.19.14
  • [FLINK-30724] - Update doc of kafka per-partition watermark to FLIP-27 source

Technical Debt

  • [FLINK-27731] - Remove Hugo Modules integration
  • [FLINK-29157] - Clarify the contract between CompletedCheckpointStore and SharedStateRegistry
  • [FLINK-29957] - Rework connector docs integration
  • [FLINK-29958] - Add new connector_artifact shortcode
  • [FLINK-29972] - Pin Flink docs to Elasticsearch Connector 3.0.0
  • [FLINK-29973] - connector_artifact should append Flink minor version
  • [FLINK-30291] - Integrate flink-connector-aws into Flink docs
  • [FLINK-30382] - Flink 1.16 to integrate KDS/KDF docs from flink-connector-aws
  • [FLINK-30383] - UseLogicalIdentifier makes datadog consider metric as custom