HomeTech GiantsAmazonJetBrains' Big Data Tools 1.6 keeps track of Flink jobs

JetBrains’ Big Data Tools 1.6 keeps track of Flink jobs

Published on

- Advertisement -

Big Data tools 1.6, a plug-in for accessing Zeppelin notebooks, can now also monitor Apache flink and integrate the Hive Metastore.


JetBrains has updated the Big Data Tools. In version 1.6, the plug-in for the IDEs IntelliJ IDEA Ultimate, DataGrip, DataSpell and PyCharm Professional offers bug fixes and general improvements as well as some new functions – including the monitoring of Apache Flink jobs directly from the development environment. The integration of the Hive Metastore is also new.

- Advertisement -

The Big Data Tools, which were first presented in 2020 and have been available in a version released for productive use since the beginning of last year, are primarily designed for connection to Apache Zeppelin notebooks, with which – comparable to Jupyter notebooks – interactive data analysis can be carried out. In addition, developers can use the plugin to monitor applications on Spark and Hadoop. With the update to version 1.6, JetBrains extends the monitoring functions to Apache Flink jobs. New jobs can also be created, submitted and monitored directly from the IDE. The Big Data Tools can be used to keep an eye on both the Job Manager and the Task Manager.

For the first time, an integration of the Hive Metastore is also available. You can connect to the metastore from within the IDE to browse Hive catalogs, columns, and tables. Metadata storage partitions can also be viewed and detailed information about the Hive Metastore can be displayed via a special tool window. In addition, the Big Data Tools now master code completion for Spark SQL, which is based on metadata in the Hive Metastore.

The JetBrains team also overhauled the Amazon EMR (Elastic MapReduce) integration and added the ability to monitor clusters and nodes. A tree view is also available for the EMR clusters, which offers various filter options. With Kafka monitoring, developers will also find extended filter options for topics.

For a full overview and more details on what’s new in Big Data Tools 1.6, see the JetBrains blog. The free plug-in is available in the Marketplace, but requires one of the commercial JetBrains IDEs IntelliJ IDEA Ultimate, DataGrip, DataSpell or PyCharm Professional.

- Advertisement -

- Advertisement -

Latest articles

microSDXC: The first memory cards with 2 TB are coming

Kioxia has the first memory card prototypes based on the microSDXC standard with a...

Google wants to translate search results more often and personalize them more

Google is announcing many innovations relating to the search for information – from concealing...

More like this