JetBrains has updated the Big Data Tools. In version 1.6, the plug-in for the IDEs IntelliJ IDEA Ultimate, DataGrip, DataSpell and PyCharm Professional offers bug fixes and general improvements as well as some new functions – including the monitoring of Apache Flink jobs directly from the development environment. The integration of the Hive Metastore is also new.
Monitor jobs and tasks in Apache Flink
The Big Data Tools, which were first presented in 2020 and have been available in a version released for productive use since the beginning of last year, are primarily designed for connection to Apache Zeppelin notebooks, with which – comparable to Jupyter notebooks – interactive data analysis can be carried out. In addition, developers can use the plugin to monitor applications on Spark and Hadoop. With the update to version 1.6, JetBrains extends the monitoring functions to Apache Flink jobs. New jobs can also be created, submitted and monitored directly from the IDE. The Big Data Tools can be used to keep an eye on both the Job Manager and the Task Manager.
For the first time, an integration of the Hive Metastore is also available. You can connect to the metastore from within the IDE to browse Hive catalogs, columns, and tables. Metadata storage partitions can also be viewed and detailed information about the Hive Metastore can be displayed via a special tool window. In addition, the Big Data Tools now master code completion for Spark SQL, which is based on metadata in the Hive Metastore.
More filter options
The JetBrains team also overhauled the Amazon EMR (Elastic MapReduce) integration and added the ability to monitor clusters and nodes. A tree view is also available for the EMR clusters, which offers various filter options. With Kafka monitoring, developers will also find extended filter options for topics.
For a full overview and more details on what’s new in Big Data Tools 1.6, see the JetBrains blog. The free plug-in is available in the Marketplace, but requires one of the commercial JetBrains IDEs IntelliJ IDEA Ultimate, DataGrip, DataSpell or PyCharm Professional.