XML Parser to parse XML data based on Path

  Plugin Name : XMLParser
  Java Class  : org.apache.hive.custom.udfs.text.XMLParser
  Status      : Active
  Version     : v1.0.0
  Author      : Rishu Shrivastava
  Last Updated: October 4, 2020

Download

Download the custom UDF jar from here.

Description

XMLParser is the custom user defined function that would parse a xml data. A xml is a markup language and one of the most popular data sharing format.

In order to read data from an xml file/data, one has an option of using XPath. Xpath is a query language for xml that extracts the information out of an xml file.

XMLParser takes XML data and XPath query as an input to parse the data and generate the output.

Example of XML and XPath:

<?xml version = "1.0"?>
<epl>
  <player id="1">
     <name>Harry Kane</name>
     <Age>27</Age>
     <club>TOT</club>
  </player>
  <player rollno = "2">
     <name>Bruno Fernandes</name>
     <Age>28</Age>
     <club>MUN</club>
  </player>
</epl>

• In order to get the list of players in a epl xml tag, use the Xpath query as /epl/player/name/text(). This will result in the list of players as: ["Harry Kane","Bruno Fernandes"]

• In order to get the list of players in club MUN, you can use the Xpath query as /epl/player[club='MUN']/text(). This will output the result as : Bruno Fernandes

Usage

_FUNC_(xml_data:String, xpath:String)

• @Input Params:
  • Xml_data of type String
  • Xpath of type string.
• @Output: An Array List of the result from the Xpath.

e.g.

• xmlparser("<epl><player id="1"><name>Harry Kane</name></player><player id="2"><name>Bruno Fernandes</name></player></epl>","/epl/player/@/*")

• xmlparser(data,"/epl/player/@/*") where data is the field having the xml format data.

Deployment

In order to use and plugin the UDFs, follow the below steps:

1. Download the hive-custom-udfs-*.jar onto your system or edge-node.

2. Upload the downloaded jar onto HDFS location. Use the below command:

   $ hadoop fs -copyFromLocal hive-custom-udfs-*.jar <path of your hdfs directory>
   

3. Login to Hive SHELL and add the jar to the Hive Classpath and define a function, as below:

   $ beeline -u jdbc:hive2://localhost:10000
   hive> create function xmlparser as 'org.apache.hive.custom.udfs.text.XMLParser' using JAR 'hdfs:///<path of hdfs directory>/hive-custom-udfs-*.jar';
   

   • The name of the function is: xmlparser
   • The function Java class is the class path: org.apache.hive.custom.udfs.text.XMLParser
   • The HDFS path: It can be any location in HDFS. e.g. /user/cloudera/udfs

4. Use the function in hive

   hive> SELECT xmlparser("<epl><player id="1"><name>Harry Kane</name></player><player id="2"><name>Bruno Fernandes</name></player></epl>","/epl/player/@/*");
   

This results ["1","2"] which are the id's of the players.

Snapshots

• Deployment of the Jar to class path:

• Querying XML data from HUE

• Querying on Hive CLI