报价宝 › 科技 ›

Apache Zeppelin使用入门指南：程式设计

报价宝来源：baojiabao.com 发布时间：2019-11-02 14:58:00 10月28日更新

报价宝综合消息Apache Zeppelin使用入门指南：程式设计

使用Apache Zeppelin

编译和启动完Zeppelin相关的程序之后，我们就可以来使用Zeppelin了。我们进入到https://www.iteblog.com:8080页面，我们可以在页面上直接操作Zeppelin，依次选择Notebook->Create new note，然后会弹出一个对话方块，我们在Note Name里面随便输入一个名字，这里我输入iteblog，然后点选Create Note就可以建立一个新的Notebook了。我们可以在新建的Notebook里面输入相关的程式码进行测试：

sc.version

sqlc

res26: String = 1.5.2

res27: org.apache.spark.sql.SQLContext = [email protected]

和Spark Shell一样，Zeppelin会初始化好SparkContext和SQLContext物件，分别命名为sc和sqlc，我们可以直接在里面使用到它。接下来我们来在Zeppelin里面载入HDFS上面的资料，如下：

sc.textFile("hdfs://www.iteblog.com/tmp/json").count

res29: Long = 200

Took 0 seconds (outdated)

我们再来使用sqlc物件读取上面的json档案来建立一个DataFrame：

val profilesJsonRdd =sqlc.jsonFile("hdfs://www.iteblog.com/tmp/json")

val profileDF=profilesJsonRdd.toDF()

val iteblog = profileDF.selectExpr("_id", "address", "age", "email")

iteblog.show()

profileDF.registerTempTable("profiles")

profilesJsonRdd: org.apache.spark.sql.DataFrame = [_id: string, about: string, address: string, age: bigint, company: string, email: string, eyeColor: string, favoriteFruit: string, gender: string, name: string, phone: string, registered: string, tags: array]

profileDF: org.apache.spark.sql.DataFrame = [_id: string, about: string, address: string, age: bigint, company: string, email: string, eyeColor: string, favoriteFruit: string, gender: string, name: string, phone: string, registered: string, tags: array]

root

|-- _id: string (nullable = true)

|-- about: string (nullable = true)

|-- address: string (nullable = true)

|-- age: long (nullable = true)

|-- company: string (nullable = true)

|-- email: string (nullable = true)

|-- eyeColor: string (nullable = true)

|-- favoriteFruit: string (nullable = true)

|-- gender: string (nullable = true)

|-- name: string (nullable = true)

|-- phone: string (nullable = true)

|-- registered: string (nullable = true)

|-- tags: array (nullable = true)

| |-- element: string (containsNull = true)

iteblog: org.apache.spark.sql.DataFrame = [_id: string, address: string, age: bigint, email: string]

+--------------------+--------------------+---+--------------------+

| _id| address|age| email|

+--------------------+--------------------+---+--------------------+

+--------------------+--------------------+---+--------------------+

only showing top 20 rows

Took 1 seconds

需要注意的是，因为Zeppelin会自动地初始化好SparkContext和SQLContext，所以我们不能再显示地建立SparkContext和SQLContext。如果你显示地建立了SQLContext，而且使用它来注册一张临时表，当你下一次使用这个临时表的时候将会出现以下的异常资讯：no such table List ([iteblog])

下面我们来使用上面注册的临时表，Zeppelin自带了SQL Interpreter，所以我们可以直接在上面编写SQL语句：

%sql select eyeColor, count(eyeColor) as count from profiles where

gender='male' group by eyeColor