Apache Spark 2.0.0 发布了，Apache Spark 是一种与 Hadoop 相似的开源集群计算环境，但是两者之间还存在一些不同之处，这些有用的不同之处使 Spark 在某些工作负载方面表现得更加优越，换句话说，Spark 启用了内存分布数据集，除了能够提供交互式查询外，它还可以优化迭代工作负载。

该版本主要更新APIs，支持SQL 2003，支持R UDF ，增强其性能。300个开发者贡献了2500补丁程序。

Apache Spark 2.0.0 APIs更新记录如下：

• Unifying DataFrame and Dataset: In Scala and Java, DataFrame and Dataset have been unified, i.e. DataFrame is just a type alias for Dataset of Row. In Python and R, given the lack of type safety, DataFrame is the main programming interface.

• SparkSession: new entry point that replaces the old SQLContext and HiveContext for DataFrame and Dataset APIs. SQLContext and HiveContext are kept for backward compatibility.

• A new, streamlined configuration API for SparkSession

• Simpler, more performant accumulator API

• A new, improved Aggregator API for typed aggregation in Datasets

Apache Spark 2.0.0 SQL更新记录如下：

• A native SQL parser that supports both ANSI-SQL as well as Hive QL

• Native DDL command implementations

• Subquery support, including

  • Uncorrelated Scalar Subqueries

  • Correlated Scalar Subqueries

  • NOT IN predicate Subqueries (in WHERE/HAVING clauses)

  • IN predicate subqueries (in WHERE/HAVING clauses)

  • (NOT) EXISTS predicate subqueries (in WHERE/HAVING clauses)

• View canonicalization support

一些新特性：

• Native CSV data source, based on Databricks’ spark-csv module

• Off-heap memory management for both caching and runtime execution

• Hive style bucketing support

• Approximate summary statistics using sketches, including approximate quantile, Bloom filter, and count-min sketch.

性能增强：

• Substantial (2 - 10X) performance speedups for common operators in SQL and DataFrames via a new technique called whole stage code generation.

• Improved Parquet scan throughput through vectorization

• Improved ORC performance

• Many improvements in the Catalyst query optimizer for common workloads

• Improved window function performance via native implementations for all window functions

• Automatic file coalescing for native data sources

更多发布信息，可查看发布说明。

下载地址：http://spark.apache.org/downloads.html