class CassandraCatalog extends CatalogPlugin with TableCatalog with SupportsNamespaces with Logging
A Spark Sql Catalog for inter-operation with Cassandra
Namespaces naturally map to C* Keyspaces, but they are always only a single element deep.
- Alphabetic
- By Inheritance
- CassandraCatalog
- Logging
- SupportsNamespaces
- TableCatalog
- CatalogPlugin
- AnyRef
- Any
- Hide All
- Show All
- Public
- All
Instance Constructors
- new CassandraCatalog()
Abstract Value Members
-
abstract
def
alterNamespace(arg0: Array[String], arg1: <repeated...>[NamespaceChange]): Unit
- Definition Classes
- SupportsNamespaces
- Annotations
- @throws( ... ) @transient()
-
abstract
def
alterTable(arg0: Identifier, arg1: <repeated...>[TableChange]): Table
- Definition Classes
- TableCatalog
- Annotations
- @throws( ... ) @transient()
Concrete Value Members
-
final
def
!=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
final
def
##(): Int
- Definition Classes
- AnyRef → Any
-
final
def
==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- def alterNamespace(namespace: Array[String], changes: NamespaceChange*): Unit
-
def
alterTable(ident: Identifier, changes: TableChange*): Table
Limited to Adding and Removing Normal Columns and Setting Properties
Limited to Adding and Removing Normal Columns and Setting Properties
The API expects that if any change is rejected, we should not apply any changes. This is basically impossible for us since we cannot batch together all of our DDL changes into a single CQL statement. To ameliorate this we perform as many checks as we can up front to try to avoid beginning our DDL requests which we know will fail.
We break up DDL into 3 phases Set Properties Remove Columns Add Columns
-
final
def
asInstanceOf[T0]: T0
- Definition Classes
- Any
-
def
clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws( ... ) @native() @HotSpotIntrinsicCandidate()
-
def
createNamespace(namespace: Array[String], metadata: Map[String, String]): Unit
Builds a Cassandra Keyspace based off the metadata provided by SparkSQL.
Builds a Cassandra Keyspace based off the metadata provided by SparkSQL. There are currently only two available replication strategies and their syntax mimics CQL
"class" -> "NetworkTopologyStrategy", "dcName" -> "rfNum"
All additional NTS properties will be considered datacenter Names (except durable_writes and replication_factor)
or
"class" -> "SimpleStrategy", "replication_factor" -> "rfNum"
In either case
"durable_writes" -> "true|false"
is also allowed to disable the commitlog (see Cassandra Docs defaults to true)
- Definition Classes
- CassandraCatalog → SupportsNamespaces
-
def
createTable(ident: Identifier, schema: StructType, partitions: Array[Transform], properties: Map[String, String]): Table
Creates a Cassandra Table
Creates a Cassandra Table
Uses properties "partition_key" and "clustering_key" to set partition key and clustering key respectively for each of these properties.
Partition key is defined as a string of comma-separated column identifiers: "col_a, col_b, col_b" Clustering key is defined as a string of comma-separated column identifiers optionally marked with clustering order: "col_a.ASC, col_b.DESC"
Additional options can be specified as table properties For Example caching='{keys=ALL,rows_per_partition=42}', default_time_to_live='33', compaction='{class=SizeTieredCompactionStrategy,bucket_high=42}'
- Definition Classes
- CassandraCatalog → TableCatalog
-
def
defaultNamespace(): Array[String]
- Definition Classes
- CatalogPlugin
-
def
dropNamespace(namespace: Array[String]): Boolean
- Definition Classes
- CassandraCatalog → SupportsNamespaces
-
def
dropTable(ident: Identifier): Boolean
- Definition Classes
- CassandraCatalog → TableCatalog
-
final
def
eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
def
equals(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
final
def
getClass(): Class[_]
- Definition Classes
- AnyRef → Any
- Annotations
- @native() @HotSpotIntrinsicCandidate()
-
def
hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @native() @HotSpotIntrinsicCandidate()
-
def
initialize(name: String, options: CaseInsensitiveStringMap): Unit
- Definition Classes
- CassandraCatalog → CatalogPlugin
-
def
invalidateTable(arg0: Identifier): Unit
- Definition Classes
- TableCatalog
-
final
def
isInstanceOf[T0]: Boolean
- Definition Classes
- Any
-
def
isTraceEnabled(): Boolean
- Attributes
- protected
- Definition Classes
- Logging
-
def
listNamespaces(namespace: Array[String]): Array[Array[String]]
Since we only allow single depth keyspace identifiers in C* we always either return an empty list of namespaces or throw a NoSuchNamespaceException
Since we only allow single depth keyspace identifiers in C* we always either return an empty list of namespaces or throw a NoSuchNamespaceException
- Definition Classes
- CassandraCatalog → SupportsNamespaces
-
def
listNamespaces(): Array[Array[String]]
- Definition Classes
- CassandraCatalog → SupportsNamespaces
-
def
listTables(namespace: Array[String]): Array[Identifier]
- Definition Classes
- CassandraCatalog → TableCatalog
-
def
loadNamespaceMetadata(namespace: Array[String]): Map[String, String]
- Definition Classes
- CassandraCatalog → SupportsNamespaces
-
def
loadTable(ident: Identifier): Table
- Definition Classes
- CassandraCatalog → TableCatalog
-
def
log: Logger
- Attributes
- protected
- Definition Classes
- Logging
-
def
logDebug(msg: ⇒ String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logDebug(msg: ⇒ String): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logError(msg: ⇒ String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logError(msg: ⇒ String): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logInfo(msg: ⇒ String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logInfo(msg: ⇒ String): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logName: String
- Attributes
- protected
- Definition Classes
- Logging
-
def
logTrace(msg: ⇒ String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logTrace(msg: ⇒ String): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logWarning(msg: ⇒ String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logWarning(msg: ⇒ String): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
name(): String
- Definition Classes
- CassandraCatalog → CatalogPlugin
-
def
namespaceExists(arg0: Array[String]): Boolean
- Definition Classes
- SupportsNamespaces
-
final
def
ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
final
def
notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native() @HotSpotIntrinsicCandidate()
-
final
def
notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native() @HotSpotIntrinsicCandidate()
-
def
purgeTable(arg0: Identifier): Boolean
- Definition Classes
- TableCatalog
- Annotations
- @throws( ... )
-
def
renameTable(oldIdent: Identifier, newIdent: Identifier): Unit
- Definition Classes
- CassandraCatalog → TableCatalog
-
final
def
synchronized[T0](arg0: ⇒ T0): T0
- Definition Classes
- AnyRef
-
def
tableExists(arg0: Identifier): Boolean
- Definition Classes
- TableCatalog
-
def
toString(): String
- Definition Classes
- AnyRef → Any
-
final
def
wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
-
final
def
wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... ) @native()
-
final
def
wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
Deprecated Value Members
-
def
finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws( classOf[java.lang.Throwable] ) @Deprecated @deprecated
- Deprecated
(Since version ) see corresponding Javadoc for more information.