package writer
Contains components for writing RDDs to Cassandra
- Alphabetic
- By Inheritance
- writer
- AnyRef
- Any
- Hide All
- Show All
- Public
- All
Type Members
-
class
AsyncExecutor[T, R] extends Logging
Asynchronously executes tasks but blocks if the limit of unfinished tasks is reached.
- case class AsyncStatementWriter[T](connector: CassandraConnector, writeConf: WriteConf, tableDef: TableDef, preparedStatement: PreparedStatement, groupingBatchBuilderBase: GroupingBatchBuilderBase[T], maybeRateLimit: (RichStatement) ⇒ Unit, successHandler: Option[Handler[RichStatement]] = None, failureHandler: Option[Handler[RichStatement]] = None) extends Closeable with Logging with Product with Serializable
- sealed trait BatchGroupingKey extends AnyRef
-
class
CassandraRowWriter extends RowWriter[CassandraRow]
A RowWriter that can write CassandraRow objects.
-
class
DefaultRowWriter[T] extends RowWriter[T]
A
RowWriter
suitable for saving objects mappable by a ColumnMapper.A
RowWriter
suitable for saving objects mappable by a ColumnMapper. Can save case class objects, java beans and tuples. -
trait
LowPriorityRowWriterFactoryImplicits extends AnyRef
Provides a low-priority implicit
RowWriterFactory
able to write objects of any class for which a ColumnMapper is defined. - case class NullKeyColumnException(columnName: String) extends NullPointerException with Product with Serializable
- case class PerRowWriteOptionValue[T](placeholder: String) extends WriteOptionValue[T] with Product with Serializable
- class QueryExecutor extends AsyncExecutor[RichStatement, AsyncResultSet]
-
class
RateLimiter extends Serializable
A leaking bucket rate limiter.
A leaking bucket rate limiter. It can be used to limit rate of anything, but typically it is used to limit rate of data transfer.
It starts with an empty bucket. When packets arrive, they are added to the bucket. The bucket has a constant size and is leaking at a constant rate. If the bucket overflows, the thread is delayed by the amount of time proportional to the amount of the overflow.
This class is thread safe and lockless.
-
class
ReplicaLocator[T] extends Serializable with Logging
A utility class for determining the Replica Set (Ip Addresses) of a particular Cassandra Row.
A utility class for determining the Replica Set (Ip Addresses) of a particular Cassandra Row. Used by the com.datastax.spark.connector.RDDFunctions.keyByCassandraReplica method. Uses the Java Driver to obtain replica information.
- trait RichStatement extends AnyRef
-
trait
RowWriter[T] extends Serializable
RowWriter
knows how to extract column names and values from custom row objects and how to convert them to values that can be written to Cassandra.RowWriter
knows how to extract column names and values from custom row objects and how to convert them to values that can be written to Cassandra.RowWriter
is required to apply any user-defined data type conversion. -
trait
RowWriterFactory[T] extends AnyRef
Creates instances of RowWriter objects for the given row type
T
.Creates instances of RowWriter objects for the given row type
T
.RowWriterFactory
is the trait you need to implement if you want to support row representations which cannot be simply mapped by a ColumnMapper. -
class
SqlRowWriter extends RowWriter[Row]
A RowWriter that can write SparkSQL
Row
objects. - case class StaticWriteOptionValue[T](value: T) extends WriteOptionValue[T] with Product with Serializable
- case class TTLOption(value: WriteOptionValue[Int]) extends WriteOption[Int] with Product with Serializable
-
class
TableWriter[T] extends Serializable with Logging
Writes RDD data into given Cassandra table.
Writes RDD data into given Cassandra table. Individual column values are extracted from RDD objects using given RowWriter Then, data are inserted into Cassandra with batches of CQL INSERT statements. Each RDD partition is processed by a single thread.
- case class TimestampOption(value: WriteOptionValue[Long]) extends WriteOption[Long] with Product with Serializable
- abstract class WritableToCassandra[T] extends AnyRef
-
case class
WriteConf(batchSize: BatchSize = BatchSize.Automatic, batchGroupingBufferSize: Int = ..., batchGroupingKey: BatchGroupingKey = WriteConf.BatchLevelParam.default, consistencyLevel: ConsistencyLevel = ..., ifNotExists: Boolean = WriteConf.IfNotExistsParam.default, ignoreNulls: Boolean = WriteConf.IgnoreNullsParam.default, parallelismLevel: Int = ..., throughputMiBPS: Option[Double] = ..., ttl: TTLOption = TTLOption.defaultValue, timestamp: TimestampOption = TimestampOption.defaultValue, taskMetricsEnabled: Boolean = WriteConf.TaskMetricsParam.default, executeAs: Option[String] = None) extends Product with Serializable
Write settings for RDD
Write settings for RDD
- batchSize
approx. number of bytes to be written in a single batch or exact number of rows to be written in a single batch;
- batchGroupingBufferSize
the number of distinct batches that can be buffered before they are written to Cassandra
- batchGroupingKey
which rows can be grouped into a single batch
- consistencyLevel
consistency level for writes, default LOCAL_QUORUM
- ifNotExists
inserting a row should happen only if it does not already exist
- parallelismLevel
number of batches to be written in parallel
- ttl
the default TTL value which is used when it is defined (in seconds)
- timestamp
the default timestamp value which is used when it is defined (in microseconds)
- taskMetricsEnabled
whether or not enable task metrics updates (requires Spark 1.2+)
- sealed trait WriteOption[+T] extends AnyRef
- sealed trait WriteOptionValue[+T] extends AnyRef
Value Members
- object AsyncExecutor
- object BatchGroupingKey
- object CassandraRowWriter extends Serializable
- object DefaultRowWriter extends Serializable
- object DefaultValue extends WriteOptionValue[Nothing] with Product with Serializable
-
object
ObjectSizeEstimator
Estimates amount of memory required to serialize Java/Scala objects
- object QueryExecutor
-
object
ReplicaLocator extends Serializable
Helper methods for mapping a set of data to their relative locations in a Cassandra Cluster.
- object RichStatement
-
object
RowWriterFactory extends LowPriorityRowWriterFactoryImplicits
Provides an implicit
RowWriterFactory
for saving CassandraRow objects. - object SqlRowWriter extends Serializable
- object TTLOption extends Serializable
- object TableWriter extends Serializable
- object TimestampOption extends Serializable
- object WriteConf extends Serializable
- object WriteOption