Uses of Interface
com.hazelcast.jet.pipeline.Sink
Package | Description |
---|---|
com.hazelcast.jet.avro |
Apache Avro file read/write support for Hazelcast Jet.
|
com.hazelcast.jet.cdc |
Contains source/sink connectors that deal with Change Data Capture (CDC)
events from various databases as well as a generic connector for Debezium
CDC sources.
|
com.hazelcast.jet.elastic |
Contains sources and sinks for Elasticsearch 7
|
com.hazelcast.jet.hadoop |
Apache Hadoop read/write support for Hazelcast Jet.
|
com.hazelcast.jet.kafka |
Apache Kafka reader/writer support for Hazelcast Jet.
|
com.hazelcast.jet.kinesis |
Amazon Kinesis Data Streams producer/consumer support for Hazelcast Jet.
|
com.hazelcast.jet.pipeline |
The Pipeline API is Jet's high-level API to build and execute
distributed computation jobs.
|
com.hazelcast.jet.pipeline.test |
This package contains various mock sources to help with pipeline testing
and development.
|
com.hazelcast.jet.s3 |
AWS S3 read/write support for Hazelcast Jet.
|
-
Uses of Sink in com.hazelcast.jet.avro
Methods in com.hazelcast.jet.avro that return Sink Modifier and Type Method Description static <R> Sink<R>
AvroSinks. files(String directoryName, Class<R> recordClass, org.apache.avro.Schema schema)
Convenience forAvroSinks.files(String, Schema, SupplierEx)
which uses eitherSpecificDatumWriter
orReflectDatumWriter
depending on the suppliedrecordClass
.static Sink<org.apache.avro.generic.IndexedRecord>
AvroSinks. files(String directoryName, org.apache.avro.Schema schema)
Convenience forAvroSinks.files(String, Schema, SupplierEx)
which usesGenericDatumWriter
.static <R> Sink<R>
AvroSinks. files(String directoryName, org.apache.avro.Schema schema, SupplierEx<org.apache.avro.io.DatumWriter<R>> datumWriterSupplier)
Returns a sink that that writes the items it receives to Apache Avro files. -
Uses of Sink in com.hazelcast.jet.cdc
Methods in com.hazelcast.jet.cdc that return Sink Modifier and Type Method Description static <K, V> Sink<ChangeRecord>
CdcSinks. map(IMap<? super K,? super V> map, FunctionEx<? super ChangeRecord,? extends K> keyFn, FunctionEx<? super ChangeRecord,? extends V> valueFn)
Returns a sink that applies the changes described by a Change Data Capture (CDC) stream to anIMap
.static <K, V> Sink<ChangeRecord>
CdcSinks. map(String mapName, FunctionEx<? super ChangeRecord,? extends K> keyFn, FunctionEx<? super ChangeRecord,? extends V> valueFn)
Returns a sink that applies the changes described by a Change Data Capture (CDC) stream to anIMap
.static <K, V> Sink<ChangeRecord>
CdcSinks. remoteMap(String mapName, ClientConfig clientConfig, FunctionEx<? super ChangeRecord,? extends K> keyFn, FunctionEx<? super ChangeRecord,? extends V> valueFn)
Returns a sink equivalent toCdcSinks.map(java.lang.String, com.hazelcast.function.FunctionEx<? super com.hazelcast.jet.cdc.ChangeRecord, ? extends K>, com.hazelcast.function.FunctionEx<? super com.hazelcast.jet.cdc.ChangeRecord, ? extends V>)
, but for a map in a remote Hazelcast cluster identified by the suppliedClientConfig
. -
Uses of Sink in com.hazelcast.jet.elastic
Methods in com.hazelcast.jet.elastic that return Sink Modifier and Type Method Description Sink<T>
ElasticSinkBuilder. build()
Create a sink that writes data into Elasticsearch based on this builder configurationstatic <T> Sink<T>
ElasticSinks. elastic(FunctionEx<? super T,? extends org.elasticsearch.action.DocWriteRequest<?>> mapToRequestFn)
Creates an Elasticsearch sink, uses a local instance of Elasticsearchstatic <T> Sink<T>
ElasticSinks. elastic(SupplierEx<org.elasticsearch.client.RestClientBuilder> clientFn, FunctionEx<? super T,? extends org.elasticsearch.action.DocWriteRequest<?>> mapToRequestFn)
Creates an Elasticsearch sink, uses a client obtained from clientFn and maps items using given mapToRequestFn -
Uses of Sink in com.hazelcast.jet.hadoop
Methods in com.hazelcast.jet.hadoop that return Sink Modifier and Type Method Description static <K, V> Sink<Map.Entry<K,V>>
HadoopSinks. outputFormat(org.apache.hadoop.conf.Configuration configuration)
Convenience forHadoopSinks.outputFormat(Configuration, FunctionEx, FunctionEx)
which expectsMap.Entry<K, V>
as input and extracts its key and value parts to be written to HDFS.static <E, K, V> Sink<E>
HadoopSinks. outputFormat(org.apache.hadoop.conf.Configuration configuration, FunctionEx<? super E,K> extractKeyF, FunctionEx<? super E,V> extractValueF)
Returns a sink that writes to Apache Hadoop HDFS. -
Uses of Sink in com.hazelcast.jet.kafka
Methods in com.hazelcast.jet.kafka that return Sink Modifier and Type Method Description Sink<E>
KafkaSinks.Builder. build()
Builds the Sink object that you pass to theGeneralStage.writeTo(Sink)
method.static <E, K, V> Sink<E>
KafkaSinks. kafka(Properties properties, FunctionEx<? super E,org.apache.kafka.clients.producer.ProducerRecord<K,V>> toRecordFn)
Returns a sink that publishes messages to Apache Kafka topic(s).static <K, V> Sink<Map.Entry<K,V>>
KafkaSinks. kafka(Properties properties, String topic)
Convenience forKafkaSinks.kafka(Properties, String, FunctionEx, FunctionEx)
which expectsMap.Entry<K, V>
as input and extracts its key and value parts to be published to Kafka.static <E, K, V> Sink<E>
KafkaSinks. kafka(Properties properties, String topic, FunctionEx<? super E,K> extractKeyFn, FunctionEx<? super E,V> extractValueFn)
Convenience forKafkaSinks.kafka(Properties, FunctionEx)
which creates aProducerRecord
using the given topic and the given key and value mapping functions -
Uses of Sink in com.hazelcast.jet.kinesis
Methods in com.hazelcast.jet.kinesis that return Sink Modifier and Type Method Description Sink<T>
KinesisSinks.Builder. build()
Construct the sink based on the options provided so far. -
Uses of Sink in com.hazelcast.jet.pipeline
Methods in com.hazelcast.jet.pipeline that return Sink Modifier and Type Method Description Sink<T>
FileSinkBuilder. build()
Creates and returns the fileSink
with the supplied components.Sink<T>
JdbcSinkBuilder. build()
Creates and returns the JDBCSink
with the supplied components.Sink<T>
JmsSinkBuilder. build()
Creates and returns the JMSSink
with the supplied components.Sink<T>
SinkBuilder. build()
Creates and returns theSink
with the components you supplied to this builder.static <T extends Map.Entry>
Sink<T>Sinks. cache(String cacheName)
Returns a sink that putsMap.Entry
s it receives into a HazelcastICache
with the specified name.static <T> Sink<T>
Sinks. files(String directoryName)
Convenience forSinks.filesBuilder(java.lang.String)
with the UTF-8 charset and with overwriting of existing files.static <T> Sink<T>
Sinks. fromProcessor(String sinkName, ProcessorMetaSupplier metaSupplier)
Returns a sink constructed directly from the given Core API processor meta-supplier.static <T> Sink<T>
Sinks. fromProcessor(String sinkName, ProcessorMetaSupplier metaSupplier, FunctionEx<? super T,?> partitionKeyFn)
Returns a sink constructed directly from the given Core API processor meta-supplier.static <T> Sink<T>
Sinks. jdbc(String updateQuery, SupplierEx<? extends CommonDataSource> dataSourceSupplier, BiConsumerEx<PreparedStatement,T> bindFn)
A shortcut for:static <T> Sink<T>
Sinks. jdbc(String updateQuery, String jdbcUrl, BiConsumerEx<PreparedStatement,T> bindFn)
A shortcut for:static <T> Sink<T>
Sinks. jmsQueue(String queueName, SupplierEx<javax.jms.ConnectionFactory> factorySupplier)
Convenience forSinks.jmsQueueBuilder(SupplierEx)
.static <T> Sink<T>
Sinks. jmsTopic(String topicName, SupplierEx<javax.jms.ConnectionFactory> factorySupplier)
Shortcut for:static <T> Sink<T>
Sinks. json(String directoryName)
Convenience forSinks.filesBuilder(java.lang.String)
with the UTF-8 charset and with overwriting of existing files.static <T> Sink<T>
Sinks. list(IList<? super T> list)
Returns a sink that adds the items it receives to the specified HazelcastIList
.static <T> Sink<T>
Sinks. list(String listName)
Returns a sink that adds the items it receives to a HazelcastIList
with the specified name.static <T> Sink<T>
Sinks. logger()
static <T> Sink<T>
Sinks. logger(FunctionEx<? super T,String> toStringFn)
Returns a sink that logs all the data items it receives, at the INFO level to the log categoryWriteLoggerP
.static <K, V> Sink<Map.Entry<K,V>>
Sinks. map(IMap<? super K,? super V> map)
Returns a sink that putsMap.Entry
s it receives into the given HazelcastIMap
.static <T, K, V> Sink<T>
Sinks. map(IMap<? super K,? super V> map, FunctionEx<? super T,? extends K> toKeyFn, FunctionEx<? super T,? extends V> toValueFn)
Returns a sink that uses the supplied functions to extract the key and value with which to put to given HazelcastIMap
.static <K, V> Sink<Map.Entry<K,V>>
Sinks. map(String mapName)
Returns a sink that putsMap.Entry
s it receives into a HazelcastIMap
with the specified name.static <T, K, V> Sink<T>
Sinks. map(String mapName, FunctionEx<? super T,? extends K> toKeyFn, FunctionEx<? super T,? extends V> toValueFn)
Returns a sink that uses the supplied functions to extract the key and value with which to put to a HazelcastIMap
with the specified name.static <E, K, V, R> Sink<E>
Sinks. mapWithEntryProcessor(int maxParallelAsyncOps, String mapName, FunctionEx<? super E,? extends K> toKeyFn, FunctionEx<? super E,? extends EntryProcessor<K,V,R>> toEntryProcessorFn)
Returns a sink that uses the items it receives to createEntryProcessor
s it submits to a HazelcastIMap
with the specified name.static <T, K, V, R> Sink<T>
Sinks. mapWithEntryProcessor(IMap<? super K,? super V> map, FunctionEx<? super T,? extends K> toKeyFn, FunctionEx<? super T,? extends EntryProcessor<K,V,R>> toEntryProcessorFn)
Returns a sink that uses the items it receives to createEntryProcessor
s it submits to a HazelcastIMap
with the specified name.static <E, K, V, R> Sink<E>
Sinks. mapWithEntryProcessor(String mapName, FunctionEx<? super E,? extends K> toKeyFn, FunctionEx<? super E,? extends EntryProcessor<K,V,R>> toEntryProcessorFn)
Convenience forSinks.mapWithEntryProcessor(int, String, FunctionEx, FunctionEx)
when the maximum number of async operations is not specified.static <T, K, V> Sink<T>
Sinks. mapWithMerging(IMap<? super K,? super V> map, FunctionEx<? super T,? extends K> toKeyFn, FunctionEx<? super T,? extends V> toValueFn, BinaryOperatorEx<V> mergeFn)
Returns a sink that uses the supplied functions to extract the key and value with which to update a HazelcastIMap
.static <K, V> Sink<Map.Entry<K,V>>
Sinks. mapWithMerging(IMap<? super K,V> map, BinaryOperatorEx<V> mergeFn)
Convenience forSinks.mapWithMerging(IMap, FunctionEx, FunctionEx, BinaryOperatorEx)
withMap.Entry
as input item.static <K, V> Sink<Map.Entry<K,V>>
Sinks. mapWithMerging(String mapName, BinaryOperatorEx<V> mergeFn)
Convenience forSinks.mapWithMerging(String, FunctionEx, FunctionEx, BinaryOperatorEx)
withMap.Entry
as input item.static <T, K, V> Sink<T>
Sinks. mapWithMerging(String mapName, FunctionEx<? super T,? extends K> toKeyFn, FunctionEx<? super T,? extends V> toValueFn, BinaryOperatorEx<V> mergeFn)
Returns a sink that uses the supplied functions to extract the key and value with which to update a HazelcastIMap
.static <K, V, E extends Map.Entry<K, V>>
Sink<E>Sinks. mapWithUpdating(IMap<? super K,? super V> map, BiFunctionEx<? super V,? super E,? extends V> updateFn)
Convenience forSinks.mapWithUpdating(IMap, FunctionEx, BiFunctionEx)
withMap.Entry
as the input item.static <T, K, V> Sink<T>
Sinks. mapWithUpdating(IMap<? super K,? super V> map, FunctionEx<? super T,? extends K> toKeyFn, BiFunctionEx<? super V,? super T,? extends V> updateFn)
Returns a sink that uses the supplied key-extracting and value-updating functions to update a HazelcastIMap
.static <K, V, E extends Map.Entry<K, V>>
Sink<E>Sinks. mapWithUpdating(String mapName, BiFunctionEx<? super V,? super E,? extends V> updateFn)
Convenience forSinks.mapWithUpdating(String, FunctionEx, BiFunctionEx)
withMap.Entry
as the input item.static <T, K, V> Sink<T>
Sinks. mapWithUpdating(String mapName, FunctionEx<? super T,? extends K> toKeyFn, BiFunctionEx<? super V,? super T,? extends V> updateFn)
Returns a sink that uses the supplied key-extracting and value-updating functions to update a HazelcastIMap
.static <T> Sink<T>
Sinks. noop()
Returns a sink which discards all received items.static <T> Sink<T>
Sinks. observable(Observable<? super T> observable)
Returns a sink that publishes to the providedObservable
.static <T> Sink<T>
Sinks. observable(String name)
Returns a sink that publishes to theObservable
with the provided name.static <T> Sink<T>
Sinks. reliableTopic(ITopic<Object> reliableTopic)
Returns a sink which publishes the items it receives to the provided distributed reliable topic.static <T> Sink<T>
Sinks. reliableTopic(String reliableTopicName)
Returns a sink which publishes the items it receives to a distributed reliable topic with the specified name.static <T extends Map.Entry>
Sink<T>Sinks. remoteCache(String cacheName, ClientConfig clientConfig)
Returns a sink that putsMap.Entry
s it receives into a HazelcastICache
with the specified name in a remote cluster identified by the suppliedClientConfig
.static <T> Sink<T>
Sinks. remoteList(String listName, ClientConfig clientConfig)
Returns a sink that adds the items it receives to a HazelcastIList
with the specified name in a remote cluster identified by the suppliedClientConfig
.static <K, V> Sink<Map.Entry<K,V>>
Sinks. remoteMap(String mapName, ClientConfig clientConfig)
Returns a sink that putsMap.Entry
s it receives into a HazelcastIMap
with the specified name in a remote cluster identified by the suppliedClientConfig
.static <T, K, V> Sink<T>
Sinks. remoteMap(String mapName, ClientConfig clientConfig, FunctionEx<? super T,? extends K> toKeyFn, FunctionEx<? super T,? extends V> toValueFn)
Returns a sink that uses the supplied functions to extract the key and value with which to put to a HazelcastIMap
in a remote cluster identified by the suppliedClientConfig
.static <E, K, V, R> Sink<E>
Sinks. remoteMapWithEntryProcessor(String mapName, ClientConfig clientConfig, FunctionEx<? super E,? extends K> toKeyFn, FunctionEx<? super E,? extends EntryProcessor<K,V,R>> toEntryProcessorFn)
Returns a sink equivalent toSinks.mapWithEntryProcessor(java.lang.String, com.hazelcast.function.FunctionEx<? super E, ? extends K>, com.hazelcast.function.FunctionEx<? super E, ? extends com.hazelcast.map.EntryProcessor<K, V, R>>)
, but for a map in a remote Hazelcast cluster identified by the suppliedClientConfig
.static <K, V> Sink<Map.Entry<K,V>>
Sinks. remoteMapWithMerging(String mapName, ClientConfig clientConfig, BinaryOperatorEx<V> mergeFn)
static <T, K, V> Sink<T>
Sinks. remoteMapWithMerging(String mapName, ClientConfig clientConfig, FunctionEx<? super T,? extends K> toKeyFn, FunctionEx<? super T,? extends V> toValueFn, BinaryOperatorEx<V> mergeFn)
Returns a sink equivalent toSinks.mapWithMerging(String, BinaryOperatorEx)
, but for a map in a remote Hazelcast cluster identified by the suppliedClientConfig
.static <K, V, E extends Map.Entry<K, V>>
Sink<E>Sinks. remoteMapWithUpdating(String mapName, ClientConfig clientConfig, BiFunctionEx<? super V,? super E,? extends V> updateFn)
static <T, K, V> Sink<T>
Sinks. remoteMapWithUpdating(String mapName, ClientConfig clientConfig, FunctionEx<? super T,? extends K> toKeyFn, BiFunctionEx<? super V,? super T,? extends V> updateFn)
Returns a sink equivalent toSinks.mapWithUpdating(java.lang.String, com.hazelcast.function.FunctionEx<? super T, ? extends K>, com.hazelcast.function.BiFunctionEx<? super V, ? super T, ? extends V>)
, but for a map in a remote Hazelcast cluster identified by the suppliedClientConfig
.static <T> Sink<T>
Sinks. remoteReliableTopic(String reliableTopicName, ClientConfig clientConfig)
Returns a sink which publishes the items it receives to a distributed reliable topic with the provided name in a remote cluster identified by the suppliedClientConfig
.static <T> Sink<T>
Sinks. socket(String host, int port)
Convenience forSinks.socket(String, int, FunctionEx, Charset)
withObject.toString
as the conversion function and UTF-8 as the charset.static <T> Sink<T>
Sinks. socket(String host, int port, FunctionEx<? super T,? extends String> toStringFn)
Convenience forSinks.socket(String, int, FunctionEx, Charset)
with UTF-8 as the charset.static <T> Sink<T>
Sinks. socket(String host, int port, FunctionEx<? super T,? extends String> toStringFn, Charset charset)
Returns a sink that connects to the specified TCP socket and writes to it a string representation of the items it receives.Methods in com.hazelcast.jet.pipeline with parameters of type Sink Modifier and Type Method Description SinkStage
GeneralStage. writeTo(Sink<? super T> sink)
Attaches a sink stage, one that accepts data but doesn't emit any.<T> SinkStage
Pipeline. writeTo(Sink<? super T> sink, GeneralStage<? extends T> stage0, GeneralStage<? extends T> stage1, GeneralStage<? extends T>... moreStages)
Attaches the supplied sink to two or more pipeline stages. -
Uses of Sink in com.hazelcast.jet.pipeline.test
Methods in com.hazelcast.jet.pipeline.test that return Sink Modifier and Type Method Description static <T> Sink<T>
AssertionSinks. assertAnyOrder(String message, Collection<? extends T> expected)
Asserts that the previous stage emitted the expected items in any order, but nothing else.static <T> Sink<T>
AssertionSinks. assertAnyOrder(Collection<? extends T> expected)
Asserts that the previous stage emitted the expected items in any order, but nothing else.static <T> Sink<T>
AssertionSinks. assertCollected(ConsumerEx<? super List<T>> assertFn)
Collects all the received items in a list and once the upstream stage is completed it executes the assertion supplied byassertFn
.static <T> Sink<T>
AssertionSinks. assertCollectedEventually(int timeoutSeconds, ConsumerEx<? super List<T>> assertFn)
Collects all the received items into a list and runs theassertFn
every time a new item is received.static <T> Sink<T>
AssertionSinks. assertContains(String message, Collection<? extends T> expected)
Asserts that the previous stage emitted all of the given items in any order.static <T> Sink<T>
AssertionSinks. assertOrdered(String message, Collection<? extends T> expected)
Asserts that the previous stage emitted the exact sequence of expected items and nothing else.static <T> Sink<T>
AssertionSinks. assertOrdered(Collection<? extends T> expected)
Asserts that the previous stage emitted the exact sequence of expected items and nothing else.Sink<T>
AssertionSinkBuilder. build()
Creates and returns theSink
with the components you supplied to this builder. -
Uses of Sink in com.hazelcast.jet.s3
Methods in com.hazelcast.jet.s3 that return Sink Modifier and Type Method Description static <T> Sink<? super T>
S3Sinks. s3(String bucketName, SupplierEx<? extends software.amazon.awssdk.services.s3.S3Client> clientSupplier)
Convenience forS3Sinks.s3(String, String, Charset, SupplierEx, FunctionEx)
UsesObject.toString()
to convert the items to lines.static <T> Sink<? super T>
S3Sinks. s3(String bucketName, String prefix, Charset charset, SupplierEx<? extends software.amazon.awssdk.services.s3.S3Client> clientSupplier, FunctionEx<? super T,String> toStringFn)
Creates an AWS S3Sink
which writes items to files into the given bucket.