Uses of Interface
com.hazelcast.jet.pipeline.BatchSource
Package | Description |
---|---|
com.hazelcast.jet.avro |
Apache Avro file read/write support for Hazelcast Jet.
|
com.hazelcast.jet.elastic |
Contains sources and sinks for Elasticsearch 7
|
com.hazelcast.jet.hadoop |
Apache Hadoop read/write support for Hazelcast Jet.
|
com.hazelcast.jet.pipeline |
The Pipeline API is Jet's high-level API to build and execute
distributed computation jobs.
|
com.hazelcast.jet.pipeline.file |
This package offers the
FileSourceBuilder
which allows you to construct various kinds of Pipeline sources that read from local or distributed files. |
com.hazelcast.jet.pipeline.test |
This package contains various mock sources to help with pipeline testing
and development.
|
com.hazelcast.jet.s3 |
AWS S3 read/write support for Hazelcast Jet.
|
-
Uses of BatchSource in com.hazelcast.jet.avro
Methods in com.hazelcast.jet.avro that return BatchSource Modifier and Type Method Description BatchSource<D>
AvroSourceBuilder. build()
Convenience forAvroSourceBuilder.build(BiFunctionEx)
.<T> BatchSource<T>
AvroSourceBuilder. build(BiFunctionEx<String,? super D,T> mapOutputFn)
Builds a custom Avro fileBatchSource
with supplied components and the output functionmapOutputFn
.static <D> BatchSource<D>
AvroSources. files(String directory, BiFunctionEx<String,org.apache.avro.generic.GenericRecord,D> mapOutputFn)
Convenience forAvroSources.filesBuilder(String, SupplierEx)
which reads all the files in the supplied directory as generic records and emits the results of transforming each generic record with the supplied mapping function.static <D> BatchSource<D>
AvroSources. files(String directory, Class<D> datumClass)
Convenience forAvroSources.filesBuilder(String, Class)
which reads all the files in the supplied directory as specific records using supplieddatumClass
. -
Uses of BatchSource in com.hazelcast.jet.elastic
Methods in com.hazelcast.jet.elastic that return BatchSource Modifier and Type Method Description BatchSource<T>
ElasticSourceBuilder. build()
Build ElasticsearchBatchSource
with supplied parametersstatic BatchSource<String>
ElasticSources. elastic()
Creates a source which queries local instance of Elasticsearch for all documentsstatic <T> BatchSource<T>
ElasticSources. elastic(FunctionEx<? super org.elasticsearch.search.SearchHit,T> mapToItemFn)
Creates a source which queries local instance of Elasticsearch for all documents.static BatchSource<String>
ElasticSources. elastic(SupplierEx<org.elasticsearch.client.RestClientBuilder> clientFn)
Creates a source which queries Elasticsearch using client obtained fromRestClientBuilder
supplier function.static <T> BatchSource<T>
ElasticSources. elastic(SupplierEx<org.elasticsearch.client.RestClientBuilder> clientFn, FunctionEx<? super org.elasticsearch.search.SearchHit,T> mapToItemFn)
Creates a source which queries Elasticsearch using client obtained fromRestClientBuilder
supplier function.static <T> BatchSource<T>
ElasticSources. elastic(SupplierEx<org.elasticsearch.client.RestClientBuilder> clientFn, SupplierEx<org.elasticsearch.action.search.SearchRequest> searchRequestFn, FunctionEx<? super org.elasticsearch.search.SearchHit,T> mapToItemFn)
Creates a source which queries Elasticsearch using client obtained fromRestHighLevelClient
supplier. -
Uses of BatchSource in com.hazelcast.jet.hadoop
Methods in com.hazelcast.jet.hadoop that return BatchSource Modifier and Type Method Description static <K, V, E> BatchSource<E>
HadoopSources. inputFormat(ConsumerEx<org.apache.hadoop.conf.Configuration> configureFn, BiFunctionEx<K,V,E> projectionFn)
Returns a source that reads records from Apache Hadoop HDFS and emits the results of transforming each record (a key-value pair) with the supplied projection function.static <K, V> BatchSource<Map.Entry<K,V>>
HadoopSources. inputFormat(org.apache.hadoop.conf.Configuration jobConf)
Convenience forHadoopSources.inputFormat(Configuration, BiFunctionEx)
withMap.Entry
as its output type.static <K, V, E> BatchSource<E>
HadoopSources. inputFormat(org.apache.hadoop.conf.Configuration configuration, BiFunctionEx<K,V,E> projectionFn)
Returns a source that reads records from Apache Hadoop HDFS and emits the results of transforming each record (a key-value pair) with the supplied projection function. -
Uses of BatchSource in com.hazelcast.jet.pipeline
Methods in com.hazelcast.jet.pipeline that return BatchSource Modifier and Type Method Description static <T> BatchSource<T>
Sources. batchFromProcessor(String sourceName, ProcessorMetaSupplier metaSupplier)
Returns a bounded (batch) source constructed directly from the given Core API processor meta-supplier.BatchSource<String>
FileSourceBuilder. build()
Deprecated.<T> BatchSource<T>
FileSourceBuilder. build(BiFunctionEx<String,String,? extends T> mapOutputFn)
Deprecated.<T> BatchSource<T>
FileSourceBuilder. build(FunctionEx<? super Path,? extends Stream<T>> readFileFn)
Deprecated.BatchSource<T>
SourceBuilder.Batch. build()
Builds and returns the batch source.static <K, V> BatchSource<Map.Entry<K,V>>
Sources. cache(String cacheName)
Returns a source that fetches entries from a HazelcastICache
with the given name and emits them asMap.Entry
.static BatchSource<String>
Sources. files(String directory)
A source to read all files in a directory in a batch way.static <T> BatchSource<T>
Sources. jdbc(SupplierEx<? extends Connection> newConnectionFn, ToResultSetFunction resultSetFn, FunctionEx<? super ResultSet,? extends T> createOutputFn)
Returns a source which connects to the specified database using the givennewConnectionFn
, queries the database and creates a result set using the the givenresultSetFn
.static <T> BatchSource<T>
Sources. jdbc(String connectionURL, String query, FunctionEx<? super ResultSet,? extends T> createOutputFn)
Convenience forSources.jdbc(SupplierEx, ToResultSetFunction, FunctionEx)
.static BatchSource<Map<String,Object>>
Sources. json(String directory)
Convenience forSources.json(String, Class)
which converts each JSON string to aMap
.static <T> BatchSource<T>
Sources. json(String directory, Class<T> type)
A source to read all files in a directory in a batch way.static <T> BatchSource<T>
Sources. list(IList<? extends T> list)
Returns a source that emits items retrieved from a HazelcastIList
.static <T> BatchSource<T>
Sources. list(String listName)
Returns a source that emits items retrieved from a HazelcastIList
.static <K, V> BatchSource<Map.Entry<K,V>>
Sources. map(IMap<? extends K,? extends V> map)
Returns a source that fetches entries from the given HazelcastIMap
and emits them asMap.Entry
.static <T, K, V> BatchSource<T>
Sources. map(IMap<? extends K,? extends V> map, Predicate<K,V> predicate, Projection<? super Map.Entry<K,V>,? extends T> projection)
Returns a source that fetches entries from the given HazelcastIMap
.static <K, V> BatchSource<Map.Entry<K,V>>
Sources. map(String mapName)
Returns a source that fetches entries from a local HazelcastIMap
with the specified name and emits them asMap.Entry
.static <T, K, V> BatchSource<T>
Sources. map(String mapName, Predicate<K,V> predicate, Projection<? super Map.Entry<K,V>,? extends T> projection)
Returns a source that fetches entries from a local HazelcastIMap
with the specified name.static <K, V> BatchSource<Map.Entry<K,V>>
Sources. remoteCache(String cacheName, ClientConfig clientConfig)
Returns a source that fetches entries from the HazelcastICache
with the specified name in a remote cluster identified by the suppliedClientConfig
and emits them asMap.Entry
.static <T> BatchSource<T>
Sources. remoteList(String listName, ClientConfig clientConfig)
Returns a source that emits items retrieved from a HazelcastIList
in a remote cluster identified by the suppliedClientConfig
.static <K, V> BatchSource<Map.Entry<K,V>>
Sources. remoteMap(String mapName, ClientConfig clientConfig)
Returns a source that fetches entries from the HazelcastIMap
with the specified name in a remote cluster identified by the suppliedClientConfig
and emits them asMap.Entry
.static <T, K, V> BatchSource<T>
Sources. remoteMap(String mapName, ClientConfig clientConfig, Predicate<K,V> predicate, Projection<? super Map.Entry<K,V>,? extends T> projection)
Returns a source that fetches entries from a remote HazelcastIMap
with the specified name in a remote cluster identified by the suppliedClientConfig
.Methods in com.hazelcast.jet.pipeline with parameters of type BatchSource Modifier and Type Method Description <T> BatchStage<T>
Pipeline. readFrom(BatchSource<? extends T> source)
Returns a pipeline stage that represents a bounded (batch) data source. -
Uses of BatchSource in com.hazelcast.jet.pipeline.file
Methods in com.hazelcast.jet.pipeline.file that return BatchSource Modifier and Type Method Description BatchSource<T>
FileSourceBuilder. build()
Builds aBatchSource
based on the current state of the builder. -
Uses of BatchSource in com.hazelcast.jet.pipeline.test
Methods in com.hazelcast.jet.pipeline.test that return BatchSource Modifier and Type Method Description static <T> BatchSource<T>
TestSources. items(Iterable<? extends T> items)
Returns a batch source which iterates through the supplied iterable and then terminates.static <T> BatchSource<T>
TestSources. items(T... items)
Returns a batch source which iterates through the supplied items and then terminates.static <T> BatchSource<T>
TestSources. itemsDistributed(Iterable<? extends T> items)
Returns a batch source which iterates through the supplied iterable and then terminates.static <T> BatchSource<T>
TestSources. itemsDistributed(T... items)
Returns a batch source which iterates through the supplied items and then terminates. -
Uses of BatchSource in com.hazelcast.jet.s3
Methods in com.hazelcast.jet.s3 that return BatchSource Modifier and Type Method Description static BatchSource<String>
S3Sources. s3(List<String> bucketNames, String prefix, SupplierEx<? extends software.amazon.awssdk.services.s3.S3Client> clientSupplier)
Convenience forS3Sources.s3(List, String, Charset, SupplierEx, BiFunctionEx)
.static <I, T> BatchSource<T>
S3Sources. s3(List<String> bucketNames, String prefix, SupplierEx<? extends software.amazon.awssdk.services.s3.S3Client> clientSupplier, FunctionEx<? super InputStream,? extends Stream<I>> readFileFn, BiFunctionEx<String,? super I,? extends T> mapFn)
Creates an AWS S3BatchSource
which lists all the objects in the bucket-list using givenprefix
, reads them using providedreadFileFn
, transforms each read item to the desired output object using givenmapFn
and emits them to downstream.static <I, T> BatchSource<T>
S3Sources. s3(List<String> bucketNames, String prefix, SupplierEx<? extends software.amazon.awssdk.services.s3.S3Client> clientSupplier, TriFunction<? super InputStream,String,String,? extends Stream<I>> readFileFn, BiFunctionEx<String,? super I,? extends T> mapFn)
Creates an AWS S3BatchSource
which lists all the objects in the bucket-list using givenprefix
, reads them using providedreadFileFn
, transforms each read item to the desired output object using givenmapFn
and emits them to downstream.static <T> BatchSource<T>
S3Sources. s3(List<String> bucketNames, String prefix, Charset charset, SupplierEx<? extends software.amazon.awssdk.services.s3.S3Client> clientSupplier, BiFunctionEx<String,String,? extends T> mapFn)
Creates an AWS S3BatchSource
which lists all the objects in the bucket-list using givenprefix
, reads them line by line, transforms each line to the desired output object using givenmapFn
and emits them to downstream.