Package | Description |
---|---|
com.hazelcast.jet.avro |
Apache Avro file read/write support for Hazelcast Jet.
|
com.hazelcast.jet.hadoop |
Apache Hadoop HDFS read/write support for Hazelcast Jet.
|
com.hazelcast.jet.pipeline |
The Pipeline API is Jet's high-level API to build and execute
distributed computation jobs.
|
com.hazelcast.jet.pipeline.test |
This package contains various mock sources to help with pipeline testing
and development.
|
com.hazelcast.jet.s3 |
AWS S3 read/write support for Hazelcast Jet.
|
Modifier and Type | Method and Description |
---|---|
BatchSource<D> |
AvroSourceBuilder.build()
Convenience for
AvroSourceBuilder.build(BiFunctionEx) . |
<T> BatchSource<T> |
AvroSourceBuilder.build(BiFunctionEx<String,? super D,T> mapOutputFn)
Builds a custom Avro file
BatchSource with supplied components
and the output function mapOutputFn . |
static <D> BatchSource<D> |
AvroSources.files(String directory,
BiFunctionEx<String,org.apache.avro.generic.GenericRecord,D> mapOutputFn)
Convenience for
AvroSources.filesBuilder(String, SupplierEx) which
reads all the files in the supplied directory as generic records and
emits the results of transforming each generic record with the supplied
mapping function. |
static <D> BatchSource<D> |
AvroSources.files(String directory,
Class<D> datumClass)
Convenience for
AvroSources.filesBuilder(String, Class) which
reads all the files in the supplied directory as specific records using
supplied datumClass . |
Modifier and Type | Method and Description |
---|---|
static <K,V> BatchSource<Map.Entry<K,V>> |
HdfsSources.hdfs(org.apache.hadoop.mapred.JobConf jobConf)
Convenience for
HdfsSources.hdfs(JobConf, BiFunctionEx)
with Map.Entry as its output type. |
static <K,V,E> BatchSource<E> |
HdfsSources.hdfs(org.apache.hadoop.mapred.JobConf jobConf,
BiFunctionEx<K,V,E> projectionFn)
Returns a source that reads records from Apache Hadoop HDFS and emits
the results of transforming each record (a key-value pair) with the
supplied mapping function.
|
Modifier and Type | Method and Description |
---|---|
static <T> BatchSource<T> |
Sources.batchFromProcessor(String sourceName,
ProcessorMetaSupplier metaSupplier)
Returns a bounded (batch) source constructed directly from the given
Core API processor meta-supplier.
|
BatchSource<T> |
SourceBuilder.Batch.build()
Builds and returns the batch source.
|
BatchSource<String> |
FileSourceBuilder.build()
Convenience for
FileSourceBuilder.build(BiFunctionEx) . |
<T> BatchSource<T> |
FileSourceBuilder.build(BiFunctionEx<String,String,? extends T> mapOutputFn)
Builds a custom file
BatchSource with supplied components and the
output function mapOutputFn . |
static <K,V> BatchSource<Map.Entry<K,V>> |
Sources.cache(String cacheName)
Returns a source that fetches entries from a Hazelcast
ICache
with the given name and emits them as Map.Entry . |
static BatchSource<String> |
Sources.files(String directory)
A source to read all files in a directory in a batch way.
|
static <T> BatchSource<T> |
Sources.jdbc(String connectionURL,
String query,
FunctionEx<? super ResultSet,? extends T> createOutputFn)
Convenience for
Sources.jdbc(SupplierEx,
ToResultSetFunction, FunctionEx) . |
static <T> BatchSource<T> |
Sources.jdbc(SupplierEx<? extends Connection> newConnectionFn,
ToResultSetFunction resultSetFn,
FunctionEx<? super ResultSet,? extends T> createOutputFn)
Returns a source which connects to the specified database using the given
newConnectionFn , queries the database and creates a result set
using the the given resultSetFn . |
static <T> BatchSource<T> |
Sources.list(IList<? extends T> list)
Returns a source that emits items retrieved from a Hazelcast
IList . |
static <T> BatchSource<T> |
Sources.list(String listName)
Returns a source that emits items retrieved from a Hazelcast
IList . |
static <K,V> BatchSource<Map.Entry<K,V>> |
Sources.map(IMap<? extends K,? extends V> map)
Returns a source that fetches entries from the given Hazelcast
IMap and emits them as Map.Entry . |
static <T,K,V> BatchSource<T> |
Sources.map(IMap<? extends K,? extends V> map,
Predicate<? super K,? super V> predicate,
FunctionEx<? super Map.Entry<K,V>,? extends T> projection)
Convenience for
Sources.map(IMap, Predicate, Projection) which uses a
FunctionEx as the projection function. |
static <T,K,V> BatchSource<T> |
Sources.map(IMap<? extends K,? extends V> map,
Predicate<? super K,? super V> predicate,
Projection<? super Map.Entry<K,V>,? extends T> projection)
Returns a source that fetches entries from the given Hazelcast
IMap . |
static <K,V> BatchSource<Map.Entry<K,V>> |
Sources.map(String mapName)
Returns a source that fetches entries from a local Hazelcast
IMap
with the specified name and emits them as Map.Entry . |
static <T,K,V> BatchSource<T> |
Sources.map(String mapName,
Predicate<? super K,? super V> predicate,
FunctionEx<? super Map.Entry<K,V>,? extends T> projection)
Convenience for
Sources.map(String, Predicate, Projection)
which uses a FunctionEx as the projection function. |
static <T,K,V> BatchSource<T> |
Sources.map(String mapName,
Predicate<? super K,? super V> predicate,
Projection<? super Map.Entry<K,V>,? extends T> projection)
Returns a source that fetches entries from a local Hazelcast
IMap with the specified name. |
static <K,V> BatchSource<Map.Entry<K,V>> |
Sources.remoteCache(String cacheName,
ClientConfig clientConfig)
Returns a source that fetches entries from the Hazelcast
ICache
with the specified name in a remote cluster identified by the supplied
ClientConfig and emits them as Map.Entry . |
static <T> BatchSource<T> |
Sources.remoteList(String listName,
ClientConfig clientConfig)
Returns a source that emits items retrieved from a Hazelcast
IList in a remote cluster identified by the supplied ClientConfig . |
static <K,V> BatchSource<Map.Entry<K,V>> |
Sources.remoteMap(String mapName,
ClientConfig clientConfig)
Returns a source that fetches entries from the Hazelcast
IMap
with the specified name in a remote cluster identified by the supplied
ClientConfig and emits them as Map.Entry . |
static <T,K,V> BatchSource<T> |
Sources.remoteMap(String mapName,
ClientConfig clientConfig,
Predicate<? super K,? super V> predicate,
FunctionEx<? super Map.Entry<K,V>,? extends T> projection)
Convenience for
Sources.remoteMap(String, ClientConfig, Predicate, Projection)
which use a FunctionEx as the projection function. |
static <T,K,V> BatchSource<T> |
Sources.remoteMap(String mapName,
ClientConfig clientConfig,
Predicate<? super K,? super V> predicate,
Projection<? super Map.Entry<K,V>,? extends T> projection)
Returns a source that fetches entries from a remote Hazelcast
IMap with the specified name in a remote cluster identified by the
supplied ClientConfig . |
Modifier and Type | Method and Description |
---|---|
<T> BatchStage<T> |
Pipeline.drawFrom(BatchSource<? extends T> source)
Returns a pipeline stage that represents a bounded (batch) data source.
|
Modifier and Type | Method and Description |
---|---|
static <T> BatchSource<T> |
TestSources.items(Iterable<? extends T> items)
Returns a batch source which iterates through the supplied iterable and then
terminates.
|
static <T> BatchSource<T> |
TestSources.items(T... items)
Returns a batch source which iterates through the supplied items and then
terminates.
|
Modifier and Type | Method and Description |
---|---|
static <T> BatchSource<T> |
S3Sources.s3(List<String> bucketNames,
String prefix,
Charset charset,
SupplierEx<? extends software.amazon.awssdk.services.s3.S3Client> clientSupplier,
BiFunctionEx<String,String,? extends T> mapFn)
Creates an AWS S3
BatchSource which lists all the objects in the
bucket-list using given prefix , reads them line by line,
transforms each line to the desired output object using given mapFn and emits them to downstream. |
static BatchSource<String> |
S3Sources.s3(List<String> bucketNames,
String prefix,
SupplierEx<? extends software.amazon.awssdk.services.s3.S3Client> clientSupplier)
Convenience for
S3Sources.s3(List, String, Charset, SupplierEx, BiFunctionEx) . |
Copyright © 2019 Hazelcast, Inc.. All rights reserved.