Package com.hazelcast.jet.avro
Class AvroSourceBuilder<D>
java.lang.Object
com.hazelcast.jet.avro.AvroSourceBuilder<D>
- Type Parameters:
D
- the type of the datum read bydatumReaderSupplier
public final class AvroSourceBuilder<D> extends Object
Builder for an Avro file source which reads records from Avro files in a
directory (but not its subdirectories) and emits output object created by
mapOutputFn
.- Since:
- 3.0
-
Method Summary
Modifier and Type Method Description BatchSource<D>
build()
Convenience forbuild(BiFunctionEx)
.<T> BatchSource<T>
build(BiFunctionEx<String,? super D,T> mapOutputFn)
Builds a custom Avro fileBatchSource
with supplied components and the output functionmapOutputFn
.AvroSourceBuilder<D>
glob(String glob)
Sets the globbing mask, seegetPathMatcher()
.AvroSourceBuilder<D>
sharedFileSystem(boolean sharedFileSystem)
Sets if files are in a shared storage visible to all members.
-
Method Details
-
glob
Sets the globbing mask, seegetPathMatcher()
. Default value is"*"
which means all files. -
sharedFileSystem
Sets if files are in a shared storage visible to all members. Default value isfalse
If
sharedFileSystem
istrue
, Jet will assume all members see the same files. They will split the work so that each member will read a part of the files. IfsharedFileSystem
isfalse
, each member will read all files in the directory, assuming the are local. -
build
Builds a custom Avro fileBatchSource
with supplied components and the output functionmapOutputFn
.The source does not save any state to snapshot. If the job is restarted, it will re-emit all entries.
Any
IOException
will cause the job to fail. The files must not change while being read; if they do, the behavior is unspecified.The default local parallelism for this processor is 4 (or available CPU count if it is less than 4).
- Type Parameters:
T
- the type of the items the source emits- Parameters:
mapOutputFn
- the function which creates output object from each record. Gets the filename and record read bydatumReader
as parameters
-
build
Convenience forbuild(BiFunctionEx)
. Source emits records read bydatumReader
to downstream without any transformation.
-