in the testing section for scio.io.Parquet There are several points where the examples appear out of date or unclear.
-
predicate filtering: the function withPredicate appears to have been renamed to withFilter
-
the sequencing of predicate vs projection is inverted - in the first example (avro projection) we have
records withProjection projection withPredicate predicate
however this fails to compile due to apparently interpreting withFilter as a scala collection method, leading to a type mismatch expecting a method returning a Boolean rather than a parquet FilterPredicate.
Fiipping the order to
records withFilter predicate withProjection projection
should resolve this (tested with the case-class projection approach, which requires other changes - see below)
- postfix operators do not work for the case-class projection method - in the second example we have
records withProjection[MyProjection]
but this fails with type application is not allowed for postfix operators. Using a normal method call records.withProjection[MyProjection] resolves this.
in the testing section for scio.io.Parquet There are several points where the examples appear out of date or unclear.
predicate filtering: the function
withPredicateappears to have been renamed towithFilterthe sequencing of predicate vs projection is inverted - in the first example (avro projection) we have
however this fails to compile due to apparently interpreting
withFilteras a scala collection method, leading to a type mismatch expecting a method returning a Boolean rather than a parquetFilterPredicate.Fiipping the order to
should resolve this (tested with the case-class projection approach, which requires other changes - see below)
records withProjection[MyProjection]but this fails with
type application is not allowed for postfix operators. Using a normal method callrecords.withProjection[MyProjection]resolves this.