ODAS Workers restart and produce a core dump when queries access tables that are backed by Parquet files and use certain, complex MAP data types in their schema.
Okera provides limited support for complex MAP data types for schemas backed by Parquet files. A user can do the following:
- Create table with MAP<STRING, STRING>
For example, this table stores key/value pairs for users. The pairs are stored as STRING types:
CREATE EXTERNAL TABLE referencedb.user_settings (
key_values MAP<STRING, STRING>
STORED AS PARQUET
- Create a table with MAP<STRING, STRUCT> and MAP<STRING, STRUCT<ARRAY>>
Note however that Okera does not support direct access of these complex types. To avoid application failure, create a VIEW on the complex tables without referring to the complex MAP types. For example, here we create a view on the base table that does not reference the complex map:
CREATE VIEW referencedb.user_settings_safe
AS SELECT id FROM referencedb.user_settings;
WARNING: If you use MAP types with STRUCT members directly, the ODAS Worker process will restart and the query fail. Use the VIEW approach shown above to access the remaining fields of the table.