How do I fix a cluster when I see the following message:
org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:Expected 1 components, got 2 (<folder/subfolder>))
Hive expects a single partition folders per geo code, but sees 2.
1. Look at the folder structure on aws
aws ls s3://<path/to/folder>/
You should see a list of unique folders as below.
2018-03-21 21:13:04 0 <subfolder1>
2018-03-21 21:13:05 0 <subfolder2>
2018-03-21 21:13:23 0 <subfolder3>
2018-03-21 21:13:26 0 <subfolder4>
2. Create a local database. On Okera machines, you can create a db named localdb which bypasses Okera completely. This enables you to test the solution on an non-production database and to rule out Okera as the root cause.
3. Replicate the table structure of the real database into localdb.
4. Run the command that failed previously.It should fail as before.
5. Set the property
6. Run this command against localdb:
hive -e "use <localdb> ;msck repair table <bad_table>"
7. Repeat the command against the production database:
hive -e "use <s3_db> ;msck repair table <bad_table>"
8. To prevent recurrence, add this flag to the set statements in the DDL: