CDH 5.3.6 Release Notes

Changes Not In Apache Hive 0.13.1

Common

Improvement

• [HADOOP-10448] - Support pluggable mechanism to specify proxy user settings

Hive

Bug

• [HIVE-7202] - DbTxnManager deadlocks in hcatalog.cli.TestSematicAnalysis.testAlterTblFFpart()
• [HIVE-9625] - Delegation tokens for HMS are not renewed
• [HIVE-10571] - HiveMetaStoreClient should close existing thrift connection before its reconnect
• [HIVE-10801] - 'drop view' fails throwing java.lang.NullPointerException
• [HIVE-11150] - Remove wrong warning message related to chgrp
• [HIVE-8318] - Null Scan optimizer throws exception when no partitions are selected
• [HIVE-7385] - Optimize for empty relation scans
• [HIVE-10808] - Inner join on Null throwing Cast Exception
• [HIVE-10841] - [WHERE col is not null] does not work sometimes for queries with many JOIN statements
• [HIVE-9620] - Cannot retrieve column statistics using HMS API if column name contains uppercase characters
• [HIVE-8863] - Cannot drop table with uppercase name after "compute statistics for columns"
• [HIVE-10956] - HS2 leaks HMS connections
• [HIVE-8298] - Incorrect results for n-way join when join expressions are not in same order across joins
• [HIVE-8895] - bugs in mergejoin
• [HIVE-10771] - "separatorChar" has no effect in "CREATE TABLE AS SELECT" statement
• [HIVE-6679] - HiveServer2 should support configurable the server side socket timeout and keepalive for various transports types where applicable
• [HIVE-10732] - Hive JDBC driver does not close operation for metadata queries
• [HIVE-8411] - Support partial partition spec for certain ALTER PARTITION statements
• [HIVE-7971] - Support alter table change/replace/add columns for existing partitions
• [HIVE-7027] - Hive job fails when referencing a view that explodes an array
• [HIVE-10646] - ColumnValue does not handle NULL_TYPE
• [HIVE-10453] - HS2 leaking open file descriptors when using UDFs
• [HIVE-9655] - Dynamic partition table insertion error
• [HIVE-10202] - Beeline outputs prompt+query on standard output when used in non-interactive mode
• [HIVE-10312] - SASL.QOP in JDBC URL is ignored for Delegation token Authentication
• [HIVE-10087] - Beeline's --silent option should suppress query from being echoed when running with -f option
• [HIVE-10098] - HS2 local task for map join fails in KMS encrypted cluster
• [HIVE-10085] - Lateral view on top of a view throws RuntimeException
• [HIVE-10086] - Hive throws error when accessing Parquet file schema using field name match
• [HIVE-7426] - ClassCastException: ...IntWritable cannot be cast to ...Text involving ql.udf.generic.GenericUDFBasePad.evaluate
• [HIVE-9934] - Vulnerability in LdapAuthenticationProviderImpl enables HiveServer2 client to degrade the authentication mechanism to "none", allowing authentication without password
• [HIVE-9828] - Semantic analyzer does not capture view parent entity for tables referred in view with union all
• [HIVE-9749] - ObjectStore schema verification logic is incorrect
• [HIVE-9755] - Hive built-in "ngram" UDAF fails when a mapper has no matches.
• [HIVE-9770] - Beeline ignores --showHeader for non-tablular output formats i.e csv,tsv,dsv
• [HIVE-8688] - serialized plan OutputStream is not being closed
• [HIVE-9716] - Map job fails when table's LOCATION does not have scheme
• [HIVE-5857] - Reduce tasks do not work in uber mode in YARN
• [HIVE-6978] - beeline always exits with 0 status, should exit with non-zero status on error
• [HIVE-8938] - Compiler should save the transform URI as input entity
• [HIVE-9023] - HiveHistoryImpl relies on removed counters to print num rows
• [HIVE-7733] - Ambiguous column reference error on query
• [HIVE-9303] - Parquet files are written with incorrect definition levels
• [HIVE-8444] - update pom to junit 4.11
• [HIVE-9474] - truncate table changes permissions on the target
• [HIVE-9234] - HiveServer2 leaks FileSystem objects in FileSystem.CACHE
• [HIVE-9462] - HIVE-8577 - breaks type evolution
• [HIVE-8577] - Cannot deserialize Avro schema with a map<string,string> with null values
• [HIVE-9482] - Hive parquet timestamp compatibility
• [HIVE-6308] - COLUMNS_V2 Metastore table not populated for tables created without an explicit column list.
• [HIVE-9502] - Parquet cannot read Map types from files written with Hive <= 0.12
• [HIVE-9393] - reduce noisy log level of ColumnarSerDe.java:116 from INFO to DEBUG
• [HIVE-7800] - Parquet Column Index Access Schema Size Checking
• [HIVE-9330] - DummyTxnManager will throw NPE if WriteEntity writeType has not been set
• [HIVE-9199] - Excessive exclusive lock used in some DDLs with DummyTxnManager
• [HIVE-8891] - Another possible cause to NucleusObjectNotFoundException from drops/rollback
• [HIVE-8889] - JDBC Driver ResultSet.getXXXXXX(String columnLabel) methods Broken
• [HIVE-8909] - Hive doesn't correctly read Parquet nested types
• [HIVE-8916] - Handle user@domain username under LDAP authentication
• [HIVE-8359] - Map containing null values are not correctly written in Parquet files
• [HIVE-8808] - HiveInputFormat caching cannot work with all input formats
• [HIVE-7011] - HiveInputFormat's split generation isn't thread safe
• [HIVE-5454] - HCatalog runs a partition listing with an empty filter
• [HIVE-8745] - Joins on decimal keys return different results whether they are run as reduce join or map join
• [HIVE-6853] - show create table for hbase tables should exclude LOCATION
• [HIVE-8784] - Querying partition does not work with JDO enabled against PostgreSQL
• [HIVE-8791] - Hive permission inheritance throws exception S3
• [HIVE-8827] - Remove SSLv2Hello from list of disabled protocols
• [HIVE-8704] - HivePassThroughOutputFormat cannot proxy more than one kind of OF (in one process)
• [HIVE-8680] - Set Max Message for Binary Thrift endpoints
• [HIVE-8713] - Unit test TestParquetTimestampUtils.testTimezone failing
• [HIVE-8757] - YARN dep in scheduler shim should be optional
• [HIVE-8708] - Add query id to explain log option
• [HIVE-8693] - Separate out fair scheduler dependency from hadoop 0.23 shim
• [HIVE-8634] - HiveServer2 fair scheduler queue mapping doesn't handle the secondary groups rules correctly
• [HIVE-8615] - beeline csv,tsv outputformat needs backward compatibility mode
• [HIVE-6847] - Improve / fix bugs in Hive scratch dir setup
• [HIVE-8627] - Compute stats on a table from impala caused the table to be corrupted
• [HIVE-8600] - Add option to log explain output for query
• [HIVE-7764] - Support all JDBC-HiveServer2 authentication modes on a secure cluster
• [HIVE-8448] - Union All might not work due to the type conversion issue
• [HIVE-7107] - Fix HiveServer1 JDBC Driver spec compliancy issue
• [HIVE-5865] - AvroDeserializer incorrectly assumes keys to Maps will always be of type 'org.apache.avro.util.Utf8'
• [HIVE-7618] - TestDDLWithRemoteMetastoreSecondNamenode unit test failure
• [HIVE-7902] - Cleanup hbase-handler/pom.xml dependency list
• [HIVE-6374] - Hive job submitted with non-default name node (fs.default.name) doesn't process locations properly
• [HIVE-7932] - It may cause NP exception when add accessed columns to ReadEntity
• [HIVE-8182] - beeline fails when executing multiple-line queries with trailing spaces
• [HIVE-8380] - NanoTime class serializes and deserializes Timestamp incorrectly
• [HIVE-7263] - Missing fixes from review of parquet-timestamp
• [HIVE-7433] - ColumnMappins.ColumnMapping should expose public accessors for its fields
• [HIVE-7109] - Resource leak in HBaseStorageHandler
• [HIVE-8345] - q-test for Avro date support
• [HIVE-7154] - TestMetrics fails intermittently on the trunk
• [HIVE-8266] - create function using <resource> statement compilation should include resource URI entity
• [HIVE-8317] - WebHCat pom should explicitly depend on jersey-core
• [HIVE-7200] - Beeline output displays column heading even if --showHeader=false is set
• [HIVE-8205] - Using strings in group type fails in ParquetSerDe
• [HIVE-7017] - Insertion into Parquet tables fails under Tez
• [HIVE-7201] - Fix TestHiveConf#testConfProperties test case
• [HIVE-8138] - Global Init file should allow specifying file name not only directory
• [HIVE-7647] - Beeline does not honor --headerInterval and --color when executing with "-e"
• [HIVE-7441] - Custom partition scheme gets rewritten with hive scheme upon concatenate
• [HIVE-8083] - Authorization DDLs should not enforce hive identifier syntax for user or group
• [HIVE-7777] - Add CSV Serde based on OpenCSV
• [HIVE-8139] - Upgrade commons-lang from 2.4 to 2.6
• [HIVE-7353] - HiveServer2 using embedded MetaStore leaks JDOPersistanceManager
• [HIVE-7373] - Hive should not remove trailing zeros for decimal numbers
• [HIVE-8034] - Don't add colon when no port is specified
• [HIVE-3756] - "LOAD DATA" does not honor permission inheritence
• [HIVE-6785] - query fails when partitioned table's table level serde is ParquetHiveSerDe and partition level serde is of different SerDe
• [HIVE-1363] - 'SHOW TABLE EXTENDED LIKE' command does not strip single/double quotes
• [HIVE-7629] - Problem in SMB Joins between two Parquet tables
• [HIVE-7620] - Hive metastore fails to start in secure mode due to "java.lang.NoSuchFieldError: SASL_PROPS" error
• [HIVE-6820] - HiveServer(2) ignores HIVE_OPTS
• [HIVE-7220] - Empty dir in external table causes issue (root_dir_external_table.q failure)
• [HIVE-8008] - NPE while reading null decimal value
• [HIVE-6741] - HiveServer2 startup fails in secure (kerberos) mode due to backward incompatible hadoop change
• [HIVE-7682] - HadoopThriftAuthBridge20S should not reset configuration unless required
• [HIVE-7890] - SessionState creates HMS Client while not impersonating
• [HIVE-6245] - HS2 creates DBs/Tables with wrong ownership when HMS setugi is true
• [HIVE-7730] - Extend ReadEntity to add accessed columns from query
• [HIVE-7889] - Query fails with char partition column
• [HIVE-6745] - HCat MultiOutputFormat hardcodes DistributedCache keynames
• [HIVE-7298] - desc database extended does not show properties of the database
• [HIVE-7824] - CLIServer.getOperationStatus eats ExceutionException
• [HIVE-6185] - DDLTask is inconsistent in creating a table and adding a partition when dealing with location
• [HIVE-7829] - Entity.getLocation can throw an NPE
• [HIVE-7635] - Query having same aggregate functions but different case throws IndexOutOfBoundsException
• [HIVE-7279] - UDF format_number() does not work on DECIMAL types
• [HIVE-7257] - UDF format_number() does not work on FLOAT types
• [HIVE-5456] - Queries fail on avro backed table with empty partition
• [HIVE-7345] - Beeline changes its prompt to reflect successful database connection even after failing to connect
• [HIVE-6622] - UDF translate doesn't take either CHAR or VARCHAR as any of its arguments
• [HIVE-7459] - Fix NPE when an empty file is included in a Hive query that uses CombineHiveInputFormat
• [HIVE-7042] - Fix stats_partscan_1_23.q and orc_createas1.q for hadoop-2
• [HIVE-7004] - Fix more unit test failures on hadoop-2
• [HIVE-6947] - More fixes for tests on hadoop-2
• [HIVE-6960] - Set Hive pom to use Hadoop-2.4
• [HIVE-7149] - Parquet not able to handle negative decimal numbers
• [HIVE-6684] - Beeline does not accept comments that are preceded by spaces
• [HIVE-6979] - Hadoop-2 test failures related to quick stats not being populated correctly
• [HIVE-7450] - Database should inherit perms of warehouse dir
• [HIVE-6858] - Unit tests decimal_udf.q, vectorization_div0.q fail with jdk-7.
• [HIVE-6602] - Multi-user HiveServer2 throws error
• [HIVE-6322] - Fix file_with_header_footer_negative.q
• [HIVE-6880] - TestHWISessionManager fails with -Phadoop-2
• [HIVE-7075] - JsonSerde raises NullPointerException when object key is not lower case
• [HIVE-7053] - Unable to fetch column stats from decimal columns
• [HIVE-7170] - Fix display_colstats_tbllvl.q in trunk
• [HIVE-7050] - Display table level column stats in DESCRIBE EXTENDED/FORMATTED TABLE
• [HIVE-6928] - Beeline should not chop off "describe extended" results by default
• [HIVE-4561] - Column stats : LOW_VALUE (or HIGH_VALUE) will always be 0.0000 ,if all the column values larger than 0.0 (or if all column values smaller than 0.0)
• [HIVE-7117] - Partitions not inheriting table permissions after alter rename partition
• [HIVE-7119] - Extended ACL's should be inherited if warehouse perm inheritance enabled
• [HIVE-7092] - Insert overwrite should not delete the original directory
• [HIVE-6913] - Hive unable to find the hashtable file during complex multi-staged map join
• [HIVE-7099] - Add Decimal datatype support for Windowing
• [HIVE-7015] - Failing to inherit group/permission should not fail the operation
• [HIVE-6916] - Export/import inherit permissions from parent directory
• [HIVE-6891] - Alter rename partition Perm inheritance and general partition/table group inheritance
• [HIVE-6648] - Permissions are not inherited correctly when tables have multiple partition columns
• [HIVE-6893] - out of sequence error in HiveMetastore server
• [HIVE-5847] - DatabaseMetadata.getColumns() doesn't show correct column size for char/varchar/decimal
• [HIVE-6792] - hive.warehouse.subdir.inherit.perms doesn't work correctly in CTAS
• [HIVE-6793] - DDLSemanticAnalyzer.analyzeShowRoles() should use HiveAuthorizationTaskFactory
• [HIVE-6612] - Misspelling "schemaTool completeted"
• [HIVE-6817] - Some hadoop2-only tests need diffs to be updated
• [HIVE-6822] - TestAvroSerdeUtils fails with -Phadoop-2
• [HIVE-6861] - more hadoop2 only golden files to fix
• [HIVE-5536] - Incorrect Operation Name is passed to hookcontext

Improvement

• [HIVE-5771] - Constant propagation optimizer for Hive
• [HIVE-9328] - Tests cannot move files due to change on HIVE-9325
• [HIVE-6024] - Load data local inpath unnecessarily creates a copy task
• [HIVE-10629] - Dropping table in an encrypted zone does not drop warehouse directory
• [HIVE-10630] - Renaming tables across encryption zones renames table even though the operation throws error
• [HIVE-5193] - Columnar Pushdown for RC/ORC File not happening in HCatLoader
• [HIVE-2828] - make timestamp accessible in the hbase KeyValue
• [HIVE-7737] - Hive logs full exception for table not found
• [HIVE-9788] - Make double quote optional in tsv/csv/dsv output
• [HIVE-5700] - enforce single date format for partition column storage
• [HIVE-9265] - Hive with encryption throws NPE to fs path without schema
• [HIVE-7858] - Parquet compression should be configurable via table property
• [HIVE-8904] - Hive should support multiple Key provider modes
• [HIVE-8864] - Fix permission inheritance with HDFS encryption
• [HIVE-8245] - Collect table read entities at same time as view read entities
• [HIVE-6430] - MapJoin hash table has large memory overhead
• [HIVE-6758] - Beeline doesn't work with -e option when started in background
• [HIVE-8471] - Cleanup TestSSL
• [HIVE-6657] - Add test coverage for Kerberos authentication implementation using Hadoop's miniKdc
• [HIVE-7432] - Remove deprecated Avro's Schema.parse usages
• [HIVE-6148] - Support arbitrary structs stored in HBase
• [HIVE-6147] - Support avro data stored in HBase columns
• [HIVE-3635] - allow 't', 'T', '1', 'f', 'F', and '0' to be allowable true/false values for the boolean hive type
• [HIVE-6584] - Add HiveHBaseTableSnapshotInputFormat
• [HIVE-7211] - Throws exception if the name of conf var starts with "hive." does not exists in HiveConf
• [HIVE-6473] - Allow writing HFiles via HBaseStorageHandler table
• [HIVE-7048] - CompositeKeyHBaseFactory should not use FamilyFilter
• [HIVE-6411] - Support more generic way of using composite key for HBaseHandler
• [HIVE-6677] - HBaseSerDe needs to be refactored
• [HIVE-8130] - Support Date in Avro
• [HIVE-6561] - Beeline should accept -i option to Initializing a SQL file
• [HIVE-5871] - Use multiple-characters as field delimiter
• [HIVE-5160] - HS2 should support .hiverc
• [HIVE-5799] - session/operation timeout for hiveserver2
• [HIVE-7554] - Parquet Hive should resolve column names in case insensitive manner
• [HIVE-6438] - Sort query result for test, removing order by clause
• [HIVE-6938] - Add Support for Parquet Column Rename
• [HIVE-7445] - Improve LOGS for Hive when a query is not able to acquire locks
• [HIVE-7196] - Configure session by single open session call
• [HIVE-5946] - DDL authorization task factory should be better tested
• [HIVE-7140] - Bump default hive.metastore.client.socket.timeout to 5 minutes

New Feature

• [HIVE-7299] - Enable metadata only optimization on Tez
• [HIVE-7390] - Make quote character optional and configurable in BeeLine CSV/TSV output
• [HIVE-6806] - CREATE TABLE should support STORED AS AVRO
• [HIVE-6380] - Specify jars/files when creating permanent UDFs
• [HIVE-5823] - Support for DECIMAL primitive type in AvroSerDe

Task

• [HIVE-8465] - Fix some minor test fails on trunk
• [HIVE-8707] - Fix ordering differences due to Java 8 HashMap function
• [HIVE-8935] - Add debug logging around token stores
• [HIVE-8862] - Fix ordering diferences on TestParse tests due to Java8
• [HIVE-7108] - Cleanup HBaseStorageHandler
• [HIVE-8224] - Support Char, Varchar in AvroSerDe
• [HIVE-5976] - Decouple input formats from STORED as keywords
• [HIVE-7034] - Explain result of TezWork is not deterministic
• [HIVE-7095] - Fix test fails for both hadoop-1 and hadoop-2
• [HIVE-7735] - Implement Char, Varchar in ParquetSerDe
• [HIVE-6394] - Implement Timestmap in ParquetSerde
• [HIVE-6367] - Implement Decimal in ParquetSerde

Test

• [HIVE-4937] - Create description annotations for vectorized UDF
• [HIVE-7085] - TestOrcHCatPigStorer.testWriteDecimal tests are failing on trunk