CDH 5.1.5 Release Notes

Changes Not In Apache Oozie 4.0.0

Oozie

Bug

• [OOZIE-2063] - Cron syntax creates duplicate actions
• [OOZIE-2034] - Disable SSLv3 (POODLEbleed vulnerability)
• [OOZIE-1989] - NPE during a rerun with forks
• [OOZIE-1920] - Capture Output for SSH Action doesn't work
• [OOZIE-1865] - Oozie servers can't talk to each other with Oozie HA and Kerberos
• [OOZIE-1907] - DB upgrade from 3.3.0 to trunk fails on derby
• [OOZIE-1879] - Workflow Rerun causes error depending on the order of forked nodes
• [OOZIE-1828] - Introduce counters JobStatus terminal states metrics
• [OOZIE-1319] - "LAST_ONLY" in execution control for coordinator job still runs all the actions
• [OOZIE-1834] - sla should-start is supposed to be optional but it is not
• [OOZIE-1838] - jdbc.connections.active sampler does not show up
• [OOZIE-1783] - Sharelib purging only occurs at Oozie startup
• [OOZIE-1809] - offset and len options are ignored in oozie job -info for workflow
• [OOZIE-1762] - Sharelib with oozie.action.ship.launcher.jar=true should copy oozie-hadoop-utils.jar
• [OOZIE-1787] - parameterize interval of SLAService updating SlaStatus
• [OOZIE-1777] - duplicated log message in Pig launcher's stdout
• [OOZIE-1748] - When using cron-like syntax, the "Time Unit" field says "MINUTE"
• [OOZIE-1737] - Oozie log streaming is slow
• [OOZIE-1668] - Coord log streaming start and end time should be of action list start and end time
• [OOZIE-1799] - Document hcatalog integration steps for Oozie in a secure cluster
• [OOZIE-1785] - Add oozie email action xsd to OozieCli.java
• [OOZIE-1797] - Workflow rerun command should use existing workflow properties
• [OOZIE-1796] - Job status should not transition from KILLED
• [OOZIE-1781] - UI - Last Modified time is not displayed for coord action in coord job info grid
• [OOZIE-1792] - Ability to kill bundle stuck in RUNNING due to inconsistent pending states
• [OOZIE-1719] - v1/jobs api returns null for parentId even when it exists
• [OOZIE-1773] - bulk API returns total = 0 when it's not
• [OOZIE-1774] - Expected/Actual Duration on UI SLA Tab doesn't show correct information
• [OOZIE-1754] - add order(sort) option and exclude filter for coord job Info
• [OOZIE-1761] - Improve sharelib purging logic
• [OOZIE-1725] - add coord EL functions to be used in SLA tag
• [OOZIE-1732] - Sharelib instrumentation fails if sharelib.system.libpath is not created.
• [OOZIE-1692] - modify log message when checking completion of child job in Map-Reduce action
• [OOZIE-1734] - Oozie returned 500 Internal Server error when user passes invalid request
• [OOZIE-1713] - Avoid creating dummy input file for each launcher job
• [OOZIE-1701] - TestXTestCase#testWaitFor and TestXTestCase#testBaseDir have the errors
• [OOZIE-1751] - Default authentication type using env variable for oozie CLI
• [OOZIE-1753] - Update create-release-artifact script for git
• [OOZIE-1718] - Coord Job Query UPDATE_COORD_JOB_CHANGE does not update last modified time
• [OOZIE-1693] - UI timeout while loading job table
• [OOZIE-1712] - Oozie page does not render in Internet Explorer 9 on Windows
• [OOZIE-1720] - Oozie Hive action doesn't honour mapred.job.name.
• [OOZIE-1543] - Minor typo in Hive Action documentation
• [OOZIE-1650] - Whitelisting docs are incorrect
• [OOZIE-1223] - too many extjs cookies can cause the web UI to stop responding
• [OOZIE-1671] - add an option to limit # of coordinator actions for log retrieval
• [OOZIE-1629] - EL function in <timeout> is not evaluated properly
• [OOZIE-1687] - Bundle can still be in RUNNINGWITHERROR status after bundle kill
• [OOZIE-1622] - Multiple CoordSubmit for same bundle
• [OOZIE-1658] - Add bundle, coord, wf and action related information to launched M/R jobs.
• [OOZIE-1610] - UnitTests fail on Windows because of wrong paths
• [OOZIE-1598] - enable html email in email action
• [OOZIE-1835] - NullPointerException from SLAEmailEventListener
• [OOZIE-1823] - OozieSharelibCLI shouldn't load ext services
• [OOZIE-1765] - JMS Notifications for Workflows not always on the correct topic
• [OOZIE-1794] - java-opts and java-opt in the Java action don't always work properly in YARN
• [OOZIE-1768] - Workflow schema 0.4.5 was removed
• [OOZIE-1714] - Update Derby driver version to latest
• [OOZIE-1699] - Some of the commands submitted to Oozie internal queue are never executed
• [OOZIE-1681] - Sqoop sharelib has no hsqldb jar version
• [OOZIE-1683] - UserGroupInformationService should close any filesystems opened by it
• [OOZIE-1670] - Workflow kill command doesn't kill child job for map-reduce action.
• [OOZIE-1663] - Queuedump to display command type
• [OOZIE-1672] - UI info fetch fails for bundle having large number of coordinators.
• [OOZIE-1666] - Child job link not working in safari and chrome
• [OOZIE-1664] - PollablePriorityDelayQueue.poll() returns elements with +ve delay
• [OOZIE-1661] - Stream logs in oozie UI
• [OOZIE-1662] - Correct some used and unused Workflow Action SELECT queries
• [OOZIE-1632] - Coordinators that undergo change endtime but are doneMaterialization, not getting picked for StatusTransit
• [OOZIE-1642] - writeUTF 64k limit for counters.
• [OOZIE-1605] - Add common custom filter applied to Wf/Coord/Bundle jobs on oozie UI
• [OOZIE-1691] - StackOverflowError in TimestampedMessageParser.parseNextLine()
• [OOZIE-1608] - Update Curator to 2.4.0 when its available to fix security hole
• [OOZIE-1684] - DB upgrade from 3.3.0 to trunk fails on Oracle
• [OOZIE-1674] - DB upgrade from 3.3.0 to trunk fails on postgres
• [OOZIE-1660] - DB connection misconfig causes all or most unit tests to fail
• [OOZIE-1655] - Change oozie.service.JPAService.validate.db.connection to true
• [OOZIE-1641] - oozie-audit.log - add remote IP.
• [OOZIE-1643] - Oozie doesn't parse Hadoop Job Id from the Hive action
• [OOZIE-1647] - oozie-setup.sh doesn't check exit code of java executions
• [OOZIE-1635] - verifySlaElement in submitXCommand.java should get sla info from action child as well
• [OOZIE-1630] - <prepare> operations fail when path doesn't have scheme
• [OOZIE-1627] - Rerun doesn't resolve workflow app name
• [OOZIE-1626] - pig action pop-up is not working properly in UI
• [OOZIE-1548] - OozieDBCLI changes to convert clob to blob and remove the discriminator column
• [OOZIE-1634] - TestJavaActionExecutor#testUpdateConfForUberMode fails against Hadoop 2
• [OOZIE-1631] - Tools module should have a direct dependency on mockito
• [OOZIE-1615] - shell action cannot find script file and fails in uber mode
• [OOZIE-1623] - JPAService doesn't need to do reads in a transaction
• [OOZIE-1604] - <java-opts> and <java-opt> not added to Application Master property in uber mode
• [OOZIE-1517] - Support using MS SQL Server as a metastore
• [OOZIE-1314] - IllegalArgumentException: wfId cannot be empty
• [OOZIE-1544] - Remove references of PigStatsUtil from OoziePigStats
• [OOZIE-1603] - cannot submit job to oozie on mysql
• [OOZIE-1600] - map-reduce actions without configuration section in workflow.xml throws "IllegalArgumentException: element cannot be null"
• [OOZIE-1542] - When extjs isn't installed, the web UI is unhelpfully blank
• [OOZIE-1562] - Allow re-run of actions of killed coordinator
• [OOZIE-1481] - Getting a coordinator job info with len=0 should return 0 actions
• [OOZIE-1541] - Typo in Oozie HA admin -servers command in documentation
• [OOZIE-1573] - coord:tzOffset() gives incorrect offset for daylight saving timezones
• [OOZIE-1592] - El Expression Reference should point to the exact chapter in java 6
• [OOZIE-1578] - Coordinator jobs with cron frequency should be invalidated upon submission if nothing would be materialized during run time
• [OOZIE-1577] - Oozie coordinator job with identical start and end time remains "RUNNING" forever
• [OOZIE-1559] - Fix missing fields from new SELECT queries and Recovery Service picking up killed control nodes
• [OOZIE-1500] - Fix many OS-specific issues on Windows
• [OOZIE-1556] - Change Bundle SELECT query to fetch only necessary columns and consolidate JPA Executors
• [OOZIE-1558] - RAT Warning from BundleActionsGetJPAExecutor.java
• [OOZIE-1547] - Change Coordinator SELECT query to fetch only necessary columns and consolidate JPA Executors
• [OOZIE-1524] - Change Workflow SELECT query to fetch only necessary columns and consolidate JPA Executors
• [OOZIE-1539] - Load more coordinator jobs eligible to be materialized in MaterializeTriggerService
• [OOZIE-1499] - Update only necessary columns and consolidate JPA Executors
• [OOZIE-1461] - provide an option to auto-deploy launcher jar onto HDFS system libpath
• [OOZIE-1520] - Sequencefile Reader fails to use doas for reading action data file
• [OOZIE-1589] - TestZKLocksService is flakey
• [OOZIE-1596] - TestOozieMySqlDBCLI.testCreateMysql fails when tests are executed in a different order
• [OOZIE-1582] - Bump up Tomcat version to 6.0.37
• [OOZIE-1568] - TestWorkflowClient.testSla is flakey
• [OOZIE-1563] - colt jar includes GPL licence
• [OOZIE-1526] - Oozie does not work with a secure HA JobTracker or ResourceManager
• [OOZIE-1557] - TestFsActionExecutor.testChmodWithGlob fails against Hadoop 2.1.x-beta
• [OOZIE-1545] - RecoveryService keeps repeatedly queueing SuspendXCommand
• [OOZIE-1529] - Disable job DAG display for workflow having huge actions.
• [OOZIE-1549] - Update hcat documentation to mention hcatalog-pig-adapter jar
• [OOZIE-1515] - Passing superset of action id range should be allowed
• [OOZIE-1509] - Do not preload all tabs in Oozie UI and make Active Jobs default
• [OOZIE-1496] - Oozie demo and streaming examples fails to run on Windows
• [OOZIE-1522] - SignalX may try to insert transition for a forked node twice
• [OOZIE-1513] - Workflow stays in running if Fork/join validation or loop detection fails
• [OOZIE-1490] - Remove unix OS enforcement from build
• [OOZIE-1372] - When using uber mode, Oozie should also make the AM container size larger
• [OOZIE-1495] - inconsistent behavior of chmod/chgrp when path doesn't exist after glob support
• [OOZIE-1463] - Remove discriminator column
• [OOZIE-1448] - A CoordActionUpdateXCommand gets queued for all workflows even if they were not launched by a coordinator
• [OOZIE-1471] - Support glob in FS action and prepare blocks
• [OOZIE-1546] - TestMapReduceActionExecutorUberJar.testMapReduceWithUberJarEnabled fails
• [OOZIE-1284] - oozie.service.SchemaService.wf.ext.schemas in oozie-site is missing some newer xsd files
• [OOZIE-1540] - When oozie.zookeeper.oozie.id is not specified, its using a space instead of the hostname
• [OOZIE-1443] - forkjoin validation should not allow a fork to go to the same node multiple times
• [OOZIE-1403] - forkjoin validation blocks some valid cases involving decision nodes
• [OOZIE-1449] - Coordinator Workflow parent relationship is broken for purge service
• [OOZIE-1447] - Sqoop actions that don't launch a map reduce job fail with an IllegalArgumentException
• [OOZIE-1440] - Build fails in certain environments due to xerces OpenJPA issue

Improvement

• [OOZIE-1724] - Make it easier to specify the HCat hive-site.xml for the Oozie Server
• [OOZIE-1801] - ZKLocksService instrumentation should say how many locks this server has
• [OOZIE-1698] - Action sharelib configuration document lacks the "oozie." prefix
• [OOZIE-1581] - Workflow performance optimizations
• [OOZIE-1504] - Allow specifying a fixed instance as the start instance of a data-in
• [OOZIE-1849] - If the underlying job finishes while a Workflow is suspended, Oozie can take a while to realize it
• [OOZIE-1826] - Add thread which detects JVM pauses
• [OOZIE-1621] - Add proper error code and error message for sharelib exceptions.
• [OOZIE-1733] - Fix test failures caused by OOZIE-1722
• [OOZIE-1722] - When an ApplicationMaster restarts, it restarts the launcher job
• [OOZIE-1680] - Add a check for a maximum frequency of 5 min on Coord jobs
• [OOZIE-1618] - dryrun should check variable substitution in workflow.xml
• [OOZIE-1609] - HA support for share lib.
• [OOZIE-1646] - HBase Table Copy between two HBase servers doesn't work with Kerberos
• [OOZIE-1474] - Fix logging issues - latency, accurate job ids, coord Job UI to show job logs
• [OOZIE-1612] - When printing Dates to log messages, we should make sure they are in oozie.processing.timezone
• [OOZIE-1468] - Add created time column in WF_ACTIONS and SLA tables
• [OOZIE-1616] - Add sharelib and launcherlib locations to the instrumentation info
• [OOZIE-1519] - Admin command to update the sharelib
• [OOZIE-1633] - Test failures related to sharelib when running against Hadoop 2
• [OOZIE-1584] - Setup sharelib using script and pickup latest(honor ship.launcher) and remove DFS dependency at startup.
• [OOZIE-1550] - Create a safeguard to kill errant recursive workflows before they bring down oozie
• [OOZIE-1606] - Update Curator to 2.3.0 and fix some misc minor ZK related things
• [OOZIE-1580] - EL variables don't get resolved in configurations imported from a <job-xml>
• [OOZIE-1597] - Cleanup database before every test
• [OOZIE-1551] - Change hadoop-2 profile to use 2.2.0
• [OOZIE-1569] - Maintain backward incompatibility for running jobs before upgrade
• [OOZIE-1570] - Make openjpa connection properties configurable
• [OOZIE-1503] - [DB optimization] revisit eagerLoadState at places
• [OOZIE-1555] - LauncherMapper to check for sys properties before opening files for action data
• [OOZIE-1462] - Compress lob columns before storing in database
• [OOZIE-1486] - Cut down on number of small files created to track a running action
• [OOZIE-1491] - Make sure HA works with a secure ZooKeeper
• [OOZIE-1560] - Log messages should have a way of identifying which server they came from when using HA
• [OOZIE-1460] - Implement and Document security for HA
• [OOZIE-1476] - Add ability to issue kill on Coordinator Action directly with id and nominal daterange
• [OOZIE-1458] - If a Credentials type is not defined, Oozie should say something

New Feature

• [OOZIE-7] - Ability to view the log information corresponding to particular coordinator actions
• [OOZIE-1746] - New API to fetch workflows corresponding to coordinator action reruns
• [OOZIE-1704] - Add ability to use Bulk API with bundle ID
• [OOZIE-1576] - Add documentation for Oozie Sqoop CLI
• [OOZIE-1575] - Add functionality to submit sqoop jobs through http on oozie server side
• [OOZIE-1566] - Add reference to Quartz module in cron documentation
• [OOZIE-1454] - Documentation for cron syntax scheduling of coordinator job
• [OOZIE-1306] - add flexibility to oozie coordinator job scheduling
• [OOZIE-1425] - param checker should validate cron syntax
• [OOZIE-1453] - Change "frequency" to string in SyncCoordAction.java
• [OOZIE-615] - Support high availability for the Oozie service

Task

• [OOZIE-1775] - TestEventGeneration.testCoordinatorActionEvent is failing and CoordRerunX should generate event