Jira takes long time to restart after upgrade

We noticed that Jira startup time increased from 3+ mins to 20+ mins after a recent upgrade (7.10 to 7.12).

After analysed a few restart relevant events in the logs, I found that there are some messages related to DB row migrations which were not there before the upgrade. Here is a example:

5/3/19
9:43:14.816 PM	
2019-05-03 21:43:14,816 JIRA-Bootstrap INFO      [c.a.j.p.devstatus.upgrade.UpgradeTask_Build03] Running upgrade task to migrate rows.

5/3/19
9:44:22.271 PM	
2019-05-03 21:44:22,271 JIRA-Bootstrap INFO      [c.a.j.p.d.u.devsummary.migration.SQLMigrationHandler] Migrating rows 

5/3/19
9:54:47.772 PM	
2019-05-03 21:54:47,772 JIRA-Bootstrap INFO      [c.a.j.p.d.u.devsummary.migration.SQLMigrationHandler] Total Migrated Rows in current pass 3012490

5/3/19
10:05:29.498 PM	
2019-05-03 22:05:29,498 JIRA-Bootstrap ERROR      [c.a.j.p.d.u.devsummary.migration.DevSummaryMigrationHandler] Error while upgrading rows.

5/3/19
10:07:58.328 PM	
2019-05-03 22:07:58,328 Caesium-1-4 INFO ServiceRunner     [c.a.j.p.devstatus.upgrade.UpgradeTask_Build03] Running upgrade task to migrate rows.

5/3/19
10:08:03.143 PM	
2019-05-03 22:08:03,143 Caesium-1-4 INFO ServiceRunner     [c.a.j.p.d.u.devsummary.migration.SQLMigrationHandler] Migrating rows 

	5/3/19
10:17:42.551 PM	
2019-05-03 22:17:42,551 Caesium-1-4 INFO ServiceRunner     [c.a.j.p.d.u.devsummary.migration.SQLMigrationHandler] Total Migrated Rows in current pass 3012490

5/3/19
10:27:50.016 PM	
2019-05-03 22:27:50,016 Caesium-1-4 ERROR ServiceRunner     [c.a.j.p.d.u.devsummary.migration.DevSummaryMigrationHandler] Error while upgrading rows.

According to above, I know why Jira takes so long to start, as the log says “upgrade task to migrate rows.“. But why it shows in every restart? the log also says “Error while upgrading rows.“. Right, it makes sense. As the row upgrade failed each time, so Jira has to try it again and again at each restart.

By now, we need to find out the root cause of why the row upgrades failed. The logs tell me that as well:

2019-05-03 22:05:29,498 JIRA-Bootstrap ERROR      [c.a.j.p.d.u.devsummary.migration.DevSummaryMigrationHandler] Error while upgrading rows.
java.util.concurrent.ExecutionException: com.querydsl.core.QueryException: Caught SQLException for delete from "ENTITY_PROPERTY" where "ENTITY_PROPERTY"."ENTITY_NAME" = ? and "ENTITY_PROPERTY"."UPDATED" < ?

Caused by: com.querydsl.core.QueryException: Caught SQLException for delete from "ENTITY_PROPERTY" where "ENTITY_PROPERTY"."ENTITY_NAME" = ? and "ENTITY_PROPERTY"."UPDATED" < ?

Caused by: java.sql.SQLException: ORA-30036: unable to extend segment by 8 in undo tablespace 'UNDOTBS1'

Caused by: Error : 30036, Position : 12, Sql = delete from "ENTITY_PROPERTY" where "ENTITY_PROPERTY"."ENTITY_NAME" = :1  and "ENTITY_PROPERTY"."UPDATED" < :2 , OriginalSql = delete from "ENTITY_PROPERTY" where "ENTITY_PROPERTY"."ENTITY_NAME" = ? and "ENTITY_PROPERTY"."UPDATED" < ?, Error Msg = ORA-30036: unable to extend segment by 8 in undo tablespace 'UNDOTBS1'

“unable to extend segment by 8 in undo tablespace ‘UNDOTBS1‘” is the root cause! As when Jira upgrades, it needs to update the database which requires the undo table where rollback information is kept. But due to the small undo tablespace, the upgrade process failed to write the rollback info. After the DBA increased the undo tablespace in the Oracle database, the upgrade/ row migration completed successfully! And the startup time drops to the normal range.

How to check the undo tablespace in Oracle: