Putting ERP to Work in Higher Ed: Spinning Wheel of Death

Examining the ongoing challenges of delivering high-quality, value-added ERP services in Higher Education.

Sunday, August 19, 2012

Spinning Wheel of Death

What fun we had last week trouble-shooting predictably unpredictable failures on our OBIEE development environment. It all began with Noetix, a set of pre-packaged content for Oracle E-Business Suite Financials that we acquired to jump-start our transition to modern BI tools. (But it was not really Noetix' fault, as you'll see).

There were some new roles (as expected) and an RPD to merge (no biggie) and some catalog content, too (par for the course in multi-user development). All things we will face again and again in our future as multiple development teams author content for thousands of users.

For roughly 38 minutes after the install was done, everything looked great. And then we saw the following behaviors:

Perpetual spinning icon when trying to open the Dashboards Menu
No response (not even a browser error) when selecting New > Dashboard
Permissions error when opening ANY Dashboard object (not only new merged content)

Throughout this time I could create ad hoc analyses, run saved analyses from the catalog (both pre-existing and new Noetix content), etc.

Attempting to open any Dashboard prompted this error:

This message-- seemingly a failure related to one of the roles created during our Noetix installation process -- cried out for corruption somewhere... We started by scouring the permissions on Dashboards features: everything was fine. Checked the policy store in Weblogic / Enterprise Manager: looked fine. Checked the permission on catalog content; tweaked the settings -- problem solved! For 38-41 minutes, upon which I saw this, again:

We reverted to our RPD: no luck. Reverted to old RPD: spinning, spinning, spinning. Restarted services. Bounced servers. Cleared caches. Any "fix" we thought we found lasted 40 minutes.

Finally, we found relevant articles on My Oracle Support:

Oracle Support Document 1390497.1 (How To Clean Corrupted Permissions On An Upgraded OBIEE 11g Presentation Server Catalog)
Oracle Support Document 1388837.1 (OBIEE 11g: Clicking on "Manage Privileges" Throws
"Error retrieving user/group data from Oracle BI Server's User Population API")

Unfortunately, our first attempt at this catalog clean up failed. Third or fourth try worked like a charm (changing from "Report" to "Clean" mode also critical). The Oracle OBIEE 11.1.1.6 documentation actually describes the process rather clearly, but we had never thought to look there!

Problem solved; onward to new problems next week!

Labels: OBIEE, Oracle, Oracle BI