Independent Project Oversight Report (IPOR) Rating

Print

8860-030 Financial Information System for California (FI$Cal)

​This project will replace the state's aging and non-integrated financial systems with a single comprehensive financial application supporting the state's fiscal and policy decision processes. The proposed solution is to implement an Enterprise Resource Planning application that will meet the state's budget, accounting, and some procurement needs. The solution will also address various fiscal information needs of the Legislature.

Project Details

Agency / State Entity General Government / Finance, Department of
Total Cost $1,063,071,368
Last Approved Start Date 08/01/2005
Last Approved Finish Date 07/01/2020
Criticality Rating High
Most Recent Project Approval Document Special Project Report 8
Date of Most Recent Project Approval Document 2/1/2018
IPOR Reporting Period Overall IPOR Rating
06/01/2020 - 06/30/2020
Red
Print the IPOR Dashboard

Key Questions

Is the project on track to satisfy the customer's business objectives?No

Is the project on track to achieve the objectives in the approved timeframe?No

Is the project on track to achieve the objectives within the approved budget?No

Project Overall Health Comments
Red
In June 2020 the project’s overall health continues to remain Red. The project could not deliver the Minimum Viable Product (MVP) scope as approved in SPR 8 by June 2020. The project plans to formally re-baseline its schedule using an Iterative Project Report (IPR) to bring the schedule back to a green status. The project is expected to submit its first draft after the June 2020 Release. The Milestone 3 (M3) and Milestone 4 (M4) Business Processes (BP) code was deployed into production in the end of June 2020. The development and testing faced several challenges of scope changes and resource contentions. The code base is tested with a few critical scenarios and are subject to many other scenarios in production use. The project must be commended for completing the UAT and deploying the integrated solution. As the project loads ledger data and starts using the deployed functionality, we will know more about the deployed BP’s fit for purpose. The project status is based on the following reasons • The project did not meet MVP Schedule as approved in SPR8. • The project did not deliver full scope as approved in SPR8. • The projects deployments continue to require extensive manual support to ensure departments can transact using FISCal on a day to day basis. • Several material departments continue to face delays with completing their MEC and YEC in a reasonable timeframe using FISCal due to functional and technical issues that need to be resolved. • The project continues to face challenges with Agile process management leading to missed early schedule and scope risk mitigation. • FISCal is not keeping the backlog UpToDate and this is resulting in differences in projects reports on progress vs. what is in the backlog (Tool used to manage Agile work). • Due to the lack of a fully functional analysis and reporting toolset, users continue to face challenges with using their financial data within FISCal. • The Risks and Issues Management process continues to miss capturing, managing and mitigating scope and schedule risks prior to becoming issues. • Future performance of FISCal and batch processing is at risk from increased volume of new data coming in as a result the complex integrated solution. The project has documented this risk. Scope Management: Scope management focus area continues to remain Red through June 2020. The project continued to change the scope of features within BP’s as late as June to meet the June 2020 release schedule. The formal re-baselining of the project scope will be required to change the status of this focus area. The process of formal re-baselining will occur after the June 2020 release using an IPR. • Scope changes at the last minute can be seen from the examples below: In June 2020, the following scope is removed from MVP though Project’s governance process. • BP127- Renumbering Bond Sub Funds (In SPR 8 this feature is part of BP42: Ledger Conversion). This feature is not required for Milestone 4; however, it must be implemented before the SCO legacy system is retired (End State). • In May 2020, the following scope changes were made o Project divided the Plan of Financial Adjustments (PFA) interface BP into three parts. Project has gone live with part 1 required for Milestone 4 in June 2020. The remaining two parts of the scope will be delivered by August 2020. o Project had identified an additional report required to reconcile balances to the FISCal Cash Basis ledger with new CVAC Budget Detail Exception Report. Due to a lack of time, the project decided to develop PeopleSoft query for June 2020 release and develop the actual reports post go-live. o The project is also deploying two major control processes (Cash Validation and control and PFA interfaces) in a phased manner instead of deploying them for all departments by June 2020 as approved in SPR8. • Another metric showing the increase in scope of effort for M4 business processes in May was observed from the increasing story points in the product backlog. • The project had removed four business processes from project scope prior to January 2020 (BP51, BP74, BP74A and BP85A (April 2020). • The SOCA solution also experienced increasing scope in terms of additional reports being identified. This resulted in the SOCA solution being removed from the June 2020 release. • The Ledger Recon tool carries risks of scope increases as users start using functionality in production in the end of July. Time Management: Time Management focus area remains Red in June 2020 due to the missed M5 schedule and missed schedule for key M3 and M4 BPs. Time management is connected to overall schedule management challenges the project has faced • The deployment schedule for M5 functionality originally scheduled for June 2020 was missed and is now planned for June 2021. • Unanticipated new tasks and design changes led to missed schedules with PFA, the full suite of Ledger Reconciliation tools, Statement of Cash Accountability (SOCA) BPs. • The project has planned to turn on CA/CVAC Cash controls in a phased manner after July 2020 instead of turning them on for all FISCal departments with the start of the integrated solution in July 2020. • The project continued to be behind in build, test, and validation tasks for M3 and M4. This has resulted in missed deployments of Direct Transfer Reversals in M3 and SOCA solution in M4. At least two other BP’s are deployed partially due to schedule issues. • MEC/YEC effort originally expected to be complete by October 15th of 2019 continues to be incomplete in June 2020. Quality Management: The project continues to have Quality Management process challenges, System/Data quality issues and Implementation quality issues as seen from the incidents, problem tickets and pending defect fixes as well as IV&V teams’ observations on the defects generated in the system from previous defect fixes. Quality Management process issues: • After one year of working on automated testing, the project has not yet fully automated all critical scripts. Additionally, Critical scripts are only a subset of all testing scripts. Due to this, we still observe changes to production breaking already working functionality. • FISCal’s automated regression testing suite currently only covers critical scripts. There is no time for manual testing of all the remaining scripts. For Agile sprints, the lack of fully automated regression testing scripts for critical functionality causes issues for sprints testing goals and test coverage. • The project does not perform code validation at a level of detail to effectively identify defects before moving changes and new deployments to production due to schedule and resource constraints. Only a selected few critical function are validated for coding. • Performance test teams do not have dedicated performance test environments. This is leading to missed testing opportunities and defects identified only in production and performance issues being observed in production. System/Data quality Issues: • FISCal continues to fix transactional data through SQL changes to the database without fixing the underlying functional problems that cause the data issues. • FISCal system’s transactional performance is degraded when departmental users query the system for substantial amounts of information like large downloads of all departmental transactions. With the new integrated solution and additional interfaces bringing more data into the system and at least one BP requiring additional queries, this risk continues for the FISCal system performance. • The FISCal system does not have functionality (In some cases) to edit and change configuration data or transactional data through the application. This issue results in the project making direct changes to the backend database which in turn results in poor data quality. The project should prioritize the development of such functionality to reduce the risks from direct SQL changes to the database. The project has yet to make considerable progress in this area. • General Ledger (GL) and Commitment control (KK) ledgers mismatches resulted in reconciliation delays. The project deployed fixes to resolve this issue with the June release and CDT will report on the effectiveness of these fixes in the next few reports. • Lack of reasonable guardrails (business rules to validate data entry). FISCal is working on adding more guardrails for accounting functionality to be entered without avoidable errors. • FISCal does not have a formal set of Operational Level objectives (OLO’s) for its users. This results in users not knowing when to log incidents from poor transaction response times. Implementation quality: Project continues to face implementation quality issues due to the following issues which continue to exist through June 2020 • User stories are not sized small enough to complete an entire user story in one Sprint. Multiple user stories span over one sprint. This dilutes the ability of the project to measure its sprint team’s velocity and that leads to missed estimates and delays. • Lack of resources and time to perform code validation- Lack of robust code validation by FISCal teams (CDT is referring code validation of customized code and not PeopleSoft code base) • Performing UAT at a very high level using only happy path scenarios. Most of the defects in production are in areas of functionality that was not tested in UAT. • A very aggressive schedule for the milestone releases with large scale user impacts. This leads to effective testing being missed. • Lack of full functional testing (current testing is done for high level functions) • The inherent complications with test data harvesting for an accounting system where test data changes and must be created new for each round of testing. • Lack of appropriately scaled environments for performance testing • Continued errors with departmental accounting configurations (COA Values and Accounting Trees) Risk and Issue Management: Project did not manage key schedule and scope risks in a timely manner which have now impacted the projects scope, schedule and cost requiring another extension to the project. At FISCal, risks must be approved by FISCal partners (PBEs) to ensure they are supportive of the approach to mitigating/resolving the risk/issue before it can be proposed, and this has been a problem getting risks documented and approved quickly and mitigated before they become issues. With an aggressive deadlines and multiple paths of effort occurring simultaneously, it is important that risks get documented and mitigated as soon as they are observed to meet schedule and scope. As the project has deployed a critical integrated solution and is getting ready to deploy M5 P1 and P2 functionality, it is critical the project focusses on addressing issues with the risk and issue management process. A few issues that have to be addressed by the project • The project is not logging risks timely. The Risk Management process continues to take more than 6 months to approve risks and mitigations. (In the last quarter, we have seen improvement in the timely identification of risks related to the June 2020 release) • The FISCal solution is integral to a departments ability to complete its MEC and YEC efficiently. Based on the current challenges with departments ability to submit their annual fund status before the October 15th budget letter recommendation, CDT believes that the projects risk register should carry an open risk to MEC/YEC delays within FISCal to ensure formal mitigation can continue and reduce the risks to Financial year 2020/2021 MEC/YEC submissions. • The project does not consider the resource contention from SCO SME’s time being split between the support for MEC/YEC, Covid-19 impacts, daily responsibilities along with sprint and UAT tasks as a project UAT schedule and product quality risk. SCO SME resource contentions • Resource contentions exist with SCO SME resources availability for multiple concurrent tasks. The SME’s time is split between providing design support, perform UAT and assisting with MEC/YEC reconciliation as well as CAFR development starting in April of a given year. • Continued delays of completion of MEC and YEC by FISCal entities through June of 2020 have resulted in continued resource contentions and stress on the few Key SCO SME’s. • CoVID 19 continues to pose a risk of SME resource contentions and poses a risk to M5 P1 and P2 UAT and deployment by June 2021.
Focus Area/Rating Comments
Governance

yellow icon Yellow
The Governance focus area remains Yellow through June 2020 with due acknowledgement of the successful deployment if integrated solution code Project leadership continues to monitor and take proactive measures to ensure project success. The project started with a very aggressive schedule and was able to quickly identify that M5 P1 and P2 BP’s could not be delivered in late February. Leadership took quick decisions to remove this scope from the June 2020 release to focus on the challenging Integrated Solution code deployment. With the June 2020 Integrated Solution code release, the project faced several risks and issues w.r.t scope and testing and project governance made quick decisions to adjust course to deliver the code before the start of the financial year. However, this does not diminish the impact of missed schedule, not managing overall project risks effectively and the large amount of work left to be completed for the M5 P1 and P2 where governance is critical. CDT IPO will also monitor the leaderships management of contingencies that may arise with the daily reconciliation issues or performance issues or functional defects in the next few months of stabilization and report accordingly. Starting with a very aggressive schedule and underestimated scope of effort the project ran out of time to deliver committed scope by June 2020. • The project governance was not able to control scope of effort from increasing or bring removed due to the competing priorities of multiple stakeholders with different priorities. • The last few MEC and YEC challenges continued through June 2020 and the 2018/2019 FY CAFR is now late. • Project governance was unable to resolve the SCO SME resource contentions in time to avoid missing M5 deadlines and key M4 BP like SOCA. Recommendations to consider for improvement: Through a robust governance process the project leadership should implement measures for increased control of Scope, improved Quality Assurance (QA) and Control, and increase the effectiveness of Risks and Issues management. • Governance must address the risk from SCO SME resource availability for project work with the steering committee and finds mitigations to schedule risks due to this resource contention for the M5 P1 and P2 functionality. • Ensure Quality Management processes are strengthened to minimize quality issues with deployed software as well as the remaining M5 milestone deployments. Assign more time for QA testing. • Identify and establish Service Level Objectives and Implement them with corresponding performance monitoring and reporting processes in place • Encourage documentation, quick approval and mitigation of project level risks using the FISCal’s Risk and Issues management process.  The project should identify critical risks that exist and new ones and mitigate them by adding the risks to the risk register • Continue to identify the root causes of MEC and YEC process challenges departments are facing and resolve technical and FISCal business process issues to ensure these challenges do not recur for the next financial year of 2019/2020 • Implement and continue to monitor the new knowledge transfer (KT) plan effectively and ensure state team can support FISCal without additional System Integrator (SI) Non-Competitive Bids (NCB)
Time Management

red icon Red
Time Management focus area remains Red in June 2020 due to the missed M5 schedule and missed schedule for key M3 and M4 BPs. • The deployment schedule for M5 functionality originally scheduled for June 2020 was missed and is now planned for June 2021. • Unanticipated new tasks and design changes led to missed schedules with PFA, Deploying the full suite of Ledger Reconciliation tools, Statement of Cash Accountability (SOCA) BPs. • The project has planned to turn on CA/CVAC Cash controls in a phased manner after July 2020 instead of turning them on for all FISCal departments with the start of the integrated solution in July 2020. • The project continued to be behind in build, test, and validation tasks for M3 and M4. This has resulted in missed deployments of Direct Transfer Reversals in M3 and SOCA solution in M4. At least two other BP’s are deployed partially due to schedule issues. • MEC/YEC effort originally expected to be complete by October 15th of 2019 continues to be incomplete in June 2020. The project has deployed the integrated solution (M3 and M4) code into production and is starting to load ledger data into the new ledgers. At the end of July these new BP’s will be used in production and their stability will be evaluated then. The project planned to deploy Cash Management Controls and PFA interfaces only to a few small departments as opposed to all FISCal departments by June 2020. Milestone 4: Project delivered the following scope in June 2020 • Cash Basis Ledgers and sub ledgers have been deployed. • CV/CVAC functionality has been deployed. Cash Validation control, a critical part of the state’s cash management goals will not be met fully in June 2020. • The project plans to turn on Cash Validation controls on for seven small CFS departments by the end of July 2020 and progressively turn on the controls for larger departments after any issues are resolved. • Statement of Cash Accountability (SOCA) solution (M4 BP43) were not ready and not deployed in June 2020. • The FISCal Project deployed tools (Automated Ledger reconciliation) to assist the daily reconciliation of FISCal ledgers with the SCO ledgers. o By the end of July after enough ledger data has been loaded into FISCal the daily reconciliation will start between the FISCal and SCO ledgers. o The next 90 days of use of these tools will validate if the tools are working to help in the reconciliation or additional scope has to be included for effective reconciliation tools. • Critical production transaction cleanup required for M4 to go live was completed and deployed into production according to the SCO PBE. Milestone 5: Project has missed schedule (known from late February) The project could not deliver the M5 P1 and P2 BP’s schedule by June 2020. The project expects a one-year delay in the deployment. Schedule was impacted by the following • Delays with completing build and validation of BP63, and BP67 • Delays with Master test data generation leading to M5 UAT delays
Cost Management

yellow icon Yellow
Cost Management status remains Yellow in June due to a risk from a one-year delay with the M5 deployment. The project requires another year for deploying M5 P1 and P2 functionality and remaining work for Milestone 4 (SOCA, PFA etc.) At this time, it is undetermined if the project costs will increase. However, there is a high likelihood the project may incur additional costs to complete all planned SPR8 scope. • Project is in the process of developing an Iterative Project Report (IPR) to re-baseline the project schedule and scope which may impact overall project cost. • Due to the delays in completing the approved M5 scope and remaining M5 (P3) scope that was scheduled for completion post June 2020 there is a potential for increased vendor cost. • As of end of June 2020, the Project has spent approximately 96% ($550 Million of $571 Million) of the one-time IT project costs and 85% ($904 Million of $1.063 Billion) of the total Project cost.
Scope Management

red icon Red
Scope management focus area continues to remain Red through June 2020. Approved SPR8 Scope has been changed for M3, M4 and M5. The formal re-baselining of the project scope will be required to change the status of this focus area. The process of formal re-baselining will occur after the June 2020 release using an IPR process. In June 2020, the following scope was removed from MVP through the Project’s Governance process. • BP127- Renumbering Bond Sub Funds (In SPR 8 this feature is part of BP42: Ledger Conversion). This feature is not required for Milestone 4; however, it must be implemented before the SCO legacy system is retired (End State). • In May 2020, the following scope changes were made o Project divided the Plan of Financial Adjustments (PFA) interface BP into three parts. Project has gone live with part 1 required for Milestone 4 in June 2020. The remaining two parts of the scope will be delivered by August 2020. o Project had identified an additional report required to reconcile balances to the FISCal Cash Basis ledger with new CVAC Budget Detail Exception Report. Due to a lack of time, the project decided to develop PeopleSoft query for June 2020 release and develop the actual reports post go-live. o The project is also deploying two major control processes (Cash Validation and control and PFA interfaces) in a phased manner instead of deploying them for all departments by June 2020 as approved in SPR8. • Another metric showing the increase in scope of effort for M4 business processes in May was observed from the increasing story points in the product backlog. • The project had removed four business processes from project scope prior to January 2020 (BP51, BP74, BP74A and BP85A (April 2020)). • The SOCA solution also experienced increasing scope in terms of additional reports being identified. This resulted in the SOCA solution being removed from the June 2020 release. • The Ledger Recon tool carries risks of scope increases as users start using functionality in production in the end of July.
Resources

yellow icon Yellow
Resources focus area remains Yellow. The project’s vacancy rate impacts on its ability to deliver all planned SPR8 scope on schedule and with quality. • The project vacancy rates are over 15% for the past one year. • The project continued to face resource issues in June 2020 with SCO SME resources availability for multiple tasks. The SME’s must provide design support, perform UAT and help with MEC/YEC reconciliation • SCO resources continue to have resource contention issues due to the CoVID 19 situation. • With the M5 P1 and P2 requiring extensive UAT, the resource contentions risk must be addressed as soon as possible to avoid risks to project scope and schedule.
Quality

yellow icon Yellow
Quality focus area remains Yellow due to continued Quality Management process challenges, System quality issues and Implementation quality issues. Quality Management: The project continues to make several enhancements and overhauls of its tools and processes related to quality management. However as new development and production bug fixes continue the Quality Management process challenges, System/Data quality issues and Implementation quality issues remain to be resolved. Quality Management process issues: • After one year of working on automated testing, the project has not yet fully automated all critical scripts. Additionally, Critical scripts are only a subset of all testing scripts. Due to this, we still observe changes to production breaking already working functionality. • FISCal’s automated regression testing suite currently only covers critical scripts. There is no time for manual testing of all the remaining scripts. For Agile sprints, the lack of fully automated regression testing scripts for critical functionality causes issues for sprints testing goals and test coverage. • The project does not perform code validation at a level of detail to effectively identify defects before moving changes and new deployments to production due to schedule and resource constraints. Only a selected few critical function are validated for coding. • Performance test teams do not have dedicated performance test environments. This is leading to missed testing opportunities and defects identified only in production and performance issues being observed in production. System/Data quality Issues: • FISCal continues to fix transactional data through SQL changes to the database without fixing the underlying functional problems that cause the data issues. • FISCal system’s transactional performance is degraded when departmental users query the system for substantial amounts of information like large downloads of all departmental transactions. With the new integrated solution and additional interfaces bringing more data into the system and at least one BP requiring additional queries, this risk continues for the FISCal system performance. • The FISCal system does not have functionality (In some cases) to edit and change configuration data or transactional data through the application. This issue results in the project making direct changes to the backend database which in turn results in poor data quality. The project should prioritize the development of such functionality to reduce the risks from direct SQL changes to the database. The project has yet to make considerable progress in this area. • General Ledger (GL) and Commitment control (KK) ledgers mismatches resulted in reconciliation delays. The project deployed fixes to resolve this issue with the June release and CDT will report on the effectiveness of these fixes in the next few reports. • Lack of reasonable guardrails (business rules to validate data entry). FISCal is working on adding more guardrails for accounting functionality to be entered without avoidable errors. • FISCal does not have a formal set of Operational Level objectives (OLO’s) for its users. This results in users not knowing when to log incidents from poor transaction response times. Implementation quality: Project continues to face implementation quality issues due to the following issues which continue to exist through June 2020 • User stories are not sized small enough to complete an entire user story in one Sprint. Multiple user stories span over one sprint. This dilutes the ability of the project to measure its sprint team’s velocity and that leads to missed estimates and delays. • Lack of resources and time to perform code validation- Lack of robust code validation by FISCal teams (CDT is referring code validation of customized code and not PeopleSoft code base) • Performing UAT at a very high level using only happy path scenarios. Most of the defects in production are in areas of functionality that was not tested in UAT. • A very aggressive schedule for the milestone releases with large scale user impacts. This leads to effective testing being missed. • Lack of full functional testing (current testing is done for high level functions) • The inherent complications with test data harvesting for an accounting system where test data changes and must be created new for each round of testing. • Lack of appropriately scaled environments for performance testing • Continued errors with departmental accounting configurations (COA Values and Accounting Trees)
Risk and Issues

yellow icon Yellow
Risks and Issues management status remains yellow based on an overall challenge with managing risks effectively. Project did not manage key schedule and scope risks in a timely manner for the overall project scope and schedule and have now impacted the projects scope, schedule and cost requiring another extension to the project. At FISCal, risks must be approved by FISCal partners (PBEs) to ensure they are supportive of the approach to mitigating/resolving the risk/issue before it can be proposed, and this has been a problem getting risks documented and approved quickly and mitigated before they become issues. With an aggressive deadlines and multiple paths of effort occurring simultaneously, it is important that risks get documented and mitigated as soon as they are observed to meet schedule and scope. As the project has deployed a critical integrated solution and is getting ready to deploy M5 P1 and P2 functionality, it is critical the project focusses on addressing issues with the risk and issue management process. A few issues that must be addressed by the project • The project is not logging risks timely. The Risk Management process continues to take more than 6 months to approve risks and mitigations. (In the last quarter, we have seen improvement in the timely identification of risks related to the June 2020 release) • The FISCal solution is integral to a departments ability to complete its MEC and YEC efficiently. Based on the current challenges with departments ability to submit their annual fund status before the October 15th budget letter recommendation, CDT believes that the projects risk register should carry an open risk to MEC/YEC delays within FISCal to ensure formal mitigation can continue and reduce the risks to Financial year 2020/2021 MEC/YEC submissions. • The project does not consider the resource contention from SCO SME’s time being split between the support for MEC/YEC, Covid-19 impacts, daily responsibilities along with sprint and UAT tasks as a project UAT schedule and product quality risk.
Transition Readiness

blue icon Blue
Transition Readiness focus area status remains Blue. • The Project has initiated a command center for June 2020 release.
Conditions for Approval
Not Applicable
Corrective Action

yellow icon Yellow
CDT is working with FISCal on a corrective action plan. FISCal will use an Iterative Project Report (IPR) process to rebase line schedule and scope.
;