U.S. Flag

Making a difference in how people find health information, today and into the future.

Operational Progress Report

Site Tags: 
Sunday, December 1, 2013
HealthCare.gov Progress and Performance Report. Some content in this report may not be accessible to persons using assistive technology. For assistance, contact 202-690-6343

Read the full
Progress and Performance Report (PDF, 256 KB)

Today, Jeff Zients offered an operational progress report on our work to improve HealthCare.gov over the past five weeks. 

In Jeff’s own words: “The bottom line- HealthCare.gov on December 1st is night and day from where it was on October 1st.”

In addition, we also released a Progress and Performance Report which provides data on how the system is performing and can be viewed here.

We’ve provided weekly reports up until now and today want to highlight our work as we begin December –detailing the many measurable improvements we’ve made to the site as well as acknowledge that more work remains to be done. 

As we’ve said, with any web project there is not a magic moment but a process of continual improvement over time and we will continue to work to make enhancements in the days, weeks and months ahead.

As we begin December with a vastly improved web experience, we are mindful of the work to do to make sure those consumers who experienced frustration over the past several weeks are able to resolve their issues and complete their enrollment and confirm that those who have enrolled know their next steps to make sure they get coverage.  While our door is open for new consumers and we invite them in, we will place particular attention on those who still need questions answered in order to complete their enrollment.  We are 2 months in to a sustained 6 month long outreach and education campaign that will continue through the end of March so that consumers have time to access and enroll in affordable health care options that best  meet their needs and budget. 

Before diving into the data, its important to provide some context for the Progress and Performance Report.

In mid-October, the President, Secretary Kathleen Sebelius, and Administrator Marilyn Tavenner asked Jeff Zients to provide short-term management assistance on HealthCare.gov.

We started by bringing in technology experts from across government agencies and from the private sector to conduct an assessment of HealthCare.gov

The assessment highlighted a number of significant problems, most notably an unacceptable user experience marked by very slow response times, inexplicable user error messages and frequent website crashes and system outages.

At the same time, the team identified the root cause problems that needed to be addressed to fix the site. 

These root causes included: hundreds of software bugs, inadequate hardware and infrastructure, and a general lack of system monitoring and incident response capabilities.

The assessment also identified weaknesses in how the project was being managed, with slow decision making and diffuse or unclear accountability.

With these root causes identified, the conclusion was that HealthCare.gov was fixable, if significant changes were made to the management approach and if we executed against the lengthy punch list of software and hardware fixes with relentless focus and discipline.

In short, we needed to get the team working with the speed and urgency of a high performing private sector tech company.

The first key change made was appointing QSSI as the General Contractor and Systems Integrator.

QSSI has provided project management expertise, and coordinates the work with CMS and other contractors. They have also provided fresh eyes, talent, and dedicated teams of experts focused on system monitoring, software fixes, and hardware upgrades.

Working with QSSI, we instituted a new management structure to have clear accountability and rapid decision making. This management structure is centered on a Command Center that includes senior leaders from CMS and each contractor and vendor involved in HealthCare.gov.

The Command Center leadership monitors the site’s performance in real time, evaluating key metrics and dashboards. There are examples of those 24 hour monitoring dashboards on page 4 of the report.

The Command Center team focuses on site monitoring and incidence response around the clock. Twice a day, the Command Center hosts standup war room meetings for real-time, data-driven decision making and prioritization of key hardware and software fixes. There is an open line -- or bridge -- connecting the Command Center with all the key programmers and managers working on the system, so that 24 hours a day we have rapid, effective response to any issues or problems the instant they appear.

This clear accountability, prioritization and quick decision making is central to the progress we’ve made in improving the site’s performance.

In addition to implementing this new management structure and getting the team working with the velocity and discipline of a high-performing private sector company, we developed a prioritized punchlist of software fixes, hardware upgrades and user enhancements. Prioritization is based on what has the biggest impact on system stability, capacity and speed and user experience.

Over the last five weeks, we’ve made substantial progress working through the punchlist.

We’ve executed hundreds of software fixes and hardware upgrades, and the site is now stable and operating at its intended capacity, with greatly improved performance.

The report outlines some data on the fixes and upgrades and how the progress can be seen in the key operating metrics.

The top graphic on page 5 shows how the team has knocked more than 400 bugs and software improvements off the punchlist over the past two months. The pace has greatly accelerated once we got the new management structure and discipline in place.

After clearing through fewer than 100 bugs across the entire month of October, the speed has more than tripled, with over 400 bugs fixed. This has eliminated critical glitches and made improvements to the consumer experience throughout the site. This includes more than 50 bug fixes that were installed just last night, many of which made improvements in the back end of the system.

So a total of more than 400 software fixes have combined to improve the user’s experience as they look for information, fill out applications, shop, and enroll.

At the same time, we’ve been working through the software items on the punchlist, the team has made substantial improvements to the underlying hardware infrastructure.

On that front, the team has executed a series of upgrades to key components of the system that have increased redundancy, reliability and scale (bottom of page 5). 

There are four components that needed a lot of work. First, there’s the front end of the system, the Registration Database. This is where we were experiencing a large bottleneck when the site first launched.

As consumers attempted to create accounts and log on to the site, they ran into error messages and website crashes.

In the last few weeks, the team has re-architected the design of the registration system and installed new, dedicated hardware; including a major new upgrade this past Friday night.

All together this has more than quadrupled the throughput of the registration database, so that many more users can successfully create new accounts and log on.

In effect, we’ve widened the system’s on ramp  – it now has four lanes instead of one or two.

Beyond Registration, the entire site rests on a Core Database that enables consumers to shop, compare plans, and enroll.

Here, we’ve made two significant changes. We’ve deployed 12 large, dedicated servers. And we’ve significantly upgraded a storage – or memory – unit to improve response time. As a result, we’ve increased the system’s database throughput by more than 3 times.

Third, we’ve brought additional application environments online, more than doubling the website’s capacity.

And last, we’ve upgraded the firewall that protects the system. The team identified that the firewall was a constraining factor on the system’s capacity and throughput, so we upgraded and reconfigured it to allow more than 5 times the network throughput.

The cumulative effect of these hardware changes, along with others, is that the underlying infrastructure of HealthCare.gov is much stronger today than it was a few weeks ago.

The system is now able to handle its intended volume of consumer visits, and it has redundancy built in to avoid the type of instability that we saw in October.

The result of these efforts can be seen in the improvement in the sites’ operating metrics, starting with response times (page 6).

Response times is the measure of how quickly a page responds to a user request. In late October, the average response time on healthcare.gov was running around 8 seconds which was clearly unacceptable and very frustrating for consumers. Driven by the software and hardware fixes, we now have much faster response times. Over the last three weeks, the average response time has been well under 1 second. This means that consumers are having a much faster, smoother experience on the site.

Page 6 also shows system error rate, another key operating metric.

This is the measure of how often, on a per page basis, the system times out or presents an error message. The team has made progress. In late October, the error rate was approximately 6%.

We got that down to about 2% by November 9th, to 1% by November 16th, and this past Friday, the average error rate was approximately .75 or three quarters of one percent.

In addition to improving system speed and reducing the error rate, we’ve also made measurable progress increasing the system’s stability (page 7).

System stability, which is typically referred to as system uptime -- is measured by the percentage of time the site is available on a given day, excluding planned downtime for scheduled maintenance.

HealthCare.gov is now seeing uptime consistently above 90%.  For the week ending November 2nd was only 42.9%. In fact, that’s what we think the system averaged through most of October as well.

The uptime improved to 71.9% by November 9th, and has been consistently above 90% since then, including 95% uptime this past week. Again, this improvement in stability is driven by the hardware and software fixes, and we expect to see further improvements given the redundancy and capacity we’ve added to the system.

And just as importantly, when we do experience system glitches or slowdowns, we can resolve issues much more quickly, due to the continuous monitoring and rapid response teams. Back in October, a typical system outage lasted several hours or more. Now, the team can generally diagnose root cause problems and make the necessary fixes within 60 minutes.

So we have a much more stable system that’s reliably open for business.

That’s important, because at the end of the day, we need high system up time so consumers are able to use the system to seek information, fill out applications, shop and enroll. It’s critical that the result of all the improvements we’ve made is that we’ve doubled the system’s capacity, and HealthCare.gov can now support its intended volumes.

The chart on the bottom of page 7 outlines the simple math.

The site now has the capacity to handle 50,000 concurrent or simultaneous users at one time. And we know that each visitor spends, on average, 20 to 30 minutes on the site per visit. So the site will support more than 800,000 consumer visits a day.

Now to be clear, there likely will be times that even with this increased capacity, it will be insufficient to handle peaks in simultaneous demand. So to prepare for those times when spikes in user volume outstrip the systems’ expanded capacity, we will deploy a new queuing system to serve consumers in an orderly fashion. It will allow consumers to request email notifications when it’s a better time to come back to the site.

So, lifting up, we’ve made significant progress in improving HealthCare.gov, and achieving a system that runs smoothly for the vast majority of consumers.

This progress is summarized on page 8 of the report.

Response times are under 1 second.

Error rates are down well under 1%.

And the system is stable, with uptimes exceeding 90%.

We now have a rapid response team and continuing monitoring in place to ensure optimal system performance and to respond quickly to glitches or other issues that crop up.

All of which means the site has the ability to serve 50,000 concurrent users and support 800,000 consumer visits a day as consumers seek information, fill out applications, shop and enroll.

As with any website, the team will continue to address additional bugs and glitches and will continuously evaluate emerging infrastructure needs.

The general contractor and rapid response team has served us well; enabling us to execute with private sector speed and focus currently and for the long term. 

While we still have work to do, we’ve made significant progress with HealthCare.gov working smoothly for the vast majority of consumers.

 

This post is no longer accepting comments. If you want to learn more about the Affordable Care Act, please visithhs.gov/healthcare. If you have question about or need assistance with health insurance enrollment, please visithttps://www.healthcare.gov/contact-us/.

 

Join the Conversation

Comments

Submitted by Ken on
Is there an email address to which bugs can be submitted? Is there a list of outstanding bugs so that a user knows whether they are doing somehting wrong or the system is doing it?
Submitted by Ken on
I have a friend who I viewed going through the application process. The application was submitted, but a "verification system down" message appeared. We tried it again with a new application. Same thing. She phoned the call center and had them fill out an application over the phone. The call center person got the same error. Her response "try again later, like the message says". We tried again and the verification system down message still comes up. What can she do? Why is this happening?
Submitted by Helen on
I have successfully enrolled an am waiting for Invoice from Insurer. I had the website up and talking to a rep. on phone from the Market Place when I had trouble getting finished with application. Just by having her walk me through it I was able to finish. I was in process of comparing plans that I had narrowed down to 7 from a total of 42 plans. I was getting frustrated because I thought I was in a loop not getting me where I wanted to go. I clicked on the word "set" in the green square to the right of the square in the enrollment step, I think the step was labeled "select a plan" with my name Helen was under it. That was it. I was then able to select the plans I wanted to compare. I printed out the side by side comparisons. The standardized summary of benefits page for all the selected plans was great. I could compare feature to feature of each company and analyze which fit my health situation. The data in this format made it possible to analyze. I made a decision the next day after studying the print outs and my application to the insurer is in process. I was determined to enroll using this website. I want this to be a great success! The comparison tool is great.
Submitted by Susan on
I have been trying to get my identity verified since October 6th. I have downloaded my driver's license 4 times. I had no problem with my identity being verified when I applied for a new mortgage, started a new job and opened a new bank account. Where is my information going? Why is this taking so long? My husband and I want to be enrolled by January 1st, but I am doubtful that we will be. Is there any recourse to this issue or do we just lose out.
Submitted by Jeri on
I've gotten as far as the final page where I should be able to pick out a plan. But all I'm getting is 500.301533 error and suggesting I log out and log back in or to try again in 30 minutes. But I always get the same result. Should I call or what?
Submitted by Todd on
I have never yet been able to get past the Verification page. I have to repeatedly go through the application and never get to verification or enrollment. For me, at least, there has been no improvement. If I can't enroll, I can't buy or be covered. And I try every day. When I call the phone center, they can't offer any assistance that meaningful. I even sent in a paper application…nothing on that has happened. WTF?
Submitted by Glynis on
Use the new 'remove' button to delete your application and start over. After you remove your application, delete your cookies, temporary internet files, and history, log out, then restart your browser. Also make sure your browser and java plugin are the current versions but don't use Firefox because it does not work well. When you complete your application be careful about the format of your answers. Follow the examples that they give exactly as they look. Also, don't add any decimals to any place they don't, like to hours worked, round up or down. And your name should be exactly as it appears on your Social Security card.
Submitted by Bob on
The log in page for the last few days just refreshes with a blank page when I click log in after entering my user name and password. It is frustrating because I was at the final stage of selecting a plan and now I cannot even log in. I know it recognizes me since if I enter a wrong digit in my password or user name it quickly replies that it is wrong. HHS site when things hang up just remove the application and start over. But if you can't log in how can you do this?
Submitted by Glynis on
Did you try to reset your password? Use the 'forgot your password' link to do that. If it gives you an error message when trying to do that then you'll need to create an entirely new account with a new username, new password, and a different email address.
Submitted by Bob on
I'm answering my own question for the benefit of others. I could not log in with any existing email addresses that I used when the system was having problems earlier in October. Rather I had to start all over again with an email address the system hadn't yet associated with me. I did not use my existing application fearing it might use the old system that wasn't working so well. By starting from scratch everything worked perfectly and I went from entering everything all over again all the way through singing up for a specific plan. But when I tried to pay it linked me to the insurance company pay website which then refused payment. Next day I called the insurance company offering to mail the check to them but they refused saying they needed to get the proper confirmation from healthcare.gov. Only then will they send out a letter to me with instructions on where to send the check. If I don't receive the letter in a week or so then call them to see what to do next.
Submitted by Dan on
I just went back to earlier blog comments, and realized that most of the questions listed in my previous post have already been asked. What I DIDN'T find were any answers. Can anyone point them out to me (if they exist)?
Submitted by Dan on
This has nothing to do with the current operational status of the enrollment system; it's about accountability: 1. How was the original system design contractor chosen? Is it true that it was a no-bid situation? 2. How much was the original contractor paid? 3. How much will that contractor be required to refund, given its obvious incompetence? 4. Who designed the original HRS program oversight system? Has that person been asked to resign? 5. Given her overall responsibility for the enrollment system, when can we expect Secretary Sibelius to be replaced? If I sound bitter, its because I am. I just can't understand how the President and Sec. Sibelius could have let this happen. The roll-out fiasco has more than erased the apparent advantage of Democratic House and Senate candidates given them by the Republican shut-down; CMMS' corrective efforts will help regain some of the lost ground, but it is unclear whether all the damage can be undone. I plan to do what I can to help my local Democrats take back the seats they lost last year, but I can only hope...
Submitted by anonymous on
Sorry...But I'm getting really tired of hearing the complaining from people who won't listen to the sound advice given here by Glynis and others...to START OVER!!!!! People, you need to learn how to help yourselves!! Listen to what you need to do... and "DO IT" to make your new application go through..There is now a "delete" option on the old applications that are "stuck". Please USE it and start over....It really works!! But if you refuse to do the work of starting over, you will get nowhere!!....It is up to you.. It has been frustrating and I too have experienced all of the glitches. But I chose to follow the suggestions of other helpful bloggers and I am now signed up for Jan. 1st. Please stop complaining and be pro-active!! This IS going to work, more and more people and realizing that they can start over and are getting through this time...Let's start being more positive here!! Unfortunately, It makes me realize how "short- sighted" people are, and that makes me kind of sad. :( I will bet than in a few months time, the popularity of the ACA is going to skyrocket....once people really understand how it works. Most people who are against it, do not fully understand it. (And that is a FACT) Research something fully before you form an opinion on it...For some, I guess "Ignorance is bliss"...good luck....Sincerely
Submitted by Karen on
I did what Glynis suggested and I got signed up this afternoon. Finally after 2 months -4 applications-and probably 50 calls to the 800 number. I am so relieved that I can quit messing with this!
Submitted by Kevin on
I've made it as far as the page where you are supposed to set the amount of my tax credit that I'm going to spend on my health care plan, but then the wheel just spins and spins with the message, "Please Wait." At EVERY hour (and I've tried non-peak hours- 12:30am and 5:30am Central) it has been like this. PLEASE PLEASE PLEASE fix this so I can get insurance by December 23rd. The site still doesn't seem to be able to handle the traffic. Thanks and keep working hard.
Submitted by Glynis on
Delete your cookies, temporary internet files, and history before you try again. Also make sure that your browser and java plugin are the current up to date versions.
Submitted by Ronald on
Glyniis: I am at the point where I am to enroll. When attempting to view available options, receive a blank page. Called for assistance Was advised that my application is valid. Do NOT remove and start over. Result was that the system still has glitches and to give it until Friday 12/6 to see if it's been fixed. Tried Thursday 12/5. Attempts to enter account brought me back to sign on screen, several times. This thing is NOT fixed. I want to enroll, but system unable to complete task. Denial is NOT a valid response.
Submitted by Glynis on
This is what HHS is advising: "If you’ve tried Healthcare.gov – especially in October or early November – and your application seems stuck or you can’t move forward, try starting the process again. First, log in to your account, select the application in progress and hit “remove.” Then close and re-open your web browser. Log back into your Healthcare.gov account and start a new application. " If you can't login then reset your password.
Submitted by Frank on
Can't login. Enter user name and password click login in just takes me back to login screen. Doesn't say invalid password or user name??
Submitted by Glynis on
forgot to mention that your new password has to be different than the last 6 passwords.
Submitted by Glynis on
Click the "forgot your password" link to reset your password.
Submitted by Jeff on
Didn't work for me. I got a message stating that I would receive an email with a new password, but it didn't come. I called the number, and they manually reset my password, and then I just got the same behavior with the new password. I only get returned to the login screen with no error message. I've tried it on multiple computers, multiple browsers, even on a computer with a brand new OS, and always get the same behavior. I've called, emailed and "Live Chatted" multiple times and everybody just keeps telling me to delete cookies, make sure latest java installed, clear history, etc. I've done all this and it doesn't help. It won't let me set up a new account because it says my email address is already attached to an account. I want to participate, but this is ridiculous.
Submitted by Glynis on
you need to use a completely different email address that wasn't attached to a previous account in addition to a new username and new password to create a new account.. Also, your password should be different than the last 6 passwords.
Submitted by Glynis on
When you create a new account you need a different email address with a new username and new password. It has to be a completely different email account. For example, if you used a gmail account then use yahoo. And you have to use a password that is different than your last 6 passwords.
Submitted by Mark on
I had success tonight like never before! Although the ACR never called me back (yet) in response to the last of my many calls to the call center, they or someone did manage to delete two of my three accounts which had flawed applications from October that apparently could not be fixed (one had me listed as a household member three times and the other had an incorrect social security number which was entered correctly, somehow incorrectly modified by the call center, and then could not be changed back to the correct number), and they deleted my partially completed application in my third account which kept getting stuck at the income questions, thus giving me a chance to start over with the third account using the newly improved although not yet great web site. If you had a flawed application, have been trying to log on and know that you are using your correct user name and password, and the screen repeatedly takes you back to a blank login page with no error warning, then from my experience it seems that perhaps the account was deleted (possibly because any application therein was flawed beyond repair). I had that experience when trying to sign into my first two accounts, but I was able to log into my third account no problem. I successfully submitted a new application tonight through that third account. It did get bogged down at one point resulting in an error message and requiring me to log out, delete cookies and history, and log back in. After logging back in, I had almost no problem completing and submitting the application. However, when I hit the submit button, the site did not in any way indicate whether the submit has been accepted, so, rather than hit the button again, I logged out, deleted cookies and history and logged back in again. When I logged back in, the site showed there was a message for me and when I clicked on the link to my application, it took me to the view eligibility results button. Shockingly, for the first time ever I was able to see the eligibility results, set my subsidy level and compare plans. I did not yet enroll as I need to decide on which plan I want, but the web site appears to have worked up to that point and I did see the enroll button and the option to get a dental plan as well. One possibly VERY important thing I did differently before making my new attempt tonight which NO ONE from the call center or the ACR ever suggested to me, was that I actually checked to see if I had the latest Java plugin installed in my browser (I used Chrome, btw). It turns out,there was no Java plugin installed, so I did that first and then confirmed that it was installed on the verification page of Oracle's Java download site. I'm not sure if this was always the reason why I was not previously able to view the eligibility results of previously submitted applications, if the repairs to the site solved the problem or if it was a combination of both. However, I would suggest to anyone having problems viewing eligibility results to make sure they have the latest version of Java plugin installed and make sure popup blockers are turned off before making any new attempt to view eligibility results. Also, as a previous poster from Illinois noted, when trying to click the links to the insurance company provider lists, the URL the marketplace links to, at least for the Illinois plans, is flawed because it does not have a ":" after the "https" and the "//." If you simply add the colon to the URL it will take you to where you want to go. This is annoying, but relatively minor. The best part of my night was finding out that my cost is going down by over $270 to $400 per month depending on which plan I end up choosing and my deductible is going down by $1000 either way. Also, to the tech teams working on the site, kudos for installing the "Remove" button. I had been asking to be able to simply delete my applications and start over since the middle of October. It made no sense that this was not permitted earlier, but I'm glad it is available now. I don't know how difficult it was to implement, but I believe it to be a major improvement to the site. I will sleep much better tonight as I am no longer AS CONCERNED about whether I will be able to have insurance coverage on January 1. As a 50 year old, I need the insurance coverage for peace of mind even though I currently have no known health conditions. I was skeptical, but am now very hopeful that this whole web site fiasco may just work out all right.

Pages