Table of Contents  
Year : 2018  |  Volume : 5  |  Issue : 2  |  Page : 95-99

The predictive accuracy of the american college of surgeons national surgical quality improvement program surgical risk calculator in patients undergoing major vascular surgery

1 Department of Vascular Surgery, Royal Brisbane and Women's Hospital, Herston QLD 4006, Australia
2 Department of Vascular Surgery, Royal Brisbane and Women's Hospital; School of Medicine, The University of Queensland, Herston QLD 4006, Australia

Date of Web Publication3-May-2018

Correspondence Address:
Dr. Alison McGill
Department of Vascular Surgery, Royal Brisbane and Women's Hospital, Herston QLD 4006
Login to access the Email id

Source of Support: None, Conflict of Interest: None

DOI: 10.4103/ijves.ijves_18_18

Rights and Permissions

Aim: The American College of Surgeons (ACS) National Surgical Quality Improvement Program (NSQIP) Surgical Risk Calculator (SRC) was developed to estimate the rates of complications for patients undergoing a variety of surgical procedures, based on the patient's preoperative demographics and medical conditions. Its predictive ability has been evaluated in a number of studies for a variety of surgical fields. There has so far been no assessment of the SRC in patients undergoing vascular surgery. This study assesses whether the ACS NSQIP SRC can accurately predict risk of complications in patients undergoing major vascular surgery at a tertiary hospital. Methods: A retrospective review of prospectively collected data was performed on all patients who underwent an open abdominal aortic aneurysm (AAA) repair, an endovascular aneurysm repair (EVAR), or a femoral-popliteal bypass graft (FPBG) from July 2016 to April 2017. A total of 95 patients had their demographics entered into the ACS NSQIP SRC, and the predicted rates of complications were compared to the observed rates of complications. Results: Statistical analysis was performed with Brier scores and C-statistics. This analysis found the ACS NSQIP SRC accurately estimated the risk of complications with a Brier score of 0.044 for EVAR, 0.068 for open AAA repair, and 0.0752 for FPBG. The C-statistics for serious complications, any complications, and discharge to a nursing home or rehabilitation indicated the model was good at accurately predicting the risk of these outcomes. Conclusion: The ACS NSQIP SRC accurately predicts the rates of complications in patients undergoing vascular surgery.

Keywords: Calculator, complications, National Surgical Quality Improvement Program, surgery, vascular

How to cite this article:
McGill A, Pinto N, Jenkins J, Favot D, Ogg M, Boyne N, Quinn S, Kruger A, Rowbotham SE. The predictive accuracy of the american college of surgeons national surgical quality improvement program surgical risk calculator in patients undergoing major vascular surgery. Indian J Vasc Endovasc Surg 2018;5:95-9

How to cite this URL:
McGill A, Pinto N, Jenkins J, Favot D, Ogg M, Boyne N, Quinn S, Kruger A, Rowbotham SE. The predictive accuracy of the american college of surgeons national surgical quality improvement program surgical risk calculator in patients undergoing major vascular surgery. Indian J Vasc Endovasc Surg [serial online] 2018 [cited 2021 Dec 8];5:95-9. Available from:

  Introduction Top

Accurate surgical outcome predictors have been sought to attempt to improve patient outcomes and inform decision-making. The American College of Surgeons (ACS) National Surgical Quality Improvement Program (NSQIP) Surgical Risk Calculator (SRC) was developed in 2014 based on the NSQIP database of more than 3.2 million surgical cases and their complications.[1] The development of NSQIP was initiated in 2006 and collected data from over 600 hospitals and has shown improved outcomes in patient care.[2] The purpose of the SRC is to provide the estimates of rates of complications in the first 30 days postoperatively, as well as the length of stay for patients undergoing a variety of surgical procedures. It is open access, online tool that is patient-specific, as it is based on the patients' preoperative demographics and medical conditions. The calculator is based on a variety of procedures across all surgical specialties, so can be applied to almost any surgical intervention. Risk is estimated for length of hospital stay as well as 14 categories of complications: serious complication, any complication, pneumonia, cardiac arrest or myocardial infarction, surgical site infection (SSI), urinary tract infection (UTI), venous thromboembolism, renal failure, colonic ileus (conditionally displayed based on the selected procedure), colon anastomosis leak (conditionally displayed based on the selected procedure), readmission, return to theater, death, or discharge to rehabilitation, or nursing home. The ACS NSQIP webpage defines “serious complication” as death, cardiac arrest, myocardial infarction, pneumonia, progressive renal insufficiency, acute renal failure, pulmonary embolism (PE), deep vein thrombosis (DVT), return to the operating room, deep incisional SSI, organ space SSI, systemic sepsis, unplanned intubation, UTI, and wound disruption. The webpage defines “any complication” as superficial incisional SSI, deep incisional SSI, organ space SSI, wound disruption, pneumonia, unplanned intubation, PE, ventilator for >48 h, progressive renal insufficiency, acute renal failure, UTI, stroke, cardiac arrest, myocardial infarction, DVT, and systemic sepsis.[3] The benefit of the SRC is that it can provide patient-specific assessment of risk, which assists both surgical decision-making and informed consent, as well as the ability for comparison of risk-adjusted outcomes.

A number of studies have evaluated the SRC and its predictive ability for a variety of surgical fields, including general surgery, orthopedics, ear, nose, and throat, plastic surgery, and gynecology using Brier's scores and C-statistic for analysis.[4],[5],[6],[7],[8],[9],[10] Three of these studies reported that the SRC underestimated the risk of complications and so could not validate the SRC in their surgical field.[7],[11],[12] Two of these studies showed that the SRC overestimated the risks of complications, compared to actual outcomes and other predictive models.[10],[13] To date, no assessment of the SRC in patients undergoing vascular surgery has been published. The purpose of the current study is to evaluate the NSQIP model to determine its applicability to three common elective major vascular procedures: elective open AAA repair, EVAR, and FPBG, by assessing whether it can accurately predict the risk of complications. These surgeries were chosen as they are the most commonly performed elective major vascular operations performed in our institution. The authors chose to evaluate the SRC as it is the most recently developed risk calculator, and, unlike other risk predictors e.g. (vascular-POSSUM), it has potentially wide application to any surgical procedure and provides specific risk estimates for the most common complications, as well as mortality and functional outcomes.

  Methods Top

Ethics exemption was obtained for this project from the hospital's Human Research Ethics Committee. All patients at our tertiary hospital who underwent an elective open AAA repair, an EVAR, or an FPBG from July 2016 to April 2017 were included in the study. A total of 95 patients were identified, comprising 35 patients undergoing EVAR, 27 patients undergoing open AAA repair, and 33 patients undergoing FPBG. The study size was chosen as it is consistent with similar study cohorts evaluating the SRC,[4],[9] and the statistical analysis used does not require a minimum number of data inputs. State-wide patient records were accessed to obtain information about demographics and comorbidities and also complications in the 30 days following surgery. Almost all preoperative and postoperative data were available for all patients. The only consistently lacking data was patient height, which was only found in approximately 60% of patient notes. When this was absent, average heights for men (1.75 m) and women (1.62 m) in Australia were used instead. Qualitative demographics such as functional status or dyspnea at rest or on exertion, if it was not explicitly recorded in the notes, could be inferred from the anesthetic records that assess exercise tolerance, as these often stated whether a patient was functionally limited by dyspnea or another symptom.

The patient demographics were entered into the ACS NSQIP SRC, and the predicted rates of complications and length of hospital stay were generated. These results were compared to the observed rates of complications and analyzed statistically. The ACS NSQIP webpage definitions of “serious complication” and “any complications” were used in this study to classify the complications experienced by this group of patients.

To verify the accuracy of the probability forecast, in this case, the risk of complications generated by the SRC for each patient and the Brier score was calculated for each operation. The Brier score is calculated from the data comparing the predicted and actual complication rates for each patient to give an overall analysis of the SRC's accuracy of prediction. The Brier score is between 0 and 1, with 0 indicating the most accurate forecast and 1 indicating the least accurate forecast. If the Brier score is close to 0, it suggests the SRC is accurate at predicting the risk of complications for an individual patient.

The C-statistic was also calculated for each complication using R (version 3.4.0), as the C-statistic gives a measure of the predictive accuracy of a logistics regression model, to assess the likelihood that the model will accurately predict an outcome better than chance.[14] A model is considered poor with a C-statistic value <0.5, good if >0.7, very good if >0.8, and a value of 1 indicates a perfect model. The C-statistic is equivalent to the area under the receiver operator curve (ROC). If the C-statistic shows the model is good, the SRC is likely to be accurate in predicting the risk of complications for a given patient.

Length of stay data were not analyzed as it was widely variable in our patient group depending on the patient home location, preoperative workup, transfer from local hospital, and incidence of complications, and most other studies did not evaluate these results.

  Results Top

A total of 95 patients and their surgery were analyzed, and their demographics and comorbidities recorded [Table 1]. The observed complication rates for each surgery along with the Brier scores for each surgery type were calculated [Table 2]. A total of 52 complications were recorded, with 11 patients having more than one complication. The Brier score for the EVAR group was calculated to be 0.045. The Brier score for open AAA repair was found to be 0.068. The Brier score for FPBG was calculated to be 0.075.
Table 1: Patient premorbid comorbidities

Click here to view
Table 2: Rates of observed complications

Click here to view

The Brier scores are low and indicate that the ACS NSQIP SRC accurately estimated the rates of complications in our patient group for these three common vascular surgeries.

The C-statistics were calculated for each type of complication across all three surgeries. Due to the low observed number of some complications, and the inclusive criteria for “serious complications” and “any complications,” ROCs were only produced for “serious complications,” “any complications,” and “discharge to nursing home or rehabilitation” [Figure 1]. The C-statistic for “serious complications” was 0.697, for “any complication” it was 0.719, and for “discharge to rehabilitation or nursing home” it was 0.913. This demonstrated that the SRC is good at predicting the risk of serious or any complications occurring, and very good at predicting the risk of discharge to rehabilitation or nursing home.
Figure 1: Receiver operator curves. (a) Serious complication: C-statistic 0.697. (b) Any complication: C-statistic 0.719. (c) Discharge to nursing home or rehabilitation: C-statistic 0.913

Click here to view

  Discussion Top

This study used the Brier's score and C-statistic to evaluate the data, as this is the same statistical evaluation used in the original validation study.[1] The Brier scores and C-statistics calculated in this article validate the use of the ACS NSQIP SRC in patients undergoing vascular surgery. They show that the SRC is a good model to accurately estimate the risk of complications.

The ability to have accurate predictors of complications of a procedure by adjusting for the risk profile of the patient benefits the surgical community in numerous ways. It aids educated decision-making that is specific to each patient. It allows differentiation between surgical options in cases where outcomes are unclear. It makes the informed consent process more accurate and provides opportunities for benchmarking, ongoing learning, and improvement of procedures. The SRC is useful because it draws data from a large cohort of many surgical procedures, is user-friendly, and free.

The development of calculators such as the SRC has the advantage of predicting complications rates that are surgery and patient specific. This is particularly relevant in vascular surgery, as the patients can have significant medical comorbidities and treatment may involve complex surgical decisions. A systematic review assessing the evidence for decision aids in surgery found that patients who used decision aids were more knowledgeable about treatment options and experienced less decisional conflict.[15] The availability of a predictive tool such as the SRC can assist in this process by providing education to patients, setting realistic expectations, and providing the opportunity to adjust perioperative management. The SRC has the advantages of being easily accessible, quick to use, and provides tools to distribute the generated results.[9]

Another possible use is in incentive-based funding, where funding is based on performance consistent with clinical predictors, which is more prevalent in countries other than Australia. If complication rates consistent with those predicted by this SRC are achieved, this suggests good local practice with appropriate risk reduction strategies.

A large number of patients treated at the RBWH travel from up to 700 km to attend hospital. Therefore, there was a risk that a review of the patient records at the RBWH might miss complications that were managed by the patients' local hospital, for example, medical complications such as pneumonia or UTI. However, the introduction of electronic medical records in Queensland allows users to view every hospital encounter for a patient and to read the emergency notes and discharge summary. This way, a search could be performed for any hospital presentation within 30 days of surgery due to post-operative complications.

There are limitations to the current study. First, the sample size was small, which, although consistent with other similar studies, meant a very low incidence of some complications. This could be increased by expanding the inclusion criteria for a larger cohort study. Second, the data collected wase from surgeries performed in only one hospital, although a major center for vascular surgery in Australia. Third, the data were collected retrospectively from data recorded by multiple practitioners and therefore relied on accurate recording of comorbidities and diagnosis of complications by other medical staff. While almost all pre- and post-operative data were available for all patients, the only example of incomplete records was the lack of recorded patient heights.

The patient cohort assessed had a consistent collection of premorbid conditions, expected in vascular surgery patients. They often had diabetes, hypertension, chronic obstructive pulmonary disease, and many were smokers. Dyspnea at rest or on exertion and acute renal failure were seen less frequently. Almost no patients had the comorbidities assessed in the SRC of disseminated cancer, sepsis, ascites, or were ventilator dependent. It seemed that the SRC had few measures of cardiac disease, which is a leading cause of morbidity and mortality in vascular patients. The only two conditions assessed were dyspnea on exertion and new congestive cardiac failure. However, the SRC does assess for risk factors for heart disease, such as hypertension, diabetes, age, and weight. The predicted complications also only include myocardial infarction or cardiac arrest for cardiac complications. Events, such as NSTEMI, pulmonary edema, or arrhythmias, were not included in this study. Other conditions relevant to vascular patients that were not considered were bleeding disorders and antiplatelet and anticoagulation medication use. As our group of patients was reasonably homogenous, the results of the current study are more likely to be applicable to other groups of patients undergoing vascular surgery in other centers.

This exploratory study is the first to evaluate the accuracy of the NSQIP SRC in vascular surgery and found that for the procedures of EVAR, open AAA repair, and FPBG, the NSQIP SRC was an accurate and good predictor of outcomes. This could be extrapolated to apply to other vascular surgeries. Further investigation would be needed to confirm this.

  Conclusion Top

The ACS NSQIP SRC appears to accurately predict the rates of complications in patients undergoing vascular surgery as shown by Brier scores of <0.08 and C-statistics of >0.7. It could be used to generate accurate patient-specific estimates of risk in patients considering vascular surgery and may be able to be applied to the estimate risk for patients undergoing other forms of vascular surgery.

Financial support and sponsorship


Conflicts of interest

There are no conflicts of interest.

  References Top

Bilimoria KY, Liu Y, Paruch JL, Zhou L, Kmiecik TE, Ko CY, et al. Development and evaluation of the universal ACS NSQIP surgical risk calculator: A decision aid and informed consent tool for patients and surgeons. J Am Coll Surg 2013;217:833-420.  Back to cited text no. 1
Cohen ME, Liu Y, Ko CY, Hall BL. Improved surgical outcomes for ACS NSQIP hospitals over time: Evaluation of hospital cohorts with up to 8 years of participation. Ann Surg 2016;263:267-73.  Back to cited text no. 2
American College of Surgeons National Quality Improvement Program. Surgical Risk Calculator. Available from: [Last updated on 2017 Jul 18; Last accessed on 2017 Aug].  Back to cited text no. 3
Cologne KG, Keller DS, Liwanag L, Devaraj B, Senagore AJ. Use of the American college of surgeons NSQIP surgical risk calculator for laparoscopic colectomy: How good is it and how can we improve it? J Am Coll Surg 2015;220:281-6.  Back to cited text no. 4
Edelstein AI, Kwasny MJ, Suleiman LI, Khakhkhar RH, Moore MA, Beal MD, et al. Can the American college of surgeons risk calculator predict 30-day complications after knee and hip arthroplasty? J Arthroplasty 2015;30:5-10.  Back to cited text no. 5
Evans RP. CORR insights(®): The ACS NSQIP risk calculator is a fair predictor of acute periprosthetic joint infection. Clin Orthop Relat Res 2016;474:2067-70.  Back to cited text no. 6
Mogal HD, Fino N, Clark C, Shen P. Comparison of observed to predicted outcomes using the ACS NSQIP risk calculator in patients undergoing pancreaticoduodenectomy. J Surg Oncol 2016;114:157-62.  Back to cited text no. 7
Paruch JL, Ko CY, Bilimoria KY. An opportunity to improve informed consent and shared decision making: The role of the ACS NSQIP surgical risk calculator in oncology. Ann Surg Oncol 2014;21:5-7.  Back to cited text no. 8
Prasad KG, Nelson BG, Deig CR, Schneider AL, Moore MG. ACS NSQIP risk calculator: An accurate predictor of complications in major head and neck surgery? Otolaryngol Head Neck Surg 2016;155:740-2.  Back to cited text no. 9
Teoh D, Halloway RN, Heim J, Vogel RI, Rivard C. Evaluation of the American College of Surgeons National Surgical Quality Improvement Program Surgical Risk Calculator in gynecologic oncology patients undergoing minimally invasive surgery. J Minim Invasive Gynecol 2017;24:48-54.  Back to cited text no. 10
Samson P, Robinson CG, Bradley J, Lee A, Broderick S, Kreisel D, et al. The national surgical quality improvement program risk calculator does not adequately stratify risk for patients with clinical stage I non-small cell lung cancer. J Thorac Cardiovasc Surg 2016;151:697-7050.  Back to cited text no. 11
Slump J, Ferguson PC, Wunder JS, Griffin A, Hoekstra HJ, Bagher S, et al. Can the ACS-NSQIP surgical risk calculator predict post-operative complications in patients undergoing flap reconstruction following soft tissue sarcoma resection? J Surg Oncol 2016;114:570-5.  Back to cited text no. 12
Simorov A, Bills N, Shostrom V, Boilesen E, Oleynikov D. Can surgical performance benchmarking be generalized across multiple outcomes databases: A comparison of university HealthSystem consortium and national surgical quality improvement program. Am J Surg 2014;208:942-8.  Back to cited text no. 13
R Core Team. R: A Language and Environment for Statistical Computing. Vienna, Austria: Foundation or Statistical Computing; 2017. Available from: [Last accessed on 2017 Jul].  Back to cited text no. 14
Knops AM, Legemate DA, Goossens A, Bossuyt PM, Ubbink DT. Decision aids for patients facing a surgical treatment decision: A systematic review and meta-analysis. Ann Surg 2013;257:860-6.  Back to cited text no. 15


  [Figure 1]

  [Table 1], [Table 2]


Similar in PUBMED
   Search Pubmed for
   Search in Google Scholar for
 Related articles
Access Statistics
Email Alert *
Add to My List *
* Registration required (free)

  In this article
   Article Figures
   Article Tables

 Article Access Statistics
    PDF Downloaded72    
    Comments [Add]    

Recommend this journal