Applicability and Validity of an e-Health Tool for the Appropriate Referral and Selection of Patients With Chronic Pain for Spinal Cord Stimulation: Results From a European Retrospective Study

Objectives: To support rational decision-making on spinal cord stimulation (SCS), a European expert panel developed an educational e-health tool using the RAND/University of California at Los Angeles Appropriateness Method. This retrospective study aimed to determine the applicability and validity of the tool using data from patients for whom SCS had been considered. Materials and Methods: A total of 12 European implant centers retrieved data from 25 to 50 consecutive patients for whom SCS was considered in 2018 – 2019. For each patient, data were captured on the clinical and psychosocial variables included in the e-health tool, center decisions on SCS, and patient outcomes. Patient outcomes included global perception of effect by the patient and observer, and pain reduction (numeric pain rating scale) at six-month follow-up. Results: In total, 483 patients were included, of whom 133 received a direct implant, 258 received an implant after a positive trial, 32 had a negative trial, and 60 did not receive SCS for reasons other than a negative trial. The most frequent indication was persistent spinal pain syndrome type 1 and type 2 (74%), followed by neuropathic pain syndromes (13%), complex regional pain syndrome (12%), and ischemic pain syndromes (0.8%). Data on the clinical and psychosocial variables were complete for 95% and 93% of patients, respectively, and missing data did not have a signi ﬁ cant impact on the study outcomes. In patients who had received


INTRODUCTION
6][7] Therefore, a European consensus study was conducted, 8 using the RAND/University of California at Los Angeles Appropriateness Method (RUAM) 9 to establish the criteria for patient selection and referral for SCS.In this study, a multidisciplinary expert panel assessed the appropriateness of 386 clinical scenarios across four main indications for SCS: chronic low back and leg pain (CBLP), complex regional pain syndrome (CRPS), neuropathic pain syndromes (NPSs), and ischemic pain syndromes (IPSs).Regarding CBLP, the International Association for the Study of Pain (IASP) published a revised classification of chronic pain, as part of the new International Classification of Diseases, 11th Revision. 10This includes the term chronic pain after spinal surgery (CPSS), which is suggested as a replacement for failed back surgery syndrome.However, CPSS excludes cases where preexisting pain was not relieved after surgery and cases where it is not clear whether surgery caused the pain. 11Therefore, a group of international multidisciplinary experts have recently proposed the term persistent spinal pain syndrome (PSPS). 113][14] Following this proposal, CBLP is referred to in this article as PSPS type 1 (persistent pain without previous surgery) and type 2 (persistent pain after spine surgery). 11In addition to the clinical scenarios, the expert panel identified eight psychosocial factors that should be considered when determining the eligibility of patients for SCS. 8 In 2019, these panel recommendations were embedded in an educational e-health tool (https://scstool.org)with the primary goal of educating implanters and referrers on the appropriate (pre)selection of patients with chronic pain for treatment with SCS.The e-health tool provides the user with a panel recommendation (ie, not recommended, recommended, and strongly recommended) upon entering of the clinical and psychosocial characteristics (Fig. 1).
This study aimed to retrospectively evaluate the applicability and validity of the appropriateness criteria embedded in the educational e-health tool on real-life patient data, exploring the relationship between panel recommendations, center decisions, and patient outcomes for SCS.

Development of the Educational e-Health Tool for SCS
The development of the educational e-health tool is described elsewhere. 8Based on a literature review and expert opinion, a multidisciplinary panel of 18 experts (ten anesthesiologists, three neurosurgeons, three psychologists, one nurse specialist, and one physiotherapist) from nine European countries selected absolute inclusion and exclusion criteria for the consideration of SCS.Afterward, the panel defined clinical variables for four indication areas (PSPS [previously CBLP], CRPS, NPSs, and IPSs), generating a total of 386 theoretical clinical scenarios that were rated using the RUAM to assess the appropriateness of (referral for) SCS.Variables for each of the indication areas included treatment history, type/nature and location of pain, anatomic abnormalities, spread of pain, and response to previous procedures.From a clinical perspective, panelists individually rated the appropriateness of each clinical scenario on a nine-point scale (reference values: 1 = inappropriate, 5 = equivocal/uncertain, and 9 = appropriate) without considering the cost of treatment, reimbursement environment, or other potential constraints and regardless of the different SCS types available.In addition to the clinical variables, the panel identified eight psychosocial factors (ie, lack of engagement, dysfunctional coping, unrealistic expectations, inadequate daily activity level, problematic social support, secondary gain, psychologic distress/mental health problems, and unwillingness to reduce high-dose opioids) to be considered when selecting or referring patients for SCS.A more detailed description of the psychosocial factors is embedded in the ehealth tool (https://scstool.org/en/Home).Based on the ratings of the clinical and psychosocial factors, the e-health tool was developed, generating patient-specific recommendations on SCS for both implanters and referrers upon completing a patient's clinical and psychosocial profile.

Study Population and Design
This study exclusively involved centers from participants of the RUAM panel study. 8In total, 12 centers from seven European countries (Belgium, France, Germany, Italy, Sweden, The Netherlands, and the United Kingdom) were included.In this retrospective study, participants were asked to compile an anonymized list of all patients for whom SCS had been considered between January 1, 2018, and June 30, 2019.Starting with the most recent case, a series with a minimum of 25 and a maximum of 50 consecutive cases had to be included for data entry.This procedure aimed at preventing selection bias and allowing sufficient spread over the participating centers.SCS devices were used from all manufacturers, and the type of SCS mode used was based on the standard practice of each included center.Data were analyzed in June 2021, ensuring that patients with a positive decision had follow-up of at least six months after the SCS implant.

Data Collection
Data entry was done using an online data capture program.For each selected patient, baseline patient data (sex, age, and numeric pain rating scale [NRS]), e-health tool variables (clinical and psychosocial), center decisions on SCS, and patient outcomes were documented.For all patients, the clinical and psychosocial data were retrospectively entered in the e-health tool to determine the appropriateness outcomes.A center's decision on SCS could be negative, deferred, or positive.In case the decision was positive, experts had to indicate if patients were receiving a trial or direct implant.If patients received a trial, they had to specify if the outcome of the trial resulted in a positive, negative, or deferred decision for SCS.To evaluate long-term pain reduction, experts were asked to document the NRS and report the global perceived effect by the patient and observer at six-month follow-up.

Data Analysis
Frequency tables and cross-tabulations were used to analyze ehealth tool recommendations in relation to center decisions and patient outcomes.The impact of missing data on e-health tool outcomes was studied by sensitivity analysis.

Ethics Committee Review and Approval
Patient data were collected retrospectively and were entered completely anonymously in an online data capture program.All participants conformed to their national/local ethics committee/ institutional requirements and received approval before data entry.

Patient Population
In total, 491 patients had been considered for SCS.Of those, eight were excluded, because they were considered for indications other than the ones included in the e-health tool (ie, widespread pain and unspecified [N = 1]).Most of the included patients received an implant (N = 391, 81%), with 346 (88%) of those having complete outcome data at six-month followup (Table 1).Improvement in pain outcomes after SCS was similar across assessments, with limited variation between the different indication areas (Table S1).

Patient Characteristics Clinical Profiles and Variables
Of the 386 different theoretical clinical profiles embedded in the e-health tool, 132 (32%) were seen in the retrospective study cohort, with 60 theoretical profiles (16%) covering 80% of the patients included in the retrospective analysis.For clinical variables in the e-health tool, 1.1% of data points were missing, applying to 5% of the patients.These missing clinical data involved at most two variables of a profile, without affecting the final tool outcome.
Details on the clinical variables by main indication area and SCS implant are provided in Tables S2 to S5.Overall, the differences between patients who received an implant and those who did not were limited, but some significant differences were found.For PSPS, the presence of scar tissue was more prevalent in patients receiving an implant than in those considered for SCS but not receiving an implant.Furthermore, not receiving an implant was more frequently associated with experiencing nociceptive pain in case of PSPS.In addition, the percentage of patients considered for SCS without a previous response to alternative treatments/ interventions (eg, neuropathic pain medication and transcutaneous electrical nerve stimulation) was considerably higher in those not receiving an implant.This suggests that the lack of response to alternative treatment options may have contributed to some extent to the center decisions on SCS.For IPSs, all patients who were considered for SCS received an implant, and hence no distinction could be made between patients who received an implant and those who did not.

Psychosocial Variables
In addition to the clinical variables, data on the eight psychosocial variables in the e-health tool were complete in 448 patients (92.8%).It was assumed that aspects not reported in the medical record were most likely absent or did not have a significant impact on the psychosocial profile of the patient, including all patients (N = 483) in the subsequent analyses.Of these patients, 308 (64%) were reported to have one or more psychosocial factor(s) to be considered when determining the appropriateness for SCS (Fig. 2a).The three most commonly reported psychosocial factors were psychologic distress/mental health problems (42.2%),inadequate daily activity level (38.1%), and dysfunctional coping (27.2%).Psychosocial factors were mostly moderate in severity (Fig. 2b).

e-Health Tool Recommendations vs Center Decisions
Figure 3 shows the appropriateness of SCS according to the ehealth tool recommendations in relation to the center decisions.SCS was either "strongly recommended" or "recommended" in most patients (87%).In the remaining 13%, SCS was "not recommended," which in 80% of these patients was driven by the presence of one or more psychosocial factors in the severe category.The percentage of patients for whom SCS was strongly recommended or recommended by the e-health tool was considerably higher in patients receiving a direct implant or an implant after a positive trial than in patients not receiving an implant after a negative trial or for other reasons such as refusal of SCS by the patient (Fig. 3).

e-Health Tool Recommendations vs Patient Outcomes
In patients who received SCS and had complete outcome data at six-month follow-up (N = 346), the e-health tool recommendations were significantly associated with three types of patient outcomes: much or very much improvement by observer perception (χ 2 [2, N = 346] = 25.0;p < 0.001), much or very much improvement by patient perception (χ 2 [2, N = 346] = 20.0;p < 0.001), and at least 50% pain reduction measured by the NRS (χ 2 [2, N = 346] = 15.6;p < 0.001) (Fig. 4).Substantial improvement ranged from 25% if the e-health tool outcome was not recommended to 83% if SCS was strongly recommended and comparable results were seen for the various outcome measures.For the strongly recommended and recommended categories, these patterns were fairly similar between patients receiving a direct implant and those receiving an implant after a positive trial (Fig. S1).

e-Health Tool Recommendations vs Trial Results
In patients who underwent a trial (N = 290), the e-health tool recommendations were significantly associated with the trial outcomes (χ 2 [2, N = 290] = 40.8;p < 0.001).Trials failed in 3% of patients for whom SCS was strongly recommended, whereas a negative trial was seen in 46% of patients for whom the e-health tool outcome was not recommended (Fig. 5).

DISCUSSION
This study retrospectively evaluated the applicability and validity of previously defined appropriateness criteria on real-life data from patients for whom SCS was considered.6][17] The appropriateness criteria were embedded in an educational e-health tool (https://scstool.org)aiming to help implanters and referrers learn to apply an integrated approach when (pre)selecting patients for SCS.
Participating experts were asked to document patient data reflecting a daily practice population in implant centers.The clinical variables reported by the experts were identical to the ones included in the e-health tool allowing retrospective evaluation of their applicability.Overall, 34% of the possible clinical scenarios were represented in this study, with 60 profiles covering 80% of the included patients.These numbers are comparable with other RAND/University of California at Los Angeles studies, underscoring the applicability of the clinical variables embedded in the e-health tool. 18In addition to the clinical variables, all psychosocial factors were categorized multiple times by the experts as either moderate or severe, confirming their relevance in the consideration of SCS.The applicability of both the clinical and psychosocial factors was further demonstrated by the very low number of missing data, which supports the assumption that these factors relate to commonly collected and documented key variables in the decisionmaking on SCS.
Following retrospective application of the clinical and psychosocial factors to the e-health tool, the panel recommendations were compared with the center decisions.SCS was either strongly recommended or recommended in approximately 90% of the patients receiving an implant.Although the number of patients with a negative trial was low (N = 32, 7%), SCS was not recommended in 41% of them.Because patients with a negative trial did not receive an implant, we could not evaluate if there would be any false negatives in the remaining 59% for whom SCS was strongly recommended or recommended.False negatives can be expected, because it has previously been suggested that screening trials may exclude good SCS candidates. 19In addition, SCS was not recommended in 38% of patients who did not receive an implant for reasons other than a negative trial.In 45% of these patients, the decision on SCS was negative, most likely explaining the high  number of patients for whom SCS was not recommended, whereas in the other patients, the decision on SCS was deferred because of reasons such as refusal by the patient or presence of concurrent diseases.However, these reasons were too diffuse to explain any correlation between the panel recommendations and center decisions.

Percentage of paƟents
When evaluating the validity of the e-health tool, retrospective application demonstrated a significant association between the panel recommendations and pain outcomes.Substantial improvement in pain outcomes ranged from 25% if the e-health tool outcome was not recommended to 65% and 83% if SCS was recommended and strongly recommended, respectively.Furthermore, this retrospective study showed that differences in pain outcomes between patients receiving a direct implant and those receiving an implant after a positive trial in whom SCS was strongly recommended or recommended were small.This would, to some extent, support the findings from the recently published TRIAL-STIM study that found no difference in pain outcomes between patients who received a screening trial and those who did not. 20The TRIAL-STIM study argued to reject the dogma that patients can only be considered for SCS after a positive trial, emphasizing the need for careful multidisciplinary assessment including psychosocial evaluation when evaluating candidates for SCS, 20 which is considered in the appropriateness criteria embedded in the e-health tool. 8lthough the trial results were confirmatory if the e-health tool outcome was strongly recommended or recommended, more patients for whom SCS was not recommended had improvement when receiving an implant after a positive trial compared with those who received a direct implant.This suggests that trials may especially be valuable when considering SCS despite a not recommended outcome but requires further study because only a limited number of patients were included in the not recommended group.This scenario did not arise in the TRIAL-STIM study because a positive multidisciplinary team recommendation was required as an entry criterion to the study. 20Following the results described earlier, trial failure was minimal (<10%) when SCS was strongly recommended or recommended, confirming the association between the panel recommendations and trial results.However, 54% of the trials in the not recommended group were positive, albeit only 25% of patients for whom SCS was not recommended showed improvement in pain outcomes.Because of the retrospective design and the low number of patients included in the not  recommended group, we could not determine the reasons explaining the positive trial results and improved six-month pain outcomes in this group.Nevertheless, the not recommended outcomes were mostly caused by contraindicated psychosocial factors, and we hypothesize that these may have been managed before or after SCS, positively affecting pain outcomes.In addition, pain outcomes were only assessed at six-month follow-up, not allowing assessment of any fluctuations in improvement or evaluation beyond this time point.One of the most important limitations of this study is its retrospective design, which may have resulted in some recall bias.Although we aimed to prevent selection bias by inclusion of consecutive patients, we cannot exclude that the retrospective study design may have contributed to the preferential inclusion of patients receiving an implant, explaining the rather low number of patients with negative trial results and the limited number of patients for whom SCS was not recommended.Another limitation is the reporting of data by or under the supervision of experts who were involved in the development of the e-health tool, currently evaluating decision-making at the level of the implant centers without validating the e-health tool from the referrer perspective.In addition, it is unknown if patients who initially reported success with SCS at six-month follow-up deteriorated over a longer time horizon because of tolerance to SCS, exacerbation of PSPS, or development of new pains. 21Although this loss of therapeutic effect has been described, we do not know if pain outcomes would be similarly reduced in patients for whom SCS was strongly recommended and recommended compared with those for whom SCS was not recommended.Finally, the data were collected from expert European centers with large SCS practices and thus may not be generalizable to non-European SCS practice or starter centers where expertise in detecting psychologic barriers may not be immediately accessible or developed.
Most of these limitations will be addressed in a prospective study aiming to validate the described associations determining the predictive value of the panel recommendations on patient outcomes.In the prospective study, we hope to evaluate the unbiased applicability of the clinical and psychosocial variables and to reduce potential selection bias by the inclusion of implanters and referrers who are unfamiliar with the e-health tool.

CONCLUSION
Patient-specific recommendations on the appropriate referral and selection of patients with chronic pain for SCS were previously developed by a European multidisciplinary expert panel using the RUAM.These recommendations were embedded in an e-health tool (https://scstool.org)that may be implemented as an initial assessment, considering both clinical and psychosocial factors important in the (pre)selection of patients for SCS.As an evaluation of its validity, the e-health tool recommendations were retrospectively applied to real-life patient data, suggesting predictive value of the e-health tool for SCS trial results and treatment outcomes in patients with chronic pain.The predictive value will be further examined in a prospective validation study including both implanters and referrers.

Authorship Statements
Simon Thomson, Frank Huygen, and Herman Stoevelaar designed the study.Herman Stoevelaar and Nicky Helsen performed the data analyses and prepared the manuscript draft together with Frank Huygen and Simon Thomson.All authors, except Herman Stoevelaar and Nicky Helsen, contributed to the data collection.All authors have reviewed and approved the final manuscript.

SUPPLEMENTARY DATA
To access the supplementary material accompanying this article, visit the online version of Neuromodulation: Technology at the Neural Interface at www.neuromodulationjournal.org and at https:// doi.org/10.1016/j.neurom.2021.12.006.

COMMENTS
Surgery has long utilized risk and outcome calculators for a variety of diseases and outcomes.These include the NSQIP Surgical Risk calculator and the revised cardiac risk index for preoperative risk (RCRI).The SCS tool is looking at being a predictive tool in a similar manner.Once functional assessment parameters can be built into the tool, we will have a more robust method for predicting appropriate patients for the most successful outcomes.
Marshall Bedder, MD Augusta, GA, USA *** It is important to have a triage system that is adequately sensitive and specific for screening referrals for neuromodulation for pain.The e-health tool proposed by Thomson and colleagues (Thomson S, Huygen F, Prangnell S, et al.Appropriate referral and selection of patients with chronic pain for spinal cord stimulation: European consensus recommendations and e-health tool.Eur J Pain. 2020;24:1169-1181) and evaluated in this article is an important step to refine and streamline the assessment process that precedes an implant.The data reported in this article adds to our understanding of the outcomes of SCS.Anuj Bhatia, MD, MBBS Toronto, Ontario, Canada *** SCS is an established treatment modality for various chronic pain conditions.Despite this, there is considerable variation in practice and what factors suggest a person would do well with an implant and who would not.Therefore, the decision to implant will be determined by clinicians' opinion, and this will be dependent on clinical experience and the disciplines represented as part of the assessment team.For conditions where patients have often undergone many different unhelpful or unsuccessful treatments, there are risks associated with exposing a person to further treatment.These need to be carefully considered to prevent poor outcomes and potential harms associated with repeated treatment failure.The tool includes screening for psychologic measures and therefore helps referrers and implants consider relevant and important factors to improve patient outcomes and reduce potential harms.This should not replace a psychologic assessment but provide a tool that can be considered a minimum standard of care in terms of SCS screening.Selina Johnson, MSc, BSc (Hons) Liverpool, England, United Kingdom

Figure 1 .
Figure 1.User interfaces of the e-health tool: clinical aspects, psychosocial factors, and composite outcome combining clinical and psychosocial variables.

Number of psychosocial factors a b Figure 2 .
Figure 2. Psychosocial variables across all indications.a. Number of psychosocial factors and (b) psychosocial factors by type and severity, both reported as thepercentage of all patients considered for SCS (N = 483).In the case of missing data, it was assumed that these psychosocial factors were absent or of less importance.

Figure 4 .
Figure 4. Retrospective comparison of the e-health tool recommendations vs patient outcomes.Improvement by perception was defined as the percentage of patients with strong improvement (much to very much) as reported by the observer or patient.Pain reduction was defined as the percentage of patients with at least 50% reduction as measured by the NRS.

Figure 3 .
Figure 3. Retrospective comparison of the e-health recommendations vs center decisions.The category "Other" included patients with a negative (27/60) or deferred decision (33/60) because of a variety of reasons such as refusal by the patient or presence of concurrent diseases.

Table 1 .
Demographics of the Patients, Center Decisions, and Clinical Indications for SCS.
*No implant, including patients with a negative or deferred decision.
SCS E-TOOL: APPLICABILITY AND VALIDITY www.neuromodulationjournal.org © 2021 The Authors.Published by Elsevier Inc. on behalf of the International Neuromodulation Society.This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/4.0/).