4 studies compared on Nest savings
Cross Study analysis of Nest thermostat savings
Internet-connected thermostats are gaining wider adoption in the marketplace and in utility energy efficiency efforts. To date, there have been relatively few published impact evaluations of this technology. The potential energy savings of these devices pose some challenges for evaluators seeking to quantify impacts and generalize results for planning and policy purposes. We decided to get to bottom of the question, “How much does Nest Thermostat save?” by comparing our Nest Thermostat Review findings with actual studies done about the subject. This paper summarizes results from four impact evaluations of this technology, as embodied in the Nest Learning Thermostat, and explores some of these issues.
The evaluations include two studies performed by utilities in Indiana, a study of heat pump homes by the Energy Trust of Oregon, and a study by Nest of customers who had enrolled in the MyEnergy service that helps track utility usage. The measured energy savings were quite consistent across studies. Heating savings ranged from 10% to 13% of heating use and electric savings ranged from 14%-18% of cooling use. This consistency was unexpected given the diversity of study populations and levels of customer engagement. The studies with the least engaged participants actually had larger evaluated savings than the study with the most engaged participants. These results may make more sense considering that the potential for energy savings from automated features may be larger for customers who have little interest in micro- managing their set points.
In addition to prior behavior patterns, thermostat savings are expected to vary based on factors such as occupancy patterns, climate, HVAC type and house construction characteristics. Evaluations need to assess how well the study population represents the population of interest and be cautious about generalizing from pilot results to different populations.
Check Amazon for Reviews and Prices
Smart thermostat saving potentials
Internet-connected (a.k.a. “smart”) thermostats generally provide two primary mechanisms for reducing energy use: optimizing thermostat set points, and adjusting how HVAC equipment is controlled. The optimization of set points can be accomplished by a schedule (which may be set explicitly or learned by the device), by occupancy detection to adjust temperatures when the home is vacant, by remote control, and by features designed to impact occupant behaviors and choices about settings. Thermostats can affect HVAC equipment efficiency through a variety of features such as algorithms that minimize the use of electric resistance auxiliary heat in heat pumps or that control the air handler fan to provide some evaporative cooling in the cooling season.
Standard programmable thermostats (PTs) have been promoted as an energy savings product for many years. The real world energy savings provided by programmable thermostats has been an area of controversy. The Energy Star program of the US Environmental Protection Agency (EPA) summarized the issue in 2003:
“Consumers are often advised that installing a programmable thermostat can save them anywhere from 10 to 30% on the space heating and cooling portion of their energy bills. While reliant on proper use of the programmable thermostat, such savings are easily true in theory; however, there needs to be more field-tested data to better substantiate savings claims. Analyses from recent field studies have suggested that programmable thermostats may be achieving considerably lower savings than their estimated potential.” [EPA 2003]
The energy savings are primarily expected to come from automatically turning down the heating set point temperature (or turning up the cooling set point) when people either aren’t at home or are sleeping (known as “setback”). The magnitude of the savings depends on the how much the temperatures are changed compared to the settings before installing the thermostat.
Field research has found that many programmable thermostats aren’t actually programmed due to usability and design problems (see Peffer et al. 2011 for a literature survey), leading to set points that aren’t much more efficient than manual thermostat set points and therefore result in uncertain energy savings. The range of field research led EPA to end the Energy Star designation for programmable thermostats in 2009. Still, the government and manufacturers have continued to explain the energy savings potential of well-programmed thermostats in terms of the possible savings relative to previous set point assumptions. The U.S. Department of Energy (DOE) lists heating savings of 5%-15% for a single eight-hour temperature setback per day compared to a constant temperature setting (DOE 2015). Although it has ended the Energy Star certification for programmable thermostats, EPA still lists savings of $180 per year for a programmable thermostat (EPA 2015). The Nest web site states that customers “could cut 20% off your heating and cooling bill” compared to maintaining a constant temperature (Nest Labs 2015), where the constant temperature is based on customer-specific set points. Other thermostat manufacturers make a variety of savings estimates:
- “Customers in the US saved an average of 23% on their heating and cooling costs” based on a comparison to an assumed 72°F constant heating set point (Ecobee 2015)
- “Homeowners saved an average of 20% on their heating and cooling energy costs” based on a comparison to an assumed 72°F constant heating set point (Carrier 2014)
- “Cut your heating bill by up to 31%” compared to a constant set point (Tado 2015)
All of the thermostat savings estimates are based on models (simulation or statistical) of how set points affect energy use and calculate the savings compared to an assumed constant temperature set point. It’s been common practice to assume a constant set point as the baseline setting behavior because:
- It provides a clear reference condition,
- Data on prior set points are rarely available, and
- Field research has found that many programmable thermostats aren’t being programmed (so the set points are constant) (Meier et al, 2010).
The savings estimates based on the constant set point assumption are a useful guide but may not reflect actual savings if the baseline assumptions aren’t met. Real world energy savings are best assessed by standard evaluation methods such as pre/post billing data analysis.
Check Amazon for Reviews and Prices
Study Designs and Methodologies
This paper summarizes energy savings from four studies of Nest thermostats:
- An evaluation of gas heating and electric cooling savings sponsored by the Indiana utility company Vectren (Aarish et al 2015a)
- An essentially identical evaluation of gas heating and electric cooling savings sponsored by the Indiana utility company NIPSCo (Aarish et al 2015b)
- A study of electric heating savings in homes heated by heat pumps sponsored by the Energy Trust of Oregon (Apex Analytics 2014)
- A study by Nest Labs of Nest customers who enrolled in the MyEnergy utility usage tracking service (Nest Labs 2015b).
This paper is too brief to fully describe the methods and many findings of all four studies, so interested parties should refer to the original reports to gain a fuller understanding of each study. Instead, this paper provides an abbreviated description of each study to provide some context for the results.
All four evaluations compare pre- and post-thermostat billing data analysis, and use treatment and comparison groups to measure energy savings. The two Indiana studies performed house level weather normalization. The Oregon study employed a pooled mixed effects model to account for partial year post- retrofit data with a focus on only assessing heating savings in heat pump homes. Nest’s analysis employed both pooled fixed effects models and house level weather normalizations consistent with the US DOE Uniform Methods Project (DOE 2013).
The Vectren and NIPSCo Indiana studies were designed, conducted, and evaluated simultaneously and may be viewed in some ways as one larger study. The studies recruited participants from customers who had received energy audits in the past but hadn’t proceeded with any retrofits. These customers were required to have gas heat, central air conditioning, and a manual thermostat. The customers were further divided into two groups – the treatment group was offered a Nest thermostat and the comparison group was offered a standard programmable thermostat. The thermostats were offered for free and were installed and set up by a contractor. The experimental design encountered an obstacle due to the need for working Wi-Fi in homes that received a Nest. This requirement led to final samples that differed in terms of demographics and prior energy use, making direct comparisons between groups less clear.
The Oregon heat pump pilot sampled from customers who lived in single-family detached homes heated by a heat pump with electric auxiliary heat, had received an energy audit, and hadn’t participated in any Energy Trust programs for the past year. The sample was further narrowed geographically and a comparison group was set aside prior to recruiting participants. The participants were screened by phone to check that they had Wi-Fi and did not plan any major changes in the next year. The thermostats were installed for free by a contractor who also set the Nest Heat Pump Balance feature to “Max Savings” (to reduce auxiliary heating use the most).
The Nest study was national in scope and included Nest customers who had purchased the thermostat on their own and had enrolled in the MyEnergy service that helps customers track their utility usage. The electric analysis focused on homes with central air conditioning loads (defined as >500 kWh/yr in estimated cooling use) and without electric heat (there were too few electrically heated homes in the sample to reliably evaluate). The gas analysis excluded homes where electric heating usage was also detected. The final natural gas sample included customers from 36 different states with an average of 4,533 annual heating degree days, comparable to Baltimore, MD. The final electric sample included customers from 39 different states and an average of 1,729 cooling degree days, comparable to Charlotte, NC.
Findings: Gas and Electric Savings
The energy savings results of the four studies are summarized in Table 1.
Table 1. Gas and Electric Savings Results
* Note: the Oregon study estimated heating usage based on data from other research
The heating savings spanned a fairly narrow range – from 10% found in the Nest MyEnergy study to 13% found in the NIPSCo study. In terms of gas usage, the NIPSCo study found significantly more therm savings than the other gas studies but also had much higher heating usage.
The Oregon heat pump study found 12% heating savings, quite similar to the gas heating savings, although the study itself attributes much of the savings to the Heat Pump Balance feature designed to reduce auxiliary heating use. Heat pumps have long been considered a poor application for programmable thermostats due to the frequent need for low efficiency auxiliary heat when recovering from a setback (US DOE 2015). The Nest algorithm is designed to avoid this penalty and also reduce other unnecessary reliance on auxiliary heating. One perhaps unexpected finding from the Oregon pilot was that 66% of participants reported improved comfort with the Nest thermostat. Reducing auxiliary heating use is often associated with reduced comfort as delivery air temperatures for auxiliary heat are typically much higher than for compressor-based heating. Nest’s heat pump algorithms are designed to give customers control over comfort and energy savings and perhaps that control provides greater overall satisfaction.
The electricity savings are also in a fairly narrow range in terms of kWh and also when savings are expressed as a percentage of the cooling usage – ranging from 14%-18%. It should be noted that the electricity savings would also include heating season savings due to reduced furnace fan run time. In the Nest MyEnergy study, the percent savings shown are the percent of total HVAC use, which includes a small amount of heating seasonal usage. It should be noted that the electricity savings are all a fairly small fraction of total electricity usage in these homes and have fairly wide confidence intervals. More studies of cooling savings would help provide better estimates of the range of savings being achieved.
Overall, the study results were quite consistent in terms of percent heating and cooling savings. But there are still questions about whether such savings can be expected in other populations, climates, and housing stocks and HVAC types.
Potential Biases
All evaluations need to consider potential sources of bias. The most common issues relate to whether the participants being evaluated represent the larger population of interest and whether the comparison group is truly representative of how participant energy usage would have changed without the retrofit.
Nest MyEnergy Customers: Too Energy Conscious?
In the Nest MyEnergy study, the analysis group consisted of customers who purchased a Nest Learning Thermostat on their own and also chose to sign up for MyEnergy. People who enroll in MyEnergy are interested in tracking their utility energy use, and so they can be expected to be more energy conscious and efficient than the average Nest customer. Although it may seem counterintuitive, this greater interest in energy efficiency may lead to lower energy savings from the thermostat. The most energy conscious customers are the ones more likely to have already had efficient thermostat settings – either because they put in the effort to properly use their old programmable thermostat or they consistently set back temperatures whenever feasible prior to having a Nest. Prior behavior has a large impact on savings potential.
The Nest MyEnergy study explored the potential bias from the sample composition through an email survey and an analysis of Nest settings. Table 2 summarizes some key findings from the survey.
Table 2. MyEnergy Customers compared to average Nest customers
Note: Survey results are based on 657 MyEnergy Nest customers and 763 Nest customers not in MyEnergy
The table shows that the MyEnergy customers reported having more efficient set points prior to installing the Nest than the average Nest customer surveyed. Compared to the other Nest customers, MyEnergy customers were more likely to have a programmable thermostat, more likely to employ two or more setbacks per day, and less likely to have practiced no setbacks prior to having the Nest. These differences all suggest that MyEnergy Nest customers have less potential for saving energy since they were already more efficient. Modeling suggested that the impact of these differences was that the MyEnergy customers had about 2% lower savings potential than the average Nest customer – their set points were calculated to be about 10% more efficient than a constant baseline compared to about 8% more efficient for the average Nest customer.
The last two rows of the table summarize the actual Nest Learning Thermostat customer set points during February and March 2014 for the surveyed homes. The MyEnergy Nest customers maintained a lower average heating set point than the average Nest customer and also had greater night temperature setbacks (primarily more frequent rather than deeper). Differences were also found for other settings, such as daytime setbacks, and for the use of Nest features such as Heat Pump Balance (more than twice as likely to select “Max Savings”). We used energy modeling to estimate the impact of these differences and calculated that the MyEnergy customers were about 2% more efficient with their Nest set points than the average Nest customer.
The conclusion from this assessment of bias was that MyEnergy customers were more efficient than the average Nest customer both before and after installing their Nest, and the magnitude of these differences was similar – implying no significant bias between the groups.
It’s also worth noting that both groups of Nest customers reported more efficient prior thermostat practices compared to studies of typical US household thermostat use. A literature review (Peffer et al. 2011) reported that 42% of US households had programmable thermostats in 2008, and 47% of programmable thermostats were running a program. In contrast, 65% of non-MyEnergy Nest customers reported having a programmable thermostat, and 71% of those were running a program. These results indicate that Nest customers tended to have more efficient set points than the average U.S. household, which reduced the potential for savings.
Another potential source of bias in the Nest MyEnergy study was the comparison group of non- participants which consisted of people who signed up for MyEnergy on their own. The fact that they chose to enroll on their own (possible self-selection bias) implies that they may differ from the MyEnergy customers that were recruited by Nest. This difference could introduce a downward bias on savings if, for example, the non-Nest MyEnergy customers were more likely to pursue other efficiency upgrades on their own – which may have led them to sign up for MyEnergy.
Overall, the analysis did not uncover any evidence of a large bias from having the study focus on MyEnergy customers, although the comparison group issue suggests that any likely bias would lead toward finding lower energy savings than the average Nest customer might achieve.
Vectren and NIPSCo Customer Engagement Issues
The Vectren and NIPSCo studies both encountered issues with customer engagement. Nest had agreed to provide the evaluators with some aggregated thermostat data for the pilot participants. In the course of extracting these data, Nest found that a surprisingly large fraction of the pilot participants had never connected their Nest thermostat to the Internet, and a large fraction that were connected had never registered the thermostat and created an account – which was needed to use the phone apps or web site access for remote control. Table 3 summarizes these basic customer engagement metrics for all four studies and also for Nest customers in Indiana who had purchased a thermostat from nest.com.
Table 3. Customer engagement data
Note: The Oregon and Nest MyEnergy results are shown for reference, but both required a thermostat internet connection and the MyEnergy study also required a Nest account.
The third column in the table shows the percent of Nest thermostats that were connected to the Internet, and the fourth column shows what percentage had registered to create an account and provide access to remote control features through phone, tablet apps and the web. The two Indiana pilots stand out for having nearly 20% of the thermostats never connecting to the Internet and only about half of the customers ever creating an account – far lower levels of engagement than found in other studies or among Nest customers overall.
The findings suggest that the contractors were not always able to fully install the thermostat and, perhaps more importantly, that many of the customers were not very interested in the technology and features like remote control through a phone app (one of the most popular Nest features). The low level of engagement may be related to the fact that the thermostats were offered for free, so that even customers who were not interested in the technology might agree to the installation. This lack of engagement with the product features might be representative of what would occur if a utility planned to give away Nest thermostats. But if the planned program design was to provide a rebate for less than 100% of the cost, then the program population could be expected to differ from this pilot group.
The low levels of pilot customer engagement caused concern that the pilots may not show significant energy savings. But the evaluations actually found slightly larger heating savings in these two groups than in the more engaged Nest MyEnergy study. This finding might be explained by differences in pre-Nest behaviors and overall savings potential. The very engaged MyEnergy customers were found to be more efficient before they installed their Nest and also more efficient after the installation. Perhaps the Indiana study participants were less efficient in their prior set points – providing a greater opportunity for savings. The majority of Nest features such as Auto-Schedule (learning) and Auto-Away work without an Internet connection.
The lessons learned about customer engagement from the MyEnergy and Indiana studies are that it may not be wise to assume any clear relationship between how people use their new thermostat and how much energy they are likely to save. The savings are just as much a function of where the customers started in terms of set points and existing behavior patterns as they are of how they use their connected thermostat. The findings also suggest that evaluators should be careful in generalizing from a pilot project to a future
program if the populations of customers may differ due to program designs since the extent and direction of potential biases are hard to estimate.
Check Amazon for Reviews and Prices
Other Findings and Issues
The four studies addressed a variety of other issues related to thermostat savings, customer satisfaction and pilot project design, which the full reports describe in more detail. Over the course of conducting the Nest MyEnergy study and reviewing these other studies, several evaluation issues have been identified that are still being worked on, including assessing effective measure life, savings longevity and non-energy benefits.
Effective Measure Lifetime and Savings Over Time
Quantifying the effective measure life of a connected thermostat poses some challenges. The hardware itself may last well over 10 years. But it may also be perceived as a consumer electronics device that is purchased for more than just efficiency reasons, and so customers may choose to upgrade to a newer model for reasons other than energy savings. Should the measure life end when the original device is replaced or when the customer no longer has a comparable product or perhaps only if they receive another incentive for a comparable product?
Beyond just the issue of actual measure lifespan is the issue of savings lifespan. Research has found that, over time, homes with standard programmable thermostats have no more efficient set points than homes with manual set points (Meier et al. 2010). The Vectren and NIPSCo pilot projects both assessed savings from standard programmable thermostats that were professionally installed and set up and looked at first year savings. Those studies found standard programmable thermostat heating savings about half as large as the Nest heating savings, but cooling savings were about the same size. But those results should be considered the maximum savings that standard programmable thermostats will ever provide given the professional installation and set-up. The Vectren study found evidence of schedule degradation already occurring in the first year, reporting that “only 37% of participants appear to have relied on their thermostat program by the end of the study period.” Therefore, savings from a standard programmable thermostat could be expected to degrade over time as more users override their schedules. More research is needed on schedule longevity (e.g., survival analysis).
For connected thermostats, the longevity of efficient set points has not been documented. But there is the real potential for savings to actually improve over time. Nest has a feature, Seasonal Savings, that offers (on an opt-in basis for customers of Nest energy partners) to gradually improve customer schedules over a few weeks and then asks the customers if they would like to keep the new schedule – providing them an estimate of the savings that could be expected from the change. Results from this algorithm have been promising and should be available in 2015.
When comparing a standard programmable thermostat to a connected thermostat, the difference in savings may be smallest during the first year, when standard thermostats are most likely to be properly programmed, and then the difference may grow over time as the standard units suffer from program attrition (e.g., they use the Hold button, stopping the programmed set points) while connected units may have their schedules stay the same or even improve over time due to special interactive features and feedback.
Another issue related to savings lifespan is software updates. Connected thermostats can continue to receive software updates over time. These updates may provide new features or adjust how existing features work. While most of these updates might be expected to improve savings, some may reduce savings. When evaluating a connected thermostat project or considering how to apply prior evaluation results to new program designs, the version number of the hardware and software may be worth tracking.
Non Energy Benefits
Non-energy benefits are a potentially significant issue when evaluating connected thermostats. Some consumers seek out and purchase connected thermostats for reasons other than energy savings – such as remote control via phone app or the appearance of the thermostat on the wall. The Oregon pilot survey found that 34% of participants reported that the Nest Learning Thermostat would be worth the full purchase price even if it didn’t save any energy at all. These potential consumer benefits should be considered when conducting cost-effectiveness screening of connected thermostats – recognizing that they are not only an energy saving device but also a consumer electronics device and part of home decor.
Savings Compared to Prior Projections
The results from the four studies indicate that Nest customers are saving about 10%-13% of their heating energy usage on average. The survey results from the Nest MyEnergy study suggested that Nest customers had previous set points that were about 8%-10% more efficient than the standard constant temperature baseline assumption (and were more efficient than the average household). These findings are actually consistent with the estimate of about 20% savings compared to a constant temperature baseline – customers had been 8%-10% better than a constant temperature and then became about 20% better, leading to 10%-12% heating savings. It appears that the modeling was accurate, and the baseline assumption is responsible for the difference in savings.
The importance of baseline condition assumptions in estimating energy savings is not unique to thermostats. Savings from common retrofits such as insulation and HVAC replacement also depend strongly on the assumed prior conditions. The added challenge for thermostats is that the prior conditions are not directly observable or easily estimated.
Check Amazon for Reviews and Prices
Other Factors Affecting Thermostat Savings
In addition to the key role of prior set point behaviors, thermostat savings can also be expected to vary significantly depending on the general occupancy schedule of the home, the climate, the type of HVAC equipment, baseline energy consumption, and the efficiency of the home. Homes that are occupied all day have little savings potential for daytime setbacks or daytime Auto-Away. Homes that are very well insulated and tight have reduced savings potential from setbacks because temperatures will drop more slowly during a setback. Homes heated with heat pumps that use auxiliary heat extensively have higher savings potential from algorithms designed to reduce that usage while heat pump homes that use little auxiliary heat may have more limited savings potential. Variations in savings potential related to demographics, housing and HVAC characteristics pose a challenge for designing pilot projects and using those results for planning purposes.
Evaluators and program planners need to consider how well the analysis sample represents the larger target population and how any differences may affect the results. For example, the heating savings found in a pilot project where the thermostats are given away for free may differ from the savings found when purchased with a rebate – in ways that are hard to assess.
Conclusions
This paper has presented results from four evaluations of the energy savings from Nest Learning Thermostats. The studies found generally similar results – showing Nest Learning Thermostat heating savings of about 10%-13% and electric savings of about 14%-18% of cooling use in homes with central air conditioning.
Although the overall results were similar across studies, differences in study designs and associated customer behaviors make this finding somewhat unexpected. The savings from the pilot studies with least engaged customers appeared to achieve slightly larger savings than the study that focused on the most energy conscious customers. These results highlight the importance of prior behaviors on the savings potential from connected thermostats. Customers that are most interested in energy efficiency may already have had efficient set points, reducing the potential for savings. Customers who were not very engaged are may be more likely to have had less efficient settings prior to the thermostat installation providing greater savings potential even with less efficient schedules than more engaged customers.
Savings can also be expected to vary significantly depending on the general occupancy patterns of the home, the climate, the type of HVAC equipment, and the efficiency of the home. Given these variations, evaluators must consider how well the analysis sample represents the larger target population and how any differences may affect the results.
Support us by checking our Honest and Real Product reviews.
Check Amazon for Reviews and Prices
References
Aarish, C., M. Perussi, A. Rietz, and D. Korn. Evaluation of the 2013–2014 Programmable and Smart Thermostat Program. Prepared by Cadmus for Vectren Corporation. 2015a.
Aarish, C., M. Perussi, A. Rietz, and D. Korn. Evaluation of the 2013–2014 Programmable and Smart Thermostat Program. Prepared by Cadmus for Northern Indiana Public Service Company. 2015b.
Apex Analytics LLC, “Energy Trust of Oregon Nest Learning Thermostat Heat Pump Control Pilot Evaluation”, October 10, 2014 accessed from http://energytrust.org/library/reports/Nest_Pilot_Study_Evaluation_wSR.pdf
Carrier Corp. “TP–WEM01 PerformanceTM Series AC/HP Wi–Fi Thermostat Carrier CôrTM Thermostat” from http://dms.hvacpartners.com/docs/1009/public/02/tp-wem-01pd.pdf
Ecobee, 2015. https://www.ecobee.com/savings/
Meier, A., C. Aragon, B. Hurwitz, D. Mujumdar, D. Perry, T. Peffer, M. Pritoni, “How People Actually Use Thermostats”, in Proceedings of the 2010 ACEEE Summer Study on Energy Efficiency in Buildings, American Council for an Energy Efficient Economy, 2010. accessed from https://www.aceee.org/files/proceedings/2010/data/papers/1963.pdf
Nest Labs, 2015a. https://nest.com/thermostat/saving-energy
Nest Labs, 2015b. https://nest.com/downloads/press/documents/energy-savings-white-paper.pdf
Peffer, T., M. Pritoni, A. Meier, C. Aragon, D. Perry, “How people use thermostats in homes: A review”, Building and Environment 46 (2011) 2529-2541.
Tado, 2015. https://www.tado.com/gb/heatingcontrol-savings
U.S. Department of Energy (DOE), “Uniform Methods Project For Determining Energy Efficiency Program Savings”, 2013.
U.S. DOE 2015 http://energy.gov/energysaver/articles/thermostats
U.S. Environmental Protection Agency (EPA) “Summary of Research Findings From the Programmable Thermostat Market”, 2003 accessed from http://www.energystar.gov/ia/partners/prod_development/revisions/downloads/thermostats/Summary.pdf
U.S. EPA 2015 http://www.energystar.gov/index.cfm?c=heat_cool.pr_hvac
