Hello everyone,
To preface this post, the content you will find below is a rich summary of the testing and history regarding our QC35 Acoustic Noise Cancelling performance and firmware 4.5.2. Our community members spent many hours providing us with feedback and we wanted to make sure we provided a comprehensive and detailed summary on our approach to evaluating your concerns and testing the product to understand if there was an issue.
When we launched QC35 I and II, we set stringent specifications for Noise Cancelling performance. Through all of our investigation and testing, we’re confident that firmware 4.5.2 did not affect the noise cancelling feature. As with any of our products, if you are not satisfied with the performance of your QC35s, please go to your regional support site to review the service options available to you.
With that being said, the newest information on the actions we have taken since our last communication are towards the bottom of this post. If you are interested in our findings from our in-home visits click here. If you would like to see the results of our engineering analysis from the returned Headphones we received, click here. If you are interested in what we are doing about it, click here.
Table of Contents
An Overview of Bose Noise Reduction Technology
Bose QC35 Noise Cancelling Headphones use a combination of both passive and active noise reduction technologies to deliver a high level of performance. This section provides a brief introduction to this technology and describes each of the different components as context for the balance of our findings.
Passive noise reduction is the result of the physical presence of the headphone covering your ears. Things like the fit and sealing of the headphones to the head, the earcup materials, as well as the design and attachment of the earcushions, which include a proprietary silicone gel layer, contribute directly to passive performance on our QC35s. Passive reduction generally increases as a function of frequency, with best reduction occurring at high frequencies.
Feedback cancellation is achieved by the presence of a microphone near the ear inside the headphone. The system reacts in real-time to the instantaneous pressure inside the earcup, generating an opposing signal to cancel out noise. The system works best at lower frequencies.
Feedforward cancellation is achieved by microphones on the outward facing surface of the headphone. This system gives advanced notice of the noise environment before it reaches the ear, allowing the electronic system to predict and generate a matched cancellation signal. This system works best at low and mi frequencies.
Total noise reduction is the combination of all of these systems working together in harmony. If any of these systems are not functioning as designed, the impact on performance can be significant.
Note the terms "Noise Cancellation" and "Noise Reduction" are used throughout. Cancellation always refers to the performance component achieved with active electronics. Reduction doe not indication a specific method, but instead refers to the overall removal of noise, combining feedback, feedforward and passive technologies.
History
How we got here
On January 19, 2017, we released firmware 1.2.9. We saw reports on the Bose Community Forum soon after that some users were having a poor audio experience. Some thought there was an issue with the Noise Cancelling feature. We investigated and found that their QC35s were connecting to the wrong Bluetooth profile, which was causing the poor experience.
In response to that, we released firmware 1.2.10 on February 7, 2017. Following this release, some people were still concerned about the Acoustic Noise Cancelling feature. We took these reports seriously and our engineering team conducted a thorough analysis of all field reports collected, initiated a product sampling program to test customer units, reexamined our software changes, and performed A/B testing using various product development and manufacturing validation test methodologies. We heard your concerns and did everything we could to confirm that the Acoustic Noise Cancellation technology on our QC35 headphones was working the way it should.
Similar concerns resurfaced with subsequent firmware releases post firmware 1.2.10 but the feedback never reached the magnitude we saw during the first release. Each time we always worked with the community to address the concerns and monitored the situation closely. The feedback never reached the magnitude we saw during the first release.
However, this changed on June 12, 2019 with our latest firmware update 4.5.2 , even though no firmware changes were made to the noise cancelling feature. Many community members shared their concern on our forum.
Initial Engineering Evaluation for Firmware 4.5.2 Feedback
Feedback from Bose Community Members
Bose has read every post on the subject and we want to thank all of those who provided important and descriptive feedback of their experiences. We use this information to help replicate customer issues, identify software bugs, modify existing features and also implement new ones.
Here are the most common things we heard from the Bose forum:
- Updating to the new firmware results in reduction of Active Noise Cancelling performance
- Inability to toggle between Noise Cancelling “High”, “Low”, and “Off”
- Some outlets are claiming that there is a change in Active Noise Cancelling
- During phone calls, the Active Noise Cancelling does not work, is broken, or amplifies the world.
- Bose reduced the ANR performance of QC35 ANR to encourage upgrades to Bose Headphones 700.
QC35 headphones are engineered to deliver world-class Noise Cancellation and we would never intentionally downgrade the performance of our products in the field. Our firmware update strategy is focused on improving product performance, adding new features, fixing bugs and maintaining interoperability.
Engineering Deep Dive - Firmware 4.5.2
Bose immediately set up an engineering team to review the software qualification test results and our code to determine if we had unintentionally made changes which could impact our Noise Cancelling feature. During our investigation we:
- Conducted an extensive review of the firmware code (internally and with an outside contractor)
- Reviewed and repeated manual and automated firmware Quality Assurance testing
- Dedicated dozens of hours from our firmware Development and Quality Assurance teams attempting to replicate customer issues
Conclusion
After completing this deep dive into the codebase and validation process, we confirmed that no changes were made to the firmware related to the noise cancelling feature, concluded that none of the firmware changes affected the Noise Cancelling feature indirectly, and that the new firmware passed all our test procedures. We also compared the noise cancelling performance running firmware version 4.5.2 to previous versions on new out-of-box units and existing ones and found no discernible differences.
Customer Headphone Sampling and Testing
Given the history and the millions of units in the field, we continued to monitor the forum closely. As reports of issues with QC35 Noise Cancellation continued to grow into July, we began questioning whether our investigation truly covered all possible scenarios. Therefore, we initiated an effort to directly contact forum members in order to obtain the QC35 headphones they claimed were having issues.
This proved to be challenging and we only were able to get a small number of headphones. Nevertheless, these units were put through a robust test process and were compared against known good reference samples. Tests included the following:
- Controlled A/B Testing on Test Fixtures – Comparing noise cancelling performance between different headphones or under uncontrolled test conditions can be extremely challenging given the number of variables which can impact results. At Bose, we use very precise methods, repeatable measurement sequences and noise environments, and acoustically controlled test fixtures. We can also switch firmware versions back and forth on a test unit without disturbing the setup.
An example of one type of QC35 Test Fixture
- Real-World Performance Measurements on Human Subjects – Noise cancellation performance was measured on multiple human test subjects using tiny microphones placed inside the ear canal. This produces a quantified measurement of the reduction of noise at a user’s ears when wearing and operating the headphones.
- Critical Listener Evaluation – In addition to objective performance measurements, Bose maintains a team of trained critical listeners to judge subjective performance of all audio products. Members of this team performed subjective testing of the headphones in controlled noise environments. Evaluations were performed running each firmware revision in various states (High/Low/Off) of the returned headphones.
Conclusion
This robust evaluation of customer-returned units did not reveal any issues with noise cancellation performance, and Bose was not able to replicate the concerns of our customers. Even still, we valued your feedback and remained determined to continue our investigation to understand why our results did not match the reports from the community.
Extended Engineering Evaluation
In an effort to better understand why our results were not matching the feedback we were receiving from some members on the forum, we began a new set of activities to dig deeper. This next phase of our investigation included 3rd party independent testing, direct customer engagement, and an initiative to get back as many headphones as possible from customers.
3rd Party Testing
To gain an independent assessment, we commissioned a third-party acoustics test laboratory to conduct a set of measurements of the impact of firmware 4.5.2 on noise cancellation performance.
Michael & Associates is the only test lab in the US accredited by the National Institute of Standards and Technology (NIST) National Laboratory Voluntary Accreditation Program (NVLAP) for the purposes of measuring hearing protection.
Without Bose involvement, Michael & Associates purchased 5 new QC35 headphones, running 4.1.3 firmware, to evaluate. The measurements were taken using a GRAS 45CA ANSI Acoustic Test Fixture, a device designed to the requirements of ISO 4869-3 for use in measuring hearing protectors, primarily for quality assurance purposes.
The headphones were placed on the test head, measured first in their initial out-of-box state, and then again after updating the firmware to 4.5.2. The headphones were updated using the Bose Connect app and were not moved or disturbed during the test.
Conclusion
For all 5 headphones, both Left and Right earcups, no loss of noise cancellation performance was measured after updating the firmware to 4.5.2. A small change was observed, with slightly higher performance for the post-update condition, by on average 0.4dBA, an imperceptible amount. Bose engineering does not believe that this increase was due to firmware changes, but rather was an artifact of the test sequence and fixturing, which could not control for all variables.
Test results show no loss of noise cancellation after updating firmware to 4.5.2
In-Home Visits
In the past, when our data did not match the experiences our customers were reporting, we have found tremendous value in learning directly from those customers and experiencing the product in their environment. At times, this can reveal unique findings that could not have been easily replicated in a lab setup.
We created a survey on the Community that screened customers to find those who identified they were experiencing issues with the noise cancellation function, still had the headphones in their possession, and were open to a visit from Bose. We heard from over 150 customers, all of whom were contacted directly by our customer service team. We followed up with 60 of the 150 customers for more in-depth interviews, and then ultimately visited 5 customers for 2-hour in-home sessions.
For these 5 in-home visits, we put together a set of interview questions and activities to try to understand these participants’ experiences better.
Our goals for the in-home visits were:
- Understand specific details of the customer’s experiences using QC35s
- Understand if the customer’s paired device and updating method resulted in a defective update or produced an unexpected device state (for example, the product is stuck in Low noise cancellation mode even if High mode is selected)
- Give Bose engineering a firsthand experience with the customer’s headphones
- Give customers the opportunity to experience a known-good QC35 in their home environment
- Exchange customer’s headphones for a brand-new pair, and bring suspect headphones back to Bose for detailed engineering analysis
During the home visits, Bose representatives – a senior acoustical engineer and a user experience researcher – brought two pairs of QC35s with 4.1.3 firmware (as manufactured). One of these headphones was set aside to be used as the reference. For the second pair, the participants were asked to update the firmware to 4.5.2 using the same method and device that they used when updating their personal QC35s.
While the update process was running, the participant and the Bose researcher engaged in an in-depth conversation about the participant’s headphones, their typical usage behaviors, their firmware updating experience, and their expectations with the noise cancelling function. Each participant was asked to describe in detail the circumstances where they first noticed a change in performance, providing as many specifics to the researcher as possible.
Following the completion of the headphones’ update, both the participant and the Bose engineer completed a series of A/B listening tasks with the three pairs of QC35s:
The Bose team set up a simulated airplane cabin noise environment using a Bose S1 speaker and a sound level meter. This noise environment was chosen for its low frequency content, where the active noise cancellation technology is most effective, making it easier to detect changes with this function. Four use states were demonstrated to the participant to use as anchors for comparison using the reference headphones and the scale below. These anchor points represent the noise levels experienced under the following conditions:
- User is not wearing the headphones
- Headphones are worn on head, but noise cancellation is Off
- Noise cancellation on, set to Low mode
- Noise cancellation on, set to High mode
Note that no audio was played through the headphones during the evaluation.
After the participants were accustomed to the reference headphones and the four anchor points on the scale, A/B comparisons began.
Participants were asked to evaluate the following A/B pairings:
- Reference QC35 (4.1.3) vs. Live Updated QC35 (4.5.2) at High Noise Cancellation
- Reference QC35 (4.1.3) vs. Live Updated QC35 (4.5.2) at Low Noise Cancellation
- Reference QC35 (4.1.3) vs. Participant’s QC35 (4.5.2) at High Noise Cancellation
- Reference QC35 (4.1.3) vs. Participant’s QC35 (4.5.2) at Low Noise Cancellation
Participants alternated between the two pairs of headphones, repeating the process as many times as needed to make an assessment.
An example of the A/B test setup.
Following this listening experience, participants discussed what they heard, and then rated how the updated headphones sounded relative to the reference headphones.
Throughout this session, the Bose engineer also listened to the pairs of headphones and made comparisons but did not share this information to ensure that the participant’s ratings were made independently.
At the end of the in-home visit, we exchanged the participant’s headphones for a brand-new pair of QC35s. The participant’s headphones were brought back to Bose for a detailed engineering analysis to determine if the headphone hardware was still performing according to factory specs.
Conclusion
Based on what we learned, the following items were determined to not be a factor for these customers:
- Updating method
- Device type interactions
- Bluetooth pairing list device quantity
- Problems with headphone fit and seal due to changes in hairstyles, headware, jewelry, etc.
- No observations of unintended mode switching, or locked out controls of noise cancellation modes
The noise environment where a product is used can influence the perception of noise reduction. Some customers did report changes to their noise environments such as changing jobs and working in a new open office or commuting on a different train after moving. One such customer reported hearing more office voice noise while not playing music.
In several conversations with customers, there was some confusion about noise cancellation function during a phone call or other Bluetooth HFP connection. QC35 contains a feature called Self Voice, sometimes referred to as Sidetone, where the audio from the voice pickup microphones is amplified and added into the audio stream. This enables you to better hear your own voice while speaking during a phone call, giving you the ability to monitor the volume of your voice. The result is a more natural conversation, which helps prevent you from talking too loudly. While noise cancellation remains active and unchanged during phone calls, a customer may perceive the added Self Voice audio as a degradation of noise cancellation function, especially when in a noisy environment.
As expected with any subjective audio test, there was some variation in how the participants perceived the noise reduction on the different headphones, but generally, the participants thought that the three headphones all sounded similar. Neither any participant nor the Bose engineer observed noticeable changes in performance after updating the headphone firmware to 4.5.2 live during the visits.
For the participant’s’ own headphones, however, there was one visit where a more substantial difference was observed. Both Participant-2 and the Bose engineer scored this participant’s personal headphones as sounding similar to the reference for Low mode performance while operating in High mode. Switching between Low and High for this headphone produced only a small change in performance, much less than with the reference headphones. Subsequent engineering analysis, covered in the next section, found an explanation for this result unrelated to product firmware.
We found the home visits to be insightful and are always appreciative of the opportunity to learn more directly from our customers. We thank everyone whom we visited and those who volunteered through our survey.
Engineering Analysis of Returned Headphones
In order to maximize the number of samples, we used the interview process described earlier to retrieve additional headphones from community members who had reported having issues and combined these with the exchanged participant units obtained during our visits.
In total, 5 headsets were taken directly from participants, and another 5 headsets were returned to the engineering team through the phone interview process. These 10 headsets were running firmware 4.5.2, as updated by the customers themselves. A reference headphone, with measured average performance, was included for comparison. For these headsets, a series of tests were conducted including:
- Electrical system evaluation
- Mechanical system evaluation
- Acoustic hardware function
- Microphone and driver function and health
- Factory end-of-line testing on the full headphone for both feedback and feedforward noise cancellation systems and music playback performance
- Independent measurements on all contributing systems to noise reduction performance
- Real world noise reduction measurements on human heads in a controlled noise environment
- Bose critical listener evaluation for noise reduction performance
Key findings:
Of the 10 customer headphones evaluated, 8 performed within specification limits. and 2 were found to be out of spec. Subjective critical listening at Bose confirmed that these 2 headphones were noticeably different from the reference QC35, and that the others were performing within the normal range of expected product variation. Measurements of noise reduction performance on human subjects was particularly revealing, showing the following results when measured in High mode:
Test results of total noise reduction performance on the 10 headphones returned by customers, compared to a reference QC35 headset. Lines showing typical performance for High and Low noise cancellation modes are shown for illustration.
Root cause analysis was conducted on the 2 lower-performing headphones, revealing the following issues:
- Participant-2 Headphone
- This headphone had one earcushion that was not fully snapped on (one snap was undone on the Right earcup)
- In addition, this headphone had replaced the original Bose earcushions with 3rd party aftermarket earcushions. These aftermarket cushions negatively impacted passive reduction performance and affected the headphone’s acoustic output to be out of spec on both the Left and Right sides.
- This headphone also had suffered mechanical damage. The acoustic subsystem responses for both Left and Right sides remained out of spec even when Bose authentic earcushions were attached.
- This participant noted that they did not use the case to store and transport their headphones while commuting every day.
- Return-4 Headphone
- This headphone had one earcushion that was not fully snapped on (one snap was undone on the Left earcup)
- Normal performance was recovered when the snap was fully engaged
One snap undone as observed on a customer returned QC35
A Bose authentic earcushion on the left, an aftermarket earcushion on the right. Though the appearance is similar, the aftermarket cushions do not contain the proprietary design features to deliver the acoustic performance and passive noise reduction benefits that authentic Bose cushions provide.
Conclusion
After evaluating customer returns from forum members and through in-home visits, we concluded that some customers’ headphones did demonstrate reduced overall noise reduction performance. However, our full engineering analysis determined that the degradation in all cases was the result of hardware related issues with earcushions, aftermarket parts, or mechanical integrity. Once again, the firmware update 4.5.2 was not found to be the cause of any degradation in overall noise reduction performance.
What are we doing about it
Updated test procedures
Even though we could not attribute any noise reduction performance loss to the QC35 4.5.2 firmware, during our exploration of the issue we did uncover some new use cases and gained additional insights from how customers update our products in the field. As a result, we performed these new tests as part of our investigation and have already updated our firmware qualification test process to include these for all new releases.
Service Videos
We learned a lot from reading your comments on the forum, speaking to those who completed the survey, and visiting some customers. As such, we decided there was an opportunity to supplement our service videos with additional information. Our newest video on our website or our support page on YouTube covers the following topics:
- Headphone Maintenance/Storage – how to properly store and maintain your headphone to prevent mechanical damage.
- Proper Cushion Attachment – it is critical that all 10 snaps on the cushion structure are properly inserted into their respective hooks on the ear cups. Even one single unsnapped cushion can noticeably reduce noise cancellation performance.
- Use of Authentic Bose Replacement Cushions – QC35 replacement cushions are the same ones we use in manufacturing – they’re just packaged differently for service. This means they utilizes our proprietary protein leather and silicone-gel technology, as well as our precision design and manufacturing to ensure a reliable seal when properly installed on a Bose QC35 headphone, thereby delivering the same acoustic characteristics for optimum performance.
- Self-Voice – During a phone call, the headphone uses a Bluetooth mode called Hands-Free Profile (HFP). In this mode, the headphones utilize the Self-Voice feature to improve the communication experience. Under these circumstances, it may appear that noise cancellation performance has been compromised.
Please watch the video below!
BTU Updater Site
During our customer visits, we realized there was some confusion around the version of firmware on a product and the version of the Voice Prompts. Our over-the-air updates via the app are able to update the headphone firmware, but for QC35, it does not update the Voice Prompts. To update to the latest Voice Prompts, customers can use the Bose Updater tool.
To reduce confusion, we have updated our messaging to more clearly explain the difference. We also tested all combinations of firmware and Voice Prompt versions to ensure this didn’t impact our noise cancelling performance and added this to our standard testing procedure for all future code releases.
Downgrade
An important piece of feedback we’ve received from the community during this discussion is that you want the ability to downgrade the firmware on your Bose QC35 product. Today, we’re re-introducing the ability to downgrade firmware QC35 II to 4.3.6 and QC35 series 1 to 2.5.5 via the Bose BTU site for a limited time. Further communication regarding the availability of the downgrade option will be posted to the community at a later date, but if you wish to take advantage of this downgrade option, we advise you to do so as soon as possible.
Just to reiterate, based on our findings and testing to date, we are confident the QC35 firmware does not impact noise cancelling performance. If you are experiencing any noise cancelling-related concerns, please contact us directly if you haven’t done so already.
We strive to continuously improve our products through software updates that fix software bugs, ensure continued interoperability as companies launch new phones and operating systems, enhance existing features, and add new ones. We encourage customers to remain on the current version of firmware and always upgrade when new releases are available to benefit from these improvements.
Self Voice
We are exploring options to allow you to modify how this feature works via the Bose Connect app. If this is an area of interest to you, please stay tuned as we hope to provide an update on feasibility and timing for this work.
Conclusion
We want to thank our wonderful Bose community for sticking with us through this process and reading this post. As we hope you can tell, we take customer feedback very seriously and we’re dedicated to making sure the QC35s you hold in your hand are the best possible product we can make. When you said you disagreed, we listened. We tested. We came to you.
We understand this is a lot to take in and we’re sure that you have plenty of thoughts. And we want to hear them. You can click here to go to a thread where you can discuss our findings and what’s to come with other members of the Bose community.
Thanks,
The Bose QC35 Engineering team
from Hacker News https://ift.tt/2JBFfIj
No comments:
Post a Comment
Note: Only a member of this blog may post a comment.