Consumer crash tests: the elusive best practice

Michael Griffiths, Road Safety Solutions
Michael Paine, Vehicle Design and Research
Jack Haley, NRMA
Australia

Presented at the Symposium Worldwide Harmonization of Crash Test Programs organised by TUV in Cologne, Germany, Dec 2-3 1999.

This paper represents the views of the authors only and does not necessarily represent the views or position of any organisation.

Abstract

Car crashworthiness tests to assist car buyers have gone through many changes since the US New Car Assessment Program (NCAP) commenced in 1978. Australia’s program started in 1992 and has been open to an evolving methodology for tests, their assessment and the presentation of results to consumers and other stakeholders. For a variety of reasons, Australia recently adopted most of the European NCAP (Euro-NCAP) protocols. Whilst current Euro-NCAP protocols have room for improvement, they achieve the main aim of assisting consumers to identify those vehicles which perform better at protecting their occupants in serious crashes.

There is much to be said for the proposition that international harmonisation of consumer crash test programs, with the consequent potential for improvements in credibility should be given high priorityAt the same timewe need to ensure that the assessment methods adopted will guide consumers to those vehicles which offer all road users the best protection from injury.

Introduction

The Australian New Car Assessment Program (ANCAP) commenced in 1992. It was inspired by, and mainly based on, the 56 km/h full frontal test developed by the US National Highway Traffic Safety Administration (NHTSA) as part of its New Car Assessment Program. Ongoing development of the Australian program has been heavily influenced by the European Experimental Vehicle Committee (EEVC) and the US Insurance Institute of Highway Safety (IIHS). During 1999, ANCAP aligned its test and assessment procedures with those of Euro-NCAP,

ANCAP has therefore had experience with most of the crash tests and assessment procedures that are in use today. ANCAP is also in the unique position of being able to provide comment without needing to promote or defend any particular system.

Background

ANCAP had considerable assistance from NHTSA during the set-up phase for its initial program. This included informal advice to create a flexible program that could incorporate better test procedures as they became available. It was clear from the observation of real world crashes that an offset test was required. TUV was doing a test of that nature for the German motoring magazine, Auto Motor und Sport and US and European facilities were in the process of developing such a test. Soon after the EEVC agreed upon the 40% offset frontal crash test, it was added to the Australian program. The original adoption of a crash speed of 60 km/hr was a best estimate, at the time, of what the harmonized speed would be. In 1995 however, following developments by the IIHS, ANCAP increased its offset crash test speed to 64 km/hr. During 1999 ANCAP replaced the full frontal crash test with the EEVC 50km/h side impact crash test.

Over this time the method of assessing the outcome of the crash tests has also evolved. Up until 1996 the assessment was based primarily on the risk of life-threatening injury, derived from Head Injury Criterion (HIC) and chest deceleration. An algorithm for combining the injury risk from full frontal and offset crash tests was developed.

In 1996 a revised assessment method was introduced (Paine 1998a). This was derived from the assessment method of the IIHS in the USA. In essence this method assigned separate ratings for injury measurements, structural performance and restraint performance and combined them into an overall rating. The change was introduced partly due to concerns about the cases where a vehicle was substantially deformed but, more by luck than good safety design, the dummy injury measurements were relatively good.

During this time Euro-NCAP developed its crash test program in consultation with the EEVC (Hobbs 1996). The Euro-NCAP protocols appear to be based largely on existing and proposed European regulations. Unfortunately this meant that the earlier experiences in the USA and Australia were less relevant to the Euro-NCAP approach resulting in different criteria being developed. This has led to some vehicles receiving a markedly different rating under Euro-NCAP compared with IIHS or previous ANCAP procedures. This can undermine the credibility of the programs in the eyes of consumers - a point not lost on some NCAP-sceptical motoring journalists. This is possibly a more acute problem in Australia, where the bulk of the motoring journalists initially seemed to take a hostile approach to the ANCAP (along with most of the motor industry).

If the Australian program had had sufficient funds, it could have based its decision on which tests to conduct on its eight years of experience and knowledge of the options available. However, it also needed to be using a system which would allow vehicle crash tests and results in other countries to be added to its catalogue of results for the use of local consumers.

Over the past few years ANCAP was faced with the decision of continuing with US-derived procedures or changing to the Euro-NCAP protocols which would give it a larger range of vehicles for its test results catalogue. The direction of vehicle regulation internationally, except perhaps in the US, is to adopt the UNECE requirements, as already subscribed to by Europe. There are more Japanese and European vehicles imported to the Australian market than US vehicles so there would also be a greater opportunity to share crash tests results with Euro-NCAP. In 1999 ANCAP made the decision to align its crash test program with Euro-NCAP.

Euro-NCAP Assessment Protocol

Euro-NCAP procedures are set out in several "protocol" documents. The following is a brief outline of the requirements.

Driver and passenger dummy measurements are taken for the head/neck, chest, upper legs and lower legs in the offset crash test Driver dummy measurements are taken for the head, chest, abdomen and pelvis. in the side impact crash test.

Each of four body regions is assigned a score out of a maximum of 4 points. For example, a HIC of 1000, which is poor, would result a head score of zero but a good HIC of 649 would earn 4 points. A "sliding scale" applies for readings between "good" and "poor" so that, for example, a HIC of 825 would earn 2 points. Where more than one injury measurement applies for the body region then the lowest value is used - this includes the passenger scores in the case of the offset crash test.

With the offset test the injury scores are subject to modifiers or penalty points. For example, excessive rearward movement of the steering wheel might result in one penalty point applying to the head score (so an unmodified score of 4 would reduce to 3). These modifiers include airbag stability, steering column movement, A-pillar movement, structural integrity, hazardous structures in the knee impact area and brake pedal movement. In this way some concerns about structural performance are taken into account in the scoring system. Of course, if the injury measurement results in a score of zero then structural performance will have no influence on the overall rating (other than perhaps contributing to the poor injury results). Previous ANCAP procedures meant that a vehicle could receive a double penalty through a combination of bad injury results and poor structural performance. Both aspects were considered relevant and the procedures had the potential to encourage manufacturers to pay greater attention to structural performance - a desirable outcome.

The four body region scores for the offset test are summed to give an offset score out of 16 points. Similarly, the four side impact scores are summed to give a side impact score out of 16 points. The offset and side impact scores are added together to give an overall score with a maximum of 32 points.

The star rating is based on the overall score. For example, an overall score of 22 points would earn 3 stars. If the unmodified injury score for the head, chest, abdomen or pelvis is zero then there is a high risk of a life-threatening injury. In this case a warning note is added to the overall rating (Euro-NCAP uses a "struck through star" to indicate this situation).

There is provision in the Euro-NCAP protocol for a side impact pole test to be conducted at the manufacturer's expense. This only applies where a maximum head score is achieved in the side impact barrier test and a "head protecting" side airbag is provided. The pole test can earn the vehicle an additional 2 points and, if the total score exceeds 32 points, a fifth star.

Comments on Euro-NCAP procedures

Vehicles without airbags

Under the protocol, vehicles without a driver's airbag cannot receive a head score more than 2 points. This could be viewed as design restrictive but is considered productive in order to encourage greater take-up of airbags in countries such as Australia.

The Euro-NCAP protocol currently requires an additional faceform test to be conducted where, with a non-airbag vehicle, the offset crash test HIC is less than 1000 and the head deceleration is less than 88g. This provision is unlikely to apply in the USA or Europe where airbags are almost universally fitted. Unfortunately it appears that 40-50% of new vehicles sold in Australia still do not have airbags fitted as standard so the additional faceform test is likely to be required in Australia from time to time.

For example, in a recent ANCAP offset crash test the HIC was 990 and the head deceleration (3ms) was 91.9g. If the deceleration had been slightly less (88g), ANCAP would have been obliged to conduct a faceform test in accordance with the protocol. Given that the maximum points that could have been awarded was 2 and that a score based on a HIC of 990 would have been essentially zero points, it seems there would have been little point in conducting the faceform test. The test could have been of advantage to the manufacturer, since it may have resulted in a higher score if the faceform test result had been favourable.

Chest Injury

There is considerable variation between IIHS and Euro-NCAP chest compression criteria. In particular the lower limit (4 points) for Euro-NCAP is at 22mm whereas IIHS rates chest compression under 50mm as good. 50mm is the upper limit under Euro-NCAP and is the criterion for zero points.

Figure 1 shows an analysis of chest compression data for 184 offset crash tests (Euro NCAP, IIHS and ANCAP). Only two tests had a driver chest compression under 22mm (21mm in each case). The mean value was 35mm (=2 points out of 4) and the standard deviation was 7mm. This suggests that the Euro-NCAP lower limit of 22mm may be too low, with the unfortunate effect of rating vehicles similarly, with less discrimination between good and bad performers..
NCAP chest compression distribution

Figure 1.  Distribution of chest compression in offset crash tests

This observation is supported by injury research. Ryan (1998) estimates that the probability of a chest injury of AIS 3 or greater is approximately 7% for a chest deflection of 22m. This risk increases to 20% for a chest deflection of 28mm. According to data presented by Ryan , the latter injury risk is comparable to the risk of serious (AIS 3 or greater) head injury risk at a HIC of 650 - the lower limit for HIC under the Euro-NCAP protocol. 28mm is therefore considered to be a more appropriate lower limit for chest deflection. (Note: It is acknowledged that the risk of severe injury (AIS 4 or greater) is generally used for setting upper limits but for lower limits the probability of serious injury (AIS 3 or greater) is considered more appropriate since it offers better accuracy and discrimination)

In the 111 cases where passenger chest compression was known, 61 cases (55%) had greater compression than that of the driver. Under Euro-NCAP protocol the average reduction in chest score due to the use of the (lower) passenger value instead of the driver value was 0.8 points.
 
 

A more difficult issue to address, and probably one that requires additional research, is whether seat-belted occupants in frontal crashes, where there is no major intrusion into the occupant space, are actually receiving a significant proportion of serious chest injuries or not. This could confirm impressions that occupants restrained by seatbelts, with poor configuration or characteristics, are still receiving notable head and abdominal injuries, whereas serious chest injuries are not as frequently observed.

Upper Legs

There have been difficulties in interpreting the two knee impact area modifiers "variable contact" and "concentrated loading". The Euro-NCAP protocol defines a knee contact area based on the point of impact of the knees. This includes "an additional penetration depth of 20mm...beyond that identified as knee penetration in the test". One difficulty is that, with many modern dashboards, the point of maximum knee penetration is not readily evident. This is because the plastic dash material usually deforms or shatters. A better approach may be to define the knee impact area (actually volume) based on defined conditions such as the seating reference point rather than the dynamic points occurring in a crash test.

Lower Legs

Many recent papers have dealt with the relative increase in seriously disabling lower leg injuries in real crashes, compared to head and torso injuries. Automobile manufacturers and others have looked at ways of reducing intrusion, and adding energy absorbing padding, into the lower foot occupant space.

Rearward movement of the brake pedal is currently used as a surrogate for risk of foot injury but this does take account footwell design improvements. The least of the two tibia scores and the brake pedal movement score is used for the lower leg score. The protocol foreshadows the introduction of a lower leg modifier based on footwell deformation (incidentally, the Protocol appears to involve unnecessarily complex measurement techniques compared with those used by IIHS and ANCAP). ANCAP and IIHS experience suggests that footwell intrusion is an important factor (Zuby et al 1996).

A second concern is the lack of a modifier for entrapment of dummy feet by deformed panels or pedals. This factor has applied in several ANCAP and IIHS crash tests.
 
 

Side Impact

There are currently no modifiers in the side impact protocol. This is of concern because there appears to be a wide variation in structural performance and small changes in the crash characteristics could lead to major differences in crash outcomes.

We currently have insufficient experience with side impact tests in Australia to provide specific recommendations but issues such as intrusion and structural integrity should be considered. There are also concerns that some designs of door interior may be loading parts of the Euro-SID dummy that are not instrumented, thereby giving a misleading indication of the risk of serious injury.

Full frontal crash tests

Full frontal crash tests are not included in the Euro-NCAP protocol. Full frontal crash tests are more demanding on occupant restraint systems than other types of crash tests. Between 1993 and 1998 ANCAP conducted both full-frontal and offset crash tests. The rating was based on both tests. In general the full frontal injury measurements for the head and chest were more influential than the corresponding offset results and the leg and vehicle structure assessments in the offset test were more influential than those of the full frontal test. Figure 2 shows the results of an analysis of 41 ANCAP tests, mostly without airbags. Injury risk is calculated from HIC and chest deceleration (Paine 1998a). It is evident that relatively few vehicles with an injury risk exceeding 50% in the full frontal test have a correspondingly high injury risk in the offset crash.
NCAP injury risk - full frontal vs offset crash tests

Figure 2 ANCAP full frontal and offset crash test results

The influence of the full frontal test on the overall rating (under the previous ANCAP system) appears to have diminished with airbag-equipped vehicles but cases have still been encountered where the airbags did not provide reasonable protection in the full frontal crash test. Examples are a 1996 small car with dual airbags (Driver HIC of 1317 in full frontal and 307 in offset, passenger HIC of 1102 in full frontal and 209 in offset) and a 1994 large car with driver's airbag (Driver HIC of 910 in full frontal and 600 in offset, non-airbag passenger HIC 1280 in full frontal and 380 in offset).

These examples might result from early "aggressive" airbag designs and, these days, the full frontal crash test is less likely to show up major differences between airbag-equipped vehicles. For example, NHTSA's listing of 1999 year model cars and passengers vans shows no vehicle with a driver rating less than 3 stars. The Japanese OSA NCAP program, which commenced in 1996 also uses the 56km/h full frontal crash test (Horigome et al 1996). OSA's current listing also shows very little spread of ratings.

In Australia, it was initially suggested by some authorities that if vehicles had to comply with a 48 km/hr full frontal crash test (as required in Australian Design Rule 69) it would result in most being fitted with airbags. Unfortunately, this has not happened, with approximately 40-50% of new vehicles sold in Australia passing the requirement without an airbag fitted. This includes a significant proportion of compact vehicles. Their prospects of offering adequate protection in more severe crashes is questionable, and many have demonstrated poor performance in the NCAP 56 km/hr full frontal crash test. Hence, there is a strong case for continuing with the full frontal crash test as part of an NCAP in countries which still have a high proportion of new vehicles being sold without airbags.

On this note, it is believed that Australia is not the only country in the world receiving vehicles which have downgraded safety systems compared to those built for the North American or European market.

Bearing in mind budget limitations, an alternative approach to full frontal crash tests would be to increase the "penalty", under the Euro-NCAP protocol, for models without a driver's airbag. At present there is effectively a two-point penalty out of a maximum possible overall score of 32 points. Consideration could be given to an automatic penalty of 8 points, which represents one star range. Manufacturers could then be given the option of funding a 56km/h full frontal crash test in order to demonstrate that a non-airbag vehicle deserved a higher rating. This practice has a precedent under the Euro-NCAP protocol for the knee impact rating. Where a model is deemed, by the assessors, to have a "variable knee contact" problem the manufacturer may submit test results to demonstrate that a penalty is not appropriate.

Effectiveness of the scoring system

Consumer crash tests need to identify both the "good" and "bad" performers, so that the best performers can be distinguished from the worst. There is not much use in setting the test requirements so low so that most vehicles appear "good" because they can pass the test, when in fact some vehicles clearly perform much better than others. Starting with the original NHTSA NCAP tests the test speed has generally been higher than that in the corresponding regulation. When the Euro-NCAP program was introduced there was little experience with the side impact test and therefore the test speed was aligned with the proposed legislation (Hobbs 1996).

Table 1 sets out the results of an analysis of 62 crash tests (mostly Euro-NCAP). The relatively small standard deviations with the side impact scores suggest that the test is not discriminating between vehicles. This could be because they all perform well but a more likely reason is that the test is not tough enough to show up differences.

Table 1. Analysis of Euro-NCAP scores
Vehicle Type (N)
Offset Score

Mean              Std Dev.

Side Impact Score

Mean               Std Dev.

Small 4WD (5) 7.29 3.24 - -
Large 4WD (8) 8.65 2.4 - -
Large/medium cars (17) 6.94 3.89 12.3 2.41
Luxury Cars (6) 10.36 1.97 14.56 1.33
Passenger Vans (4) 5.94 5.28 14.84 0.78
Small cars (22) 5.73 3.83 11.71 2.84
Overall (62) 7.03 3.77 12.57 2.6

Concern about the lack of spread of side impact scores, and the weight given to the side impact test in the star rating, is illustrated in figure 3. In particular, the passenger vans (people movers) all have high scores in the side impact crash test but show a very wide range of performance in the offset crash test. This is not unexpected since the side impact test is much less demanding for vehicle configurations where the occupants sit higher from the ground than with conventional vehicles. The same outcome could be expected with four-wheel-drive/sports utility vehicles (none have been tested).

Euro-NCAP star rating from offset and side impact results
Figure 3. Combination of offset and side impact scores to derive a star rating.
Note: Under Euro-NCAP the overall scores are rounded to integers before a star rating is assigned. This process results in the same ranges as those illustrated in the graph. For example an overall score of 16.5 points would earn 3 stars because it is rounded up to 17 points - the lower limit for 3 stars under the protocol.


For passenger vans and four-wheel drive vehicles it may therefore be appropriate to consider an alternative to the side impact crash test. A side impact pole test might be appropriate but a 56km/h full frontal test should also be considered.

Under the current Protocol it would be very rare for a vehicle to be awarded just 1 star. In effect the rating system ranges from 2 to 4 stars. This masks the wide range in performance during the offset crash test and decreases the program’s usefulness to consumers.

The graph also illustrates how difficult it will be for a vehicle to attain 5 stars, even with an extra two points from the optional "pole test".

Offset crash tests and four-wheel-drive vehicles

There have been concerns expressed that the 64km/h offset crash test is too severe on large vehicles such as some four-wheel-drives. The main criticism is that the crushable barrier bottoms out early in the crash and the vehicle structure must absorb a higher proportion of the crash forces than is the case with lighter vehicles. This is only a valid criticism if it is based solely on large vehicle to small vehicle crashes. In country areas of Australia a large vehicle is just as likely to be involved in a head-on collision with another large vehicle as with a small vehicle. Furthermore, collisions with intrusive objects such as trees are common.

ANCAP offset tests of large four-wheel-drive vehicles have demonstrated a wide range of crashworthiness in this class. One vehicle in particular showed very little energy absorption at the front and most of the crash forces appear to have been transferred into the footwell area of the passenger compartment. The legs of the driver dummy were forced so far upwards that large tension forces were recorded in the femur. This result could not be used in the rating because, with a human occupant, the lower legs would have been crushed by compression forces before the femur reached high tension forces. Another concern with this test was that there was an extremely large rearward displacement of the brake pedal and it is suspected that the pedal contacted the groin of the dummy. This would have been a life-threatening injury but, of course, the dummy was not instrumented for such a loading.

Serious fuel leaks have been observed after offset crash tests of four-wheel-drive vehicles with the obvious increased risk of a post-crash fire in an on-road crash.. These vehicles had secondary fuel tanks mounted under the passenger compartment. During the crash the tailshaft moved sideways and caused the leak - in one case a fuel line was severed and in the other a rotating coupling on the tailshaft gouged into the fuel tank. Design changes resulted from the deficiencies revealed in the ANCAP test.

Aggressivity

A further criticism of offset tests is that they are claimed to be forcing manufacturers of large vehicles, including four-wheel-drives, to adopt designs that are more aggressive towards other vehicles. This is not necessarily the case, as is evident from real world crash experience in Australia - some designs which offer good crash protection for occupants also provide comparatively good protection for occupants of other vehicles (Paine et al 1998b).

A similar claim made about the full frontal crash test in regard to the design of passenger vans in the US has been shown by NHTSA not to be valid. A comparison of the stiffness of the fronts of vans over the period of full frontal crash tests of vans by the NHTSA showed no consistent pattern in frontal stiffness between good and bad crash test performers. In fact the better performers appeared to have softer fronts than the less good performers (Park et al, 1999). The paper concluded that-:

  • the maximum dynamic crush and the time period of the acceleration pulse have increased over time. The maximum acceleration of the vehicle structure has decreased over time.
  • the approximate linear stiffness in the first 200 mm of crush has decreased over time, and
  • there is a correlation between lower stiffness and aggressivity parameters and better NCAP scores.
  • Correlation with real world crashes

    There is a need to relate crash test characteristics and outcomes with those of real world crashes. In this way better informed decisions can be made about the future direction of NCAP programs. Key issues that need to be addressed are types of tests to be conducted, test speeds and configurations, number and type of dummies, types of injuries to be assessed and, for the rating system, the relative weight given to various injuries and types of tests.

    Several comparisons have been made between crash test results and injury outcomes in real world crashes. Hackney et al (1996) report on an analysis the impact speeds in real world crashes and a comparison of injury outcomes with those predicted from NCAP tests in the USA.

    Newstead et al (1996) describe an assessment of the correlation between ANCAP results and real world crash data. This included an analysis of injury data from insurance records.

    Whilst these comparisons are a good start they do not allow assessment of the predictive for specific injuries, such as say the head, chestor lower legs, for the different makes and models. Comparisons of this kind will require in-depth studies with good quality injury data.

    Suggested improvements to the Euro-NCAP protocols

    It is acknowledged that there will always be differences of opinion over the relative weight applied to the various factors in the rating system. On the basis of experience with IIHS and ANCAP procedures the following items offer potential areas for improvement in the Euro-NCAP protocol:

  • The offset crash test carries equal weight to the side impact crash test in the overall star rating. There may be a case for giving greater weight to the offset test.
  • Major structural problems such as excessive a-pillar movement, door failure and the like can only total 2 modifier (penalty) points, or 12.5% of the maximum possible 16 points for an offset test. A larger penalty should be considered.
  • Under previous ANCAP procedures less weight was given to the passenger injury results on the basis of the lower occupancy of this seating position. IIHS tests do not include a passenger dummy. The weight given to the passenger score under the Euro-NCAP protocol should be reviewed.
  • Include dash movement as a factor in the offset test. Dash movement has been a serious problem in some ANCAP tests (Euro-NCAP protocol does provide for one penalty point where the dash separates from the a-pillar but this is considered too lenient. The protocol also penalises the head score if there is excessive rearward movement of the steering wheel. A concern is that most steering wheel assemblies collapse when impacted by the occupant, even through an airbag, and so residual movement may not give a good indication of dynamic performance).
  • Improve the objectivity for the assessment of the upper leg modifiers. Define the knee impact area in terms of vehicle reference points rather than knee impact points.
  • Include foot entrapment as a factor in the lower leg rating
  • Include a penalty for the need to use tools to open a door after the offset test.
  • Include a penalty for fuel leaks after either test.
  • Include penalties for restraint concerns such as severe head strikes during rebound, seat belt failures, seat failures and airbag deployment problems (Estep 1996).
  • In order that calculations can be independently verified it is recommended that scores be calculated on the basis of the number of decimal places quoted in the published injury values and not the "laboratory" values.