NYS000179

Submitted: December 16, 2011

United States Nuclear Regulatory Commission Official Hearing Exhibit

Entergy Nuclear Operations, Inc.
In the Matter of:
(Indian Point Nuclear Generating Units 2 and 3)
ASLBP #: 07-858-03-LR-BD01
Docket #: 05000247 | 05000286
Exhibit #: NYS000179-00-BD01 Identified: 10/15/2012
Admitted: 10/15/2012 Withdrawn:
Rejected: Stricken:
Other:
Problem Statement
"On February 16, 2009, Unit 2 entered a 7 day shutdown AOT due to an underground leak in the
condensate storage tank return line. "

Team Members

Team Lead Steve Manzione

Root Cause Qualified Evaluator Anthony DeDonato

P & C Engineering Nelson Azevedo

Robert Lee

K-T Member Lizabeth Lee

EFin Greg Bouderau

Civil Engineering KaiLo

Mechanical Engineering John Bencivenga

Operations Jan Mayer

CA&A Mike Tumicki

Fleet Joe Abisamra

Root Cause Analysis Report • 20f39

IPEC00199052

IPEC00199052
Event Narrative

On February 15,2009, a CR was entered at 1629 EST. An Operator observed water filling the floor guard
collar on the CST return line and spilling onto the floor on the 18' AFB pump area. Operations secured
recirculation of CST -Hotwell. The Chemistry Department was contacted for sampling the spilling fluid.
The Chemist reported 54ppb ofHydrazine in the water, which identified the water as condensate. CST was
declared inoperable on Monday, February 16,2009 at 0205 EST. Unit 2 was operating at 100% Rx
power throughout the event.
The source of the leak was determined to be just outside of the Auxiliary Feed Pump Room (AFPR) based on
observations of the leakage at the pipe collar in the AFPR, the leakage observed from Manhole #5 just outside the
AFPR door, and Engineering experience and judgment of the leak to be near a pipe bend. Work forces were
mobilized and boring through the concrete slab and excavation of the pipes was commenced. The area was full of
water, just below the concrete slab. A vacuum truck was used to remove the water and fine debris from the area.
Larger objects were removed by hand.
Workers reported the material was mostly clay under the floor. There was no evidence of any sand around the
pipes. Some construction debris was also unearthed. Based on the condition of portions of the coating, the through
wall leak and other defects on the pipe, a decision was made to replace the section of the pipe.

Timeline of Major Events

Sunday, 2/15/09 @ approximately 1500 Water was identified leaking into the IP2 Auxiliary Feedwater
Pump Building through a vertical pipe sleeve for the 8 CST return
line.

Sunday, 2/15/09 @ approximately 1600 Chemistry results show 54ppb Hydrazine, which indicates it is
condensate. The FSS contacts the Engineering Duty Manager
(ED M). A conference call is held between Engineering
management and supervisors.

Sunday, 2/15/09 @ approximately 1900 The EDM arrives on site, inspects the pipe sleeve, confirms water
is rising up and requests another chemistry sample for
confirmation of condensate.

Sunday, 2/15/09 @ approximately 2100 Further Engineering inspection reveals water is leaking into
Manhole #5 at two locations through the masonry joints.
(Manhole #5 is located in the FRV Room approximately 5' west
of the underground location of the 8" line.) The Watch Chemist
is instructed to take samples of incoming flows into Manhole #5 .

Root Cause Analysis Report • 30f39

IPEC00199053

IPEC00199053
Event Narrative

Timeline of Major Events (continued)

Monday, 2116/09 @ approximately 0205 The CST was declared inoperable and entered the 7-day AOT.
Chemistry samples confirm the water is condensate in both the
pipe sleeve and the effluents entering Manhole #5. The group
decided to meet at 10:00 AM the following morning.

Monday, 2116/09 @ approximately 1200 Civil Engineering maps out the line location underground and
determines the area to be excavated based on the configuration of
the pipe. Construction is mobilized.

Monday, 2116/09 @ approximately 1900 Core drilling operations commenced in preparation of slab
removal for excavation of the area. First indications that water
saturation is present under the slab. Demolition by jackhammer is
not allowed due to possible undermining.

Monday, 2117/09 @ approximately 0500 Core drilling operations in progress; the first 2' x 2' section of the
slab is removed. Standing water is present under the slab. Three
small sump pumps are needed to keep up with the water.

Monday, 2117/09 @ approximately 1700 Removal of the concrete slab was completed.

Tuesday, 2117/09 @ approximately 2200 Shoring is installed in the excavation; a containment area for
removed soil is set up.

Tuesday, 2117/09 @ approximately 2400 The vacuum truck arrives on site.

Wednesday, 2118/09 @ approximately 0230 Excavation of the site begins.

Wednesday, 2118/09 @ approximately 1800 Chemistry increases the amount of Hydrazine in the Condensate
System and monitors the level ofHydrazine in the area.

Wednesday, 2118/09 @ approximately 2100 Operations calculates the make up to the CST is approximately
17gpm. Chemistry confirms the leak in the 8" line to the CST
based on rising Hydrazine levels in Manhole #5 .

Root Cause Analysis Report • 40f39

IPEC00199054

IPEC00199054
Event Narrative

Timeline of Major Events (continued)

Thursday, 2/19/09 @ approximately 0100 12" pipe is exposed and no leakage is seen.

Thursday, 2/19/09 @ approximately 0330 8" pipe is exposed and the leak is seen at the horizontal section of
the pipe.

Thursday, 2/19/09 @ approximately 0400 Engineering assesses the leak is located at the 5:00 position of the
pipe and determines the pipe is structurally sound to accept a
housekeeping patch.

Thursday, 2/19/09 @ approximately 0600 A full circle clamp with longer bolts is installed over the leak; it
slows it down enough for the sump pumps to keep up with
dewatering, allowing further inspections.

Thursday, 2/19/09 @ approximately 1200 Abatement of the coal-tar coating begins.

Thursday, 2/19/09 @ approximately 2100 Abatement is completed; visual inspection and UT of the line is
started.

Thursday, 2/19/09 @ approximately 2230 Visual inspection and UT of the line is completed. Several areas
of minor degradation were found on the lower elbow and
horizontal section of the pipe. A decision was made to replace the
elbow and damaged section of the pipe.

Friday, 2120 @ approximately 0100 Shop work on the replacement pipe is started.

Friday, 2120 @ approximately 0500 Shop weld is completed. NDE was performed on the weld, MT
SAT.

Friday, 2120 @ approximately 0500 Entered a 72-hour AOT for 22ABFP in order to accommodate the
pipe and elbow replacement and isolated line #1509 .

Root Cause Analysis Report • 50f39

IPEC00199055

IPEC00199055
Event Narrative

Timeline of Major Events (continued)

Friday 2120/09 @ approximately 1200 Pipe and elbow section removed.

Friday 2120/09 @ approximately 1700 Pipe fitted into place. Weld out of two field welds
begins.

Saturday 2121/09 @ approximately 0400 Work completed, NDE performed, MT reading SAT,
Operations commence clearance of PTO and filling line.

Saturday 2121/09 @ approximately 0530 Line verified fill, in service VT -1 leak inspection
performed by Operations; no leaks observed.

Saturday 2121/09 @ approximately 0600ABFP 22 declared operable, exited the 72-hour AOT.

Saturday 2121/09 @ approximately 0630 CST Line declared operable, exited the 7-day AOT. All
work secured .

Root Cause Analysis Report • 60f39

IPEC00199056

IPEC00199056
Event Narrative

Background Information

• Lines 1505 and 1509 are carbon steel, schedule 40. Line 1505 is the 12" supply from the CST to the AFPs.
Line 1509 is the 8" CST return line. These lines were deemed not to require cathodic protection during
original plant design due to favorable soil resistivity and drainage characteristics. As a defense against
localized corrosion attack, however, lines 1505 and 1509 were externally coated with coal tar enamel and
have a coal tar enamel saturated felt overwrap. These pipes are sloped from the CST elevation to the AFPB,
and are each approximately 320-330 feet in length.

• Several months prior, a similar event occurred when water was observed at the same pipe sleeve.
Excavation of the CST lines at 2 locations was in-progress at this time of inspection. The leak was
attributed to groundwater due to the open excavations. No hydrazine was detected. This was based
on Chemistry testing for activity, pH and hydrazine.

• The backfill that was used in this area during original installation contained various size rocks and
other foreign material such as cans and wire. The backfill used was for a light load area.

• Groundwater is suspected to also infiltrate the area.

• Line 1509 does not experience "movement". The buried pipe is installed below the freeze line. Other
than seismic activity, there is no ground movement accounted for in the design nor anticipated to
occur. Thermal expansion is very small due to the small delta T that may be experienced if the water
from the CST was at its high temperature of approximately 100 degrees OF. However, thermal
growth would not cause pipe movement since the pipe is restrained by the ground in the vertical rise
at both ends.

• The area is used as a walkway. If there is significant heavy load on top of the slab and the fill has 3"
to 8" large rocks (confirmed by observers), the heavy load can force the pipe to deflect downward
and pinch on the rock, causing damage to the coating.

• Underneath the 6'x 6' cut hole on the floor slab, the fill was washed out locally.

• AFW building is supported by the foundation and containment building wall. The foundation is into
the bedrock and loss of fill will not affect the load transfer capability from the slab, to the wall, to the
foundation and onto the bedrock. The containment mat foundation is into the bedrock and will not be
affected by the leak.

Root Cause Analysis Report • 70f39

IPEC00199057

IPEC00199057
Event Narrative

Background Information (continued)

• Sections of the bituminous type wrap were discovered damaged some voids around the area of the
leak and 90~ elbows.

• The specific 1989 ASME Code requirement to be met for the buried pipe #1509 is IWA-5244(c)
which requires verification of non-impairment of flow in the non-isolated and non-redundant line to
the heat sink. This specific requirement is met by performance of a pump test which verifies the
ability to obtain and control recirculation flow. This inspection is required to be performed three
times within the 10 year interval. Successful performance ofPT-Q34 in conjunction with the PI-
3Y4A Inservice Inspection services as verification of non-impairment of flow.

• Seismic concerns: based on outside study, historical seismic activities around IPEC from 1974 until
2007 fell into a range of modified Mercalli scale of 2.4 to 3.0. The range of seismic activity at the
plant is less than 3.0 (0.007g) and IPEC is designed to M = 6.5, well below the plant's design ground
response spectrum. The leak location has exterior surface/coating damages that appeared to be
caused by impact from large, angular external object. It is highly unlikely that uniform seismic
ground movement with a buried pipe could cause such a surface impact.

• An Entergy Engineering Fleet Call was conducted on March 4,2009, which discussed the failure of
the CST return line #1509.

CST Operation and Secondary leak detection

The condensate Storage Tank (CST) supplies makeup water to the Condensate System and to the Auxiliary
Feed Water Pumps (AFP) during hot shutdown decay removal via a common 12" underground pipe. The
CST is sized to supply a minimum inventory of 360,000 gals for 24 hrs of decay removal in hot standby
following a plant trip as well as additional inventory for Condensate operation. This 12" supply line first
feeds an 8" common AFW supply header in the ABFP, and then is routed to the Turbine Bldg via LCV-
1158 which will automatically isolate make up to the Condensate System to protect this minimum inventory
for AFW in this 600,000 gal design capacity tank. An 8" return header off condensate pump discharge is
routed underground back to the CST for inventory control. AFPs recirc back to the CST via individual 3"
lines that tap into this 8" Condensate Return header to maintain minimum design pump flow whenever
running AFPs are not feeding Steam Generators.

Normal operational lineup in the Turbine building is as follows, the CST supply valves (LCV-1127 &
l128A) to the condenser hotwells are throttled open in manual control. Condensate System return is
controlled in manual via LCV-1129 which is normally kept shut in warm weather. In the winter, LCV-
1129 is throttled open to maintain CST temperature between 50 and 65°F. Summer CST temperatures can
range in the 80's.

Root Cause Analysis Report • 80f39

IPEC00199058

IPEC00199058
Event Narrative

CST Operation and Secondary Leak Detection (continued)

A mobile unit supplies primary water to U2 via portable demineralizers to the three Ul Condensate Storage
Tanks. From here, water factory Deaerator Booster Pumps supplyU2 condensate makeup to the Drains
Collecting Tank (normal path) or to the U2 CST. Flow to the Drains Collecting Tank (DCT) is adjusted
using Booster Pump (BP) flow (PI-53 1) or Primary Water Flow to the DCT (FIT-lOOOl) meters for rough
monitoring of manual changes.

Operators control secondary makeup by monitoring average Hotwellievel in the Control Room. They
attempt to match BP flow to the DCT with condensate losses. These losses include 100 gpm S/G
Blowdown (or 70 gpm cold), 20 gpm Aux Steam heating (winter), 10-20 gpm condensate leakage and
steam loss. Other than S/G blowdown there are no meters to track process losses. Operators allow CST
level to track down slowly to makeup for the difference between DCT supply and secondary losses

Operations initially estimated the return line loss at l7gpm by shutting LCV-1158 and 1129 and
subtracting the difference in CST rates oflevel change. This assumes no leak by LCV-1158 and no siphon
break in the Return line.

A 10 gpm loss equals 14,000 gallons a day. This equates to a CST level loss of9 in/day or 3/8 inlhr.
This would be hard to detect by monitoring CST level changes because the indicator in the CCR has
intervals of feet, and such small changes cannot be visibly seen unless tracked over a few days. Since the
CST is not the primary means of making up secondary losses, a 10 gpm leak would go unnoticed by
watching CST levels. Hotwellievel is maintained at about 4' or 76,000 gals. At 19,000gal/fi., a 10 gpm
loss equates to about 3/8 inlhr change in hotwellievel. Since operators maintain hotwellievel within a
close band, leak rate changes can't be detected by hotwellievel.

Root Cause Analysis Report • 90f39

IPEC00199059

IPEC00199059
Event Narrative

IPEC Buried Piping and Tank Inspection and Monitoring Program

The IPEC Buried Piping and Tank Inspection and Monitoring Program, hereafter simply referred to as
BPT Program, is under development. The foundational elements of the program have been completed per
scheduled milestones identified in Entergy fleet procedure EN-DC-343, which went into effect on Nov. 19,
2007. The fleet procedure required that all systems having buried portions of piping be included in the
program, including but not limited to those systems that were identified in the IPEC License Renewal
Application (LRA).

Once all buried piping systems were identified, the piping was assessed as having High, Medium or Low
Impact, based on the consequences of a failure of the piping in the following areas:

• Safety (High = Nuclear Safety Related; Medium = Augmented Quality/Category M; Low = non-
safety related)
• Public risk (High = potential radiological consequence; Medium = environment discharge or
hazardous fluid; Low = non-contaminated, non-hazardous fluids)
• Economic impact of equipment failure on plant operation (High = >$lM; Medium = $lOOK-
$lM; Low = <$100K.)

Table 1 presents the details for performing the impact assessment.

Using the impact assessment results, the High Impact systems are Corrosion Risk assessed with
consideration of the following four (4) factors:

1. Soil resistivity
2. drainage
3. material
4. coatings/cathodic protection.

Tables 2 and 3 present the details for performing the corrosion risk assessments. Corrosion risk
assessments were performed sequentially for the High, Medium and Low Impact system .

Root Cause Analysis Report • 100f39

IPEC00199060

IPEC00199060
Event Narrative

IPEC Buried Piping and Tank Inspection and Monitoring Program (continued)

In conjunction with the corrosion risk assessments, the inspection priorities for performing the initial
inspections and subsequent inspection intervals were determined for each buried piping system based on the
results of the Impact and Corrosion Risk assessments. Table 4 provides the guidance for scheduling these
inspections.

Buried pipe inspection parameters will include:

• External pipe coating and wrap condition

• Pipe wall thickness
• Cathodic Protection effectiveness (if applicable)

Current plans are for a Central Engineering Programs document for buried pipe and tanks be developed
(target issue by end of 2009), and for each site to manage its buried pipe activities (surveys, excavations,
inspections, etc.) using IDDEAL Scheduleworks, or similar software.

IPEC License Renewal Application (LRA) commitments for Buried Piping and Tanks Program:

IP2 Commitment NL-07-039 Sept. 28, 2013

IP3 Commitment NL-07-l53 Dec. 12,2015

The excavations and inspections of the IP2 AFW lines to the CST were performed in response to the ISE
Panel recommendation to complete same by the end of 2008 .

Root Cause Analysis Report • 11 of39

IPEC00199061

IPEC00199061
Event Narrative

Utility Experience

Most utilities are in the process of early Buried Piping and Tank Program development. Some (as IPEC, in
2005) have received INPO AFI's for non-functioning Cathodic Protection systems for their buried piping.
At IPEC, however, Cathodic Protection systems are generally not provided for buried piping systems
(exceptions being the sewage treatment pipeline, and underground diesel fuel oil lines). Resolution of the
IPEC API is focused on correcting deficiencies in the installed cathodic protection system.

According to the Unit 2 and Unit 3 USF ARs, the basis for not providing cathodic protection systems for
buried piping was an engineering study performed during original licensing of the plants. Determinations
of the soil resistivities at locations away from the river were concluded to be sufficiently high to preclude
the need for cathodic protection for buried piping. The study recommended the application of protective
coating to prevent local corrosion attack. Based on recent resistivity testing, the original resistivity
determinations remain consistent.

An EPRI guidelines document (1016456) for an effective buried pipe program was issued in Dec. 2008.
Future revision to the document is planned and will include:

• Buried tanks
• Non-metallic pipe

Industry initiatives are being taken to identify underground piping assessment technologies to perform
assessments of coating integrity/condition, to identify degraded pipe locations and to quantify associated
wall loss.

The EPRI buried piping guidance document has identified several methods that are used in the gas and oil
pipe lines industries, but has not endorsed them.

Root Cause Analysis Report • 120f39

IPEC00199062

IPEC00199062
Event Narrative

Table 1 - Impact Assessment

High Medium Low

Safety (Class per Augmented QP and Fire

Safety-related Non-Safety related
EN-DC-167) Protection

Radioactive
Chemical/Oil-Treated Untreated Water, SW,
Public Risk Contamination e.g.
System Gases Demineralized Water
Tritium

Economics (Cost of buried

> $1 M or potential
equipment failure to the $IOOK - $IM < $IOOK
shutdown
plant)

Notes: 1. Any buried section with at least one High Impact gets an overall High Impact rating.

2. Any buried section with no High Impact rating but at least one Medium Impact rating
gets an overall rating of Medium Impact.

3. Any buried section with all Low Impact ratings is to be rated as Low Impact.

Root Cause Analysis Report • 13 of39

IPEC00199063

IPEC00199063
Event Narrative
Table 2 - Corrosion Risk Assessment

Soil Resistivity
Corrosivity Rating Soil Resistivity Risk Weight
Q-cm (Note 1)
> 20,000 Essential Non-Corrosive 1
10,001 - 20,000 Mildly Corrosive 2
5,001 - 10,000 Moderately Corrosive 4
3,001 - 5,000 Corrosive 5
1,000 - 3,000 Highly Corrosive 8
< 1,000 Extremely Corrosive 10
Drainage Drainage Risk Weight
Poor Continually Wet 4.0
Fair Generally Moist 2.0
Good Generally Dry 1.0
Material Material Risk Weight
Carbon and Low Alloy Steel 2.0
Cast and Ductile Iron 1.5
Stainless Steel 1.5
Copper Alloys 1.0
Concrete 0.5
Cathodic Protection Coating CP/Coating Risk Weight
NoCP Degraded Coating 2.0
NoCP Sound Coating 2.0
NoCP No Coating 1.0
Degraded CP Degraded Coating 1.0
Degraded CP Sound Coating 1.0
Degraded CP No Coating 0.5
Sound CP Degraded Coating 0.5
Sound CP Sound Coating 0.5
Sound CP No Coating 0.5

Note: Soil resistivity measurements must be taken at least once per 10 years unless areas are excavated
and backfilled or if soil conditions are known to have changed for any reason .

Root Cause Analysis Report • 140f39

IPEC00199064

IPEC00199064
Event Narrative

Table 3 - Corrosion Risk Tabulation

Corrosion Condition Risk Weight Points

Soil Conditions
Resistivity 1 - 10
Drainage 1-4

Materials
Material 0.5 -2

Component Protection
Cathodic Protection / Coating 0.5 -2

Final Corrosion Risk Tabulation

Multiply all weights together in Steps 5.5 [2] (a) thru (d) 0.25 - 160

Corrosion Risk: High 61 - 160 points

Medium 30 - 60 points
Low 0- 29 points

Root Cause Analysis Report • 150f39

IPEC00199065

IPEC00199065
Event Narrative

Table 4 - Inspection Intervals vs Inspection Priority

Impact - Corrosion Risk Inspection Priority Initial Inspection Inspection Interval

(Years) (Years)
High-High High 5 8
High-Medium High 5 8
Medium-High High 5 8
High-Low Medium 8 10
Medium-Medium Medium 8 10
Low-High Medium 8 10
Medium-Low Low 10 15
Low-Medium Low 10 15
Low-Low Low 10 15

Notes: 1. High priority initial inspections shall be scheduled within 5 years. Subsequent
high priority inspections shall be scheduled within 8 years thereafter.
2. Medium priority initial inspections shall be scheduled within 8 years.
Subsequent medium priority inspections shall be scheduled within 10 years
thereafter.
3. Low priority initial inspections shall be scheduled within 10 years. Subsequent
low priority inspections shall be scheduled within 15 years thereafter.
4. Regardless of the above inspection schedule (reference EN-DC-343),
compliance with IPEC LRA commitments prevail.
5. Once initial inspections are performed and conditions before known, a re-
prioritization may maintain, decrease or increase a component future
inspection priority.

The CST line (#1509) was assessed per EN-DC-343 to be a "High" inspection priority .

Root Cause Analysis Report • 160f39

IPEC00199066

IPEC00199066
Event Narrative

Previous Inspections of CST 8" Line

As a result of the Indian Point Independent Safety Evaluation (lSE) Report dated July 31, 2008, the following
recommendation (R-7) was issued via LOCR IP3LO-2008-00151, CA-19: "to explore options for reducing the
vulnerability of buried piping to the occurrence of any future unanticipated leak. Such options include excavating a
few selected locations to confirm the presence of protective coating on the piping, as well as to measure and confirm
the existence of sufficient wall thickness of the thus exposed piping using existing inspection techniques." Two
areas of the Unit 2 CST lines were selected for inspection. The following information was retrieved from CR IP2-
2008-04754.

The three CST pipes (Aux Feed Pump supply, CST return and CST overflow) were exposed at two locations for
approximately 10' piping runs each.

The three inspected lines were: 12" Line 1505, AFP Suction line,
8" Line 1509, Condensate supply to the CST
10" Overflow Line (no line number assigned, corrugated metal pipe to Manhole #5

Upper and lower holes were excavated. An inspection in the upper hole identified five areas which required coating
repair. UT thickness measurements were also performed on those areas where the base metal was exposed and these
inspections confirmed that the pipe thickness remains at nominal thickness (i.e. within the manufacturer's
tolerance). All of these activities were performed under WO 164495.

The visual inspection of these pipes at the lower excavation revealed that they were in generally good condition,
with the coating intact and in acceptable condition. A minor coating repair was required at one location on 8" Line
1509 and the 10" overflow line required repair at the top portion of the pipe at the crests of the corrugations,
possibly indicative of coating damage during the digging. Based on the results of these pipe visual inspections (at
the upper and lower holes), and the coating repairs performed, there was no evidence of any significant pipe
degradation that would warrant the re-inspection of these pipes at the same locations. Future inspection of these
lines will be performed at different location(s) along their length. The scheduling of the future inspections will be
controlled under the IPEC Buried Pipe Program. Specifically for the CST lines, CA-5 of CR IP2-2008-04754
reads, "Given the results of the leak in the AFP Building, determine if the scope and/or frequency of future buried
condensate lines should be modified. This should cover both IP2 and IP3." See elevation drawing on the next
page.

Completed assessments of these lines determined these lines to be of HIGH impact (two lines are safety related),
MEDIUM corrosion risk and HIGH inspection priority. The HIGH inspection priority results from the safety
function performed by Lines 1505 and 1509. The pipe material, soil resistivity and site condition factors result in
these lines being of medium corrosion risk. Accordingly, with the initial inspection of these lines performed in
OctoberlNovember 2008, re-inspection of these lines is required within eight years, or by September 2016 .

Root Cause Analysis Report • 170f39

IPEC00199067

IPEC00199067
 ~ 1..-0 1~ t.-e - Zoot:; ~ I'::> I o~ ~l '1
o
o
..... Lt~~.s It.lSfUn;:p A1fAoti Z
Iz..~ I.AN'E:" ISoS z/z.
gK /.-lr-lfr [Soq
10" OV€:«.fWW l.,JtJt;

~'
P-n

1 I

-it I~·el.,

+---~"1J1..4:

• ~,4."r'7r,. .
• '/i!"V!41. II..

, •Fi../tT .1M:>·
W • LlNE'*lfii
-'¥II N,l;;OViUI;
:X
4
e:t',
~----~"

PROFILE LOOKING NORTH

li(.1.0\'0'
(F-'lU.Cj,3)

"'U
m
()
""0
m a
() a.......
o
o <D
<D
<D
<D a(J)
o
(j)
00 ex>
Event Narrative

The configuration of the underground structures in the area is laid out as follows:

The ABFP Building column wall #22 is a solid concrete wall anchored directly to the bedrock that spans from the
east wall to the west wall. The ABFP Building wall begins at approximately the 12' elevation (6' below grade) and
is connected to the containment shield wall on the east end and the ABFP shield wall on the west end. The column
wall #21 is a solid wall anchored directly to the bedrock. The east wall running north to south does not penetrate
below grade. The west wall running north to south does not penetrate below grade but the structure for Manhole #5
is located in this area. There are two concrete slabs under the pipes in this area. One is the pad for the stairs and
the other is believed to be related to Manhole #5. They are between 4-8" below the pipe. Based on this, the area
can be described as being similar to a "bath tub" that drains slowly.

The slab construction in this area is approximately 6-8" thick and was poured as a monolithic slab on grade with
sporadic reinforcement installed. There was one area that had what appeared to be 6 x 6 welded wire mesh. But it
was not consistent throughout the slab and was not heavily supported when the slab was originally poured. This
arrangement would not be unusual for a walkway design.

Once excavation began, it was apparent that there was significant amount of standing water beneath the slab and
the backfill was saturated. The backfill also had significantly reduced in volume and was not supporting the slab in
this area. The gap between the slab and top of the backfill was approximately 6" to 10", covering an area
approximately 8' x 8'. There also was an area that appeared to be washed out measuring 2' x 2' and greater than
6' deep.

The backfill in the area contained rocks measuring up to 8" and other debris such as several aluminum cans and
other debris. The large rocks were found throughout the excavation area and a concentration was found closer to
the pipe, especially in the area of the 2' x 2' sinkhole. The debris and rocks would hamper the achievement of
proper compaction of the area due to the creation of voids and an increase in the amount of un-compactable
material.

The configuration of the underground structures that encompass the area allows some ground water to collect due
to run off from the north hill and surrounding area. The elevation of the 8" pipe is estimated to be at or just above
the groundwater table area; this is based on the surveys of the installed test wells in the area. Some groundwater
infiltration has been seen leaching into the bottom of the excavation site, creating a wet environment around these
lines.

The 12" line (#1505) and the 8" line (#1509) are carbon steel, schedule 40. These lines were deemed not to require
cathodic protection during original plant design due to favorable soil resistivity and drainage characteristics. As a
defense against localized corrosion attack, however, lines 1505 and 1509 are externally coated with coal tar enamel
and utilize a coal tar enamel saturated felt overwrap for further protection of the coating.

It is a general coating practice to pull a glass matt into the hot enamel as reinforcement, and the outer side of the
coating is saturated asbestos felt. The same corresponding lines at Unit 3 are above ground, insulated and heat
traced, except outside the ABFB as described on page 35.

The bituminous coatings, as a class, are the most widely used protective media. These classes include asphalt
enamels, the greases or waxes, and other mastics which consist of an asphalt or coal-tar base plus an inert binder.
Of the large group of coatings tested under the joint effort of the National Bureau of Standards and the American
Petroleum Institute, concluded that asphalts and coal-tar enamels were the best.

Root Cause Analysis Report • 190f39

IPEC00199069

IPEC00199069
Event Narrative
Coal-tar coatings have been used for over 100 years to protect ferrous metals against underground corrosion. In
1913, an early form of coal-tar enamel was used in protecting gates, locks and penstocks of the Panama Canal.
Examination after 35 years of service showed them to be in perfect condition. Coal-tar pitch is almost completely
inert to moisture and soil chemicals. Coal-tar coatings and coal-tar pitch used as pipe coatings and for
waterproofing have been dug up after 20 to 50 years of service underground. Coal-tar pitch does not absorb any
appreciable water and is not affected to any appreciable extent by soil bacteria. These properties make it eminently
more suitable for waterproofing and coating of buried steel pipe lines to protect them from the corrosion action of
wet soil. If properly applied, the coating should be able to protect the lines in there currently installed environment
and is within the life span of the protective coating.

Flow accelerated corrosion (FAC) was not part of the failure mechanism as described below.

Flow accelerated corrosion of carbon steel in water occurs due to the dissolution of the normally protective
magnetic film that forms on the surface. (Mechanical removal, i.e. cavitation-erosion, does not normally occur
under FAC conditions.) This corrosion was outside to inside on the pipe, and temperature attributes for FAC are
generally between 212 of and 572 of.

The leak in the condensate piping was caused by external corrosion. Patterns of corrosion on the piping and
observations of the backfill indicate that the corrosion on the pipe likely occurred at localized areas of coating
damage that occurred during installation of the pipe. In comparison, other corrosion found on the removed elbow
which affected a larger area but it was not as deep is more typical of corrosion related to difficulties in applying a
good quality coating on more complex surfaces such as elbows and other fittings. For additional details on the
failure analysis, see Structural Integrity Associates (SIA) Report No. 0900235.402 .

Root Cause Analysis Report • 200f39

IPEC00199070

IPEC00199070
Event Narrative

Conclusions

From the information gathered, the event was caused by the failure of the protective coal-tar epoxy coating that was
applied at the time of original construction. Based on historical date, the coating, if properly applied, remains
undamaged is sufficient to provide corrosion protection of the pipe. This is based on outside studies and the results
of the previous inspection and analysis performed in 2008.

The coating failure was a direct result from the installation and type of backfill.

The eventual location of the "through wall leak" on the straight horizontal pipe at this location was due to a
localized coating failure in this area that made it the most susceptible area to degrade once the mechanism for
corrosion started. There were other localized areas on the straight pipe section and 90 0 elbow that were in a very
advanced state of corrosion, also would have, given more time, would have eventually produced additional leak
locations.

The ground water infiltration, soil composition, and the location of the leak at the lowest point of the system were
all contributing factors to the leak developing in this area .

Root Cause Analysis Report • 2lof39

IPEC00199071

IPEC00199071
Root Cause Evaluation

The Direct Cause (DC) was a through-wall defect in the CST return line located below grade in the ABF Building.
There is evidence that the pipe coating had degraded allowing corrosion to eventually penetrate its way through the
pipe wall.

A. ROOT CAUSE(S)

1. RC I - The Root Cause (RC-1) is the installation specification 9321-01-8-4 in effect at the time of plant
construction. There is evidence that sections of the pipe coating were damaged by rocks that were
present in the backfill for the CST lines. The pipe coating material is fiber-based saturated with coal-
tar. The material is then applied to the pipe. Since it is a fiber, the coating is susceptible to damage
from the various size rocks found in close proximity to and in some cases, up against the pipes
themselves.

The rocks present in the backfill caused coating degradation in some areas of the pipes, making the
pipes susceptible to external corrosion. It is evident that soil conditions influence the corrosion rate on
those sections of pipe where there is coating degradation. For example, in the sample inspection holes,
some coating degradation was found accompanied by minor surface rust. Ultrasonic Testing (UTs) of
these areas found virtually no pipe wall loss. The soil conditions in the sample inspection holes were
mostly dry, with minimal ground water present. By contrast, the soil in the failure location was found
to be moist, which is consistent with the water table in the area estimated to be at the eight to ten foot
elevation. The absence of gravel and sand surrounding the pipe promotes wicking of the ground water
through the soil which also contributes to the moist conditions in the area. Therefore, the combination
of degraded pipe coating and high water table served to accelerate corrosion of the failed pipe .

Root Cause Analysis Report • 220f39

IPEC00199072

IPEC00199072
Root Cause Evaluation

B. CONTRIBUTING CAUSE(S)

CC l - The water table in the area of the leak is between eight to ten feet with the pipe elevation at
approximately ten feet. The backfill specification did not specify the use of clean sand and gravel under the
pipe that would have limited the wicking of the ground water to the soil surrounding the pipes. This kept
the soil in the area moist, and at times wet. These soil conditions would find its way into defects in the
coating causing corrosion external to the pipe.

CC2 - The inspection techniques used to preemptively detect underground pipe through wall leaks was
ineffective. Buried Piping and Tank Inspection and Monitoring Program does not identify the low point of
a pipe line as a suggested sample test point. The procedure inspection locations are based on risk and
impact assessments, ease of access, limitations of inspections and ability to isolate lines .

Root Cause Analysis Report • 230f39

IPEC00199073

IPEC00199073
Root Cause Evaluation
C. ORGANIZATIONAL AND PROGRAMMATIC WEAKNESS EVALUATION:

The team performed an Organization and Programmatic Issues Review in accordance with Attachment 9.5 of
EN-LI-118.

1. Organization and Programmatic Issues for Installation Concerns

OP2J - Is there evidence that personnel exhibited insufficient awareness if the impact of actions on
safety and reliability?

Yes. The drawings and specifications allowed the use of material (already available on site) from
blasted areas as fill for this buried pipe.

OP4A - Is there evidence that there are insufficient details in a procedure to perform the task?

Yes, at the time of installation, UE&C Specification No. 9321-01-8-4, Placing and Compaction of
Backfill, was used as backfill guidance. This spec only stated the following, "Place fill in 12" layers
and compact with tamper in small areas or by dozer or trucks in open areas. Top 12" shall be clean
and compacted as noted on drawings." Drawing 9321-F -1002 states to "top off with gravel."
Drawing 9321-F-I024, Containment Building Backfilling and Grading North and East Side, has a note
which reads, "Surface of fill to be random size blasted rock from Unit 3 excavation."

Presently for Unit 2, Con Edison Specification 02200 governs excavation and backfilling. For Unit 3,
today's specification is UE&C Specification 9321-05-8-4, Placing and Compaction of Backfill, which
is very similar to the original Unit 2 specification.

OPSE - Is there evidence of inadequate job skills, work practices or decision making?

Yes, there could have been the potential for poor job skills and work practices when applying the pipe
coating, because the fill used was a contributor to the coating damage that was observed.

All of the above identified issues occurred over 30 years ago, and there are corrective actions to
address them. Present equipment and construction specifications and quality control/assurance review,
the Engineering modification process would minimize these occurrences from repeating themselves.
Interim corrective actions will be issued with this report to address these concerns.

2. Organization and Programmatic Issues for an Inadequate Buried Pipe Program (EN-DC-343)

OP4A - Is there evidence that there are insufficient details in a procedure to perform the task?

Yes, EN-DC-343, Buried Piping and Tank Inspection and Monitoring Program does not identify the
low point of a pipe line as a suggested sample test point. The procedure inspection locations are based
on risk and impact assessments, ease of access, limitations of inspections and ability to isolate lines.

A corrective action to address this issue would be for the Entergy Fleet team to assess the addition of
this point in the procedure, EN-DC-343 .

Root Cause Analysis Report • 240f39

IPEC00199074

IPEC00199074
Root Cause Evaluation
D. Safety Culture Evaluation

The root cause and contributing causes were reviewed against EN-LI-1l8, Attachment 9.6, Safety Culture
Evaluation. It was determined that 12 of the 13 impact areas were not applicable to the causes identified
within this root cause analysis. However the Human Performance portion of the Safety Culture was
impacted because complete, accurate and up-to-date Design Specifications were not in place during original
backfill and supervisory and management oversight of work activities and contractors were not effectively
implemented to prevent inappropriate backfill practices.

E. Equipment Failure Evaluation - see Attachment.

Root Cause Analysis Report • 250f39

IPEC00199075

IPEC00199075
Generic Implications
Extent of Condition Review

As previously stated in this evaluation, the condition of the CST return line through-wall degradation and eventual
leakage, was corrosion of the carbon steel piping material caused by coating imperfections and soil/ground water
conditions.

Based on the cause of the corrosion and a review of the IPEC Buried Piping Program, it has been determined that
all buried coated carbon steel piping could be susceptible to the same corrosion mechanism since the same materials
and construction practices used to install the CST return pipe could have been used in other systems. Although all
buried piping could be susceptible to external corrosion, recent inspections and operating experience indicate that
piping buried at the lower site elevations could have a higher susceptibility because of the closer proximity to the
ground water. This was confirmed by the Fall of 2008 excavations which indicated that piping with areas of
degraded coating experienced essentially no degradation other than minor surface corrosion. On the other hand, the
auxiliary steam pipe between IPI and IP3, and the IP2 condensate return lines, experienced significant degradation
in areas where the protective coating or insulation was either missing or degraded. In the case of the auxiliary
steam piping, the incorrect type of insulation was installed (water retentive vs. water shedding). Both the IP2 CST
return line and the auxiliary steam piping were buried under the 15' ground level while the two excavated locations
were at the 60' + elevation.

Unlike IP2, the piping at IP3 from the CST down to the transformer yard is supported above grade, heat traced and
insulated.

The impact and risk assessments performed for the Buried Piping Program identified the following systems having
buried piping that are deemed high priority for inspection:
IP2 IP3

City Water City Water

EDG Fuel Oil EDG Fuel Oil
Service Water Service Water
CST Piping CST Piping
Aux Steam
Steam Generator Blowdown

Root Cause Analysis Report • 260f39

IPEC00199076

IPEC00199076
Generic Implications

Extent of Condition Review (continued)

Those systems that are assessed to be medium inspection priority, have been scheduled in a long-term inspection
plan, with a delayed first inspection (increased from 5 to 8 years) and longer interval until the second inspection
(increased from 8 to 10 years).

Similarly, buried tanks of high inspection priority include the EDG fuel oil storage tanks for both IP2 and IP3.
However, these tanks and associated piping are enclosed in concrete vaults and were provided with an engineered
backfill and therefore, have reduced exposure to the environmental conditions that are associated with the failed
lines and no radiological consequence of failure.

Extent of Cause Review

The root cause of this event is attributed to backfill specification No. 9321-01-8-4 used for the buried CST pipe.
The specification did not provide limits regarding the type of fill, including rocks size that could be used. Other
backfill specifications in effect at the time of construction were also considered. These include backfill for the
circulating water piping and de-icing line and around the containment building. A review of these specifications
found them to be adequate for their applications.

Other installation specifications associated with Systems, Structures and Components (SSC) important to safety
was also considered for deficiencies. These installation specifications were found to be not applicable based on
years of operating experience and the plants surveillance test, predictive and preventive maintenance and corrective
action programs. These programs are effective in the prediction and identification of equipment and component
problems before they become self revealing.

Backfill specifications do not fall into the same category as the specifications mentioned in the previous paragraph.
Inspections of buried components is presently limited through code based piping and tank tests. The IPEC buried
piping program will address this issue of underground pipe inspections.

Proposed Corrective Actions (GIl)

As stated above, a number of systems have buried carbon steel piping in locations where the soil conditions could
be conducive to external corrosion.

Discussions with buried piping inspection vendors as well as EPRI and other utilities indicated that there are several
methods which can be used to assess the condition of buried piping and/or the condition of the protective coating.
The three most effective and most widely used methods are discussed below:

1. Guided Wave UT (GWUT) - This approach has been demonstrated by EPRI and by GWUT vendors to be
an effective tool to identify areas of concern. For GWUT, an inspection collar is installed in an area of the pipe with
known thickness. The GWUT collar then sends torsional and longitudinal sound waves down the pipe to compare
the continuous cross sectional area of the pipe relative to the average cross sectional area under the collar.

Root Cause Analysis Report • 270f39

IPEC00199077

IPEC00199077
Generic Implications
Proposed Corrective Actions (GIl) (continued)

This provides a continuous assessment of the remaining pipe cross sectional area for pipe lengths up to several
hundred feet, depending on the number of pipe fittings such as elbows located in the inspected area. In addition to
the remaining cross sectional area, the GWUT also assess the uniformity of the cross section. This allows the
inspector to determine whether any wall loss is uniform around the entire pipe circumference or whether it is
predominantly concentrated at a location around the circumference. This inspection tool has been demonstrated to
be effective in confirming the absence of corrosion and locating areas for additional inspection. However, since the
GWUT does not provide actual pipe wall thicknesses, it can not be used to perform structural evaluations or assess
piping operability.
2. Coating Assessment - This assessment technique uses electrical current to locate coating holidays. This
method sends electrical current down the pipe from a given location (usually at the location where the pipe
penetrates the ground) and then the inspector uses a special probe to walk the area above the pipe, to locate areas of
stray current. The location and magnitude of the stray current areas provides an indication of the location and
extent of coating degradation. Similar to the GWUT, this inspection technique is effective in locating areas for
further evaluation but does not provide actual pipe wall thickness measurements to support structural evaluations.
Therefore these inspection results can not be used to assess piping operability.
3. Direct OD Surface Inspection - This approach is the most effective in assessing the condition of the
coating and also provides access to straight beam UT which allows for direct measurement of the remaining pipe
wall thickness. However, this approach requires excavation ranging in depth from a few feet up to 20 or 25 feet,
and has the potential for causing damage to the pipe and/or pipe coating during the excavation itself. This approach
requires extensive resources and can not be used when the outside temperature has the potential to result in
freezing. This approach is best used in combination with either GWUT or coating inspections when areas of
concern have already been pre screened.
The recommended locations to be inspected are:

1. IP2 8" Condensate Return Line in the excavated area in the FRV Room.

2. IP2 12" Condensate Supply Line in the excavated area in the FRV Room.

3. IP2 24" SW Line 408 in the Transformer Yard outside the PAB where it exits the ground.

4. IP3 12" Condensate Supply Line outside the Auxiliary Feedwater Pump Building where it goes
underground.

5. IP3 8" Condensate Return Line outside the Auxiliary Feedwater Pump Building where it exits the
ground.

6. IP3 24" Line 408 in the backup pump valve pit.

NOTE: If any of the above area(s) are not accessible, substitute area(s) may be selected by IPEC
Engineering.

Root Cause Analysis Report • 280f39

IPEC00199078

IPEC00199078
Previous Occurrence Evaluation

Internal

A search of internal operating experience data was performed to determine if the same or similar conditions had
previously occurred at IPEC or other Entergy sites. No CR origination date restriction was placed on the search.
The search included IP2 and IP3. CR descriptions were searched for the words "buried" or "underground." The
searches resulted in 89 hits that were related to CAT "A" or CAT "B" CRs. In addition, 11 CAT "c" hits in
HQN, NOE and LAR CRs were reviewed. The causes and corrective actions from pertinent CRs in the search
results were considered during this root cause evaluation. Attachment V contains a list of the pertinent Entergy
CRs.

A search was also performed for all IPEC CAT "c" CRs that contained the words "buried" or "underground." The
search returned 100 hits. The CRs were reviewed and one CR of note was CR IP3-2007-01852. The CR
documented a steam leak on an 8" auxiliary steam line. This past event was determined to not be similar in that the
issue was the original coating of the steam line being incorrect for the application. See additional information
regarding this leak below.

Unit 1 to Unit 3 Aux Steam Buried Pipe Leak - On April 7, 2007, CR IP3-2007-01852 was initiated stating,
"There are large underground steam leaks just went and north of the U3 ETA tank moat. Two separate leaks exist,
one on the north comer of the moat and the other about 10' west of the moat. The pavement around these leaks is
very soft and may impede transformer replacement/work if heavy equipment is needed to travel over this area.

Work Order IP3-07-19295, was written requesting excavation of the material surrounding the 8" Aux Steam line
leaks. Further actions shall be taken as a function of the results of the visual inspection of the leak areas. Work
Order IP3-07-19911, was written to perform the necessary repair of the steam line.

The 8" 0 Aux Steam line has a minimum of 9" thick gilsotherm insulation over which a 1'6" thick soil cover is
provided. The Aux Steam and the 4" 0 Condensate line have special supports which are located every 15'. Details
of the supports are as follows:

Root Cause Analysis Report • 290f39

IPEC00199079

IPEC00199079
Previous Occurrence Evaluation

External

A search of external operating experience data was performed for this event using the INPO web. The searches
were conducted but not limited to SOERs, SERs, SENs, Topical Reports, NRC GLs, NRC INs, NRC Bulletins,
Plant Events, LERs, EPIX, NPRDs and vendor notifications. A date restriction was not placed on the searches.
Searches consisted ofOE containing "buried: and "pipe" of which 247 hits were returned. Twelve pertinent CRs
from the OE search were noted for which the information provided was considered in this root cause evaluation. A
summary of the pertinent OE is contained in Attachment VI .

Root Cause Analysis Report • 300f39

IPEC00199080

IPEC00199080
Safety Significance Evaluation
There is no safety significance to the operating plant as a direct result of the degraded condition on the 8" CST
return line based on the following:

1. Overall integrity of the subject line was evaluated under Calculation IP-CALC-09-00032 (EC 13322) to be
structurally adequate per ASME CC-N513-2 with the through wall leak and the subject line remained
operable.

2. Presence of the through wall leak can drain the 8" Condensate return but it will not siphon the CST. Since
the CST overflow line enters/exits the CST at the same elevation as the 8" return line, (i.e. 115'5" pipe
centerline (Drawing 9321-F -2264)) and the 8" return line is equipped with a %" siphon breaker hole drilled at
the top of the pipe inside the tank (9-9237-DWG-19), the volume loss in the CST is effectively limited to the
volume of water present in the return and overflow lines. This siphon breaker ensures that the CST water
level remains at a minimum of 34.01' from the tank bottom which in tum ensures approximately 645,000
gallons (Graph TC-6, Rev 1 and Calculation FIX-00024) is maintained in the tank.

3. The loss in inventory returned back to the CST, with the AFW pumps in operation re-circulating to the CST
through a postulated degradation is estimated to be less than 15 GPM. This estimate is conservative since
the increase in pressure at the area from pump operation is expected to be less than 1 psig. (Normal head
pressure from the CST is about 40 psig. Estimated differential pressure drop through 300' of 8" schedule 40
pipe at a flow rate of650 gpm is less than 1 psid from the Appendix B Table B-14 of Crane Technical Paper
410). The increase of 1 psid over the initial 40 psid is about 1% in flow. Based on the pump out rate of
approximately 10-12 gpm noted during uncovering of the subject pipe, the estimated increase is expected to
be significantly less than 15 gpm stated. Assume a 15 gpm loss of inventory through the breach, the
estimated volume loss from the CST for a 24-hour period would be about 21,600 gallons (1.13' in tank
level).

4. In support of decay heat removal, Technical Specification (TS) 3.7.6 requires a minimum of CST volume of
360,000 gallons for 24 hours following a trip at 100.6% rate thermal power (RTP). Due to piping
degradation, some of the recirculation flow from operating AFW Pump would be lost and additional volume
in the CST is required for this for 24 hours. Based on the condition which existed, CST volume would have
to be increased by approximately 21,600 gallons to accommodate inventory loss through the breach above the
360,000 gallons resulting in a required volume of3 81,600 gallons. Rounding this to 400,000 gallons, it is
equivalent to an approximately indicating level of 17'. A review of IP2 CST trend since 1/1/2007 to the
present, minimum indicated CST level is above 19'. Additionally, multiple barriers are in place to support
CST inventory about the 17' indicating level. These barriers include a low-level alarm at 19.5' and 19.2',
and automatic closure of LCV-1158 at 18.21'.

5. There is no significant environmental impact since the leakage is discharged via the site storm drainage
system because the CST is already identified in the IPEC SPEDES Permit. Leakage was directed to the
nearby storm drainage (e.g. Manhole #5). Environmental Engineering was informed of the condition and has
been monitoring these discharges. It was noted that a minute level of Tritium was detected in the samples.

6. There is no industrial impact as a result of the degradation. The leakage was underground below the concrete
flooring. Subsequent excavation and repair to the subject pipe was conducted in accordance with the
accepted work practice.

7. City water is available as a backup to the CST in the event that the CST level drops below the level required
to perform its safety function.

Root Cause Analysis Report • 31 of39

IPEC00199081

IPEC00199081
Safety Significance Evaluation

In sunnnary, there is no safety significance to the health and safety of the public from nuclear, safety, industrial or
environmental associated with the subject event due to the condition of the degraded 8" CST return line. Multiple
barriers are in place to insure the minimum required CST inventory is not challenged as a direct result of the piping
degradation and any discharge is monitored consistent with the SPEDES Permit.

Root Cause Analysis Report • 320f39

IPEC00199082

IPEC00199082
Corrective Action Plan
A. Immediate Actions: {Describe actions performed on discovery and remedial actions which were taken to mitigate the consequences of the event
or condition}

1. Replaced the damaged section of the 8" CST return piping per Work Order 00183296.

2. Released to the industry as OE28335; (Preliminary) Leaking Underground Condensate Return Line Pipe. (lPEC,IP2) Consequently, this will
be reviewed by each site as it enters the OE screening process from INPO. (CA-06 of this CR.)

B. Interim Actions (if required): {Document any interim actions needed to minimize the likelihood of recurrence WHILE THE LONG-TERM ACTIONS
ARE COMPLETED. Each interim action should only last until a specific long-term action is completed.}

1. Send out the failed section of pipe for failure analysis. Assign to P&C Eng, Due Complete

2. Track failure analysis for the failed section of the pipe and document the results of the testing and analysis. Assign to P&C Eng, Due
5/2112009

3. Discuss this Root Cause Analysis with the ESP population, through ESP Continuing Training "Kick-Off' session. Assign to P&C Eng,
Due 10/15/2009

4. Review modification and corrective maintenance packages ready for installation that are related to buried piping and backfilling. Assign to
Design Eng, Due 612512009

Root Cause Analysis Report • 330f39

IPEC00199083

IPEC00199083
Corrective Action Plan

Long Term Actions: {DESCRIBE recommended actions to mitigate consequences and prevent recurrence. IDENTIFY which actions are Corrective
Actions to Prevent Recurrence (CAPRs) and any which are Long-Term CAs per EN-LI-I02.

Identified Causes Corrective Actions Resp. Dept. Due Date

RC b CAPR and CC I • Update the buried piping backfill and excavation specification for IPEC as a site. Design Eng 12/17/2009

CC2 Implement improved inspection techniques for buried piping. P&C Eng 9/1012009

DC Send out removed pipe for failure analysis. Track and evaluate results P&C Eng 5/2112009

RC I Research the original construction of this buried pipe for any additional backfill Design Eng 9/24/2009
guidance that may have been available
CC I Evaluate the need/feasibility for cathodic protection to be used on selected buried P&C Eng 12/17/2009
piping. Initiate Engineering changes and present to the URT with results, as necessary.
CC I Evaluate the need for a drainage system and monitoring for CST lines, near Manhole Design Eng 12/17/2009
#5. Initiate Engineering Changes and present to the URT, as necessary.
CC2 Ensure inspection locations are based on risk. Considering highly moist environments Eng- HQN 6/1512009
to be included in the procedure. Include Corporate Engineering Programs (CEP) for
inspection guidance.
CC2 Evaluate the need to add cathodic protection to those areas of buried pipe whose P&C Eng 11115/2009
inspections have indicated pipe defects.
EOC Once additional inspections are complete, initiate additional CAs as required and P&C Eng 9/24/2009
present the results to CARB.
Other Evaluate the use of existing monitoring wells for buried pipe and tank leaks as early Projects 7/23/2009
leak detection. Update monitoring wells testing requirements as necessary.
OE Issue an internal fleet-learning NOE in accordance with EN-LI-I02 Attachment 9.5 to OE Complete
[fleet manager identified by CARB] to evaluate this for the fleet" (Preliminary) Coordinator
OE Issue/revise an internal/external OE to the Industry for this Root Cause Evaluation in OE 6/18/2009
accordance with EN-OE-IOO Coordinator

Root Cause Analysis Report • 340f39

IPEC00199084

IPEC00199084
Corrective Action Plan
Identified Causes Corrective Actions Resp. Dept. Due Date
EOC Perform pipe inspection at the location; IP2 8" Condensate Return Line in the P&C Eng 9/10/2009
excavated area in the FRV Room"
EOC Perform pipe inspection at the location; IP2 12" Condensate Supply Line in the P&C Eng 9/10/2009
excavated area in the FRV Room"
EOC Perform pipe inspection at the location; IP2 24" SW Line 408 in the Transformer Yard P&C Eng 9/10/2009
outside the PAB where it exits the ground"
EOC Perform pipe inspection at the location; IP3 12" Condensate Supply Line outside the P&C Eng 9/10/2009
Auxiliary Feedwater Pump Building where it goes underground.
EOC Perform pipe inspection at the location; IP3 8" Condensate Return Line outside the P&C Eng 9/10/2009
Auxiliary Feedwater Pump Building where it exits the ground.
EOC Perform pipe inspection at the 10cation;IP3 24" Line 408 in the backup pump valve pit P&C Eng 9/10/2009

Effectiveness Review Issue an LOCA to track and document the RCA effectiveness review CA&A 5/28/2009

Root Cause Analysis Report • 350f39

IPEC00199085

IPEC00199085
Effectiveness Review Plan
Effectiveness reviews are required for all CAPRs ( and may also encompass the entire corrective action plan). This section should contain an
Effectiveness Review strategy that includes the following:

• METHOD - describe the method that will be used to verifY that the actions taken had the desired outcome.
• ATTRIBUTES - Describe the process attributes to be monitored or evaluated.
• SUCCESS - Establish the acceptance criteria for the attributes to be monitored or evaluated.
• TIMELINESS - Define the optimum time to perform the effectiveness review.}

THE FOLLOWING DEFINES THE EFFECTIVENESS REVIEW PLAN FOR THE CAPRs IN THIS CORRECTIVE ACTION PLAN:

CAPR (State the CAPR(s) here) Resp. Dept. Due Date

1. Design Engineering to update the buried piping backfill and excavation specification for
IPEC as a site.

Method 1. Document review of the new specification issued for IPEC. P&CEng

Attributes 1. Verify new specification thoroughly addresses: Design Eng

Details of gravel, sand rock size
Layering requirements, dimensions
Type offill

Success 1. The details specified in Attributes section have been adhered to. Design Eng

Timeliness 1. Perform a review 3 months after completion of the CAPR (Nine months has P&CEng
been allowed to complete the specification).

See Att. 9.10 in LI-118 (REPEAT SEQUENCE ABOVE FOR EACH CAPR)

Root Cause Analysis Report • 360f39

IPEC00199086

IPEC00199086
References

Documents reviewed:

1. EN-LI-1l8, Root Cause Analysis Process

2. EN-DC-343, Buried Piping and Tank Inspection and Monitoring Program

3. EN-LI-1l9, Apparent Cause Evaluation (ACE) Process

4. Corrosion and Cathodic Protection Theory, Bushman and Associates, Inc.

5. Siebert, CA and Bush, SH - Literature Survey on Underground Pipe Lines, Their Corrosion and Protection.
Engineering Research Institute, University of Michigan

6. Jonas, O. (PE; PhD) - Water Chemistry and Corrosion Control, Training Course Material

7. LO-IP3LO-2008-00l5l, CA 19, Attachment 1, IPEC Buried Piping Impact Assessment, Corrosion Risk Ranking
and Inspection Prioritization Methodology

8. IP-RPT-09-000ll, EC 13500, Corrosion/Cathodic Protection Field Survey of Underground Structures at IPEC

during October 2008

9. Watson, TRB - Why Metals Corrode

10. EPRI Report 1016456, Draft 3, September 2008 - Recommendations for an Effective Program to Control the
Degradation of Buried Pipe

11. IP Drawing 932l-F -2019 - Flow Diagram Boiler Feedwater

12. IP Drawing 9321-2018 - Flow Diagram Condensate and Boiler Feed Pump Suction

13. IP Drawing 932l-F-l024 - Containment Building Backfilling and Grading North and East Side

14. IP Drawing 932l-F-1209 - Shield Wall Area Concrete Foundation Sections

15. Drawing 932l-F-1002 - Plot Plan - UFSAR Figure No 1.2-3

16. Drawing 932l-F-1208 - Shield Wall Area Concrete Plan Elevation 18'6"

17. Drawing 9321-F -1231 - Shield Wall Elevations and Section - Sheet No 2

18. Drawing 932l-F-2263 - Yard Area - Condensate Storage Tank to Turbine Building Condensate Piping Plan - Sheet
No 1

19. Drawing 932l-F-2264 - Yard Area - Condensate Storage Tank to Turbine Building Condensate Piping Plan - Sheet
N02

20. Structural Integrity Associates Inc, Report No. 0900235.401

Root Cause Analysis Report • 370f39

IPEC00199087

IPEC00199087
References

Personnel contacted:

1. Greg Bouderau

2. Joe Peterson

3. Joe Ruch

4. Bob Allen

5. Bob Danko - Stone and Webster

6. Rich Meyer Jr - Stone and Webster

7. John Bouse - Stone and Webster

8. Michael O'Brien - Stone and Webster

9. Charles Kiall - Stone and Webster

10. Robert Okuniewski - Stone and Webster

11. Tom Camilliere - Stone and Webster

12. Bernard Barreiro - Stone and Webster

13. John Curry

14. Charlie Jackson

15. Bob Eifler

16. Kai La

17. Jan Mayer

18. Kevin Curley

19. Mike Tesoriero

20. Nelson Azevedo

21. Bob Lee

22. Matt Barvenik - GZA Geo Environmental Incorporated

Root Cause Analysis Report • 38 of39

IPEC00199088

IPEC00199088
References

Analysis Methodologies employed: {List all analysis methods used}

1. Why Analysis

2. K-T Method

3. SIA Report "Analysis of8" Condensate Water Storage Tank Return Line CD-183", Report No. 0900235.402

Attachments: in PCRS

I. Equipment Failure Evaluation

II. K-T Analysis

III. Why Staircase

IV. Operating Experience

Root Cause Analysis Report • 390f39

IPEC00199089

IPEC00199089