Summarizing articles in a new way

From: Nadav Rotem (nadavrotem_at_mail.ru)
Date: Sat Dec 06 2003 - 09:07:49 EST

  • Next message: Dom Lachowicz: "Commit: hspell provider for enchant"

    I started working on a new way of summarizing texts. This time the goal it to take a group of
    articles and join them to a single article; I do so by taking one article from the group and replacing parts of it with parts of the other articles that have the same signature.

    The code is still hackish , leaks buckets and is NOT ready for any use. but it's in the cvs under examples/;

    Here (Attached) is a quick demo of an article I joint from various sources;
                                                                                                                                 
    > [nadav_at_gringo news]$ ../join-art cnn.txt abc.txt fox.txt msn.txt usatoday.txt yahoo.txt > ../new_news.txt

    new_news: Generated articles
    news_replace: a list of replaced blocks

    -Nadav Rotem

    The storm brought several inches of snow to Virginia and Maryland on Friday, and road crews were preparing for a second onslaught with flurries overnight expected to turn to heavy snow. FORECASTERS SAID the system could hit even harder over the weekend, with total snow accumulations of up to two feet in Massachusetts and near-blizzard conditions in parts of Maine and New Hampshire. These included a blizzard warning for Saturday for the New York City area. Many schoolchildren got an early start to their weekend when dozens of schools closed early on Friday. More than 80 fender-benders were reported on the New Jersey Turnpike, where the speed limit was reduced from 65 mph to 35 mph because of dangerous conditions. Winds from 20 to 40 mph with higher gusts are forecast Saturday near the coast from New York City north to Boston. "We're hoping the forecasters are wrong, but if they're not, we're trusting that people will be staying home with their families and off the road so we can get our job done," said Anna Farneski, spokeswoman for the Bergen County, N.J. The warning means forecasters expect sustained wind speeds or frequent gusts of 35 mph or faster with considerable falling or blowing and drifting snow that will drop visibility to zero at times. "I love storms, but I don't like shoveling," Stevenson said. Forecasters in Massachusetts predicted up to 18 inches of snow through Sunday. By Friday evening, The National Weather Service had winter storm warnings in effect from Virginia and West Virginia northeast into southern New England. About 100 flights were canceled at Philadelphia International Airport on Friday, and some flights were delayed at Washington Dulles International Airport. Flights were canceled at New York's La Guardia, New Jersey's Newark Liberty International Airport, Boston and Philadelphia airports, and others were delayed by several hours. Kennedy airports and New Jersey's Newark Liberty International, according to the Port Authority of New York and New Jersey. The storm had already brought a mixture of ice, freezing rain, sleet and snow in mountain regions of North Carolina, Virginia and West Virginia on Thursday. The Weather Service said the complex storm system brought in snow and sleet on Friday, which changed to rain in some places as temperatures warmed through the day. About 100 flights were canceled at Philadelphia International Airport on Friday, and some flights were delayed at Washington Dulles International Airport. Cancellations and delays also were expected Saturday at New York's La Guardia and John F. "We're loving it," said Tana Graham, visiting New York City from Jackson, Miss. “The salt is here, the trucks are ready, the chains are on, and New York is prepared for the assault,” Mayor Michael Bloomberg said Friday. "I just bought 100 pounds of ice melt and we took out our snow shovels last night. "The salt is here, the trucks are ready, the chains are on, and New York is prepared for the assault," New York City Mayor Michael Bloomberg said Friday. At least one death was blamed on the storm, that of a 23-year-old man who was killed in an accident on a slick road in Virginia. By Jacqueline Larma, AP Friday's snow was like a warm-up for the main event with even heavier snow and high winds making travel even harder on Saturday, forecasters said. Four to 8 inches were expected in the Baltimore region.

    replacing
    [Overnight, the heavy snows caused traffic accidents and airport delays.] with
    [ The storm brought several inches of snow to Virginia and Maryland on Friday, and road crews were preparing for a second onslaught with flurries overnight expected to turn to heavy snow.]

    replacing
    [ Visibility was down to a quarter-mile in parts of New England as winds and snow made driving conditions treacherous.] with
    [ FORECASTERS SAID the system could hit even harder over the weekend, with total snow accumulations of up to two feet in Massachusetts and near-blizzard conditions in parts of Maine and New Hampshire.]

    replacing
    [ In New York City a blizzard warning has been issued from 10 a.m. until 10 p.m. Saturday.] with
    [ These included a blizzard warning for Saturday for the New York City area.]

    replacing
    [ In Connecticut, traffic accidents were widespread Friday, with a portion of I-95 shut down during rush hour.] with
    [ Many schoolchildren got an early start to their weekend when dozens of schools closed early on Friday.]

    replacing
    [ Authorities urged residents to limit travel Saturday.] with
    [ More than 80 fender-benders were reported on the New Jersey Turnpike, where the speed limit was reduced from 65 mph to 35 mph because of dangerous conditions.]

    replacing
    [ Winds were expected to be 15-20 mph Saturday, with gusts as high as 40 mph.] with
    [ Winds from 20 to 40 mph with higher gusts are forecast Saturday near the coast from New York City north to Boston.]

    replacing
    [ "We're hoping people take caution," said Julie Bisi, a spokeswoman for Connecticut's emergency management office.] with
    [ "We're hoping the forecasters are wrong, but if they're not, we're trusting that people will be staying home with their families and off the road so we can get our job done," said Anna Farneski, spokeswoman for the Bergen County, N.J.]

    replacing
    [ "The blowing and drifting could cause problems."] with
    [ The warning means forecasters expect sustained wind speeds or frequent gusts of 35 mph or faster with considerable falling or blowing and drifting snow that will drop visibility to zero at times.]

    replacing
    [ Forecasters said the height of the storm is expected to come at midday Saturday.] with
    [ "I love storms, but I don't like shoveling," Stevenson said.]

    replacing
    [ Snow totals are predicted to be 10-18 inches.] with
    [ Forecasters in Massachusetts predicted up to 18 inches of snow through Sunday.]

    replacing
    [ "The nor'easter has yet to crank up," National Weather Service meteorologist Kimberly Buttrick said Friday night from Taunton, Massachusetts.] with
    [ By Friday evening, The National Weather Service had winter storm warnings in effect from Virginia and West Virginia northeast into southern New England.]

    replacing
    [ The snow and low clouds were wreaking havoc on air travel in the Northeast, with the worst delays for some incoming flights running close to 15 hours.] with
    [ About 100 flights were canceled at Philadelphia International Airport on Friday, and some flights were delayed at Washington Dulles International Airport.]

    replacing
    [ Ground stops were in effect late Friday at Newark International Airport in New Jersey, Philadelphia International Airport in Pennsylvania, and Cleveland Hopkins International Airport in Ohio.] with
    [ Flights were canceled at New York's La Guardia, New Jersey's Newark Liberty International Airport, Boston and Philadelphia airports, and others were delayed by several hours.]

    replacing
    [ A ground stop means traffic at other airports bound for that destination is not allowed to depart.] with
    [ Kennedy airports and New Jersey's Newark Liberty International, according to the Port Authority of New York and New Jersey.]

    replacing
    [ Winter storm warnings were in effect from eastern Ohio to Connecticut and from West Virginia into Massachusetts.] with
    [ The storm had already brought a mixture of ice, freezing rain, sleet and snow in mountain regions of North Carolina, Virginia and West Virginia on Thursday.]

    replacing
    [ Freezing rain, sleet and snow from the first storm blanketed parts of North Carolina, Virginia and West Virginia and more is expected as the storm moves up the Atlantic coast.] with
    [ The Weather Service said the complex storm system brought in snow and sleet on Friday, which changed to rain in some places as temperatures warmed through the day.]

    replacing
    [ Washington is expected to get 4-6 inches, and Philadelphia is bracing for 8-10 inches.] with
    [ About 100 flights were canceled at Philadelphia International Airport on Friday, and some flights were delayed at Washington Dulles International Airport.]

    replacing
    [ New York is expected to get at least 8-14 inches of snow.] with
    [ Cancellations and delays also were expected Saturday at New York's La Guardia and John F.]

    replacing
    [ New York City has 200,000 tons of salt to be dispersed on roads, and 1,700 snowplows at the ready.] with
    [ "We're loving it," said Tana Graham, visiting New York City from Jackson, Miss.]

    replacing
    [ New York City Mayor Michael Bloomberg said 3,000 city workers would be on duty to make sure the city's 6,000 miles of roads stay clear.] with
    [ “The salt is here, the trucks are ready, the chains are on, and New York is prepared for the assault,” Mayor Michael Bloomberg said Friday.]

    replacing
    [ "This is snow that we know how to take care of;] with
    [ "I just bought 100 pounds of ice melt and we took out our snow shovels last night.]

    replacing
    [ they've got the right equipment," he said in a news conference Friday night.] with
    [ "The salt is here, the trucks are ready, the chains are on, and New York is prepared for the assault," New York City Mayor Michael Bloomberg said Friday.]

    replacing
    [ Bloomberg said the roads were fine, but when the storm blows in Saturday, conditions could be bad.] with
    [ At least one death was blamed on the storm, that of a 23-year-old man who was killed in an accident on a slick road in Virginia.]

    replacing
    [ "The winds are forecast to be very strong, really blizzard conditions, that's when you'll have to worry about trees coming down, power lines," he said.] with
    [ By Jacqueline Larma, AP Friday's snow was like a warm-up for the main event with even heavier snow and high winds making travel even harder on Saturday, forecasters said.]

    replacing
    [ The weather grounded President Bush's planned Marine One helicopter flight to a re-election campaign luncheon in Baltimore, Maryland, where 6 inches of snow by midmorning had clogged streets and slowed traffic.] with
    [ Four to 8 inches were expected in the Baltimore region.]



    This archive was generated by hypermail 2.1.4 : Sat Dec 06 2003 - 09:04:52 EST