Shopping sites ratchet up battle against bots
Skip to main content
Sister Publication Links
  • Automotive News Canada
  • Automotive News Europe
  • Automotive News China
  • Automobilwoche
AN-LOGO-BLUE
Subscribe
  • Subscribe
  • Account
  • login
  • HOME
  • NEWS
    • Dealers
    • Automakers & Suppliers
    • News by Brand
    • Cars & Concepts
    • Coronavirus Coverage
    • China
    • Shift
    • Mobility Report
    • Special Reports
    • Digital Edition Archive
    • This Week's Issue
    • GHSP
      Mich. supplier showcases technologies to fight distraction, virus
      Shoppers to choose how they use digital retail
      Ford's ‘Finish Strong' ad sets right tone for '21
      EcoMotion’s Orlie Gruper says the 2020 event was quickly shifted to an all-virtual format.
      High-profile events opt for virtual route
    • Hyundai Motor Group sets first overseas fuel cell plant in China
      Inside Huawei's push to develop smart roads that talk to driverless cars
      Vehicle production line at GAC-Honda, Honda's joint venture with GAC Motor Co.
      Honda books sales growth in 2020; Nissan slips
      Mercedes-Benz China
      Mercedes, Audi, BMW post record high deliveries for 2020
    • Detroit Aerial Panorama during sunset
      Developing future workers to fuel growth in mobility
      Getting around in Las Vegas often involves long waits for a crowded monorail ride, below, or taking escalators up and down to cross elevated pedestrian walkways along the Strip.
      Las Vegas offers best and worst in transportation challenges
      In pandemic environment, RV industry ‘roaring back'
      Where is transportation headed?
    • Revel
      The moped-sharing startup with thousands of EVs and hundreds of collisions
      ELMS_Headquarters_i.jpg
      EV maker Electric Last Mile to move HQ in suburban Detroit
      Uber logo on a car window
      Uber appeals $59M Calif. fine over sexual assault data request
      Michigan cops go electric with new Mustang Mach-E
    • Elon Musk interview
      Tesla’s Model Y, on sale since March, is a showcase of EV technologies competitors will benchmark.
      Technologies of Electrification
      Cadillac’s Lyriq EV will be unveiled Aug. 6.
      Future Product Pipeline
      A CALL TO ACTION
    • Hot topics at virtual CES: AVs, air taxis and drones
      CarMax sees longer return policy as its next trend
      Japanese version of the Mazda MX-30
      Coming to the U.S.: Mazda EV with rotary backup
      Stronger bottom lines despite 2020 sales dive
    • Access F&I
    • Fixed Ops Journal
    • Marketing
    • Used Cars
    • Retail Technology
    • Sales
    • Best Practices
    • Dealership Buy/Sell
    • NADA
    • NADA Show
    • Automakers
    • Manufacturing
    • Suppliers
    • Regulations & Safety
    • Executives
    • Talk From The Top
    • Leading Women Network
    • Guide to Economic Development
    • PACE Awards
    • Management Briefing Seminars
    • World Congress
    • Aston Martin
    • BMW
      • Mini
      • Rolls-Royce
    • Daimler
      • Mercedes Benz
      • Smart
    • Fiat Chrysler
      • Alfa Romeo
      • Chrysler
      • Dodge
      • Ferrari
      • Fiat
      • Jeep
      • Maserati
      • Ram
    • Ford
      • Lincoln
    • General Motors
      • Buick
      • Cadillac
      • Chevrolet
      • GMC
      • Holden
    • Honda
      • Acura
    • Hyundai
      • Genesis
      • Kia
    • Mazda
    • McLaren
    • Mitsubishi
    • Nissan
      • Infiniti
    • PSA
      • Citroen
      • Opel
      • Peugeot
    • Renault
    • Subaru
    • Suzuki
    • Tata
      • Jaguar
      • Land Rover
    • Tesla
    • Toyota
      • Lexus
    • Volkswagen
      • Audi
      • Bentley
      • Bugatti
      • Lamborghini
      • Porsche
      • Seat
      • Skoda
    • Volvo
    • (Discontinued Brands)
    • Virtual reveals (Sponsored)
      • GENESIS: 2021 GV80
      • KIA: 2021 K5
      • LEXUS: 2021 IS
      • NISSAN: 2021 Rogue
      • TOYOTA: 2021 Venza and 2021 Sienna
    • Auto Shows
      • Detroit Auto Show
      • New York Auto Show
      • Los Angeles Auto Show
      • Chicago Auto Show
      • Geneva Auto Show
      • Paris Auto Show
      • Frankfurt Auto Show
      • Toronto Auto Show
      • Tokyo Auto Show
      • Shanghai Auto Show
      • Beijing Auto Show
    • Future Product Pipeline
    • Photo Galleries
    • Car Cutaways
    • Design
  • OPINION
    • Blogs
    • Cartoons
    • Keith Crain
    • Automotive Views with Jason Stein
    • Columnists
    • China Commentary
    • Editorials
    • Letters to the Editor
    • Send us a Letter
    • Nikon turns lens on auto factory, build quality
      Bosch_information_domain_computer web.jpg
      Bosch, Continental battle to supply the brain of the connected car
      A victory lap for Detroit car design
      Stellantis_logo_blue_background web.jpg
      In 'merger of equals,' PSA is buying FCA, Stellantis prospectus says
    • view gallery
      1 photos
      The Bridge
      view gallery
      12 photos
      Leo Michael's best of 2020
      view gallery
      1 photos
      UAW Deal
      view gallery
      1 photos
      Exit Voluntarily
    • Shifting gears away from the stick shift
      SEMA still a wonderful circus
      Penske still has plenty of races to win
      Ford's turn in the hot seat
    • January 12, 2021 | Automotive industry shined in 2020
      December 8, 2020 | In a year that lacked luster, All-Stars showed how to shine
      November 24, 2020 | Female talent in automotive is expanding
      November 17, 2020 | Automakers are getting creative with vehicle reveals
    • Daron Gifford
      Biden era holds big promise for future of electric vehicles
      A show we love to hate ... but hate to miss
      Dave Versical
      Female CMO is new norm
      Magna car output web.jpg
      Who will build the Apple car?
    • Is sales recovery nearing an end?
      Beijing's uphill battle to boost EV sales
      Nasdaq-like Star Board poised to become key finance platform
      Virus outbreak upends ranks of EV startups
    • Ford's ‘Finish Strong' ad sets right tone for '21
      UAW settlement brings justice, voting rights
      Dealers should be treated as partners in EVs
      Bergstrom Chevrolet-Buick-GMC-Cadillac of Oshkosh donates a new truck to Habitat for Humanity.
      Commitment to safety begets lasting loyalty
    • Is banning ICEs in our best interests?
      GM should share EV risk with dealers
      Ready to buy Mirai, but where to fuel?
      From Toyota store to hydrogen highway
  • DATA CENTER
  • VIDEO
    • AutoNews Now
    • First Shift
    • Special Video Reports
    • Weekend Drive
    • AutoNews Now: Stellantis may reconsider Peugeot's U.S. return
      AutoNews Now: Toyota to pay $180M in U.S. emissions settlement
      AutoNews Now: NTSB to automakers: Improve EV emergency response guides
      AutoNews Now: Cadillac's electric Celestiq teased at CES
    • First Shift: GM said to be considering electric Corvette crossover
      First Shift: Subaru to slow output due to microchip shortage
      First Shift: Trump administration delays increased fuel-efficiency fines
      First Shift: Ford, NADA rethink political donations in wake of Capitol siege
    • N.J. dealer helps position peers for ‘electric revolution’
      DCH Millburn Audi
      'Finding the diamond in the rough': How 2020's No. 1 dealership retains talent
      Don Johnson Motors
      'Distinctive culture' cuts turnover, carves career paths at Wis. dealership group
      Why 2020 could be a record year for buy-sells
    • Why the pickup is the auto industry's 'battleground'
      Carlos Ghosn's quest to restore his reputation
      Why Ford must execute to avoid 'deep trouble'
      Why Honda is 'locked and loaded' for 2020
  • EVENTS & AWARDS
    • Events
    • Awards
    • Congress Conversations
    • Retail Forum: NADA
    • Canada Congress
    • Europe Congress Conversations
    • Leading Women Conference
    • Fixed Ops Journal Forum
    • ANE Shift
    • Shift: Mobility at a Crossroads
    • Shift: The Future of Mobility (CES)
    • 100 Leading Women
    • 40 Under 40 Retail
    • All-Stars
    • Best Dealerships To Work For
    • PACE Program
    • Rising Stars
    • Europe Rising Stars
  • JOBS
  • AN Solutions
  • +MORE
    • Leading Women Network
    • Podcasts
    • Webinars
    • In the Driver's Seat
    • Publishing Partners
    • Classifieds
    • Companies on the Move
    • People on the Move
    • Newsletters
    • Contact Us
    • Media Kit
    • RSS Feeds
    • Shift: A Podcast About Mobility
    • Special Reports Podcasts
    • Daily Drive Podcasts
    • AAM
    • Gentex
    • Reputation.com
    • Ricardo
    • Ricardo
    • Allstate: Want more from your F&I?
    • Ally: Navigating the future of automotive retailing
    • Amazon Web Services: Any place, any time, any channel
    • Amazon Web Services: The power of the cloud
    • Amazon Web Services: Universal translator: Harnessing sensor data to build better automotive software
    • Epic Games: Transforming the auto industry with digital assets
    • FTI Consulting: Crisis as a catalyst for change
    • Google: 5 trends shaping the auto industry's approach to a new normal
    • IHS Markit: Automotive loyalty in the wake of the COVID-19 recession
    • IHS Markit: Autonomous vehicles: Automotive and transportation disruption
    • IHS Markit: COVID-19: The future mobility delusion
    • IHS Markit: The battery electric vehicle (BEV)
    • Level5: 2020 Automotive E-Commerce Report
    • Naked Lime: Bring social reputation together as part of big-picture marketing
    • Wells Fargo Auto: Switching gears from LIBOR to SOFR
    • Ally: Do It Right
    • DealerSocket
    • Deloitte: Cyber everywhere: Preparing for automotive safety in the face of cyber threats
    • Facebook: The road to a zero-friction future
    • Guide To Economic Development
    • PayPal Credit: How consumer financing helps drive sales for online auto parts retailers
MENU
Breadcrumb
  1. Home
  2. Retail
July 13, 2014 01:00 AM

Shopping sites ratchet up battle against bots

Vince Bond Jr.
  • Tweet
  • Share
  • Share
  • Email
  • More
    Print
    BLOOMBERG
    For shopping sites such as Cars.com, TrueCar and AutoTrader, whose listings of vehicles in dealer inventories attract millions of legitimate consumers a month, it's become a constant battle to weed out shady players that can compromise traffic data.
    Traffic jam

    Car shopping sites AutoTrader, Cars.com and TrueCar have to detect suspicious bot activity amid a torrent of Web traffic each day. Their hands are full.

    • AutoTrader: Averaging more than 18 million unique monthly visitors in 2014; topped 16 million unique monthly visitors last year

    • Cars.com: Around 13.5 million unique monthly visitors in 2013 across desktop and mobile platforms

    • TrueCar: Tallied 3.9 million unique visitors a month in the first quarter of 2014

    Cars.com site analysts knew something was up about seven years ago when they saw traffic suddenly surge on BMW listings in Albany, N.Y.

    A little digging found that the activity was no cause for celebration. It wasn't coveted in-market car shoppers who were driving up page views on dealers' inventory listings but rather Internet bots -- snippets of nefarious software code masquerading as human Web users -- that had invaded the site and skewed its traffic metrics.

    It was the opening salvo in a cyberwar that Cars.com is still fighting today, against an enemy that is only becoming more sophisticated.

    "That's when we started doing something about it," William Swislow, Cars.com's chief information officer, said of the bot menace. "We spend a lot of time and creativity basically recognizing when someone who appears to be accessing the site is a real person or a bot."

    The risk goes beyond skewed numbers. Site managers say bots can swipe inventory listing data for reuse by outside parties and can even skim off ad revenue that's generated from online clicks.

    For shopping sites such as Cars.com, TrueCar and AutoTrader, whose listings of vehicles in dealer inventories attract millions of legitimate visitors a month, it is critical to weed out shady players and protect the integrity of the data they're reporting in order to sustain their relationship with dealers. Any distortion of the metrics could confound dealerships that are seeing heavy activity on the sites but not getting a proportionate number of leads.

    The sites say the reports they provide to dealers -- with metrics such as how many e-mail leads and phone calls were generated by a listing and how many times vehicle detail pages were clicked -- are scrubbed clean of bot activity. But figuring out what the bots are after and how to beat them is a constant challenge.

    "It's hard to know who they are in many cases and what they want to do with the data," said Scott Hernalsteen, AutoTrader's senior director of enterprise analytics.

    "If you look at the success of AutoTrader over the years, we've built up a large number of listings," he said. "They could be trying to acquire [the information], accumulate it to repackage it and sell it to others. At the end of the day, we typically don't know."

    Scott HernalsteenAutoTrader

    "It's hard to know who they are in many cases and what they want to do with the data."

    Precisely how much online traffic is generated by bots -- and how many marketing dollars are wasted on these bogus users -- isn't known either. Estimates of the wasted ad spending range from $1 billion a year to nearly $12 billion.

    Solve Media, a digital advertising and security firm, said its research found that suspicious Web traffic rose 40 percent in 2013, with bot traffic peaking at 61 percent of all traffic during the fourth quarter.

    A common template for bot activity is criminals creating a network of bogus Web sites. They can be one-page junk sites, or more sophisticated sites that look legitimate because they've copied content from other sources, said Jeffrey Tognetti, product development team lead at DealerX, a digital marketing company.

    The site operators create a bot and send it to car shopping or dealer sites with hopes that it will be recognized as a potential buyer. If that happens, an ad retargeting platform -- a service that shows ads to people on the Web sites they visit based on their online shopping history -- could key in on the bot and follow it to the fake site, where it would display an ad.

    As a result, "the person who owns the real estate, who owns that Web site, gets paid … for the impression," Tognetti said. "And then often the bot itself clicks on the ad and gets paid for the click."

    As the efforts to detect them have become more sophisticated, so have the bots themselves. AutoTrader's Hernalsteen said his site, which uses an array of filters if bots make it past their initial wave of defense, receives tens of thousands of attempts to copy content each day.

    The bots sometimes originate in foreign locales, but their creators can disguise them as U.S.-based users to fool Web sites that are equipped to block foreign traffic.

    "Early on we've seen different sources where you'd have international IP addresses hitting a publisher's Web site. Those were easy to sniff out," said Scott Cunningham, vice president of technology and ad operations for the Interactive Advertising Bureau. "Over time, they've gotten really good at masquerading."

    One clear warning sign for shopping sites is browsing speed: Seeing one IP address, or computer identification number, hit hundreds of pages within a minute is a dead giveaway that a bot is at work. But some bots are being designed to closely mimic human behavior by browsing slower to avoid getting flagged.

    John Williams, TrueCar's senior vice president of platform operations, says the site counters these bots with anomaly detection software developed in-house. The software identifies patterns in the traffic, pointing out in some cases whether similar IP addresses appear to be working in concert. If addresses appear to be bots, Williams said, the site can block them, or impose strict rate limits on them when browsing the site.

    A rate limit means there is a maximum rate at which a given set of addresses can load pages, Williams says. An error is returned to the user when the limit is reached.

    TrueCar also spots suspicious patterns through data analysis when looking at factors such as how many car makes a user looks at before buying.

    "We often have these things pop up where [we say] 'Hey, there are some small percentages who look at 300 different makes and they never buy anything.' That's not a person," Williams said. "We just go ahead and block those also."

    Cars.com requires every e-mail lead submitted to go through an automated system that tries to discern if it's legitimate. Plus, Swislow said the fraud team manually reviews any lead that looks suspicious. He said his team has made significant progress against bot traffic over the past several years.

    But even with dedicated security teams and technologies working to spot bots, he says, the enemy can't be eradicated.

    "The nature of the Internet is every time you take steps to defend your site against unauthorized use, there are going to be people who will try to get around it," Swislow said. "I wouldn't make the claim [that] we have no issues whatsoever, but we think we have a pretty good handle on it."

    RECOMMENDED FOR YOU
    Vehicle buyers cling to familiarity amid pandemic, study finds
    Letter
    to the
    Editor

     

     

    Send us a letter

    Have an opinion about this story? Click here to submit a Letter to the Editor, and we may publish it in print.

    Recommended for You
    Vehicle buyers cling to familiarity amid pandemic, study finds
    Vehicle buyers cling to familiarity amid pandemic, study finds
    Shoppers to choose how they use digital retail
    Cadillacs, customers connect virtually via ‘Live' platform
    Cadillacs, customers connect virtually via ‘Live' platform
    Sponsored Content: 5-Step Guide to Modernizing the Buyer’s Journey
    Sign up for free newsletters
    EMAIL ADDRESS

    Please enter a valid email address.

    Please enter your email address.

    Please verify captcha.

    Please select at least one newsletter to subscribe.

    See more newsletter options at autonews.com/newsletters.

    You can unsubscribe at any time through links in these emails. For more information, see our Privacy Policy.

    Digital Edition
    Automotive News 1-11-2021
    THIS WEEK'S EDITION
    See our archive
    Fixed Ops Journal
    Fixed Ops Journal 12-14-20
    Read the issue
    See our archive
    RETAIL TECHNOLOGY NEWSLETTER: Sign up for our Tuesday afternoon newsletter that breaks down the latest developments in how technology is changing the auto sales landscape.
    WEEKLY NEWSLETTER: Sign up to start your week with the top stories from the latest issue of Automotive News, delivered to your inbox first thing each Monday morning.
    Get Free Newsletters

    Sign up and get the best of Automotive News delivered straight to your email inbox, free of charge. Choose your news – we will deliver.

    Subscribe Today

    Get 24/7 access to in-depth, authoritative coverage of the auto industry from a global team of reporters and editors covering the news that’s vital to your business.

    Subscribe Now
    Connect With Us
    • Facebook
    • Instagram
    • LinkedIn
    • Twitter

    Our mission

    The Automotive News mission is to be the primary source of industry news, data and understanding for the industry's decision-makers interested in North America.

    AN-LOGO-BLUE
    Contact Us

    1155 Gratiot Avenue
    Detroit, Michigan
    48207-2997

    (877) 812-1584

    Email us

    Automotive News
    ISSN 0005-1551 (print)
    ISSN 1557-7686 (online)

    Fixed Ops Journal
    ISSN 2576-1064 (print)
    ISSN 2576-1072 (online)

    Resources
    • About us
    • Contact Us
    • Media Kit
    • Subscribe
    • Manage your account
    • Reprints
    • Ad Choices Ad Choices
    • Sitemap
    Legal
    • Terms and Conditions
    • Privacy Policy
    • Privacy Request
    Automotive News
    Copyright © 1996-2021. Crain Communications, Inc. All Rights Reserved.
    • HOME
    • NEWS
      • Dealers
        • Access F&I
        • Fixed Ops Journal
        • Marketing
        • Used Cars
        • Retail Technology
        • Sales
        • Best Practices
        • Dealership Buy/Sell
        • NADA
        • NADA Show
      • Automakers & Suppliers
        • Automakers
        • Manufacturing
        • Suppliers
        • Regulations & Safety
        • Executives
        • Talk From The Top
        • Leading Women Network
        • Guide to Economic Development
        • PACE Awards
        • Management Briefing Seminars
        • World Congress
      • News by Brand
        • Aston Martin
        • BMW
          • Mini
          • Rolls-Royce
        • Daimler
          • Mercedes Benz
          • Smart
        • Fiat Chrysler
          • Alfa Romeo
          • Chrysler
          • Dodge
          • Ferrari
          • Fiat
          • Jeep
          • Maserati
          • Ram
        • Ford
          • Lincoln
        • General Motors
          • Buick
          • Cadillac
          • Chevrolet
          • GMC
          • Holden
        • Honda
          • Acura
        • Hyundai
          • Genesis
          • Kia
        • Mazda
        • McLaren
        • Mitsubishi
        • Nissan
          • Infiniti
        • PSA
          • Citroen
          • Opel
          • Peugeot
        • Renault
        • Subaru
        • Suzuki
        • Tata
          • Jaguar
          • Land Rover
        • Tesla
        • Toyota
          • Lexus
        • Volkswagen
          • Audi
          • Bentley
          • Bugatti
          • Lamborghini
          • Porsche
          • Seat
          • Skoda
        • Volvo
        • (Discontinued Brands)
      • Cars & Concepts
        • Virtual reveals (Sponsored)
          • GENESIS: 2021 GV80
          • KIA: 2021 K5
          • LEXUS: 2021 IS
          • NISSAN: 2021 Rogue
          • TOYOTA: 2021 Venza and 2021 Sienna
        • Auto Shows
          • Detroit Auto Show
          • New York Auto Show
          • Los Angeles Auto Show
          • Chicago Auto Show
          • Geneva Auto Show
          • Paris Auto Show
          • Frankfurt Auto Show
          • Toronto Auto Show
          • Tokyo Auto Show
          • Shanghai Auto Show
          • Beijing Auto Show
        • Future Product Pipeline
        • Photo Galleries
        • Car Cutaways
        • Design
      • Coronavirus Coverage
      • China
      • Shift
      • Mobility Report
      • Special Reports
      • Digital Edition Archive
      • This Week's Issue
    • OPINION
      • Blogs
      • Cartoons
      • Keith Crain
      • Automotive Views with Jason Stein
      • Columnists
      • China Commentary
      • Editorials
      • Letters to the Editor
      • Send us a Letter
    • DATA CENTER
    • VIDEO
      • AutoNews Now
      • First Shift
      • Special Video Reports
      • Weekend Drive
    • EVENTS & AWARDS
      • Events
        • Congress Conversations
        • Retail Forum: NADA
        • Canada Congress
        • Europe Congress Conversations
        • Leading Women Conference
        • Fixed Ops Journal Forum
        • ANE Shift
        • Shift: Mobility at a Crossroads
        • Shift: The Future of Mobility (CES)
      • Awards
        • 100 Leading Women
        • 40 Under 40 Retail
        • All-Stars
        • Best Dealerships To Work For
        • PACE Program
        • Rising Stars
        • Europe Rising Stars
    • JOBS
    • AN Solutions
    • +MORE
      • Leading Women Network
      • Podcasts
        • Shift: A Podcast About Mobility
        • Special Reports Podcasts
        • Daily Drive Podcasts
      • Webinars
      • In the Driver's Seat
        • AAM
        • Gentex
        • Reputation.com
        • Ricardo
        • Ricardo
      • Publishing Partners
        • Allstate: Want more from your F&I?
        • Ally: Navigating the future of automotive retailing
        • Amazon Web Services: Any place, any time, any channel
        • Amazon Web Services: The power of the cloud
        • Amazon Web Services: Universal translator: Harnessing sensor data to build better automotive software
        • Epic Games: Transforming the auto industry with digital assets
        • FTI Consulting: Crisis as a catalyst for change
        • Google: 5 trends shaping the auto industry's approach to a new normal
        • IHS Markit: Automotive loyalty in the wake of the COVID-19 recession
        • IHS Markit: Autonomous vehicles: Automotive and transportation disruption
        • IHS Markit: COVID-19: The future mobility delusion
        • IHS Markit: The battery electric vehicle (BEV)
        • Level5: 2020 Automotive E-Commerce Report
        • Naked Lime: Bring social reputation together as part of big-picture marketing
        • Wells Fargo Auto: Switching gears from LIBOR to SOFR
        • Ally: Do It Right
        • DealerSocket
        • Deloitte: Cyber everywhere: Preparing for automotive safety in the face of cyber threats
        • Facebook: The road to a zero-friction future
        • Guide To Economic Development
        • PayPal Credit: How consumer financing helps drive sales for online auto parts retailers
      • Classifieds
      • Companies on the Move
      • People on the Move
      • Newsletters
      • Contact Us
      • Media Kit
      • RSS Feeds