Website Terms and Conditions Held Applicable to Content Copied by Robot or Spider

Back to Legal Insights
Back to Legal Insights

There is a lot of controversy in computer law circles about the circumstances under which the user of a website is bound by provisions contained in the website's terms and conditions of use. Existing case law reflects inconsistent results, with the outcome often turning on whether a "click-on" mechanism was employed, whether the site user actually read the terms and conditions, how many times the user had visited the site, and other factors that might influence a court's determination as to whether it is fair to impose on the user terms and conditions that he or she may never have read or may have read but did not intend to agree to. See "Browse-Wrap License Agreement Not Valid."

But what happens when the website is visited not by a human being, but rather by a "robot" or "spider" that crawls around the site and harvests content? Obviously, the robot or spider did not "read," or at least did not "understand," the website terms and conditions. Is there an argument for enforcing such terms and conditions against a person or firm who directed the robot or spider, but who never read the terms and conditions and may not even know they exist?

Absolutely. A federal district judge in northern California recently took exactly this position in a case, which, if followed by appellate courts, will have widespread ramifications in the computer law world. See Cairo, Inc. v. Crossmedia Services, Inc., 2005 WL 756610 (N.D. Cal Apr. 1, 2005). In this case, Cairo, Inc. caused its "spiders" or "crawlers" to visit various retailers' websites and harvest information about weekly sale prices for products. This information was put into a database maintained by Cairo and accessible on its website. The court noted that "Cairo's computer search programs cannot read the Terms of Use posted on a web site, and they do not report the presence of such Terms of Use back to Cairo. On a day-to-day basis, Cairo does not actually know whether the web pages it searches contain Terms of Use, much less what the specific content of those Terms of Use is." Id. at *3 (citation omitted).

The defendant, Crossmedia, discovered by reviewing its server logs that Cairo was copying promotional materials from its site and posting a version on Cairo's site. Crossmedia sent a demand letter accusing Cairo of breaching the terms of use of Crossmedia's website, and Cairo responded by filing a declaratory judgment action in federal court in its home state of California. Crossmedia moved to dismiss based upon the forum-selection clause contained in its website's terms of use, which required that any lawsuits be filed in state or federal courts in Chicago. Granting Crossmedia's motion to dismiss, the court held that the terms of use were enforceable, notwithstanding that Cairo had never read them and may not have known that they existed.

Cairo relied upon Specht v. Netscape Communications Corp., 306 F3d 17 (2d Cir 2002), in which the Second Circuit held that users who downloaded Netscape software from Netscape's website were not bound by an agreement to arbitrate disputes, because the users would not have seen Netscape's terms and conditions without scrolling down their computer screens, and there was no reason for them to do so. In Specht, therefore, there was no evidence that the individual users had read the terms and conditions. In Cairo, the court distinguished Specht by pointing out that "Cairo's repeated and automated use of Crossmedia's web pages can form the basis of imputing knowledge to Cairo of the terms on which Crossmedia's services were offered * * *." 2005 WL 756610 at *5.

Comment: As odd as the outcome of this case seems, it is likely to be followed by other courts. The key is that Cairo was a sophisticated user making repeated visits to sites knowing that such sites typically contain terms and conditions prohibiting the very activity Cairo was conducting. Common sense tells us that a party should not be allowed to accomplish through a crawler something that the party itself would be prohibited from doing. Interestingly, the court did not mention a practice arguably relevant to this case: including in your website a notice, readable by crawlers, that requests that the site not be crawled. Good Internet etiquette may dictate that crawlers be set to respect such online "no trespassing" signs but whether this practice rises to the level of "custom or usage in the trade" is another matter. Internet etiquette likely will be ignored when competitors are pillaging each other's sites for valuable information.

One final point: Crossmedia also sued Cairo for copyright infringement, but the court never ruled on this claim, because it dismissed based on the forum-selection clause. The application of copyright law to listings of factual information is an interesting topic in itself; for one thing, it is clear that a lot of data found on websites and in online databases enjoys no copyright protection, because there is not even a small amount of creative expression in presenting the data. Until recently, the conventional wisdom has been that there is no reliable way to protect such online data under contract law without having a user employ a "click-on" acceptance of website terms and conditions. The Cairo ruling, if followed, will open an important new avenue for protection of online data and other content.

Key Contributors

See all contributors See less contributors
Saved Pages

Use the arrows to arrange content.  Download pages as a .pdf file or share links via email..

{{ item.Title }} {{ item.AttorneyPosition }}, {{ item.AttorneyLocation }} , C. {{ item.AttorneyCell }} , P. {{ item.AttorneyPhone }} , F. {{ item.AttorneyFax }} {{ item.TypeText }} Remove
You have no pages saved
            {{ state | json }}