com.norconex.collector.http.handler
Interface IRobotsTxtProvider

All Superinterfaces:
Serializable
All Known Implementing Classes:
DefaultRobotsTxtProvider

public interface IRobotsTxtProvider
extends Serializable

Given a URL, extract any "robots.txt" rules.

Author:
Pascal Essiembre

Method Summary
 RobotsTxt getRobotsTxt(org.apache.http.impl.client.DefaultHttpClient httpClient, String url)
           
 

Method Detail

getRobotsTxt

RobotsTxt getRobotsTxt(org.apache.http.impl.client.DefaultHttpClient httpClient,
                       String url)


Copyright © 2009-2013 Norconex Inc.. All Rights Reserved.