Reference

class urlfetch.Response(r, **kwargs)

A Response object.

>>> import urlfetch
>>> response = urlfetch.get("http://docs.python.org/")
>>> response.total_time
0.033042049407959
>>> response.status, response.reason, response.version
(200, 'OK', 10)
>>> type(response.body), len(response.body)
(<type 'str'>, 8719)
>>> type(response.text), len(response.text)
(<type 'unicode'>, 8719)
>>> response.getheader('server')
'Apache/2.2.16 (Debian)'
>>> response.getheaders()
[
    ('content-length', '8719'),
    ('x-cache', 'MISS from localhost'),
    ('accept-ranges', 'bytes'),
    ('vary', 'Accept-Encoding'),
    ('server', 'Apache/2.2.16 (Debian)'),
    ('last-modified', 'Tue, 26 Jun 2012 19:23:18 GMT'),
    ('connection', 'close'),
    ('etag', '"13cc5e4-220f-4c36507ded580"'),
    ('date', 'Wed, 27 Jun 2012 06:50:30 GMT'),
    ('content-type', 'text/html'),
    ('x-cache-lookup', 'MISS from localhost:8080')
]
>>> response.headers
{
    'content-length': '8719',
    'x-cache': 'MISS from localhost',
    'accept-ranges': 'bytes',
    'vary': 'Accept-Encoding',
    'server': 'Apache/2.2.16 (Debian)',
    'last-modified': 'Tue, 26 Jun 2012 19:23:18 GMT',
    'connection': 'close',
    'etag': '"13cc5e4-220f-4c36507ded580"',
    'date': 'Wed, 27 Jun 2012 06:50:30 GMT',
    'content-type': 'text/html',
    'x-cache-lookup': 'MISS from localhost:8080'
}

Raises:

ContentLimitExceeded

status

Status code returned by server.

status_code

An alias of status.

reason

Reason phrase returned by server.

version

HTTP protocol version used by server. 10 for HTTP/1.0, 11 for HTTP/1.1.

total_time

total time

property ok

True if status is less than 400.

raise_for_status()

Raise HTTPError if status is 400 or higher.

read(chunk_size=65536)

Read content (for streaming and large files)

Parameters:

chunk_size (int) – size of chunk, default is 65536, i.e. 64KiB.

iter_lines(delimiter=None)

Yield decoded chunks split on delimiter (default \n).

classmethod from_httplib(r, **kwargs)

Make an Response object from a httplib response object.

body

Response body.

Raises:

ContentLimitExceeded, ContentDecodingError

property content

encoding

Charset from Content-Type, or None.

text

Response body in str.

Decoded with encoding if Content-Type has a charset, otherwise UTF-8. Raises ContentDecodingError on failure.

json

Load response body as json.

Raises:

ContentDecodingError

headers

Response headers.

Response headers is a dict with all keys in lower case.

>>> import urlfetch
>>> response = urlfetch.get("http://docs.python.org/")
>>> response.headers
{
    'content-length': '8719',
    'x-cache': 'MISS from localhost',
    'accept-ranges': 'bytes',
    'vary': 'Accept-Encoding',
    'server': 'Apache/2.2.16 (Debian)',
    'last-modified': 'Tue, 26 Jun 2012 19:23:18 GMT',
    'connection': 'close',
    'etag': '"13cc5e4-220f-4c36507ded580"',
    'date': 'Wed, 27 Jun 2012 06:50:30 GMT',
    'content-type': 'text/html',
    'x-cache-lookup': 'MISS from localhost:8080'
}

cookies

Cookies in dict

cookiestring

Cookie string

links

Links parsed from HTTP Link header

close()

Close the connection.

class urlfetch.Session(headers=None, cookies=None, auth=None)

A session object.

urlfetch.Session holds common headers and a cookiejar. Cookies from responses, putcookie(), and the cookies argument are only sent back to matching hosts. A cookie without an explicit domain is bound to the host of the next request.

Parameters:

• headers (dict) – Init headers.

• cookies (dict) – Init cookies (bound to the first request host).

• auth (tuple) – (username, password) for basic authentication.

headers

headers

putheader(header, value)

Add an header to default headers.

popheader(header)

Remove an header from default headers.

putcookie(key, value='', domain=None, path='/')

Add a cookie.

If domain is omitted, the cookie is bound to the host of the next request.

popcookie(key)

Remove a cookie from pending cookies and the cookiejar.

property cookies

Cookies in dict (pending cookies plus those from the cookiejar).

property cookiestring

Cookie string.

It’s assignalbe, and will change cookies correspondingly.

>>> s = Session()
>>> s.cookiestring = 'foo=bar; 1=2'
>>> s.cookies
{'1': '2', 'foo': 'bar'}

snapshot()

_bind_pending(url)

_cookie_header(url)

_store_cookies(r)

request(*args, **kwargs)

Issue a request.

fetch(*args, **kwargs)

Fetch an URL

get(*args, **kwargs)

Issue a get request.

post(*args, **kwargs)

Issue a post request.

put(*args, **kwargs)

Issue a put request.

delete(*args, **kwargs)

Issue a delete request.

head(*args, **kwargs)

Issue a head request.

options(*args, **kwargs)

Issue a options request.

trace(*args, **kwargs)

Issue a trace request.

patch(*args, **kwargs)

Issue a patch request.

urlfetch.request(url, method='GET', params=None, data=None, headers=None, timeout=None, files=None, randua=False, auth=None, length_limit=None, proxies=None, trust_env=True, max_redirects=0, source_address=None, validate_certificate=None, ssl_context=None, json=None, **kwargs)

request an URL

Parameters:

• url (string) – URL to be fetched.

• method (string) – (optional) HTTP method, one of GET, DELETE, HEAD, OPTIONS, PUT, POST, TRACE, PATCH. GET is the default.

• params (dict/string) – (optional) Dict or string to attach to url as querystring.

• headers (dict) – (optional) HTTP request headers.

• timeout (float) – (optional) Timeout in seconds

• files – (optional) Files to be sended

• randua – (optional) If True or path string, use a random user-agent in headers, instead of 'urlfetch/' + __version__

• auth (tuple) – (optional) (username, password) for basic authentication

• length_limit (int) – (optional) If None, no limits on content length, if the limit reached raised exception ‘Content length is more than …’

• proxies (dict) – (optional) HTTP proxy, like {‘http’: ‘127.0.0.1:8888’, ‘https’: ‘127.0.0.1:563’}

• trust_env (bool) – (optional) If True, urlfetch will get infomations from env, such as HTTP_PROXY, HTTPS_PROXY

• max_redirects (int) – (integer, optional) Max redirects allowed within a request. Default is 0, which means redirects are not allowed.

• source_address (tuple) – (optional) A tuple of (host, port) to specify the source_address to bind to.

• json – (optional) JSON-serializable object sent as the request body with Content-Type: application/json. Cannot be combined with data or files.

• ssl_context (ssl.SSLContext) – (optional) SSL context for HTTPS connections. Overrides validate_certificate.

• validate_certificate (bool) – (optional) If False, urlfetch skips all the necessary certificate and hostname checks

Returns:

A Response object

Raises:

URLError, UrlfetchException, TooManyRedirects,

urlfetch.fetch(*args, **kwargs)

fetch an URL.

fetch() is a wrapper of request(). It calls get() by default. If one of parameter data or parameter files is supplied, post() is called.

urlfetch.get(url, *, method='GET', params=None, data=None, headers=None, timeout=None, files=None, randua=False, auth=None, length_limit=None, proxies=None, trust_env=True, max_redirects=0, source_address=None, validate_certificate=None, ssl_context=None, json=None, **kwargs)

Issue a get request

urlfetch.post(url, *, method='POST', params=None, data=None, headers=None, timeout=None, files=None, randua=False, auth=None, length_limit=None, proxies=None, trust_env=True, max_redirects=0, source_address=None, validate_certificate=None, ssl_context=None, json=None, **kwargs)

Issue a post request

urlfetch.head(url, *, method='HEAD', params=None, data=None, headers=None, timeout=None, files=None, randua=False, auth=None, length_limit=None, proxies=None, trust_env=True, max_redirects=0, source_address=None, validate_certificate=None, ssl_context=None, json=None, **kwargs)

Issue a head request

urlfetch.put(url, *, method='PUT', params=None, data=None, headers=None, timeout=None, files=None, randua=False, auth=None, length_limit=None, proxies=None, trust_env=True, max_redirects=0, source_address=None, validate_certificate=None, ssl_context=None, json=None, **kwargs)

Issue a put request

urlfetch.delete(url, *, method='DELETE', params=None, data=None, headers=None, timeout=None, files=None, randua=False, auth=None, length_limit=None, proxies=None, trust_env=True, max_redirects=0, source_address=None, validate_certificate=None, ssl_context=None, json=None, **kwargs)

Issue a delete request

urlfetch.options(url, *, method='OPTIONS', params=None, data=None, headers=None, timeout=None, files=None, randua=False, auth=None, length_limit=None, proxies=None, trust_env=True, max_redirects=0, source_address=None, validate_certificate=None, ssl_context=None, json=None, **kwargs)

Issue a options request

urlfetch.trace(url, *, method='TRACE', params=None, data=None, headers=None, timeout=None, files=None, randua=False, auth=None, length_limit=None, proxies=None, trust_env=True, max_redirects=0, source_address=None, validate_certificate=None, ssl_context=None, json=None, **kwargs)

Issue a trace request

urlfetch.patch(url, *, method='PATCH', params=None, data=None, headers=None, timeout=None, files=None, randua=False, auth=None, length_limit=None, proxies=None, trust_env=True, max_redirects=0, source_address=None, validate_certificate=None, ssl_context=None, json=None, **kwargs)

Issue a patch request

Exceptions

class urlfetch.UrlfetchException

Base exception. All exceptions and errors will subclass from this.

class urlfetch.ContentLimitExceeded

Content length is beyond the limit.

class urlfetch.URLError

Error parsing or handling the URL.

class urlfetch.ContentDecodingError

Failed to decode the content.

class urlfetch.TooManyRedirects

Too many redirects.

class urlfetch.Timeout

Request timed out.

class urlfetch.HTTPError(response)

HTTP status code is 400 or higher.

helpers

urlfetch.parse_url(url)

Return a dictionary of parsed url

Including scheme, netloc, path, params, query, fragment, uri, username, password, host, port and http_host

urlfetch.get_proxies_from_environ()

Get proxies from os.environ.

urlfetch.mb_code(s, coding=None, errors='replace')

encoding/decoding helper.

urlfetch.random_useragent(filename=True)

Returns a User-Agent string randomly from file.

Parameters:

filename (string) – (Optional) Path to the file from which a random useragent is generated. By default it’s True, a file shipped with this module will be used.

Returns:

An user-agent string.

urlfetch.url_concat(url, args, keep_existing=True)

Concatenate url and argument dictionary

>>> url_concat("http://example.com/foo?a=b", dict(c="d"))
'http://example.com/foo?a=b&c=d'

Parameters:

• url (string) – URL being concat to.

• args (dict) – Args being concat.

• keep_existing (bool) – (Optional) Whether to keep the args which are alreay in url, default is True.

urlfetch.choose_boundary()

Generate a multipart boundry.

Returns:

A boundary string

urlfetch.encode_multipart(data, files)

Encode multipart.

Parameters:

• data (dict) – Data to be encoded

• files (dict) – Files to be encoded

Returns:

Encoded binary string

Raises:

UrlfetchException