Changeset 3364

Show
Ignore:
Timestamp:
01/13/10 23:09:50 (10 years ago)
Author:
yemuru
Message:

New version of registerhttpcallback.repy. supports different type of raising exception.

Location:
seattle/trunk/seattlelib
Files:
2 modified

Legend:

Unmodified
Added
Removed
  • seattle/trunk/seattlelib/httpretrieve.repy

    r3354 r3364  
    1010 
    1111<Purpose> 
    12   Provides a method for retrieving content from web servers using the HTTP 
    13   protocol. The content can be accessed as a file like object, or saved to 
    14   a file or returned as a string. 
     12  provides a http content from a web server using http protocol. It sends a http request to 
     13  any http server through socket connection to get the http content. Then once the http server 
     14  replies with http header and content, the http header is checked for any error message. 
     15  Then provides http content in a format of string, saved in a file or as a file like object.  
    1516""" 
    1617 
    1718 
    1819 
    19 include urlparse.repy 
     20 
     21include urlparse.repy   
    2022include sockettimeout.repy 
     23include http_hierarchy_error.repy 
    2124include urllib.repy 
    2225 
    2326 
    2427 
    25 class HttpConnectionError(Exception): 
    26   """ 
    27   Error indicating that the web server has unexpectedly dropped the 
    28   connection. 
    29   """ 
    30  
    31  
    32  
    33  
    34 class HttpBrokenServerError(Exception): 
    35   """ 
    36   Error indicating that the web server has sent us complete garbage instead 
    37   of something resembling HTTP. 
    38   """ 
    39  
    40  
    41  
    42  
    43 def httpretrieve_open(url, postdata=None, querydata=None, \ 
    44     httpheaders=None, timeout=None): 
     28 
     29 
     30def httpretrieve_open(url, http_query=None, http_post=None, http_header=None, header_timeout=30, content_timeout=30, httpheader_limit=8192, httpcontent_limit=4194304): 
    4531  """ 
    4632  <Purpose> 
    47      Returns a file-like object that can be used to read the content from 
    48      an HTTP server. Follows 3xx redirects. 
     33     Returns file like object that that can read the http content form a http server. The file like 
     34     object gets string from http server using read method.   
    4935 
    5036  <Arguments> 
    5137    url: 
    52            The URL to perform a GET or POST request on. 
    53     postdata (optional): 
    54            A dictionary of form data or a string to POST to the server. 
    55            Passing a non-None value results in a POST request being sent 
    56            to the server. 
    57     querydata (optional): 
    58            A dictionary of form data or a string to send as the query 
    59            string to the server. 
    60  
    61            If postdata is omitted, the URL is retrieved with GET. If 
    62            both postdata and querydata are omitted, there is no query 
    63            string sent in the request. 
    64  
    65            For both querydata and postdata, strings are sent *unmodified*. 
    66            This means you probably should encode them first, with 
    67            urllib_quote(). 
    68     httpheaders (optional): 
    69            A dictionary of supplemental HTTP request headers to add to the 
    70            request. 
    71     timeout (optional): 
    72            A timeout for establishing a connection to the web server, 
    73            sending headers, and reading the response headers. 
    74  
    75            If excluded or None, never times out. 
    76  
     38           String of a http web server-URL 
     39    http_post: 
     40           dictionary of data to post to server(unencoded string, the library encodes the post it self) 
     41    http_query: 
     42           dictionary of query to send to server(unencoded string, the library encodes the query it self) 
     43    http_header: 
     44           dictionary of http header to add the the http header request 
     45    header_timeout: 
     46           socket timeout for receiving header from server(default value set to 30 seconds) 
     47    content_timeout: 
     48           socket timeout for receiving content from server(default value set to 30 seconds) 
     49    httpheader_limit: 
     50           length limit for when a server sends http header(default value set to 8kb(8192 charactors))  
     51    httpcontent_limit: 
     52            limits the the amount of content a server can send to the retrieval  
     53  
    7754  <Exceptions> 
    78     ValueError if given an invalid URL, or malformed limit or timeout 
    79       values. This is also raised if the user attempts to call a method 
    80       on the file-like object after closing it. 
    81  
    82     HttpConnectionError if opening the connection fails, or if the 
    83       connection is closed by the server before we expect. 
    84  
    85     SocketTimeoutError if the timeout is exceeded. 
    86  
    87     HttpBrokenServerError if the response or the Location response header 
    88       is malformed. 
     55        HttpUserInputError 
     56            ->  If given a invalid URL  
     57            ->  If given a none http protocol server 
     58            ->  if file like object read is given a negative number or a none int as a limit 
     59            ->  if the file like object is called after it is closed 
     60 
     61        HttpConnectionError 
     62            ->  if opening connection with server fails       
     63            ->  if sending http request to http server fails   
     64 
     65        HttpHeaderReceivingError 
     66            ->  If the timeout(default timeout set to 5 seconds) for receiving exceeds    
     67            ->  If the http header length exceeds (default  
     68             
     69        HttpHeaderFormatError 
     70            ->  If The http header is too long(default set to 8 kb(8192 charactors)) 
     71            ->  If the http header statuscode format is invalid. The right http header 
     72                    status code includes HTTP<version> http_status_number http_status_msg                  
     73            ->  If the http server gives a http header with a invalid content length 
     74                    or invalid redirect location 
     75             
     76        HttpContentReceivingError: 
     77            ->  if the http server fails to send http content  
     78            ->  if the timeout(default timeout set to 5 seconds) for receiving exceeds 
     79            ->  if the server socket connection fails for any reason besides connection closing 
     80                    during receiving http content 
     81                         
     82        HttpContentLengthError: 
     83            ->  if content length exceeds(default set to 4096 kb(4194304 charactors))  
     84                    (ONLY WORKS CONTENT LENGTH IS GIVEN BY THE HTTP SERVER) 
     85            ->  If the total received length is not the same as the content length(this check will fail 
     86                     if the content length isnt given) 
     87            ->  If read is called with limits and it returns a empty string but the total read is                                       
     88                     not equal to the content length(this check will fail if the content length isnt given)                 
     89                                             
     90        HttpStatuscodeError: 
     91            -> if the http response status code isnt ok or redirect, it will raise an exception 
     92                   depending up on the http protocol status number   
     93         
    8994 
    9095  <Side Effects> 
    91     None 
     96    None  
    9297 
    9398  <Returns> 
    94     Returns a file-like object which can be used to read the body of 
    95     the response from the web server. The protocol version spoken by the 
    96     server, status code, and response headers are available as members of 
    97     the object. 
     99    Returns file like obj which can read the http content from http web server.  
    98100  """ 
    99  
    100   # TODO: Make sure the timeout actually works correctly everywhere it 
    101   # should. I'm 99% sure it's broken somewhere. 
    102  
    103   starttime = getruntime() 
    104  
    105   # Check if the URL is valid and get host, path, port and query 
    106   parsedurl = urlparse_urlsplit(url) 
    107   host = parsedurl['hostname'] 
    108   path = parsedurl['path'] 
    109   port = parsedurl.get('port') 
    110   port = port or 80 
    111  
    112   if parsedurl['scheme'] != 'http': 
    113     raise ValueError("URL doesn't seem to be for the HTTP protocol.") 
    114   if host is None: 
    115     raise ValueError("Missing hostname.") 
    116   if parsedurl['query'] is not None and parsedurl['query'] != "": 
    117     raise ValueError("URL cannot include a query string.") 
    118  
    119   # Open connection to the web server 
     101   
     102  # check if url is valid and get host, path, port and query from the given url 
     103  (host, port, path, url_query) = _httpretrieve_parse_given_url(url) 
     104 
     105  # get connection to the http web server 
    120106  try: 
    121107    sock = timeout_openconn(host, port) 
    122  
     108     
    123109  except Exception, e: 
    124     if repr(e).startswith("timeout("): 
    125       raise HttpConnectionError("Socket timed out connecting to host/port.") 
    126     raise 
    127  
    128   # build an HTTP request using the given port, host, path and query 
    129   method = "GET" 
    130   if postdata is not None: 
    131     method = "POST" 
    132     if type(postdata) is dict: 
    133       postdata = urllib_quote_parameters(postdata) 
    134     if type(postdata) is not str: 
    135       raise TypeError("postdata should be a dict of form data or string") 
     110    raise HttpConnectionError('Error: opening a connection failed with given http server, Given: ' + str(url) + ' ' + str(e)) 
     111     
     112  # build a http format request using the given port, host, path and query 
     113  httpheader = _httpretrieve_buildhttprequest(http_header, port, host, path, url_query, http_query, http_post) 
     114   
     115  # send http format request to http web server   
     116  _httpretrieve_sendhttprequest(sock, httpheader)  
     117   
     118  # receive the http header lines in a form of list from the http web server 
     119  httpheaderlines = _httpretrieve_receive_httpheader(sock, header_timeout, httpheader_limit) 
     120   
     121  # get the http status number and http status msg from http header response 
     122  (http_status_number, http_status_msg) = _httpretrieve_get_httpstatuscode(httpheaderlines) 
     123 
     124  if http_status_number == '200':# ok message 
     125    # gets the content length if given. 
     126    contentlength = _httpretrieve_get_contentlength(httpheaderlines) 
     127    # return a filelikeobj to read the http content from the http server  
     128    return _httpretrieve_filelikeobject(sock, contentlength, httpcontent_limit, content_timeout)  
     129   
     130  elif http_status_number == '301' or http_status_number == '302': # redirect 
     131    # redirect to the new location via recursion   
     132    sock.close() 
     133    # get the redirection location 
     134    redirect_location = _httpretrieve_httpredirect(httpheaderlines)  
     135    # redirect to the new location using recursion 
     136    return httpretrieve_open(redirect_location)      
     137 
    136138  else: 
    137     postdata = "" 
    138  
    139   if path == "": 
    140     raise ValueError("Invalid path -- empty string.") 
    141   if type(querydata) is dict: 
    142     querydata = urllib_quote_parameters(querydata) 
    143   if type(querydata) is str and querydata != "": 
    144     encquerydata = "?" + querydata 
    145   else: 
    146     encquerydata = "" 
    147  
    148   httpheader = method + ' ' + path + encquerydata + ' HTTP/1.0\r\n' 
    149   if httpheaders is not None: 
    150     if type(httpheaders) is not dict: 
    151       raise TypeError("Expected HTTP headers as a dictionary.") 
    152     else: 
    153       if "Host" not in httpheaders: 
    154         httpheader += "Host: " + host + ':' + str(port) + "\r\n" 
    155       for key, val in httpheaders.items(): 
    156         httpheader += key + ": " + val + '\r\n' 
    157  
    158   if method == "POST": 
    159     httpheader += 'Content-Length: ' + str(len(postdata)) + '\r\n' 
    160   httpheader += '\r\n' 
    161   if method == "POST": 
    162     httpheader += postdata 
    163  
    164   # send HTTP request to the web server 
    165   sock.send(httpheader) 
    166  
    167   # receive the header lines from the web server 
    168   if timeout is None: 
    169     sock.settimeout(0) 
    170   elif getruntime() - starttime >= timeout: 
    171     raise SocketTimeoutError("Timed out") 
    172   else: 
    173     sock.settimeout(timeout - (getruntime() - starttime)) 
    174  
    175   headers_str = "" 
    176   while True: 
    177     try: 
    178       headers_str += sock.recv(1) 
    179       if headers_str.endswith("\r\n\r\n"): 
    180         break 
    181     except Exception, e: 
    182       if str(e) == "Socket closed": 
    183         break 
    184       else: 
    185         raise 
    186  
    187   httpheaderlines = headers_str.split("\r\n") 
    188   while httpheaderlines[-1] == "": 
    189     httpheaderlines = httpheaderlines[:-1] 
    190  
    191   # get the status code and status message from the HTTP response 
    192   statusline, httpheaderlines = httpheaderlines[0], httpheaderlines[1:] 
    193   headersplit = statusline.split(' ', 2) 
    194   if len(headersplit) < 3: 
    195     raise HttpBrokenServerError("Server returned garbage for HTTP response.") 
    196   if not headersplit[0].startswith('HTTP'): 
    197     raise HttpBrokenServerError("Server returned garbage for HTTP response.") 
    198   statusmsg = headersplit[2] 
    199   try: 
    200     statusnum = int(headersplit[1]) 
    201   except ValueError, e: 
    202     raise HttpBrokenServerError("Server returned garbage for HTTP response.") 
    203  
    204   responseheaders = _httpretrieve_parse_responseheaders(httpheaderlines) 
    205  
    206   if statusnum == 301 or statusnum == 302: 
    207     # redirect to the new location via recursion 
    208     sock.close() 
    209     try: 
    210       redirect_location = responseheaders["Location"][0] 
    211     except (KeyError, IndexError), ke: 
    212       raise HttpBrokenServerError("Server returned garbage for HTTP" + \ 
    213           " response. Redirect response missing Location header.") 
    214     else: 
    215       return httpretrieve_open(redirect_location) 
    216  
    217   else: 
    218     return _httpretrieve_filelikeobject(sock, responseheaders, \ 
    219         (headersplit[0], statusnum, statusmsg)) 
    220  
    221  
    222  
    223  
    224 def httpretrieve_save_file(url, filename, querydata=None, postdata=None, \ 
    225     httpheaders=None, timeout=None): 
     139    # if given receive content length inorder to check http content error is received fully 
     140    contentlength = _httpretrieve_get_contentlength(httpheaderlines) 
     141    # receive the http content error  
     142    http_errorcontent = _httpretrieve_receive_httperror_content(sock, contentlength) 
     143    # raise exception depending up on the http status number and add on the http error content after a 
     144    # discription that says 'Http error content: ' 
     145    _httpretrieve_raise_httpstatuscode_error(http_status_number, http_status_msg, http_errorcontent)    
     146 
     147 
     148   
     149 
     150 
     151def httpretrieve_save_file(url, filename, http_query=None, http_post=None, http_header=None, header_timeout=30, content_timeout=30, httpheader_limit=8192, httpcontent_limit=4194304): 
    226152  """ 
    227153  <Purpose> 
    228     Perform an HTTP request, and save the content of the response to a 
    229     file. 
     154     Saves http content of the given URL to current directory   
    230155 
    231156  <Arguments> 
     157    url: 
     158           String of a http web server URL 
    232159    filename: 
    233            The file name to save the response to. 
    234     Other arguments: 
    235            See documentation for httpretrieve_open(). 
    236  
     160           The file name for the http content to be saved in 
     161    http_post: 
     162           dictionary of data to post to server(unencoded string, the library encodes the post it self) 
     163    http_query: 
     164           dictionary of query to send to server(unencoded string, the library encodes the query it self) 
     165    http_header: 
     166           dictionary of http header to add the the http header request 
     167    header_timeout: 
     168           socket timeout for receiving header from server(default value set to 30 seconds) 
     169    content_timeout: 
     170           socket timeout for receiving content from server(default value set to 30 seconds) 
     171    httpheader_limit: 
     172           length limit for when a server sends http header(default value set to 8kb(8192 charactors))  
     173    httpcontent_limit: 
     174            limits the the amount of content a server can send to the retrieval  
     175            
    237176  <Exceptions> 
    238     This function will raise any exception raised by Repy file objects 
    239     in opening, writing to, and closing the file. 
    240  
    241     This function will all also raise any exception raised by 
    242     httpretrieve_open(), for the same reasons. 
     177   
     178    HttpRetrieveClientError: cant create a file to save the http content too 
     179    Also includes:  all the exception from httpretrieve_open  
    243180 
    244181  <Side Effects> 
    245     Writes the body of the response to 'filename'. 
     182    same as httpretrieve_open   
    246183 
    247184  <Returns> 
    248     None 
     185    None  
    249186  """ 
    250  
     187   
    251188  httpcontent = '' 
    252   newfile = open(filename, 'w') 
    253  
    254   http_obj = httpretrieve_open(url, querydata=querydata, postdata=postdata, \ 
    255       httpheaders=httpheaders, timeout=timeout) 
    256  
    257   # Read from the file-like HTTP object into our file. 
     189  try: 
     190    # create a new file with the given filename 
     191    newfile = open(filename, 'w') 
     192  except Exception, e: 
     193    raise HttpRetrieveClientError('Error on creating a file to saving http content' + str(e)) 
     194 
     195  http_obj = httpretrieve_open(url, http_query, http_post, http_header, header_timeout, content_timeout, httpheader_limit, httpcontent_limit) 
     196 
     197  # keep on reading 1024 and writing it into a file, until it receives an empty string 
     198  # which means the http server content is completely read  
    258199  while True: 
    259     httpcontent = http_obj.read(4096) 
     200    httpcontent = http_obj.read(1024) 
    260201    if httpcontent == '': 
    261       # we're done reading 
    262       newfile.close() 
     202      # done reading close file and file like obj and exit loop  
     203      newfile.close()   
    263204      http_obj.close() 
    264205      break 
     
    266207 
    267208 
    268  
    269  
    270 def httpretrieve_get_string(url, querydata=None, postdata=None, \ 
    271     httpheaders=None, timeout=30): 
     209   
     210def httpretrieve_get_string(url, http_query=None, http_post=None, http_header=None, header_timeout=30, content_timeout=30, httpheader_limit=8192, httpcontent_limit=4194304): 
    272211  """ 
    273212  <Purpose> 
    274     Performs an HTTP request on the given URL, using POST or GET, 
    275     returning the content of the response as a string. Uses 
    276     httpretrieve_open. 
     213     retruns string of the http content from a given URL 
    277214 
    278215  <Arguments> 
    279     See httpretrieve_open. 
    280  
     216    url: 
     217           String of a http web server-URL 
     218    http_post: 
     219           dictionary of data to post to server(unencoded string, the library encodes the post it self) 
     220    http_query: 
     221           dictionary of query to send to server(unencoded string, the library encodes the query it self) 
     222    http_header: 
     223           dictionary of http header to add the the http header request 
     224    header_timeout: 
     225           socket timeout for receiving header from server(default value set to 30 seconds) 
     226    content_timeout: 
     227           socket timeout for receiving content from server(default value set to 30 seconds) 
     228    httpheader_limit: 
     229           length limit for when a server sends http header(default value set to 8kb(8192 charactors))  
     230    httpcontent_limit: 
     231            limits the the amount of content a server can send to the retrieval  
     232  
    281233  <Exceptions> 
    282     See httpretrieve_open. 
     234     same as httpretrieve_open 
    283235 
    284236  <Side Effects> 
    285     None. 
     237     same as httpretrieve_open  
    286238 
    287239  <Returns> 
    288     Returns the body of the HTTP response (no headers). 
    289   """ 
    290  
    291   http_obj = httpretrieve_open(url, querydata=querydata, postdata=postdata, \ 
    292       httpheaders=httpheaders, timeout=timeout) 
     240     returns a string of the http content.  
     241  """        
     242   
     243  http_obj = httpretrieve_open(url, http_query, http_post, http_header, header_timeout, content_timeout, httpheader_limit, httpcontent_limit) 
     244  # retrieve the http content from server using httpretrieve file like object 
    293245  httpcontent = http_obj.read() 
    294246  http_obj.close() 
     247  # return the http content in a form of string 
    295248  return httpcontent 
    296249 
    297250 
    298251 
    299  
    300252class _httpretrieve_filelikeobject: 
    301   # This class implements a file-like object used for performing HTTP 
    302   # requests and retrieving responses. 
    303  
    304   def __init__(self, sock, headers, httpstatus): 
    305     self._sock = sock 
    306     self._fileobjclosed = False 
    307     self._totalcontentisreceived = False 
    308     self._totalread = 0 
    309     self.headers = headers 
    310     self.httpstatus = httpstatus 
    311  
    312  
    313  
    314   def read(self, limit=None, timeout=None): 
     253  # file like object used to receive the http content with a length limit    
     254  def __init__(self, sock, contentlength, httpcontent_limit, content_timeout): 
     255    self.sock = sock 
     256    if contentlength == None: 
     257      self.contentlengthisknown = False 
     258    else: 
     259      self.contentlengthisknown = True 
     260      self.contentlength = contentlength 
     261    self.httpcontent_limit = httpcontent_limit   
     262    self.content_timeout = content_timeout   
     263    self.fileobjclosed = False 
     264    self.totalcontentisreceived = False  
     265    self.totalread = 0 
     266     
     267 
     268  def read(self, limit = None): 
    315269    """ 
    316270    <Purpose> 
    317       Behaves like Python's file.read(), with the potential to raise 
    318       additional informative exceptions. 
     271        reads the http content from http server using the file like object    
    319272 
    320273    <Arguments> 
    321       limit (optional): 
    322             The maximum amount of data to read. If omitted or None, this 
    323             reads all available data. 
    324  
     274        limit(optional): 
     275             maximum number of bytes to read. If not specified the whole file is read. 
     276             Can be 0 or any positive int 
     277    
    325278    <Exceptions> 
    326       See file.read()'s documentation, as well as that of 
    327       httpretrieve_open(). 
     279 
     280        HttpContentReceivingError: 
     281            ->  if the http server fails to send http content  
     282            ->  if the timeout(default timeout set to 5 seconds) for receiving exceeds 
     283            ->  if the server socket connection fails for any reason besides connection closing 
     284                    during receiving http content 
     285        HttpContentLengthError: 
     286            ->  if content length exceeds(default set to 4096 kb(4194304 charactors)) 
     287                    (ONLY WORKS CONTENT LENGTH IS GIVEN BY THE HTTP SERVER) 
     288            ->  If the total received length is not the same as the content length(this check will fail 
     289                     if the content length isnt given) 
     290            ->  If read is called with limits and it returns a empty string but the total read is                                       
     291                     not equal to the content length(this check will fail if the content length isnt given) 
     292                      
     293        HttpStatuscodeError: 
     294            if the http response status code isnt ok or redirect, it will raise an exception 
     295            depending up on the http protocol status number   
     296 
    328297 
    329298    <Side Effects> 
    330       None. 
     299       None 
     300 
    331301 
    332302    <Returns> 
    333       See file.read(). 
     303      returns the content of http server in a form of string or an empty string if the content is completely read. 
    334304    """ 
    335  
    336     if self._fileobjclosed == True: 
    337       raise ValueError("I/O operation on closed file") 
    338  
    339     if self._totalcontentisreceived: 
     305     
     306    # raises an exception, if read is called after the filelikeobj is closed  
     307    if self.fileobjclosed == True: 
     308      raise HttpUserInputError('Http Error: filelikeobj is closed') 
     309 
     310    # if read is called after all http content received return an empty string 
     311    if self.totalcontentisreceived: 
    340312      return '' 
    341  
    342     if limit is not None: 
    343       # Sanity check type/value of limit 
    344       if type(limit) is not int: 
    345         raise TypeError("Expected an integer or None for limit") 
     313    
     314    # check if limit is given 
     315    if limit == None:  
     316      readhaslimit = False  
     317      left_to_read = 1024 
     318    else:  
     319    # check if limit is a valid number 
     320      if not type(limit) == int: 
     321        raise HttpUserInputError('User input Error: given a none int to receive' + str(e))   
    346322      elif limit < 0: 
    347         raise ValueError("Expected a non-negative integer for limit") 
    348  
    349       lefttoread = limit 
    350     else: 
    351       lefttoread = None 
    352  
    353     if timeout is None: 
    354       self._sock.settimeout(0) 
    355     else: 
    356       self._sock.settimeout(timeout) 
    357  
    358     # Try to read up to limit, or until there is nothing left. 
     323        # raise an exception if limit is a negative number 
     324        raise HttpUserInputError('User input Error: given a negative number to receive, given: ' + str(limit)) 
     325      readhaslimit = True 
     326      left_to_read = limit   
     327 
     328    # set a timeout for receiveing content from server  
     329    self.sock.settimeout(self.content_timeout) 
     330 
     331    # if limit is given it will receiveby subtracting what is left until the limit is reached  
     332    # if limit isnt given it will receive1024 until the server closes connection        
    359333    httpcontent = '' 
    360334    while True: 
    361335      try: 
    362         content = self._sock.recv(lefttoread or 4096) 
    363       except Exception, e: 
    364         if str(e) == "Socket closed": 
    365           self._totalcontentisreceived = True 
    366           break 
    367         else: 
    368           raise 
    369        
    370       httpcontent += content 
    371       self._totalread += len(content) 
    372       if limit is not None: 
    373         if len(content) == lefttoread: 
    374           break 
    375         else: 
    376           lefttoread -= len(content) 
    377       if content == "": 
    378         self._totalcontentisreceived = True 
     336        content = self.sock.recv(left_to_read) 
     337 
     338      except SocketTimeoutError: 
     339        # raise an exception if receiveis taking too long to respond 
     340        self.sock.close()   
     341        raise HttpContentReceivingError('Timeout Error on receiving content: server taking too long to send content')   
     342 
     343      except Exception , e: 
     344        # socket closed - signal for when the server is done sending content 
     345        # if there is any other exceptions close connection and raise an error   
     346        if 'Socket closed' not in str(e):  
     347          self.sock.close()             
     348          raise HttpContentReceivingError('Error on receiving content:' + str(e)) 
     349        self.totalcontentisreceived = True 
    379350        break 
    380351 
     352      else: 
     353        # By default, httpretrieve permits content length to be less than 4,096 kilobytes(4194304 charactors) 
     354        if len(content) >= self.httpcontent_limit: 
     355          raise HttpContentLengthError('content length exceeded ' + self.httpcontent_limit) 
     356 
     357        # add what is received 
     358        httpcontent += content 
     359        if readhaslimit: 
     360          # keep subtracting what is left to receieve until it reachs the given limit amount 
     361          self.totalread += len(content) 
     362          if len(content) == left_to_read: 
     363            break 
     364          else: 
     365            left_to_read -= len(content) 
     366 
     367    # check if there was an error during reciving http content 
     368    self._check_recieving_error(readhaslimit, httpcontent) 
     369 
    381370    return httpcontent 
    382  
    383  
     371   
    384372 
    385373  def close(self): 
    386374    """ 
    387375    <Purpose> 
    388       Close the file-like object. 
     376      close the file like object  
    389377 
    390378    <Arguments> 
    391379      None 
    392  
     380    
    393381    <Exceptions> 
    394       None 
     382      None  
    395383 
    396384    <Side Effects> 
    397       Disconnects from the HTTP server. 
     385      closes socket connection for the http client to http server 
    398386 
    399387    <Returns> 
    400       Nothing 
    401     """ 
    402     self._fileobjclosed = True 
     388      Nothing  
     389    """    
     390    self.fileobjclosed = True# flag used to raise an exception if the file like object is called after closed 
     391    self.sock.close() 
     392 
     393 
     394  def _check_recieving_error(self, readhaslimit, httpcontent): 
     395    if len(httpcontent) == 0: 
     396      self.sock.close() 
     397      raise HttpContentLengthError('Error on recieving content: received a http header but didnt receive any http content') 
     398     
     399    if self.contentlengthisknown:                     
     400      # if limit is given and content length is given and total content is received, check the total read equals the content length     
     401      if readhaslimit and self.totalcontentisreceived: 
     402        if self.totalread != self.contentlength: 
     403          self.sock.close()                 
     404          raise HttpContentLengthError('Total length read with limit did not match the content length: total read: ' + str(self.totalread) + ' content length: ' + str(self.contentlength)) 
     405 
     406      # if called read without limit and content length is given; check if the received length is the same as the content length        
     407      if readhaslimit == False: 
     408        if len(httpcontent) != self.contentlength: 
     409          self.sock.close() 
     410          raise HttpContentLengthError('Total received length did not match the content length: received: ' + str(len(httpcontent)) + ' content length : ' + str(self.contentlength))      
     411 
     412 
     413 
     414 
     415 
     416def _httpretrieve_parse_given_url(url): 
     417  # checks if the URL is in the right format and returns a string of host, port, path and query by parsing the URL   
     418  try: 
     419   # returns a dictionary of {scheme, netloc, path, quer, fragment, username, password, hostname and port} form the url                      
     420    urlparse = urlparse_urlsplit(url)   
     421  except Exception, e: 
     422    raise HttpUserInputError('Given URL error: ' + str(e)) 
     423  else: 
     424    # check if the protocol is http  
     425    if urlparse['scheme'] != 'http': 
     426      raise HttpUserInputError('Given URL error: the given protocol ' + urlparse['scheme'] + ' isnt supported')        
     427    if urlparse['hostname'] == None: 
     428      raise HttpUserInputError('Given URL error: host name is not given')  
     429 
     430    # get only the host path, port, query from the urlparse dictionary 
     431    host = urlparse['hostname'] 
     432    path = urlparse['path'] 
     433    query = urlparse['query'] 
     434     
     435    # use default port 80 if the port isnt given                     
     436    if urlparse['port'] == None: 
     437      port = 80  
     438    else: 
     439      port = urlparse['port'] 
     440 
     441    return host, port, path, query 
     442 
     443 
     444 
     445 
     446 
     447def _httpretrieve_buildhttprequest(http_header, port, host, path, url_query, dict_query, http_post): 
     448  # send http request to the http web server using socket connection   
     449   
     450  if http_post != None: 
     451    # there is a posted data, thus use http POST command 
     452    
     453    # check if the given post data is valid 
     454    if not type(http_post) == dict: 
     455      raise HttpUserInputError('The given http_post is not a dictionary, given: ' + str(type(http_post))) 
     456 
     457    # change the given http post dictionary into a encoded post data with a key and val   
     458    try:  
     459      http_post = urllib_quote_parameters(http_post) 
     460    except Exception, e: 
     461      raise HttpUserInputError('Error encoding the given http post dictionary ' +  str(http_post) + str(e)) 
     462 
     463 
     464    # build the main http request header which includes the GET/POST and the Host name field 
     465    httpheader = _httpretrieve_httprequestmain_header('POST', url_query, dict_query, path, host, port) 
     466 
     467    # if given add a client http header to the request 
     468    httpheader += _httpretrieve_parse_clienthttpheader(http_header) 
     469 
     470    # indicate the http post content length 
     471    httpheader += 'Content-Length: ' + str(len(http_post)) + '\r\n'   
     472    # add a new line to indicate that the http header is done and the http post is followed. 
     473    httpheader += '\r\n' 
     474    # include the posted data after the http header empty line 
     475    httpheader += http_post 
     476         
     477 
     478  else: 
     479    # there is no posted data, use http GET method    
     480    httpheader = _httpretrieve_httprequestmain_header('GET', url_query, dict_query, path, host, port) 
     481    # if http header is given, add client headers to the request  
     482    httpheader += _httpretrieve_parse_clienthttpheader(http_header) 
     483    # add a new line to indicate that the header request is complete 
     484    httpheader += '\r\n' 
     485 
     486  # return header with a new line which is signal for http header is done  
     487  return httpheader  
     488 
     489 
     490 
     491 
     492def _httpretrieve_httprequestmain_header(http_command, url_query, dict_query, path, host, port): 
     493  # builds the first two main http request headers which include the GET/POST and the HOST name 
     494   
     495  # before building the httprequest make sure there isnt two fields of query given by the client 
     496  if url_query != '' and dict_query != None: 
     497    # cant have two different fields with query 
     498    raise HttpUserInputError('Cant input a http query with the url and an extra parameter dictionary with a http query') 
     499 
     500  elif dict_query != None: 
     501    # user has given a http query  
     502    try:  
     503      encoded_query = '?' + urllib_quote_parameters(dict_query) 
     504    except Exception, e: 
     505      raise HttpUserInputError('Error encoding the given http query dictionary ' +  str(dict_query) + str(e)) 
     506 
     507  elif url_query != '': 
     508    # if there is a query include the query on the main header('?' is used as a seperater between path) 
     509    encoded_query = '?' + url_query 
     510  else: 
     511    # there is no query 
     512    encoded_query = '' 
     513     
     514 
     515  # if there is no path add a '/' on the request and if there is a path use the given path 
     516  addpath = '/' 
     517  if path != '': 
     518    addpath = path 
     519 
     520  # FIRST header which includes the POST/GET request   
     521  main_httpheader = http_command + ' ' + addpath + encoded_query + ' HTTP/1.0\r\n' 
     522 
     523 
     524  # if port is 80, dont need to include the port upon request 
     525  addport = '' 
     526  if port != 80: 
     527    # if the port is not 80 the host needs to include the port number on Host header 
     528    # (':' is used as a separater between host and port) 
     529    addport = ':' + str(port) 
     530 
     531  # SECOND line of the header request which include the host name with port if the port is not 80    
     532  main_httpheader += 'Host: ' + host + addport + '\r\n' 
     533 
     534  # return the firs two lines of the http request  
     535  return main_httpheader 
     536 
     537 
     538 
     539 
     540def _httpretrieve_parse_clienthttpheader(http_header): 
     541  # builds a http header from the given http header dictionary 
     542  if http_header == None: 
     543    # if the http header isnt given return a empty string 
     544    return '' 
     545  elif not type(http_header) == dict: 
     546    # raise an exception if the http header isnt dictionary 
     547    raise HttpUserInputError('The given http_post is not a dictionary, given: ' + str(type(http_header))) 
     548  else:  
     549    # take the given key and val from the http_header dictionary and add them to the http header with 
     550    # correct http format 
     551    clienthttpheader = '' 
     552    for key, val in http_header.items(): 
     553      # make simple checks on the key and val  
     554      if not type(key) == str: 
     555        raise HttpUserInputError('The given http header key isnt str given ' + str(type(key))) 
     556      if not type(val) == str: 
     557        raise HttpUserInputError('The given http header value isnt str given ' + str( type(val))) 
     558      if key == '' or val == '': 
     559        # raise an exception if the key or value is a empty field 
     560        raise HttpUserInputError('The given empty http header feild of key or value') 
     561      if key.capitalize() != key: 
     562        raise HttpUserInputError('The given http header key is not capitalized, given: ' + key) 
     563 
     564      # add the key and value to the http header field 
     565      clienthttpheader += key + ' : ' + val + '\r\n'   
     566 
     567    # return the string of the http header   
     568    return clienthttpheader 
     569 
     570 
     571 
     572 
     573def _httpretrieve_sendhttprequest(sock, httpheader): 
     574  # send the request, and if there is any error raise an excetion 
     575  try: 
     576    sock.send(httpheader) 
     577  except Exception, e: 
     578    if 'Socket closed' not in str(e):  
     579      sock.close() 
     580    raise HttpConnectionError('Connection error: on sending http request to server ' + str(e)) 
     581 
     582 
     583 
     584   
     585def _httpretrieve_receive_httpheader(sock, header_timeout, httpheader_limit): 
     586  # receives the http header leaving alone rest of the http response and returns in list 
     587  # of each header as a line. default httpheader limit is set to 8 kb                         
     588 
     589  # set a time out if the server fails to send http header  
     590  sock.settimeout(header_timeout) 
     591 
     592  httpheader_received = 0  
     593  httpheader = ''  
     594  while True: 
     595    # receive until a empty line (\n\n or \r\n\r\n ) which separates the 
     596    # http header from the http content 
     597    if '\r\n\r\n' in httpheader: 
     598      # return split to return a list of httpheader lines 
     599      return httpheader.split('\r\n') 
     600    if '\n\n' in httpheader: 
     601      # return split to return a list of httpheader lines 
     602      return httpheader.split('\n') 
     603 
     604    if httpheader_limit == httpheader_received: 
     605      sock.close()                   
     606      raise HttpHeaderFormatError('Http header length Error: The http header is too long, exceeded 8 kb') 
     607                         
    403608    try: 
    404       self._sock.close() 
     609      # receive one character at a time inorder to check for the empty line 
     610      content = sock.recv(1) 
     611      # keep track of the received characters to raise an exception if the limit is exceeded                   
     612      httpheader_received += 1                   
     613 
     614    except SocketTimeoutError: 
     615      raise HttpHeaderReceivingError('Timeout Error on receiving header: server taking too long to send http header') 
    405616    except Exception, e: 
    406       pass # don't care 
    407  
    408  
    409  
    410  
    411 def _httpretrieve_parse_responseheaders(headerlines): 
    412   # Parse rfc822-style headers (this could be abstracted out to an rfc822 
    413   # library that would be quite useful for internet protocols). Returns 
    414   # a dictionary mapping headers to arrays of values. E.g.: 
    415   # 
    416   # Foo: a 
    417   # Bar: 
    418   #   b 
    419   # Bar: c 
    420   # 
    421   # Becomes: {"Foo": ["a"], "Bar": ["b", "c"]} 
    422  
    423   i = 0 
    424   lastheader = None 
    425   lastheader_str = "" 
    426   res = {} 
    427   try: 
    428     while True: 
    429       # non-CRLF whitespace characters 
    430       if headerlines[i][0] in (" ", "\t") and lastheader is not None: 
    431         lastheader_str += headerlines[i] 
     617      sock.close()  
     618      raise HttpHeaderReceivingError('Error on recieving http header: ' + str(e)) 
     619    else: 
     620      # if there was not receiving error add keep on adding the receieved content 
     621      httpheader += content 
     622 
     623 
     624 
     625 
     626def _httpretrieve_get_httpstatuscode(httpHeaderLines):  
     627  # checks if the http status code is valid and return the status number and msg  
     628 
     629  # http response header includes 3 "words": HTTP<version> http_status_number http_status_msg  
     630  httpstatusheader = httpHeaderLines[0] 
     631  headersplit = httpstatusheader.split(' ', 2) 
     632 
     633  # length of the header has to be 3 or greater because depending up on the http_status_msg might be more than one word 
     634  if len(headersplit) != 3: 
     635    raise HttpHeaderFormatError('Invalid Http header status code format: Correct format is HTTP<version> http_status_number http_status_msg: Given '  + httpstatusheader) 
     636  if not httpstatusheader.startswith('HTTP'): 
     637    raise HttpHeaderFormatError('Invalid Http header status code format: Http header status code should start of with HTTP<version> but given: '  + httpstatusheader) 
     638 
     639  # the first split is the http version 
     640  http_version = headersplit[0]                       
     641 
     642  # check if http_status_number is valid int 
     643  try:  
     644    int(headersplit[1]) 
     645  except ValueError, e: 
     646    raise HttpHeaderFormatError('Invalid Http header status code format: Status number should be a int, Given: ' + str(headersplit[1]) + str(e)) 
     647  else: 
     648    http_status_number = headersplit[1] 
     649   
     650  # what ever is left is the http status msg 
     651  http_status_msg = headersplit[2] 
     652   
     653  # return the values 
     654  return http_status_number, http_status_msg 
     655 
     656 
     657 
     658 
     659def _httpretrieve_receive_httperror_content(sock, contentlength): 
     660  # receives the http error content which is located after the http error header                         
     661  httperror_content = ''  
     662  while True: 
     663    try: 
     664      content = sock.recv(1024)                   
     665 
     666    except SocketTimeoutError: 
     667      raise HttpContentReceivingError('Timeout Error on receiving http error conent: server taking too long to send http error content') 
     668    except Exception, e: 
     669      # socket closed - signal for when the server is done sending error content 
     670      # if there is any other exceptions close connection and raise an error besides socket closing   
     671      if 'Socket closed' not in str(e):  
     672        sock.close()             
     673        raise HttpContentReceivingError('Error on receiving http error content: ' + str(e)) 
     674      break 
     675         
     676    else: 
     677      # if there was not a receiving error keep on adding the receieved content 
     678      httperror_content += content 
     679 
     680  # return the received http error content. If the content length is given check 
     681  # if the content length maches the received content 
     682  if contentlength != None: 
     683    if contentlength != len(httperror_content): 
     684      raise HttpContentLengthError('Error on receiving http error content: received conent length: ' + str(len(httperror_content)) + ' actual content length: ' + str(contentlength))    
     685  return httperror_content       
     686 
     687 
     688 
     689     
     690def _httpretrieve_raise_httpstatuscode_error(http_status_number, http_status_msg, http_errorcontent):  
     691  # raises an exception using the http_status_number 1xx for Informational, 2xx for Success 3xx for Redirection, 
     692  # 4xx for Client Error, and 5xx Server Error 
     693  
     694  # raise a detailed error message explaining the http_status_number and http_status_msg for popular http errors   
     695  if http_status_number == '202': 
     696    raise HttpError202('Http response error: ' + http_status_number + ' ' + http_status_msg +  ' http proccesing not responding. Http error content: ' + http_errorcontent)       
     697  elif http_status_number == '204': 
     698    raise HttpError204('Http response error: ' + http_status_number + ' ' + http_status_msg +  ' thier is no http body content. Http error content: ' + http_errorcontent)  
     699  elif http_status_number == '300': 
     700    raise HttpError300('Http response error: ' + http_status_number + ' ' + http_status_msg +  ' multiple redirect isnt suported. Http error content: ' + http_errorcontent)                 
     701  elif http_status_number == '404': 
     702    raise HttpError404('Http response error: ' + http_status_number + ' ' + http_status_msg +  ' cant find anything matching the given url. Http error content: ' + http_errorcontent)                 
     703  elif http_status_number == '403': 
     704    raise HttpError403('Http response error: ' + http_status_number + ' ' + http_status_msg +  ' the request was illegal. Http error content: ' + http_errorcontent)                 
     705  elif http_status_number == '400': 
     706    raise HttpError400('Http response error: ' + http_status_number + ' ' + http_status_msg +  ' the request contians bad syntex. Http error content: ' + http_errorcontent)                 
     707  elif http_status_number == '500': 
     708    raise HttpError500('Http response error: ' + http_status_number + ' ' + http_status_msg +  ' The server encountered an unexpected condition. Http error content: ' + http_errorcontent)                  
     709  elif http_status_number == '502': 
     710    raise HttpError502('Http response error: ' + http_status_number + ' ' + http_status_msg +  ' acting like a gateway received an invalid response. Http error content: ' + http_errorcontent)                 
     711 
     712  # if the http number wasnt any of the popular http error msgs, raise an exception using 
     713  # the defualt http status number with http status msg   
     714  elif http_status_number >= '100' and http_status_number < '200':  
     715    raise HttpError1xx('Http response error: Information ' + http_status_number + ' ' + http_status_msg + '.Http error content: ' + http_errorcontent)  
     716  elif http_status_number > '200' and http_status_number < '300':  
     717    raise HttpError2xx('Http response error: success error ' + http_status_number + ' ' + http_status_msg + '.Http error content: ' + http_errorcontent) 
     718  elif http_status_number >= '300' and http_status_number < '400':  
     719    raise HttpError3xx('Http response error: Redirection error' + http_status_number + ' ' + http_status_msg + '.Http error content: ' + http_errorcontent) 
     720  elif http_status_number >= '400' and http_status_number < '500':  
     721    raise HttpError4xx('Http response error: client error ' + http_status_number + ' ' + http_status_msg + '.Http error content: ' + http_errorcontent)   
     722  elif http_status_number >= '500' and http_status_number < '600':  
     723    raise HttpError5xx('Http response error: server error: ' + http_status_number + ' ' + http_status_msg + '.Http error content: ' + http_errorcontent) 
     724  else: 
     725    raise HttpStatusCodeError('Http response error: invalid http status response, given ' + http_status_number + '.Http error content: ' + http_errorcontent)   
     726 
     727 
     728 
     729 
     730def _httpretrieve_httpredirect(httpheaderlines): 
     731  # given http header retruns the redirect location    
     732 
     733  # if there is a redirect location given by the server it will look like 
     734  # eg.'Location: http://www.google.com' 
     735  for headerline in httpheaderlines: 
     736    if headerline.startswith('Location:'): 
     737      # if found redirect need to strip out 'Location: ' to return the url    
     738      redirect = headerline[len('Location: '):] 
     739                         
     740      # check if the redirect has given a location then return it 
     741      if len(redirect) == 0: 
     742        raise HttpHeaderFormatError('Http header redirection format error: http server gave a redierect location with no URL') 
     743      return redirect 
     744                         
     745  # there wasn't a redirect location given 
     746  raise HttpHeaderFormatError('Http header redirection format error: http redirect header didnt include the location')   
     747 
     748 
     749 
     750 
     751def _httpretrieve_get_contentlength(httpheaderlines): 
     752  # returns the content legth if given or returns None if not given by server 
     753 
     754  # if there is a content length given by the server it will look like 
     755  # eg.'Content-Length: 34' 
     756  for headerline in httpheaderlines: 
     757    if headerline.startswith('Content-Length:'): 
     758      # if found content length need to strip out 'Content-Length: ' to return the length    
     759       
     760      try:  
     761        contentlength = int(headerline[len('Content-Length: '):]) 
     762      except ValueError, e: 
     763        raise HttpHeaderFormatError('Http header Content-Length format error: http server provided content length that isnt a int ' + str(e))                 
     764 
     765       
     766      # check if the content length is valid and retrun it  
     767      if contentlength < 0:                 
     768        raise HttpHeaderFormatError('Http header Content-Length format error: provided content length with invalid number ' + str(contentlength)) 
    432769      else: 
    433         if lastheader is not None: 
    434           if lastheader not in res: 
    435             res[lastheader] = [] 
    436           res[lastheader].append(lastheader_str.strip()) 
    437         lastheader, lastheader_str = headerlines[i].split(":", 1) 
    438       i += 1 
    439       if i >= len(headerlines): 
    440         if lastheader is not None: 
    441           if lastheader not in res: 
    442             res[lastheader] = [] 
    443           res[lastheader].append(lastheader_str.strip()) 
    444         break 
    445     return res 
    446   except IndexError, idx: 
    447     raise HttpBrokenServerError("Server returned garbage for HTTP" + \ 
    448         " response. Bad header.") 
     770        return contentlength 
     771                         
     772  # there wasn't a content-length line or the content length was given but didnt give a int  
     773  return None 
     774 
     775 
  • seattle/trunk/seattlelib/registerhttpcallback.repy

    r3332 r3364  
    3030 
    3131 
    32 def registerhttpcallback(url, callbackfunc, httprequest_limit = 131072, httppost_limit = 2048): 
     32def registerhttpcallback(url, callbackfunc, httprequest_limit = 131072, httppost_limit = 2048, httpstatuscode_numb = 200): 
    3333  """ 
    3434  <Purpose> 
     
    5757 
    5858            RESTRICTIONS: 
    59               -> Follow the http_hierarchy_error(HttpStatusCodeError)to raise an exception. eg. raise 
    60                   HttpError404 for file not found 
    61               -> To redirect raise HttpError301(url) or HttpError302(url). The url has to be valid.   
    62               -> server only excepts tuple type of [httpcontent, httpheader] where the httpcontent is a 
    63                   string of the content intended to display. httpheader is a dictionary used only if you have a 
    64                   extra header you want to add on the http response.  
     59              Retun a type tuple list on the callback function. Which could include content, http header 
     60                and optional http status code. Where the content has to be type string, http header has to be type 
     61                dict, and http status code has to be int. 
     62               
     63               
    6564 
    6665     httprequest_limit: 
     
    9493  <Returns> 
    9594     A handle to the listener. This can be used to stop the server from waiting for a connection.   
     95 
    9696  """   
    9797  def run_webserver(ip, port, sock, thiscommhandle, listencommhandle): 
     
    118118      # place the call back callbackfunc with dictionary that includes http_command, 
    119119      # http_version, path, query, posted_data, and all the http requests headers 
    120       webpage_content = callbackfunc(httprequest_dictionary, httprequest_query, httprequest_posted_data) 
     120      callbackfunc_val = callbackfunc(httprequest_dictionary, httprequest_query, httprequest_posted_data) 
     121 
     122      # check the callbackfunc_val and parse the callbackfunc_content, callbckfunc_httpheader, and httpstautscode_numb from the returned tuple 
     123      (callbackfunc_content, callbckfunc_httpheader, httpstautscode_numb) = _registerhttpcallback_parse_callbackfunc_val(callbackfunc_val, httpstatuscode_numb) 
    121124         
    122125      # callback callbackfunc excecuted, send the processed dynamic web page data to client(web browser) 
    123       _registerhttpcallback_send_httpresponse(sock, webpage_content) 
     126      _registerhttpcallback_send_httpresponse(sock, callbackfunc_content, callbckfunc_httpheader, httpstautscode_numb) 
    124127 
    125128       
    126129    except HttpStatusCodeError, e: 
     130      # using the httpstatuscode dictionary get the statuscode number and statuscode constant from the given httperror 
     131      httpstautscode_numb = _registerhttpcallback_get_http_statuscode(e) 
     132 
     133      # build html error page using the exception that was raised  
     134      error_page_content = _registerhttpcallback_build_htmlerrorpage(httpstautscode_numb, str(e)) 
     135 
    127136      # send any error that occur during processing server to the client(web browser) 
    128       _registerhttpcallback_send_httpformat_error(sock, e) 
     137      _registerhttpcallback_send_httpheader_n_content(sock, httpstautscode_numb, error_page_content, None)  
     138       
    129139       
    130140       
    131141    except Exception, e: 
    132       # if the program failed to catch an internal error raise an exception and send it to client(web browser) 
    133       try: 
    134         raise HttpError500('Server failed internally: ' + str(e)) 
    135       except Exception, e: 
    136         _registerhttpcallback_send_httpformat_error(sock, e) 
     142      # if the program failed to catch an internal error and send it to client(web browser) 
     143      # build html error page using the exception that was raised  
     144      error_page_content = _registerhttpcallback_build_htmlerrorpage(httpstautscode_numb, str(e)) 
     145 
     146      # send any error that occur during processing server to the client(web browser) 
     147      _registerhttpcallback_send_httpheader_n_content(sock, 500, error_page_content, None)  
     148 
     149 
    137150 
    138151 
     
    214227 
    215228 
     229 
    216230       
    217231def _registerhttpcallback_receive_client_request(sock, httprequest_limit): 
     
    399413 
    400414 
     415 
    401416def _registerhttpcallback_receive_httpposted_data(sock, httprequest_dictionary, httppoost_limit): 
    402417  # receive the posted data which sent right after the http request with a empty line 
     
    502517 
    503518 
    504 def _registerhttpcallback_send_httpresponse(sock, callbackfunc_val): 
    505   # sends a response to the client(web browser) with a ok http header and the http web page content 
     519def _registerhttpcallback_send_httpresponse(sock, callbackfunc_content, callbckfunc_httpheader, httpstautscode_numb): 
     520  # sends a response to the client(web browser) with a ok http header and the http web page content  
     521 
     522  if httpstautscode_numb == 200: 
     523    # send a http format 200 ok msg 
     524 
     525    # check if the callback function content has data to be sent 
     526    if len(callbackfunc_content) == 0: 
     527      raise HttpUserInputError('Callback func didnt return any content to send') 
     528   
     529    # send a standard http 200 ok msg 
     530    _registerhttpcallback_send_httpheader_n_content(sock, 200, callbackfunc_content, callbckfunc_httpheader) 
     531 
     532 
     533  elif httpstautscode_numb == 204: 
     534    # send a http format 204 no content msg, which only includes a http header no content 
     535 
     536    # make sure the callback function client didnt give a content 
     537    if len(callbackfunc_content) != 0: 
     538      raise HttpUserInputError('Callback func should return a empty content on a http status code 204, given: ' + str(callbackfunc_content)) 
     539 
     540    # send the 204 msg 
     541    _registerhttpcallback_send_httpheader_n_content(sock, 204, '', callbckfunc_httpheader) 
     542 
     543 
     544  elif httpstautscode_numb > 200 and httpstautscode_numb < 300: 
     545    # unsupported http status codes ok msg's thus raise an exception 
     546    raise HttpUserInputError('The http status code ' + str(httpstautscode_numb) + ' isnt supported by registerhttpcallback' )    
     547 
     548   
     549  elif httpstautscode_numb == 301 or httpstautscode_numb == 302:  
     550    # send a http format redirect to client 
     551 
     552    # checks if client has given the location in the callback func http header 
     553    try: 
     554      redirect_loc = callbckfunc_httpheader['Location'] 
     555    except Exception,e: 
     556      raise HttpUserInputError('Redirect with no Location field specified in the http header dict. Raised ' + str(e)) 
     557 
     558    if len(redirect_loc) == 0: 
     559      raise HttpUserInputError('Given a empty redirect Location in the http header dict') 
     560 
     561    if not type(redirect_loc) == str: 
     562      raise HttpUserInputError('Callback func didnt return str for the content, returned ' + str(type(callbackfunc_content))) 
     563 
     564    # error page to follow the http header for a http error 
     565    error_page_content = _registerhttpcallback_build_htmlerrorpage(httpstautscode_numb, callbackfunc_content)   
     566 
     567    # send a http format redirect 
     568    _registerhttpcallback_send_httpheader_n_content(sock, httpstautscode_numb, error_page_content, callbckfunc_httpheader) 
     569     
     570 
     571  elif httpstautscode_numb > 300 and httpstautscode_numb < 400: 
     572    # unsupported http status codes redirects thus raise an exception  
     573    raise HttpUserInputError('The redirect type http status code ' + str(httpstautscode_numb) + ' isnt supported by registerhttpcallback' )    
     574 
     575 
     576  elif httpstautscode_numb > 500: 
     577    # raise an exception if the status code is above 500 
     578    raise HttpUserInputError('the given http status code number is invalid. Given ' + str(httpstautscode_numb)) 
     579 
     580     
     581  else: 
     582    # for any other 400's and 500's send a http format error to client 
     583 
     584    # error page to follow the http header for a http error  
     585    error_page_content = _registerhttpcallback_build_htmlerrorpage(httpstautscode_numb, callbackfunc_content)  
     586     
     587    # send a http format error msg to the client 
     588    _registerhttpcallback_send_httpheader_n_content(sock, httpstautscode_numb, error_page_content, callbckfunc_httpheader) 
     589 
     590 
     591 
     592 
     593 
     594def _registerhttpcallback_parse_callbackfunc_val(callbackfunc_val, httpstatuscode_numb): 
     595  # check the callbackfunction returned value type and parse the given  
    506596  if not type(callbackfunc_val) == list: 
    507597    raise HttpUserInputError('Callback func didnt return list, returned ' + str(type(callbackfunc_val))) 
    508598 
    509   try: 
    510     webpage_content = callbackfunc_val[0] 
    511     callbckfunc_httpheader = callbackfunc_val[1] 
    512   except Exception, e: 
    513     raise HttpUserInputError('Callback func returned data failed ' + str(e)) 
    514  
    515   # check the given web page content  
    516   if not type(webpage_content) == str: 
    517     raise HttpUserInputError('Callback func didnt return str for the content, returned ' + str(type(webpage_content))) 
    518   if len(webpage_content) == 0: 
    519     raise HttpUserInputError('Callback func didnt return any content') 
    520    
    521    
    522   # build the http ok response header  
    523   httpheader = 'HTTP/1.0 200 OK\n' 
    524   httpheader += 'Content-Length: ' + str(len(webpage_content)) + '\n' 
    525   #check if there is a given http header and add it to the response 
    526   httpheader += _registerhttpcallback_parse_callbckfunc_httpheader(callbckfunc_httpheader) 
     599  # the list has to be 2 or 3 which include (content, httpheader, httpstatuscode) 
     600  # where httpstatuscode is optional and the default is set to 200  
     601  if len(callbackfunc_val) > 4 or len(callbackfunc_val) < 2: 
     602    raise HttpUserInputError('need to retrun at least 2 elements in callback func tuple. Returned ' + str(len(callbackfunc_val))) 
     603   
     604  # check and parse the given content  
     605  callbackfunc_content = callbackfunc_val[0] 
     606  if not type(callbackfunc_content) == str: 
     607    raise HttpUserInputError('Callback func didnt return str for the content, returned ' + str(type(callbackfunc_content))) 
     608 
     609  # check and parse the given httpheader.(The dict values are checked in _registerhttpcallback_parse_callbckfunc_httpheader) 
     610  callbckfunc_httpheader = callbackfunc_val[1] 
     611  if not type(callbckfunc_httpheader) == dict and callbckfunc_httpheader != None: 
     612    # raise an exception if the http header isnt dictionary 
     613    raise HttpUserInputError('The given http header is not a dictionary, given: ' + str(type(callbckfunc_httpheader))) 
     614 
     615  # if the callback function list contains three elements. The third element is the status code number  
     616  if len(callbackfunc_val) == 3: 
     617    # check and parse the http status code number to the given 
     618    httpstautscode_numb = callbackfunc_val[2] 
     619    if not type(httpstautscode_numb) == int: 
     620      raise HttpUserInputError('Callback func didnt return int for the http status code, returned ' + str(type(httpstautscode_numb))) 
     621 
    527622     
     623  # retun the parsed values  
     624  return callbackfunc_content, callbckfunc_httpheader, httpstautscode_numb   
     625 
     626 
     627 
     628 
     629   
     630 
     631def _registerhttpcallback_build_htmlerrorpage(httpstautscode_numb, extra_content): 
     632  # builds a html page to display for redirect or error msg  
     633  # get the http constant form the client 
     634  httpstatus_constant = _registerhttpcallback_get_http_statusconstant(httpstautscode_numb)   
     635   
     636  # build error content for error descritption 
     637  httpcontent = '<html>' 
     638  httpcontent += '<head><title>' + str(httpstautscode_numb) + ' ' + httpstatus_constant + '</title></head>' 
     639  httpcontent += '<body><h1>' + str(httpstautscode_numb) + ' ' + httpstatus_constant + '</h1>' 
     640  if extra_content != '': 
     641    # add a extra content to display if given  
     642    httpcontent += '<p>' + extra_content + '</p></body>' 
     643  httpcontent += '</html>' 
     644  # to end the error content 
     645  httpcontent += '\n\n' 
     646 
     647 
     648 
     649 
     650   
     651def _registerhttpcallback_send_httpheader_n_content(sock, httpstautscode_numb, content_to_send, callbckfunc_httpheader): 
     652  # send a http format response including httpheader and the webpage content to client  
     653 
     654  # get the http constant form the client 
     655  httpstatus_constant = _registerhttpcallback_get_http_statusconstant(httpstautscode_numb) 
     656 
     657  httpheader = 'HTTP/1.0 ' + str(httpstautscode_numb) + ' ' + httpstatus_constant + '\n' 
     658  httpheader += 'Content-Length: ' + str(len(callbackfunc_content)) + '\n' 
     659  # check if there is a given http header and add it to the response 
     660  httpheader += _registerhttpcallback_parse_callbckfunc_httpheader(callbckfunc_httpheader)    
     661  # Server field followed by empty line to end the http header 
    528662  httpheader += 'Server: Seattle Testbed\n\n' 
    529663 
    530664  # http header followed by http content and close the connection 
    531665  try: 
    532     sock.send(httpheader) 
    533     # sends data by chunk of 1024 charactors at a time 
    534     _registerhttpcallback_sendbychunk(webpage_content, 1024, sock) 
     666    # send the http header to client  
     667    _registerhttpcallback_sendall(sock, httpheader) 
     668    # send the content to client.(check if content is given because 204 doesnt include a content) 
     669    if callbackfunc_content != '': 
     670      # send the content to client 
     671      _registerhttpcallback_sendall(sock, callbackfunc_content) 
     672    # close connection with client 
    535673    sock.close()  
    536674  except Exception, e: 
    537     raise HttpConnectionError('server failed to send the http content ' + str(e))   
    538  
    539  
    540  
    541  
    542  
    543 def _registerhttpcallback_sendbychunk(full_data, chunck_size, sock): 
    544   # sends a given data to a socket connection by the chunk amount one at a time.  
    545   length_of_data = len(full_data) 
    546   start = 0  
    547   end = chunck_size 
    548   # used to signal when there is no more data left to parse by chunk size 
    549   done = False 
    550    
    551   while not done : 
    552     if length_of_data <= end: 
    553       # last end of data 
    554       end = length_of_data 
    555       done = True 
    556  
    557     # parse the given data by the given chunk size and keep on sending by chunk bytes 
    558     chuncked_data = full_data[start:end] 
    559  
     675    raise HttpConnectionError('Server failed to send the http response ' + str(e))   
     676 
     677 
     678       
     679 
     680 
     681def _registerhttpcallback_sendall(sock, data): 
     682  # sends a given data to a socket connection by making sure the data is sent.  
     683  amountsent = 0 
     684  while amountsent < data: 
    560685    try: 
    561       sock.send(chuncked_data)   
     686      # send the data by parsing any data that didnt get sent 
     687      amountsent = amountsent + sock.send(data[amountsent:]) 
    562688    except Exception, e: 
    563       raise Exception('faild on sending data: ' + str(e)) 
    564     else: 
    565       # change start and end to send the next part of the data 
    566       start = end 
    567       end += chunck_size 
    568  
     689      # raise an exception if the socket fails 
     690      raise Exception(str(e)) 
     691     
    569692 
    570693 
     
    576699    # if the http header isnt given return a empty string 
    577700    return '' 
    578   elif not type(callbckfunc_httpheader) == dict: 
    579     # raise an exception if the http header isnt dictionary 
    580     raise HttpUserInputError('The given http header is not a dictionary, given: ' + str(type(callbckfunc_httpheader))) 
     701   
    581702  else:  
    582703    # take the given key and val from the callbckfunc_httpheader dictionary and add them to the http header with 
     
    594715      if key.capitalize() != key: 
    595716        raise HttpUserInputError('The callback func given http header key is not capitalized, given: ' + key) 
     717      if key == 'Content-Length' 
     718        raise HttpUserInputError('Content-Length http header field is set to default. Cant specify the content length in http header dict') 
     719      if key == 'Server' 
     720        raise HttpUserInputError('Server http header field is set to default. Cant specify the content length in http header dict') 
    596721 
    597722      # add the key and value to the http header field 
     
    601726    return httpheaders 
    602727 
    603    
    604  
    605  
    606  
    607 def _registerhttpcallback_send_httpformat_error(sock, e): 
    608   # send  correct format http header with a  http content that displays detailed error msg and close connection  
    609  
    610   # using the httpstatuscode dictionary get the statuscode number and statuscode constant from the given httperror 
    611   (statuscode_numb, client_error_msg, statuscode_constant) = _registerhttpcallback_get_http_statuscode(e) 
    612    
    613   # build http body error msg to client(web browser) 
    614   error_msg = client_error_msg 
    615  
    616   # error content body 
    617   httpcontent = '<html>' 
    618   httpcontent += '<head><title>' + str(statuscode_numb) + ' ' + statuscode_constant + '</title></head>' 
    619   httpcontent += '<body><h1>' + str(statuscode_numb) + ' ' + statuscode_constant + '</h1>' 
    620   httpcontent += '<p>' + error_msg + '</p></body>' 
    621   httpcontent += '</html>' 
    622   # to end the error content 
    623   httpcontent += '\n\n' 
    624    
    625   # build the http header to send     
    626   httpheader = 'HTTP/1.0 ' + str(statuscode_numb)  + ' ' + statuscode_constant + '\n' 
    627  
    628   # for redirect add the location of the redirection to the http header     
    629   if statuscode_numb == 301 or statuscode_numb == 302: 
    630     if client_error_msg == '': 
    631       raise HttpUserInputError('Internal server error: callback func client should put the location on raising redirect') 
    632     elif not client_error_msg.startswith('http://'): 
    633       raise HttpUserInputError('Internal server error: calback func client redirect is invalid, Given: ' + client_error_msg) 
    634     else: 
    635       httpheader += 'Location: ' + str(client_error_msg) + '\n' 
    636    
    637   # finish up the http header 
    638   httpheader += 'Content-Length: ' + str(len(httpcontent)) + '\n' 
    639   httpheader += 'Server: Seattle Testbed\n\n' 
    640    
    641   # send the http response header and body to the client(web browser) and close connection 
    642   try: 
    643     sock.send(httpheader) 
    644     sock.send(httpcontent) 
    645     sock.close() 
    646   except Exception, e: 
    647     raise HttpConnectionError('server failed internally on send http error ' + str(statuscode_numb) + ' ' + statuscode_constant + ' ' + error_msg + ' Raised' + str(e))  
    648728 
    649729 
     
    655735  # httpstatus code dictionary with the statuscode constant 
    656736  httpstatuscode_dict = { 
    657       HttpError100: (100, 'Continue'), 
    658       HttpError101: (101, 'Switching Protocols'), 
    659       HttpError102: (102, 'Processing'), 
    660       HttpError201: (201 ,'Created'), 
    661       HttpError202: (202, 'Accepted') 
    662       HttpError203: (203, 'Non-Authoritative Information'), 
    663       HttpError204: (204, 'No Content'), 
    664       HttpError205: (205, 'Reset Content'), 
    665       HttpError206: (206, 'Partial Content'), 
    666       HttpError207: (207, 'Multi-Status'), 
    667       HttpError226: (226, 'IM Used'), 
    668       HttpError300: (300, 'Multiple Choices'), 
    669       HttpError301: (301, 'Moved Permanently'), 
    670       HttpError302: (302, 'Found'), 
    671       HttpError303: (303, 'See Other'), 
    672       HttpError304: (304, 'Not Modified'), 
    673       HttpError305: (305, 'Use Proxy'), 
    674       HttpError306: (306, 'Unused'), 
    675       HttpError307: (307, 'Temporary Redirect'), 
    676       HttpError400: (400, 'Bad Request'), 
    677       HttpError401: (401, 'Unauthorized'), 
    678       HttpError402: (402, 'Payment Required'), 
    679       HttpError403: (403, 'Forbidden'), 
    680       HttpError404: (404, 'Not Found'), 
    681       HttpError405: (405, 'Method Not Allowed'), 
    682       HttpError406: (406, 'Not Acceptable'), 
    683       HttpError407: (407, 'Proxy Authentication Required'), 
    684       HttpError408: (408, 'Request Timeout'), 
    685       HttpError409: (409, 'Conflict'), 
    686       HttpError410: (410, 'Gone'), 
    687       HttpError411: (411, 'Length Required'), 
    688       HttpError412: (412, 'Precondition Failed'), 
    689       HttpError413: (413, 'Request Entity Too Large'), 
    690       HttpError414: (414, 'Request-URI Too Long'), 
    691       HttpError415: (415, 'Unsupported Media Type'), 
    692       HttpError416: (416, 'Requested Range Not Satisfiable'), 
    693       HttpError417: (417, 'Expectation Failed'), 
    694       HttpError418: (418, 'Im a teapot'), 
    695       HttpError422: (422, 'Unprocessable Entity'), 
    696       HttpError423: (423, 'Locked'), 
    697       HttpError424: (424, 'Failed Dependency'), 
    698       HttpError425: (425, 'Unordered Collection'), 
    699       HttpError426: (426, 'Upgrade Required'), 
    700       HttpError500: (500, 'Internal Server Error'), 
    701       HttpError501: (501, 'Not Implemented'), 
    702       HttpError502: (502, 'Bad Gateway'), 
    703       HttpError503: (503, 'Service Unavailable'), 
    704       HttpError504: (504, 'Gateway Timeout'), 
    705       HttpError505: (505, 'HTTP Version Not Supported'), 
    706       HttpError506: (506, 'Variant Also Negotiates'), 
    707       HttpError507: (507, 'Insufficient Storage'), 
    708       HttpError510: (510, 'Not Extended')} 
    709    
    710   # retrieves the status number and constant from the given exception class using the dictionary  
     737      HttpError100: 100, 
     738      HttpError101: 101, 
     739      HttpError102: 102, 
     740      HttpError201: 201, 
     741      HttpError202: 202 
     742      HttpError203: 203, 
     743      HttpError204: 204,  
     744      HttpError205: 205, 
     745      HttpError206: 206, 
     746      HttpError207: 207, 
     747      HttpError226: 226, 
     748      HttpError300: 300, 
     749      HttpError301: 301, 
     750      HttpError302: 302, 
     751      HttpError303: 303, 
     752      HttpError304: 304, 
     753      HttpError305: 305, 
     754      HttpError306: 306, 
     755      HttpError307: 307, 
     756      HttpError400: 400, 
     757      HttpError401: 401, 
     758      HttpError402: 402, 
     759      HttpError403: 403, 
     760      HttpError404: 404, 
     761      HttpError405: 405, 
     762      HttpError406: 406, 
     763      HttpError407: 407, 
     764      HttpError408: 408, 
     765      HttpError409: 409, 
     766      HttpError410: 410, 
     767      HttpError411: 411, 
     768      HttpError412: 412, 
     769      HttpError413: 413, 
     770      HttpError414: 414, 
     771      HttpError415: 415, 
     772      HttpError416: 416, 
     773      HttpError417: 417, 
     774      HttpError418: 418, 
     775      HttpError422: 422, 
     776      HttpError423: 423, 
     777      HttpError424: 424, 
     778      HttpError425: 425, 
     779      HttpError426: 426, 
     780      HttpError500: 500, 
     781      HttpError501: 501, 
     782      HttpError502: 502, 
     783      HttpError503: 503, 
     784      HttpError504: 504, 
     785      HttpError505: 505, 
     786      HttpError506: 506, 
     787      HttpError507: 507, 
     788      HttpError510: 510} 
     789   
     790  # retrieves the status number from the given exception class using the dictionary  
    711791  try: 
    712     (statuscode_numb, statuscode_constant) = httpstatuscode_dict[type(e)] 
     792    statuscode_numb = httpstatuscode_dict[type(e)] 
    713793  except Exception, e: 
    714     raise HttpServerError('Internal error on generating error msg: ' + str(e)) 
    715  
    716   # get any extra error msg that the callback fucntion raised  
    717   client_error_msg = str(e) 
     794    raise HttpServerError('Internal error on generating error msg for: ' + str(type(e)) + ' .Raised: ' + str(e)) 
    718795 
    719796  # return what is retrieved 
    720   return statuscode_numb, client_error_msg, statuscode_constant 
    721  
    722  
    723  
    724  
     797  return statuscode_numb 
     798 
     799 
     800 
     801 
     802def _registerhttpcallback_get_http_statusconstant(statuscode_numb): 
     803  # retrieves the status constant from the given status number using the dictionary  
     804 
     805  # httpstatus code dictionary with the statuscode constant 
     806  httpstatuscode_dict = { 
     807      100: 'Continue', 
     808      101: 'Switching Protocols', 
     809      102: 'Processing', 
     810      201:  'Created', 
     811      202: 'Accepted',   
     812      203: 'Non-Authoritative Information', 
     813      204: 'No Content', 
     814      205: 'Reset Content', 
     815      206: 'Partial Content', 
     816      207: 'Multi-Status', 
     817      226: 'IM Used', 
     818      300: 'Multiple Choices', 
     819      301: 'Moved Permanently', 
     820      302: 'Found', 
     821      303: 'See Other', 
     822      304: 'Not Modified', 
     823      305: 'Use Proxy', 
     824      306: 'Unused', 
     825      307: 'Temporary Redirect', 
     826      400: 'Bad Request', 
     827      401: 'Unauthorized', 
     828      402: 'Payment Required', 
     829      403: 'Forbidden', 
     830      404: 'Not Found', 
     831      405: 'Method Not Allowed', 
     832      406: 'Not Acceptable', 
     833      407: 'Proxy Authentication Required', 
     834      408: 'Request Timeout', 
     835      409: 'Conflict', 
     836      410: 'Gone', 
     837      411: 'Length Required', 
     838      412: 'Precondition Failed', 
     839      413: 'Request Entity Too Large', 
     840      414: 'Request-URI Too Long', 
     841      415: 'Unsupported Media Type', 
     842      416: 'Requested Range Not Satisfiable', 
     843      417: 'Expectation Failed', 
     844      418: 'Im a teapot', 
     845      422: 'Unprocessable Entity', 
     846      423: 'Locked', 
     847      424: 'Failed Dependency', 
     848      425: 'Unordered Collection', 
     849      426: 'Upgrade Required', 
     850      500: 'Internal Server Error', 
     851      501: 'Not Implemented', 
     852      502: 'Bad Gateway', 
     853      503: 'Service Unavailable', 
     854      504: 'Gateway Timeout', 
     855      505: 'HTTP Version Not Supported', 
     856      506: 'Variant Also Negotiates', 
     857      507: 'Insufficient Storage', 
     858      510: 'Not Extended'} 
     859   
     860  # retrieves the status constant from the given status number using the dictionary  
     861  try: 
     862    statuscode_constant = httpstatuscode_dict[statuscode_numb] 
     863  except Exception, e: 
     864    raise HttpServerError('The given http stauscode number isnt recognized. Given: ' + str(statuscode_numb) + ' Raised: ' + str(e)) 
     865 
     866  # return what is retrieved statsus constant 
     867  return  statuscode_constant 
     868 
     869 
     870 
     871 
     872 
     873   
     874 
     875