DB Connector Unable to Retrieve Chunks of Data

Document ID : KB000106300
Last Modified Date : 28/08/2018
Show Technical Document Details
Issue:
When running the connector the data load may be incomplete and the following is observed on the command line:
Retrying the page_urls for the page group startIndexes: 1001 -> 8001 
Retrying the page_urls for the page group startIndexes: 1001 -> 8001 
Retrying the page_urls for the page group startIndexes: 1001 -> 8001 

Additionally, the following error may be seen in the log file:
[2018-07-12 20:03:04 Z] ERROR: DBConnector.run - RallyResponseError in DBConnector.run processing entity: Story  -- Unable to retrieve 10 chunks of data
  File "X:\dbeif\connector.py", line 184, in run
    stats[entity] = self._processEntity(entity, csv_dir_name, populate_db, trash_csv)
   File "X:\dbeif\connector.py", line 259, in _processEntity
    entity_schema, pk_field = self.retrieveAndSaveAgileCentralData(entity)
   File "X:\dbeif\utils\metrixeur.py", line 13, in wrapper
    result = f(*args, **kwargs)
   File "X:\dbeif\connector.py", line 226, in retrieveAndSaveAgileCentralData
    = self.agicen_conn.getAgileCentralData(workspace_name, project, entity, project_scope_up, project_scope_down, self.remapping)
   File "X:\dbeif\agile_central_connection.py", line 615, in getAgileCentralData
    for item in response:
   File "X:\Python\lib\site-packages\pyral\rallyresp.py", line 313, in __next__
    self._page[:]  = self.__retrieveNextPage()
   File "X:\Python\lib\site-packages\pyral\rallyresp.py", line 410, in __retrieveNextPage
    chapter = self.__retrievePages()
   File "X:\Python\lib\site-packages\pyral\rallyresp.py", line 478, in __retrievePages
    raise RallyResponseError("Unable to retrieve %d chunks of data" % num_threads)

 <class 'pyral.rallyresp.RallyResponseError'> Unable to retrieve 10 chunks of data
[2018-07-12 20:03:04 Z] ERROR: DBConnectorRunner.run - Unable to retrieve 10 chunks of data
Cause:
This is caused by a configuration file that is requesting too much data and/or complex data.

We have also found that using the Milestones object in the Fetch list can cause performance issues.
Resolution:
  • Break the entity types out into multiple config files.  This is best practice per the documentation.
  • Restrict the number of records pulled by adding a selector to restrict data based on an attribute such as CreationDate.  Using an attribute like this, you can also split the selector data into multiple configs.  For example in one config you might use CreationDate <= 2017-07-01 and in a second config you would use CreationDate > 2017-07-01
  • Remove the Milestones field from the Fetch list 
Additional Information:
https://help.rallydev.com/agile-central-database-connector