python - AWS S3 Boto Download Timing Out on some Jenkins Jobs and Not Others -


the problem: have python script ran via multi-configuration jenkins job. job downloads script , passes parameters it. spawns 10 times because there 10 sets of data being loaded in parallel. on ocasions , not 2 or 3 out of jobs s3 timeout error. i'm not sure why because not same jobs , not time.

error:

downloading: /var/lib/jenkins/workspace/deploy-es-data/label/swarm/wos/wos_1/json_data/wos/20150707/wos_1/wok5_20150410045810_wos_0124.json.gz downloading: /var/lib/jenkins/workspace/deploy-es-data/label/swarm/wos/wos_1/json_data/wos/20150707/wos_1/wok5_20150410045810_wos_0125.json.gz downloading: /var/lib/jenkins/workspace/deploy-es-data/label/swarm/wos/wos_1/json_data/wos/20150707/wos_1/wok5_20150410045810_wos_0126.json.gz downloading: /var/lib/jenkins/workspace/deploy-es-data/label/swarm/wos/wos_1/json_data/wos/20150707/wos_1/wok5_20150410045810_wos_0127.json.gz downloading: /var/lib/jenkins/workspace/deploy-es-data/label/swarm/wos/wos_1/json_data/wos/20150707/wos_1/wok5_20150410045810_wos_0128.json.gz downloading: /var/lib/jenkins/workspace/deploy-es-data/label/swarm/wos/wos_1/json_data/wos/20150707/wos_1/wok5_20150410045810_wos_0129.json.gz downloading: /var/lib/jenkins/workspace/deploy-es-data/label/swarm/wos/wos_1/json_data/wos/20150707/wos_1/wok5_20150410045810_wos_0130.json.gz downloading: /var/lib/jenkins/workspace/deploy-es-data/label/swarm/wos/wos_1/json_data/wos/20150707/wos_1/wok5_20150410045810_wos_0131.json.gz downloading: /var/lib/jenkins/workspace/deploy-es-data/label/swarm/wos/wos_1/json_data/wos/20150707/wos_1/wok5_20150410045810_wos_0132.json.gz downloading: /var/lib/jenkins/workspace/deploy-es-data/label/swarm/wos/wos_1/json_data/wos/20150707/wos_1/wok5_20150410045810_wos_0133.json.gz downloading: /var/lib/jenkins/workspace/deploy-es-data/label/swarm/wos/wos_1/json_data/wos/20150707/wos_1/wok5_20150410045810_wos_0134.json.gz downloading: /var/lib/jenkins/workspace/deploy-es-data/label/swarm/wos/wos_1/json_data/wos/20150707/wos_1/wok5_20150410045810_wos_0135.json.gz > downloading: > /var/lib/jenkins/workspace/deploy-es-data/label/swarm/wos/wos_1/json_data/wos/20150707/wos_1/wok5_20150410045810_wos_0135.json.gz > traceback (most recent call last):   file > "/var/lib/jenkins/workspace/deploy-es-data/label/swarm/wos/wos_1/es_single_load.py", > line 1094, in <module> >     directory = download_s3(wos_data, 'tr-ips-ses-data',access_key, secret_key)   file > "/var/lib/jenkins/workspace/deploy-es-data/label/swarm/wos/wos_1/es_single_load.py", > line 31, in download_s3 >     l.get_contents_to_filename(d)   file "/usr/lib/python2.7/dist-packages/boto/s3/key.py", line 1712, in > get_contents_to_filename >     response_headers=response_headers)   file "/usr/lib/python2.7/dist-packages/boto/s3/key.py", line 1650, in > get_contents_to_file >     response_headers=response_headers)   file "/usr/lib/python2.7/dist-packages/boto/s3/key.py", line 1482, in > get_file >     query_args=none)   file "/usr/lib/python2.7/dist-packages/boto/s3/key.py", line 1535, in > _get_file_internal >     bytes in self:   file "/usr/lib/python2.7/dist-packages/boto/s3/key.py", line 386, in next >     data = self.resp.read(self.buffersize)   file "/usr/lib/python2.7/dist-packages/boto/connection.py", line 413, in > read >     return http_client.httpresponse.read(self, amt)   file "/usr/lib64/python2.7/httplib.py", line 573, in read >     s = self.fp.read(amt)   file "/usr/lib64/python2.7/socket.py", line 380, in read >     data = self._sock.recv(left)   file "/usr/lib64/python2.7/ssl.py", line 714, in recv >     return self.read(buflen)   file "/usr/lib64/python2.7/ssl.py", line 608, in read >     v = self._sslobj.read(len or 1024) ssl.sslerror: ('the read operation timed out',) build step 'execute shell' marked build > failure finished: failure

python:

def download_s3(wos,bucket, access_key, secret_key):     # connect bucket     conn = boto.connect_s3(access_key, secret_key)     bucket = conn.get_bucket(bucket)     local_path = os.path.dirname(os.path.realpath(__file__))     bucket_list = bucket.list()     l in bucket_list:         keystring = str(l.key)         d = local_path + '/' + keystring         search_string = '/' + wos + '/'         if search_string in str(l) , '20150707' in str(l):             if str(wos) == 'wos_1' , '/wos_10/' in str(l):                 continue             try:                 print 'downloading: %s' % d                 l.get_contents_to_filename(d)             except oserror:                 if not os.path.exists(d):                     os.makedirs(d)         else:             continue      conn.close()     return d,local_path


Comments

Post a Comment

Popular posts from this blog

toolbar - How to add link to user registration inside toobar in admin joomla 3 custom component -

im trying write component admin side , need add user creation. dont want duplicate existing codes add link. know how add task not simple link toolbar. thx help. if wanting button opens new user screen can try in component: $bar = jtoolbar::getinstance('toolbar'); $bar->appendbutton('link', 'users', 'new user','index.php?option=com_users&task=user.add'); you need add view.html.php file wherever you're creating toolbar. is need?
Read more

linux - disk space limitation when creating war file -

i have 1 gb disk space limitation hosting app , have used 700mb of it. however want create war file directory big (around 600mb) using command : jar -cvf root.war folderpath that's why encounter quota limitation , can't following. i'm looking way delete folder content when it's applying archive conserve disk space take note can't install zip tool in ssh shell ~ the code works on test directories, test first anyway, deletes files. it's not efficient. assumptions: 300mb free space, in destructively compress 600mb folderpath directory. none of files big -- if file in folderpath 300mb or larger, fail. notes: if ' $out ' file exists, nothing. first find pipes file list while , if ' $out ' doesn't exist in loop, create it, don't update -- if does exist, update it. each pass adds 1 more file, deletes unless file directory. after loop finishes delete empty directories too. unset u ; in=folderpath ; out=roo...
Read more

How to provide Authorization & Authentication using Asp.net, C#? -

Image
i have asp.net application in c#. i have provide page authorization specific user/usergroup. i have created 2 tables in database: 1) pagename 2)permissions[insert, update, view, delete] but i'm unable match tables , permissions assigning... can 1 please me, how created tables in database , how coding. please me. you can below explanation. configure access specific file , folder, set forms-based authentication. request page in application redirected logon.aspx automatically. in web.config file, type or paste following code. this code grants users access default1.aspx page , subdir1 folder . <configuration> <system.web> <authentication mode="forms" > <forms loginurl="login.aspx" name=".aspnetauth" protection="none" path="/" timeout="20" > </forms> </authentication> <!-- section denies access files in...
Read more