Current File : //usr/local/lib/python2.7/multiprocessing/pool.pyc
Pac @ sT d g Z d d l Z d d l Z d d l Z d d l Z d d l Z d d l m Z m Z m Z d d l
m Z m Z d Z
d Z d Z e j Z d Z d e f d
YZ e f e d Z d e f d YZ d
e f d YZ e Z d e f d YZ d e f d YZ d e f d YZ d e f d YZ d S( t PooliN( t Processt cpu_countt TimeoutError( t Finalizet debugi i i c C s
t | S( N( t map( t args( ( s0 /usr/local/lib/python2.7/multiprocessing/pool.pyt mapstar@ s t MaybeEncodingErrorc B s) e Z d Z d Z d Z d Z RS( sV Wraps possible unpickleable errors, so they can be
safely sent through the socket.c C sA t | | _ t | | _ t t | j | j | j d S( N( t reprt exct valuet superR t __init__( t selfR R ( ( s0 /usr/local/lib/python2.7/multiprocessing/pool.pyR K s c C s d | j | j f S( Ns( Error sending result: '%s'. Reason: '%s'( R R ( R ( ( s0 /usr/local/lib/python2.7/multiprocessing/pool.pyt __str__P s c C s d t | S( Ns <MaybeEncodingError: %s>( t str( R ( ( s0 /usr/local/lib/python2.7/multiprocessing/pool.pyt __repr__T s ( t __name__t
__module__t __doc__R R R ( ( ( s0 /usr/local/lib/python2.7/multiprocessing/pool.pyR G s c C s | d k s6 t | t t f k r0 | d k s6 t | j } | j } t | d rt | j j | j
j n | d k r | | n d } xI| d k s | r| | k ry
| } Wn" t t f k
r t
d Pn X| d k r t
d Pn | \ } }
} } }
y t | | |
f } Wn t k
rO} t | f } n Xy | | |
| f WnM t k
r} t | | d } t
d | | | |
t | f f n Xd } } } } } }
| d 7} q Wt
d | d S( Ni t _writers) worker got EOFError or IOError -- exitings worker got sentinel -- exitingi s0 Possible encoding error while sending result: %ss worker exiting after %d tasks( t Nonet typet intt longt AssertionErrort putt gett hasattrR t closet _readert EOFErrort IOErrorR t Truet Exceptiont FalseR ( t inqueuet outqueuet initializert initargst maxtasksR R t completedt taskt jobt it funcR t kwdst resultt et wrapped( ( s0 /usr/local/lib/python2.7/multiprocessing/pool.pyt workerX sB 6
!
c B s e Z d Z e Z d d d d d Z d Z d Z d Z d Z d i d Z
d d Z d d Z d d
Z
d i d d Z d d d Z e d
Z e d Z e d Z e d Z d Z d Z d Z d Z e d Z e d Z RS( sH
Class which supports an async version of the `apply()` builtin
c C s/ | j t j | _ i | _ t | _ | | _ | | _ | | _ | d k r| y
t
} Wq| t k
rx d } q| Xn | d k r t d n | d k r t
| d r t d n | | _ g | _ | j t j d t j d | f | _ t | j _ t | j _ | j j t j d t j d | j | j | j | j | j f | _ t | j _ t | j _ | j j t j d t j d | j | j | j f | _ t | j _ t | j _ | j j t! | | j" d | j | j# | j | j | j | j | j | j f d d | _$ d S( Ni s& Number of processes must be at least 1t __call__s initializer must be a callablet targetR t exitpriorityi (% t
_setup_queuest Queuet
_taskqueuet _cachet RUNt _statet _maxtasksperchildt _initializert _initargsR R t NotImplementedErrort
ValueErrorR t TypeErrort
_processest _poolt _repopulate_poolt threadingt ThreadR t _handle_workerst _worker_handlerR# t daemont startt
_handle_taskst
_quick_putt _outqueuet
_task_handlert _handle_resultst
_quick_gett _result_handlerR t _terminate_poolt _inqueuet
_terminate( R t processesR( R) t maxtasksperchild( ( s0 /usr/local/lib/python2.7/multiprocessing/pool.pyR sX
c C sw t } xj t t t | j D]M } | j | } | j d k r" t d | | j t } | j | =q" q" W| S( s Cleanup after any worker processes which have exited due to reaching
their specified lifetime. Returns True if any workers were cleaned up.
s cleaning up worker %dN(
R% t reversedt ranget lenRE t exitcodeR R t joinR# ( R t cleanedR. R4 ( ( s0 /usr/local/lib/python2.7/multiprocessing/pool.pyt _join_exited_workers s "
c
C s x t | j t | j D] } | j d t d | j | j | j | j | j
f } | j j | | j j
d d | _ t | _ | j t d q Wd S( s Bring the number of pool processes up to the specified number,
for use after reaping workers which have exited.
R6 R R t
PoolWorkers added workerN( RZ RD R[ RE R R4 RU RO R? R@ R> t appendt namet replaceR# RK RL R ( R R. t w( ( s0 /usr/local/lib/python2.7/multiprocessing/pool.pyRF s #
c C s | j r | j n d S( sE Clean up any exited workers and start replacements for them.
N( R_ RF ( R ( ( s0 /usr/local/lib/python2.7/multiprocessing/pool.pyt _maintain_pool s c C sP d d l m } | | _ | | _ | j j j | _ | j j j | _ d S( Ni ( t SimpleQueue(
t queuesRf RU RO R t sendRN R t recvRR ( R Rf ( ( s0 /usr/local/lib/python2.7/multiprocessing/pool.pyR8 s
c C s. | j t k s t | j | | | j S( s1
Equivalent of `apply()` builtin
( R= R< R t apply_asyncR ( R R/ R R0 ( ( s0 /usr/local/lib/python2.7/multiprocessing/pool.pyt apply s c C s. | j t k s t | j | | | j S( s/
Equivalent of `map()` builtin
( R= R< R t map_asyncR ( R R/ t iterablet chunksize( ( s0 /usr/local/lib/python2.7/multiprocessing/pool.pyR s i c s | j t k s t | d k rf t | j | j j f d t | D j f S| d k sx t t j
| | } t | j | j j f d t | D j f d D Sd S( sZ
Equivalent of `itertools.imap()` -- can be MUCH slower than `Pool.map()`
i c 3 s0 | ]& \ } } j | | f i f Vq d S( N( t _job( t .0R. t x( R/ R1 ( s0 /usr/local/lib/python2.7/multiprocessing/pool.pys <genexpr> s c 3 s0 | ]& \ } } j | t | f i f Vq d S( N( Ro R ( Rp R. Rq ( R1 ( s0 /usr/local/lib/python2.7/multiprocessing/pool.pys <genexpr>
s c s s" | ] } | D] } | Vq
q d S( N( ( Rp t chunkt item( ( s0 /usr/local/lib/python2.7/multiprocessing/pool.pys <genexpr> s N( R= R< R t IMapIteratorR; R: R t enumeratet _set_lengthR t
_get_tasks( R R/ Rm Rn t task_batches( ( R/ R1 s0 /usr/local/lib/python2.7/multiprocessing/pool.pyt imap s c s | j t k s t | d k rf t | j | j j f d t | D j f S| d k sx t t j
| | } t | j | j j f d t | D j f d D Sd S( sK
Like `imap()` method but ordering of results is arbitrary
i c 3 s0 | ]& \ } } j | | f i f Vq d S( N( Ro ( Rp R. Rq ( R/ R1 ( s0 /usr/local/lib/python2.7/multiprocessing/pool.pys <genexpr> s c 3 s0 | ]& \ } } j | t | f i f Vq d S( N( Ro R ( Rp R. Rq ( R1 ( s0 /usr/local/lib/python2.7/multiprocessing/pool.pys <genexpr> s c s s" | ] } | D] } | Vq
q d S( N( ( Rp Rr Rs ( ( s0 /usr/local/lib/python2.7/multiprocessing/pool.pys <genexpr>! s N( R= R< R t IMapUnorderedIteratorR; R: R Ru Rv R Rw ( R R/ Rm Rn Rx ( ( R/ R1 s0 /usr/local/lib/python2.7/multiprocessing/pool.pyt imap_unordered s c C sV | j t k s t t | j | } | j j | j d | | | f g d f | S( s>
Asynchronous equivalent of `apply()` builtin
N( R= R< R t ApplyResultR; R: R Ro R ( R R/ R R0 t callbackR1 ( ( s0 /usr/local/lib/python2.7/multiprocessing/pool.pyRj # s +c s | j t k s t t | d s3 t | } n | d k r} t t | t | j d \ } } | r} | d 7} q} n t | d k r d } n t j
| | | } t | j | t | | | j
j f d t | D d f S( s<
Asynchronous equivalent of `map()` builtin
t __len__i i i c 3 s0 | ]& \ } } j | t | f i f Vq d S( N( Ro R ( Rp R. Rq ( R1 ( s0 /usr/local/lib/python2.7/multiprocessing/pool.pys <genexpr>= s N( R= R< R R t listR t divmodR[ RE R Rw t MapResultR; R: R Ru ( R R/ Rm Rn R} t extraRx ( ( R1 s0 /usr/local/lib/python2.7/multiprocessing/pool.pyRl , s ( c C so t j } xB | j t k s6 | j rP | j t k rP | j t j d q W| j j
d t d d S( Ng?s worker handler exiting(
RG t current_threadR= R< R; t TERMINATERe t timet sleepR: R R R ( t poolt thread( ( s0 /usr/local/lib/python2.7/multiprocessing/pool.pyRI A s *
c C s
t j } xt | j d D]q\ } } d } d } zGy x t | D] \ } } | j rm t d Pn y | | WqJ t k
r }
| d \ } } y | | j | t
|
f Wq t k
r q XqJ XqJ W| r t d | | d n w PWn} t k
r|}
| r | d n d \ } } | | k rX| | j | d t
|
f n | r}t d | | d q}n XWd d } } } Xq Wt d y@ t d | j d t d x | D] } | d qWWn t
k
rt d
n Xt d d S(
Nis' task handler found thread._state != RUNi s doing set_length()i i s task handler got sentinels/ task handler sending sentinel to result handlers( task handler sending sentinel to workerss/ task handler got IOError when sending sentinelss task handler exiting( i i ( RG R t iterR R Ru R= R R$ t _setR% t KeyErrorR R" ( t taskqueueR R' R t cacheR t taskseqt
set_lengthR, R. R2 R- t indt ext p( ( s0 /usr/local/lib/python2.7/multiprocessing/pool.pyRM N sR
!
c C s t j } x y
| } Wn" t t f k
r@ t d d SX| j rm | j t k s_ t t d Pn | d k r t d Pn | \ } } } y | | j | | Wn t
k
r n Xd } } } q Wx | r| j t k ry
| } Wn" t t f k
r t d d SX| d k r:t d q n | \ } } } y | | j | | Wn t
k
rtn Xd } } } q Wt | d rt d y5 x. t d D] } | j
j sPn | qWWqt t f k
rqXn t d t | | j d S( Ns. result handler got EOFError/IOError -- exitings, result handler found thread._state=TERMINATEs result handler got sentinels& result handler ignoring extra sentinelR s" ensuring that outqueue is not fulli
s7 result handler exiting: len(cache)=%s, thread._state=%s( RG R R" R! R R= R R R R R R RZ R t pollR[ ( R' R R R R, R- R. t obj( ( s0 /usr/local/lib/python2.7/multiprocessing/pool.pyRQ s\
c c sD t | } x1 t t j | | } | s1 d S| | f Vq Wd S( N( R t tuplet itertoolst islice( R/ t itt sizeRq ( ( s0 /usr/local/lib/python2.7/multiprocessing/pool.pyRw s c C s t d d S( Ns: pool objects cannot be passed between processes or pickled( RA ( R ( ( s0 /usr/local/lib/python2.7/multiprocessing/pool.pyt
__reduce__ s c C s5 t d | j t k r1 t | _ t | j _ n d S( Ns closing pool( R R= R< t CLOSERJ ( R ( ( s0 /usr/local/lib/python2.7/multiprocessing/pool.pyR s
c C s- t d t | _ t | j _ | j d S( Ns terminating pool( R R R= RJ RV ( R ( ( s0 /usr/local/lib/python2.7/multiprocessing/pool.pyt terminate s
c C sn t d | j t t f k s% t | j j | j j | j j x | j D] } | j qV Wd S( Ns joining pool(
R R= R R R RJ R] RP RS RE ( R R ( ( s0 /usr/local/lib/python2.7/multiprocessing/pool.pyR] s
c C sW t d | j j x9 | j rR | j j rR | j j t j d q Wd S( Ns7 removing tasks from inqueue until task handler finishedi ( R t _rlockt acquiret is_aliveR R Ri R R ( R& t task_handlerR ( ( s0 /usr/local/lib/python2.7/multiprocessing/pool.pyt _help_stuff_finish s
c
C s t d t | _ t | _ t d | j | | t | | j sc t | d k sc t t | _ | j d t d t j
| k r | j d n | r t | d d r t d x- | D]" } | j
d k r | j q q Wn t d t j
| k r$| j d n t d t j
| k rP| j d n | rt | d d rt d
x; | D]0 } | j rzt d | j | j qzqzWn d S( Ns finalizing pools& helping task handler/workers to finishi s joining worker handlerg}Ô%ITR s terminating workerss joining task handlers joining result handlers joining pool workerss cleaning up worker %d( R R R= R R[ R R R R RG R R] R R\ R t pid(
t clsR R& R' R t worker_handlerR t result_handlerR R ( ( s0 /usr/local/lib/python2.7/multiprocessing/pool.pyRT s8
$
N( ( ( ( R R R R R R R_ RF Re R8 Rk R Ry R{ Rj Rl t staticmethodRI RM RQ Rw R R R R] R t classmethodRT ( ( ( s0 /usr/local/lib/python2.7/multiprocessing/pool.pyR s0 :
4< R| c B sD e Z d Z d Z d Z d d Z d d Z d Z RS( c C sS t j t j | _ t j | _ | | _ t | _ | | _
| | | j <d S( N( RG t Conditiont Lockt _condt job_countert nextRo R; R% t _readyt _callback( R R R} ( ( s0 /usr/local/lib/python2.7/multiprocessing/pool.pyR s c C s | j S( N( R ( R ( ( s0 /usr/local/lib/python2.7/multiprocessing/pool.pyt ready& s c C s | j s t | j S( N( R R t _success( R ( ( s0 /usr/local/lib/python2.7/multiprocessing/pool.pyt
successful) s c C sB | j j z | j s, | j j | n Wd | j j Xd S( N( R R R t waitt release( R t timeout( ( s0 /usr/local/lib/python2.7/multiprocessing/pool.pyR - s
c C s< | j | | j s t n | j r/ | j S| j d S( N( R R R R t _value( R R ( ( s0 /usr/local/lib/python2.7/multiprocessing/pool.pyR 5 s
c C s | \ | _ | _ | j r7 | j r7 | j | j n | j j z t | _ | j j Wd | j j X| j | j
=d S( N( R R R R R R# R t notifyR R; Ro ( R R. R ( ( s0 /usr/local/lib/python2.7/multiprocessing/pool.pyR > s
N( R R R R R R R R R ( ( ( s0 /usr/local/lib/python2.7/multiprocessing/pool.pyR| s R c B s e Z d Z d Z RS( c C s t j | | | t | _ d g | | _ | | _ | d k r` d | _ t | _ | | j =n | | t
| | | _ d S( Ni ( R| R R# R R R t
_chunksizet _number_leftR Ro t bool( R R Rn t lengthR} ( ( s0 /usr/local/lib/python2.7/multiprocessing/pool.pyR R s
c C s | \ } } | r | | j | | j | d | j +| j d 8_ | j d k r| j rn | j | j n | j | j =| j j z t | _ | j j
Wd | j j XqnW t | _
| | _ | j | j =| j j z t | _ | j j
Wd | j j Xd S( Ni i ( R R R R R; Ro R R R# R R R R% R ( R R. t success_resultt successR1 ( ( s0 /usr/local/lib/python2.7/multiprocessing/pool.pyR ^ s* "
( R R R R ( ( ( s0 /usr/local/lib/python2.7/multiprocessing/pool.pyR P s Rt c B s> e Z d Z d Z d d Z e Z d Z d Z RS( c C sk t j t j | _ t j | _ | | _ t j | _
d | _ d | _
i | _ | | | j <d S( Ni ( RG R R R R R Ro R; t collectionst dequet _itemst _indexR t _lengtht _unsorted( R R ( ( s0 /usr/local/lib/python2.7/multiprocessing/pool.pyR s c C s | S( N( ( R ( ( s0 /usr/local/lib/python2.7/multiprocessing/pool.pyt __iter__ s c C s | j j z y | j j } Wn t k
r | j | j k rN t n | j j | y | j j } Wq t k
r | j | j k r t n t q Xn XWd | j j
X| \ } } | r | S| d S( N( R R R t popleftt
IndexErrorR R t
StopIterationR R R ( R R Rs R R ( ( s0 /usr/local/lib/python2.7/multiprocessing/pool.pyR s&
c C s | j j z | j | k r | j j | | j d 7_ xJ | j | j k r | j j | j } | j j | | j d 7_ qA W| j j n
| | j | <| j | j k r | j | j
=n Wd | j j Xd S( Ni ( R R R R Ra R t popR R R; Ro R ( R R. R ( ( s0 /usr/local/lib/python2.7/multiprocessing/pool.pyR s
c C s^ | j j z< | | _ | j | j k rH | j j | j | j =n Wd | j j Xd S( N( R R R R R R; Ro R ( R R ( ( s0 /usr/local/lib/python2.7/multiprocessing/pool.pyRv s
N( R R R R R R t __next__R Rv ( ( ( s0 /usr/local/lib/python2.7/multiprocessing/pool.pyRt } s
Rz c B s e Z d Z RS( c C st | j j zR | j j | | j d 7_ | j j | j | j k r^ | j | j =n Wd | j j Xd S( Ni (
R R R Ra R R R R; Ro R ( R R. R ( ( s0 /usr/local/lib/python2.7/multiprocessing/pool.pyR s
( R R R ( ( ( s0 /usr/local/lib/python2.7/multiprocessing/pool.pyRz s t
ThreadPoolc B sB e Z d d l m Z d d d d Z d Z e d Z RS( i ( R c C s t j | | | | d S( N( R R ( R RW R( R) ( ( s0 /usr/local/lib/python2.7/multiprocessing/pool.pyR s c C s@ t j | _ t j | _ | j j | _ | j j | _ d S( N( R9 RU RO R RN R RR ( R ( ( s0 /usr/local/lib/python2.7/multiprocessing/pool.pyR8 s c C sW | j j z5 | j j | j j d g | | j j Wd | j j Xd S( N( t not_emptyR t queuet cleart extendR t
notify_allR ( R& R R ( ( s0 /usr/local/lib/python2.7/multiprocessing/pool.pyR s
N( ( R R t dummyR R R R8 R R ( ( ( s0 /usr/local/lib/python2.7/multiprocessing/pool.pyR s ( t __all__RG R9 R R R t multiprocessingR R R t multiprocessing.utilR R R< R R t countR R R$ R R R4 t objectR R| t AsyncResultR Rt Rz R ( ( ( s0 /usr/local/lib/python2.7/multiprocessing/pool.pyt <module># s, , .-I
Mini Shell