Pub / Sub Subscriber: CPU Usage eventually spikes to 100% #4600
Description
This is a distillation of this report from many other issues. Large thanks to @anorth2 and @dmontag for reporting and helping refine the issue.
Issues that have been partially resolved (except for the CPU spike) are being collapsed into this one:
- PubSub: Subscriber Client stops acking messages after several seconds #4274
- PubSub with Python 3.5 spawns threads that don't die, eventually bringing the process to consume all CPU available #4563
- Python PubSub consumer high cpu usage after approx 1 hour #3965
- (more to come?)
Core issue: there is a spinlock bug in gRPC (present in all recent versions 1.6.x, 1.7.x, 1.8.1). A "bandaid" fix exists but has not been merged (as of noon Pacific on December 15, 2017).
Update: The fix was rolled back, but will be rolled forward (grpc/grpc#13918).
Potential workaround: compile grpcio from source (e.g.) while including the bandaid fix or just use the 64-bit manylinux wheel that I already created (I may be open to creating Mac OS X wheels, not sure about Windows)