POST request does not have accesible body when Content-Type is not specified #2376
Description
I am coming here from rack/rackup#35 and the issue is that with Rack 3+, I started to observe issues like this in em-http-request:
1) EventMachine::HttpRequest should perform successful POST
Failure/Error: http.response.should match(/data/)
expected "" to match /data/
Diff:
@@ -1 +1 @@
-/data/
+""
# ./spec/client_spec.rb:186:in 'block (4 levels) in <top (required)>'
# /usr/share/gems/gems/eventmachine-1.2.7/lib/em/deferrable.rb:151:in 'EventMachine::Deferrable#set_deferred_status'
# /usr/share/gems/gems/eventmachine-1.2.7/lib/em/deferrable.rb:191:in 'EventMachine::Deferrable#succeed'
# ./lib/em-http/client.rb:113:in 'EventMachine::HttpClient#unbind'
# ./lib/em-http/client.rb:71:in 'EventMachine::HttpClient#on_request_complete'
# ./lib/em-http/http_connection.rb:205:in 'block in EventMachine::HttpConnection#post_init'
# ./lib/em-http/http_connection.rb:220:in 'HTTP::Parser#<<'
# ./lib/em-http/http_connection.rb:220:in 'EventMachine::HttpConnection#receive_data'
# ./lib/em-http/http_connection.rb:26:in 'EventMachine::HttpStubConnection#receive_data'
# /usr/share/gems/gems/eventmachine-1.2.7/lib/eventmachine.rb:195:in 'EventMachine.run_machine'
# /usr/share/gems/gems/eventmachine-1.2.7/lib/eventmachine.rb:195:in 'EventMachine.run'
# ./spec/client_spec.rb:180:in 'block (2 levels) in <top (required)>'
2) EventMachine::HttpRequest should escape body on POST
Failure/Error: http.response.should == "stuff=string%26string"
expected: "stuff=string%26string"
got: "" (using ==)
# ./spec/client_spec.rb:212:in 'block (4 levels) in <top (required)>'
# /usr/share/gems/gems/eventmachine-1.2.7/lib/em/deferrable.rb:151:in 'EventMachine::Deferrable#set_deferred_status'
# /usr/share/gems/gems/eventmachine-1.2.7/lib/em/deferrable.rb:191:in 'EventMachine::Deferrable#succeed'
# ./lib/em-http/client.rb:113:in 'EventMachine::HttpClient#unbind'
# ./lib/em-http/client.rb:71:in 'EventMachine::HttpClient#on_request_complete'
# ./lib/em-http/http_connection.rb:205:in 'block in EventMachine::HttpConnection#post_init'
# ./lib/em-http/http_connection.rb:220:in 'HTTP::Parser#<<'
# ./lib/em-http/http_connection.rb:220:in 'EventMachine::HttpConnection#receive_data'
# ./lib/em-http/http_connection.rb:26:in 'EventMachine::HttpStubConnection#receive_data'
# /usr/share/gems/gems/eventmachine-1.2.7/lib/eventmachine.rb:195:in 'EventMachine.run_machine'
# /usr/share/gems/gems/eventmachine-1.2.7/lib/eventmachine.rb:195:in 'EventMachine.run'
# ./spec/client_spec.rb:206:in 'block (2 levels) in <top (required)>'
3) EventMachine::HttpRequest should perform successful POST with Ruby Hash/Array as params
Failure/Error: http.response.should match(/key1=1&key2\[0\]=2&key2\[1\]=3/)
expected "" to match /key1=1&key2\[0\]=2&key2\[1\]=3/
Diff:
@@ -1 +1 @@
-/key1=1&key2\[0\]=2&key2\[1\]=3/
+""
# ./spec/client_spec.rb:226:in 'block (4 levels) in <top (required)>'
# /usr/share/gems/gems/eventmachine-1.2.7/lib/em/deferrable.rb:151:in 'EventMachine::Deferrable#set_deferred_status'
# /usr/share/gems/gems/eventmachine-1.2.7/lib/em/deferrable.rb:191:in 'EventMachine::Deferrable#succeed'
# ./lib/em-http/client.rb:113:in 'EventMachine::HttpClient#unbind'
# ./lib/em-http/client.rb:71:in 'EventMachine::HttpClient#on_request_complete'
# ./lib/em-http/http_connection.rb:205:in 'block in EventMachine::HttpConnection#post_init'
# ./lib/em-http/http_connection.rb:220:in 'HTTP::Parser#<<'
# ./lib/em-http/http_connection.rb:220:in 'EventMachine::HttpConnection#receive_data'
# ./lib/em-http/http_connection.rb:26:in 'EventMachine::HttpStubConnection#receive_data'
# /usr/share/gems/gems/eventmachine-1.2.7/lib/eventmachine.rb:195:in 'EventMachine.run_machine'
# /usr/share/gems/gems/eventmachine-1.2.7/lib/eventmachine.rb:195:in 'EventMachine.run'
# ./spec/client_spec.rb:219:in 'block (2 levels) in <top (required)>'
4) EventMachine::HttpRequest should stream a file off disk
Failure/Error: http.response.should match('google')
expected "" to match "google"
# ./spec/client_spec.rb:800:in 'block (4 levels) in <top (required)>'
# /usr/share/gems/gems/eventmachine-1.2.7/lib/em/deferrable.rb:151:in 'EventMachine::Deferrable#set_deferred_status'
# /usr/share/gems/gems/eventmachine-1.2.7/lib/em/deferrable.rb:191:in 'EventMachine::Deferrable#succeed'
# ./lib/em-http/client.rb:113:in 'EventMachine::HttpClient#unbind'
# ./lib/em-http/client.rb:71:in 'EventMachine::HttpClient#on_request_complete'
# ./lib/em-http/http_connection.rb:205:in 'block in EventMachine::HttpConnection#post_init'
# ./lib/em-http/http_connection.rb:220:in 'HTTP::Parser#<<'
# ./lib/em-http/http_connection.rb:220:in 'EventMachine::HttpConnection#receive_data'
# ./lib/em-http/http_connection.rb:26:in 'EventMachine::HttpStubConnection#receive_data'
# /usr/share/gems/gems/eventmachine-1.2.7/lib/eventmachine.rb:195:in 'EventMachine.run_machine'
# /usr/share/gems/gems/eventmachine-1.2.7/lib/eventmachine.rb:195:in 'EventMachine.run'
# ./spec/client_spec.rb:795:in 'block (2 levels) in <top (required)>'
5) EventMachine::HttpRequest streams POST request from disk via Pathname
Failure/Error: http.response.should match('google')
expected "" to match "google"
# ./spec/client_spec.rb:811:in 'block (4 levels) in <top (required)>'
# /usr/share/gems/gems/eventmachine-1.2.7/lib/em/deferrable.rb:151:in 'EventMachine::Deferrable#set_deferred_status'
# /usr/share/gems/gems/eventmachine-1.2.7/lib/em/deferrable.rb:191:in 'EventMachine::Deferrable#succeed'
# ./lib/em-http/client.rb:113:in 'EventMachine::HttpClient#unbind'
# ./lib/em-http/client.rb:71:in 'EventMachine::HttpClient#on_request_complete'
# ./lib/em-http/http_connection.rb:205:in 'block in EventMachine::HttpConnection#post_init'
# ./lib/em-http/http_connection.rb:220:in 'HTTP::Parser#<<'
# ./lib/em-http/http_connection.rb:220:in 'EventMachine::HttpConnection#receive_data'
# ./lib/em-http/http_connection.rb:26:in 'EventMachine::HttpStubConnection#receive_data'
# /usr/share/gems/gems/eventmachine-1.2.7/lib/eventmachine.rb:195:in 'EventMachine.run_machine'
# /usr/share/gems/gems/eventmachine-1.2.7/lib/eventmachine.rb:195:in 'EventMachine.run'
# ./spec/client_spec.rb:807:in 'block (2 levels) in <top (required)>'
6) EventMachine::HttpRequest streams POST request from IO object
Failure/Error: http.response.should match('google')
expected "" to match "google"
# ./spec/client_spec.rb:822:in 'block (4 levels) in <top (required)>'
# /usr/share/gems/gems/eventmachine-1.2.7/lib/em/deferrable.rb:151:in 'EventMachine::Deferrable#set_deferred_status'
# /usr/share/gems/gems/eventmachine-1.2.7/lib/em/deferrable.rb:191:in 'EventMachine::Deferrable#succeed'
# ./lib/em-http/client.rb:113:in 'EventMachine::HttpClient#unbind'
# ./lib/em-http/client.rb:71:in 'EventMachine::HttpClient#on_request_complete'
# ./lib/em-http/http_connection.rb:205:in 'block in EventMachine::HttpConnection#post_init'
# ./lib/em-http/http_connection.rb:220:in 'HTTP::Parser#<<'
# ./lib/em-http/http_connection.rb:220:in 'EventMachine::HttpConnection#receive_data'
# ./lib/em-http/http_connection.rb:26:in 'EventMachine::HttpStubConnection#receive_data'
# /usr/share/gems/gems/eventmachine-1.2.7/lib/eventmachine.rb:195:in 'EventMachine.run_machine'
# /usr/share/gems/gems/eventmachine-1.2.7/lib/eventmachine.rb:195:in 'EventMachine.run'
# ./spec/client_spec.rb:818:in 'block (2 levels) in <top (required)>'
7) EventMachine::HttpRequest request should execute request middleware before dispatching request
Failure/Error: req.response.should match(/data modified/)
expected "" to match /data modified/
Diff:
@@ -1 +1 @@
-/data modified/
+""
# ./spec/middleware_spec.rb:110:in 'block (5 levels) in <top (required)>'
# /usr/share/gems/gems/eventmachine-1.2.7/lib/em/deferrable.rb:151:in 'EventMachine::Deferrable#set_deferred_status'
# /usr/share/gems/gems/eventmachine-1.2.7/lib/em/deferrable.rb:191:in 'EventMachine::Deferrable#succeed'
# ./lib/em-http/client.rb:113:in 'EventMachine::HttpClient#unbind'
# ./lib/em-http/client.rb:71:in 'EventMachine::HttpClient#on_request_complete'
# ./lib/em-http/http_connection.rb:205:in 'block in EventMachine::HttpConnection#post_init'
# ./lib/em-http/http_connection.rb:220:in 'HTTP::Parser#<<'
# ./lib/em-http/http_connection.rb:220:in 'EventMachine::HttpConnection#receive_data'
# ./lib/em-http/http_connection.rb:26:in 'EventMachine::HttpStubConnection#receive_data'
# /usr/share/gems/gems/eventmachine-1.2.7/lib/eventmachine.rb:195:in 'EventMachine.run_machine'
# /usr/share/gems/gems/eventmachine-1.2.7/lib/eventmachine.rb:195:in 'EventMachine.run'
# ./spec/middleware_spec.rb:103:in 'block (3 levels) in <top (required)>'
8) EventMachine::HttpRequest jsonify should use middleware to JSON encode and JSON decode the body
Failure/Error: resp.response = MultiJson.load(resp.response)
MultiJson::ParseError:
JSON::ParserError
# /usr/share/gems/gems/multi_json-1.15.0/lib/multi_json/adapter.rb:20:in 'MultiJson::Adapter.load'
# /usr/share/gems/gems/multi_json-1.15.0/lib/multi_json.rb:122:in 'MultiJson#load'
# ./spec/middleware_spec.rb:124:in 'JSONify#response'
# /usr/share/gems/gems/eventmachine-1.2.7/lib/em/deferrable.rb:151:in 'EventMachine::Deferrable#set_deferred_status'
# /usr/share/gems/gems/eventmachine-1.2.7/lib/em/deferrable.rb:191:in 'EventMachine::Deferrable#succeed'
# ./lib/em-http/client.rb:113:in 'EventMachine::HttpClient#unbind'
# ./lib/em-http/client.rb:71:in 'EventMachine::HttpClient#on_request_complete'
# ./lib/em-http/http_connection.rb:205:in 'block in EventMachine::HttpConnection#post_init'
# ./lib/em-http/http_connection.rb:220:in 'HTTP::Parser#<<'
# ./lib/em-http/http_connection.rb:220:in 'EventMachine::HttpConnection#receive_data'
# ./lib/em-http/http_connection.rb:26:in 'EventMachine::HttpStubConnection#receive_data'
# /usr/share/gems/gems/eventmachine-1.2.7/lib/eventmachine.rb:195:in 'EventMachine.run_machine'
# /usr/share/gems/gems/eventmachine-1.2.7/lib/eventmachine.rb:195:in 'EventMachine.run'
# ./spec/middleware_spec.rb:129:in 'block (3 levels) in <top (required)>'
# ------------------
# --- Caused by: ---
# JSON::ParserError:
# JSON::ParserError
# /usr/share/gems/gems/multi_json-1.15.0/lib/multi_json/adapter.rb:20:in 'MultiJson::Adapter.load'
The analysis here helped me to identify and understand the root cause. The issue is related to removal of rewind. The change was documented here and it says:
Previously
.rewindwas called after consuming form and multipart data. Use
Rack::RewindableInput::Middlewareto make the body rewindable, and call
.rewindexplicitly to match this behavior.
However, it does not imply (to me at least) that Rack would "consume" the body when Content-Type is not specified. An excerpt from the RFC9110
A sender <...> SHOULD generate a Content-Type header field <...>
If a Content-Type header field is not present, the recipient MAY
either assume a media type of "application/octet-stream"
or examine the data to determine its type.'
This was not a problem previously in Rack 2.2, because the body was rewound, which is not the case for Rack 3+ anymore.